Turn off color for downstairs dump Creating region directory at: /var/tmp/test_repair Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/test_repair/8810 in 0.29817548 Downstairs region 1 created at /var/tmp/test_repair/8820 in 0.04777248 Downstairs region 2 created at /var/tmp/test_repair/8830 in 0.04818753 Region set with 3 regions was created /var/tmp/bins/crutest with -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 1184 1185 1186 Dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.064728153Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.065732226Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.065883488Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188} {"msg":"Crucible 87a8e332-1d1a-405b-94ae-d54dda238ce3 has session id: 71c25937-e690-4d97-856e-1605e6bd9a09","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.065909707Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Upstairs opts: Upstairs UUID: 87a8e332-1d1a-405b-94ae-d54dda238ce3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.066110828Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Crucible stats registered with UUID: 87a8e332-1d1a-405b-94ae-d54dda238ce3","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.066137077Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.066955698Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188} {"msg":"87a8e332-1d1a-405b-94ae-d54dda238ce3 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.066991247Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067011146Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067032155Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067052064Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067072283Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067092772Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067118511Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067220156Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067255764Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.06734588Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:26:51.067502713Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:26:51.067538541Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:26:51.06756624Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:26:51.067588609Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:26:51.067615168Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:26:51.067641706Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:26:51.067668075Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067688294Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067712073Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:26:51.067746642Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.06777238Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067794539Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:26:51.067818908Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067838677Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067859356Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067881115Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067903634Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:26:51.067926313Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.067963235Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"ds_connection connected from Ok(127.0.0.1:62082)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.068182314Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.069694634Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.069727392Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"ds_connection connected from Ok(127.0.0.1:40227)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.069870795Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"ds_connection connected from Ok(127.0.0.1:45142)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.069912093Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"io task","client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.070220099Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.070260487Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.070930566Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.070969494Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.071022382Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.07105776Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.071075459Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.071615654Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.071658912Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.071730569Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.071770947Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.071800985Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072205037Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072231495Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072246385Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072263084Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072280763Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072379298Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"downstairs","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072399268Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"downstairs","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072413837Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"downstairs","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072427526Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"downstairs","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072441075Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"downstairs","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072455275Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"downstairs","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072468784Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"downstairs","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072482214Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"downstairs","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072496123Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"downstairs","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072509092Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"downstairs","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072522412Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"downstairs","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072535451Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"downstairs","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} Wait for a query_work_queue command to finish before sending IO {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.07255186Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"":"downstairs","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072568829Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072581239Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"0","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072595058Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"1","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072608558Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"client":"2","":"downstairs","session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072622347Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072634877Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"87a8e332-1d1a-405b-94ae-d54dda238ce3 is now active with session: 71c25937-e690-4d97-856e-1605e6bd9a09","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072647716Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072659565Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188,"session_id":"71c25937-e690-4d97-856e-1605e6bd9a09"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:01.072671675Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Fill test Read and Verify all blocks (0..600 range:false) Wrote out file "/tmp/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 5 Acked 1004 Read 88 Done Done Done false 6 Acked 1005 Read 256 Done Done Done false 7 Acked 1006 Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1003 1003 1003 Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 7 6 5 4 3 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 8 7 6 5 4 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:27:05.164070038Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1188} Begin loop 1 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.194312617Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.1946862Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.194727198Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190} {"msg":"Crucible 20a6a60a-7519-4b1a-a252-77a031d442b7 has session id: 7a74806d-82ea-421d-adc0-a75239dfa1b8","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.195073122Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"Upstairs opts: Upstairs UUID: 20a6a60a-7519-4b1a-a252-77a031d442b7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.19510477Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"Crucible stats registered with UUID: 20a6a60a-7519-4b1a-a252-77a031d442b7","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.195119979Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.195413296Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190} {"msg":"20a6a60a-7519-4b1a-a252-77a031d442b7 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.195461854Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.195477563Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"0","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.195496552Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"0","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.195510281Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.195523961Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.19553907Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"2","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.195562339Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"2","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.195586718Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"io task","client":"0","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.195608437Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"io task","client":"2","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.195626026Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"io task","client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"ds_connection connected from Ok(127.0.0.1:40900)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.195908923Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"io task","client":"2","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"ds_connection connected from Ok(127.0.0.1:52521)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.196203379Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"io task","client":"0","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:27:05.196289035Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"io task","client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:27:05.196317523Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"io task","client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:27:05.196346372Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.19639525Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.196419689Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.196446228Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"io task","client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.196729604Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"2","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.196756573Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"2","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.196771642Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"0","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.196788741Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"0","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.197326756Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"0","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.19746773Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"2","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.197813874Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"0","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.197837813Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"0","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.197851942Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.197927659Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"2","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.197950517Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"2","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:05.197967107Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.197446413Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"io task","client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"ds_connection connected from Ok(127.0.0.1:46647)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.197667113Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"io task","client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.198757792Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.19880363Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.199289827Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.199825802Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.199848151Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.199865681Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"downstairs","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.19988806Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"downstairs","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.199906159Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"downstairs","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.199923218Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"downstairs","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.199939797Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"downstairs","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.199957206Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"downstairs","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.199973896Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"downstairs","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.199990645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"downstairs","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.200008424Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"downstairs","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.200024683Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"downstairs","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"Generation requested: 2 >= found:2Wait for a query_work_queue command to finish before sending IO ","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.200044972Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"downstairs","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.200061291Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"downstairs","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.200077351Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"":"downstairs","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.20009366Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.200108329Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"0","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.200124598Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"1","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.200143177Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"client":"2","":"downstairs","session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.200160087Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.200175216Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"20a6a60a-7519-4b1a-a252-77a031d442b7 is now active with session: 7a74806d-82ea-421d-adc0-a75239dfa1b8","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.200191155Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.200205905Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190,"session_id":"7a74806d-82ea-421d-adc0-a75239dfa1b8"} {"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:15.200220834Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 252 len 20480 data: 2 2 2 2 2 02/30 Write block 392 len 28672 data: 2 2 2 2 2 2 2 03/30 Read block 578 len 20480 04/30 Read block 39 len 36864 05/30 Write block 196 len 12288 data: 2 2 2 06/30 Read block 180 len 16384 07/30 Write block 586 len 12288 data: 2 2 2 08/30 Write block 430 len 16384 data: 2 2 2 2 09/30 Write block 213 len 40960 data: 2 2 2 2 2 2 2 2 2 2 10/30 Read block 337 len 24576 11/30 Write block 57 len 16384 data: 2 2 2 2 12/30 Read block 385 len 4096 13/30 Write block 29 len 4096 data: 2 14/30 Read block 344 len 20480 15/30 Write block 125 len 24576 data: 2 2 2 2 2 2 16/30 Write block 560 len 28672 data: 2 2 2 2 2 2 2 17/30 Flush 18/30 Write block 202 len 40960 data: 2 2 2 2 2 2 2 2 2 2 19/30 Read block 310 len 40960 20/30 Write block 353 len 40960 data: 2 2 2 2 2 2 2 2 2 2 21/30 Write block 165 len 36864 data: 2 2 2 2 2 2 2 2 2 22/30 Read block 382 len 28672 23/30 Write block 553 len 28672 data: 2 2 2 2 2 2 2 24/30 Read block 423 len 8192 25/30 Read block 146 len 8192 26/30 Flush 27/30 Write block 541 len 4096 data: 2 28/30 Write block 7 len 20480 data: 2 2 2 2 2 29/30 Read block 372 len 36864 30/30 Read block 503 len 16384 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 1 Done Done Done false 28 Acked 1027 Write 5 Done Done Done false 29 Acked 1028 Read 9 Done Done Done false 30 Acked 1029 Read 4 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:27:15.533271133Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1190} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 T T T 1 020-039 2 2 2 2 2 2 F F F 2 040-059 2 2 2 2 2 2 F F F 3 060-079 2 2 2 2 2 2 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 2 2 2 3 3 3 F F F 9 180-199 2 2 2 2 2 2 F F F 10 200-219 2 2 2 3 3 3 F F F 11 220-239 2 2 2 2 2 2 F F F 12 240-259 2 2 2 2 2 2 F F F 13 260-279 1 1 1 1 1 1 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 1 1 1 1 1 1 F F F 17 340-359 2 2 2 3 3 3 F F F 18 360-379 2 2 2 3 3 3 F F F 19 380-399 2 2 2 2 2 2 F F F 20 400-419 1 1 1 1 1 1 F F F 21 420-439 2 2 2 2 2 2 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 2 2 2 3 3 3 T T T 28 560-579 2 2 2 2 2 2 F F F 29 580-599 2 2 2 2 2 2 F F F Max gen: 2, Max flush: 3 A Difference in extent metadata is expected here On loop 1 Verifying data now No change after verify Loop: 1 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 3 3 3 4 4 4 F F F 1 020-039 2 2 2 2 2 2 F F F 2 040-059 2 2 2 2 2 2 F F F 3 060-079 2 2 2 2 2 2 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 2 2 2 3 3 3 F F F 9 180-199 2 2 2 2 2 2 F F F 10 200-219 2 2 2 3 3 3 F F F 11 220-239 2 2 2 2 2 2 F F F 12 240-259 2 2 2 2 2 2 F F F 13 260-279 1 1 1 1 1 1 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 1 1 1 1 1 1 F F F 17 340-359 2 2 2 3 3 3 F F F 18 360-379 2 2 2 3 3 3 F F F 19 380-399 2 2 2 2 2 2 F F F 20 400-419 1 1 1 1 1 1 F F F 21 420-439 2 2 2 2 2 2 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 3 3 3 4 4 4 F F F 28 560-579 2 2 2 2 2 2 F F F 29 580-599 2 2 2 2 2 2 F F F Max gen: 3, Max flush: 4 Begin loop 2 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.241521586Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.242020033Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.242055731Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198} {"msg":"Crucible f7570730-fe86-4d18-af29-48e30109a425 has session id: b5f33824-2571-4840-bea4-a64ce26bcc6d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.242161066Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"Upstairs opts: Upstairs UUID: f7570730-fe86-4d18-af29-48e30109a425, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.242239573Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"Crucible stats registered with UUID: f7570730-fe86-4d18-af29-48e30109a425","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.242275451Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:4","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.242533479Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198} {"msg":"f7570730-fe86-4d18-af29-48e30109a425 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.242625685Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.242650204Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.242670492Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.242684512Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"1","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.242698111Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"1","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.242711571Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"2","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.24272536Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"2","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.242740909Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.242754669Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.242768568Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:27:30.242954879Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:27:30.242994157Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"ds_connection connected from Ok(127.0.0.1:63416)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.243018946Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:27:30.243046335Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.243069244Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.243094023Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"ds_connection connected from Ok(127.0.0.1:45468)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.243117762Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.243191678Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.243711624Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"2","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.243749222Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"2","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.243779101Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"1","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.24380747Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"1","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.244426461Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"1","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.244453909Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"2","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.244942946Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"2","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.244966695Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"2","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.244980805Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.245045972Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"1","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.245064641Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"1","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:30.24507792Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.244065951Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"ds_connection connected from Ok(127.0.0.1:39480)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.244345528Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.245605019Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.245650627Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.246137664Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.246730857Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.246758726Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"[0]R flush_numbers[0..12]: [4, 2, 2, 2, 1, 1, 2, 1, 3, 2, 3, 2]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.246777614Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"downstairs","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"[0]R generation[0..12]: [3, 2, 2, 2, 1, 1, 2, 1, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.246795174Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"downstairs","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.246819693Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"downstairs","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"[1]R flush_numbers[0..12]: [4, 2, 2, 2, 1, 1, 2, 1, 3, 2, 3, 2]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.246837842Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"downstairs","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"[1]R generation[0..12]: [3, 2, 2, 2, 1, 1, 2, 1, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.246855091Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"downstairs","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.24687252Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"downstairs","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"[2]R flush_numbers[0..12]: [4, 2, 2, 2, 1, 1, 2, 1, 3, 2, 3, 2]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.246889539Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"downstairs","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"[2]R generation[0..12]: [3, 2, 2, 2, 1, 1, 2, 1, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.246906088Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"downstairs","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.246923588Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"Max found gen is 4","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.246943247Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"downstairs","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"Generation requested: 4 >= found:4","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.246960276Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"downstairs","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"Next flush: 5","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.246976275Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"downstairs","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.246992235Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"":"downstairs","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.247008994Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.247024063Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"0","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.247042232Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"1","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.247058852Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"client":"2","":"downstairs","session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.24707858Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.24709416Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"f7570730-fe86-4d18-af29-48e30109a425 is now active with session: b5f33824-2571-4840-bea4-a64ce26bcc6d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.247109979Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.247124758Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198,"session_id":"b5f33824-2571-4840-bea4-a64ce26bcc6d"} {"msg":"The guest has finished waiting for activation with:4","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:40.247139998Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 179 len 8192 data: 2 2 03/30 Write block 254 len 40960 data: 3 3 3 2 2 2 2 2 2 2 04/30 Flush 05/30 Write block 6 len 28672 data: 2 3 3 3 3 3 2 06/30 Write block 71 len 12288 data: 2 2 2 07/30 Read block 592 len 28672 08/30 Read block 261 len 12288 09/30 Write block 571 len 24576 data: 2 2 2 2 2 2 10/30 Read block 167 len 28672 11/30 Write block 334 len 20480 data: 2 2 2 2 2 12/30 Read block 376 len 4096 13/30 Write block 357 len 16384 data: 3 3 3 3 14/30 Write block 403 len 24576 data: 2 2 2 2 2 2 15/30 Read block 5 len 36864 16/30 Write block 364 len 40960 data: 2 2 2 2 2 2 2 2 2 2 17/30 Read block 520 len 12288 18/30 Read block 291 len 24576 19/30 Read block 344 len 20480 20/30 Read block 103 len 16384 21/30 Read block 444 len 4096 22/30 Write block 285 len 40960 data: 2 2 2 2 2 2 2 2 2 2 23/30 Read block 382 len 40960 24/30 Write block 502 len 36864 data: 2 2 2 2 2 2 2 2 2 25/30 Read block 489 len 20480 26/30 Write block 460 len 12288 data: 2 2 2 27/30 Read block 526 len 20480 28/30 Read block 253 len 16384 29/30 Read block 155 len 4096 30/30 Read block 335 len 32768 ---------------------------------------------------------------- Crucible gen:4 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 20 Acked 1019 Read 5 Done Done Done false 21 Acked 1020 Read 4 Done Done Done false 22 Acked 1021 Read 1 Done Done Done false 23 Acked 1022 Write 10 Done Done Done false 24 Acked 1023 Read 10 Done Done Done false 25 Acked 1024 Write 9 Done Done Done false 26 Acked 1025 Read 5 Done Done Done false 27 Acked 1026 Write 3 Done Done Done false 28 Acked 1027 Read 5 Done Done Done false 29 Acked 1028 Read 4 Done Done Done false 30 Acked 1029 Read 1 Done Done Done false 31 Acked 1030 Read 8 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 0 1 Done 11 12 12 35 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1018 1018 Downstairs last five completed: 1018 1017 1016 1015 1014 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:27:40.342737317Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1198} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 7 7 7 F F F 1 020-039 2 2 2 2 2 2 F F F 2 040-059 2 2 2 2 2 2 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 4 4 4 6 6 6 F F F 9 180-199 4 4 4 6 6 6 F F F 10 200-219 2 2 2 3 3 3 F F F 11 220-239 2 2 2 2 2 2 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 4 4 4 6 6 6 F F F 14 280-299 1 1 1 1 1 1 T T T 15 300-319 1 1 1 1 1 1 F F F 16 320-339 4 4 4 7 7 7 F F F 17 340-359 4 4 4 7 7 7 F F F 18 360-379 4 4 4 7 7 7 F F F 19 380-399 2 2 2 2 2 2 F F F 20 400-419 4 4 4 7 7 7 F F F 21 420-439 2 2 2 2 2 2 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 T T T 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 T T T 26 520-539 1 1 1 1 1 1 F F F 27 540-559 3 3 3 4 4 4 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 2 2 2 2 2 2 F F F Max gen: 4, Max flush: 7 A Difference in extent metadata is expected here On loop 2 Verifying data now No change after verify Loop: 2 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 7 7 7 F F F 1 020-039 2 2 2 2 2 2 F F F 2 040-059 2 2 2 2 2 2 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 4 4 4 6 6 6 F F F 9 180-199 4 4 4 6 6 6 F F F 10 200-219 2 2 2 3 3 3 F F F 11 220-239 2 2 2 2 2 2 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 4 4 4 6 6 6 F F F 14 280-299 5 5 5 8 8 8 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 4 4 4 7 7 7 F F F 17 340-359 4 4 4 7 7 7 F F F 18 360-379 4 4 4 7 7 7 F F F 19 380-399 2 2 2 2 2 2 F F F 20 400-419 4 4 4 7 7 7 F F F 21 420-439 2 2 2 2 2 2 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 5 5 5 8 8 8 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 5 5 5 8 8 8 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 3 3 3 4 4 4 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 2 2 2 2 2 2 F F F Max gen: 5, Max flush: 8 Begin loop 3 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.191250615Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.191763941Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.191790079Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206} {"msg":"Crucible 51f9704f-f57b-4276-a68d-bb7b2ac370bc has session id: 562497b6-a180-4b51-a6eb-58be5c30d507","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.191983601Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"Upstairs opts: Upstairs UUID: 51f9704f-f57b-4276-a68d-bb7b2ac370bc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.192030048Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"Crucible stats registered with UUID: 51f9704f-f57b-4276-a68d-bb7b2ac370bc","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.192056037Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:6","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.192305895Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206} {"msg":"51f9704f-f57b-4276-a68d-bb7b2ac370bc active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.192397491Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.19241609Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.192437289Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.192451089Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"1","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.192464858Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"1","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.192477918Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"2","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.192491517Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"2","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.192504686Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.192518576Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.192536045Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"ds_connection connected from Ok(127.0.0.1:41434)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.192711976Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"ds_connection connected from Ok(127.0.0.1:64196)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.192759674Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:27:55.192938436Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:27:55.192970615Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:27:55.192988554Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.193001273Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.193014302Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.193112948Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.193631614Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"1","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.193658672Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"1","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.193673592Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"2","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.193687111Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"2","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.194302812Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"2","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.194330081Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"1","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.194884125Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"2","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.194910604Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"2","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.194925303Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.194937743Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"1","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.194950762Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"1","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:27:55.194964322Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.194375781Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"ds_connection connected from Ok(127.0.0.1:34649)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.194787562Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.195692389Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.195724908Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.196217795Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.19675269Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.196780819Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"[0]R flush_numbers[0..12]: [7, 2, 2, 7, 1, 1, 2, 1, 6, 6, 3, 2]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.196798608Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"downstairs","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"[0]R generation[0..12]: [4, 2, 2, 4, 1, 1, 2, 1, 4, 4, 2, 2]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.196815797Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"downstairs","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.196840006Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"downstairs","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"[1]R flush_numbers[0..12]: [7, 2, 2, 7, 1, 1, 2, 1, 6, 6, 3, 2]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.196857455Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"downstairs","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"[1]R generation[0..12]: [4, 2, 2, 4, 1, 1, 2, 1, 4, 4, 2, 2]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.196874224Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"downstairs","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.196892244Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"downstairs","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"[2]R flush_numbers[0..12]: [7, 2, 2, 7, 1, 1, 2, 1, 6, 6, 3, 2]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.196909143Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"downstairs","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"[2]R generation[0..12]: [4, 2, 2, 4, 1, 1, 2, 1, 4, 4, 2, 2]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.196925922Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"downstairs","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.196946941Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"downstairs","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"Max found gen is 6","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.19696307Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"downstairs","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"Generation requested: 6 >= found:6","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.196979589Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"downstairs","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"Next flush: 9","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.196995659Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"downstairs","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.197011808Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"":"downstairs","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.197028287Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.197043426Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"0","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.197060006Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"1","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.197076565Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"client":"2","":"downstairs","session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.197095464Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.197110823Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"51f9704f-f57b-4276-a68d-bb7b2ac370bc is now active with session: 562497b6-a180-4b51-a6eb-58be5c30d507","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.197126822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.197141682Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206,"session_id":"562497b6-a180-4b51-a6eb-58be5c30d507"} {"msg":"The guest has finished waiting for activation with:6","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:05.197156741Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 274 len 8192 data: 2 2 02/30 Write block 518 len 28672 data: 2 2 2 2 2 2 2 03/30 Flush 04/30 Write block 216 len 28672 data: 3 3 3 3 3 3 3 05/30 Read block 465 len 24576 06/30 Read block 297 len 8192 07/30 Write block 23 len 28672 data: 2 2 2 2 2 2 3 08/30 Write block 221 len 20480 data: 4 4 2 2 2 09/30 Write block 435 len 32768 data: 2 2 2 2 2 2 2 2 10/30 Write block 325 len 4096 data: 2 11/30 Read block 130 len 16384 12/30 Flush 13/30 Write block 20 len 20480 data: 2 2 2 3 3 14/30 Write block 293 len 40960 data: 3 3 2 2 2 2 2 2 2 2 15/30 Read block 282 len 12288 16/30 Read block 47 len 36864 17/30 Read block 208 len 28672 18/30 Flush 19/30 Flush 20/30 Write block 367 len 8192 data: 3 3 21/30 Flush 22/30 Write block 389 len 36864 data: 2 2 2 3 3 3 3 3 3 23/30 Read block 290 len 24576 24/30 Write block 0 len 20480 data: 2 2 2 2 2 25/30 Read block 402 len 36864 26/30 Write block 170 len 16384 data: 3 3 3 3 27/30 Read block 218 len 40960 28/30 Read block 140 len 8192 29/30 Write block 392 len 32768 data: 4 4 4 4 4 4 3 2 30/30 Read block 383 len 28672 ---------------------------------------------------------------- Crucible gen:6 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 9 Done Done Done false 23 Acked 1022 Read 6 Done Done Done false 24 Acked 1023 Write 5 Done Done Done false 25 Acked 1024 Read 9 Done Done Done false 26 Acked 1025 Write 4 Done Done Done false 27 Acked 1026 Read 10 Done Done Done false 28 Acked 1027 Read 2 Done Done Done false 29 Acked 1028 Write 8 Done Done Done false 30 Acked 1029 Read 7 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 9 9 9 27 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:28:05.389938885Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1206} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 4 4 4 7 7 7 T T T 1 020-039 6 6 6 11 11 11 F F F 2 040-059 2 2 2 2 2 2 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 4 4 4 6 6 6 T T T 9 180-199 4 4 4 6 6 6 F F F 10 200-219 6 6 6 10 10 10 F F F 11 220-239 6 6 6 10 10 10 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 9 9 9 F F F 14 280-299 6 6 6 11 11 11 F F F 15 300-319 6 6 6 11 11 11 F F F 16 320-339 6 6 6 10 10 10 F F F 17 340-359 4 4 4 7 7 7 F F F 18 360-379 6 6 6 13 13 13 F F F 19 380-399 2 2 2 2 2 2 T T T 20 400-419 4 4 4 7 7 7 F F F 21 420-439 6 6 6 10 10 10 F F F 22 440-459 6 6 6 10 10 10 F F F 23 460-479 5 5 5 8 8 8 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 6 6 6 9 9 9 F F F 26 520-539 6 6 6 9 9 9 F F F 27 540-559 3 3 3 4 4 4 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 2 2 2 2 2 2 F F F Max gen: 6, Max flush: 13 A Difference in extent metadata is expected here On loop 3 Verifying data now No change after verify Loop: 3 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 7 7 7 14 14 14 F F F 1 020-039 6 6 6 11 11 11 F F F 2 040-059 2 2 2 2 2 2 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 7 7 7 14 14 14 F F F 9 180-199 4 4 4 6 6 6 F F F 10 200-219 6 6 6 10 10 10 F F F 11 220-239 6 6 6 10 10 10 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 9 9 9 F F F 14 280-299 6 6 6 11 11 11 F F F 15 300-319 6 6 6 11 11 11 F F F 16 320-339 6 6 6 10 10 10 F F F 17 340-359 4 4 4 7 7 7 F F F 18 360-379 6 6 6 13 13 13 F F F 19 380-399 7 7 7 14 14 14 F F F 20 400-419 4 4 4 7 7 7 F F F 21 420-439 6 6 6 10 10 10 F F F 22 440-459 6 6 6 10 10 10 F F F 23 460-479 5 5 5 8 8 8 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 6 6 6 9 9 9 F F F 26 520-539 6 6 6 9 9 9 F F F 27 540-559 3 3 3 4 4 4 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 2 2 2 2 2 2 F F F Max gen: 7, Max flush: 14 Begin loop 4 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.023357851Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.023662447Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.023684806Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214} {"msg":"Crucible 2982aa3f-16fa-4559-a841-9d8e79cad845 has session id: 6044df02-0e27-4066-90e0-ff934062386a","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.024079877Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"Upstairs opts: Upstairs UUID: 2982aa3f-16fa-4559-a841-9d8e79cad845, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.024112266Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"Crucible stats registered with UUID: 2982aa3f-16fa-4559-a841-9d8e79cad845","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.024126655Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"session_id":"6044df02-0e27-4066-90e0-ff934062386a"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:8","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.024525286Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214} {"msg":"2982aa3f-16fa-4559-a841-9d8e79cad845 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.024547785Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.024569114Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"0","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.024601943Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"0","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.024626902Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.02465009Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.024678249Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"2","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.024702128Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"2","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.024724417Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"io task","client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.024748606Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"io task","client":"0","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.024783184Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"io task","client":"2","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:28:20.024898869Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"io task","client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:28:20.024940587Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"io task","client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:28:20.024968295Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.024989705Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.025013713Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"ds_connection connected from Ok(127.0.0.1:55824)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.025036942Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"io task","client":"2","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"ds_connection connected from Ok(127.0.0.1:45018)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.025136558Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"io task","client":"0","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.025193085Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"io task","client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.025766468Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"2","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.025793027Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"2","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.025901222Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"0","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.025929681Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"0","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.026286024Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"2","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.026399429Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"0","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.026730083Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"2","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.026778131Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"2","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.02679299Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.026934594Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"0","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.026969552Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"0","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:20.026992491Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.02642672Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"io task","client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"ds_connection connected from Ok(127.0.0.1:50117)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.026609271Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"io task","client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.027856583Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.027892792Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.028390328Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.028899415Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.028928943Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"[0]R flush_numbers[0..12]: [14, 11, 2, 7, 1, 1, 2, 1, 14, 6, 10, 10]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.028951742Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"downstairs","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"[0]R generation[0..12]: [7, 6, 2, 4, 1, 1, 2, 1, 7, 4, 6, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.028967772Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"downstairs","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.02898822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"downstairs","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"[1]R flush_numbers[0..12]: [14, 11, 2, 7, 1, 1, 2, 1, 14, 6, 10, 10]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.02900274Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"downstairs","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"[1]R generation[0..12]: [7, 6, 2, 4, 1, 1, 2, 1, 7, 4, 6, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.029017339Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"downstairs","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.029032629Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"downstairs","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"[2]R flush_numbers[0..12]: [14, 11, 2, 7, 1, 1, 2, 1, 14, 6, 10, 10]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.029046338Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"downstairs","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"[2]R generation[0..12]: [7, 6, 2, 4, 1, 1, 2, 1, 7, 4, 6, 6]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.029060597Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"downstairs","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.029075316Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"downstairs","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"Max found gen is 8","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:28:30.029088796Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"downstairs","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"Generation requested: 8 >= found:8","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.029104665Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"downstairs","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"Next flush: 15","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.029117654Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"downstairs","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.029130634Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"":"downstairs","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.029144093Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.029155963Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"0","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.029169372Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"1","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.029190211Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"client":"2","":"downstairs","session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.02920829Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.0292209Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"2982aa3f-16fa-4559-a841-9d8e79cad845 is now active with session: 6044df02-0e27-4066-90e0-ff934062386a","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.029233729Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.029245808Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214,"session_id":"6044df02-0e27-4066-90e0-ff934062386a"} {"msg":"The guest has finished waiting for activation with:8","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:30.029265798Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 476 len 8192 data: 2 2 02/30 Flush 03/30 Flush 04/30 Flush 05/30 Write block 470 len 32768 data: 2 2 2 2 2 2 3 3 06/30 Read block 292 len 8192 07/30 Read block 130 len 28672 08/30 Read block 436 len 32768 09/30 Read block 569 len 40960 10/30 Write block 73 len 20480 data: 3 2 2 2 2 11/30 Read block 4 len 28672 12/30 Flush 13/30 Write block 406 len 8192 data: 3 3 14/30 Read block 90 len 4096 15/30 Write block 232 len 20480 data: 2 2 2 2 2 16/30 Write block 368 len 4096 data: 4 17/30 Write block 441 len 20480 data: 3 3 2 2 2 18/30 Write block 335 len 36864 data: 3 3 3 3 2 2 2 2 2 19/30 Write block 455 len 12288 data: 2 2 2 20/30 Read block 344 len 12288 21/30 Write block 419 len 8192 data: 2 2 22/30 Write block 328 len 40960 data: 2 2 2 2 2 2 3 4 4 4 23/30 Read block 37 len 16384 24/30 Flush 25/30 Write block 390 len 36864 data: 3 3 5 5 5 5 5 5 4 26/30 Flush 27/30 Write block 227 len 12288 data: 2 2 2 28/30 Read block 219 len 24576 29/30 Read block 125 len 36864 30/30 Write block 52 len 8192 data: 2 2 ---------------------------------------------------------------- Crucible gen:8 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 3 Done Done Done false 28 Acked 1027 Read 6 Done Done Done false 29 Acked 1028 Read 9 Done Done Done false 30 Acked 1029 Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:28:30.197826022Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1214} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 7 7 7 14 14 14 F F F 1 020-039 6 6 6 11 11 11 F F F 2 040-059 2 2 2 2 2 2 T T T 3 060-079 8 8 8 18 18 18 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 7 7 7 14 14 14 F F F 9 180-199 4 4 4 6 6 6 F F F 10 200-219 6 6 6 10 10 10 F F F 11 220-239 8 8 8 19 19 19 T T T 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 9 9 9 F F F 14 280-299 6 6 6 11 11 11 F F F 15 300-319 6 6 6 11 11 11 F F F 16 320-339 8 8 8 19 19 19 F F F 17 340-359 8 8 8 19 19 19 F F F 18 360-379 8 8 8 19 19 19 F F F 19 380-399 8 8 8 20 20 20 F F F 20 400-419 8 8 8 19 19 19 F F F 21 420-439 8 8 8 19 19 19 F F F 22 440-459 8 8 8 19 19 19 F F F 23 460-479 8 8 8 18 18 18 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 6 6 6 9 9 9 F F F 26 520-539 6 6 6 9 9 9 F F F 27 540-559 3 3 3 4 4 4 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 2 2 2 2 2 2 F F F Max gen: 8, Max flush: 20 A Difference in extent metadata is expected here On loop 4 Verifying data now No change after verify Loop: 4 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 7 7 7 14 14 14 F F F 1 020-039 6 6 6 11 11 11 F F F 2 040-059 9 9 9 21 21 21 F F F 3 060-079 8 8 8 18 18 18 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 7 7 7 14 14 14 F F F 9 180-199 4 4 4 6 6 6 F F F 10 200-219 6 6 6 10 10 10 F F F 11 220-239 9 9 9 21 21 21 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 9 9 9 F F F 14 280-299 6 6 6 11 11 11 F F F 15 300-319 6 6 6 11 11 11 F F F 16 320-339 8 8 8 19 19 19 F F F 17 340-359 8 8 8 19 19 19 F F F 18 360-379 8 8 8 19 19 19 F F F 19 380-399 8 8 8 20 20 20 F F F 20 400-419 8 8 8 19 19 19 F F F 21 420-439 8 8 8 19 19 19 F F F 22 440-459 8 8 8 19 19 19 F F F 23 460-479 8 8 8 18 18 18 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 6 6 6 9 9 9 F F F 26 520-539 6 6 6 9 9 9 F F F 27 540-559 3 3 3 4 4 4 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 2 2 2 2 2 2 F F F Max gen: 9, Max flush: 21 Begin loop 5 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.716199205Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.71672907Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.716764738Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222} {"msg":"Crucible d529d84d-8909-42f4-a973-cc14907ac79d has session id: c9a08758-67ac-428f-9da8-64498e1f193f","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.717018416Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"Upstairs opts: Upstairs UUID: d529d84d-8909-42f4-a973-cc14907ac79d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.717101223Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"Crucible stats registered with UUID: d529d84d-8909-42f4-a973-cc14907ac79d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.717121951Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:10","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.717425047Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222} {"msg":"d529d84d-8909-42f4-a973-cc14907ac79d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.717487934Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.717513473Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"0","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.717540542Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"0","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.717566321Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.717592629Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.717617028Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"2","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.717640287Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"2","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.717662906Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"io task","client":"0","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.717687355Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"io task","client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.717709954Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"io task","client":"2","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"ds_connection connected from Ok(127.0.0.1:49016)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.71780055Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"io task","client":"2","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"ds_connection connected from Ok(127.0.0.1:48083)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.718019769Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"io task","client":"0","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:28:44.718060578Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"io task","client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:28:44.718092406Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"io task","client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:28:44.718198661Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.7182313Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.718256978Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.718377633Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"io task","client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.718907328Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"0","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.718934507Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"0","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.718949216Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"2","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.718973335Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"2","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.719549548Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"0","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.719696541Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"2","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.720068244Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"0","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.720101723Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"0","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.720121571Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.720254155Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"2","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.720281654Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"2","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:44.720297813Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.720041038Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"io task","client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"ds_connection connected from Ok(127.0.0.1:33055)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.720245289Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"io task","client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.721651063Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.721700061Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722174289Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722705764Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722729263Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"[0]R flush_numbers[0..12]: [14, 11, 21, 18, 1, 1, 2, 1, 14, 6, 10, 21]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722744572Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"downstairs","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"[0]R generation[0..12]: [7, 6, 9, 8, 1, 1, 2, 1, 7, 4, 6, 9]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722759032Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"downstairs","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722773561Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"downstairs","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"[1]R flush_numbers[0..12]: [14, 11, 21, 18, 1, 1, 2, 1, 14, 6, 10, 21]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.7227873Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"downstairs","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"[1]R generation[0..12]: [7, 6, 9, 8, 1, 1, 2, 1, 7, 4, 6, 9]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722804969Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"downstairs","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722819349Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"downstairs","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"[2]R flush_numbers[0..12]: [14, 11, 21, 18, 1, 1, 2, 1, 14, 6, 10, 21]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722832958Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"downstairs","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"[2]R generation[0..12]: [7, 6, 9, 8, 1, 1, 2, 1, 7, 4, 6, 9]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722846197Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"downstairs","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722860157Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"downstairs","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"Max found gen is 10","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722872946Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"downstairs","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"Generation requested: 10 >= found:10","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722886135Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"downstairs","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} Wait for a query_work_queue command to finish before sending IO {"msg":"Next flush: 22","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722902365Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"downstairs","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722915924Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"":"downstairs","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722929453Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722941493Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"0","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722954942Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"1","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722968072Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"client":"2","":"downstairs","session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722981721Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.722993941Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"d529d84d-8909-42f4-a973-cc14907ac79d is now active with session: c9a08758-67ac-428f-9da8-64498e1f193f","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.72300877Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.723021019Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222,"session_id":"c9a08758-67ac-428f-9da8-64498e1f193f"} {"msg":"The guest has finished waiting for activation with:10","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:28:54.723033189Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 543 len 16384 data: 2 2 2 2 02/30 Read block 177 len 8192 03/30 Write block 364 len 32768 data: 3 3 3 4 5 3 3 3 04/30 Read block 514 len 28672 05/30 Write block 501 len 8192 data: 2 3 06/30 Write block 170 len 40960 data: 4 4 4 4 2 2 2 2 2 3 07/30 Write block 38 len 28672 data: 2 2 2 2 2 2 2 08/30 Flush 09/30 Write block 384 len 4096 data: 2 10/30 Flush 11/30 Write block 334 len 16384 data: 4 5 5 5 12/30 Write block 232 len 32768 data: 3 3 3 3 3 2 2 2 13/30 Write block 471 len 8192 data: 3 3 14/30 Write block 55 len 20480 data: 2 2 3 3 3 15/30 Read block 503 len 4096 16/30 Read block 341 len 28672 17/30 Read block 262 len 40960 18/30 Read block 7 len 20480 19/30 Read block 271 len 24576 20/30 Write block 72 len 16384 data: 3 4 3 3 21/30 Read block 128 len 4096 22/30 Read block 152 len 36864 23/30 Flush 24/30 Write block 308 len 16384 data: 2 2 2 2 25/30 Flush 26/30 Write block 25 len 40960 data: 3 3 3 3 4 2 2 2 2 2 27/30 Write block 521 len 8192 data: 3 3 28/30 Read block 522 len 32768 29/30 Read block 350 len 28672 30/30 Read block 547 len 20480 ---------------------------------------------------------------- Crucible gen:10 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 26 Acked 1025 Write 10 Done Done Done false 27 Acked 1026 Write 2 Done Done Done false 28 Acked 1027 Read 8 Done Done Done false 29 Acked 1028 Read 7 Done Done Done false 30 Acked 1029 Read 5 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 0 2 Done 4 4 5 13 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1024 1024 1024 Downstairs last five completed: 1024 1023 1022 1021 1020 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:28:54.934112169Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1222} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 7 7 7 14 14 14 F F F 1 020-039 10 10 10 22 22 22 T T T 2 040-059 10 10 10 24 24 24 F F F 3 060-079 10 10 10 24 24 24 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 10 10 10 22 22 22 F F F 9 180-199 4 4 4 6 6 6 F F F 10 200-219 6 6 6 10 10 10 F F F 11 220-239 10 10 10 24 24 24 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 9 9 9 F F F 14 280-299 6 6 6 11 11 11 F F F 15 300-319 10 10 10 25 25 25 F F F 16 320-339 10 10 10 24 24 24 F F F 17 340-359 8 8 8 19 19 19 F F F 18 360-379 10 10 10 22 22 22 F F F 19 380-399 10 10 10 23 23 23 F F F 20 400-419 8 8 8 19 19 19 F F F 21 420-439 8 8 8 19 19 19 F F F 22 440-459 8 8 8 19 19 19 F F F 23 460-479 10 10 10 24 24 24 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 10 10 10 22 22 22 F F F 26 520-539 6 6 6 9 9 9 T T T 27 540-559 10 10 10 22 22 22 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 2 2 2 2 2 2 F F F Max gen: 10, Max flush: 25 A Difference in extent metadata is expected here On loop 5 Verifying data now No change after verify Loop: 5 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 7 7 7 14 14 14 F F F 1 020-039 11 11 11 26 26 26 F F F 2 040-059 10 10 10 24 24 24 F F F 3 060-079 10 10 10 24 24 24 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 10 10 10 22 22 22 F F F 9 180-199 4 4 4 6 6 6 F F F 10 200-219 6 6 6 10 10 10 F F F 11 220-239 10 10 10 24 24 24 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 6 6 6 9 9 9 F F F 14 280-299 6 6 6 11 11 11 F F F 15 300-319 10 10 10 25 25 25 F F F 16 320-339 10 10 10 24 24 24 F F F 17 340-359 8 8 8 19 19 19 F F F 18 360-379 10 10 10 22 22 22 F F F 19 380-399 10 10 10 23 23 23 F F F 20 400-419 8 8 8 19 19 19 F F F 21 420-439 8 8 8 19 19 19 F F F 22 440-459 8 8 8 19 19 19 F F F 23 460-479 10 10 10 24 24 24 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 10 10 10 22 22 22 F F F 26 520-539 11 11 11 26 26 26 F F F 27 540-559 10 10 10 22 22 22 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 2 2 2 2 2 2 F F F Max gen: 11, Max flush: 26 Begin loop 6 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.441354271Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.441747613Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.441781171Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230} {"msg":"Crucible 90d21d36-42c6-4838-bdee-0fd7258ffddf has session id: 7819c195-9306-49d8-8fbe-a9a385f7bbd5","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.441916495Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"Upstairs opts: Upstairs UUID: 90d21d36-42c6-4838-bdee-0fd7258ffddf, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.441961653Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"Crucible stats registered with UUID: 90d21d36-42c6-4838-bdee-0fd7258ffddf","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.441991112Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:12","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.442293078Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230} {"msg":"90d21d36-42c6-4838-bdee-0fd7258ffddf active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.442327266Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.442342555Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"0","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.442361004Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"0","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.442374764Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"1","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.442388393Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"1","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.442401653Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.442423092Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.44245332Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"io task","client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.442474009Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"io task","client":"0","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.442488038Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"io task","client":"1","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:29:09.442634292Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"io task","client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:29:09.44267166Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"io task","client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:29:09.442705648Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.442728867Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.442751496Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"ds_connection connected from Ok(127.0.0.1:49478)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.442775445Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"io task","client":"1","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"ds_connection connected from Ok(127.0.0.1:58514)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.442895019Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"io task","client":"0","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.442934788Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"io task","client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.443427405Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"1","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.443451493Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"1","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.443567138Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"0","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.443596687Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"0","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.443927591Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"1","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.444051865Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"0","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.44439652Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"1","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.444427458Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"1","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.444451607Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.444546873Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"0","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.444566292Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"0","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:09.444579731Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.444317705Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"io task","client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"ds_connection connected from Ok(127.0.0.1:62872)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.444721806Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"io task","client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.445768498Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.445797646Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.446309522Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.446843818Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.446869216Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"[0]R flush_numbers[0..12]: [14, 26, 24, 24, 1, 1, 2, 1, 22, 6, 10, 24]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.446884636Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"downstairs","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"[0]R generation[0..12]: [7, 11, 10, 10, 1, 1, 2, 1, 10, 4, 6, 10]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.446899115Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"downstairs","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.446913484Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"downstairs","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"[1]R flush_numbers[0..12]: [14, 26, 24, 24, 1, 1, 2, 1, 22, 6, 10, 24]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.446927094Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"downstairs","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"[1]R generation[0..12]: [7, 11, 10, 10, 1, 1, 2, 1, 10, 4, 6, 10]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.446940653Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"downstairs","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.446960412Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"downstairs","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"[2]R flush_numbers[0..12]: [14, 26, 24, 24, 1, 1, 2, 1, 22, 6, 10, 24]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.446974511Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"downstairs","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"[2]R generation[0..12]: [7, 11, 10, 10, 1, 1, 2, 1, 10, 4, 6, 10]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.446987951Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"downstairs","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.44700219Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"downstairs","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"Max found gen is 12","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.44701525Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"downstairs","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"Generation requested: 12 >= found:12","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.447028419Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"downstairs","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} Wait for a query_work_queue command to finish before sending IO {"msg":"Next flush: 27","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.447044578Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"downstairs","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.447058388Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"":"downstairs","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.447071977Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.447083886Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"0","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.447097246Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"1","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.447110365Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"client":"2","":"downstairs","session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.447123504Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.447135674Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"90d21d36-42c6-4838-bdee-0fd7258ffddf is now active with session: 7819c195-9306-49d8-8fbe-a9a385f7bbd5","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.447148513Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.447162223Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230,"session_id":"7819c195-9306-49d8-8fbe-a9a385f7bbd5"} {"msg":"The guest has finished waiting for activation with:12","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:19.447174592Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 432 len 32768 data: 3 3 2 3 3 3 3 3 02/30 Write block 356 len 20480 data: 3 4 4 4 4 03/30 Write block 238 len 24576 data: 3 3 2 2 2 2 04/30 Write block 287 len 24576 data: 3 3 3 3 3 3 05/30 Read block 258 len 8192 06/30 Write block 254 len 36864 data: 4 4 4 3 3 3 3 3 3 07/30 Write block 89 len 12288 data: 2 2 2 08/30 Write block 378 len 40960 data: 2 2 2 2 2 2 3 2 2 2 09/30 Read block 563 len 40960 10/30 Write block 245 len 24576 data: 2 2 2 2 2 2 11/30 Read block 418 len 16384 12/30 Flush 13/30 Write block 172 len 24576 data: 5 5 3 3 3 3 14/30 Write block 266 len 28672 data: 2 2 2 2 2 2 2 15/30 Read block 451 len 4096 16/30 Write block 309 len 28672 data: 3 3 3 2 2 2 2 17/30 Write block 167 len 12288 data: 3 3 3 18/30 Read block 155 len 20480 19/30 Read block 499 len 4096 20/30 Flush 21/30 Write block 455 len 16384 data: 3 3 3 2 22/30 Write block 588 len 20480 data: 3 2 2 2 2 23/30 Write block 158 len 16384 data: 2 2 2 2 24/30 Write block 3 len 20480 data: 3 3 2 3 4 25/30 Read block 122 len 28672 26/30 Write block 80 len 24576 data: 2 2 2 2 2 2 27/30 Read block 164 len 8192 28/30 Read block 153 len 4096 29/30 Write block 502 len 28672 data: 4 3 3 3 3 3 3 30/30 Read block 210 len 20480 ---------------------------------------------------------------- Crucible gen:12 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 21 Acked 1020 Write 4 Done Done Done false 22 Acked 1021 Write 5 Done Done Done false 23 Acked 1022 Write 4 Done Done Done false 24 Acked 1023 Write 5 Done Done Done false 25 Acked 1024 Read 7 Done Done Done false 26 Acked 1025 Write 6 Done Done Done false 27 Acked 1026 Read 2 Done Done Done false 28 Acked 1027 Read 1 Done Done Done false 29 Acked 1028 Write 7 Done Done Done false 30 Acked 1029 Read 5 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 0 1 Done 9 10 10 29 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1019 1019 1019 Downstairs last five completed: 1019 1018 1017 1016 1015 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:29:19.626299884Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1230} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 7 7 7 14 14 14 T T T 1 020-039 11 11 11 26 26 26 F F F 2 040-059 10 10 10 24 24 24 F F F 3 060-079 10 10 10 24 24 24 F F F 4 080-099 12 12 12 27 27 27 T T T 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 1 1 1 1 1 1 T T T 8 160-179 12 12 12 28 28 28 T T T 9 180-199 4 4 4 6 6 6 F F F 10 200-219 6 6 6 10 10 10 F F F 11 220-239 12 12 12 27 27 27 F F F 12 240-259 12 12 12 27 27 27 F F F 13 260-279 12 12 12 28 28 28 F F F 14 280-299 12 12 12 27 27 27 F F F 15 300-319 12 12 12 28 28 28 F F F 16 320-339 10 10 10 24 24 24 F F F 17 340-359 12 12 12 27 27 27 F F F 18 360-379 12 12 12 27 27 27 F F F 19 380-399 12 12 12 27 27 27 F F F 20 400-419 8 8 8 19 19 19 F F F 21 420-439 12 12 12 27 27 27 F F F 22 440-459 8 8 8 19 19 19 T T T 23 460-479 10 10 10 24 24 24 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 10 10 10 22 22 22 T T T 26 520-539 11 11 11 26 26 26 F F F 27 540-559 10 10 10 22 22 22 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 2 2 2 2 2 2 T T T Max gen: 12, Max flush: 28 A Difference in extent metadata is expected here On loop 6 Verifying data now No change after verify Loop: 6 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 29 29 29 F F F 1 020-039 11 11 11 26 26 26 F F F 2 040-059 10 10 10 24 24 24 F F F 3 060-079 10 10 10 24 24 24 F F F 4 080-099 13 13 13 29 29 29 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 13 13 13 29 29 29 F F F 8 160-179 13 13 13 29 29 29 F F F 9 180-199 4 4 4 6 6 6 F F F 10 200-219 6 6 6 10 10 10 F F F 11 220-239 12 12 12 27 27 27 F F F 12 240-259 12 12 12 27 27 27 F F F 13 260-279 12 12 12 28 28 28 F F F 14 280-299 12 12 12 27 27 27 F F F 15 300-319 12 12 12 28 28 28 F F F 16 320-339 10 10 10 24 24 24 F F F 17 340-359 12 12 12 27 27 27 F F F 18 360-379 12 12 12 27 27 27 F F F 19 380-399 12 12 12 27 27 27 F F F 20 400-419 8 8 8 19 19 19 F F F 21 420-439 12 12 12 27 27 27 F F F 22 440-459 13 13 13 29 29 29 F F F 23 460-479 10 10 10 24 24 24 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 13 13 13 29 29 29 F F F 26 520-539 11 11 11 26 26 26 F F F 27 540-559 10 10 10 22 22 22 F F F 28 560-579 4 4 4 7 7 7 F F F 29 580-599 13 13 13 29 29 29 F F F Max gen: 13, Max flush: 29 Begin loop 7 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.888526786Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.889101199Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.889130707Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238} {"msg":"Crucible 6d93399a-c29a-4b78-861c-815ba0d73dbe has session id: ffd22f14-daac-49f9-b473-b5167625508c","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.889231043Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"Upstairs opts: Upstairs UUID: 6d93399a-c29a-4b78-861c-815ba0d73dbe, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.88928241Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"Crucible stats registered with UUID: 6d93399a-c29a-4b78-861c-815ba0d73dbe","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.889312229Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:14","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.889576846Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238} {"msg":"6d93399a-c29a-4b78-861c-815ba0d73dbe active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.889657243Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.889680112Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.88971301Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.889739649Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"1","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.889763888Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"1","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.889788107Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"2","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.889809196Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"2","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.889831175Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"io task","client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.889855273Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"io task","client":"2","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.889877563Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"io task","client":"1","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:29:34.890011296Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"io task","client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"ds_connection connected from Ok(127.0.0.1:48800)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.890060974Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"io task","client":"2","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:29:34.890100322Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"io task","client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:29:34.890134571Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.890157839Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.890176378Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"ds_connection connected from Ok(127.0.0.1:61636)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.890190388Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"io task","client":"1","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.890212887Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"io task","client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.890753662Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"1","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.8907805Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"1","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.89079534Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"2","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.890813279Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"2","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.891502767Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"1","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.891529926Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"2","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.892016033Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"2","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.892042271Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"2","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.892061371Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.89208091Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"1","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.892094699Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"1","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:34.892107639Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.890724166Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"io task","client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"ds_connection connected from Ok(127.0.0.1:53930)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.890951765Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"io task","client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.892100142Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.89213172Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.892755981Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893368693Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893395431Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"[0]R flush_numbers[0..12]: [29, 26, 24, 24, 29, 1, 2, 29, 29, 6, 10, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893411411Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"downstairs","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"[0]R generation[0..12]: [13, 11, 10, 10, 13, 1, 2, 13, 13, 4, 6, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.89342621Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"downstairs","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893440559Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"downstairs","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"[1]R flush_numbers[0..12]: [29, 26, 24, 24, 29, 1, 2, 29, 29, 6, 10, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893454399Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"downstairs","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"[1]R generation[0..12]: [13, 11, 10, 10, 13, 1, 2, 13, 13, 4, 6, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893472088Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"downstairs","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893486527Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"downstairs","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"[2]R flush_numbers[0..12]: [29, 26, 24, 24, 29, 1, 2, 29, 29, 6, 10, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893500147Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"downstairs","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"[2]R generation[0..12]: [13, 11, 10, 10, 13, 1, 2, 13, 13, 4, 6, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893513676Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"downstairs","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893527645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"downstairs","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"Max found gen is 14","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893540415Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"downstairs","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"Generation requested: 14 >= found:14","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893553614Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"downstairs","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"Next flush: 30","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893566244Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"downstairs","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"All extents match","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:29:44.893579033Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"":"downstairs","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893594392Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893606351Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"0","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893619621Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"1","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.89363282Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"client":"2","":"downstairs","session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.89364625Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893658489Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"6d93399a-c29a-4b78-861c-815ba0d73dbe is now active with session: ffd22f14-daac-49f9-b473-b5167625508c","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893671228Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893684898Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238,"session_id":"ffd22f14-daac-49f9-b473-b5167625508c"} {"msg":"The guest has finished waiting for activation with:14","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:44.893697177Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 579 len 16384 data: 2 2 2 2 02/30 Flush 03/30 Write block 405 len 24576 data: 3 4 4 3 2 2 04/30 Write block 285 len 8192 data: 3 3 05/30 Read block 500 len 4096 06/30 Read block 464 len 20480 07/30 Read block 403 len 32768 08/30 Write block 426 len 16384 data: 2 2 2 2 09/30 Read block 154 len 36864 10/30 Write block 563 len 20480 data: 3 3 3 3 2 11/30 Write block 376 len 4096 data: 2 12/30 Write block 477 len 4096 data: 4 13/30 Read block 93 len 40960 14/30 Read block 271 len 24576 15/30 Write block 349 len 32768 data: 2 2 2 2 3 3 3 4 16/30 Write block 416 len 32768 data: 2 2 2 3 3 2 2 2 17/30 Read block 521 len 8192 18/30 Read block 508 len 36864 19/30 Read block 576 len 40960 20/30 Read block 551 len 24576 21/30 Read block 553 len 16384 22/30 Read block 546 len 8192 23/30 Write block 390 len 16384 data: 4 4 6 6 24/30 Write block 582 len 36864 data: 3 2 2 2 3 3 4 3 3 25/30 Write block 192 len 16384 data: 2 2 2 2 26/30 Read block 545 len 32768 27/30 Read block 498 len 20480 28/30 Read block 231 len 40960 29/30 Read block 461 len 12288 30/30 Read block 21 len 24576 ---------------------------------------------------------------- Crucible gen:14 GIO:true work queues: Upstairs:1 downstairs:29 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 3 Acked 1002 Write 6 Done Done Done false 4 Acked 1003 Write 2 Done Done Done false 5 Acked 1004 Read 1 Done Done Done false 6 Acked 1005 Read 5 Done Done Done false 7 Acked 1006 Read 8 Done Done Done false 8 Acked 1007 Write 4 Done Done Done false 9 Acked 1008 Read 9 Done Done Done false 10 Acked 1009 Write 5 Done Done Done false 11 Acked 1010 Write 1 Done Done Done false 12 Acked 1011 Write 1 Done Done Done false 13 Acked 1012 Read 10 Done Done Done false 14 Acked 1013 Read 6 Done Done Done false 15 Acked 1014 Write 8 Done Done Done false 16 Acked 1015 Write 8 Done Done Done false 17 Acked 1016 Read 2 Done Done Done false 18 Acked 1017 Read 9 Done Done Done false 19 NotAcked 1018 Flush 0 Done Sent Sent false 20 Acked 1019 Read 10 Done Sent Sent false 21 Acked 1020 Read 6 Done Sent Sent false 22 Acked 1021 Read 4 Done Sent Sent false 23 Acked 1022 Read 2 Done Sent Sent false 24 Acked 1023 Write 4 Done Sent Sent false 25 Acked 1024 Write 9 Done Sent Sent false 26 Acked 1025 Write 4 Done Sent Sent false 27 Acked 1026 Read 8 Done Sent Sent false 28 Acked 1027 Read 5 Done Sent Sent false 29 Acked 1028 Read 10 Done Sent Sent false 30 Acked 1029 Read 3 Done Sent Sent false 31 Acked 1030 Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 13 13 26 Done 29 16 16 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1001 1001 Downstairs last five completed: 1001 1000 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:29:44.921132257Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1238} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 29 29 29 F F F 1 020-039 11 11 11 26 26 26 F F F 2 040-059 10 10 10 24 24 24 F F F 3 060-079 10 10 10 24 24 24 F F F 4 080-099 13 13 13 29 29 29 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 13 13 13 29 29 29 F F F 8 160-179 13 13 13 29 29 29 F F F 9 180-199 4 4 4 6 6 6 T F F <--- 10 200-219 6 6 6 10 10 10 F F F 11 220-239 12 12 12 27 27 27 F F F 12 240-259 12 12 12 27 27 27 F F F 13 260-279 12 12 12 28 28 28 F F F 14 280-299 14 14 14 31 31 31 F F F 15 300-319 12 12 12 28 28 28 F F F 16 320-339 10 10 10 24 24 24 F F F 17 340-359 14 14 14 31 31 31 F F F 18 360-379 14 14 14 31 31 31 F F F 19 380-399 12 12 12 27 27 27 T F F <--- 20 400-419 14 14 14 31 31 31 F F F 21 420-439 14 14 14 31 31 31 F F F 22 440-459 13 13 13 29 29 29 F F F 23 460-479 14 14 14 31 31 31 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 13 13 13 29 29 29 F F F 26 520-539 11 11 11 26 26 26 F F F 27 540-559 10 10 10 22 22 22 F F F 28 560-579 14 14 14 31 31 31 F F F 29 580-599 14 14 14 30 30 30 T F F <--- Max gen: 14, Max flush: 31 Error: Difference in extent metadata found! A Difference in extent metadata is expected here On loop 7 Verifying data now No change after verify Loop: 7 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 29 29 29 F F F 1 020-039 11 11 11 26 26 26 F F F 2 040-059 10 10 10 24 24 24 F F F 3 060-079 10 10 10 24 24 24 F F F 4 080-099 13 13 13 29 29 29 F F F 5 100-119 1 1 1 1 1 1 F F F 6 120-139 2 2 2 2 2 2 F F F 7 140-159 13 13 13 29 29 29 F F F 8 160-179 13 13 13 29 29 29 F F F 9 180-199 15 15 15 32 32 32 F F F 10 200-219 6 6 6 10 10 10 F F F 11 220-239 12 12 12 27 27 27 F F F 12 240-259 12 12 12 27 27 27 F F F 13 260-279 12 12 12 28 28 28 F F F 14 280-299 14 14 14 31 31 31 F F F 15 300-319 12 12 12 28 28 28 F F F 16 320-339 10 10 10 24 24 24 F F F 17 340-359 14 14 14 31 31 31 F F F 18 360-379 14 14 14 31 31 31 F F F 19 380-399 15 15 15 32 32 32 F F F 20 400-419 14 14 14 31 31 31 F F F 21 420-439 14 14 14 31 31 31 F F F 22 440-459 13 13 13 29 29 29 F F F 23 460-479 14 14 14 31 31 31 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 13 13 13 29 29 29 F F F 26 520-539 11 11 11 26 26 26 F F F 27 540-559 10 10 10 22 22 22 F F F 28 560-579 14 14 14 31 31 31 F F F 29 580-599 15 15 15 32 32 32 F F F Max gen: 15, Max flush: 32 Begin loop 8 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.65919468Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.659473716Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.659494886Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246} {"msg":"Crucible 1f4fa312-e02a-496b-9c5a-48039cb9d3c8 has session id: b3d7f635-283d-4077-90af-8b03210b5beb","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.660014241Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"Upstairs opts: Upstairs UUID: 1f4fa312-e02a-496b-9c5a-48039cb9d3c8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.66004617Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"Crucible stats registered with UUID: 1f4fa312-e02a-496b-9c5a-48039cb9d3c8","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.660060809Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:16","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.660392104Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246} {"msg":"1f4fa312-e02a-496b-9c5a-48039cb9d3c8 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.660418992Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.660432822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"0","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.660450821Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"0","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.66046432Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"1","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.66047779Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"1","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.660492139Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.660515408Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.660544026Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"io task","client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.660560316Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"io task","client":"0","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.660574175Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"io task","client":"1","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"ds_connection connected from Ok(127.0.0.1:64867)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.660857542Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"io task","client":"0","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"ds_connection connected from Ok(127.0.0.1:43046)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.660968497Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"io task","client":"1","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:29:59.661134739Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"io task","client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:29:59.661175617Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"io task","client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:29:59.661259943Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.661286442Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.661306731Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.661371198Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"io task","client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.661773199Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"0","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.661799618Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"0","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.661814367Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"1","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.661827927Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"1","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.662412789Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"0","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.662545723Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"1","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.662928615Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"0","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.662953204Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"0","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.662967824Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.663052829Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"1","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.663071869Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"1","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:29:59.663085408Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.662617391Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"io task","client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"ds_connection connected from Ok(127.0.0.1:34770)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.662767974Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"io task","client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.663953339Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.663979837Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.664471814Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665005949Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665026819Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"[0]R flush_numbers[0..12]: [29, 26, 24, 24, 29, 1, 2, 29, 29, 32, 10, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665049567Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"downstairs","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"[0]R generation[0..12]: [13, 11, 10, 10, 13, 1, 2, 13, 13, 15, 6, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665064147Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"downstairs","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665078136Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"downstairs","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"[1]R flush_numbers[0..12]: [29, 26, 24, 24, 29, 1, 2, 29, 29, 32, 10, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665092005Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"downstairs","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"[1]R generation[0..12]: [13, 11, 10, 10, 13, 1, 2, 13, 13, 15, 6, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665105515Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"downstairs","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665119504Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"downstairs","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"[2]R flush_numbers[0..12]: [29, 26, 24, 24, 29, 1, 2, 29, 29, 32, 10, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665132883Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"downstairs","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"[2]R generation[0..12]: [13, 11, 10, 10, 13, 1, 2, 13, 13, 15, 6, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665146493Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"downstairs","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665162032Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"downstairs","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"Max found gen is 16","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665178772Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"downstairs","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"Generation requested: 16 >= found:16","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665194921Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"downstairs","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"Next flush: 33","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.66521084Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"downstairs","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"All extents match","v"Wait for a query_work_queue command to finish before sending IO :0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665230469Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"":"downstairs","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665247108Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665261948Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"0","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665281047Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"1","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665297836Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"client":"2","":"downstairs","session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665314305Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665329385Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"1f4fa312-e02a-496b-9c5a-48039cb9d3c8 is now active with session: b3d7f635-283d-4077-90af-8b03210b5beb","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665345414Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665360323Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246,"session_id":"b3d7f635-283d-4077-90af-8b03210b5beb"} {"msg":"The guest has finished waiting for activation with:16","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:09.665375392Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 333 len 8192 data: 3 5 02/30 Write block 246 len 28672 data: 3 3 3 3 3 2 3 03/30 Read block 563 len 8192 04/30 Read block 303 len 20480 05/30 Write block 185 len 12288 data: 2 2 2 06/30 Write block 536 len 24576 data: 2 2 2 2 2 3 07/30 Write block 279 len 24576 data: 2 2 2 2 2 2 08/30 Read block 541 len 8192 09/30 Read block 276 len 8192 10/30 Flush 11/30 Write block 553 len 16384 data: 3 3 3 3 12/30 Read block 249 len 32768 13/30 Write block 501 len 20480 data: 3 5 4 4 4 14/30 Write block 26 len 32768 data: 4 4 4 5 3 3 3 3 15/30 Write block 181 len 20480 data: 2 2 2 2 3 16/30 Write block 97 len 16384 data: 2 2 2 2 17/30 Read block 350 len 40960 18/30 Read block 97 len 28672 19/30 Flush 20/30 Write block 497 len 40960 data: 2 2 2 2 4 6 5 5 5 4 21/30 Write block 76 len 40960 data: 3 3 2 2 3 3 3 3 3 3 22/30 Write block 261 len 36864 data: 4 4 3 2 2 3 3 3 3 23/30 Write block 392 len 40960 data: 7 7 6 6 6 6 5 3 2 2 24/30 Read block 293 len 8192 25/30 Write block 284 len 36864 data: 3 4 4 4 4 4 4 4 4 26/30 Read block 407 len 12288 27/30 Read block 73 len 4096 28/30 Read block 154 len 40960 29/30 Write block 123 len 4096 data: 2 30/30 Read block 228 len 20480 ---------------------------------------------------------------- Crucible gen:16 GIO:true work queues: Upstairs:0 downstairs:11 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 20 Acked 1019 Write 10 Done Done Done false 21 Acked 1020 Write 10 Done Done Done false 22 Acked 1021 Write 9 Done Done Done false 23 Acked 1022 Write 10 Done Done Done false 24 Acked 1023 Read 2 Done Done Done false 25 Acked 1024 Write 9 Done Done Done false 26 Acked 1025 Read 3 Done Done Done false 27 Acked 1026 Read 1 Done Done Done false 28 Acked 1027 Read 10 Done Done Done false 29 Acked 1028 Write 1 Done Done Done false 30 Acked 1029 Read 5 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 0 1 Done 10 11 11 32 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1018 1018 Downstairs last five completed: 1018 1017 1016 1015 1014 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:30:09.79120953Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1246} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 29 29 29 F F F 1 020-039 16 16 16 34 34 34 F F F 2 040-059 10 10 10 24 24 24 F F F 3 060-079 10 10 10 24 24 24 T T T 4 080-099 16 16 16 34 34 34 T T T 5 100-119 16 16 16 34 34 34 F F F 6 120-139 2 2 2 2 2 2 T T T 7 140-159 13 13 13 29 29 29 F F F 8 160-179 13 13 13 29 29 29 F F F 9 180-199 16 16 16 34 34 34 F F F 10 200-219 6 6 6 10 10 10 F F F 11 220-239 12 12 12 27 27 27 F F F 12 240-259 16 16 16 33 33 33 F F F 13 260-279 16 16 16 33 33 33 T T T 14 280-299 16 16 16 33 33 33 T T T 15 300-319 12 12 12 28 28 28 F F F 16 320-339 16 16 16 33 33 33 F F F 17 340-359 14 14 14 31 31 31 F F F 18 360-379 14 14 14 31 31 31 F F F 19 380-399 15 15 15 32 32 32 T T T 20 400-419 14 14 14 31 31 31 T T T 21 420-439 14 14 14 31 31 31 F F F 22 440-459 13 13 13 29 29 29 F F F 23 460-479 14 14 14 31 31 31 F F F 24 480-499 1 1 1 1 1 1 T T T 25 500-519 16 16 16 34 34 34 T T T 26 520-539 16 16 16 33 33 33 F F F 27 540-559 16 16 16 34 34 34 F F F 28 560-579 14 14 14 31 31 31 F F F 29 580-599 15 15 15 32 32 32 F F F Max gen: 16, Max flush: 34 A Difference in extent metadata is expected here On loop 8 Verifying data now No change after verify Loop: 8 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 29 29 29 F F F 1 020-039 16 16 16 34 34 34 F F F 2 040-059 10 10 10 24 24 24 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 17 17 17 35 35 35 F F F 5 100-119 16 16 16 34 34 34 F F F 6 120-139 17 17 17 35 35 35 F F F 7 140-159 13 13 13 29 29 29 F F F 8 160-179 13 13 13 29 29 29 F F F 9 180-199 16 16 16 34 34 34 F F F 10 200-219 6 6 6 10 10 10 F F F 11 220-239 12 12 12 27 27 27 F F F 12 240-259 16 16 16 33 33 33 F F F 13 260-279 17 17 17 35 35 35 F F F 14 280-299 17 17 17 35 35 35 F F F 15 300-319 12 12 12 28 28 28 F F F 16 320-339 16 16 16 33 33 33 F F F 17 340-359 14 14 14 31 31 31 F F F 18 360-379 14 14 14 31 31 31 F F F 19 380-399 17 17 17 35 35 35 F F F 20 400-419 17 17 17 35 35 35 F F F 21 420-439 14 14 14 31 31 31 F F F 22 440-459 13 13 13 29 29 29 F F F 23 460-479 14 14 14 31 31 31 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 17 17 17 35 35 35 F F F 26 520-539 16 16 16 33 33 33 F F F 27 540-559 16 16 16 34 34 34 F F F 28 560-579 14 14 14 31 31 31 F F F 29 580-599 15 15 15 32 32 32 F F F Max gen: 17, Max flush: 35 Begin loop 9 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.53357598Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.534110515Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.534140634Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254} {"msg":"Crucible 60466546-cea0-4d59-bfd4-a8f43c4f4540 has session id: dcd9ac61-3f66-4963-b582-1e43651dffc0","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.534282407Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"Upstairs opts: Upstairs UUID: 60466546-cea0-4d59-bfd4-a8f43c4f4540, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.534396702Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"Crucible stats registered with UUID: 60466546-cea0-4d59-bfd4-a8f43c4f4540","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.53442688Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:18","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.534728056Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254} {"msg":"60466546-cea0-4d59-bfd4-a8f43c4f4540 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.534812533Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.534830932Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.534850331Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.53487136Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"1","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.534897049Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"1","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.534922307Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"2","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.534952776Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"2","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.534978595Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"io task","client":"2","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.535004553Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"io task","client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.535030072Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"io task","client":"1","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"ds_connection connected from Ok(127.0.0.1:39172)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.535065701Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"io task","client":"2","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:30:25.535099709Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"io task","client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:30:25.535129418Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"io task","client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:30:25.535155946Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"ds_connection connected from Ok(127.0.0.1:40362)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.535177695Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"io task","client":"1","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.535201974Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.535224273Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.535398985Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"io task","client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.535863993Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"2","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.535889512Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"2","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.535906511Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"1","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.53593428Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"1","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.536517463Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"1","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.536556391Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"2","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.537060868Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"1","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.537087076Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"1","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.537101946Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.537158073Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"2","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.537177182Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"2","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:25.537190722Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.536249128Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"io task","client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"ds_connection connected from Ok(127.0.0.1:39817)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.53644134Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"io task","client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.537837685Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.537867063Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.538454586Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539045498Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539075007Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"[0]R flush_numbers[0..12]: [29, 34, 24, 35, 35, 34, 35, 29, 29, 34, 10, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539093296Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"downstairs","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"[0]R generation[0..12]: [13, 16, 10, 17, 17, 16, 17, 13, 13, 16, 6, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539110595Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"downstairs","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539128414Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"downstairs","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"[1]R flush_numbers[0..12]: [29, 34, 24, 35, 35, 34, 35, 29, 29, 34, 10, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539145353Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"downstairs","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"[1]R generation[0..12]: [13, 16, 10, 17, 17, 16, 17, 13, 13, 16, 6, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539162063Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"downstairs","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539179582Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"downstairs","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"[2]R flush_numbers[0..12]: [29, 34, 24, 35, 35, 34, 35, 29, 29, 34, 10, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539196621Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"downstairs","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"[2]R generation[0..12]: [13, 16, 10, 17, 17, 16, 17, 13, 13, 16, 6, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.53921341Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"downstairs","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"Wait for a query_work_queue command to finish before sending IO [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539234699Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"downstairs","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"Max found gen is 18","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539252299Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"downstairs","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"Generation requested: 18 >= found:18","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539265568Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"downstairs","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"Next flush: 36","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539278437Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"downstairs","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539291427Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"":"downstairs","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539304556Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539318905Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"0","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539332075Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"1","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539345054Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"client":"2","":"downstairs","session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539358563Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539370713Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"60466546-cea0-4d59-bfd4-a8f43c4f4540 is now active with session: dcd9ac61-3f66-4963-b582-1e43651dffc0","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539383432Z"Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539404661Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254,"session_id":"dcd9ac61-3f66-4963-b582-1e43651dffc0"} {"msg":"The guest has finished waiting for activation with:18","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:35.539416891Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Flush 03/30 Write block 319 len 20480 data: 2 2 2 2 2 04/30 Read block 401 len 12288 05/30 Read block 306 len 4096 06/30 Read block 564 len 24576 07/30 Read block 3 len 24576 08/30 Read block 163 len 28672 09/30 Read block 17 len 8192 10/30 Read block 144 len 24576 11/30 Write block 578 len 24576 data: 2 3 3 3 4 3 12/30 Read block 16 len 20480 13/30 Read block 160 len 40960 14/30 Flush 15/30 Write block 129 len 16384 data: 3 3 2 2 16/30 Write block 369 len 24576 data: 4 4 4 3 3 2 17/30 Read block 232 len 32768 18/30 Read block 508 len 40960 19/30 Write block 341 len 20480 data: 3 3 3 2 2 20/30 Read block 515 len 40960 21/30 Write block 201 len 16384 data: 2 3 3 3 22/30 Read block 244 len 12288 23/30 Read block 257 len 24576 24/30 Write block 148 len 24576 data: 2 2 2 2 2 2 25/30 Read block 364 len 28672 26/30 Read block 114 len 12288 27/30 Read block 80 len 20480 28/30 Read block 274 len 40960 29/30 Read block 14 len 40960 30/30 Write block 349 len 36864 data: 3 3 3 3 4 4 4 5 5 ---------------------------------------------------------------- Crucible gen:18 GIO:true work queues: Upstairs:0 downstairs:16 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 15 Acked 1014 Write 4 Done Done Done false 16 Acked 1015 Write 6 Done Done Done false 17 Acked 1016 Read 8 Done Done Done false 18 Acked 1017 Read 10 Done Done Done false 19 Acked 1018 Write 5 Done Done Done false 20 Acked 1019 Read 10 Done Done Done false 21 Acked 1020 Write 4 Done Done Done false 22 Acked 1021 Read 3 Done Done Done false 23 Acked 1022 Read 6 Done Done Done false 24 Acked 1023 Write 6 Done Done Done false 25 Acked 1024 Read 7 Done Done Done false 26 Acked 1025 Read 3 Done Done Done false 27 Acked 1026 Read 5 Done Done Done false 28 Acked 1027 Read 10 Done Done Done false 29 Acked 1028 Read 10 Done Done Done false 30 Acked 1029 Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 15 15 15 45 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1013 1013 1013 Downstairs last five completed: 1013 1012 1011 1010 1009 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:30:35.645054522Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1254} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 29 29 29 F F F 1 020-039 16 16 16 34 34 34 F F F 2 040-059 10 10 10 24 24 24 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 17 17 17 35 35 35 F F F 5 100-119 16 16 16 34 34 34 F F F 6 120-139 17 17 17 35 35 35 T T T 7 140-159 13 13 13 29 29 29 T T T 8 160-179 13 13 13 29 29 29 F F F 9 180-199 16 16 16 34 34 34 F F F 10 200-219 6 6 6 10 10 10 T T T 11 220-239 12 12 12 27 27 27 F F F 12 240-259 16 16 16 33 33 33 F F F 13 260-279 17 17 17 35 35 35 F F F 14 280-299 17 17 17 35 35 35 F F F 15 300-319 18 18 18 38 38 38 F F F 16 320-339 18 18 18 38 38 38 F F F 17 340-359 14 14 14 31 31 31 T T T 18 360-379 14 14 14 31 31 31 T T T 19 380-399 17 17 17 35 35 35 F F F 20 400-419 17 17 17 35 35 35 F F F 21 420-439 14 14 14 31 31 31 F F F 22 440-459 13 13 13 29 29 29 F F F 23 460-479 14 14 14 31 31 31 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 17 17 17 35 35 35 F F F 26 520-539 16 16 16 33 33 33 F F F 27 540-559 16 16 16 34 34 34 F F F 28 560-579 18 18 18 38 38 38 F F F 29 580-599 18 18 18 38 38 38 F F F Max gen: 18, Max flush: 38 A Difference in extent metadata is expected here On loop 9 Verifying data now No change after verify Loop: 9 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 29 29 29 F F F 1 020-039 16 16 16 34 34 34 F F F 2 040-059 10 10 10 24 24 24 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 17 17 17 35 35 35 F F F 5 100-119 16 16 16 34 34 34 F F F 6 120-139 19 19 19 39 39 39 F F F 7 140-159 19 19 19 39 39 39 F F F 8 160-179 13 13 13 29 29 29 F F F 9 180-199 16 16 16 34 34 34 F F F 10 200-219 19 19 19 39 39 39 F F F 11 220-239 12 12 12 27 27 27 F F F 12 240-259 16 16 16 33 33 33 F F F 13 260-279 17 17 17 35 35 35 F F F 14 280-299 17 17 17 35 35 35 F F F 15 300-319 18 18 18 38 38 38 F F F 16 320-339 18 18 18 38 38 38 F F F 17 340-359 19 19 19 39 39 39 F F F 18 360-379 19 19 19 39 39 39 F F F 19 380-399 17 17 17 35 35 35 F F F 20 400-419 17 17 17 35 35 35 F F F 21 420-439 14 14 14 31 31 31 F F F 22 440-459 13 13 13 29 29 29 F F F 23 460-479 14 14 14 31 31 31 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 17 17 17 35 35 35 F F F 26 520-539 16 16 16 33 33 33 F F F 27 540-559 16 16 16 34 34 34 F F F 28 560-579 18 18 18 38 38 38 F F F 29 580-599 18 18 18 38 38 38 F F F Max gen: 19, Max flush: 39 Begin loop 10 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.044103828Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.044586465Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.044616274Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262} {"msg":"Crucible 5f5409b7-489b-4df5-a77a-96eac1258afe has session id: cf17b760-deae-44ad-81ba-665e918d980e","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.044767047Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"Upstairs opts: Upstairs UUID: 5f5409b7-489b-4df5-a77a-96eac1258afe, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.044810815Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"Crucible stats registered with UUID: 5f5409b7-489b-4df5-a77a-96eac1258afe","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.044835474Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:20","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.045120291Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262} {"msg":"5f5409b7-489b-4df5-a77a-96eac1258afe active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.045189737Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.045214366Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"0","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.045234645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"0","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.045248775Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.045262794Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.045280023Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"2","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.045293583Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"2","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.045312152Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"io task","client":"2","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.045328321Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"io task","client":"0","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.04534705Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"io task","client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"ds_connection connected from Ok(127.0.0.1:45432)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.0455641Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"io task","client":"0","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"ds_connection connected from Ok(127.0.0.1:58205)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.045616148Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"io task","client":"2","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:30:51.045799309Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"io task","client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:30:51.045839017Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"io task","client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:30:51.045861336Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.045874855Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.045896604Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.045951822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"io task","client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.046504326Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"2","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.046531855Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"2","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.046550414Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"0","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.046564293Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"0","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.047143506Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"2","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.047263541Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"0","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.047662092Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"2","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.047688141Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"2","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.04770301Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.047789406Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"0","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.047808265Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"0","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:30:51.047821765Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.046922888Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"io task","client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"ds_connection connected from Ok(127.0.0.1:65100)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.046986465Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"io task","client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.048103923Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.048150541Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.048789871Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049378084Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049405243Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"[0]R flush_numbers[0..12]: [29, 34, 24, 35, 35, 34, 39, 39, 29, 34, 39, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049420642Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"downstairs","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"[0]R generation[0..12]: [13, 16, 10, 17, 17, 16, 19, 19, 13, 16, 19, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049434871Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"downstairs","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049449281Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"downstairs","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"[1]R flush_numbers[0..12]: [29, 34, 24, 35, 35, 34, 39, 39, 29, 34, 39, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.04946438Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"downstairs","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"[1]R generation[0..12]: [13, 16, 10, 17, 17, 16, 19, 19, 13, 16, 19, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049478869Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"downstairs","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049493108Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"downstairs","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"[2]R flush_numbers[0..12]: [29, 34, 24, 35, 35, 34, 39, 39, 29, 34, 39, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049506728Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"downstairs","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"[2]R generation[0..12]: [13, 16, 10, 17, 17, 16, 19, 19, 13, 16, 19, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049520097Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"downstairs","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049534287Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"downstairs","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"Max found gen is 20","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049547166Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"downstairs","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"Generation requested: 20 >= found:20","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:31:01.049560325Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"downstairs","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"Next flush: 40","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049573365Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"downstairs","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049590284Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"":"downstairs","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049603913Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049615953Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"0","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049629182Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"1","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049642372Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"client":"2","":"downstairs","session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049656741Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.04966913Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"5f5409b7-489b-4df5-a77a-96eac1258afe is now active with session: cf17b760-deae-44ad-81ba-665e918d980e","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.04968195Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049693839Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262,"session_id":"cf17b760-deae-44ad-81ba-665e918d980e"} {"msg":"The guest has finished waiting for activation with:20","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:01.049705998Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 510 len 40960 data: 3 2 2 2 2 2 2 2 3 3 02/30 Read block 410 len 8192 03/30 Flush 04/30 Write block 423 len 28672 data: 3 2 2 3 3 3 3 05/30 Write block 411 len 36864 data: 2 2 2 2 2 3 3 3 4 06/30 Read block 571 len 16384 07/30 Write block 312 len 20480 data: 3 3 3 3 2 08/30 Write block 246 len 8192 data: 4 4 09/30 Flush 10/30 Flush 11/30 Write block 147 len 32768 data: 2 3 3 3 3 3 3 2 12/30 Write block 120 len 24576 data: 2 2 2 3 2 3 13/30 Flush 14/30 Write block 365 len 32768 data: 4 4 5 6 5 5 5 4 15/30 Read block 470 len 36864 16/30 Read block 484 len 28672 17/30 Write block 587 len 36864 data: 4 5 4 4 3 3 2 2 2 18/30 Write block 431 len 36864 data: 3 4 4 3 4 4 4 4 4 19/30 Read block 31 len 40960 20/30 Flush 21/30 Write block 428 len 12288 data: 4 4 3 22/30 Read block 113 len 28672 23/30 Read block 122 len 36864 24/30 Flush 25/30 Write block 396 len 28672 data: 7 7 6 4 3 3 2 26/30 Write block 183 len 4096 data: 3 27/30 Read block 102 len 4096 28/30 Write block 26 len 20480 data: 5 5 5 6 4 29/30 Write block 588 len 4096 data: 6 30/30 Read block 395 len 24576 ---------------------------------------------------------------- Crucible gen:20 GIO:true work queues: Upstairs:0 downstairs:6 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 26 Acked 1025 Write 7 Done Done Done false 27 Acked 1026 Write 1 Done Done Done false 28 Acked 1027 Read 1 Done Done Done false 29 Acked 1028 Write 5 Done Done Done false 30 Acked 1029 Write 1 Done Done Done false 31 Acked 1030 Read 6 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 1 1 Done 6 6 5 17 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1024 1024 1024 Downstairs last five completed: 1024 1023 1022 1021 1020 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:31:01.147363982Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1262} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 29 29 29 F F F 1 020-039 16 16 16 34 34 34 T T T 2 040-059 10 10 10 24 24 24 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 17 17 17 35 35 35 F F F 5 100-119 16 16 16 34 34 34 F F F 6 120-139 20 20 20 44 44 44 F F F 7 140-159 20 20 20 44 44 44 F F F 8 160-179 13 13 13 29 29 29 F F F 9 180-199 16 16 16 34 34 34 T T T 10 200-219 19 19 19 39 39 39 F F F 11 220-239 12 12 12 27 27 27 F F F 12 240-259 20 20 20 42 42 42 F F F 13 260-279 17 17 17 35 35 35 F F F 14 280-299 17 17 17 35 35 35 F F F 15 300-319 20 20 20 41 41 41 F F F 16 320-339 18 18 18 38 38 38 F F F 17 340-359 19 19 19 39 39 39 F F F 18 360-379 20 20 20 45 45 45 F F F 19 380-399 17 17 17 35 35 35 T T T 20 400-419 20 20 20 41 41 41 T T T 21 420-439 20 20 20 46 46 46 F F F 22 440-459 13 13 13 29 29 29 F F F 23 460-479 14 14 14 31 31 31 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 20 20 20 40 40 40 F F F 26 520-539 16 16 16 33 33 33 F F F 27 540-559 16 16 16 34 34 34 F F F 28 560-579 18 18 18 38 38 38 F F F 29 580-599 20 20 20 45 45 45 T T T Max gen: 20, Max flush: 46 A Difference in extent metadata is expected here On loop 10 Verifying data now No change after verify Loop: 10 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 29 29 29 F F F 1 020-039 21 21 21 47 47 47 F F F 2 040-059 10 10 10 24 24 24 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 17 17 17 35 35 35 F F F 5 100-119 16 16 16 34 34 34 F F F 6 120-139 20 20 20 44 44 44 F F F 7 140-159 20 20 20 44 44 44 F F F 8 160-179 13 13 13 29 29 29 F F F 9 180-199 21 21 21 47 47 47 F F F 10 200-219 19 19 19 39 39 39 F F F 11 220-239 12 12 12 27 27 27 F F F 12 240-259 20 20 20 42 42 42 F F F 13 260-279 17 17 17 35 35 35 F F F 14 280-299 17 17 17 35 35 35 F F F 15 300-319 20 20 20 41 41 41 F F F 16 320-339 18 18 18 38 38 38 F F F 17 340-359 19 19 19 39 39 39 F F F 18 360-379 20 20 20 45 45 45 F F F 19 380-399 21 21 21 47 47 47 F F F 20 400-419 21 21 21 47 47 47 F F F 21 420-439 20 20 20 46 46 46 F F F 22 440-459 13 13 13 29 29 29 F F F 23 460-479 14 14 14 31 31 31 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 20 20 20 40 40 40 F F F 26 520-539 16 16 16 33 33 33 F F F 27 540-559 16 16 16 34 34 34 F F F 28 560-579 18 18 18 38 38 38 F F F 29 580-599 21 21 21 47 47 47 F F F Max gen: 21, Max flush: 47 Begin loop 11 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.824321026Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.82468089Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.824715528Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270} {"msg":"Crucible 14a8db2b-25ee-4614-9088-026003a26414 has session id: 25a88033-54b7-418b-9d91-227eb629444a","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.82489371Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"Upstairs opts: Upstairs UUID: 14a8db2b-25ee-4614-9088-026003a26414, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.824945888Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"Crucible stats registered with UUID: 14a8db2b-25ee-4614-9088-026003a26414","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.824967516Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"session_id":"25a88033-54b7-418b-9d91-227eb629444a"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:22","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.825362968Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270} {"msg":"14a8db2b-25ee-4614-9088-026003a26414 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.825400426Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.825414375Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"0","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.825434065Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"0","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.825447694Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.825461983Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.825474993Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"2","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.825496232Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"2","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.825561039Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"io task","client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.825581478Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"io task","client":"0","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.825601477Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"io task","client":"2","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:31:11.825731111Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"io task","client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:31:11.825765349Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"io task","client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:31:11.825796148Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.825820117Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.825842695Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"ds_connection connected from Ok(127.0.0.1:44056)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.825865334Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"io task","client":"0","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"ds_connection connected from Ok(127.0.0.1:43868)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.825889144Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"io task","client":"2","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.826142032Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"io task","client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.826688996Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"0","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.826722465Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"0","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.826749723Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"2","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.82682296Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"2","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.827285918Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"0","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.827395853Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"2","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.827863671Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"0","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.82788653Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"0","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.827901159Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.827962537Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"2","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.827980766Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"2","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:11.827994365Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.827395756Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"io task","client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"ds_connection connected from Ok(127.0.0.1:46461)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.827687893Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"io task","client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.828583571Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.828610199Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829168573Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829747097Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829778875Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"[0]R flush_numbers[0..12]: [29, 47, 24, 35, 35, 34, 44, 44, 29, 47, 39, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829795444Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"downstairs","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"[0]R generation[0..12]: [13, 21, 10, 17, 17, 16, 20, 20, 13, 21, 19, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829810094Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"downstairs","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829824443Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"downstairs","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"[1]R flush_numbers[0..12]: [29, 47, 24, 35, 35, 34, 44, 44, 29, 47, 39, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829838212Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"downstairs","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"[1]R generation[0..12]: [13, 21, 10, 17, 17, 16, 20, 20, 13, 21, 19, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829851822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"downstairs","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829865781Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"downstairs","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"[2]R flush_numbers[0..12]: [29, 47, 24, 35, 35, 34, 44, 44, 29, 47, 39, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.82987942Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"downstairs","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"[2]R generation[0..12]: [13, 21, 10, 17, 17, 16, 20, 20, 13, 21, 19, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.82989308Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"downstairs","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829907169Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"downstairs","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"Max found gen is 22","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829920109Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"downstairs","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"Generation requested: 22 >= found:22","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829933348Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"downstairs","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"Next flush: 48","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829946117Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"downstairs","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} Wait for a query_work_queue command to finish before sending IO {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829962476Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"":"downstairs","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829978706Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.829991215Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"0","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.830004634Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"1","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.830017914Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"client":"2","":"downstairs","session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.830031193Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.830043363Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"14a8db2b-25ee-4614-9088-026003a26414 is now active with session: 25a88033-54b7-418b-9d91-227eb629444a","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.830056302Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.830068362Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270,"session_id":"25a88033-54b7-418b-9d91-227eb629444a"} {"msg":"The guest has finished waiting for activation with:22","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:21.830080801Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 319 len 24576 data: 3 3 3 3 3 2 02/30 Read block 374 len 8192 03/30 Read block 356 len 20480 04/30 Write block 301 len 16384 data: 3 3 2 2 05/30 Write block 356 len 12288 data: 6 6 5 06/30 Write block 448 len 8192 data: 2 2 07/30 Read block 42 len 36864 08/30 Flush 09/30 Write block 80 len 8192 data: 4 4 10/30 Read block 244 len 4096 11/30 Write block 526 len 4096 data: 2 12/30 Flush 13/30 Write block 169 len 20480 data: 4 5 5 6 6 14/30 Read block 230 len 32768 15/30 Read block 226 len 16384 16/30 Read block 360 len 16384 17/30 Flush 18/30 Write block 120 len 4096 data: 3 19/30 Write block 167 len 16384 data: 4 4 5 6 20/30 Read block 343 len 32768 21/30 Read block 269 len 20480 22/30 Read block 271 len 40960 23/30 Read block 456 len 32768 24/30 Read block 5 len 4096 25/30 Write block 390 len 12288 data: 5 5 8 26/30 Write block 581 len 24576 data: 4 5 4 3 3 4 27/30 Write block 54 len 4096 data: 2 28/30 Read block 238 len 12288 29/30 Write block 8 len 4096 data: 4 30/30 Read block 384 len 36864 ---------------------------------------------------------------- Crucible gen:22 GIO:true work queues: Upstairs:0 downstairs:13 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 18 Acked 1017 Write 1 Done Done Done false 19 Acked 1018 Write 4 Done Done Done false 20 Acked 1019 Read 8 Done Done Done false 21 Acked 1020 Read 5 Done Done Done false 22 Acked 1021 Read 10 Done Done Done false 23 Acked 1022 Read 8 Done Done Done false 24 Acked 1023 Read 1 Done Done Done false 25 Acked 1024 Write 3 Done Done Done false 26 Acked 1025 Write 6 Done Done Done false 27 Acked 1026 Write 1 Done Done Done false 28 Acked 1027 Read 3 Done Done Done false 29 Acked 1028 Write 1 Done Done Sent false 30 Acked 1029 Read 9 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 2 3 Done 13 12 11 36 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1016 1016 1016 Downstairs last five completed: 1016 1015 1014 1013 1012 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:31:21.873927985Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1270} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 29 29 29 T T T 1 020-039 21 21 21 47 47 47 F F F 2 040-059 10 10 10 24 24 24 T T T 3 060-079 17 17 17 35 35 35 F F F 4 080-099 22 22 22 49 49 49 F F F 5 100-119 16 16 16 34 34 34 F F F 6 120-139 20 20 20 44 44 44 T T T 7 140-159 20 20 20 44 44 44 F F F 8 160-179 22 22 22 50 50 50 T T T 9 180-199 21 21 21 47 47 47 F F F 10 200-219 19 19 19 39 39 39 F F F 11 220-239 12 12 12 27 27 27 F F F 12 240-259 20 20 20 42 42 42 F F F 13 260-279 17 17 17 35 35 35 F F F 14 280-299 17 17 17 35 35 35 F F F 15 300-319 22 22 22 48 48 48 F F F 16 320-339 22 22 22 48 48 48 F F F 17 340-359 22 22 22 48 48 48 F F F 18 360-379 20 20 20 45 45 45 F F F 19 380-399 21 21 21 47 47 47 T T T 20 400-419 21 21 21 47 47 47 F F F 21 420-439 20 20 20 46 46 46 F F F 22 440-459 22 22 22 48 48 48 F F F 23 460-479 14 14 14 31 31 31 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 20 20 20 40 40 40 F F F 26 520-539 22 22 22 49 49 49 F F F 27 540-559 16 16 16 34 34 34 F F F 28 560-579 18 18 18 38 38 38 F F F 29 580-599 21 21 21 47 47 47 T T T Max gen: 22, Max flush: 50 A Difference in extent metadata is expected here On loop 11 Verifying data now No change after verify Loop: 11 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 51 51 51 F F F 1 020-039 21 21 21 47 47 47 F F F 2 040-059 23 23 23 51 51 51 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 22 22 22 49 49 49 F F F 5 100-119 16 16 16 34 34 34 F F F 6 120-139 23 23 23 51 51 51 F F F 7 140-159 20 20 20 44 44 44 F F F 8 160-179 23 23 23 51 51 51 F F F 9 180-199 21 21 21 47 47 47 F F F 10 200-219 19 19 19 39 39 39 F F F 11 220-239 12 12 12 27 27 27 F F F 12 240-259 20 20 20 42 42 42 F F F 13 260-279 17 17 17 35 35 35 F F F 14 280-299 17 17 17 35 35 35 F F F 15 300-319 22 22 22 48 48 48 F F F 16 320-339 22 22 22 48 48 48 F F F 17 340-359 22 22 22 48 48 48 F F F 18 360-379 20 20 20 45 45 45 F F F 19 380-399 23 23 23 51 51 51 F F F 20 400-419 21 21 21 47 47 47 F F F 21 420-439 20 20 20 46 46 46 F F F 22 440-459 22 22 22 48 48 48 F F F 23 460-479 14 14 14 31 31 31 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 20 20 20 40 40 40 F F F 26 520-539 22 22 22 49 49 49 F F F 27 540-559 16 16 16 34 34 34 F F F 28 560-579 18 18 18 38 38 38 F F F 29 580-599 23 23 23 51 51 51 F F F Max gen: 23, Max flush: 51 Begin loop 12 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.87832226Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.878804478Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.878838927Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278} {"msg":"Crucible 67e51be3-f8be-4bd6-add8-90b8fca1c840 has session id: 14a4efa9-9496-430f-a65b-c1640fb323ff","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.878932722Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"Upstairs opts: Upstairs UUID: 67e51be3-f8be-4bd6-add8-90b8fca1c840, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.87896976Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"Crucible stats registered with UUID: 67e51be3-f8be-4bd6-add8-90b8fca1c840","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.878992449Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:24","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.879392561Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278} {"msg":"67e51be3-f8be-4bd6-add8-90b8fca1c840 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.879467067Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.879486396Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.879506795Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.879520845Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"1","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.879534484Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"1","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.879547863Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"2","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.879565693Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"2","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.879579402Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"io task","client":"2","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.879593241Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"io task","client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.87960684Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"io task","client":"1","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"ds_connection connected from Ok(127.0.0.1:55286)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.879771633Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"io task","client":"2","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"ds_connection connected from Ok(127.0.0.1:57404)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.879964384Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"io task","client":"1","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:31:36.880086428Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"io task","client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:31:36.880127396Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"io task","client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:31:36.880155055Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.880169704Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.880183274Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.88026908Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"io task","client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.880849313Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"1","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.880884221Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"1","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.880916139Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"2","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.880945258Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"2","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.881524011Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"2","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.88155217Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"1","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.882068626Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"2","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.882093415Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"2","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.882108274Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.882169811Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"1","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.88218877Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"1","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:36.88220257Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.881569411Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"io task","client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"ds_connection connected from Ok(127.0.0.1:56518)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.88181629Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"io task","client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.882756556Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.882790715Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.883402566Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.883934711Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.88396098Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"[0]R flush_numbers[0..12]: [51, 47, 51, 35, 49, 34, 51, 44, 51, 47, 39, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.883976109Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"downstairs","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"[0]R generation[0..12]: [23, 21, 23, 17, 22, 16, 23, 20, 23, 21, 19, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.883990588Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"downstairs","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884004878Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"downstairs","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"[1]R flush_numbers[0..12]: [51, 47, 51, 35, 49, 34, 51, 44, 51, 47, 39, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884018617Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"downstairs","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"[1]R generation[0..12]: [23, 21, 23, 17, 22, 16, 23, 20, 23, 21, 19, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884032186Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"downstairs","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884046176Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"downstairs","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"[2]R flush_numbers[0..12]: [51, 47, 51, 35, 49, 34, 51, 44, 51, 47, 39, 27]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884059945Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"downstairs","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"[2]R generation[0..12]: [23, 21, 23, 17, 22, 16, 23, 20, 23, 21, 19, 12]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884073595Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"downstairs","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884087804Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"downstairs","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"Max found gen is 24","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884100763Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"downstairs","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"Generation requested: 24 >= found:24","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884113823Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"downstairs","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"Next flush: 52","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:31:46.884126642Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"downstairs","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884139921Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"":"downstairs","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884155121Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.88416721Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"0","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884180499Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"1","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884193869Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"client":"2","":"downstairs","session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884207028Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884219188Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"67e51be3-f8be-4bd6-add8-90b8fca1c840 is now active with session: 14a4efa9-9496-430f-a65b-c1640fb323ff","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884231797Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884243677Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278,"session_id":"14a4efa9-9496-430f-a65b-c1640fb323ff"} {"msg":"The guest has finished waiting for activation with:24","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:31:46.884255596Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 445 len 16384 data: 3 2 2 3 02/30 Read block 46 len 36864 03/30 Write block 101 len 20480 data: 2 2 2 2 2 04/30 Write block 244 len 28672 data: 2 3 5 5 4 4 4 05/30 Read block 132 len 28672 06/30 Read block 8 len 24576 07/30 Read block 534 len 16384 08/30 Read block 582 len 12288 09/30 Write block 219 len 20480 data: 4 4 5 5 3 10/30 Read block 594 len 16384 11/30 Flush 12/30 Flush 13/30 Write block 561 len 8192 data: 3 3 14/30 Write block 550 len 28672 data: 2 2 2 4 4 4 4 15/30 Read block 401 len 24576 16/30 Read block 437 len 20480 17/30 Flush 18/30 Write block 350 len 24576 data: 4 4 4 5 5 5 19/30 Read block 415 len 4096 20/30 Read block 101 len 36864 21/30 Read block 479 len 4096 22/30 Write block 462 len 24576 data: 3 2 2 2 2 2 23/30 Write block 98 len 36864 data: 3 3 3 3 3 3 3 3 2 24/30 Read block 259 len 12288 25/30 Read block 72 len 28672 26/30 Flush 27/30 Write block 544 len 32768 data: 3 3 3 2 2 2 3 3 28/30 Read block 564 len 16384 29/30 Write block 405 len 36864 data: 4 5 5 4 3 3 3 3 3 30/30 Read block 217 len 32768 ---------------------------------------------------------------- Crucible gen:24 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 8 Done Done Done false 28 Acked 1027 Read 4 Done Done Done false 29 Acked 1028 Write 9 Done Done Done false 30 Acked 1029 Read 8 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:31:46.976490102Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1278} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 51 51 51 F F F 1 020-039 21 21 21 47 47 47 F F F 2 040-059 23 23 23 51 51 51 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 24 24 24 55 55 55 F F F 5 100-119 24 24 24 55 55 55 F F F 6 120-139 23 23 23 51 51 51 F F F 7 140-159 20 20 20 44 44 44 F F F 8 160-179 23 23 23 51 51 51 F F F 9 180-199 21 21 21 47 47 47 F F F 10 200-219 24 24 24 52 52 52 F F F 11 220-239 24 24 24 52 52 52 F F F 12 240-259 24 24 24 52 52 52 F F F 13 260-279 17 17 17 35 35 35 F F F 14 280-299 17 17 17 35 35 35 F F F 15 300-319 22 22 22 48 48 48 F F F 16 320-339 22 22 22 48 48 48 F F F 17 340-359 24 24 24 55 55 55 F F F 18 360-379 20 20 20 45 45 45 F F F 19 380-399 23 23 23 51 51 51 F F F 20 400-419 21 21 21 47 47 47 T T T 21 420-439 20 20 20 46 46 46 F F F 22 440-459 24 24 24 52 52 52 F F F 23 460-479 24 24 24 55 55 55 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 20 20 20 40 40 40 F F F 26 520-539 22 22 22 49 49 49 F F F 27 540-559 24 24 24 54 54 54 T T T 28 560-579 24 24 24 54 54 54 F F F 29 580-599 23 23 23 51 51 51 F F F Max gen: 24, Max flush: 55 A Difference in extent metadata is expected here On loop 12 Verifying data now No change after verify Loop: 12 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 51 51 51 F F F 1 020-039 21 21 21 47 47 47 F F F 2 040-059 23 23 23 51 51 51 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 24 24 24 55 55 55 F F F 5 100-119 24 24 24 55 55 55 F F F 6 120-139 23 23 23 51 51 51 F F F 7 140-159 20 20 20 44 44 44 F F F 8 160-179 23 23 23 51 51 51 F F F 9 180-199 21 21 21 47 47 47 F F F 10 200-219 24 24 24 52 52 52 F F F 11 220-239 24 24 24 52 52 52 F F F 12 240-259 24 24 24 52 52 52 F F F 13 260-279 17 17 17 35 35 35 F F F 14 280-299 17 17 17 35 35 35 F F F 15 300-319 22 22 22 48 48 48 F F F 16 320-339 22 22 22 48 48 48 F F F 17 340-359 24 24 24 55 55 55 F F F 18 360-379 20 20 20 45 45 45 F F F 19 380-399 23 23 23 51 51 51 F F F 20 400-419 25 25 25 56 56 56 F F F 21 420-439 20 20 20 46 46 46 F F F 22 440-459 24 24 24 52 52 52 F F F 23 460-479 24 24 24 55 55 55 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 20 20 20 40 40 40 F F F 26 520-539 22 22 22 49 49 49 F F F 27 540-559 25 25 25 56 56 56 F F F 28 560-579 24 24 24 54 54 54 F F F 29 580-599 23 23 23 51 51 51 F F F Max gen: 25, Max flush: 56 Begin loop 13 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.42873876Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.429256016Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.429288625Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286} {"msg":"Crucible 03299e3c-72d9-4d16-b8ac-65409b1fb08d has session id: add0753c-7e94-4329-8569-5124ba6f6b77","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.42939747Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"Upstairs opts: Upstairs UUID: 03299e3c-72d9-4d16-b8ac-65409b1fb08d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.429435418Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"Crucible stats registered with UUID: 03299e3c-72d9-4d16-b8ac-65409b1fb08d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.429460487Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:26","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.429895346Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286} {"msg":"03299e3c-72d9-4d16-b8ac-65409b1fb08d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.430008301Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.43002618Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.430047099Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.430061139Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"1","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.430074738Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"1","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.430090757Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"2","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.430115066Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"2","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.430138835Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"io task","client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.430153564Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"io task","client":"1","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.430168383Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"io task","client":"2","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:32:01.430339806Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"io task","client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"ds_connection connected from Ok(127.0.0.1:47374)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.430377864Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"io task","client":"1","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:32:01.430429961Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"io task","client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:32:01.430518487Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.430547886Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.430571205Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"ds_connection connected from Ok(127.0.0.1:40731)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.430594424Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"io task","client":"2","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.430609313Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"io task","client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.431136248Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"2","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.431164777Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"2","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.431181066Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"1","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.431195036Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"1","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.431786228Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"1","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.431813747Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"2","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.432374191Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"2","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.432401569Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"2","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.432416409Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.432428828Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"1","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.432442577Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"1","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:01.432455737Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.430809137Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"io task","client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"ds_connection connected from Ok(127.0.0.1:59748)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.43094574Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"io task","client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.4324566Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.432492218Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433045812Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433653374Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433673943Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"[0]R flush_numbers[0..12]: [51, 47, 51, 35, 55, 55, 51, 44, 51, 47, 52, 52]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433693022Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"downstairs","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"[0]R generation[0..12]: [23, 21, 23, 17, 24, 24, 23, 20, 23, 21, 24, 24]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433708141Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"downstairs","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433722631Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"downstairs","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"[1]R flush_numbers[0..12]: [51, 47, 51, 35, 55, 55, 51, 44, 51, 47, 52, 52]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.43373662Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"downstairs","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"[1]R generation[0..12]: [23, 21, 23, 17, 24, 24, 23, 20, 23, 21, 24, 24]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.43375036Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"downstairs","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433764339Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"downstairs","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"[2]R flush_numbers[0..12]: [51, 47, 51, 35, 55, 55, 51, 44, 51, 47, 52, 52]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433778068Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"downstairs","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"[2]R generation[0..12]: [23, 21, 23, 17, 24, 24, 23, 20, 23, 21, 24, 24]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433791887Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"downstairs","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433805917Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"downstairs","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"Max found gen is 26","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433818836Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"downstairs","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"Generation requested: 26 >= found:26","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433831916Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"downstairs","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"Next flush: 57","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:32:11.433844805Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"downstairs","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433857815Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"":"downstairs","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433870944Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433884783Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"0","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433898113Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"1","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433911052Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"client":"2","":"downstairs","session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433924201Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433936431Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"03299e3c-72d9-4d16-b8ac-65409b1fb08d is now active with session: add0753c-7e94-4329-8569-5124ba6f6b77","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.43394932Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433961239Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286,"session_id":"add0753c-7e94-4329-8569-5124ba6f6b77"} {"msg":"The guest has finished waiting for activation with:26","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:11.433973469Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 548 len 32768 data: 3 3 4 4 3 5 5 5 02/30 Write block 586 len 4096 data: 5 03/30 Read block 346 len 24576 04/30 Read block 241 len 16384 05/30 Read block 256 len 8192 06/30 Read block 417 len 4096 07/30 Read block 408 len 28672 08/30 Read block 264 len 12288 09/30 Read block 436 len 32768 10/30 Write block 303 len 4096 data: 3 11/30 Write block 533 len 28672 data: 2 2 2 3 3 3 3 12/30 Flush 13/30 Write block 269 len 12288 data: 4 3 3 14/30 Read block 427 len 32768 15/30 Read block 62 len 40960 16/30 Read block 118 len 28672 17/30 Read block 278 len 12288 18/30 Read block 586 len 40960 19/30 Write block 34 len 4096 data: 3 20/30 Flush 21/30 Write block 563 len 24576 data: 4 4 4 4 3 2 22/30 Read block 255 len 12288 23/30 Write block 84 len 36864 data: 4 4 2 2 2 3 3 3 2 24/30 Write block 564 len 40960 data: 5 5 5 4 3 2 2 3 3 3 25/30 Read block 180 len 32768 26/30 Read block 127 len 28672 27/30 Read block 197 len 40960 28/30 Read block 256 len 20480 29/30 Write block 323 len 24576 data: 4 3 3 2 2 3 30/30 Write block 142 len 8192 data: 2 2 ---------------------------------------------------------------- Crucible gen:26 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 21 Acked 1020 Write 6 Done Done Done false 22 Acked 1021 Read 3 Done Done Done false 23 Acked 1022 Write 9 Done Done Done false 24 Acked 1023 Write 10 Done Done Done false 25 Acked 1024 Read 8 Done Done Done false 26 Acked 1025 Read 7 Done Done Done false 27 Acked 1026 Read 10 Done Done Done false 28 Acked 1027 Read 5 Done Done Done false 29 Acked 1028 Write 6 Sent Sent Sent false 30 Acked 1029 Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 2 2 2 6 Done 8 8 8 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1019 1019 1019 Downstairs last five completed: 1019 1018 1017 1016 1015 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:32:11.590694366Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1286} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 51 51 51 F F F 1 020-039 26 26 26 58 58 58 F F F 2 040-059 23 23 23 51 51 51 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 24 24 24 55 55 55 T T T 5 100-119 24 24 24 55 55 55 F F F 6 120-139 23 23 23 51 51 51 F F F 7 140-159 20 20 20 44 44 44 T T T 8 160-179 23 23 23 51 51 51 F F F 9 180-199 21 21 21 47 47 47 F F F 10 200-219 24 24 24 52 52 52 F F F 11 220-239 24 24 24 52 52 52 F F F 12 240-259 24 24 24 52 52 52 F F F 13 260-279 26 26 26 58 58 58 F F F 14 280-299 17 17 17 35 35 35 F F F 15 300-319 26 26 26 57 57 57 F F F 16 320-339 22 22 22 48 48 48 T T T 17 340-359 24 24 24 55 55 55 F F F 18 360-379 20 20 20 45 45 45 F F F 19 380-399 23 23 23 51 51 51 F F F 20 400-419 25 25 25 56 56 56 F F F 21 420-439 20 20 20 46 46 46 F F F 22 440-459 24 24 24 52 52 52 F F F 23 460-479 24 24 24 55 55 55 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 20 20 20 40 40 40 F F F 26 520-539 26 26 26 57 57 57 F F F 27 540-559 26 26 26 57 57 57 F F F 28 560-579 24 24 24 54 54 54 T T T 29 580-599 26 26 26 57 57 57 F F F Max gen: 26, Max flush: 58 A Difference in extent metadata is expected here On loop 13 Verifying data now No change after verify Loop: 13 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 51 51 51 F F F 1 020-039 26 26 26 58 58 58 F F F 2 040-059 23 23 23 51 51 51 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 27 27 27 59 59 59 F F F 5 100-119 24 24 24 55 55 55 F F F 6 120-139 23 23 23 51 51 51 F F F 7 140-159 27 27 27 59 59 59 F F F 8 160-179 23 23 23 51 51 51 F F F 9 180-199 21 21 21 47 47 47 F F F 10 200-219 24 24 24 52 52 52 F F F 11 220-239 24 24 24 52 52 52 F F F 12 240-259 24 24 24 52 52 52 F F F 13 260-279 26 26 26 58 58 58 F F F 14 280-299 17 17 17 35 35 35 F F F 15 300-319 26 26 26 57 57 57 F F F 16 320-339 27 27 27 59 59 59 F F F 17 340-359 24 24 24 55 55 55 F F F 18 360-379 20 20 20 45 45 45 F F F 19 380-399 23 23 23 51 51 51 F F F 20 400-419 25 25 25 56 56 56 F F F 21 420-439 20 20 20 46 46 46 F F F 22 440-459 24 24 24 52 52 52 F F F 23 460-479 24 24 24 55 55 55 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 20 20 20 40 40 40 F F F 26 520-539 26 26 26 57 57 57 F F F 27 540-559 26 26 26 57 57 57 F F F 28 560-579 27 27 27 59 59 59 F F F 29 580-599 26 26 26 57 57 57 F F F Max gen: 27, Max flush: 59 Begin loop 14 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.327043117Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.327685397Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.327716276Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294} {"msg":"Crucible 279c2e10-8e65-4f03-9099-89cadc760e29 has session id: f8d7cb36-d94a-4311-bb17-dcd12661e45c","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.327798472Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"Upstairs opts: Upstairs UUID: 279c2e10-8e65-4f03-9099-89cadc760e29, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.32784045Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"Crucible stats registered with UUID: 279c2e10-8e65-4f03-9099-89cadc760e29","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.327867519Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:28","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.328147786Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294} {"msg":"279c2e10-8e65-4f03-9099-89cadc760e29 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.328176725Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.328197453Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"0","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.328223032Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"0","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.328244691Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.32826583Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.328286439Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"2","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.328307678Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"2","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.328332467Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"io task","client":"2","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.328355396Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"io task","client":"0","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.328376775Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"io task","client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:32:26.328595175Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"io task","client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"ds_connection connected from Ok(127.0.0.1:53242)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.328680811Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"io task","client":"2","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:32:26.32870103Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"io task","client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:32:26.328717339Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.328730089Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.328742988Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.328764497Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"io task","client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"ds_connection connected from Ok(127.0.0.1:46128)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.328847293Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"io task","client":"0","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.329295022Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"2","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.329320691Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"2","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.329464764Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"0","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.329484453Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"0","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.329853356Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"2","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.329968241Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"0","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.330352633Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"2","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.330382962Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"2","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.330398351Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.330425149Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"0","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.330439399Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"0","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:26.330452538Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.329823902Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"io task","client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"ds_connection connected from Ok(127.0.0.1:37939)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.330021343Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"io task","client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.331050235Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.331079944Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.331546602Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332165233Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332198592Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"[0]R flush_numbers[0..12]: [51, 58, 51, 35, 59, 55, 51, 59, 51, 47, 52, 52]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.33222514Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"downstairs","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"[0]R generation[0..12]: [23, 26, 23, 17, 27, 24, 23, 27, 23, 21, 24, 24]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332244129Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"downstairs","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332262098Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"downstairs","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"[1]R flush_numbers[0..12]: [51, 58, 51, 35, 59, 55, 51, 59, 51, 47, 52, 52]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332278978Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"downstairs","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"[1]R generation[0..12]: [23, 26, 23, 17, 27, 24, 23, 27, 23, 21, 24, 24]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332295917Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"downstairs","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332313106Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"downstairs","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"[2]R flush_numbers[0..12]: [51, 58, 51, 35, 59, 55, 51, 59, 51, 47, 52, 52]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332330185Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"downstairs","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"[2]R generation[0..12]: [23, 26, 23, 17, 27, 24, 23, 27, 23, 21, 24, 24]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:32:36.332346714Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"downstairs","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332368424Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"downstairs","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"Max found gen is 28","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332384883Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"downstairs","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"Generation requested: 28 >= found:28","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332401362Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"downstairs","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"Next flush: 60","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332417301Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"downstairs","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.33243331Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"":"downstairs","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.33244951Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332466539Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"0","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332483508Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"1","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332499867Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"client":"2","":"downstairs","session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332516327Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332531466Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"279c2e10-8e65-4f03-9099-89cadc760e29 is now active with session: f8d7cb36-d94a-4311-bb17-dcd12661e45c","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332547195Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332562104Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294,"session_id":"f8d7cb36-d94a-4311-bb17-dcd12661e45c"} {"msg":"The guest has finished waiting for activation with:28","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:36.332577084Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 120 len 36864 data: 4 3 3 4 3 4 3 3 3 02/30 Read block 252 len 4096 03/30 Write block 161 len 4096 data: 3 04/30 Write block 183 len 32768 data: 4 3 4 3 3 2 2 2 05/30 Write block 248 len 40960 data: 5 5 5 3 4 3 5 5 5 4 06/30 Flush 07/30 Write block 327 len 36864 data: 3 4 3 3 3 3 4 6 6 08/30 Read block 441 len 8192 09/30 Write block 426 len 32768 data: 4 4 5 5 4 4 5 5 10/30 Read block 429 len 24576 11/30 Flush 12/30 Write block 400 len 4096 data: 4 13/30 Flush 14/30 Write block 43 len 28672 data: 3 3 2 2 2 2 2 15/30 Flush 16/30 Write block 318 len 12288 data: 2 4 4 17/30 Write block 175 len 4096 data: 4 18/30 Read block 126 len 36864 19/30 Read block 486 len 28672 20/30 Write block 314 len 20480 data: 4 4 3 2 3 21/30 Read block 31 len 16384 22/30 Read block 534 len 12288 23/30 Write block 429 len 8192 data: 6 5 24/30 Read block 115 len 36864 25/30 Write block 567 len 36864 data: 5 4 3 3 4 4 4 3 3 26/30 Read block 556 len 28672 27/30 Write block 215 len 36864 data: 3 4 4 4 5 5 6 6 4 28/30 Read block 588 len 36864 29/30 Write block 82 len 40960 data: 4 4 5 5 3 3 3 4 4 4 30/30 Read block 191 len 24576 ---------------------------------------------------------------- Crucible gen:28 GIO:true work queues: Upstairs:0 downstairs:15 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 16 Acked 1015 Write 3 Done Done Done false 17 Acked 1016 Write 1 Done Done Done false 18 Acked 1017 Read 9 Done Done Done false 19 Acked 1018 Read 7 Done Done Done false 20 Acked 1019 Write 5 Done Done Done false 21 Acked 1020 Read 4 Done Done Done false 22 Acked 1021 Read 3 Done Done Done false 23 Acked 1022 Write 2 Done Done Done false 24 Acked 1023 Read 9 Done Done Done false 25 Acked 1024 Write 9 Done Done Done false 26 Acked 1025 Read 7 Done Done Done false 27 Acked 1026 Write 9 Done Done Done false 28 Acked 1027 Read 9 Done Done Done false 29 Acked 1028 Write 10 Done Done Done false 30 Acked 1029 Read 6 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 1 1 Done 15 15 14 44 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1014 1014 1014 Downstairs last five completed: 1014 1013 1012 1011 1010 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:32:36.46231972Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1294} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 51 51 51 F F F 1 020-039 26 26 26 58 58 58 F F F 2 040-059 28 28 28 63 63 63 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 27 27 27 59 59 59 T T T 5 100-119 24 24 24 55 55 55 F F F 6 120-139 28 28 28 60 60 60 F F F 7 140-159 27 27 27 59 59 59 F F F 8 160-179 28 28 28 60 60 60 T T T 9 180-199 28 28 28 60 60 60 F F F 10 200-219 24 24 24 52 52 52 T T T 11 220-239 24 24 24 52 52 52 T T T 12 240-259 28 28 28 60 60 60 F F F 13 260-279 26 26 26 58 58 58 F F F 14 280-299 17 17 17 35 35 35 F F F 15 300-319 26 26 26 57 57 57 T T T 16 320-339 28 28 28 61 61 61 T T T 17 340-359 24 24 24 55 55 55 F F F 18 360-379 20 20 20 45 45 45 F F F 19 380-399 23 23 23 51 51 51 F F F 20 400-419 28 28 28 62 62 62 F F F 21 420-439 28 28 28 61 61 61 T T T 22 440-459 24 24 24 52 52 52 F F F 23 460-479 24 24 24 55 55 55 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 20 20 20 40 40 40 F F F 26 520-539 26 26 26 57 57 57 F F F 27 540-559 26 26 26 57 57 57 F F F 28 560-579 27 27 27 59 59 59 T T T 29 580-599 26 26 26 57 57 57 F F F Max gen: 28, Max flush: 63 A Difference in extent metadata is expected here On loop 14 Verifying data now No change after verify Loop: 14 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 51 51 51 F F F 1 020-039 26 26 26 58 58 58 F F F 2 040-059 28 28 28 63 63 63 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 29 29 29 64 64 64 F F F 5 100-119 24 24 24 55 55 55 F F F 6 120-139 28 28 28 60 60 60 F F F 7 140-159 27 27 27 59 59 59 F F F 8 160-179 29 29 29 64 64 64 F F F 9 180-199 28 28 28 60 60 60 F F F 10 200-219 29 29 29 64 64 64 F F F 11 220-239 29 29 29 64 64 64 F F F 12 240-259 28 28 28 60 60 60 F F F 13 260-279 26 26 26 58 58 58 F F F 14 280-299 17 17 17 35 35 35 F F F 15 300-319 29 29 29 64 64 64 F F F 16 320-339 29 29 29 64 64 64 F F F 17 340-359 24 24 24 55 55 55 F F F 18 360-379 20 20 20 45 45 45 F F F 19 380-399 23 23 23 51 51 51 F F F 20 400-419 28 28 28 62 62 62 F F F 21 420-439 29 29 29 64 64 64 F F F 22 440-459 24 24 24 52 52 52 F F F 23 460-479 24 24 24 55 55 55 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 20 20 20 40 40 40 F F F 26 520-539 26 26 26 57 57 57 F F F 27 540-559 26 26 26 57 57 57 F F F 28 560-579 29 29 29 64 64 64 F F F 29 580-599 26 26 26 57 57 57 F F F Max gen: 29, Max flush: 64 Begin loop 15 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.899207784Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.899738819Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.899767268Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302} {"msg":"Crucible d111cd00-75f0-478f-b369-89e0cd717260 has session id: 35053891-6d67-4217-8105-a2f507dd9dd1","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.899900002Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"Upstairs opts: Upstairs UUID: d111cd00-75f0-478f-b369-89e0cd717260, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.8999425Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"Crucible stats registered with UUID: d111cd00-75f0-478f-b369-89e0cd717260","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.899972798Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:30","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.900315342Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302} {"msg":"d111cd00-75f0-478f-b369-89e0cd717260 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.900456436Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.900477224Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.900496684Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.900510273Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"1","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.900523782Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"1","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.900541552Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"2","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.90057175Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"2","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.900589769Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"io task","client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.900603939Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"io task","client":"1","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.900617608Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"io task","client":"2","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"ds_connection connected from Ok(127.0.0.1:55907)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.900820539Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"io task","client":"1","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:32:51.900863277Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"io task","client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:32:51.901051998Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"io task","client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:32:51.901092626Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.901115385Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.901140194Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"ds_connection connected from Ok(127.0.0.1:56372)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.901165103Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"io task","client":"2","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.901189931Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"io task","client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.9016482Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"2","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.901682708Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"2","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.901736036Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"1","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.901759655Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"1","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.90229739Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"2","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.902338168Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"1","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.902857813Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"1","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.902885022Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"1","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.902900771Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.902963799Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"2","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.902982628Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"2","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:32:51.902996037Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.902495763Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"io task","client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"ds_connection connected from Ok(127.0.0.1:34730)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.902591469Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"io task","client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.903775233Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.903802192Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.904324537Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.904883842Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.90491833Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"[0]R flush_numbers[0..12]: [51, 58, 63, 35, 64, 55, 60, 59, 64, 60, 64, 64]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.904939449Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"downstairs","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"[0]R generation[0..12]: [23, 26, 28, 17, 29, 24, 28, 27, 29, 28, 29, 29]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.904957978Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"downstairs","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.904975897Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"downstairs","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"[1]R flush_numbers[0..12]: [51, 58, 63, 35, 64, 55, 60, 59, 64, 60, 64, 64]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.904992736Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"downstairs","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"[1]R generation[0..12]: [23, 26, 28, 17, 29, 24, 28, 27, 29, 28, 29, 29]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905020475Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"downstairs","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905043024Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"downstairs","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"[2]R flush_numbers[0..12]: [51, 58, 63, 35, 64, 55, 60, 59, 64, 60, 64, 64]","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2024-07-31T18:33:01.905063943Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"downstairs","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"[2]R generation[0..12]: [23, 26, 28, 17, 29, 24, 28, 27, 29, 28, 29, 29]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905081172Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"downstairs","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905098652Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"downstairs","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"Max found gen is 30","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905114621Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"downstairs","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"Generation requested: 30 >= found:30","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.90513107Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"downstairs","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"Next flush: 65","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905147789Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"downstairs","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905164128Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"":"downstairs","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905183348Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905198707Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"0","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905215276Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"1","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905231945Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"client":"2","":"downstairs","session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905248485Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"session_idRegion: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905270353Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"d111cd00-75f0-478f-b369-89e0cd717260 is now active with session: 35053891-6d67-4217-8105-a2f507dd9dd1","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905286193Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905301312Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302,"session_id":"35053891-6d67-4217-8105-a2f507dd9dd1"} {"msg":"The guest has finished waiting for activation with:30","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:01.905316491Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 387 len 8192 data: 3 2 02/30 Read block 247 len 24576 03/30 Write block 542 len 28672 data: 2 3 4 4 4 3 4 04/30 Write block 36 len 24576 data: 2 2 3 3 3 3 05/30 Write block 419 len 36864 data: 5 4 3 3 4 3 3 5 5 06/30 Read block 356 len 4096 07/30 Read block 117 len 20480 08/30 Write block 533 len 20480 data: 3 3 3 4 4 09/30 Read block 567 len 16384 10/30 Read block 198 len 36864 11/30 Write block 181 len 32768 data: 3 3 5 4 5 4 4 3 12/30 Write block 155 len 12288 data: 2 2 2 13/30 Write block 234 len 24576 data: 4 4 4 3 4 4 14/30 Read block 146 len 28672 15/30 Read block 328 len 8192 16/30 Write block 532 len 40960 data: 2 4 4 4 5 5 4 4 3 4 17/30 Write block 36 len 20480 data: 3 3 4 4 4 18/30 Read block 203 len 20480 19/30 Write block 267 len 12288 data: 4 4 5 20/30 Write block 195 len 16384 data: 3 3 3 3 21/30 Read block 361 len 12288 22/30 Write block 471 len 28672 data: 4 4 3 3 3 4 5 23/30 Read block 429 len 12288 24/30 Read block 469 len 32768 25/30 Read block 438 len 36864 26/30 Read block 282 len 28672 27/30 Read block 405 len 8192 28/30 Read block 400 len 20480 29/30 Read block 592 len 24576 30/30 Write block 289 len 28672 data: 5 5 5 5 4 4 3 ---------------------------------------------------------------- Crucible gen:30 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Read 3 Done Done Done false 25 Acked 1024 Read 8 Done Done Done false 26 Acked 1025 Read 9 Done Done Done false 27 Acked 1026 Read 7 Done Done Done false 28 Acked 1027 Read 2 Done Done Done false 29 Acked 1028 Read 5 Done Done Done false 30 Acked 1029 Read 6 Done Done Done false 31 Acked 1030 Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:33:02.06092976Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1302} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 51 51 51 F F F 1 020-039 30 30 30 65 65 65 F F F 2 040-059 30 30 30 65 65 65 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 29 29 29 64 64 64 F F F 5 100-119 24 24 24 55 55 55 F F F 6 120-139 28 28 28 60 60 60 F F F 7 140-159 30 30 30 65 65 65 F F F 8 160-179 29 29 29 64 64 64 F F F 9 180-199 30 30 30 65 65 65 F F F 10 200-219 29 29 29 64 64 64 F F F 11 220-239 30 30 30 65 65 65 F F F 12 240-259 28 28 28 60 60 60 F F F 13 260-279 30 30 30 65 65 65 F F F 14 280-299 17 17 17 35 35 35 T T T 15 300-319 29 29 29 64 64 64 F F F 16 320-339 29 29 29 64 64 64 F F F 17 340-359 24 24 24 55 55 55 F F F 18 360-379 20 20 20 45 45 45 F F F 19 380-399 30 30 30 65 65 65 F F F 20 400-419 30 30 30 65 65 65 F F F 21 420-439 30 30 30 65 65 65 F F F 22 440-459 24 24 24 52 52 52 F F F 23 460-479 30 30 30 65 65 65 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 20 20 20 40 40 40 F F F 26 520-539 30 30 30 65 65 65 F F F 27 540-559 30 30 30 65 65 65 F F F 28 560-579 29 29 29 64 64 64 F F F 29 580-599 26 26 26 57 57 57 F F F Max gen: 30, Max flush: 65 A Difference in extent metadata is expected here On loop 15 Verifying data now No change after verify Loop: 15 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 51 51 51 F F F 1 020-039 30 30 30 65 65 65 F F F 2 040-059 30 30 30 65 65 65 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 29 29 29 64 64 64 F F F 5 100-119 24 24 24 55 55 55 F F F 6 120-139 28 28 28 60 60 60 F F F 7 140-159 30 30 30 65 65 65 F F F 8 160-179 29 29 29 64 64 64 F F F 9 180-199 30 30 30 65 65 65 F F F 10 200-219 29 29 29 64 64 64 F F F 11 220-239 30 30 30 65 65 65 F F F 12 240-259 28 28 28 60 60 60 F F F 13 260-279 30 30 30 65 65 65 F F F 14 280-299 31 31 31 66 66 66 F F F 15 300-319 29 29 29 64 64 64 F F F 16 320-339 29 29 29 64 64 64 F F F 17 340-359 24 24 24 55 55 55 F F F 18 360-379 20 20 20 45 45 45 F F F 19 380-399 30 30 30 65 65 65 F F F 20 400-419 30 30 30 65 65 65 F F F 21 420-439 30 30 30 65 65 65 F F F 22 440-459 24 24 24 52 52 52 F F F 23 460-479 30 30 30 65 65 65 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 20 20 20 40 40 40 F F F 26 520-539 30 30 30 65 65 65 F F F 27 540-559 30 30 30 65 65 65 F F F 28 560-579 29 29 29 64 64 64 F F F 29 580-599 26 26 26 57 57 57 F F F Max gen: 31, Max flush: 66 Begin loop 16 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.3603503Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.360708593Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.360728512Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310} {"msg":"Crucible 3c637c55-e0cb-4c66-988b-87d970efd422 has session id: 488286d2-1a34-4d6f-b029-35d50de50b4d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.360965881Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"Upstairs opts: Upstairs UUID: 3c637c55-e0cb-4c66-988b-87d970efd422, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.361059737Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"Crucible stats registered with UUID: 3c637c55-e0cb-4c66-988b-87d970efd422","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.361080776Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:32","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.361408391Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310} {"msg":"3c637c55-e0cb-4c66-988b-87d970efd422 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.361437789Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.361452219Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.361471808Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.361486867Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"1","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.361510336Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"1","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.361541544Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"2","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.361568363Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"2","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.361589632Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"io task","client":"2","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.361604052Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"io task","client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.361617551Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"io task","client":"1","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"ds_connection connected from Ok(127.0.0.1:61160)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.361890488Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"io task","client":"2","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"ds_connection connected from Ok(127.0.0.1:41337)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.361948956Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"io task","client":"1","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:33:16.362024292Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"io task","client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:33:16.362047211Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"io task","client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:33:16.362127197Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.362145096Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.362158476Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.362225312Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"io task","client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.362622974Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"2","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.362660812Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"2","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.362695211Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"1","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.362722719Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"1","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.363229186Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"1","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.363337351Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"2","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.363733042Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"1","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.363759551Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"1","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.3637742Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.363834508Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"2","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.363853527Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"2","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:16.363866856Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.36384095Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"io task","client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"ds_connection connected from Ok(127.0.0.1:41560)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.364123917Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"io task","client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.365027595Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.365059193Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.365540221Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366116644Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366142193Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"[0]R flush_numbers[0..12]: [51, 65, 65, 35, 64, 55, 60, 65, 64, 65, 64, 65]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366159822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"downstairs","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"[0]R generation[0..12]: [23, 30, 30, 17, 29, 24, 28, 30, 29, 30, 29, 30]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366176921Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"downstairs","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366194471Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"downstairs","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"[1]R flush_numbers[0..12]: [51, 65, 65, 35, 64, 55, 60, 65, 64, 65, 64, 65]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.36621134Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"downstairs","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"[1]R generation[0..12]: [23, 30, 30, 17, 29, 24, 28, 30, 29, 30, 29, 30]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366228279Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"downstairs","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366245438Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"downstairs","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"[2]R flush_numbers[0..12]: [51, 65, 65, 35, 64, 55, 60, 65, 64, 65, 64, 65]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366262387Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"downstairs","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"[2]R generation[0..12]: [23, 30, 30, 17, 29, 24, 28, 30, 29, 30, 29, 30]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366279427Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"downstairs","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366296716Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"downstairs","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"Max found gen is 32","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366312805Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"downstairs","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"Generation requested: 32 >= found:32","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366329064Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"Next flush: 67","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366348593Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"downstairs","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366364612Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"":"downstairs","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366384192Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366406471Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"0","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.36642356Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"1","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366439979Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"client":"2","":"downstairs","session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366456878Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366472158Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"3c637c55-e0cb-4c66-988b-87d970efd422 is now active with session: 488286d2-1a34-4d6f-b029-35d50de50b4d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366487907Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366502576Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310,"session_id":"488286d2-1a34-4d6f-b029-35d50de50b4d"} {"msg":"The guest has finished waiting for activation with:32","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:26.366517736Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 438 len 12288 data: 5 5 3 02/30 Read block 380 len 24576 03/30 Write block 384 len 4096 data: 4 04/30 Write block 195 len 40960 data: 4 4 4 4 2 2 3 4 4 4 05/30 Read block 278 len 16384 06/30 Read block 90 len 20480 07/30 Read block 362 len 12288 08/30 Flush 09/30 Write block 530 len 36864 data: 2 2 3 5 5 5 6 6 5 10/30 Read block 296 len 4096 11/30 Write block 402 len 20480 data: 3 3 3 5 6 12/30 Write block 430 len 28672 data: 6 5 6 6 4 5 5 13/30 Flush 14/30 Write block 138 len 12288 data: 2 2 2 15/30 Read block 24 len 20480 16/30 Read block 246 len 4096 17/30 Write block 270 len 4096 data: 4 18/30 Write block 377 len 4096 data: 2 19/30 Flush 20/30 Write block 445 len 36864 data: 4 3 3 4 3 2 2 2 2 21/30 Read block 33 len 4096 22/30 Write block 142 len 4096 data: 3 23/30 Read block 270 len 20480 24/30 Write block 229 len 40960 data: 3 2 2 4 4 5 5 5 4 5 25/30 Write block 144 len 28672 data: 2 2 2 3 4 4 4 26/30 Write block 93 len 12288 data: 2 2 2 27/30 Write block 428 len 32768 data: 6 7 7 6 7 7 5 6 28/30 Write block 570 len 20480 data: 4 5 5 5 4 29/30 Write block 318 len 8192 data: 4 5 30/30 Read block 500 len 16384 ---------------------------------------------------------------- Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:11 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 20 Acked 1019 Write 9 Done Done Done false 21 Acked 1020 Read 1 Done Done Done false 22 Acked 1021 Write 1 Done Done Done false 23 Acked 1022 Read 5 Done Done Done false 24 Acked 1023 Write 10 Done Done Done false 25 Acked 1024 Write 7 Done Done Done false 26 Acked 1025 Write 3 Done Done Done false 27 Acked 1026 Write 8 Done Done Done false 28 Acked 1027 Write 5 Done Done Done false 29 Acked 1028 Write 2 Done Done Sent false 30 Acked 1029 Read 4 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 2 2 Done 11 11 9 31 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1018 1018 Downstairs last five completed: 1018 1017 1016 1015 1014 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:33:26.454220803Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1310} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 51 51 51 F F F 1 020-039 30 30 30 65 65 65 F F F 2 040-059 30 30 30 65 65 65 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 29 29 29 64 64 64 T T T 5 100-119 24 24 24 55 55 55 F F F 6 120-139 32 32 32 69 69 69 F F F 7 140-159 32 32 32 69 69 69 T T T 8 160-179 29 29 29 64 64 64 F F F 9 180-199 32 32 32 67 67 67 F F F 10 200-219 32 32 32 67 67 67 F F F 11 220-239 30 30 30 65 65 65 T T T 12 240-259 28 28 28 60 60 60 F F F 13 260-279 32 32 32 69 69 69 F F F 14 280-299 31 31 31 66 66 66 F F F 15 300-319 29 29 29 64 64 64 T T T 16 320-339 29 29 29 64 64 64 F F F 17 340-359 24 24 24 55 55 55 F F F 18 360-379 32 32 32 69 69 69 F F F 19 380-399 32 32 32 67 67 67 F F F 20 400-419 32 32 32 68 68 68 F F F 21 420-439 32 32 32 68 68 68 T T T 22 440-459 32 32 32 67 67 67 T T T 23 460-479 30 30 30 65 65 65 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 20 20 20 40 40 40 F F F 26 520-539 32 32 32 68 68 68 F F F 27 540-559 30 30 30 65 65 65 F F F 28 560-579 29 29 29 64 64 64 T T T 29 580-599 26 26 26 57 57 57 F F F Max gen: 32, Max flush: 69 A Difference in extent metadata is expected here On loop 16 Verifying data now No change after verify Loop: 16 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 23 23 23 51 51 51 F F F 1 020-039 30 30 30 65 65 65 F F F 2 040-059 30 30 30 65 65 65 F F F 3 060-079 17 17 17 35 35 35 F F F 4 080-099 33 33 33 70 70 70 F F F 5 100-119 24 24 24 55 55 55 F F F 6 120-139 32 32 32 69 69 69 F F F 7 140-159 33 33 33 70 70 70 F F F 8 160-179 29 29 29 64 64 64 F F F 9 180-199 32 32 32 67 67 67 F F F 10 200-219 32 32 32 67 67 67 F F F 11 220-239 33 33 33 70 70 70 F F F 12 240-259 28 28 28 60 60 60 F F F 13 260-279 32 32 32 69 69 69 F F F 14 280-299 31 31 31 66 66 66 F F F 15 300-319 33 33 33 70 70 70 F F F 16 320-339 29 29 29 64 64 64 F F F 17 340-359 24 24 24 55 55 55 F F F 18 360-379 32 32 32 69 69 69 F F F 19 380-399 32 32 32 67 67 67 F F F 20 400-419 32 32 32 68 68 68 F F F 21 420-439 33 33 33 70 70 70 F F F 22 440-459 33 33 33 70 70 70 F F F 23 460-479 30 30 30 65 65 65 F F F 24 480-499 17 17 17 35 35 35 F F F 25 500-519 20 20 20 40 40 40 F F F 26 520-539 32 32 32 68 68 68 F F F 27 540-559 30 30 30 65 65 65 F F F 28 560-579 33 33 33 70 70 70 F F F 29 580-599 26 26 26 57 57 57 F F F Max gen: 33, Max flush: 70 Begin loop 17 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.304186254Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.304764237Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.304795806Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318} {"msg":"Crucible b064e1df-0f74-4733-b860-26759d38edce has session id: 8d3bfc61-0c39-4eb7-8a19-f050616b68d7","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.304882192Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"Upstairs opts: Upstairs UUID: b064e1df-0f74-4733-b860-26759d38edce, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.304972927Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"Crucible stats registered with UUID: b064e1df-0f74-4733-b860-26759d38edce","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.304995606Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:34","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.305387498Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318} {"msg":"b064e1df-0f74-4733-b860-26759d38edce active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.305490773Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.305510122Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"0","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.305529491Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"0","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.305543081Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"1","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.3055565Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"1","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.305575839Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.305589788Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.305621647Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"io task","client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.305638806Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"io task","client":"0","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.305652826Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"io task","client":"1","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:33:41.305811748Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"io task","client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:33:41.305885265Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"io task","client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:33:41.305921453Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.30598523Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.306011069Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"ds_connection connected from Ok(127.0.0.1:54123)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.306035478Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"io task","client":"0","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"ds_connection connected from Ok(127.0.0.1:33742)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.306059427Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"io task","client":"1","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.306081446Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"io task","client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.306674058Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"0","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.306716766Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"0","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.306747185Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"1","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.306774423Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"1","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.307388305Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"0","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.307415833Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"1","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.308005896Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"1","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.308028295Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"1","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.308041994Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.308054104Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"0","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.308067473Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"0","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:41.308080782Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.307828908Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"io task","client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"ds_connection connected from Ok(127.0.0.1:41450)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.308076776Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"io task","client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.309124897Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.309152506Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.309614954Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310172358Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310201887Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"[0]R flush_numbers[0..12]: [51, 65, 65, 35, 70, 55, 69, 70, 64, 67, 67, 70]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310220056Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"downstairs","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"[0]R generation[0..12]: [23, 30, 30, 17, 33, 24, 32, 33, 29, 32, 32, 33]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310237425Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"downstairs","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310255215Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"downstairs","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"[1]R flush_numbers[0..12]: [51, 65, 65, 35, 70, 55, 69, 70, 64, 67, 67, 70]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310272394Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"downstairs","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"[1]R generation[0..12]: [23, 30, 30, 17, 33, 24, 32, 33, 29, 32, 32, 33]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310289523Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"downstairs","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310306932Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"downstairs","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"[2]R flush_numbers[0..12]: [51, 65, 65, 35, 70, 55, 69, 70, 64, 67, 67, 70]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310323831Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"downstairs","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"[2]R generation[0..12]: [23, 30, 30, 17, 33, 24, 32, 33, 29, 32, 32, 33]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.31034058Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"downstairs","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.31035862Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"downstairs","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"Max found gen is 34","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310374629Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"downstairs","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 34 >= found:34","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310394938Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"downstairs","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"Next flush: 71","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310411767Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"downstairs","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310431346Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"":"downstairs","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310448266Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310463255Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"0","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310479704Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"1","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310496033Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"client":"2","":"downstairs","session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310512452Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310527612Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"b064e1df-0f74-4733-b860-26759d38edce is now active with session: 8d3bfc61-0c39-4eb7-8a19-f050616b68d7","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.310543751Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.31055877Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318,"session_id":"8d3bfc61-0c39-4eb7-8a19-f050616b68d7"} {"msg":"The guest has finished waiting for activation with:34","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:33:51.31057367Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 299 len 36864 data: 3 3 4 4 4 3 2 2 2 02/30 Write block 53 len 36864 data: 3 3 3 3 4 4 4 3 2 03/30 Read block 204 len 16384 04/30 Write block 284 len 8192 data: 4 5 05/30 Write block 47 len 16384 data: 3 3 3 2 06/30 Flush 07/30 Write block 415 len 24576 data: 3 4 4 4 6 5 08/30 Flush 09/30 Write block 17 len 40960 data: 2 2 2 3 3 3 4 4 4 6 10/30 Read block 181 len 32768 11/30 Flush 12/30 Write block 401 len 24576 data: 4 4 4 4 6 7 13/30 Read block 552 len 24576 14/30 Read block 351 len 36864 15/30 Write block 362 len 28672 data: 3 2 4 5 5 6 7 16/30 Read block 543 len 40960 17/30 Read block 24 len 32768 18/30 Flush 19/30 Write block 313 len 28672 data: 4 5 5 4 3 5 6 20/30 Read block 31 len 16384 21/30 Write block 143 len 24576 data: 3 3 3 3 4 5 22/30 Read block 267 len 28672 23/30 Read block 205 len 8192 24/30 Read block 405 len 20480 25/30 Write block 484 len 4096 data: 2 26/30 Read block 172 len 16384 27/30 Read block 500 len 8192 28/30 Read block 45 len 4096 29/30 Write block 295 len 40960 data: 4 3 3 3 4 4 5 5 5 4 30/30 Read block 531 len 12288 ---------------------------------------------------------------- Crucible gen:34 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 19 Acked 1018 Write 7 Done Done Done false 20 Acked 1019 Read 4 Done Done Done false 21 Acked 1020 Write 6 Done Done Done false 22 Acked 1021 Read 7 Done Done Done false 23 Acked 1022 Read 2 Done Done Done false 24 Acked 1023 Read 5 Done Done Done false 25 Acked 1024 Write 1 Done Done Done false 26 Acked 1025 Read 4 Done Done Done false 27 Acked 1026 Read 2 Done Done Done false 28 Acked 1027 Read 1 Done Done Done false 29 Acked 1028 Write 10 Done Done Done false 30 Acked 1029 Read 3 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 1 1 Done 12 12 11 35 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1017 1017 1017 Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:33:51.423228653Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1318} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 34 34 73 73 73 F F F 1 020-039 34 34 34 73 73 73 F F F 2 040-059 34 34 34 71 71 71 F F F 3 060-079 34 34 34 71 71 71 F F F 4 080-099 33 33 33 70 70 70 F F F 5 100-119 24 24 24 55 55 55 F F F 6 120-139 32 32 32 69 69 69 F F F 7 140-159 33 33 33 70 70 70 T T T 8 160-179 29 29 29 64 64 64 F F F 9 180-199 32 32 32 67 67 67 F F F 10 200-219 32 32 32 67 67 67 F F F 11 220-239 33 33 33 70 70 70 F F F 12 240-259 28 28 28 60 60 60 F F F 13 260-279 32 32 32 69 69 69 F F F 14 280-299 34 34 34 71 71 71 T T T 15 300-319 34 34 34 71 71 71 T T T 16 320-339 29 29 29 64 64 64 F F F 17 340-359 24 24 24 55 55 55 F F F 18 360-379 34 34 34 74 74 74 F F F 19 380-399 32 32 32 67 67 67 F F F 20 400-419 34 34 34 74 74 74 F F F 21 420-439 34 34 34 72 72 72 F F F 22 440-459 33 33 33 70 70 70 F F F 23 460-479 30 30 30 65 65 65 F F F 24 480-499 17 17 17 35 35 35 T T T 25 500-519 20 20 20 40 40 40 F F F 26 520-539 32 32 32 68 68 68 F F F 27 540-559 30 30 30 65 65 65 F F F 28 560-579 33 33 33 70 70 70 F F F 29 580-599 26 26 26 57 57 57 F F F Max gen: 34, Max flush: 74 A Difference in extent metadata is expected here On loop 17 Verifying data now No change after verify Loop: 17 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 34 34 73 73 73 F F F 1 020-039 34 34 34 73 73 73 F F F 2 040-059 34 34 34 71 71 71 F F F 3 060-079 34 34 34 71 71 71 F F F 4 080-099 33 33 33 70 70 70 F F F 5 100-119 24 24 24 55 55 55 F F F 6 120-139 32 32 32 69 69 69 F F F 7 140-159 35 35 35 75 75 75 F F F 8 160-179 29 29 29 64 64 64 F F F 9 180-199 32 32 32 67 67 67 F F F 10 200-219 32 32 32 67 67 67 F F F 11 220-239 33 33 33 70 70 70 F F F 12 240-259 28 28 28 60 60 60 F F F 13 260-279 32 32 32 69 69 69 F F F 14 280-299 35 35 35 75 75 75 F F F 15 300-319 35 35 35 75 75 75 F F F 16 320-339 29 29 29 64 64 64 F F F 17 340-359 24 24 24 55 55 55 F F F 18 360-379 34 34 34 74 74 74 F F F 19 380-399 32 32 32 67 67 67 F F F 20 400-419 34 34 34 74 74 74 F F F 21 420-439 34 34 34 72 72 72 F F F 22 440-459 33 33 33 70 70 70 F F F 23 460-479 30 30 30 65 65 65 F F F 24 480-499 35 35 35 75 75 75 F F F 25 500-519 20 20 20 40 40 40 F F F 26 520-539 32 32 32 68 68 68 F F F 27 540-559 30 30 30 65 65 65 F F F 28 560-579 33 33 33 70 70 70 F F F 29 580-599 26 26 26 57 57 57 F F F Max gen: 35, Max flush: 75 Begin loop 18 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.047767811Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.048064797Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.048080746Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326} {"msg":"Crucible fbf4e3de-8dbb-4634-8a9c-7f0e37715e6b has session id: 15d06d5e-8828-424f-9f6a-6f4f49cd3a25","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.048551794Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"Upstairs opts: Upstairs UUID: fbf4e3de-8dbb-4634-8a9c-7f0e37715e6b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.048596962Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"Crucible stats registered with UUID: fbf4e3de-8dbb-4634-8a9c-7f0e37715e6b","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.048620361Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:36","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.048961145Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326} {"msg":"fbf4e3de-8dbb-4634-8a9c-7f0e37715e6b active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.048992664Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.049033582Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"0","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.04906249Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"0","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.049089589Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"1","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.049116838Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"1","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.049139107Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.049161136Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.049182295Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"io task","client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.049204014Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"io task","client":"0","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.049226023Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"io task","client":"1","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"ds_connection connected from Ok(127.0.0.1:54396)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.049248902Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"io task","client":"1","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"ds_connection connected from Ok(127.0.0.1:52215)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.049271751Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"io task","client":"0","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:34:06.049603185Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"io task","client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:34:06.049648033Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"io task","client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:34:06.049675082Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.049695121Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.04971595Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.049736819Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"io task","client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.050156519Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"1","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.050195168Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"1","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.050223476Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"0","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.050250695Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"0","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.05079288Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"1","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.050831928Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"0","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.051388922Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"1","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.051414091Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"1","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.05142878Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.051441599Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"0","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.051455189Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"0","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:06.051468408Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.051100038Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"io task","client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"ds_connection connected from Ok(127.0.0.1:59004)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.051222053Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"io task","client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.052312562Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.05235263Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.052922213Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053544624Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053567873Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"[0]R flush_numbers[0..12]: [73, 73, 71, 71, 70, 55, 69, 75, 64, 67, 67, 70]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053582972Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"downstairs","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"[0]R generation[0..12]: [34, 34, 34, 34, 33, 24, 32, 35, 29, 32, 32, 33]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053597382Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"downstairs","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053611401Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"downstairs","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"[1]R flush_numbers[0..12]: [73, 73, 71, 71, 70, 55, 69, 75, 64, 67, 67, 70]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.05362551Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"downstairs","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"[1]R generation[0..12]: [34, 34, 34, 34, 33, 24, 32, 35, 29, 32, 32, 33]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.05363966Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"downstairs","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053653709Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"downstairs","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"[2]R flush_numbers[0..12]: [73, 73, 71, 71, 70, 55, 69, 75, 64, 67, 67, 70]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053667298Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"downstairs","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"[2]R generation[0..12]: [34, 34, 34, 34, 33, 24, 32, 35, 29, 32, 32, 33]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053680768Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"downstairs","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053694737Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"downstairs","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"Max found gen is 36","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053707617Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"downstairs","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"Generation requested: 36 >= found:36","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053720626Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"downstairs","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"Next flush: 76","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2024-07-31T18:34:16.053741465Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"downstairs","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053755054Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"":"downstairs","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053768584Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053780463Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"0","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053793973Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"1","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053807322Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"client":"2","":"downstairs","session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053820801Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053833101Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"fbf4e3de-8dbb-4634-8a9c-7f0e37715e6b is now active with session: 15d06d5e-8828-424f-9f6a-6f4f49cd3a25","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.05384592Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.05385765Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326,"session_id":"15d06d5e-8828-424f-9f6a-6f4f49cd3a25"} {"msg":"The guest has finished waiting for activation with:36","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:16.053869729Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 571 len 24576 data: 6 6 6 5 4 3 03/30 Read block 361 len 8192 04/30 Write block 584 len 12288 data: 4 4 6 05/30 Write block 82 len 20480 data: 5 5 6 6 4 06/30 Write block 311 len 20480 data: 4 4 5 6 6 07/30 Read block 157 len 40960 08/30 Read block 402 len 36864 09/30 Flush 10/30 Write block 507 len 24576 data: 4 4 3 4 3 3 11/30 Read block 367 len 4096 12/30 Read block 364 len 8192 13/30 Write block 56 len 4096 data: 4 14/30 Read block 4 len 4096 15/30 Write block 41 len 28672 data: 4 3 4 4 3 3 4 16/30 Read block 439 len 8192 17/30 Write block 375 len 16384 data: 2 3 3 3 18/30 Read block 465 len 8192 19/30 Write block 593 len 20480 data: 3 3 3 2 2 20/30 Read block 476 len 16384 21/30 Write block 45 len 24576 data: 4 4 5 4 4 3 22/30 Read block 301 len 36864 23/30 Write block 111 len 40960 data: 2 2 2 2 2 2 2 2 2 5 24/30 Read block 267 len 16384 25/30 Read block 91 len 16384 26/30 Write block 152 len 28672 data: 4 4 3 3 3 3 3 27/30 Read block 96 len 28672 28/30 Read block 410 len 24576 29/30 Read block 107 len 8192 30/30 Write block 358 len 28672 data: 6 5 5 3 4 3 5 ---------------------------------------------------------------- Crucible gen:36 GIO:true work queues: Upstairs:0 downstairs:21 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 10 Acked 1009 Write 6 Done Done Done false 11 Acked 1010 Read 1 Done Done Done false 12 Acked 1011 Read 2 Done Done Done false 13 Acked 1012 Write 1 Done Done Done false 14 Acked 1013 Read 1 Done Done Done false 15 Acked 1014 Write 7 Done Done Done false 16 Acked 1015 Read 2 Done Done Done false 17 Acked 1016 Write 4 Done Done Done false 18 Acked 1017 Read 2 Done Done Done false 19 Acked 1018 Write 5 Done Done Done false 20 Acked 1019 Read 4 Done Done Done false 21 Acked 1020 Write 6 Done Done Done false 22 Acked 1021 Read 9 Done Done Done false 23 Acked 1022 Write 10 Done Done Done false 24 Acked 1023 Read 4 Done Done Done false 25 Acked 1024 Read 4 Done Done Done false 26 Acked 1025 Write 7 Done Done Done false 27 Acked 1026 Read 7 Done Done Done false 28 Acked 1027 Read 6 Done Done Done false 29 Acked 1028 Read 2 Done Done Done false 30 Acked 1029 Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 20 20 20 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1008 1008 1008 Downstairs last five completed: 1008 1007 1006 1005 1004 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:34:16.100364409Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1326} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 34 34 73 73 73 F F F 1 020-039 34 34 34 73 73 73 F F F 2 040-059 34 34 34 71 71 71 T T T 3 060-079 34 34 34 71 71 71 F F F 4 080-099 36 36 36 77 77 77 F F F 5 100-119 24 24 24 55 55 55 T T T 6 120-139 32 32 32 69 69 69 T T T 7 140-159 35 35 35 75 75 75 T T T 8 160-179 29 29 29 64 64 64 F F F 9 180-199 32 32 32 67 67 67 F F F 10 200-219 32 32 32 67 67 67 F F F 11 220-239 33 33 33 70 70 70 F F F 12 240-259 28 28 28 60 60 60 F F F 13 260-279 32 32 32 69 69 69 F F F 14 280-299 35 35 35 75 75 75 F F F 15 300-319 36 36 36 77 77 77 F F F 16 320-339 29 29 29 64 64 64 F F F 17 340-359 24 24 24 55 55 55 T T T 18 360-379 34 34 34 74 74 74 T T T 19 380-399 32 32 32 67 67 67 F F F 20 400-419 34 34 34 74 74 74 F F F 21 420-439 34 34 34 72 72 72 F F F 22 440-459 33 33 33 70 70 70 F F F 23 460-479 30 30 30 65 65 65 F F F 24 480-499 35 35 35 75 75 75 F F F 25 500-519 20 20 20 40 40 40 T T T 26 520-539 32 32 32 68 68 68 F F F 27 540-559 30 30 30 65 65 65 F F F 28 560-579 36 36 36 77 77 77 F F F 29 580-599 36 36 36 77 77 77 T T T Max gen: 36, Max flush: 77 A Difference in extent metadata is expected here On loop 18 Verifying data now No change after verify Loop: 18 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 34 34 34 73 73 73 F F F 1 020-039 34 34 34 73 73 73 F F F 2 040-059 37 37 37 78 78 78 F F F 3 060-079 34 34 34 71 71 71 F F F 4 080-099 36 36 36 77 77 77 F F F 5 100-119 37 37 37 78 78 78 F F F 6 120-139 37 37 37 78 78 78 F F F 7 140-159 37 37 37 78 78 78 F F F 8 160-179 29 29 29 64 64 64 F F F 9 180-199 32 32 32 67 67 67 F F F 10 200-219 32 32 32 67 67 67 F F F 11 220-239 33 33 33 70 70 70 F F F 12 240-259 28 28 28 60 60 60 F F F 13 260-279 32 32 32 69 69 69 F F F 14 280-299 35 35 35 75 75 75 F F F 15 300-319 36 36 36 77 77 77 F F F 16 320-339 29 29 29 64 64 64 F F F 17 340-359 37 37 37 78 78 78 F F F 18 360-379 37 37 37 78 78 78 F F F 19 380-399 32 32 32 67 67 67 F F F 20 400-419 34 34 34 74 74 74 F F F 21 420-439 34 34 34 72 72 72 F F F 22 440-459 33 33 33 70 70 70 F F F 23 460-479 30 30 30 65 65 65 F F F 24 480-499 35 35 35 75 75 75 F F F 25 500-519 37 37 37 78 78 78 F F F 26 520-539 32 32 32 68 68 68 F F F 27 540-559 30 30 30 65 65 65 F F F 28 560-579 36 36 36 77 77 77 F F F 29 580-599 37 37 37 78 78 78 F F F Max gen: 37, Max flush: 78 Begin loop 19 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.233524154Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.23405772Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.234091018Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334} {"msg":"Crucible 3dd1690e-6e68-4b86-9b16-118f3ee1afc5 has session id: 55033bc8-61f8-4cc9-940f-625a9633664d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.234313968Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"Upstairs opts: Upstairs UUID: 3dd1690e-6e68-4b86-9b16-118f3ee1afc5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.234347506Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"Crucible stats registered with UUID: 3dd1690e-6e68-4b86-9b16-118f3ee1afc5","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.234362905Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:38","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.234658132Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334} {"msg":"3dd1690e-6e68-4b86-9b16-118f3ee1afc5 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.234729398Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.234759237Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"0","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.234779576Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"0","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.234802365Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.234828474Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.234854003Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"2","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.234879471Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"2","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.234897441Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"io task","client":"2","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.23491162Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"io task","client":"0","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.234929619Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"io task","client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"ds_connection connected from Ok(127.0.0.1:45341)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.235236994Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"io task","client":"0","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"ds_connection connected from Ok(127.0.0.1:49988)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.235280092Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"io task","client":"2","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:34:31.235309621Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"io task","client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:34:31.23533643Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"io task","client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:34:31.235362939Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.235382918Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.235407837Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.235429326Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"io task","client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.235924073Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"2","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.235951121Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"2","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.23596594Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"0","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.236011069Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"0","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.236541114Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"0","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.236680957Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"2","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.237075159Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"0","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.237099558Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"0","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.237113977Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.237187943Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"2","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.237206903Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"2","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:31.237220302Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.23644612Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"io task","client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"ds_connection connected from Ok(127.0.0.1:44828)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.236547635Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"io task","client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.237561868Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.237588676Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.238241446Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.2387943Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.238818519Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"[0]R flush_numbers[0..12]: [73, 73, 78, 71, 77, 78, 78, 78, 64, 67, 67, 70]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.238833808Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"downstairs","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"[0]R generation[0..12]: [34, 34, 37, 34, 36, 37, 37, 37, 29, 32, 32, 33]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.238848147Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"downstairs","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.238862547Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"downstairs","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"[1]R flush_numbers[0..12]: [73, 73, 78, 71, 77, 78, 78, 78, 64, 67, 67, 70]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.238876296Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"downstairs","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"[1]R generation[0..12]: [34, 34, 37, 34, 36, 37, 37, 37, 29, 32, 32, 33]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.238889835Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"downstairs","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.238903755Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"downstairs","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"[2]R flush_numbers[0..12]: [73, 73, 78, 71, 77, 78, 78, 78, 64, 67, 67, 70]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.238917504Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"downstairs","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"[2]R generation[0..12]: [34, 34, 37, 34, 36, 37, 37, 37, 29, 32, 32, 33]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.238931153Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"downstairs","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.238945123Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"downstairs","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"Max found gen is 38","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.238961702Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"downstairs","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"Generation requested: 38 >= found:38","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:34:41.238975271Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"downstairs","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"Next flush: 79","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.238989711Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"downstairs","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.23900277Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"":"downstairs","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.239016109Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.239028249Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"0","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.239041438Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"1","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.239054468Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"client":"2","":"downstairs","session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.239067777Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.239079906Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"3dd1690e-6e68-4b86-9b16-118f3ee1afc5 is now active with session: 55033bc8-61f8-4cc9-940f-625a9633664d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.239092596Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.239104256Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334,"session_id":"55033bc8-61f8-4cc9-940f-625a9633664d"} {"msg":"The guest has finished waiting for activation with:38","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:41.239116305Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 542 len 36864 data: 3 4 5 5 5 4 5 4 5 02/30 Write block 369 len 4096 data: 6 03/30 Write block 230 len 12288 data: 3 3 5 04/30 Write block 382 len 8192 data: 3 3 05/30 Read block 545 len 28672 06/30 Read block 282 len 16384 07/30 Write block 526 len 12288 data: 3 2 2 08/30 Write block 430 len 12288 data: 8 7 8 09/30 Read block 191 len 36864 10/30 Write block 228 len 40960 data: 3 4 4 4 6 5 6 6 6 5 11/30 Read block 582 len 36864 12/30 Read block 477 len 12288 13/30 Write block 308 len 24576 data: 3 4 4 5 5 6 14/30 Write block 18 len 28672 data: 3 3 4 4 4 5 5 15/30 Read block 478 len 28672 16/30 Read block 419 len 12288 17/30 Read block 271 len 40960 18/30 Write block 166 len 4096 data: 3 19/30 Read block 442 len 40960 20/30 Read block 301 len 8192 21/30 Write block 314 len 8192 data: 7 7 22/30 Write block 188 len 12288 data: 4 3 3 23/30 Write block 354 len 24576 data: 6 6 7 7 7 6 24/30 Write block 145 len 8192 data: 4 4 25/30 Read block 390 len 24576 26/30 Read block 42 len 4096 27/30 Write block 417 len 24576 data: 5 5 7 6 4 4 28/30 Read block 518 len 28672 29/30 Read block 367 len 24576 30/30 Read block 125 len 24576 ---------------------------------------------------------------- Crucible gen:38 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 28 Acked 1027 Write 6 Done Done Done false 29 Acked 1028 Read 7 Done Done Done false 30 Acked 1029 Read 6 Done Done Done false 31 Acked 1030 Read 6 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1026 1026 1026 Downstairs last five completed: 1026 1025 1024 1023 1022 Upstairs last five completed: 31 30 27 29 28 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:34:41.319174869Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1334} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 79 79 79 F F F 1 020-039 38 38 38 79 79 79 F F F 2 040-059 37 37 37 78 78 78 F F F 3 060-079 34 34 34 71 71 71 F F F 4 080-099 36 36 36 77 77 77 F F F 5 100-119 37 37 37 78 78 78 F F F 6 120-139 37 37 37 78 78 78 F F F 7 140-159 38 38 38 79 79 79 F F F 8 160-179 38 38 38 79 79 79 F F F 9 180-199 38 38 38 79 79 79 F F F 10 200-219 32 32 32 67 67 67 F F F 11 220-239 38 38 38 79 79 79 F F F 12 240-259 28 28 28 60 60 60 F F F 13 260-279 32 32 32 69 69 69 F F F 14 280-299 35 35 35 75 75 75 F F F 15 300-319 38 38 38 79 79 79 F F F 16 320-339 29 29 29 64 64 64 F F F 17 340-359 38 38 38 79 79 79 F F F 18 360-379 38 38 38 79 79 79 F F F 19 380-399 38 38 38 79 79 79 F F F 20 400-419 34 34 34 74 74 74 T T T 21 420-439 38 38 38 79 79 79 T T T 22 440-459 33 33 33 70 70 70 F F F 23 460-479 30 30 30 65 65 65 F F F 24 480-499 35 35 35 75 75 75 F F F 25 500-519 37 37 37 78 78 78 F F F 26 520-539 38 38 38 79 79 79 F F F 27 540-559 38 38 38 79 79 79 F F F 28 560-579 36 36 36 77 77 77 F F F 29 580-599 37 37 37 78 78 78 F F F Max gen: 38, Max flush: 79 A Difference in extent metadata is expected here On loop 19 Verifying data now No change after verify Loop: 19 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 79 79 79 F F F 1 020-039 38 38 38 79 79 79 F F F 2 040-059 37 37 37 78 78 78 F F F 3 060-079 34 34 34 71 71 71 F F F 4 080-099 36 36 36 77 77 77 F F F 5 100-119 37 37 37 78 78 78 F F F 6 120-139 37 37 37 78 78 78 F F F 7 140-159 38 38 38 79 79 79 F F F 8 160-179 38 38 38 79 79 79 F F F 9 180-199 38 38 38 79 79 79 F F F 10 200-219 32 32 32 67 67 67 F F F 11 220-239 38 38 38 79 79 79 F F F 12 240-259 28 28 28 60 60 60 F F F 13 260-279 32 32 32 69 69 69 F F F 14 280-299 35 35 35 75 75 75 F F F 15 300-319 38 38 38 79 79 79 F F F 16 320-339 29 29 29 64 64 64 F F F 17 340-359 38 38 38 79 79 79 F F F 18 360-379 38 38 38 79 79 79 F F F 19 380-399 38 38 38 79 79 79 F F F 20 400-419 39 39 39 80 80 80 F F F 21 420-439 39 39 39 80 80 80 F F F 22 440-459 33 33 33 70 70 70 F F F 23 460-479 30 30 30 65 65 65 F F F 24 480-499 35 35 35 75 75 75 F F F 25 500-519 37 37 37 78 78 78 F F F 26 520-539 38 38 38 79 79 79 F F F 27 540-559 38 38 38 79 79 79 F F F 28 560-579 36 36 36 77 77 77 F F F 29 580-599 37 37 37 78 78 78 F F F Max gen: 39, Max flush: 80 Begin loop 20 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.774912946Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.775368244Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.775399303Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342} {"msg":"Crucible 45116513-e3bf-4e77-9b21-abf867a60929 has session id: facb6757-1949-4017-8ef9-076027ae9c27","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.775618063Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"Upstairs opts: Upstairs UUID: 45116513-e3bf-4e77-9b21-abf867a60929, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.775660891Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"Crucible stats registered with UUID: 45116513-e3bf-4e77-9b21-abf867a60929","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.77568452Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:40","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.776004895Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342} {"msg":"45116513-e3bf-4e77-9b21-abf867a60929 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.776043523Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.776058372Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"0","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.776081331Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"0","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.776095701Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"1","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.77610941Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"1","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.776123009Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.776136719Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.776199326Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"io task","client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.776219715Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"io task","client":"0","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.776234294Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"io task","client":"1","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:34:55.776435574Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"io task","client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:34:55.776475003Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"io task","client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"ds_connection connected from Ok(127.0.0.1:51873)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.776500802Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"io task","client":"1","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:34:55.776573268Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.776597947Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.776621576Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"ds_connection connected from Ok(127.0.0.1:42157)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.776647735Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"io task","client":"0","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.776692453Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"io task","client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.777245837Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"0","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.777272946Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"0","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.777287885Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"1","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.777301694Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"1","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.777863388Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"0","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.777988142Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"1","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.778343445Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"0","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.778367214Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"0","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.778382083Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.778504388Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"1","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.778522517Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"1","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:34:55.778535566Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.777946617Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"io task","client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"ds_connection connected from Ok(127.0.0.1:55164)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.77809345Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"io task","client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.778965619Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.778996838Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.779537442Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780137324Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780165613Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"[0]R flush_numbers[0..12]: [79, 79, 78, 71, 77, 78, 78, 79, 79, 79, 67, 79]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780182022Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"downstairs","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"[0]R generation[0..12]: [38, 38, 37, 34, 36, 37, 37, 38, 38, 38, 32, 38]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780196731Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"downstairs","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780211281Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"downstairs","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"[1]R flush_numbers[0..12]: [79, 79, 78, 71, 77, 78, 78, 79, 79, 79, 67, 79]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.78022566Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"downstairs","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"[1]R generation[0..12]: [38, 38, 37, 34, 36, 37, 37, 38, 38, 38, 32, 38]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780239829Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"downstairs","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780253929Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"downstairs","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"[2]R flush_numbers[0..12]: [79, 79, 78, 71, 77, 78, 78, 79, 79, 79, 67, 79]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780267828Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"downstairs","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"[2]R generation[0..12]: [38, 38, 37, 34, 36, 37, 37, 38, 38, 38, 32, 38]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780281347Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"downstairs","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780295267Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"downstairs","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"Max found gen is 40","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780313006Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"downstairs","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"Wait for a query_work_queue command to finish before sending IO } {"msg":"Generation requested: 40 >= found:40","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780329795Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"downstairs","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"Next flush: 81","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780342825Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"downstairs","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780355754Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"":"downstairs","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780369663Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780381543Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"0","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780394712Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"1","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780407922Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"client":"2","":"downstairs","session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780420991Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780433091Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"45116513-e3bf-4e77-9b21-abf867a60929 is now active with session: facb6757-1949-4017-8ef9-076027ae9c27","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.78044593Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780458099Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342,"session_id":"facb6757-1949-4017-8ef9-076027ae9c27"} {"msg":"The guest has finished waiting for activation with:40","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:05.780470059Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 393 len 40960 data: 8 7 7 8 8 7 5 5 5 5 02/30 Write block 450 len 36864 data: 3 3 3 3 2 4 4 4 3 03/30 Flush 04/30 Write block 508 len 12288 data: 5 4 5 05/30 Read block 49 len 12288 06/30 Read block 390 len 32768 07/30 Read block 262 len 8192 08/30 Flush 09/30 Write block 570 len 20480 data: 5 7 7 7 6 10/30 Read block 546 len 40960 11/30 Read block 150 len 20480 12/30 Write block 316 len 12288 data: 5 4 6 13/30 Read block 539 len 16384 14/30 Write block 563 len 24576 data: 5 6 6 6 6 5 15/30 Read block 303 len 32768 16/30 Write block 105 len 24576 data: 4 3 2 2 2 2 17/30 Write block 582 len 20480 data: 6 5 5 5 7 18/30 Read block 488 len 24576 19/30 Read block 34 len 12288 20/30 Write block 523 len 16384 data: 3 3 2 4 21/30 Read block 407 len 8192 22/30 Read block 256 len 20480 23/30 Write block 568 len 4096 data: 6 24/30 Write block 430 len 32768 data: 9 8 9 8 6 7 6 5 25/30 Write block 580 len 36864 data: 4 5 7 6 6 6 8 5 7 26/30 Read block 124 len 40960 27/30 Write block 137 len 8192 data: 2 3 28/30 Read block 432 len 24576 29/30 Write block 335 len 36864 data: 7 6 6 4 3 3 4 4 4 30/30 Write block 204 len 20480 data: 5 3 3 3 3 ---------------------------------------------------------------- Crucible gen:40 GIO:true work queues: Upstairs:0 downstairs:22 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 9 Acked 1008 Write 5 Done Done Done false 10 Acked 1009 Read 10 Done Done Done false 11 Acked 1010 Read 5 Done Done Done false 12 Acked 1011 Write 3 Done Done Done false 13 Acked 1012 Read 4 Done Done Done false 14 Acked 1013 Write 6 Done Done Done false 15 Acked 1014 Read 8 Done Done Done false 16 Acked 1015 Write 6 Done Done Done false 17 Acked 1016 Write 5 Done Done Done false 18 Acked 1017 Read 6 Done Done Done false 19 Acked 1018 Read 3 Done Done Done false 20 Acked 1019 Write 4 Done Done Done false 21 Acked 1020 Read 2 Done Done Done false 22 Acked 1021 Read 5 Done Done Done false 23 Acked 1022 Write 1 Done Done Done false 24 Acked 1023 Write 8 Done Done Done false 25 Acked 1024 Write 9 Done Done Done false 26 Acked 1025 Read 10 Done Done Done false 27 Acked 1026 Write 2 Done Done Done false 28 Acked 1027 Read 6 Done Done Done false 29 Acked 1028 Write 9 Done Done Done false 30 Acked 1029 Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 21 21 21 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1007 1007 1007 Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:35:05.845420138Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1342} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 79 79 79 F F F 1 020-039 38 38 38 79 79 79 F F F 2 040-059 37 37 37 78 78 78 F F F 3 060-079 34 34 34 71 71 71 F F F 4 080-099 36 36 36 77 77 77 F F F 5 100-119 37 37 37 78 78 78 T T T 6 120-139 37 37 37 78 78 78 T T T 7 140-159 38 38 38 79 79 79 F F F 8 160-179 38 38 38 79 79 79 F F F 9 180-199 38 38 38 79 79 79 F F F 10 200-219 32 32 32 67 67 67 T T T 11 220-239 38 38 38 79 79 79 F F F 12 240-259 28 28 28 60 60 60 F F F 13 260-279 32 32 32 69 69 69 F F F 14 280-299 35 35 35 75 75 75 F F F 15 300-319 38 38 38 79 79 79 T T T 16 320-339 29 29 29 64 64 64 T T T 17 340-359 38 38 38 79 79 79 T T T 18 360-379 38 38 38 79 79 79 F F F 19 380-399 40 40 40 81 81 81 F F F 20 400-419 40 40 40 81 81 81 F F F 21 420-439 39 39 39 80 80 80 T T T 22 440-459 40 40 40 81 81 81 F F F 23 460-479 30 30 30 65 65 65 F F F 24 480-499 35 35 35 75 75 75 F F F 25 500-519 40 40 40 82 82 82 F F F 26 520-539 38 38 38 79 79 79 T T T 27 540-559 38 38 38 79 79 79 F F F 28 560-579 36 36 36 77 77 77 T T T 29 580-599 37 37 37 78 78 78 T T T Max gen: 40, Max flush: 82 A Difference in extent metadata is expected here On loop 20 Verifying data now No change after verify Loop: 20 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 79 79 79 F F F 1 020-039 38 38 38 79 79 79 F F F 2 040-059 37 37 37 78 78 78 F F F 3 060-079 34 34 34 71 71 71 F F F 4 080-099 36 36 36 77 77 77 F F F 5 100-119 41 41 41 83 83 83 F F F 6 120-139 41 41 41 83 83 83 F F F 7 140-159 38 38 38 79 79 79 F F F 8 160-179 38 38 38 79 79 79 F F F 9 180-199 38 38 38 79 79 79 F F F 10 200-219 41 41 41 83 83 83 F F F 11 220-239 38 38 38 79 79 79 F F F 12 240-259 28 28 28 60 60 60 F F F 13 260-279 32 32 32 69 69 69 F F F 14 280-299 35 35 35 75 75 75 F F F 15 300-319 41 41 41 83 83 83 F F F 16 320-339 41 41 41 83 83 83 F F F 17 340-359 41 41 41 83 83 83 F F F 18 360-379 38 38 38 79 79 79 F F F 19 380-399 40 40 40 81 81 81 F F F 20 400-419 40 40 40 81 81 81 F F F 21 420-439 41 41 41 83 83 83 F F F 22 440-459 40 40 40 81 81 81 F F F 23 460-479 30 30 30 65 65 65 F F F 24 480-499 35 35 35 75 75 75 F F F 25 500-519 40 40 40 82 82 82 F F F 26 520-539 41 41 41 83 83 83 F F F 27 540-559 38 38 38 79 79 79 F F F 28 560-579 41 41 41 83 83 83 F F F 29 580-599 41 41 41 83 83 83 F F F Max gen: 41, Max flush: 83 Begin loop 21 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.248196004Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249017095Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249049334Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350} {"msg":"Crucible 89afa780-d0ad-4071-8386-58edae91c7d7 has session id: 3d09b769-ad7e-4f33-b714-7357bfa2fbaa","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249073713Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"Upstairs opts: Upstairs UUID: 89afa780-d0ad-4071-8386-58edae91c7d7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249105791Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"Crucible stats registered with UUID: 89afa780-d0ad-4071-8386-58edae91c7d7","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.24913097Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:42","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249368079Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350} {"msg":"89afa780-d0ad-4071-8386-58edae91c7d7 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249428067Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249454045Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"0","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249474944Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"0","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249497343Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249523352Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249549141Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"2","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249574789Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"2","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249599108Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"io task","client":"2","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249624147Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"io task","client":"0","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249646766Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"io task","client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"ds_connection connected from Ok(127.0.0.1:49106)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249687764Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"io task","client":"2","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:35:21.249715953Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"io task","client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:35:21.249747312Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"io task","client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:35:21.24977728Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249803789Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249824358Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"ds_connection connected from Ok(127.0.0.1:43596)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.249898954Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"io task","client":"0","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.250071486Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"io task","client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.25064416Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"2","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.250682448Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"2","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.250713607Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"0","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.250741255Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"0","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.251320458Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"0","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.251460672Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"2","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.251804685Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"0","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.251827144Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"0","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.251840814Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.251913671Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"2","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.251933079Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"2","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:21.251949589Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.250920831Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"io task","client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"ds_connection connected from Ok(127.0.0.1:37306)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.251018156Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"io task","client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.252177482Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.25221345Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.252717867Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253354247Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253380596Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"[0]R flush_numbers[0..12]: [79, 79, 78, 71, 77, 83, 83, 79, 79, 79, 83, 79]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253395645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"downstairs","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"[0]R generation[0..12]: [38, 38, 37, 34, 36, 41, 41, 38, 38, 38, 41, 38]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253409914Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"downstairs","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253424414Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"downstairs","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"[1]R flush_numbers[0..12]: [79, 79, 78, 71, 77, 83, 83, 79, 79, 79, 83, 79]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253438043Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"downstairs","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"[1]R generation[0..12]: [38, 38, 37, 34, 36, 41, 41, 38, 38, 38, 41, 38]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253455262Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"downstairs","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253472951Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"downstairs","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"[2]R flush_numbers[0..12]: [79, 79, 78, 71, 77, 83, 83, 79, 79, 79, 83, 79]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.25348996Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"downstairs","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"[2]R generation[0..12]: [38, 38, 37, 34, 36, 41, 41, 38, 38, 38, 41, 38]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.25351317Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"downstairs","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253531059Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWTWait for a query_work_queue command to finish before sending IO ","pid":1350,"":"downstairs","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"Max found gen is 42","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253550978Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"downstairs","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"Generation requested: 42 >= found:42","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253567447Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"downstairs","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"Next flush: 84","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253583176Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"downstairs","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253599026Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"":"downstairs","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253615195Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253630094Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"0","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253646573Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"1","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253662743Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"client":"2","":"downstairs","session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253679342Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253694531Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"89afa780-d0ad-4071-8386-58edae91c7d7 is now active with session: 3d09b769-ad7e-4f33-b714-7357bfa2fbaa","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.25371047Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.25372532Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350,"session_id":"3d09b769-ad7e-4f33-b714-7357bfa2fbaa"} {"msg":"The guest has finished waiting for activation with:42","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:31.253740499Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 495 len 32768 data: 2 2 3 3 3 3 5 7 02/30 Read block 302 len 4096 03/30 Write block 272 len 16384 data: 3 2 3 3 04/30 Read block 216 len 32768 05/30 Read block 402 len 16384 06/30 Write block 190 len 20480 data: 4 2 3 3 3 07/30 Write block 396 len 36864 data: 9 9 8 6 6 6 6 5 5 08/30 Read block 344 len 16384 09/30 Read block 109 len 28672 10/30 Write block 21 len 40960 data: 5 5 6 6 5 7 6 6 7 5 11/30 Read block 415 len 20480 12/30 Write block 593 len 24576 data: 4 4 4 3 3 2 13/30 Write block 347 len 36864 data: 2 2 4 5 5 5 6 7 7 14/30 Read block 131 len 24576 15/30 Read block 468 len 36864 16/30 Write block 133 len 20480 data: 2 2 2 2 3 17/30 Read block 35 len 16384 18/30 Read block 11 len 12288 19/30 Write block 592 len 16384 data: 4 5 5 5 20/30 Read block 554 len 12288 21/30 Read block 493 len 28672 22/30 Write block 275 len 40960 data: 4 2 2 2 3 3 3 3 3 5 23/30 Write block 225 len 32768 data: 3 2 3 4 5 5 5 7 24/30 Read block 299 len 16384 25/30 Read block 310 len 40960 26/30 Read block 210 len 16384 27/30 Read block 185 len 12288 28/30 Read block 401 len 28672 29/30 Write block 208 len 20480 data: 4 3 3 3 2 30/30 Write block 207 len 40960 data: 4 5 4 4 4 3 3 3 4 5 ---------------------------------------------------------------- Crucible gen:42 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 23 Acked 1022 Write 10 Done Done Done false 24 Acked 1023 Write 8 Done Done Done false 25 Acked 1024 Read 4 Done Done Done false 26 Acked 1025 Read 10 Done Done Done false 27 Acked 1026 Read 4 Done Done Done false 28 Acked 1027 Read 3 Done Done Done false 29 Acked 1028 Read 7 Done Done Done false 30 Acked 1029 Write 5 Done Done Done false 31 Acked 1030 Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 8 8 8 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1021 1021 1021 Downstairs last five completed: 1021 1020 1019 1018 1017 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:35:31.329345771Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1350} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 79 79 79 F F F 1 020-039 42 42 42 84 84 84 F F F 2 040-059 37 37 37 78 78 78 F F F 3 060-079 34 34 34 71 71 71 F F F 4 080-099 36 36 36 77 77 77 F F F 5 100-119 41 41 41 83 83 83 F F F 6 120-139 42 42 42 84 84 84 F F F 7 140-159 38 38 38 79 79 79 F F F 8 160-179 38 38 38 79 79 79 F F F 9 180-199 42 42 42 84 84 84 F F F 10 200-219 41 41 41 83 83 83 T T T 11 220-239 38 38 38 79 79 79 T T T 12 240-259 28 28 28 60 60 60 F F F 13 260-279 42 42 42 84 84 84 T T T 14 280-299 35 35 35 75 75 75 T T T 15 300-319 41 41 41 83 83 83 F F F 16 320-339 41 41 41 83 83 83 F F F 17 340-359 42 42 42 84 84 84 F F F 18 360-379 38 38 38 79 79 79 F F F 19 380-399 42 42 42 84 84 84 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 41 41 41 83 83 83 F F F 22 440-459 40 40 40 81 81 81 F F F 23 460-479 30 30 30 65 65 65 F F F 24 480-499 42 42 42 84 84 84 F F F 25 500-519 42 42 42 84 84 84 F F F 26 520-539 41 41 41 83 83 83 F F F 27 540-559 38 38 38 79 79 79 F F F 28 560-579 41 41 41 83 83 83 F F F 29 580-599 42 42 42 84 84 84 F F F Max gen: 42, Max flush: 84 A Difference in extent metadata is expected here On loop 21 Verifying data now No change after verify Loop: 21 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 38 38 38 79 79 79 F F F 1 020-039 42 42 42 84 84 84 F F F 2 040-059 37 37 37 78 78 78 F F F 3 060-079 34 34 34 71 71 71 F F F 4 080-099 36 36 36 77 77 77 F F F 5 100-119 41 41 41 83 83 83 F F F 6 120-139 42 42 42 84 84 84 F F F 7 140-159 38 38 38 79 79 79 F F F 8 160-179 38 38 38 79 79 79 F F F 9 180-199 42 42 42 84 84 84 F F F 10 200-219 43 43 43 85 85 85 F F F 11 220-239 43 43 43 85 85 85 F F F 12 240-259 28 28 28 60 60 60 F F F 13 260-279 43 43 43 85 85 85 F F F 14 280-299 43 43 43 85 85 85 F F F 15 300-319 41 41 41 83 83 83 F F F 16 320-339 41 41 41 83 83 83 F F F 17 340-359 42 42 42 84 84 84 F F F 18 360-379 38 38 38 79 79 79 F F F 19 380-399 42 42 42 84 84 84 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 41 41 41 83 83 83 F F F 22 440-459 40 40 40 81 81 81 F F F 23 460-479 30 30 30 65 65 65 F F F 24 480-499 42 42 42 84 84 84 F F F 25 500-519 42 42 42 84 84 84 F F F 26 520-539 41 41 41 83 83 83 F F F 27 540-559 38 38 38 79 79 79 F F F 28 560-579 41 41 41 83 83 83 F F F 29 580-599 42 42 42 84 84 84 F F F Max gen: 43, Max flush: 85 Begin loop 22 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.107369468Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.107908132Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.107943021Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358} {"msg":"Crucible 7af6142e-a0f5-4080-9646-805a22f265da has session id: 8cbb7bad-56fa-45ed-80b5-8643e8bb3005","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.10796671Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"Upstairs opts: Upstairs UUID: 7af6142e-a0f5-4080-9646-805a22f265da, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.108003728Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"Crucible stats registered with UUID: 7af6142e-a0f5-4080-9646-805a22f265da","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.108030907Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:44","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.108403049Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358} {"msg":"7af6142e-a0f5-4080-9646-805a22f265da active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.108428268Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.108442838Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.108462027Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.108475736Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"1","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.108489496Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"1","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.108502525Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"2","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.108515784Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"2","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.108559272Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"io task","client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.108575961Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"io task","client":"1","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.108589561Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"io task","client":"2","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:35:46.108878687Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"io task","client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:35:46.108916025Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"io task","client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"ds_connection connected from Ok(127.0.0.1:44552)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.108943414Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"io task","client":"1","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:35:46.108976843Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.109001732Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.10902613Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.109095687Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"io task","client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"ds_connection connected from Ok(127.0.0.1:62092)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.109140725Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"io task","client":"2","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.109632262Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"2","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.109660081Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"2","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.10967591Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"1","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.109698489Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"1","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.110254193Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"1","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.110374068Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"2","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.110728961Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"1","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.11075241Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"1","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.110766799Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.110854065Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"2","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.110873364Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"2","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:46.110890473Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.110078925Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"io task","client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"ds_connection connected from Ok(127.0.0.1:41554)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.110246338Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"io task","client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.111447792Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.11148482Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.111961168Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112526231Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.11255411Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"[0]R flush_numbers[0..12]: [79, 84, 78, 71, 77, 83, 84, 79, 79, 84, 85, 85]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112572579Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"downstairs","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"[0]R generation[0..12]: [38, 42, 37, 34, 36, 41, 42, 38, 38, 42, 43, 43]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112590658Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"downstairs","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112608547Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"downstairs","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"[1]R flush_numbers[0..12]: [79, 84, 78, 71, 77, 83, 84, 79, 79, 84, 85, 85]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112625697Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"downstairs","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"[1]R generation[0..12]: [38, 42, 37, 34, 36, 41, 42, 38, 38, 42, 43, 43]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112642726Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"downstairs","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112660135Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"downstairs","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"[2]R flush_numbers[0..12]: [79, 84, 78, 71, 77, 83, 84, 79, 79, 84, 85, 85]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112681854Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"downstairs","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"[2]R generation[0..12]: [38, 42, 37, 34, 36, 41, 42, 38, 38, 42, 43, 43]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112699223Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"downstairs","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112716792Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"downstairs","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"Max found gen is 44","v":0,"name":"crucible","Wait for a query_work_queue command to finish before sending IO level":30,"time":"2024-07-31T18:35:56.112737171Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"downstairs","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"Generation requested: 44 >= found:44","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112754161Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"downstairs","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"Next flush: 86","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.11277029Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"downstairs","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112786419Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"":"downstairs","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112803168Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112818128Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"0","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112834937Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"1","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112851566Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"client":"2","":"downstairs","session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112868425Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112883594Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"7af6142e-a0f5-4080-9646-805a22f265da is now active with session: 8cbb7bad-56fa-45ed-80b5-8643e8bb3005","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112899814Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112914673Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358,"session_id":"8cbb7bad-56fa-45ed-80b5-8643e8bb3005"} {"msg":"The guest has finished waiting for activation with:44","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:35:56.112929883Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 76 len 40960 data: 4 4 3 3 5 5 6 6 7 7 02/30 Read block 291 len 24576 03/30 Flush 04/30 Write block 160 len 36864 data: 3 4 2 2 2 3 4 5 5 05/30 Read block 141 len 36864 06/30 Write block 286 len 24576 data: 5 5 5 6 6 6 07/30 Read block 440 len 8192 08/30 Flush 09/30 Write block 53 len 32768 data: 4 4 4 5 5 5 5 4 10/30 Write block 15 len 4096 data: 2 11/30 Read block 538 len 28672 12/30 Read block 160 len 4096 13/30 Write block 280 len 4096 data: 4 14/30 Read block 222 len 28672 15/30 Read block 123 len 36864 16/30 Read block 571 len 4096 17/30 Read block 61 len 4096 18/30 Write block 476 len 4096 data: 5 19/30 Read block 337 len 24576 20/30 Read block 546 len 12288 21/30 Read block 555 len 20480 22/30 Read block 119 len 16384 23/30 Flush 24/30 Write block 574 len 36864 data: 7 5 4 2 3 4 5 6 8 25/30 Read block 565 len 40960 26/30 Write block 9 len 4096 data: 4 27/30 Read block 538 len 32768 28/30 Write block 52 len 8192 data: 3 5 29/30 Read block 123 len 20480 30/30 Read block 427 len 8192 ---------------------------------------------------------------- Crucible gen:44 GIO:true work queues: Upstairs:0 downstairs:7 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 9 Done Done Done false 25 Acked 1024 Read 10 Done Done Done false 26 Acked 1025 Write 1 Done Done Done false 27 Acked 1026 Read 8 Done Done Done false 28 Acked 1027 Write 2 Done Done Done false 29 Acked 1028 Read 5 Done Done Done false 30 Acked 1029 Read 2 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 0 1 Done 7 6 7 20 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:35:56.161465798Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1358} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 88 88 88 T T T 1 020-039 42 42 42 84 84 84 F F F 2 040-059 44 44 44 88 88 88 T T T 3 060-079 44 44 44 88 88 88 F F F 4 080-099 44 44 44 86 86 86 F F F 5 100-119 41 41 41 83 83 83 F F F 6 120-139 42 42 42 84 84 84 F F F 7 140-159 38 38 38 79 79 79 F F F 8 160-179 44 44 44 87 87 87 F F F 9 180-199 42 42 42 84 84 84 F F F 10 200-219 43 43 43 85 85 85 F F F 11 220-239 43 43 43 85 85 85 F F F 12 240-259 28 28 28 60 60 60 F F F 13 260-279 43 43 43 85 85 85 F F F 14 280-299 44 44 44 88 88 88 F F F 15 300-319 41 41 41 83 83 83 F F F 16 320-339 41 41 41 83 83 83 F F F 17 340-359 42 42 42 84 84 84 F F F 18 360-379 38 38 38 79 79 79 F F F 19 380-399 42 42 42 84 84 84 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 41 41 41 83 83 83 F F F 22 440-459 40 40 40 81 81 81 F F F 23 460-479 44 44 44 88 88 88 F F F 24 480-499 42 42 42 84 84 84 F F F 25 500-519 42 42 42 84 84 84 F F F 26 520-539 41 41 41 83 83 83 F F F 27 540-559 38 38 38 79 79 79 F F F 28 560-579 41 41 41 83 83 83 T T T 29 580-599 42 42 42 84 84 84 T T T Max gen: 44, Max flush: 88 A Difference in extent metadata is expected here On loop 22 Verifying data now No change after verify Loop: 22 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 45 45 45 89 89 89 F F F 1 020-039 42 42 42 84 84 84 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 44 44 44 88 88 88 F F F 4 080-099 44 44 44 86 86 86 F F F 5 100-119 41 41 41 83 83 83 F F F 6 120-139 42 42 42 84 84 84 F F F 7 140-159 38 38 38 79 79 79 F F F 8 160-179 44 44 44 87 87 87 F F F 9 180-199 42 42 42 84 84 84 F F F 10 200-219 43 43 43 85 85 85 F F F 11 220-239 43 43 43 85 85 85 F F F 12 240-259 28 28 28 60 60 60 F F F 13 260-279 43 43 43 85 85 85 F F F 14 280-299 44 44 44 88 88 88 F F F 15 300-319 41 41 41 83 83 83 F F F 16 320-339 41 41 41 83 83 83 F F F 17 340-359 42 42 42 84 84 84 F F F 18 360-379 38 38 38 79 79 79 F F F 19 380-399 42 42 42 84 84 84 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 41 41 41 83 83 83 F F F 22 440-459 40 40 40 81 81 81 F F F 23 460-479 44 44 44 88 88 88 F F F 24 480-499 42 42 42 84 84 84 F F F 25 500-519 42 42 42 84 84 84 F F F 26 520-539 41 41 41 83 83 83 F F F 27 540-559 38 38 38 79 79 79 F F F 28 560-579 45 45 45 89 89 89 F F F 29 580-599 45 45 45 89 89 89 F F F Max gen: 45, Max flush: 89 Begin loop 23 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.96315837Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.963640407Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.963668896Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366} {"msg":"Crucible 73bc798b-2da7-42ef-804d-0621794f3da6 has session id: 5045c998-ca46-480f-b94e-579c12747b8e","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.963889756Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"Upstairs opts: Upstairs UUID: 73bc798b-2da7-42ef-804d-0621794f3da6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.963938843Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"Crucible stats registered with UUID: 73bc798b-2da7-42ef-804d-0621794f3da6","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.963967502Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:46","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.964238189Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366} {"msg":"73bc798b-2da7-42ef-804d-0621794f3da6 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.964302766Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.964327745Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.964347724Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.964361874Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"1","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.964375443Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"1","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.964388332Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"2","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.964401542Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"2","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.964419301Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"io task","client":"2","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.96443362Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"io task","client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.964447599Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"io task","client":"1","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"ds_connection connected from Ok(127.0.0.1:41913)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.964670949Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"io task","client":"1","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:36:10.964802293Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"io task","client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:36:10.964835501Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"io task","client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:36:10.96486767Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.964900398Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.964927847Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"ds_connection connected from Ok(127.0.0.1:50311)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.964951036Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"io task","client":"2","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.964974335Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"io task","client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.965479621Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"1","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.96550663Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"1","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.965521989Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"2","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.965536499Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"2","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.96614304Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"2","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.966170719Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"1","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.966677395Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"2","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.966703604Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"2","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.966721184Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.966742883Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"1","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.966761042Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"1","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:10.966774551Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.965372998Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"io task","client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"ds_connection connected from Ok(127.0.0.1:54649)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.965514101Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"io task","client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.966719695Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.966748884Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.967316357Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.967916679Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.967942718Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"[0]R flush_numbers[0..12]: [89, 84, 89, 88, 86, 83, 84, 79, 87, 84, 85, 85]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.967958687Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"downstairs","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"[0]R generation[0..12]: [45, 42, 45, 44, 44, 41, 42, 38, 44, 42, 43, 43]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.967973446Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"downstairs","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.967988046Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"downstairs","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"[1]R flush_numbers[0..12]: [89, 84, 89, 88, 86, 83, 84, 79, 87, 84, 85, 85]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968001795Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"downstairs","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"[1]R generation[0..12]: [45, 42, 45, 44, 44, 41, 42, 38, 44, 42, 43, 43]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968015525Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"downstairs","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968029544Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"downstairs","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"[2]R flush_numbers[0..12]: [89, 84, 89, 88, 86, 83, 84, 79, 87, 84, 85, 85]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968043173Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"downstairs","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"[2]R generation[0..12]: [45, 42, 45, 44, 44, 41, 42, 38, 44, 42, 43, 43]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968060942Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"downstairs","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968075372Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"downstairs","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"Max found gen is 46","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968088541Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"downstairs","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"Generation requested: 46 >= found:46","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:36:20.968102391Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"downstairs","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"Next flush: 90","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.96811755Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"downstairs","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968130799Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"":"downstairs","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968143999Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968155818Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"0","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968168957Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"1","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968182017Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"client":"2","":"downstairs","session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968195186Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968207316Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"73bc798b-2da7-42ef-804d-0621794f3da6 is now active with session: 5045c998-ca46-480f-b94e-579c12747b8e","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968220025Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968231794Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366,"session_id":"5045c998-ca46-480f-b94e-579c12747b8e"} {"msg":"The guest has finished waiting for activation with:46","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:20.968243944Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 154 len 24576 data: 4 4 4 4 4 3 02/30 Flush 03/30 Write block 496 len 8192 data: 3 4 04/30 Write block 378 len 20480 data: 4 3 3 3 4 05/30 Read block 518 len 32768 06/30 Write block 476 len 36864 data: 6 6 2 2 2 2 2 2 3 07/30 Write block 375 len 40960 data: 3 4 4 5 4 4 4 5 4 5 08/30 Read block 513 len 40960 09/30 Read block 500 len 8192 10/30 Write block 254 len 16384 data: 6 6 6 5 11/30 Read block 14 len 8192 12/30 Read block 491 len 4096 13/30 Read block 158 len 12288 14/30 Write block 17 len 4096 data: 3 15/30 Write block 63 len 36864 data: 2 2 2 2 2 2 2 2 3 16/30 Read block 166 len 28672 17/30 Write block 100 len 20480 data: 4 4 4 4 4 18/30 Read block 161 len 20480 19/30 Flush 20/30 Write block 314 len 32768 data: 8 8 6 5 7 7 5 4 21/30 Flush 22/30 Write block 181 len 20480 data: 4 4 6 5 6 23/30 Read block 248 len 16384 24/30 Read block 286 len 32768 25/30 Read block 395 len 28672 26/30 Write block 506 len 24576 data: 5 5 6 5 6 4 27/30 Write block 197 len 20480 data: 5 5 3 3 4 28/30 Write block 345 len 40960 data: 3 2 3 3 5 6 6 6 7 8 29/30 Write block 143 len 32768 data: 4 4 5 5 5 6 5 5 30/30 Read block 500 len 24576 ---------------------------------------------------------------- Crucible gen:46 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 5 Done Done Done false 23 Acked 1022 Read 4 Done Done Done false 24 Acked 1023 Read 8 Done Done Done false 25 Acked 1024 Read 7 Done Done Done false 26 Acked 1025 Write 6 Done Done Done false 27 Acked 1026 Write 5 Done Done Done false 28 Acked 1027 Write 10 Done Done Done false 29 Acked 1028 Write 8 Done Done Done false 30 Acked 1029 Read 6 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 9 9 9 27 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:36:21.044944155Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1366} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 46 46 46 91 91 91 F F F 1 020-039 42 42 42 84 84 84 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 46 46 46 91 91 91 F F F 4 080-099 44 44 44 86 86 86 F F F 5 100-119 46 46 46 91 91 91 F F F 6 120-139 42 42 42 84 84 84 F F F 7 140-159 46 46 46 90 90 90 T T T 8 160-179 44 44 44 87 87 87 F F F 9 180-199 42 42 42 84 84 84 T T T 10 200-219 43 43 43 85 85 85 T T T 11 220-239 43 43 43 85 85 85 F F F 12 240-259 46 46 46 91 91 91 F F F 13 260-279 43 43 43 85 85 85 F F F 14 280-299 44 44 44 88 88 88 F F F 15 300-319 46 46 46 92 92 92 F F F 16 320-339 46 46 46 92 92 92 F F F 17 340-359 42 42 42 84 84 84 T T T 18 360-379 46 46 46 91 91 91 F F F 19 380-399 46 46 46 91 91 91 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 41 41 41 83 83 83 F F F 22 440-459 40 40 40 81 81 81 F F F 23 460-479 46 46 46 91 91 91 F F F 24 480-499 46 46 46 91 91 91 F F F 25 500-519 42 42 42 84 84 84 T T T 26 520-539 41 41 41 83 83 83 F F F 27 540-559 38 38 38 79 79 79 F F F 28 560-579 45 45 45 89 89 89 F F F 29 580-599 45 45 45 89 89 89 F F F Max gen: 46, Max flush: 92 A Difference in extent metadata is expected here On loop 23 Verifying data now No change after verify Loop: 23 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 46 46 46 91 91 91 F F F 1 020-039 42 42 42 84 84 84 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 46 46 46 91 91 91 F F F 4 080-099 44 44 44 86 86 86 F F F 5 100-119 46 46 46 91 91 91 F F F 6 120-139 42 42 42 84 84 84 F F F 7 140-159 47 47 47 93 93 93 F F F 8 160-179 44 44 44 87 87 87 F F F 9 180-199 47 47 47 93 93 93 F F F 10 200-219 47 47 47 93 93 93 F F F 11 220-239 43 43 43 85 85 85 F F F 12 240-259 46 46 46 91 91 91 F F F 13 260-279 43 43 43 85 85 85 F F F 14 280-299 44 44 44 88 88 88 F F F 15 300-319 46 46 46 92 92 92 F F F 16 320-339 46 46 46 92 92 92 F F F 17 340-359 47 47 47 93 93 93 F F F 18 360-379 46 46 46 91 91 91 F F F 19 380-399 46 46 46 91 91 91 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 41 41 41 83 83 83 F F F 22 440-459 40 40 40 81 81 81 F F F 23 460-479 46 46 46 91 91 91 F F F 24 480-499 46 46 46 91 91 91 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 41 41 41 83 83 83 F F F 27 540-559 38 38 38 79 79 79 F F F 28 560-579 45 45 45 89 89 89 F F F 29 580-599 45 45 45 89 89 89 F F F Max gen: 47, Max flush: 93 Begin loop 24 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.758239014Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.758621616Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.758649034Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374} {"msg":"Crucible 4803fa37-9da5-4735-af07-29c6251f41fb has session id: 0b43cda7-2c49-4437-85b0-d749ca30098f","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.758994889Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"Upstairs opts: Upstairs UUID: 4803fa37-9da5-4735-af07-29c6251f41fb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759042626Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"Crucible stats registered with UUID: 4803fa37-9da5-4735-af07-29c6251f41fb","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759068625Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:48","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759369541Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374} {"msg":"4803fa37-9da5-4735-af07-29c6251f41fb active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759455437Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759475856Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"0","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759507565Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"0","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759533983Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759557412Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759579401Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"2","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.75960432Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"2","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759630009Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"io task","client":"0","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759654358Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"io task","client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759679597Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"io task","client":"2","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:36:35.75982901Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"io task","client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:36:35.759862318Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"io task","client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:36:35.759891927Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759915695Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759945624Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"ds_connection connected from Ok(127.0.0.1:52140)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759969573Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"io task","client":"2","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.759992202Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"io task","client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"ds_connection connected from Ok(127.0.0.1:35132)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.760080598Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"io task","client":"0","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.760634172Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"2","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.760661251Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"2","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.76067667Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"0","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.760690639Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"0","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.761292671Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"2","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.76132107Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"0","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.761868394Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"0","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.761894853Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"0","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.761909062Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.761988919Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"2","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.762025637Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"2","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:35.762043666Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.760560229Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"io task","client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"ds_connection connected from Ok(127.0.0.1:56329)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.760865275Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"io task","client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.762275339Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.762308767Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.762861571Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763384037Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763409026Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"[0]R flush_numbers[0..12]: [91, 84, 89, 91, 86, 91, 84, 93, 87, 93, 93, 85]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763427675Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"downstairs","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"[0]R generation[0..12]: [46, 42, 45, 46, 44, 46, 42, 47, 44, 47, 47, 43]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763445054Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"downstairs","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763463003Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"downstairs","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"[1]R flush_numbers[0..12]: [91, 84, 89, 91, 86, 91, 84, 93, 87, 93, 93, 85]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763480462Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"downstairs","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"[1]R generation[0..12]: [46, 42, 45, 46, 44, 46, 42, 47, 44, 47, 47, 43]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763497602Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"downstairs","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763515191Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"downstairs","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"[2]R flush_numbers[0..12]: [91, 84, 89, 91, 86, 91, 84, 93, 87, 93, 93, 85]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.76353204Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"downstairs","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"[2]R generation[0..12]: [46, 42, 45, 46, 44, 46, 42, 47, 44, 47, 47, 43]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763553789Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763575518Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"downstairs","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"Max found gen is 48","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763591847Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"downstairs","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"Generation requested: 48 >= found:48","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763608016Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"downstairs","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"Next flush: 94","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763624216Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"downstairs","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763640145Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"":"downstairs","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763656774Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763671313Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"0","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763684703Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"1","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763697992Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"client":"2","":"downstairs","session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763711432Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763723641Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"4803fa37-9da5-4735-af07-29c6251f41fb is now active with session: 0b43cda7-2c49-4437-85b0-d749ca30098f","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763736421Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2024-07-31T18:36:45.76374834Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374,"session_id":"0b43cda7-2c49-4437-85b0-d749ca30098f"} {"msg":"The guest has finished waiting for activation with:48","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:36:45.763764549Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 256 len 20480 data: 7 6 4 4 4 02/30 Write block 329 len 24576 data: 4 4 4 4 5 7 03/30 Write block 108 len 20480 data: 3 3 3 3 3 04/30 Write block 473 len 24576 data: 4 4 4 7 7 3 05/30 Read block 365 len 24576 06/30 Read block 471 len 16384 07/30 Read block 219 len 28672 08/30 Write block 105 len 4096 data: 5 09/30 Write block 9 len 28672 data: 5 4 4 3 2 2 3 10/30 Read block 188 len 4096 11/30 Read block 182 len 16384 12/30 Read block 451 len 32768 13/30 Read block 540 len 8192 14/30 Read block 266 len 40960 15/30 Write block 165 len 28672 data: 4 5 6 6 6 7 6 16/30 Write block 136 len 36864 data: 3 4 4 3 3 2 4 5 5 17/30 Read block 328 len 36864 18/30 Read block 444 len 32768 19/30 Read block 566 len 8192 20/30 Read block 472 len 24576 21/30 Flush 22/30 Write block 526 len 12288 data: 5 3 3 23/30 Write block 64 len 24576 data: 3 3 3 3 3 3 24/30 Read block 85 len 16384 25/30 Write block 460 len 8192 data: 3 3 26/30 Flush 27/30 Write block 302 len 32768 data: 6 6 5 3 3 3 4 5 28/30 Read block 410 len 12288 29/30 Write block 24 len 32768 data: 7 6 8 7 7 8 6 4 30/30 Write block 440 len 40960 data: 4 4 4 3 3 5 4 4 5 4 ---------------------------------------------------------------- Crucible gen:48 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 8 Done Done Done false 28 Acked 1027 Read 3 Sent Done Done false 29 Acked 1028 Write 8 Sent Done Done false 30 Acked 1029 Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 3 1 1 5 Done 1 3 3 7 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:36:45.854504925Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1374} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 94 94 94 F F F 1 020-039 42 42 42 84 84 84 T T T 2 040-059 45 45 45 89 89 89 F F F 3 060-079 48 48 48 95 95 95 F F F 4 080-099 44 44 44 86 86 86 F F F 5 100-119 48 48 48 94 94 94 F F F 6 120-139 48 48 48 94 94 94 F F F 7 140-159 48 48 48 94 94 94 F F F 8 160-179 48 48 48 94 94 94 F F F 9 180-199 47 47 47 93 93 93 F F F 10 200-219 47 47 47 93 93 93 F F F 11 220-239 43 43 43 85 85 85 F F F 12 240-259 48 48 48 94 94 94 F F F 13 260-279 48 48 48 94 94 94 F F F 14 280-299 44 44 44 88 88 88 F F F 15 300-319 46 46 46 92 92 92 T T T 16 320-339 48 48 48 94 94 94 F F F 17 340-359 47 47 47 93 93 93 F F F 18 360-379 46 46 46 91 91 91 F F F 19 380-399 46 46 46 91 91 91 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 41 41 41 83 83 83 F F F 22 440-459 40 40 40 81 81 81 T T T 23 460-479 48 48 48 95 95 95 F F F 24 480-499 46 46 46 91 91 91 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 48 48 48 95 95 95 F F F 27 540-559 38 38 38 79 79 79 F F F 28 560-579 45 45 45 89 89 89 F F F 29 580-599 45 45 45 89 89 89 F F F Max gen: 48, Max flush: 95 A Difference in extent metadata is expected here On loop 24 Verifying data now No change after verify Loop: 24 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 94 94 94 F F F 1 020-039 49 49 49 96 96 96 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 48 48 48 95 95 95 F F F 4 080-099 44 44 44 86 86 86 F F F 5 100-119 48 48 48 94 94 94 F F F 6 120-139 48 48 48 94 94 94 F F F 7 140-159 48 48 48 94 94 94 F F F 8 160-179 48 48 48 94 94 94 F F F 9 180-199 47 47 47 93 93 93 F F F 10 200-219 47 47 47 93 93 93 F F F 11 220-239 43 43 43 85 85 85 F F F 12 240-259 48 48 48 94 94 94 F F F 13 260-279 48 48 48 94 94 94 F F F 14 280-299 44 44 44 88 88 88 F F F 15 300-319 49 49 49 96 96 96 F F F 16 320-339 48 48 48 94 94 94 F F F 17 340-359 47 47 47 93 93 93 F F F 18 360-379 46 46 46 91 91 91 F F F 19 380-399 46 46 46 91 91 91 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 41 41 41 83 83 83 F F F 22 440-459 49 49 49 96 96 96 F F F 23 460-479 48 48 48 95 95 95 F F F 24 480-499 46 46 46 91 91 91 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 48 48 48 95 95 95 F F F 27 540-559 38 38 38 79 79 79 F F F 28 560-579 45 45 45 89 89 89 F F F 29 580-599 45 45 45 89 89 89 F F F Max gen: 49, Max flush: 96 Begin loop 25 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.199802065Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.200355499Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.200390068Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382} {"msg":"Crucible 5515da52-cbbb-4b06-ad7e-a31313cc3bf6 has session id: db7a68a3-fdb6-4d49-a795-4a49af40e296","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.200459184Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"Upstairs opts: Upstairs UUID: 5515da52-cbbb-4b06-ad7e-a31313cc3bf6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.20055581Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"Crucible stats registered with UUID: 5515da52-cbbb-4b06-ad7e-a31313cc3bf6","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.200588538Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:50","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.200881395Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382} {"msg":"5515da52-cbbb-4b06-ad7e-a31313cc3bf6 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.200951501Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.20097741Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"0","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.200998889Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"0","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.201013299Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"1","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.201033018Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"1","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.201057356Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.201076236Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.201090235Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"io task","client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.201110354Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"io task","client":"0","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.201135003Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"io task","client":"1","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:37:00.201217659Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"io task","client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:37:00.201254367Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"io task","client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"ds_connection connected from Ok(127.0.0.1:61982)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.201280096Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"io task","client":"0","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:37:00.201307545Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.201430499Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.201471737Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"ds_connection connected from Ok(127.0.0.1:54874)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.201496656Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"io task","client":"1","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.201523385Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"io task","client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.20205545Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"0","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.202080099Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"0","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.202094338Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"1","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.202108187Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"1","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.20269303Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"1","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.202719909Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"0","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.203229165Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"0","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.203256134Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"0","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.203271653Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.20332906Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"1","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.20334823Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"1","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:00.203361669Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.202615677Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"io task","client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"ds_connection connected from Ok(127.0.0.1:37819)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.202900563Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"io task","client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.203919416Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.203952594Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.2044691Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205012255Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205034514Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"[0]R flush_numbers[0..12]: [94, 96, 89, 95, 86, 94, 94, 94, 94, 93, 93, 85]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205051913Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"downstairs","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"[0]R generation[0..12]: [48, 49, 45, 48, 44, 48, 48, 48, 48, 47, 47, 43]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205069182Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"downstairs","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205086741Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"downstairs","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"[1]R flush_numbers[0..12]: [94, 96, 89, 95, 86, 94, 94, 94, 94, 93, 93, 85]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.20510369Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"downstairs","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"[1]R generation[0..12]: [48, 49, 45, 48, 44, 48, 48, 48, 48, 47, 47, 43]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.2051208Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"downstairs","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205138239Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"downstairs","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"[2]R flush_numbers[0..12]: [94, 96, 89, 95, 86, 94, 94, 94, 94, 93, 93, 85]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205155208Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"downstairs","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"[2]R generation[0..12]: [48, 49, 45, 48, 44, 48, 48, 48, 48, 47, 47, 43]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205172117Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"downstairs","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:37:10.205194786Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"downstairs","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"Max found gen is 50","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205213675Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"downstairs","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"Generation requested: 50 >= found:50","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205230295Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"downstairs","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"Next flush: 97","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205246534Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"downstairs","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205262613Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"":"downstairs","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205278912Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205293832Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"0","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205310241Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"1","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.20532681Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"client":"2","":"downstairs","session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205343529Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205358829Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"5515da52-cbbb-4b06-ad7e-a31313cc3bf6 is now active with session: db7a68a3-fdb6-4d49-a795-4a49af40e296","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205374748Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205389457Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382,"session_id":"db7a68a3-fdb6-4d49-a795-4a49af40e296"} {"msg":"The guest has finished waiting for activation with:50","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:10.205404416Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 73 len 36864 data: 5 4 4 5 5 4 4 6 6 02/30 Read block 360 len 32768 03/30 Write block 551 len 12288 data: 5 4 6 04/30 Write block 340 len 20480 data: 4 5 5 5 3 05/30 Read block 397 len 8192 06/30 Read block 499 len 28672 07/30 Write block 158 len 36864 data: 5 4 4 5 3 3 3 5 6 08/30 Write block 472 len 16384 data: 5 5 5 5 09/30 Write block 146 len 24576 data: 6 6 7 6 6 4 10/30 Write block 196 len 28672 data: 5 6 6 4 4 5 5 11/30 Flush 12/30 Write block 487 len 24576 data: 2 2 2 2 2 2 13/30 Write block 207 len 36864 data: 5 6 5 5 5 4 4 4 5 14/30 Read block 300 len 16384 15/30 Flush 16/30 Write block 235 len 8192 data: 7 7 17/30 Flush 18/30 Write block 529 len 12288 data: 2 3 3 19/30 Write block 131 len 16384 data: 3 3 3 3 20/30 Write block 171 len 20480 data: 7 7 7 4 5 21/30 Read block 558 len 8192 22/30 Write block 572 len 8192 data: 8 8 23/30 Read block 159 len 8192 24/30 Read block 78 len 24576 25/30 Write block 577 len 16384 data: 3 4 5 6 26/30 Read block 300 len 24576 27/30 Read block 410 len 36864 28/30 Read block 382 len 32768 29/30 Read block 446 len 4096 30/30 Write block 352 len 16384 data: 7 8 9 8 ---------------------------------------------------------------- Crucible gen:50 GIO:true work queues: Upstairs:0 downstairs:13 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 18 Acked 1017 Write 3 Done Done Done false 19 Acked 1018 Write 4 Done Done Done false 20 Acked 1019 Write 5 Done Done Done false 21 Acked 1020 Read 2 Done Done Done false 22 Acked 1021 Write 2 Done Done Done false 23 Acked 1022 Read 2 Done Done Done false 24 Acked 1023 Read 6 Done Done Done false 25 Acked 1024 Write 4 Done Done Done false 26 Acked 1025 Read 6 Done Done Done false 27 Acked 1026 Read 9 Done Done Done false 28 Acked 1027 Read 8 Done Done Done false 29 Acked 1028 Read 1 Done Done Done false 30 Acked 1029 Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 12 12 12 36 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1016 1016 1016 Downstairs last five completed: 1016 1015 1014 1013 1012 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:37:10.363214723Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1382} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 94 94 94 F F F 1 020-039 49 49 49 96 96 96 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 50 50 50 97 97 97 F F F 4 080-099 50 50 50 97 97 97 F F F 5 100-119 48 48 48 94 94 94 F F F 6 120-139 48 48 48 94 94 94 T T T 7 140-159 50 50 50 97 97 97 F F F 8 160-179 50 50 50 97 97 97 T T T 9 180-199 50 50 50 97 97 97 F F F 10 200-219 50 50 50 98 98 98 F F F 11 220-239 50 50 50 99 99 99 F F F 12 240-259 48 48 48 94 94 94 F F F 13 260-279 48 48 48 94 94 94 F F F 14 280-299 44 44 44 88 88 88 F F F 15 300-319 49 49 49 96 96 96 F F F 16 320-339 48 48 48 94 94 94 F F F 17 340-359 50 50 50 97 97 97 T T T 18 360-379 46 46 46 91 91 91 F F F 19 380-399 46 46 46 91 91 91 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 41 41 41 83 83 83 F F F 22 440-459 49 49 49 96 96 96 F F F 23 460-479 50 50 50 97 97 97 F F F 24 480-499 50 50 50 98 98 98 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 48 48 48 95 95 95 T T T 27 540-559 50 50 50 97 97 97 F F F 28 560-579 45 45 45 89 89 89 T T T 29 580-599 45 45 45 89 89 89 T T T Max gen: 50, Max flush: 99 A Difference in extent metadata is expected here On loop 25 Verifying data now No change after verify Loop: 25 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 48 94 94 94 F F F 1 020-039 49 49 49 96 96 96 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 50 50 50 97 97 97 F F F 4 080-099 50 50 50 97 97 97 F F F 5 100-119 48 48 48 94 94 94 F F F 6 120-139 51 51 51 100 100 100 F F F 7 140-159 50 50 50 97 97 97 F F F 8 160-179 51 51 51 100 100 100 F F F 9 180-199 50 50 50 97 97 97 F F F 10 200-219 50 50 50 98 98 98 F F F 11 220-239 50 50 50 99 99 99 F F F 12 240-259 48 48 48 94 94 94 F F F 13 260-279 48 48 48 94 94 94 F F F 14 280-299 44 44 44 88 88 88 F F F 15 300-319 49 49 49 96 96 96 F F F 16 320-339 48 48 48 94 94 94 F F F 17 340-359 51 51 51 100 100 100 F F F 18 360-379 46 46 46 91 91 91 F F F 19 380-399 46 46 46 91 91 91 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 41 41 41 83 83 83 F F F 22 440-459 49 49 49 96 96 96 F F F 23 460-479 50 50 50 97 97 97 F F F 24 480-499 50 50 50 98 98 98 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 51 51 51 100 100 100 F F F 27 540-559 50 50 50 97 97 97 F F F 28 560-579 51 51 51 100 100 100 F F F 29 580-599 51 51 51 100 100 100 F F F Max gen: 51, Max flush: 100 Begin loop 26 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.984666985Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.984972411Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.98499194Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390} {"msg":"Crucible 3000dc49-2b2b-4c72-b94d-175eeee54e2b has session id: 0e8f62fd-009f-4bb0-9a29-f96c0ad14add","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.985385702Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"Upstairs opts: Upstairs UUID: 3000dc49-2b2b-4c72-b94d-175eeee54e2b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.98542968Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"Crucible stats registered with UUID: 3000dc49-2b2b-4c72-b94d-175eeee54e2b","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.985453658Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:52","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.985781733Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390} {"msg":"3000dc49-2b2b-4c72-b94d-175eeee54e2b active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.985813422Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.985836721Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"0","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.985867539Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"0","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.985890768Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"1","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.985914387Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"1","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.985940776Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.985965705Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.985989344Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"io task","client":"0","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.986013343Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"io task","client":"1","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.986036321Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"io task","client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:37:24.986133877Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"io task","client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:37:24.986171735Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"io task","client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:37:24.986201064Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.986223123Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.986248281Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"ds_connection connected from Ok(127.0.0.1:57472)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.98627135Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"io task","client":"0","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"ds_connection connected from Ok(127.0.0.1:64075)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.986296109Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"io task","client":"1","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.986365246Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"io task","client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.987091732Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"0","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.987118581Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"0","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.98713386Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"1","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.98714807Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"1","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.987630167Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"1","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.987749462Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"0","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.988117274Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"1","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.988155292Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"1","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.988183351Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.988219389Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"0","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.988248658Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"0","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:24.988271947Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.987112365Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"io task","client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"ds_connection connected from Ok(127.0.0.1:53206)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.987235859Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"io task","client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.988246292Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.98827971Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989016486Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989598189Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989624567Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"[0]R flush_numbers[0..12]: [94, 96, 89, 97, 97, 94, 100, 97, 100, 97, 98, 99]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989640667Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"downstairs","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"[0]R generation[0..12]: [48, 49, 45, 50, 50, 48, 51, 50, 51, 50, 50, 50]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989655196Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"downstairs","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989669345Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"downstairs","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"[1]R flush_numbers[0..12]: [94, 96, 89, 97, 97, 94, 100, 97, 100, 97, 98, 99]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989683074Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"downstairs","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"[1]R generation[0..12]: [48, 49, 45, 50, 50, 48, 51, 50, 51, 50, 50, 50]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989696584Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"downstairs","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989710433Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"downstairs","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"[2]R flush_numbers[0..12]: [94, 96, 89, 97, 97, 94, 100, 97, 100, 97, 98, 99]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989723843Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"downstairs","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"[2]R generation[0..12]: [48, 49, 45, 50, 50, 48, 51, 50, 51, 50, 50, 50]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989743242Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"downstairs","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989758571Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"downstairs","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"Max found gen is 52","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.98977173Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"downstairs","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"Generation requested: 52 >= found:52","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:37:34.98978514Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"downstairs","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"Next flush: 101","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989798529Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"downstairs","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989811739Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"":"downstairs","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989824828Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989836487Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"0","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989849597Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"1","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989862846Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"client":"2","":"downstairs","session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989876725Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989888805Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"3000dc49-2b2b-4c72-b94d-175eeee54e2b is now active with session: 0e8f62fd-009f-4bb0-9a29-f96c0ad14add","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989901454Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989913244Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390,"session_id":"0e8f62fd-009f-4bb0-9a29-f96c0ad14add"} {"msg":"The guest has finished waiting for activation with:52","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:34.989925243Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 559 len 20480 data: 3 3 4 4 6 02/30 Write block 130 len 40960 data: 4 4 4 4 4 3 4 5 5 4 03/30 Read block 489 len 8192 04/30 Read block 454 len 12288 05/30 Write block 598 len 4096 data: 3 06/30 Write block 148 len 32768 data: 8 7 7 5 5 5 5 5 07/30 Write block 378 len 20480 data: 6 5 5 5 6 08/30 Read block 238 len 40960 09/30 Read block 29 len 32768 10/30 Write block 425 len 32768 data: 4 6 6 7 8 10 9 10 11/30 Flush 12/30 Write block 228 len 24576 data: 5 6 6 6 8 6 13/30 Read block 590 len 36864 14/30 Read block 499 len 40960 15/30 Flush 16/30 Write block 123 len 32768 data: 5 4 5 4 4 4 4 5 17/30 Write block 538 len 28672 data: 6 5 4 5 4 5 6 18/30 Flush 19/30 Write block 16 len 12288 data: 2 4 4 20/30 Flush 21/30 Flush 22/30 Flush 23/30 Write block 276 len 40960 data: 3 3 3 4 5 4 4 4 6 6 24/30 Read block 485 len 4096 25/30 Read block 332 len 20480 26/30 Write block 65 len 16384 data: 4 4 4 4 27/30 Read block 173 len 12288 28/30 Write block 81 len 28672 data: 7 7 7 8 8 5 4 29/30 Write block 279 len 28672 data: 5 6 5 5 5 7 7 30/30 Write block 442 len 20480 data: 5 4 4 6 5 ---------------------------------------------------------------- Crucible gen:52 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 23 Acked 1022 Write 10 Done Done Done false 24 Acked 1023 Read 1 Done Done Done false 25 Acked 1024 Read 5 Done Done Done false 26 Acked 1025 Write 4 Done Done Done false 27 Acked 1026 Read 3 Done Done Done false 28 Acked 1027 Write 7 Done Done Done false 29 Acked 1028 Write 7 Done Done Done false 30 Acked 1029 Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1021 1021 1021 Downstairs last five completed: 1021 1020 1019 1018 1017 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:37:35.040501303Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1390} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 52 52 52 104 104 104 F F F 1 020-039 49 49 49 96 96 96 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 50 50 50 97 97 97 T T T 4 080-099 50 50 50 97 97 97 T T T 5 100-119 48 48 48 94 94 94 F F F 6 120-139 52 52 52 103 103 103 F F F 7 140-159 52 52 52 101 101 101 F F F 8 160-179 51 51 51 100 100 100 F F F 9 180-199 50 50 50 97 97 97 F F F 10 200-219 50 50 50 98 98 98 F F F 11 220-239 52 52 52 102 102 102 F F F 12 240-259 48 48 48 94 94 94 F F F 13 260-279 48 48 48 94 94 94 T T T 14 280-299 44 44 44 88 88 88 T T T 15 300-319 49 49 49 96 96 96 F F F 16 320-339 48 48 48 94 94 94 F F F 17 340-359 51 51 51 100 100 100 F F F 18 360-379 52 52 52 101 101 101 F F F 19 380-399 52 52 52 101 101 101 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 52 52 52 101 101 101 F F F 22 440-459 49 49 49 96 96 96 T T T 23 460-479 50 50 50 97 97 97 F F F 24 480-499 50 50 50 98 98 98 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 52 52 52 103 103 103 F F F 27 540-559 52 52 52 103 103 103 F F F 28 560-579 52 52 52 101 101 101 F F F 29 580-599 52 52 52 101 101 101 F F F Max gen: 52, Max flush: 104 A Difference in extent metadata is expected here On loop 26 Verifying data now No change after verify Loop: 26 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 52 52 52 104 104 104 F F F 1 020-039 49 49 49 96 96 96 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 53 53 53 105 105 105 F F F 4 080-099 53 53 53 105 105 105 F F F 5 100-119 48 48 48 94 94 94 F F F 6 120-139 52 52 52 103 103 103 F F F 7 140-159 52 52 52 101 101 101 F F F 8 160-179 51 51 51 100 100 100 F F F 9 180-199 50 50 50 97 97 97 F F F 10 200-219 50 50 50 98 98 98 F F F 11 220-239 52 52 52 102 102 102 F F F 12 240-259 48 48 48 94 94 94 F F F 13 260-279 53 53 53 105 105 105 F F F 14 280-299 53 53 53 105 105 105 F F F 15 300-319 49 49 49 96 96 96 F F F 16 320-339 48 48 48 94 94 94 F F F 17 340-359 51 51 51 100 100 100 F F F 18 360-379 52 52 52 101 101 101 F F F 19 380-399 52 52 52 101 101 101 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 52 52 52 101 101 101 F F F 22 440-459 53 53 53 105 105 105 F F F 23 460-479 50 50 50 97 97 97 F F F 24 480-499 50 50 50 98 98 98 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 52 52 52 103 103 103 F F F 27 540-559 52 52 52 103 103 103 F F F 28 560-579 52 52 52 101 101 101 F F F 29 580-599 52 52 52 101 101 101 F F F Max gen: 53, Max flush: 105 Begin loop 27 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.68052945Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.680829836Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.680850685Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398} {"msg":"Crucible 82b87852-3a24-4c46-b590-2f0b80bc9e1d has session id: 57d9bfe3-0a2b-42b0-8bc0-30c56801d13f","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.681270276Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"Upstairs opts: Upstairs UUID: 82b87852-3a24-4c46-b590-2f0b80bc9e1d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.681315594Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"Crucible stats registered with UUID: 82b87852-3a24-4c46-b590-2f0b80bc9e1d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.681343142Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:54","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.68161628Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398} {"msg":"82b87852-3a24-4c46-b590-2f0b80bc9e1d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.681646488Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.681670907Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"0","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.681700265Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"0","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.681722755Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.681745993Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.681768472Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"2","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.681794311Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"2","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.68182047Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"io task","client":"0","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.681847019Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"io task","client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.681873108Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"io task","client":"2","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:37:49.68202638Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"io task","client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:37:49.682066179Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"io task","client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"ds_connection connected from Ok(127.0.0.1:65337)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.682090547Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"io task","client":"0","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"ds_connection connected from Ok(127.0.0.1:37505)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.682175493Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"io task","client":"2","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:37:49.682197992Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.682211592Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.682225101Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.682381214Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"io task","client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.682916619Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"0","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.682943767Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"0","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.682959507Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"2","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.682973156Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"2","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.683586777Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"2","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.683614486Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"0","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.684147881Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"2","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.68417279Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"2","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.684187549Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.684199849Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"0","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.684213398Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"0","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:49.684230167Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.683021427Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"io task","client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"ds_connection connected from Ok(127.0.0.1:43075)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.683123452Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"io task","client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.684368674Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.684403832Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.684956196Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685504391Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.68552575Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"[0]R flush_numbers[0..12]: [104, 96, 89, 105, 105, 94, 103, 101, 100, 97, 98, 102]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685540969Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"downstairs","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"[0]R generation[0..12]: [52, 49, 45, 53, 53, 48, 52, 52, 51, 50, 50, 52]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685555249Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"downstairs","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685569568Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"downstairs","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"[1]R flush_numbers[0..12]: [104, 96, 89, 105, 105, 94, 103, 101, 100, 97, 98, 102]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685583437Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"downstairs","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"[1]R generation[0..12]: [52, 49, 45, 53, 53, 48, 52, 52, 51, 50, 50, 52]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685597296Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"downstairs","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685611236Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"downstairs","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"[2]R flush_numbers[0..12]: [104, 96, 89, 105, 105, 94, 103, 101, 100, 97, 98, 102]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685629525Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"downstairs","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"[2]R generation[0..12]: [52, 49, 45, 53, 53, 48, 52, 52, 51, 50, 50, 52]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685643624Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"downstairs","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685657594Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"downstairs","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"Max found gen is 54","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685670423Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"downstairs","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"Generation requested: 54 >= found:54","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685683732Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"downstairs","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"Next flush: 106","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:37:59.685696612Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"downstairs","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685712481Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"":"downstairs","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.68572591Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.68573797Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"0","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685751379Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"1","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685764539Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"client":"2","":"downstairs","session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685778108Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685790327Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"82b87852-3a24-4c46-b590-2f0b80bc9e1d is now active with session: 57d9bfe3-0a2b-42b0-8bc0-30c56801d13f","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685803137Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685815156Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398,"session_id":"57d9bfe3-0a2b-42b0-8bc0-30c56801d13f"} {"msg":"The guest has finished waiting for activation with:54","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:37:59.685827296Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 464 len 32768 data: 3 3 3 3 2 2 3 5 02/30 Flush 03/30 Write block 245 len 20480 data: 4 6 6 6 6 04/30 Write block 10 len 8192 data: 5 5 05/30 Write block 570 len 40960 data: 6 8 9 9 8 6 5 4 5 6 06/30 Read block 481 len 4096 07/30 Write block 576 len 8192 data: 6 5 08/30 Write block 566 len 4096 data: 7 09/30 Read block 405 len 4096 10/30 Write block 238 len 24576 data: 6 5 3 3 3 3 11/30 Read block 328 len 12288 12/30 Read block 30 len 4096 13/30 Flush 14/30 Write block 114 len 16384 data: 3 3 3 3 15/30 Read block 472 len 8192 16/30 Flush 17/30 Write block 573 len 24576 data: 10 9 7 7 6 6 18/30 Read block 118 len 12288 19/30 Read block 184 len 32768 20/30 Read block 103 len 4096 21/30 Write block 276 len 20480 data: 4 4 4 6 7 22/30 Read block 7 len 24576 23/30 Read block 366 len 28672 24/30 Write block 251 len 40960 data: 4 5 4 7 7 8 7 5 5 5 25/30 Write block 182 len 40960 data: 5 7 6 7 5 5 5 4 5 3 26/30 Write block 186 len 12288 data: 6 6 6 27/30 Read block 553 len 20480 28/30 Write block 344 len 28672 data: 4 4 3 4 4 6 7 29/30 Read block 276 len 8192 30/30 Read block 13 len 12288 ---------------------------------------------------------------- Crucible gen:54 GIO:true work queues: Upstairs:0 downstairs:14 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 17 Acked 1016 Write 6 Done Done Done false 18 Acked 1017 Read 3 Done Done Done false 19 Acked 1018 Read 8 Done Done Done false 20 Acked 1019 Read 1 Done Done Done false 21 Acked 1020 Write 5 Done Done Done false 22 Acked 1021 Read 6 Done Done Done false 23 Acked 1022 Read 7 Done Done Done false 24 Acked 1023 Write 10 Done Done Done false 25 Acked 1024 Write 10 Done Done Done false 26 Acked 1025 Write 3 Done Done Done false 27 Acked 1026 Read 5 Done Done Done false 28 Acked 1027 Write 7 Done Done Done false 29 Acked 1028 Read 2 Done Done Done false 30 Acked 1029 Read 3 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 14 14 14 42 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1015 1015 1015 Downstairs last five completed: 1015 1014 1013 1012 1011 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:37:59.755949604Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1398} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 54 54 54 107 107 107 F F F 1 020-039 49 49 49 96 96 96 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 53 53 53 105 105 105 F F F 4 080-099 53 53 53 105 105 105 F F F 5 100-119 54 54 54 108 108 108 F F F 6 120-139 52 52 52 103 103 103 F F F 7 140-159 52 52 52 101 101 101 F F F 8 160-179 51 51 51 100 100 100 F F F 9 180-199 50 50 50 97 97 97 T T T 10 200-219 50 50 50 98 98 98 F F F 11 220-239 54 54 54 107 107 107 F F F 12 240-259 54 54 54 107 107 107 T T T 13 260-279 53 53 53 105 105 105 T T T 14 280-299 53 53 53 105 105 105 T T T 15 300-319 49 49 49 96 96 96 F F F 16 320-339 48 48 48 94 94 94 F F F 17 340-359 51 51 51 100 100 100 T T T 18 360-379 52 52 52 101 101 101 F F F 19 380-399 52 52 52 101 101 101 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 52 52 52 101 101 101 F F F 22 440-459 53 53 53 105 105 105 F F F 23 460-479 54 54 54 106 106 106 F F F 24 480-499 50 50 50 98 98 98 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 52 52 52 103 103 103 F F F 27 540-559 52 52 52 103 103 103 F F F 28 560-579 54 54 54 107 107 107 T T T 29 580-599 52 52 52 101 101 101 F F F Max gen: 54, Max flush: 108 A Difference in extent metadata is expected here On loop 27 Verifying data now No change after verify Loop: 27 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 54 54 54 107 107 107 F F F 1 020-039 49 49 49 96 96 96 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 53 53 53 105 105 105 F F F 4 080-099 53 53 53 105 105 105 F F F 5 100-119 54 54 54 108 108 108 F F F 6 120-139 52 52 52 103 103 103 F F F 7 140-159 52 52 52 101 101 101 F F F 8 160-179 51 51 51 100 100 100 F F F 9 180-199 55 55 55 109 109 109 F F F 10 200-219 50 50 50 98 98 98 F F F 11 220-239 54 54 54 107 107 107 F F F 12 240-259 55 55 55 109 109 109 F F F 13 260-279 55 55 55 109 109 109 F F F 14 280-299 55 55 55 109 109 109 F F F 15 300-319 49 49 49 96 96 96 F F F 16 320-339 48 48 48 94 94 94 F F F 17 340-359 55 55 55 109 109 109 F F F 18 360-379 52 52 52 101 101 101 F F F 19 380-399 52 52 52 101 101 101 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 52 52 52 101 101 101 F F F 22 440-459 53 53 53 105 105 105 F F F 23 460-479 54 54 54 106 106 106 F F F 24 480-499 50 50 50 98 98 98 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 52 52 52 103 103 103 F F F 27 540-559 52 52 52 103 103 103 F F F 28 560-579 55 55 55 109 109 109 F F F 29 580-599 52 52 52 101 101 101 F F F Max gen: 55, Max flush: 109 Begin loop 28 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.44177397Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.442119474Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.442138793Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406} {"msg":"Crucible d6303ecb-0f81-4aeb-8ea0-4460d26682c2 has session id: eb08056a-a47f-40b1-82cd-de693282fda0","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.442508536Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"Upstairs opts: Upstairs UUID: d6303ecb-0f81-4aeb-8ea0-4460d26682c2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.442545134Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"Crucible stats registered with UUID: d6303ecb-0f81-4aeb-8ea0-4460d26682c2","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.442561183Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:56","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.44284894Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406} {"msg":"d6303ecb-0f81-4aeb-8ea0-4460d26682c2 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.442958135Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.442983874Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.443004713Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.443018872Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"1","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.443032361Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"1","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.443045851Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"2","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.44305914Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"2","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.443105218Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"io task","client":"2","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.443136046Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"io task","client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.443161445Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"io task","client":"1","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"ds_connection connected from Ok(127.0.0.1:58214)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.443309078Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"io task","client":"2","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:38:14.443347627Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"io task","client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:38:14.443379285Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"io task","client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:38:14.44350108Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.443531818Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.443557447Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"ds_connection connected from Ok(127.0.0.1:42410)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.443582616Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"io task","client":"1","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.443606984Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"io task","client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.444074733Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"1","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.444102852Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"1","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.444117941Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"2","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.44413301Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"2","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.444737482Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"1","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.444765741Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"2","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.445319645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"2","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.445357243Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"2","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.445376042Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.445390061Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"1","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.445403901Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"1","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:14.44541703Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.445516439Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"io task","client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"ds_connection connected from Ok(127.0.0.1:42540)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.445884512Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"io task","client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.446850007Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.446876065Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.447329494Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.447862139Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.447885209Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"[0]R flush_numbers[0..12]: [107, 96, 89, 105, 105, 108, 103, 101, 100, 109, 98, 107]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.447899728Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"downstairs","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"[0]R generation[0..12]: [54, 49, 45, 53, 53, 54, 52, 52, 51, 55, 50, 54]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.447913707Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"downstairs","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.447927807Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"downstairs","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"[1]R flush_numbers[0..12]: [107, 96, 89, 105, 105, 108, 103, 101, 100, 109, 98, 107]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.447941666Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"downstairs","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"[1]R generation[0..12]: [54, 49, 45, 53, 53, 54, 52, 52, 51, 55, 50, 54]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.447960055Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"downstairs","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.447974544Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"downstairs","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"[2]R flush_numbers[0..12]: [107, 96, 89, 105, 105, 108, 103, 101, 100, 109, 98, 107]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.447988214Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"downstairs","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"[2]R generation[0..12]: [54, 49, 45, 53, 53, 54, 52, 52, 51, 55, 50, 54]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.448016442Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"downstairs","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.448031372Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"downstairs","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"Max found gen is 56","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.448044951Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"downstairs","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"Wait for a query_work_queue command to finish before sending IO msg":"Generation requested: 56 >= found:56","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.4480617Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"downstairs","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"Next flush: 110","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.44807484Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"downstairs","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.448087869Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"":"downstairs","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.448100788Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.448112628Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"0","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.448125597Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"1","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.448138407Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"client":"2","":"downstairs","session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.448151526Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.448163676Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"d6303ecb-0f81-4aeb-8ea0-4460d26682c2 is now active with session: eb08056a-a47f-40b1-82cd-de693282fda0","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.448178505Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.448190544Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406,"session_id":"eb08056a-a47f-40b1-82cd-de693282fda0"} {"msg":"The guest has finished waiting for activation with:56","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:24.448202484Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 114 len 24576 data: 4 4 4 4 3 3 02/30 Write block 373 len 12288 data: 4 3 4 03/30 Read block 386 len 28672 04/30 Write block 20 len 36864 data: 5 6 6 7 8 7 9 8 8 05/30 Flush 06/30 Flush 07/30 Flush 08/30 Write block 439 len 32768 data: 6 5 5 6 5 5 7 6 09/30 Write block 193 len 20480 data: 4 4 5 6 7 10/30 Read block 380 len 12288 11/30 Read block 516 len 16384 12/30 Read block 578 len 4096 13/30 Write block 262 len 4096 data: 5 14/30 Flush 15/30 Write block 429 len 12288 data: 9 11 10 16/30 Read block 540 len 24576 17/30 Read block 136 len 16384 18/30 Write block 468 len 28672 data: 3 3 4 6 6 6 6 19/30 Write block 268 len 20480 data: 5 6 5 4 4 20/30 Read block 579 len 32768 21/30 Write block 16 len 24576 data: 3 5 5 4 6 7 22/30 Read block 196 len 40960 23/30 Read block 269 len 28672 24/30 Read block 251 len 40960 25/30 Read block 271 len 16384 26/30 Write block 185 len 28672 data: 8 7 7 7 5 6 4 27/30 Read block 204 len 4096 28/30 Read block 121 len 8192 29/30 Read block 431 len 16384 30/30 Write block 119 len 16384 data: 4 6 4 4 ---------------------------------------------------------------- Crucible gen:56 GIO:true work queues: Upstairs:0 downstairs:16 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 15 Acked 1014 Write 3 Done Done Done false 16 Acked 1015 Read 6 Done Done Done false 17 Acked 1016 Read 4 Done Done Done false 18 Acked 1017 Write 7 Done Done Done false 19 Acked 1018 Write 5 Done Done Done false 20 Acked 1019 Read 8 Done Done Done false 21 Acked 1020 Write 6 Done Done Done false 22 Acked 1021 Read 10 Done Done Done false 23 Acked 1022 Read 7 Done Done Done false 24 Acked 1023 Read 10 Done Done Done false 25 Acked 1024 Read 4 Done Done Done false 26 Acked 1025 Write 7 Done Done Done false 27 Acked 1026 Read 1 Done Done Done false 28 Acked 1027 Read 2 Done Done Done false 29 Acked 1028 Read 4 Done Done Done false 30 Acked 1029 Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 15 15 15 45 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1013 1013 1013 Downstairs last five completed: 1013 1012 1011 1010 1009 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:38:24.531794033Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1406} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 54 54 54 107 107 107 T T T 1 020-039 56 56 56 110 110 110 T T T 2 040-059 45 45 45 89 89 89 F F F 3 060-079 53 53 53 105 105 105 F F F 4 080-099 53 53 53 105 105 105 F F F 5 100-119 56 56 56 110 110 110 T T T 6 120-139 52 52 52 103 103 103 T T T 7 140-159 52 52 52 101 101 101 F F F 8 160-179 51 51 51 100 100 100 F F F 9 180-199 56 56 56 113 113 113 T T T 10 200-219 50 50 50 98 98 98 F F F 11 220-239 54 54 54 107 107 107 F F F 12 240-259 55 55 55 109 109 109 F F F 13 260-279 56 56 56 113 113 113 T T T 14 280-299 55 55 55 109 109 109 F F F 15 300-319 49 49 49 96 96 96 F F F 16 320-339 48 48 48 94 94 94 F F F 17 340-359 55 55 55 109 109 109 F F F 18 360-379 56 56 56 110 110 110 F F F 19 380-399 52 52 52 101 101 101 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 56 56 56 113 113 113 T T T 22 440-459 56 56 56 113 113 113 F F F 23 460-479 54 54 54 106 106 106 T T T 24 480-499 50 50 50 98 98 98 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 52 52 52 103 103 103 F F F 27 540-559 52 52 52 103 103 103 F F F 28 560-579 55 55 55 109 109 109 F F F 29 580-599 52 52 52 101 101 101 F F F Max gen: 56, Max flush: 113 A Difference in extent metadata is expected here On loop 28 Verifying data now No change after verify Loop: 28 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 114 114 114 F F F 1 020-039 57 57 57 114 114 114 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 53 53 53 105 105 105 F F F 4 080-099 53 53 53 105 105 105 F F F 5 100-119 57 57 57 114 114 114 F F F 6 120-139 57 57 57 114 114 114 F F F 7 140-159 52 52 52 101 101 101 F F F 8 160-179 51 51 51 100 100 100 F F F 9 180-199 57 57 57 114 114 114 F F F 10 200-219 50 50 50 98 98 98 F F F 11 220-239 54 54 54 107 107 107 F F F 12 240-259 55 55 55 109 109 109 F F F 13 260-279 57 57 57 114 114 114 F F F 14 280-299 55 55 55 109 109 109 F F F 15 300-319 49 49 49 96 96 96 F F F 16 320-339 48 48 48 94 94 94 F F F 17 340-359 55 55 55 109 109 109 F F F 18 360-379 56 56 56 110 110 110 F F F 19 380-399 52 52 52 101 101 101 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 57 57 57 114 114 114 F F F 22 440-459 56 56 56 113 113 113 F F F 23 460-479 57 57 57 114 114 114 F F F 24 480-499 50 50 50 98 98 98 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 52 52 52 103 103 103 F F F 27 540-559 52 52 52 103 103 103 F F F 28 560-579 55 55 55 109 109 109 F F F 29 580-599 52 52 52 101 101 101 F F F Max gen: 57, Max flush: 114 Begin loop 29 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.429305345Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.429933666Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.429972994Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414} {"msg":"Crucible 3076a390-3a40-4c13-9d50-9900be1d2199 has session id: e3656351-2982-4805-b83e-555b0b611dc0","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.430003273Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"Upstairs opts: Upstairs UUID: 3076a390-3a40-4c13-9d50-9900be1d2199, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.430039071Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"Crucible stats registered with UUID: 3076a390-3a40-4c13-9d50-9900be1d2199","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.43005437Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:58","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.430354636Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414} {"msg":"3076a390-3a40-4c13-9d50-9900be1d2199 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.430522319Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.430540818Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.430560337Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.430573926Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"1","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.430587376Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"1","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.430600695Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"2","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.430613994Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"2","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.430632463Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"io task","client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.430646443Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"io task","client":"1","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.430659872Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"io task","client":"2","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:38:39.430946999Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"io task","client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:38:39.431085692Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"io task","client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"ds_connection connected from Ok(127.0.0.1:51423)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.431121031Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"io task","client":"2","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:38:39.431150069Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.431172128Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.431197677Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.431224396Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"io task","client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"ds_connection connected from Ok(127.0.0.1:51874)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.431256844Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"io task","client":"1","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.43156174Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"2","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.431590189Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"2","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.431698314Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"1","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.431751761Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"1","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.432056797Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"2","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.432173171Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"1","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.432597472Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"2","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.432619611Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"2","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.4326333Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.432699617Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"1","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.432718996Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"1","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:39.432732405Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.431872378Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"io task","client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"ds_connection connected from Ok(127.0.0.1:44438)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.43225244Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"io task","client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.433273583Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.433306921Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.4337536Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434293315Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434315814Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"[0]R flush_numbers[0..12]: [114, 114, 89, 105, 105, 114, 114, 101, 100, 114, 98, 107]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434333553Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"downstairs","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"[0]R generation[0..12]: [57, 57, 45, 53, 53, 57, 57, 52, 51, 57, 50, 54]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434350952Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"downstairs","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434368721Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"downstairs","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"[1]R flush_numbers[0..12]: [114, 114, 89, 105, 105, 114, 114, 101, 100, 114, 98, 107]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.43439116Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"downstairs","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"[1]R generation[0..12]: [57, 57, 45, 53, 53, 57, 57, 52, 51, 57, 50, 54]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.43440869Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"downstairs","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434426179Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"downstairs","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"[2]R flush_numbers[0..12]: [114, 114, 89, 105, 105, 114, 114, 101, 100, 114, 98, 107]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434443228Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"downstairs","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"[2]R generation[0..12]: [57, 57, 45, 53, 53, 57, 57, 52, 51, 57, 50, 54]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434460167Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"downstairs","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"Wait for a query_work_queue command to finish before sending IO [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434480926Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"downstairs","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"Max found gen is 58","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434497295Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"downstairs","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"Generation requested: 58 >= found:58","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434513695Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"downstairs","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"Next flush: 115","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434529574Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"downstairs","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434545483Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"":"downstairs","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434561743Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434576572Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"0","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434592981Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"1","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.43460934Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"client":"2","":"downstairs","session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.43462804Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434643629Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"3076a390-3a40-4c13-9d50-9900be1d2199 is now active with session: e3656351-2982-4805-b83e-555b0b611dc0","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434659558Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434674347Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414,"session_id":"e3656351-2982-4805-b83e-555b0b611dc0"} {"msg":"The guest has finished waiting for activation with:58","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:38:49.434689526Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 550 len 40960 data: 6 6 5 7 6 6 5 3 3 4 02/30 Write block 242 len 12288 data: 4 4 3 03/30 Write block 119 len 20480 data: 5 7 5 5 6 04/30 Flush 05/30 Write block 531 len 8192 data: 4 4 06/30 Write block 520 len 4096 data: 3 07/30 Read block 475 len 20480 08/30 Flush 09/30 Write block 531 len 20480 data: 5 5 6 6 6 10/30 Write block 73 len 32768 data: 6 5 5 6 6 5 5 7 11/30 Flush 12/30 Write block 357 len 12288 data: 8 8 7 13/30 Write block 178 len 28672 data: 3 4 3 5 6 8 7 14/30 Write block 198 len 36864 data: 7 5 5 6 6 5 6 4 4 15/30 Read block 200 len 40960 16/30 Read block 246 len 32768 17/30 Write block 273 len 32768 data: 3 4 5 5 5 5 7 8 18/30 Read block 462 len 40960 19/30 Write block 221 len 28672 data: 7 7 5 3 4 3 4 20/30 Read block 297 len 8192 21/30 Write block 569 len 36864 data: 4 7 9 10 11 10 8 8 7 22/30 Read block 222 len 12288 23/30 Read block 319 len 4096 24/30 Write block 429 len 36864 data: 10 12 11 11 9 7 8 7 6 25/30 Flush 26/30 Write block 320 len 36864 data: 6 5 4 5 4 4 3 4 5 27/30 Read block 336 len 36864 28/30 Read block 270 len 40960 29/30 Read block 121 len 28672 30/30 Write block 357 len 16384 data: 9 9 8 6 ---------------------------------------------------------------- Crucible gen:58 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 9 Done Done Done false 28 Acked 1027 Read 9 Done Done Done false 29 Acked 1028 Read 10 Done Done Done false 30 Acked 1029 Read 7 Done Done Done false 31 Acked 1030 Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:38:49.493325761Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1414} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 114 114 114 F F F 1 020-039 57 57 57 114 114 114 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 58 58 58 117 117 117 F F F 4 080-099 58 58 58 117 117 117 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 58 58 58 115 115 115 F F F 7 140-159 52 52 52 101 101 101 F F F 8 160-179 58 58 58 118 118 118 F F F 9 180-199 58 58 58 118 118 118 F F F 10 200-219 58 58 58 118 118 118 F F F 11 220-239 58 58 58 118 118 118 F F F 12 240-259 58 58 58 115 115 115 F F F 13 260-279 58 58 58 118 118 118 F F F 14 280-299 58 58 58 118 118 118 F F F 15 300-319 49 49 49 96 96 96 F F F 16 320-339 48 48 48 94 94 94 T T T 17 340-359 58 58 58 118 118 118 T T T 18 360-379 56 56 56 110 110 110 T T T 19 380-399 52 52 52 101 101 101 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 58 58 58 119 119 119 F F F 22 440-459 56 56 56 113 113 113 F F F 23 460-479 57 57 57 114 114 114 F F F 24 480-499 50 50 50 98 98 98 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 58 58 58 117 117 117 F F F 27 540-559 58 58 58 115 115 115 F F F 28 560-579 58 58 58 119 119 119 F F F 29 580-599 52 52 52 101 101 101 F F F Max gen: 58, Max flush: 119 A Difference in extent metadata is expected here On loop 29 Verifying data now No change after verify Loop: 29 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 114 114 114 F F F 1 020-039 57 57 57 114 114 114 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 58 58 58 117 117 117 F F F 4 080-099 58 58 58 117 117 117 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 58 58 58 115 115 115 F F F 7 140-159 52 52 52 101 101 101 F F F 8 160-179 58 58 58 118 118 118 F F F 9 180-199 58 58 58 118 118 118 F F F 10 200-219 58 58 58 118 118 118 F F F 11 220-239 58 58 58 118 118 118 F F F 12 240-259 58 58 58 115 115 115 F F F 13 260-279 58 58 58 118 118 118 F F F 14 280-299 58 58 58 118 118 118 F F F 15 300-319 49 49 49 96 96 96 F F F 16 320-339 59 59 59 120 120 120 F F F 17 340-359 59 59 59 120 120 120 F F F 18 360-379 59 59 59 120 120 120 F F F 19 380-399 52 52 52 101 101 101 F F F 20 400-419 42 42 42 84 84 84 F F F 21 420-439 58 58 58 119 119 119 F F F 22 440-459 56 56 56 113 113 113 F F F 23 460-479 57 57 57 114 114 114 F F F 24 480-499 50 50 50 98 98 98 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 58 58 58 117 117 117 F F F 27 540-559 58 58 58 115 115 115 F F F 28 560-579 58 58 58 119 119 119 F F F 29 580-599 52 52 52 101 101 101 F F F Max gen: 59, Max flush: 120 Begin loop 30 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.851386183Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.851920538Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.851954226Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422} {"msg":"Crucible f46d6148-2a57-49e9-8a11-27d5d5c1737d has session id: 99a975ed-b20c-4232-9ea7-ee4b2149469d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.852051582Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"Upstairs opts: Upstairs UUID: f46d6148-2a57-49e9-8a11-27d5d5c1737d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.85208903Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"Crucible stats registered with UUID: f46d6148-2a57-49e9-8a11-27d5d5c1737d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.852115519Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:60","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.8525299Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422} {"msg":"f46d6148-2a57-49e9-8a11-27d5d5c1737d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.852611866Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.852638355Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"0","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.852658683Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"0","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.852672873Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.852686602Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.852699781Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"2","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.852713051Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"2","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.85272588Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"io task","client":"0","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.85273979Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"io task","client":"2","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.852753169Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"io task","client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"ds_connection connected from Ok(127.0.0.1:53876)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.852879273Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"io task","client":"0","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:39:03.852929231Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"io task","client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:39:03.852961529Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"io task","client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:39:03.853133331Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.85315979Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.853175489Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"ds_connection connected from Ok(127.0.0.1:39813)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.853189769Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"io task","client":"2","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.853210078Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"io task","client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.853820689Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"2","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.853853898Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"2","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.853872767Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"0","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.853890056Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"0","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.854364414Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"0","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.854477269Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"2","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.854908578Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"0","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.854936397Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"0","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.854951756Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.855019883Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"2","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.855039902Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"2","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:03.855053482Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.853926725Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"io task","client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"ds_connection connected from Ok(127.0.0.1:43478)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.854122266Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"io task","client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.855282882Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.85532808Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.855887294Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856463087Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856488046Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"[0]R flush_numbers[0..12]: [114, 114, 89, 117, 117, 115, 115, 101, 118, 118, 118, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856507075Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"downstairs","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"[0]R generation[0..12]: [57, 57, 45, 58, 58, 58, 58, 52, 58, 58, 58, 58]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856524774Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"downstairs","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856548073Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"downstairs","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"[1]R flush_numbers[0..12]: [114, 114, 89, 117, 117, 115, 115, 101, 118, 118, 118, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856565902Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"downstairs","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"[1]R generation[0..12]: [57, 57, 45, 58, 58, 58, 58, 52, 58, 58, 58, 58]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856583151Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"downstairs","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856600541Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"downstairs","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"[2]R flush_numbers[0..12]: [114, 114, 89, 117, 117, 115, 115, 101, 118, 118, 118, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.85661755Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"downstairs","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"[2]R generation[0..12]: [57, 57, 45, 58, 58, 58, 58, 52, 58, 58, 58, 58]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856634689Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"downstairs","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible"Wait for a query_work_queue command to finish before sending IO ,"level":30,"time":"2024-07-31T18:39:13.856655388Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"downstairs","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"Max found gen is 60","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856671887Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"downstairs","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"Generation requested: 60 >= found:60","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856688297Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"downstairs","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"Next flush: 121","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856704306Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"downstairs","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856720395Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"":"downstairs","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856736524Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856752943Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"0","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856769603Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"1","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856785862Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"client":"2","":"downstairs","session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856805041Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.85682069Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"f46d6148-2a57-49e9-8a11-27d5d5c1737d is now active with session: 99a975ed-b20c-4232-9ea7-ee4b2149469d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.85683695Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856851689Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422,"session_id":"99a975ed-b20c-4232-9ea7-ee4b2149469d"} {"msg":"The guest has finished waiting for activation with:60","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:13.856866928Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 264 len 20480 data: 3 3 4 5 6 02/30 Flush 03/30 Write block 328 len 36864 data: 6 5 5 5 5 6 8 8 7 04/30 Write block 219 len 28672 data: 6 6 8 8 6 4 5 05/30 Write block 425 len 32768 data: 5 7 7 8 11 13 12 12 06/30 Read block 353 len 8192 07/30 Flush 08/30 Write block 415 len 24576 data: 4 5 6 6 8 7 09/30 Read block 140 len 28672 10/30 Read block 262 len 24576 11/30 Write block 240 len 24576 data: 4 4 5 5 4 5 12/30 Read block 532 len 32768 13/30 Write block 429 len 16384 data: 12 14 13 13 14/30 Read block 134 len 36864 15/30 Write block 176 len 4096 data: 4 16/30 Flush 17/30 Write block 309 len 24576 data: 6 5 6 6 7 9 18/30 Write block 570 len 36864 data: 8 10 11 12 11 9 9 8 7 19/30 Read block 358 len 4096 20/30 Read block 183 len 20480 21/30 Write block 252 len 32768 data: 6 5 8 8 9 8 6 6 22/30 Write block 365 len 36864 data: 6 6 7 8 7 6 6 5 5 23/30 Read block 252 len 36864 24/30 Read block 249 len 8192 25/30 Write block 329 len 20480 data: 6 6 6 6 7 26/30 Read block 324 len 8192 27/30 Write block 307 len 8192 data: 4 5 28/30 Read block 260 len 4096 29/30 Write block 367 len 20480 data: 8 9 8 7 7 30/30 Write block 214 len 28672 data: 5 6 6 5 5 7 7 ---------------------------------------------------------------- Crucible gen:60 GIO:true work queues: Upstairs:0 downstairs:14 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 17 Acked 1016 Write 6 Done Done Done false 18 Acked 1017 Write 9 Done Done Done false 19 Acked 1018 Read 1 Done Done Done false 20 Acked 1019 Read 5 Done Done Done false 21 Acked 1020 Write 8 Done Done Done false 22 Acked 1021 Write 9 Done Done Done false 23 Acked 1022 Read 9 Done Done Done false 24 Acked 1023 Read 2 Done Done Done false 25 Acked 1024 Write 5 Done Done Done false 26 Acked 1025 Read 2 Done Done Done false 27 Acked 1026 Write 2 Done Done Done false 28 Acked 1027 Read 1 Done Done Done false 29 Acked 1028 Write 5 Done Done Done false 30 Acked 1029 Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 13 13 13 39 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1015 1015 1015 Downstairs last five completed: 1015 1014 1013 1012 1011 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:39:13.942130259Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1422} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 114 114 114 F F F 1 020-039 57 57 57 114 114 114 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 58 58 58 117 117 117 F F F 4 080-099 58 58 58 117 117 117 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 58 58 58 115 115 115 F F F 7 140-159 52 52 52 101 101 101 F F F 8 160-179 60 60 60 123 123 123 F F F 9 180-199 58 58 58 118 118 118 F F F 10 200-219 60 60 60 122 122 122 T T T 11 220-239 60 60 60 122 122 122 T T T 12 240-259 60 60 60 123 123 123 T T T 13 260-279 60 60 60 121 121 121 F F F 14 280-299 58 58 58 118 118 118 F F F 15 300-319 49 49 49 96 96 96 T T T 16 320-339 60 60 60 122 122 122 T T T 17 340-359 59 59 59 120 120 120 F F F 18 360-379 59 59 59 120 120 120 T T T 19 380-399 52 52 52 101 101 101 F F F 20 400-419 60 60 60 123 123 123 F F F 21 420-439 60 60 60 123 123 123 F F F 22 440-459 56 56 56 113 113 113 F F F 23 460-479 57 57 57 114 114 114 F F F 24 480-499 50 50 50 98 98 98 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 58 58 58 117 117 117 F F F 27 540-559 58 58 58 115 115 115 F F F 28 560-579 58 58 58 119 119 119 T T T 29 580-599 52 52 52 101 101 101 F F F Max gen: 60, Max flush: 123 A Difference in extent metadata is expected here On loop 30 Verifying data now No change after verify Loop: 30 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 114 114 114 F F F 1 020-039 57 57 57 114 114 114 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 58 58 58 117 117 117 F F F 4 080-099 58 58 58 117 117 117 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 58 58 58 115 115 115 F F F 7 140-159 52 52 52 101 101 101 F F F 8 160-179 60 60 60 123 123 123 F F F 9 180-199 58 58 58 118 118 118 F F F 10 200-219 61 61 61 124 124 124 F F F 11 220-239 61 61 61 124 124 124 F F F 12 240-259 61 61 61 124 124 124 F F F 13 260-279 60 60 60 121 121 121 F F F 14 280-299 58 58 58 118 118 118 F F F 15 300-319 61 61 61 124 124 124 F F F 16 320-339 61 61 61 124 124 124 F F F 17 340-359 59 59 59 120 120 120 F F F 18 360-379 61 61 61 124 124 124 F F F 19 380-399 52 52 52 101 101 101 F F F 20 400-419 60 60 60 123 123 123 F F F 21 420-439 60 60 60 123 123 123 F F F 22 440-459 56 56 56 113 113 113 F F F 23 460-479 57 57 57 114 114 114 F F F 24 480-499 50 50 50 98 98 98 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 58 58 58 117 117 117 F F F 27 540-559 58 58 58 115 115 115 F F F 28 560-579 61 61 61 124 124 124 F F F 29 580-599 52 52 52 101 101 101 F F F Max gen: 61, Max flush: 124 Begin loop 31 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.770306098Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.77069462Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.770761607Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430} {"msg":"Crucible 37496457-d2fa-4f96-ae30-98df773738e4 has session id: 860eb6f0-e221-4d3a-a903-e42784c4ce97","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.770881951Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"Upstairs opts: Upstairs UUID: 37496457-d2fa-4f96-ae30-98df773738e4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.770921319Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"Crucible stats registered with UUID: 37496457-d2fa-4f96-ae30-98df773738e4","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.770944048Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:62","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.771359239Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430} {"msg":"37496457-d2fa-4f96-ae30-98df773738e4 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.771383167Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.771397697Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"0","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.771416116Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"0","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.771429835Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.771443405Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.771456384Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"2","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.771471843Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"2","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.771499372Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"io task","client":"0","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.771526611Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"io task","client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.77155162Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"io task","client":"2","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:39:28.771900603Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"io task","client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:39:28.771954691Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"io task","client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:39:28.77197512Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"ds_connection connected from Ok(127.0.0.1:36872)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.771988209Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"io task","client":"2","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.772002339Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.772015398Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"ds_connection connected from Ok(127.0.0.1:48333)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.772028648Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"io task","client":"0","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.772149382Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"io task","client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.772643549Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"2","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.772670518Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"2","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.772685747Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"0","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.772700166Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"0","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.773303178Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"0","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.773338416Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"2","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.773866222Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"2","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.773893981Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"2","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.77390928Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.773921859Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"0","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.773935139Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"0","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:28.773948178Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.772626965Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"io task","client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"ds_connection connected from Ok(127.0.0.1:43481)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.772712751Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"io task","client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.774033879Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.774083177Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.774584303Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775098759Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775118999Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"[0]R flush_numbers[0..12]: [114, 114, 89, 117, 117, 115, 115, 101, 123, 118, 124, 124]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775138687Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"downstairs","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"[0]R generation[0..12]: [57, 57, 45, 58, 58, 58, 58, 52, 60, 58, 61, 61]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775153157Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"downstairs","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775168206Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"downstairs","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"[1]R flush_numbers[0..12]: [114, 114, 89, 117, 117, 115, 115, 101, 123, 118, 124, 124]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775196165Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"downstairs","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"[1]R generation[0..12]: [57, 57, 45, 58, 58, 58, 58, 52, 60, 58, 61, 61]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775210404Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"downstairs","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775224543Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"downstairs","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"[2]R flush_numbers[0..12]: [114, 114, 89, 117, 117, 115, 115, 101, 123, 118, 124, 124]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775238433Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"downstairs","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"[2]R generation[0..12]: [57, 57, 45, 58, 58, 58, 58, 52, 60, 58, 61, 61]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775251862Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"downstairs","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775265972Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"downstairs","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"Max found gen is 62","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775278711Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"downstairs","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"Generation requested: 62 >= found:62","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.77529173Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"downstairs","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"Next flush: 125","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.77530468Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775321139Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"":"downstairs","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775334338Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775348888Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"0","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775362067Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"1","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775374996Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"client":"2","":"downstairs","session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775388196Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775400315Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"37496457-d2fa-4f96-ae30-98df773738e4 is now active with session: 860eb6f0-e221-4d3a-a903-e42784c4ce97","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775413055Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775424864Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430,"session_id":"860eb6f0-e221-4d3a-a903-e42784c4ce97"} {"msg":"The guest has finished waiting for activation with:62","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:38.775436973Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 407 len 12288 data: 6 5 4 02/30 Read block 156 len 24576 03/30 Read block 462 len 4096 04/30 Read block 483 len 8192 05/30 Write block 476 len 20480 data: 8 8 4 3 3 06/30 Read block 497 len 8192 07/30 Read block 270 len 32768 08/30 Read block 440 len 32768 09/30 Read block 8 len 24576 10/30 Read block 32 len 20480 11/30 Write block 84 len 12288 data: 9 9 6 12/30 Write block 232 len 4096 data: 9 13/30 Read block 480 len 32768 14/30 Flush 15/30 Write block 302 len 32768 data: 7 7 6 4 4 5 6 7 16/30 Flush 17/30 Write block 256 len 28672 data: 10 9 7 7 6 5 6 18/30 Read block 278 len 8192 19/30 Write block 419 len 16384 data: 9 8 5 5 20/30 Read block 298 len 36864 21/30 Read block 399 len 36864 22/30 Write block 313 len 24576 data: 8 10 9 7 6 8 23/30 Read block 305 len 16384 24/30 Write block 234 len 28672 data: 7 8 8 6 7 6 5 25/30 Read block 401 len 40960 26/30 Write block 205 len 12288 data: 5 5 6 27/30 Write block 367 len 24576 data: 9 10 9 8 8 6 28/30 Write block 478 len 16384 data: 5 4 4 3 29/30 Write block 425 len 28672 data: 6 8 8 9 13 15 14 30/30 Read block 193 len 8192 ---------------------------------------------------------------- Crucible gen:62 GIO:true work queues: Upstairs:0 downstairs:14 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 17 Acked 1016 Write 7 Done Done Done false 18 Acked 1017 Read 2 Done Done Done false 19 Acked 1018 Write 4 Done Done Done false 20 Acked 1019 Read 9 Done Done Done false 21 Acked 1020 Read 9 Done Done Done false 22 Acked 1021 Write 6 Done Done Done false 23 Acked 1022 Read 4 Done Done Done false 24 Acked 1023 Write 7 Done Done Done false 25 Acked 1024 Read 10 Done Done Done false 26 Acked 1025 Write 3 Done Done Done false 27 Acked 1026 Write 6 Done Done Done false 28 Acked 1027 Write 4 Done Done Done false 29 Acked 1028 Write 7 Done Done Done false 30 Acked 1029 Read 2 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 0 1 Done 13 14 14 41 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1015 1015 1015 Downstairs last five completed: 1015 1014 1013 1012 1011 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:39:38.830295524Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1430} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 114 114 114 F F F 1 020-039 57 57 57 114 114 114 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 58 58 58 117 117 117 F F F 4 080-099 62 62 62 125 125 125 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 58 58 58 115 115 115 F F F 7 140-159 52 52 52 101 101 101 F F F 8 160-179 60 60 60 123 123 123 F F F 9 180-199 58 58 58 118 118 118 F F F 10 200-219 61 61 61 124 124 124 T T T 11 220-239 62 62 62 125 125 125 T T T 12 240-259 61 61 61 124 124 124 T T T 13 260-279 60 60 60 121 121 121 T T T 14 280-299 58 58 58 118 118 118 F F F 15 300-319 62 62 62 126 126 126 T T T 16 320-339 61 61 61 124 124 124 F F F 17 340-359 59 59 59 120 120 120 F F F 18 360-379 61 61 61 124 124 124 T T T 19 380-399 52 52 52 101 101 101 F F F 20 400-419 62 62 62 125 125 125 T T T 21 420-439 60 60 60 123 123 123 T T T 22 440-459 56 56 56 113 113 113 F F F 23 460-479 62 62 62 125 125 125 T T T 24 480-499 62 62 62 125 125 125 T T T 25 500-519 47 47 47 93 93 93 F F F 26 520-539 58 58 58 117 117 117 F F F 27 540-559 58 58 58 115 115 115 F F F 28 560-579 61 61 61 124 124 124 F F F 29 580-599 52 52 52 101 101 101 F F F Max gen: 62, Max flush: 126 A Difference in extent metadata is expected here On loop 31 Verifying data now No change after verify Loop: 31 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 114 114 114 F F F 1 020-039 57 57 57 114 114 114 F F F 2 040-059 45 45 45 89 89 89 F F F 3 060-079 58 58 58 117 117 117 F F F 4 080-099 62 62 62 125 125 125 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 58 58 58 115 115 115 F F F 7 140-159 52 52 52 101 101 101 F F F 8 160-179 60 60 60 123 123 123 F F F 9 180-199 58 58 58 118 118 118 F F F 10 200-219 63 63 63 127 127 127 F F F 11 220-239 63 63 63 127 127 127 F F F 12 240-259 63 63 63 127 127 127 F F F 13 260-279 63 63 63 127 127 127 F F F 14 280-299 58 58 58 118 118 118 F F F 15 300-319 63 63 63 127 127 127 F F F 16 320-339 61 61 61 124 124 124 F F F 17 340-359 59 59 59 120 120 120 F F F 18 360-379 63 63 63 127 127 127 F F F 19 380-399 52 52 52 101 101 101 F F F 20 400-419 63 63 63 127 127 127 F F F 21 420-439 63 63 63 127 127 127 F F F 22 440-459 56 56 56 113 113 113 F F F 23 460-479 63 63 63 127 127 127 F F F 24 480-499 63 63 63 127 127 127 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 58 58 58 117 117 117 F F F 27 540-559 58 58 58 115 115 115 F F F 28 560-579 61 61 61 124 124 124 F F F 29 580-599 52 52 52 101 101 101 F F F Max gen: 63, Max flush: 127 Begin loop 32 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.777780832Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.778191412Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.778222281Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438} {"msg":"Crucible f8a2d3ff-558e-412a-bd14-83145e408fd5 has session id: 895e033f-a456-4766-83a9-e685bf9cee25","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.778479179Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"Upstairs opts: Upstairs UUID: f8a2d3ff-558e-412a-bd14-83145e408fd5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.778522537Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"Crucible stats registered with UUID: f8a2d3ff-558e-412a-bd14-83145e408fd5","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.778547085Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:64","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.778821073Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438} {"msg":"f8a2d3ff-558e-412a-bd14-83145e408fd5 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.778845151Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.778859131Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"0","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.77887801Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"0","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.778899099Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.778922798Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.778947247Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"2","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.778970206Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"2","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.778995115Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"io task","client":"0","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.779021283Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"io task","client":"2","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.779044552Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"io task","client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"ds_connection connected from Ok(127.0.0.1:48327)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.779221234Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"io task","client":"2","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"ds_connection connected from Ok(127.0.0.1:37657)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.779272132Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"io task","client":"0","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:39:53.779511291Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"io task","client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:39:53.779550029Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"io task","client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:39:53.779577007Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.779597467Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.779619166Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.779643914Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"io task","client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.780060955Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"2","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.780084244Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"2","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.780101793Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"0","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.780115372Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"0","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.780721454Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"2","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.780748043Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"0","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.781396292Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"2","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.78144356Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"2","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.781466719Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.781482568Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"0","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.781499518Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"0","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:39:53.781515667Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.780455768Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"io task","client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"ds_connection connected from Ok(127.0.0.1:62441)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.780592362Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"io task","client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.781672362Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.78169989Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782187458Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782717823Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782738782Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"[0]R flush_numbers[0..12]: [114, 114, 89, 117, 125, 115, 115, 101, 123, 118, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782754071Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"downstairs","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"[0]R generation[0..12]: [57, 57, 45, 58, 62, 58, 58, 52, 60, 58, 63, 63]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.78276852Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"downstairs","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.78278289Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"downstairs","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"[1]R flush_numbers[0..12]: [114, 114, 89, 117, 125, 115, 115, 101, 123, 118, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782796849Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"downstairs","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"[1]R generation[0..12]: [57, 57, 45, 58, 62, 58, 58, 52, 60, 58, 63, 63]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782810878Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"downstairs","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782825148Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"downstairs","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"[2]R flush_numbers[0..12]: [114, 114, 89, 117, 125, 115, 115, 101, 123, 118, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782839157Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"downstairs","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"[2]R generation[0..12]: [57, 57, 45, 58, 62, 58, 58, 52, 60, 58, 63, 63]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782852686Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"downstairs","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782868016Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"downstairs","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"Max found gen is 64","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782881175Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"downstairs","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"Generation requested: 64 >= found:64","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782894214Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"downstairs","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"Next flush: 128","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:40:03.782907294Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"downstairs","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782926523Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"":"downstairs","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782940072Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782952082Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"0","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.782965451Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"1","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.78297853Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"client":"2","":"downstairs","session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.78299172Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.783010429Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"f8a2d3ff-558e-412a-bd14-83145e408fd5 is now active with session: 895e033f-a456-4766-83a9-e685bf9cee25","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.783033348Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.783046337Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438,"session_id":"895e033f-a456-4766-83a9-e685bf9cee25"} {"msg":"The guest has finished waiting for activation with:64","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:03.783058987Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 458 len 4096 data: 4 02/30 Read block 332 len 8192 03/30 Read block 152 len 36864 04/30 Write block 418 len 36864 data: 7 10 9 6 6 5 4 7 9 05/30 Write block 35 len 20480 data: 2 4 4 5 5 06/30 Write block 484 len 16384 data: 4 2 2 3 07/30 Write block 420 len 24576 data: 10 7 7 6 5 8 08/30 Read block 441 len 32768 09/30 Read block 39 len 40960 10/30 Read block 528 len 12288 11/30 Write block 127 len 8192 data: 5 5 12/30 Write block 527 len 20480 data: 4 4 3 4 6 13/30 Read block 263 len 16384 14/30 Write block 187 len 40960 data: 8 8 6 7 5 4 5 5 6 7 15/30 Flush 16/30 Write block 39 len 16384 data: 6 5 5 4 17/30 Read block 296 len 28672 18/30 Write block 593 len 8192 data: 6 6 19/30 Read block 209 len 36864 20/30 Flush 21/30 Write block 364 len 24576 data: 6 7 7 10 11 10 22/30 Read block 556 len 40960 23/30 Read block 133 len 36864 24/30 Read block 460 len 8192 25/30 Read block 373 len 4096 26/30 Write block 186 len 12288 data: 8 9 9 27/30 Read block 575 len 16384 28/30 Read block 261 len 36864 29/30 Write block 186 len 32768 data: 9 10 10 7 8 6 5 6 30/30 Read block 368 len 40960 ---------------------------------------------------------------- Crucible gen:64 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 21 Acked 1020 Write 6 Done Done Done false 22 Acked 1021 Read 10 Done Done Done false 23 Acked 1022 Read 9 Done Done Done false 24 Acked 1023 Read 2 Done Done Done false 25 Acked 1024 Read 1 Done Done Done false 26 Acked 1025 Write 3 Done Done Done false 27 Acked 1026 Read 4 Done Done Done false 28 Acked 1027 Read 9 Done Done Done false 29 Acked 1028 Write 8 Done Done Done false 30 Acked 1029 Read 10 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 0 1 Done 9 10 10 29 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1019 1019 1019 Downstairs last five completed: 1019 1018 1017 1016 1015 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:40:03.859887822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1438} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 114 114 114 F F F 1 020-039 64 64 64 129 129 129 F F F 2 040-059 64 64 64 129 129 129 F F F 3 060-079 58 58 58 117 117 117 F F F 4 080-099 62 62 62 125 125 125 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 64 64 64 128 128 128 F F F 7 140-159 52 52 52 101 101 101 F F F 8 160-179 60 60 60 123 123 123 F F F 9 180-199 64 64 64 128 128 128 T T T 10 200-219 63 63 63 127 127 127 F F F 11 220-239 63 63 63 127 127 127 F F F 12 240-259 63 63 63 127 127 127 F F F 13 260-279 63 63 63 127 127 127 F F F 14 280-299 58 58 58 118 118 118 F F F 15 300-319 63 63 63 127 127 127 F F F 16 320-339 61 61 61 124 124 124 F F F 17 340-359 59 59 59 120 120 120 F F F 18 360-379 63 63 63 127 127 127 T T T 19 380-399 52 52 52 101 101 101 F F F 20 400-419 64 64 64 128 128 128 F F F 21 420-439 64 64 64 128 128 128 F F F 22 440-459 64 64 64 128 128 128 F F F 23 460-479 63 63 63 127 127 127 F F F 24 480-499 64 64 64 128 128 128 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 64 64 64 128 128 128 F F F 27 540-559 58 58 58 115 115 115 F F F 28 560-579 61 61 61 124 124 124 F F F 29 580-599 64 64 64 129 129 129 F F F Max gen: 64, Max flush: 129 A Difference in extent metadata is expected here On loop 32 Verifying data now No change after verify Loop: 32 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 114 114 114 F F F 1 020-039 64 64 64 129 129 129 F F F 2 040-059 64 64 64 129 129 129 F F F 3 060-079 58 58 58 117 117 117 F F F 4 080-099 62 62 62 125 125 125 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 64 64 64 128 128 128 F F F 7 140-159 52 52 52 101 101 101 F F F 8 160-179 60 60 60 123 123 123 F F F 9 180-199 65 65 65 130 130 130 F F F 10 200-219 63 63 63 127 127 127 F F F 11 220-239 63 63 63 127 127 127 F F F 12 240-259 63 63 63 127 127 127 F F F 13 260-279 63 63 63 127 127 127 F F F 14 280-299 58 58 58 118 118 118 F F F 15 300-319 63 63 63 127 127 127 F F F 16 320-339 61 61 61 124 124 124 F F F 17 340-359 59 59 59 120 120 120 F F F 18 360-379 65 65 65 130 130 130 F F F 19 380-399 52 52 52 101 101 101 F F F 20 400-419 64 64 64 128 128 128 F F F 21 420-439 64 64 64 128 128 128 F F F 22 440-459 64 64 64 128 128 128 F F F 23 460-479 63 63 63 127 127 127 F F F 24 480-499 64 64 64 128 128 128 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 64 64 64 128 128 128 F F F 27 540-559 58 58 58 115 115 115 F F F 28 560-579 61 61 61 124 124 124 F F F 29 580-599 64 64 64 129 129 129 F F F Max gen: 65, Max flush: 130 Begin loop 33 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.117943611Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.118331353Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.118352432Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446} {"msg":"Crucible 4089f8be-e2cc-41e8-a332-3e0e0d521def has session id: 8e496ab2-c1ca-49fa-9b39-9fb0c30245aa","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.118748514Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"Upstairs opts: Upstairs UUID: 4089f8be-e2cc-41e8-a332-3e0e0d521def, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.118801711Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"Crucible stats registered with UUID: 4089f8be-e2cc-41e8-a332-3e0e0d521def","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.11883076Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:66","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.119133676Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446} {"msg":"4089f8be-e2cc-41e8-a332-3e0e0d521def active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.119234171Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.11925818Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"0","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.119290958Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"0","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.119317217Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.119344886Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.119365365Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"2","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.119387964Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"2","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.119411923Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"io task","client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.119436491Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"io task","client":"2","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.11946417Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"io task","client":"0","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:40:18.119747547Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"io task","client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:40:18.119787505Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"io task","client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"ds_connection connected from Ok(127.0.0.1:39119)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.119815384Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"io task","client":"2","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:40:18.119846122Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.119871511Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.11989732Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.119921749Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"io task","client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"ds_connection connected from Ok(127.0.0.1:47713)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.119955927Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"io task","client":"0","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.120304921Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"0","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.120342729Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"0","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.120370668Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"2","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.120397337Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"2","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.120867105Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"2","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.120983379Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"0","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.121412109Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"2","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.121439538Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"2","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.121454727Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.121473486Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"0","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.121487776Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"0","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:18.121500995Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.121623583Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"io task","client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"ds_connection connected from Ok(127.0.0.1:61219)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.122213795Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"io task","client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.123132892Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.123161221Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.123654618Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124214882Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124238711Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"[0]R flush_numbers[0..12]: [114, 129, 129, 117, 125, 115, 128, 101, 123, 130, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.12425794Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"downstairs","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"[0]R generation[0..12]: [57, 64, 64, 58, 62, 58, 64, 52, 60, 65, 63, 63]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124272569Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"downstairs","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124286809Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"downstairs","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"[1]R flush_numbers[0..12]: [114, 129, 129, 117, 125, 115, 128, 101, 123, 130, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124300448Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"downstairs","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"[1]R generation[0..12]: [57, 64, 64, 58, 62, 58, 64, 52, 60, 65, 63, 63]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124313827Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"downstairs","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124327857Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"downstairs","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"[2]R flush_numbers[0..12]: [114, 129, 129, 117, 125, 115, 128, 101, 123, 130, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124341556Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"downstairs","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"[2]R generation[0..12]: [57, 64, 64, 58, 62, 58, 64, 52, 60, 65, 63, 63]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124355286Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"downstairs","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124369315Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"downstairs","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"Max found gen is 66","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124382064Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"downstairs","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"Generation requested: 66 >= found:66","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124395164Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"downstairs","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"Next flush: 131","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124411123Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"downstairs","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124424272Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"":"downstairs","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} Wait for a query_work_queue command to finish before sending IO {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124440422Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124452891Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"0","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.12446624Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"1","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.12447936Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"client":"2","":"downstairs","session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124492469Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124505279Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"4089f8be-e2cc-41e8-a332-3e0e0d521def is now active with session: 8e496ab2-c1ca-49fa-9b39-9fb0c30245aa","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124518098Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124529867Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446,"session_id":"8e496ab2-c1ca-49fa-9b39-9fb0c30245aa"} {"msg":"The guest has finished waiting for activation with:66","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:28.124541927Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 56 len 4096 data: 6 02/30 Read block 503 len 12288 03/30 Write block 352 len 28672 data: 8 9 10 9 8 10 10 04/30 Read block 523 len 4096 05/30 Write block 264 len 24576 data: 4 4 5 6 7 7 06/30 Write block 456 len 24576 data: 5 5 5 2 4 4 07/30 Write block 487 len 24576 data: 4 3 3 3 3 3 08/30 Read block 591 len 32768 09/30 Read block 588 len 32768 10/30 Flush 11/30 Write block 383 len 20480 data: 5 6 3 3 4 12/30 Write block 187 len 16384 data: 11 11 8 9 13/30 Write block 584 len 12288 data: 7 7 9 14/30 Read block 402 len 32768 15/30 Read block 413 len 20480 16/30 Read block 484 len 12288 17/30 Read block 196 len 12288 18/30 Read block 414 len 36864 19/30 Read block 376 len 24576 20/30 Read block 46 len 4096 21/30 Read block 393 len 20480 22/30 Read block 286 len 12288 23/30 Flush 24/30 Write block 153 len 8192 data: 6 6 25/30 Flush 26/30 Write block 7 len 24576 data: 5 5 6 6 6 4 27/30 Read block 274 len 28672 28/30 Write block 131 len 28672 data: 5 5 5 5 4 5 6 29/30 Write block 279 len 8192 data: 8 9 30/30 Read block 357 len 24576 ---------------------------------------------------------------- Crucible gen:66 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 26 Acked 1025 Write 6 Done Done Done false 27 Acked 1026 Read 7 Done Done Done false 28 Acked 1027 Write 7 Done Done Done false 29 Acked 1028 Write 2 Done Done Done false 30 Acked 1029 Read 6 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 0 1 Done 5 4 5 14 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1024 1024 1024 Downstairs last five completed: 1024 1023 1022 1021 1020 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:40:28.207858399Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1446} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 114 114 114 T T T 1 020-039 64 64 64 129 129 129 F F F 2 040-059 66 66 66 131 131 131 F F F 3 060-079 58 58 58 117 117 117 F F F 4 080-099 62 62 62 125 125 125 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 64 64 64 128 128 128 T T T 7 140-159 66 66 66 133 133 133 F F F 8 160-179 60 60 60 123 123 123 F F F 9 180-199 66 66 66 132 132 132 F F F 10 200-219 63 63 63 127 127 127 F F F 11 220-239 63 63 63 127 127 127 F F F 12 240-259 63 63 63 127 127 127 F F F 13 260-279 66 66 66 131 131 131 T T T 14 280-299 58 58 58 118 118 118 T T T 15 300-319 63 63 63 127 127 127 F F F 16 320-339 61 61 61 124 124 124 F F F 17 340-359 66 66 66 131 131 131 F F F 18 360-379 65 65 65 130 130 130 F F F 19 380-399 66 66 66 132 132 132 F F F 20 400-419 64 64 64 128 128 128 F F F 21 420-439 64 64 64 128 128 128 F F F 22 440-459 66 66 66 131 131 131 F F F 23 460-479 66 66 66 131 131 131 F F F 24 480-499 66 66 66 131 131 131 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 64 64 64 128 128 128 F F F 27 540-559 58 58 58 115 115 115 F F F 28 560-579 61 61 61 124 124 124 F F F 29 580-599 66 66 66 132 132 132 F F F Max gen: 66, Max flush: 133 A Difference in extent metadata is expected here On loop 33 Verifying data now No change after verify Loop: 33 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 67 67 67 134 134 134 F F F 1 020-039 64 64 64 129 129 129 F F F 2 040-059 66 66 66 131 131 131 F F F 3 060-079 58 58 58 117 117 117 F F F 4 080-099 62 62 62 125 125 125 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 67 67 67 134 134 134 F F F 7 140-159 66 66 66 133 133 133 F F F 8 160-179 60 60 60 123 123 123 F F F 9 180-199 66 66 66 132 132 132 F F F 10 200-219 63 63 63 127 127 127 F F F 11 220-239 63 63 63 127 127 127 F F F 12 240-259 63 63 63 127 127 127 F F F 13 260-279 67 67 67 134 134 134 F F F 14 280-299 67 67 67 134 134 134 F F F 15 300-319 63 63 63 127 127 127 F F F 16 320-339 61 61 61 124 124 124 F F F 17 340-359 66 66 66 131 131 131 F F F 18 360-379 65 65 65 130 130 130 F F F 19 380-399 66 66 66 132 132 132 F F F 20 400-419 64 64 64 128 128 128 F F F 21 420-439 64 64 64 128 128 128 F F F 22 440-459 66 66 66 131 131 131 F F F 23 460-479 66 66 66 131 131 131 F F F 24 480-499 66 66 66 131 131 131 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 64 64 64 128 128 128 F F F 27 540-559 58 58 58 115 115 115 F F F 28 560-579 61 61 61 124 124 124 F F F 29 580-599 66 66 66 132 132 132 F F F Max gen: 67, Max flush: 134 Begin loop 34 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.582055849Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.582705509Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.582732247Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454} {"msg":"Crucible f07d3899-1be1-4446-9df1-b2d91a24062d has session id: 63ad5144-4042-41b2-af65-3e9efa519ee6","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.582753846Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"Upstairs opts: Upstairs UUID: f07d3899-1be1-4446-9df1-b2d91a24062d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.582788185Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"Crucible stats registered with UUID: f07d3899-1be1-4446-9df1-b2d91a24062d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.582814534Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:68","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.583038003Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454} {"msg":"f07d3899-1be1-4446-9df1-b2d91a24062d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.583073822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.58309891Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"0","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.583132279Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"0","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.583158678Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"1","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.583183276Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"1","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.583213085Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.583279642Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.583406926Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"io task","client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.583495082Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"io task","client":"1","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.58352908Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"io task","client":"0","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:40:42.583556329Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"io task","client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:40:42.583609197Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"io task","client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:40:42.583672683Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.583695073Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.583709562Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.583723001Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"io task","client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"ds_connection connected from Ok(127.0.0.1:51487)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.58374236Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"io task","client":"0","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"ds_connection connected from Ok(127.0.0.1:61636)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.58375683Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"io task","client":"1","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.584185999Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"0","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.584222198Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"0","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.584249516Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"1","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.584276485Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"1","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.584870428Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"1","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.584898566Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"0","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.585440541Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"0","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.58546728Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"0","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.585481749Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.585539496Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"1","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.585559026Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"1","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:42.585572575Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.584594303Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"io task","client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"ds_connection connected from Ok(127.0.0.1:34264)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.584733296Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"io task","client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.586181998Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.586216687Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.586685615Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.58722545Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587249369Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"[0]R flush_numbers[0..12]: [134, 129, 131, 117, 125, 115, 134, 133, 123, 132, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587264258Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"downstairs","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"[0]R generation[0..12]: [67, 64, 66, 58, 62, 58, 67, 66, 60, 66, 63, 63]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587278627Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"downstairs","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587292777Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"downstairs","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"[1]R flush_numbers[0..12]: [134, 129, 131, 117, 125, 115, 134, 133, 123, 132, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587306736Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"downstairs","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"[1]R generation[0..12]: [67, 64, 66, 58, 62, 58, 67, 66, 60, 66, 63, 63]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587320385Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"downstairs","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587334485Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"downstairs","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"[2]R flush_numbers[0..12]: [134, 129, 131, 117, 125, 115, 134, 133, 123, 132, 127, 127]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587348404Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"downstairs","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"[2]R generation[0..12]: [67, 64, 66, 58, 62, 58, 67, 66, 60, 66, 63, 63]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587362163Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"downstairs","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587376273Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"downstairs","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"Max found gen is 68","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587389362Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"downstairs","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"Generation requested: 68 >= found:68","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:40:52.587402401Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"downstairs","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"Next flush: 135","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587419111Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"downstairs","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.58743547Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"":"downstairs","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587449079Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587461059Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"0","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587474548Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"1","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587487948Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"client":"2","":"downstairs","session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587501437Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587513736Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"f07d3899-1be1-4446-9df1-b2d91a24062d is now active with session: 63ad5144-4042-41b2-af65-3e9efa519ee6","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587526726Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587538555Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454,"session_id":"63ad5144-4042-41b2-af65-3e9efa519ee6"} {"msg":"The guest has finished waiting for activation with:68","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:40:52.587550675Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 547 len 40960 data: 5 6 5 7 7 6 8 7 7 6 02/30 Read block 32 len 36864 03/30 Write block 186 len 40960 data: 10 12 12 9 10 7 6 7 6 7 04/30 Read block 123 len 8192 05/30 Read block 312 len 16384 06/30 Write block 83 len 40960 data: 8 10 10 7 5 4 5 5 5 3 07/30 Flush 08/30 Write block 482 len 28672 data: 3 3 5 3 3 5 4 09/30 Read block 16 len 40960 10/30 Read block 173 len 12288 11/30 Read block 92 len 4096 12/30 Write block 374 len 32768 data: 4 5 5 5 7 6 6 6 13/30 Read block 55 len 32768 14/30 Read block 137 len 8192 15/30 Read block 84 len 28672 16/30 Write block 273 len 24576 data: 4 5 6 6 6 6 17/30 Write block 445 len 4096 data: 8 18/30 Write block 459 len 24576 data: 3 5 5 4 3 4 19/30 Read block 82 len 20480 20/30 Flush 21/30 Write block 5 len 8192 data: 3 4 22/30 Read block 438 len 24576 23/30 Read block 280 len 20480 24/30 Read block 103 len 36864 25/30 Write block 224 len 4096 data: 5 26/30 Write block 337 len 8192 data: 7 5 27/30 Read block 479 len 8192 28/30 Read block 207 len 12288 29/30 Read block 195 len 36864 30/30 Read block 140 len 16384 ---------------------------------------------------------------- Crucible gen:68 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 21 Acked 1020 Write 2 Done Done Done false 22 Acked 1021 Read 6 Done Done Done false 23 Acked 1022 Read 5 Done Done Done false 24 Acked 1023 Read 9 Done Done Done false 25 Acked 1024 Write 1 Done Done Done false 26 Acked 1025 Write 2 Done Done Done false 27 Acked 1026 Read 2 Done Done Done false 28 Acked 1027 Read 3 Done Done Done false 29 Acked 1028 Read 9 Done Done Done false 30 Acked 1029 Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 1 2 Done 10 9 9 28 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1019 1019 1019 Downstairs last five completed: 1019 1018 1017 1016 1015 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:40:52.671808513Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1454} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 67 67 67 134 134 134 T T T 1 020-039 64 64 64 129 129 129 F F F 2 040-059 66 66 66 131 131 131 F F F 3 060-079 58 58 58 117 117 117 F F F 4 080-099 68 68 68 135 135 135 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 67 67 67 134 134 134 F F F 7 140-159 66 66 66 133 133 133 F F F 8 160-179 60 60 60 123 123 123 F F F 9 180-199 68 68 68 135 135 135 F F F 10 200-219 63 63 63 127 127 127 F F F 11 220-239 63 63 63 127 127 127 T T T 12 240-259 63 63 63 127 127 127 F F F 13 260-279 68 68 68 136 136 136 F F F 14 280-299 67 67 67 134 134 134 F F F 15 300-319 63 63 63 127 127 127 F F F 16 320-339 61 61 61 124 124 124 T T T 17 340-359 66 66 66 131 131 131 F F F 18 360-379 68 68 68 136 136 136 F F F 19 380-399 68 68 68 136 136 136 F F F 20 400-419 64 64 64 128 128 128 F F F 21 420-439 64 64 64 128 128 128 F F F 22 440-459 68 68 68 136 136 136 F F F 23 460-479 68 68 68 136 136 136 F F F 24 480-499 68 68 68 136 136 136 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 64 64 64 128 128 128 F F F 27 540-559 68 68 68 135 135 135 F F F 28 560-579 61 61 61 124 124 124 F F F 29 580-599 66 66 66 132 132 132 F F F Max gen: 68, Max flush: 136 A Difference in extent metadata is expected here On loop 34 Verifying data now No change after verify Loop: 34 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 64 64 64 129 129 129 F F F 2 040-059 66 66 66 131 131 131 F F F 3 060-079 58 58 58 117 117 117 F F F 4 080-099 68 68 68 135 135 135 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 67 67 67 134 134 134 F F F 7 140-159 66 66 66 133 133 133 F F F 8 160-179 60 60 60 123 123 123 F F F 9 180-199 68 68 68 135 135 135 F F F 10 200-219 63 63 63 127 127 127 F F F 11 220-239 69 69 69 137 137 137 F F F 12 240-259 63 63 63 127 127 127 F F F 13 260-279 68 68 68 136 136 136 F F F 14 280-299 67 67 67 134 134 134 F F F 15 300-319 63 63 63 127 127 127 F F F 16 320-339 69 69 69 137 137 137 F F F 17 340-359 66 66 66 131 131 131 F F F 18 360-379 68 68 68 136 136 136 F F F 19 380-399 68 68 68 136 136 136 F F F 20 400-419 64 64 64 128 128 128 F F F 21 420-439 64 64 64 128 128 128 F F F 22 440-459 68 68 68 136 136 136 F F F 23 460-479 68 68 68 136 136 136 F F F 24 480-499 68 68 68 136 136 136 F F F 25 500-519 47 47 47 93 93 93 F F F 26 520-539 64 64 64 128 128 128 F F F 27 540-559 68 68 68 135 135 135 F F F 28 560-579 61 61 61 124 124 124 F F F 29 580-599 66 66 66 132 132 132 F F F Max gen: 69, Max flush: 137 Begin loop 35 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.871346713Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.871745285Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.871774693Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462} {"msg":"Crucible b20a3b9e-8f03-4b5b-82de-d6a528458fd8 has session id: 8a117f4c-7b09-4127-85c0-1f1240a4b1fc","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.872002962Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"Upstairs opts: Upstairs UUID: b20a3b9e-8f03-4b5b-82de-d6a528458fd8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.872038511Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"Crucible stats registered with UUID: b20a3b9e-8f03-4b5b-82de-d6a528458fd8","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.87205545Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:70","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.872341037Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462} {"msg":"b20a3b9e-8f03-4b5b-82de-d6a528458fd8 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.872374545Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.872398094Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.872428673Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.872453612Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"1","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.87248153Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"1","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.872505429Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"2","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.872530668Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"2","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.872592485Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"io task","client":"2","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.872621214Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"io task","client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.872647652Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"io task","client":"1","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"ds_connection connected from Ok(127.0.0.1:35650)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.872676151Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"io task","client":"1","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"ds_connection connected from Ok(127.0.0.1:38010)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.872852163Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"io task","client":"2","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:41:06.87292665Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"io task","client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:41:06.872955458Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"io task","client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:41:06.872978607Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.872992276Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.873005676Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.873097702Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"io task","client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.873644536Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"1","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.873682464Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"1","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.873710973Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"2","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.873759961Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"2","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.874279776Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"1","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.874307715Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"2","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.874873299Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"1","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.874894038Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"1","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.874907627Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.874919866Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"2","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.874933236Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"2","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:06.874946205Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.874772305Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"io task","client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"ds_connection connected from Ok(127.0.0.1:52017)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.875147588Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"io task","client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.876264796Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.876299134Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.8768186Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877418682Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877445271Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"[0]R flush_numbers[0..12]: [137, 129, 131, 117, 135, 115, 134, 133, 123, 135, 127, 137]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.87746231Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"downstairs","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"[0]R generation[0..12]: [69, 64, 66, 58, 68, 58, 67, 66, 60, 68, 63, 69]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877477069Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"downstairs","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877491509Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"downstairs","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"[1]R flush_numbers[0..12]: [137, 129, 131, 117, 135, 115, 134, 133, 123, 135, 127, 137]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877505748Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"downstairs","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"[1]R generation[0..12]: [69, 64, 66, 58, 68, 58, 67, 66, 60, 68, 63, 69]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877519347Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"downstairs","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877533416Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"downstairs","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"[2]R flush_numbers[0..12]: [137, 129, 131, 117, 135, 115, 134, 133, 123, 135, 127, 137]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877547076Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"downstairs","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"[2]R generation[0..12]: [69, 64, 66, 58, 68, 58, 67, 66, 60, 68, 63, 69]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877560705Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"downstairs","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877574745Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"downstairs","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"Max found gen is 70","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877587524Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"downstairs","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"Generation requested: 70 >= found:70","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877600464Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"downstairs","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} Wait for a query_work_queue command to finish before sending IO {"msg":"Next flush: 138","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877621232Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"downstairs","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877635142Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"":"downstairs","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877648191Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877660001Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"0","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.87767333Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"1","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877687889Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"client":"2","":"downstairs","session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877709658Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877732847Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"b20a3b9e-8f03-4b5b-82de-d6a528458fd8 is now active with session: 8a117f4c-7b09-4127-85c0-1f1240a4b1fc","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877756656Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877775495Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462,"session_id":"8a117f4c-7b09-4127-85c0-1f1240a4b1fc"} {"msg":"The guest has finished waiting for activation with:70","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:16.877788185Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 475 len 4096 data: 6 02/30 Write block 511 len 16384 data: 5 4 3 3 03/30 Read block 143 len 4096 04/30 Write block 301 len 24576 data: 6 8 8 7 5 5 05/30 Write block 80 len 4096 data: 8 06/30 Write block 164 len 8192 data: 4 6 07/30 Write block 533 len 16384 data: 7 7 7 7 08/30 Write block 22 len 28672 data: 7 8 9 8 10 9 9 09/30 Write block 404 len 16384 data: 6 7 8 7 10/30 Write block 103 len 12288 data: 5 5 6 11/30 Write block 20 len 8192 data: 7 8 12/30 Write block 488 len 40960 data: 5 4 4 4 4 2 2 3 4 5 13/30 Write block 559 len 24576 data: 5 4 5 5 7 7 14/30 Flush 15/30 Write block 511 len 12288 data: 6 5 4 16/30 Write block 259 len 8192 data: 8 7 17/30 Read block 458 len 36864 18/30 Read block 264 len 4096 19/30 Write block 113 len 24576 data: 3 5 5 5 5 4 20/30 Read block 184 len 12288 21/30 Read block 455 len 36864 22/30 Flush 23/30 Write block 449 len 16384 data: 5 4 4 4 24/30 Write block 370 len 40960 data: 9 9 7 6 5 6 6 6 8 7 25/30 Write block 561 len 32768 data: 6 6 8 8 7 8 7 7 26/30 Read block 226 len 12288 27/30 Write block 149 len 20480 data: 8 8 6 6 7 28/30 Write block 540 len 36864 data: 5 6 5 6 7 6 6 6 7 29/30 Write block 449 len 32768 data: 6 5 5 5 4 3 5 6 30/30 Read block 443 len 4096 ---------------------------------------------------------------- Crucible gen:70 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 4 Done Done Done false 25 Acked 1024 Write 10 Done Done Done false 26 Acked 1025 Write 8 Done Done Done false 27 Acked 1026 Read 3 Done Done Done false 28 Acked 1027 Write 5 Done Done Done false 29 Acked 1028 Write 9 Done Done Done false 30 Acked 1029 Write 8 Done Done Done false 31 Acked 1030 Read 1 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 8 8 8 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:41:16.903327913Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1462} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 70 70 70 138 138 138 F F F 2 040-059 66 66 66 131 131 131 F F F 3 060-079 58 58 58 117 117 117 F F F 4 080-099 70 70 70 138 138 138 F F F 5 100-119 70 70 70 140 140 140 F F F 6 120-139 67 67 67 134 134 134 F F F 7 140-159 66 66 66 133 133 133 T T T 8 160-179 70 70 70 138 138 138 F F F 9 180-199 68 68 68 135 135 135 F F F 10 200-219 63 63 63 127 127 127 F F F 11 220-239 69 69 69 137 137 137 F F F 12 240-259 70 70 70 139 139 139 F F F 13 260-279 70 70 70 139 139 139 F F F 14 280-299 67 67 67 134 134 134 F F F 15 300-319 70 70 70 138 138 138 F F F 16 320-339 69 69 69 137 137 137 F F F 17 340-359 66 66 66 131 131 131 F F F 18 360-379 68 68 68 136 136 136 T T T 19 380-399 68 68 68 136 136 136 F F F 20 400-419 70 70 70 138 138 138 F F F 21 420-439 64 64 64 128 128 128 F F F 22 440-459 68 68 68 136 136 136 T T T 23 460-479 70 70 70 138 138 138 F F F 24 480-499 70 70 70 138 138 138 F F F 25 500-519 70 70 70 139 139 139 F F F 26 520-539 70 70 70 138 138 138 F F F 27 540-559 70 70 70 138 138 138 T T T 28 560-579 70 70 70 138 138 138 T T T 29 580-599 66 66 66 132 132 132 F F F Max gen: 70, Max flush: 140 A Difference in extent metadata is expected here On loop 35 Verifying data now No change after verify Loop: 35 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 70 70 70 138 138 138 F F F 2 040-059 66 66 66 131 131 131 F F F 3 060-079 58 58 58 117 117 117 F F F 4 080-099 70 70 70 138 138 138 F F F 5 100-119 70 70 70 140 140 140 F F F 6 120-139 67 67 67 134 134 134 F F F 7 140-159 71 71 71 141 141 141 F F F 8 160-179 70 70 70 138 138 138 F F F 9 180-199 68 68 68 135 135 135 F F F 10 200-219 63 63 63 127 127 127 F F F 11 220-239 69 69 69 137 137 137 F F F 12 240-259 70 70 70 139 139 139 F F F 13 260-279 70 70 70 139 139 139 F F F 14 280-299 67 67 67 134 134 134 F F F 15 300-319 70 70 70 138 138 138 F F F 16 320-339 69 69 69 137 137 137 F F F 17 340-359 66 66 66 131 131 131 F F F 18 360-379 71 71 71 141 141 141 F F F 19 380-399 68 68 68 136 136 136 F F F 20 400-419 70 70 70 138 138 138 F F F 21 420-439 64 64 64 128 128 128 F F F 22 440-459 71 71 71 141 141 141 F F F 23 460-479 70 70 70 138 138 138 F F F 24 480-499 70 70 70 138 138 138 F F F 25 500-519 70 70 70 139 139 139 F F F 26 520-539 70 70 70 138 138 138 F F F 27 540-559 71 71 71 141 141 141 F F F 28 560-579 71 71 71 141 141 141 F F F 29 580-599 66 66 66 132 132 132 F F F Max gen: 71, Max flush: 141 Begin loop 36 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.131171072Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.131710227Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.131739195Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470} {"msg":"Crucible f5060ce8-4ad6-4289-9425-939704a2a296 has session id: 5ecae0d1-8931-4f91-a344-6e985d01a22c","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.131895448Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"Upstairs opts: Upstairs UUID: f5060ce8-4ad6-4289-9425-939704a2a296, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.131936086Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"Crucible stats registered with UUID: f5060ce8-4ad6-4289-9425-939704a2a296","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.131958935Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:72","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.13228878Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470} {"msg":"f5060ce8-4ad6-4289-9425-939704a2a296 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.132322088Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.132337607Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"0","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.132355166Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"0","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.132376685Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.132401934Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.132427473Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"2","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.132452942Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"2","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.132475061Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"io task","client":"2","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.13249773Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"io task","client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.132520709Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"io task","client":"0","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:41:31.132688921Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"io task","client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:41:31.132728739Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"io task","client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"ds_connection connected from Ok(127.0.0.1:45496)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.132756318Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"io task","client":"0","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:41:31.132783277Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.132805045Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.13291988Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"ds_connection connected from Ok(127.0.0.1:61840)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.132956629Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"io task","client":"2","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.132984867Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"io task","client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.133527262Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"2","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.13355415Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"2","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.13356875Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"0","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.133582159Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"0","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.134226589Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"0","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.134343944Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"2","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.134738365Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"0","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.134765194Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"0","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.134779883Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.134792853Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"2","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.134806632Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"2","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:31.134819902Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.134838944Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"io task","client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"ds_connection connected from Ok(127.0.0.1:55085)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.135092872Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"io task","client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.135992821Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.136033059Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.136551224Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137096309Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137118658Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"[0]R flush_numbers[0..12]: [137, 138, 131, 117, 138, 140, 134, 141, 138, 135, 127, 137]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137136137Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"downstairs","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"[0]R generation[0..12]: [69, 70, 66, 58, 70, 70, 67, 71, 70, 68, 63, 69]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137153666Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"downstairs","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137171315Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"downstairs","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"[1]R flush_numbers[0..12]: [137, 138, 131, 117, 138, 140, 134, 141, 138, 135, 127, 137]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137188385Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"downstairs","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"[1]R generation[0..12]: [69, 70, 66, 58, 70, 70, 67, 71, 70, 68, 63, 69]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137205344Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"downstairs","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137222853Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"downstairs","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"[2]R flush_numbers[0..12]: [137, 138, 131, 117, 138, 140, 134, 141, 138, 135, 127, 137]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137240422Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"downstairs","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"[2]R generation[0..12]: [69, 70, 66, 58, 70, 70, 67, 71, 70, 68, 63, 69]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137257191Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"downstairs","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137274681Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"downstairs","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"Wait for a query_work_queue command to finish before sending IO Max found gen is 72","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137298249Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"downstairs","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"Generation requested: 72 >= found:72","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137315029Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"downstairs","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"Next flush: 142","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137338328Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"downstairs","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137367716Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"":"downstairs","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137395945Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137421574Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"0","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137452852Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"1","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137482231Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"client":"2","":"downstairs","session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.13751459Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137543788Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"f5060ce8-4ad6-4289-9425-939704a2a296 is now active with session: 5ecae0d1-8931-4f91-a344-6e985d01a22c","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137575227Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137604145Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470,"session_id":"5ecae0d1-8931-4f91-a344-6e985d01a22c"} {"msg":"The guest has finished waiting for activation with:72","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:41.137634044Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 43 len 40960 data: 5 5 5 5 6 5 5 4 2 4 02/30 Write block 503 len 24576 data: 6 6 6 6 6 7 03/30 Write block 573 len 12288 data: 13 12 10 04/30 Write block 257 len 36864 data: 10 8 9 8 6 7 4 5 5 05/30 Write block 560 len 32768 data: 5 7 7 9 9 8 9 8 06/30 Write block 483 len 24576 data: 4 6 4 4 6 6 07/30 Read block 406 len 36864 08/30 Read block 43 len 28672 09/30 Write block 495 len 12288 data: 4 5 6 10/30 Write block 386 len 28672 data: 4 5 3 3 6 6 9 11/30 Write block 305 len 28672 data: 6 6 6 7 8 6 7 12/30 Write block 115 len 20480 data: 6 6 6 5 6 13/30 Write block 22 len 40960 data: 8 9 10 9 11 10 10 9 7 5 14/30 Write block 511 len 40960 data: 7 6 5 4 3 3 3 4 4 4 15/30 Write block 448 len 12288 data: 6 7 6 16/30 Read block 393 len 32768 17/30 Read block 90 len 24576 18/30 Write block 201 len 40960 data: 7 7 6 7 6 6 7 7 6 6 19/30 Read block 488 len 40960 20/30 Write block 328 len 24576 data: 7 7 7 7 7 8 21/30 Write block 554 len 12288 data: 8 8 7 22/30 Read block 2 len 32768 23/30 Read block 120 len 16384 24/30 Read block 480 len 4096 25/30 Write block 549 len 32768 data: 6 8 8 7 9 9 9 8 26/30 Flush 27/30 Write block 435 len 8192 data: 9 8 28/30 Read block 577 len 20480 29/30 Write block 75 len 16384 data: 6 7 7 6 30/30 Read block 437 len 16384 ---------------------------------------------------------------- Crucible gen:72 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 28 Acked 1027 Write 2 Done Done Done false 29 Acked 1028 Read 5 Done Done Done false 30 Acked 1029 Write 4 Done Done Done false 31 Acked 1030 Read 4 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1026 1026 1026 Downstairs last five completed: 1026 1025 1024 1023 1022 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:41:41.163930917Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1470} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 72 72 72 142 142 142 F F F 2 040-059 72 72 72 142 142 142 F F F 3 060-079 58 58 58 117 117 117 T T T 4 080-099 70 70 70 138 138 138 F F F 5 100-119 72 72 72 142 142 142 F F F 6 120-139 67 67 67 134 134 134 F F F 7 140-159 71 71 71 141 141 141 F F F 8 160-179 70 70 70 138 138 138 F F F 9 180-199 68 68 68 135 135 135 F F F 10 200-219 72 72 72 142 142 142 F F F 11 220-239 69 69 69 137 137 137 F F F 12 240-259 72 72 72 142 142 142 F F F 13 260-279 72 72 72 142 142 142 F F F 14 280-299 67 67 67 134 134 134 F F F 15 300-319 72 72 72 142 142 142 F F F 16 320-339 72 72 72 143 143 143 F F F 17 340-359 66 66 66 131 131 131 F F F 18 360-379 71 71 71 141 141 141 F F F 19 380-399 72 72 72 142 142 142 F F F 20 400-419 70 70 70 138 138 138 F F F 21 420-439 64 64 64 128 128 128 T T T 22 440-459 72 72 72 142 142 142 F F F 23 460-479 70 70 70 138 138 138 F F F 24 480-499 72 72 72 142 142 142 F F F 25 500-519 72 72 72 142 142 142 F F F 26 520-539 72 72 72 142 142 142 F F F 27 540-559 72 72 72 143 143 143 F F F 28 560-579 72 72 72 142 142 142 F F F 29 580-599 66 66 66 132 132 132 F F F Max gen: 72, Max flush: 143 A Difference in extent metadata is expected here On loop 36 Verifying data now No change after verify Loop: 36 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 72 72 72 142 142 142 F F F 2 040-059 72 72 72 142 142 142 F F F 3 060-079 73 73 73 144 144 144 F F F 4 080-099 70 70 70 138 138 138 F F F 5 100-119 72 72 72 142 142 142 F F F 6 120-139 67 67 67 134 134 134 F F F 7 140-159 71 71 71 141 141 141 F F F 8 160-179 70 70 70 138 138 138 F F F 9 180-199 68 68 68 135 135 135 F F F 10 200-219 72 72 72 142 142 142 F F F 11 220-239 69 69 69 137 137 137 F F F 12 240-259 72 72 72 142 142 142 F F F 13 260-279 72 72 72 142 142 142 F F F 14 280-299 67 67 67 134 134 134 F F F 15 300-319 72 72 72 142 142 142 F F F 16 320-339 72 72 72 143 143 143 F F F 17 340-359 66 66 66 131 131 131 F F F 18 360-379 71 71 71 141 141 141 F F F 19 380-399 72 72 72 142 142 142 F F F 20 400-419 70 70 70 138 138 138 F F F 21 420-439 73 73 73 144 144 144 F F F 22 440-459 72 72 72 142 142 142 F F F 23 460-479 70 70 70 138 138 138 F F F 24 480-499 72 72 72 142 142 142 F F F 25 500-519 72 72 72 142 142 142 F F F 26 520-539 72 72 72 142 142 142 F F F 27 540-559 72 72 72 143 143 143 F F F 28 560-579 72 72 72 142 142 142 F F F 29 580-599 66 66 66 132 132 132 F F F Max gen: 73, Max flush: 144 Begin loop 37 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.352618552Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.353458583Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.353481152Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478} {"msg":"Crucible 78f67159-5da1-4379-8555-bba33e29d17a has session id: f5c133b1-a9e3-4364-a4b8-846d937a387b","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.353496381Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"Upstairs opts: Upstairs UUID: 78f67159-5da1-4379-8555-bba33e29d17a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.35351726Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"Crucible stats registered with UUID: 78f67159-5da1-4379-8555-bba33e29d17a","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.35353285Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:74","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.353694432Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478} {"msg":"78f67159-5da1-4379-8555-bba33e29d17a active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.353780748Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.353799197Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"0","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.353822406Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"0","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.353837445Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"1","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.353850825Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"1","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.353866164Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.353889753Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.353913822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"io task","client":"0","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.353938731Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"io task","client":"1","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.353961159Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"io task","client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"ds_connection connected from Ok(127.0.0.1:42804)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.354089653Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"io task","client":"0","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"ds_connection connected from Ok(127.0.0.1:34272)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.354132791Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"io task","client":"1","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:41:55.354403669Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"io task","client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:41:55.354431928Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"io task","client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:41:55.354448627Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.354461186Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.354483495Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.354509334Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"io task","client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.354958903Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"0","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.354994061Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"0","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.355073248Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"1","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.355090237Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"1","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.355639621Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"0","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.35566685Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"1","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.356227334Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"1","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.356255992Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"1","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.356272672Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.356285401Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"0","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.35629889Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"0","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:41:55.35631209Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.356049734Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"io task","client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"ds_connection connected from Ok(127.0.0.1:35512)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.356159499Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"io task","client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.3571987Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.357232729Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.357676578Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358214713Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358237042Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"[0]R flush_numbers[0..12]: [137, 142, 142, 144, 138, 142, 134, 141, 138, 135, 142, 137]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358255021Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"downstairs","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"[0]R generation[0..12]: [69, 72, 72, 73, 70, 72, 67, 71, 70, 68, 72, 69]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.35827263Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"downstairs","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358290589Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"downstairs","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"[1]R flush_numbers[0..12]: [137, 142, 142, 144, 138, 142, 134, 141, 138, 135, 142, 137]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358307848Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"downstairs","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"[1]R generation[0..12]: [69, 72, 72, 73, 70, 72, 67, 71, 70, 68, 72, 69]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358325028Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"downstairs","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358342577Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"downstairs","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"[2]R flush_numbers[0..12]: [137, 142, 142, 144, 138, 142, 134, 141, 138, 135, 142, 137]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358359956Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"downstairs","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"[2]R generation[0..12]: [69, 72, 72, 73, 70, 72, 67, 71, 70, 68, 72, 69]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358377205Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"downstairs","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358395404Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"downstairs","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"Max found gen is 74","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358415694Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"downstairs","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"Generation requested: 74 >= found:74","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:42:05.358432623Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"downstairs","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"Next flush: 145","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358449222Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"downstairs","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358465451Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"":"downstairs","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.35848223Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.35849716Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"0","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358513779Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"1","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358530198Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"client":"2","":"downstairs","session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358546657Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358561987Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"78f67159-5da1-4379-8555-bba33e29d17a is now active with session: f5c133b1-a9e3-4364-a4b8-846d937a387b","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358577986Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358592885Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478,"session_id":"f5c133b1-a9e3-4364-a4b8-846d937a387b"} {"msg":"The guest has finished waiting for activation with:74","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:05.358608074Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 35 len 20480 data: 3 5 5 6 7 02/30 Write block 564 len 32768 data: 10 9 10 9 8 5 9 11 03/30 Flush 04/30 Write block 475 len 4096 data: 7 05/30 Read block 353 len 4096 06/30 Read block 269 len 12288 07/30 Write block 369 len 12288 data: 11 10 10 08/30 Write block 371 len 8192 data: 11 8 09/30 Write block 292 len 24576 data: 6 5 5 5 4 4 10/30 Flush 11/30 Flush 12/30 Write block 502 len 36864 data: 8 7 7 7 7 7 8 6 7 13/30 Read block 435 len 28672 14/30 Write block 505 len 20480 data: 8 8 8 9 7 15/30 Read block 317 len 12288 16/30 Read block 93 len 12288 17/30 Read block 149 len 8192 18/30 Read block 404 len 28672 19/30 Read block 437 len 4096 20/30 Flush 21/30 Write block 247 len 28672 data: 7 7 7 6 5 7 6 22/30 Write block 522 len 16384 data: 4 4 4 3 23/30 Read block 40 len 16384 24/30 Read block 24 len 16384 25/30 Write block 222 len 40960 data: 9 7 6 6 4 5 6 7 7 7 26/30 Write block 262 len 28672 data: 8 5 6 6 6 7 8 27/30 Read block 188 len 40960 28/30 Write block 34 len 28672 data: 4 4 6 6 7 8 6 29/30 Read block 498 len 40960 30/30 Write block 164 len 4096 data: 5 ---------------------------------------------------------------- Crucible gen:74 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 7 Done Done Done false 23 Acked 1022 Write 4 Done Done Done false 24 Acked 1023 Read 4 Done Done Done false 25 Acked 1024 Read 4 Done Done Done false 26 Acked 1025 Write 10 Done Done Done false 27 Acked 1026 Write 7 Done Done Done false 28 Acked 1027 Read 10 Done Done Done false 29 Acked 1028 Write 7 Done Done Done false 30 Acked 1029 Read 10 Sent Done Done false 31 Acked 1030 Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 2 1 1 4 Done 8 9 9 26 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:42:05.386442116Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1478} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 74 74 74 145 145 145 T T T 2 040-059 72 72 72 142 142 142 T T T 3 060-079 73 73 73 144 144 144 F F F 4 080-099 70 70 70 138 138 138 F F F 5 100-119 72 72 72 142 142 142 F F F 6 120-139 67 67 67 134 134 134 F F F 7 140-159 71 71 71 141 141 141 F F F 8 160-179 70 70 70 138 138 138 T T T 9 180-199 68 68 68 135 135 135 F F F 10 200-219 72 72 72 142 142 142 F F F 11 220-239 69 69 69 137 137 137 T T T 12 240-259 72 72 72 142 142 142 T T T 13 260-279 72 72 72 142 142 142 T T T 14 280-299 74 74 74 146 146 146 F F F 15 300-319 72 72 72 142 142 142 F F F 16 320-339 72 72 72 143 143 143 F F F 17 340-359 66 66 66 131 131 131 F F F 18 360-379 74 74 74 146 146 146 F F F 19 380-399 72 72 72 142 142 142 F F F 20 400-419 70 70 70 138 138 138 F F F 21 420-439 73 73 73 144 144 144 F F F 22 440-459 72 72 72 142 142 142 F F F 23 460-479 74 74 74 146 146 146 F F F 24 480-499 72 72 72 142 142 142 F F F 25 500-519 74 74 74 148 148 148 F F F 26 520-539 72 72 72 142 142 142 T T T 27 540-559 72 72 72 143 143 143 F F F 28 560-579 74 74 74 145 145 145 F F F 29 580-599 66 66 66 132 132 132 F F F Max gen: 74, Max flush: 148 A Difference in extent metadata is expected here On loop 37 Verifying data now No change after verify Loop: 37 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 75 75 75 149 149 149 F F F 2 040-059 75 75 75 149 149 149 F F F 3 060-079 73 73 73 144 144 144 F F F 4 080-099 70 70 70 138 138 138 F F F 5 100-119 72 72 72 142 142 142 F F F 6 120-139 67 67 67 134 134 134 F F F 7 140-159 71 71 71 141 141 141 F F F 8 160-179 75 75 75 149 149 149 F F F 9 180-199 68 68 68 135 135 135 F F F 10 200-219 72 72 72 142 142 142 F F F 11 220-239 75 75 75 149 149 149 F F F 12 240-259 75 75 75 149 149 149 F F F 13 260-279 75 75 75 149 149 149 F F F 14 280-299 74 74 74 146 146 146 F F F 15 300-319 72 72 72 142 142 142 F F F 16 320-339 72 72 72 143 143 143 F F F 17 340-359 66 66 66 131 131 131 F F F 18 360-379 74 74 74 146 146 146 F F F 19 380-399 72 72 72 142 142 142 F F F 20 400-419 70 70 70 138 138 138 F F F 21 420-439 73 73 73 144 144 144 F F F 22 440-459 72 72 72 142 142 142 F F F 23 460-479 74 74 74 146 146 146 F F F 24 480-499 72 72 72 142 142 142 F F F 25 500-519 74 74 74 148 148 148 F F F 26 520-539 75 75 75 149 149 149 F F F 27 540-559 72 72 72 143 143 143 F F F 28 560-579 74 74 74 145 145 145 F F F 29 580-599 66 66 66 132 132 132 F F F Max gen: 75, Max flush: 149 Begin loop 38 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.631722182Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.632145732Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.632173271Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486} {"msg":"Crucible 2a51eb66-7e23-433e-9067-f3409a9e294d has session id: 27cf3046-4918-46c2-8d59-e794c7a06ee5","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.632428339Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"Upstairs opts: Upstairs UUID: 2a51eb66-7e23-433e-9067-f3409a9e294d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.632473307Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"Crucible stats registered with UUID: 2a51eb66-7e23-433e-9067-f3409a9e294d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.632497036Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:76","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.632738335Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486} {"msg":"2a51eb66-7e23-433e-9067-f3409a9e294d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.632802772Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.632830711Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"0","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.632877468Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"0","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.632904517Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.632927616Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.632952175Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"2","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.632970734Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"2","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.632984323Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"io task","client":"2","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.633003143Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"io task","client":"0","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.633026841Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"io task","client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:42:19.633140166Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"io task","client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:42:19.633188574Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"io task","client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:42:19.633218202Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"ds_connection connected from Ok(127.0.0.1:35808)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.633240101Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"io task","client":"2","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.6332652Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.633301249Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"ds_connection connected from Ok(127.0.0.1:63184)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.633375485Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"io task","client":"0","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.633407164Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"io task","client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.634107521Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"2","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.63413601Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"2","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.634152229Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"0","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.634175338Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"0","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.634754421Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"0","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.634783249Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"2","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.635318395Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"0","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.635345773Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"0","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.635360553Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.635372842Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"2","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.635386011Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"2","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:19.635399551Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.635037651Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"io task","client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"ds_connection connected from Ok(127.0.0.1:58692)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.635139806Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"io task","client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.636367449Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.636395847Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.636916923Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637479647Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637514495Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"[0]R flush_numbers[0..12]: [137, 149, 149, 144, 138, 142, 134, 141, 149, 135, 142, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637534754Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"downstairs","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"[0]R generation[0..12]: [69, 75, 75, 73, 70, 72, 67, 71, 75, 68, 72, 75]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637553023Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"downstairs","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637570903Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"downstairs","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"[1]R flush_numbers[0..12]: [137, 149, 149, 144, 138, 142, 134, 141, 149, 135, 142, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637588282Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"downstairs","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"[1]R generation[0..12]: [69, 75, 75, 73, 70, 72, 67, 71, 75, 68, 72, 75]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637605351Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"downstairs","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.63762274Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"downstairs","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"[2]R flush_numbers[0..12]: [137, 149, 149, 144, 138, 142, 134, 141, 149, 135, 142, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637639959Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"downstairs","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"[2]R generation[0..12]: [69, 75, 75, 73, 70, 72, 67, 71, 75, 68, 72, 75]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637656839Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"downstairs","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} Wait for a query_work_queue command to finish before sending IO {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637682697Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"downstairs","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"Max found gen is 76","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637700386Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"downstairs","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"Generation requested: 76 >= found:76","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637717066Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"downstairs","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"Next flush: 150","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637733155Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"downstairs","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637749284Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"":"downstairs","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637765623Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637780463Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"0","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637796992Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"1","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637813231Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"client":"2","":"downstairs","session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637829731Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.63784502Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"2a51eb66-7e23-433e-9067-f3409a9e294d is now active with session: 27cf3046-4918-46c2-8d59-e794c7a06ee5","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637861229Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:29.637876148Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486,"session_id":"27cf3046-4918-46c2-8d59-e794c7a06ee5"} {"msg":"The guest has finished waiting for activation with:76","v":0,"name":"crucible","level":30,"time":"Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2024-07-31T18:42:29.637891368Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 100 len 8192 data: 5 5 02/30 Read block 563 len 8192 03/30 Read block 497 len 32768 04/30 Read block 263 len 16384 05/30 Read block 591 len 16384 06/30 Flush 07/30 Write block 563 len 4096 data: 10 08/30 Read block 228 len 12288 09/30 Read block 498 len 16384 10/30 Write block 551 len 24576 data: 9 8 10 10 10 9 11/30 Write block 497 len 16384 data: 7 4 4 4 12/30 Write block 184 len 8192 data: 8 9 13/30 Read block 464 len 4096 14/30 Write block 373 len 32768 data: 7 6 7 7 7 9 8 7 15/30 Read block 280 len 8192 16/30 Write block 230 len 36864 data: 8 8 10 7 8 9 9 7 8 17/30 Write block 383 len 40960 data: 6 7 4 5 6 4 4 7 7 10 18/30 Read block 237 len 4096 19/30 Read block 401 len 4096 20/30 Write block 290 len 28672 data: 7 7 7 6 6 6 5 21/30 Read block 46 len 20480 22/30 Write block 341 len 24576 data: 6 6 6 5 5 4 23/30 Write block 208 len 20480 data: 8 7 7 6 5 24/30 Write block 524 len 28672 data: 5 4 6 5 5 4 5 25/30 Write block 206 len 28672 data: 7 8 9 8 8 7 6 26/30 Read block 207 len 32768 27/30 Read block 288 len 40960 28/30 Write block 364 len 32768 data: 7 8 8 11 12 12 11 12 29/30 Read block 289 len 28672 30/30 Write block 288 len 36864 data: 6 7 8 8 8 7 7 7 6 ---------------------------------------------------------------- Crucible gen:76 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 23 Acked 1022 Write 6 Done Done Done false 24 Acked 1023 Write 5 Done Done Done false 25 Acked 1024 Write 7 Done Done Done false 26 Acked 1025 Write 7 Done Done Done false 27 Acked 1026 Read 8 Done Done Done false 28 Acked 1027 Read 10 Done Done Done false 29 Acked 1028 Write 8 Done Done Done false 30 Acked 1029 Read 7 Done Done Done false 31 Acked 1030 Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 8 8 8 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1021 1021 1021 Downstairs last five completed: 1021 1020 1019 1018 1017 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:42:29.661510066Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1486} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 75 75 75 149 149 149 F F F 2 040-059 75 75 75 149 149 149 F F F 3 060-079 73 73 73 144 144 144 F F F 4 080-099 70 70 70 138 138 138 F F F 5 100-119 76 76 76 150 150 150 F F F 6 120-139 67 67 67 134 134 134 F F F 7 140-159 71 71 71 141 141 141 F F F 8 160-179 75 75 75 149 149 149 F F F 9 180-199 76 76 76 151 151 151 F F F 10 200-219 72 72 72 142 142 142 T T T 11 220-239 76 76 76 151 151 151 F F F 12 240-259 75 75 75 149 149 149 F F F 13 260-279 75 75 75 149 149 149 F F F 14 280-299 76 76 76 151 151 151 T T T 15 300-319 72 72 72 142 142 142 F F F 16 320-339 72 72 72 143 143 143 F F F 17 340-359 66 66 66 131 131 131 T T T 18 360-379 76 76 76 151 151 151 T T T 19 380-399 76 76 76 151 151 151 F F F 20 400-419 70 70 70 138 138 138 F F F 21 420-439 73 73 73 144 144 144 F F F 22 440-459 72 72 72 142 142 142 F F F 23 460-479 74 74 74 146 146 146 F F F 24 480-499 76 76 76 151 151 151 F F F 25 500-519 76 76 76 151 151 151 F F F 26 520-539 75 75 75 149 149 149 T T T 27 540-559 76 76 76 151 151 151 F F F 28 560-579 76 76 76 151 151 151 F F F 29 580-599 66 66 66 132 132 132 F F F Max gen: 76, Max flush: 151 A Difference in extent metadata is expected here On loop 38 Verifying data now No change after verify Loop: 38 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 75 75 75 149 149 149 F F F 2 040-059 75 75 75 149 149 149 F F F 3 060-079 73 73 73 144 144 144 F F F 4 080-099 70 70 70 138 138 138 F F F 5 100-119 76 76 76 150 150 150 F F F 6 120-139 67 67 67 134 134 134 F F F 7 140-159 71 71 71 141 141 141 F F F 8 160-179 75 75 75 149 149 149 F F F 9 180-199 76 76 76 151 151 151 F F F 10 200-219 77 77 77 152 152 152 F F F 11 220-239 76 76 76 151 151 151 F F F 12 240-259 75 75 75 149 149 149 F F F 13 260-279 75 75 75 149 149 149 F F F 14 280-299 77 77 77 152 152 152 F F F 15 300-319 72 72 72 142 142 142 F F F 16 320-339 72 72 72 143 143 143 F F F 17 340-359 77 77 77 152 152 152 F F F 18 360-379 77 77 77 152 152 152 F F F 19 380-399 76 76 76 151 151 151 F F F 20 400-419 70 70 70 138 138 138 F F F 21 420-439 73 73 73 144 144 144 F F F 22 440-459 72 72 72 142 142 142 F F F 23 460-479 74 74 74 146 146 146 F F F 24 480-499 76 76 76 151 151 151 F F F 25 500-519 76 76 76 151 151 151 F F F 26 520-539 77 77 77 152 152 152 F F F 27 540-559 76 76 76 151 151 151 F F F 28 560-579 76 76 76 151 151 151 F F F 29 580-599 66 66 66 132 132 132 F F F Max gen: 77, Max flush: 152 Begin loop 39 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.884791238Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.885203639Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.885226197Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494} {"msg":"Crucible 9635c8c4-ecc7-4725-b6c8-3a2c600a7c7b has session id: 9acacea1-7390-4a71-8d86-521ef9db4dbd","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.885373141Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"Upstairs opts: Upstairs UUID: 9635c8c4-ecc7-4725-b6c8-3a2c600a7c7b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.885415399Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"Crucible stats registered with UUID: 9635c8c4-ecc7-4725-b6c8-3a2c600a7c7b","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.885439028Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:78","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.885837659Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494} {"msg":"9635c8c4-ecc7-4725-b6c8-3a2c600a7c7b active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.885869198Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.885883727Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"0","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.885902676Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"0","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.885916735Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"1","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.885930705Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"1","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.885945504Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.885969093Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.885993282Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"io task","client":"0","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.88602726Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"io task","client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.886055569Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"io task","client":"1","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:42:43.886178043Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"io task","client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"ds_connection connected from Ok(127.0.0.1:57549)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.886212842Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"io task","client":"1","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"ds_connection connected from Ok(127.0.0.1:56635)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.88623883Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"io task","client":"0","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:42:43.886268219Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"io task","client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:42:43.886302207Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.886327316Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.886351905Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.886375164Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"io task","client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.886921478Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"0","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.886948107Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"0","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.886962416Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"1","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.886976196Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"1","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.887566498Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"0","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.887602807Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"1","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.888157861Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"0","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.888184839Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"0","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.888201089Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.888241107Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"1","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.888258436Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"1","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:43.888272466Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.887968963Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"io task","client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"ds_connection connected from Ok(127.0.0.1:52988)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.888137085Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"io task","client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.889335539Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.889367977Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.88996527Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.89059455Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890616169Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"[0]R flush_numbers[0..12]: [137, 149, 149, 144, 138, 150, 134, 141, 149, 151, 152, 151]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890631438Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"downstairs","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"[0]R generation[0..12]: [69, 75, 75, 73, 70, 76, 67, 71, 75, 76, 77, 76]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890645898Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"downstairs","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890660127Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"downstairs","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"[1]R flush_numbers[0..12]: [137, 149, 149, 144, 138, 150, 134, 141, 149, 151, 152, 151]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890674027Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"downstairs","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"[1]R generation[0..12]: [69, 75, 75, 73, 70, 76, 67, 71, 75, 76, 77, 76]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890688086Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"downstairs","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890702305Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"downstairs","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"[2]R flush_numbers[0..12]: [137, 149, 149, 144, 138, 150, 134, 141, 149, 151, 152, 151]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890716234Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"downstairs","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"[2]R generation[0..12]: [69, 75, 75, 73, 70, 76, 67, 71, 75, 76, 77, 76]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890734454Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"downstairs","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890749263Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"downstairs","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"Max found gen is 78","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890762472Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"downstairs","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"Generation requested: 78 >= found:78","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890775682Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"downstairs","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"Next flush: 153","v":0,"name":Wait for a query_work_queue command to finish before sending IO "crucible","level":30,"time":"2024-07-31T18:42:53.890792791Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"downstairs","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.89080613Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"":"downstairs","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.89082158Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890842419Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"0","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890867678Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"1","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890892426Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"client":"2","":"downstairs","session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890915855Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890937974Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"9635c8c4-ecc7-4725-b6c8-3a2c600a7c7b is now active with session: 9acacea1-7390-4a71-8d86-521ef9db4dbd","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890962453Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.890984432Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494,"session_id":"9acacea1-7390-4a71-8d86-521ef9db4dbd"} {"msg":"The guest has finished waiting for activation with:78","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:42:53.891005321Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 91 len 32768 data: 6 4 3 3 3 2 3 4 02/30 Read block 498 len 12288 03/30 Read block 224 len 40960 04/30 Write block 490 len 20480 data: 5 5 5 3 3 05/30 Write block 235 len 32768 data: 10 10 8 9 7 6 5 6 06/30 Read block 390 len 20480 07/30 Write block 80 len 28672 data: 9 8 8 9 11 11 8 08/30 Read block 139 len 24576 09/30 Read block 228 len 12288 10/30 Read block 188 len 4096 11/30 Read block 53 len 16384 12/30 Write block 126 len 32768 data: 5 6 6 5 6 6 6 6 13/30 Write block 79 len 36864 data: 6 10 9 9 10 12 12 9 6 14/30 Flush 15/30 Write block 576 len 36864 data: 10 9 8 7 7 7 9 7 8 16/30 Read block 256 len 36864 17/30 Write block 589 len 40960 data: 5 5 4 5 7 7 6 4 4 4 18/30 Flush 19/30 Write block 481 len 20480 data: 4 4 5 7 5 20/30 Write block 523 len 8192 data: 5 6 21/30 Read block 305 len 8192 22/30 Write block 514 len 12288 data: 5 4 4 23/30 Read block 335 len 28672 24/30 Write block 301 len 8192 data: 7 9 25/30 Read block 269 len 36864 26/30 Read block 287 len 28672 27/30 Read block 526 len 20480 28/30 Read block 136 len 20480 29/30 Write block 275 len 32768 data: 7 7 7 7 9 10 6 6 30/30 Write block 497 len 24576 data: 8 5 5 5 6 9 ---------------------------------------------------------------- Crucible gen:78 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 20 Acked 1019 Write 5 Done Done Done false 21 Acked 1020 Write 2 Done Done Done false 22 Acked 1021 Read 2 Done Done Done false 23 Acked 1022 Write 3 Done Done Done false 24 Acked 1023 Read 7 Done Done Done false 25 Acked 1024 Write 2 Done Done Done false 26 Acked 1025 Read 9 Done Done Done false 27 Acked 1026 Read 7 Done Done Done false 28 Acked 1027 Read 5 Done Done Done false 29 Acked 1028 Read 5 Done Done Done false 30 Acked 1029 Write 8 Done Sent Sent false 31 Acked 1030 Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 2 2 5 Done 11 10 10 31 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1018 1018 Downstairs last five completed: 1018 1017 1016 1015 1014 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:42:53.918162584Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1494} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 75 75 75 149 149 149 F F F 2 040-059 75 75 75 149 149 149 F F F 3 060-079 78 78 78 153 153 153 F F F 4 080-099 78 78 78 153 153 153 F F F 5 100-119 76 76 76 150 150 150 F F F 6 120-139 78 78 78 153 153 153 F F F 7 140-159 71 71 71 141 141 141 F F F 8 160-179 75 75 75 149 149 149 F F F 9 180-199 76 76 76 151 151 151 F F F 10 200-219 77 77 77 152 152 152 F F F 11 220-239 78 78 78 153 153 153 F F F 12 240-259 78 78 78 153 153 153 F F F 13 260-279 75 75 75 149 149 149 T T T 14 280-299 77 77 77 152 152 152 T T T 15 300-319 72 72 72 142 142 142 T T T 16 320-339 72 72 72 143 143 143 F F F 17 340-359 77 77 77 152 152 152 F F F 18 360-379 77 77 77 152 152 152 F F F 19 380-399 76 76 76 151 151 151 F F F 20 400-419 70 70 70 138 138 138 F F F 21 420-439 73 73 73 144 144 144 F F F 22 440-459 72 72 72 142 142 142 F F F 23 460-479 74 74 74 146 146 146 F F F 24 480-499 78 78 78 153 153 153 T T T 25 500-519 76 76 76 151 151 151 T T T 26 520-539 77 77 77 152 152 152 T T T 27 540-559 76 76 76 151 151 151 F F F 28 560-579 78 78 78 154 154 154 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 78, Max flush: 154 A Difference in extent metadata is expected here On loop 39 Verifying data now No change after verify Loop: 39 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 75 75 75 149 149 149 F F F 2 040-059 75 75 75 149 149 149 F F F 3 060-079 78 78 78 153 153 153 F F F 4 080-099 78 78 78 153 153 153 F F F 5 100-119 76 76 76 150 150 150 F F F 6 120-139 78 78 78 153 153 153 F F F 7 140-159 71 71 71 141 141 141 F F F 8 160-179 75 75 75 149 149 149 F F F 9 180-199 76 76 76 151 151 151 F F F 10 200-219 77 77 77 152 152 152 F F F 11 220-239 78 78 78 153 153 153 F F F 12 240-259 78 78 78 153 153 153 F F F 13 260-279 79 79 79 155 155 155 F F F 14 280-299 79 79 79 155 155 155 F F F 15 300-319 79 79 79 155 155 155 F F F 16 320-339 72 72 72 143 143 143 F F F 17 340-359 77 77 77 152 152 152 F F F 18 360-379 77 77 77 152 152 152 F F F 19 380-399 76 76 76 151 151 151 F F F 20 400-419 70 70 70 138 138 138 F F F 21 420-439 73 73 73 144 144 144 F F F 22 440-459 72 72 72 142 142 142 F F F 23 460-479 74 74 74 146 146 146 F F F 24 480-499 79 79 79 155 155 155 F F F 25 500-519 79 79 79 155 155 155 F F F 26 520-539 79 79 79 155 155 155 F F F 27 540-559 76 76 76 151 151 151 F F F 28 560-579 78 78 78 154 154 154 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 79, Max flush: 155 Begin loop 40 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.166895609Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.167288411Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.16731938Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502} {"msg":"Crucible bca29cbb-8cb2-49bb-a433-92599dc32361 has session id: 270572bc-ffb7-4e68-804b-d4c640300528","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.167441534Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"Upstairs opts: Upstairs UUID: bca29cbb-8cb2-49bb-a433-92599dc32361, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.167480582Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"Crucible stats registered with UUID: bca29cbb-8cb2-49bb-a433-92599dc32361","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.167507491Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:80","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.167784558Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502} {"msg":"bca29cbb-8cb2-49bb-a433-92599dc32361 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.167849465Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.167874724Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"0","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.167895023Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"0","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.167909052Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.167922692Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.167935841Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"2","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.16794915Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"2","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.167974409Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"io task","client":"0","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.167988418Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"io task","client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.168001968Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"io task","client":"2","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"ds_connection connected from Ok(127.0.0.1:53213)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.168243866Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"io task","client":"0","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"ds_connection connected from Ok(127.0.0.1:64916)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.168355151Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"io task","client":"2","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:43:08.168476816Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"io task","client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:43:08.168520184Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"io task","client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:43:08.168550762Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.168572131Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.16859724Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.16880302Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"io task","client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.168965463Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"0","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.168992582Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"0","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.169008371Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"2","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.16902218Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"2","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.169594503Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"2","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.169732457Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"0","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.17009411Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"2","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.170120829Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"2","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.170136048Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.170162977Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"0","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.170181316Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"0","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:08.170195095Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.169353069Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"io task","client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"ds_connection connected from Ok(127.0.0.1:34258)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.169646015Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"io task","client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.170875918Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.170915516Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.17147499Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172050893Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172074602Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"[0]R flush_numbers[0..12]: [137, 149, 149, 153, 153, 150, 153, 141, 149, 151, 152, 153]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172090221Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"downstairs","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"[0]R generation[0..12]: [69, 75, 75, 78, 78, 76, 78, 71, 75, 76, 77, 78]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.17210471Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"downstairs","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.17211919Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"downstairs","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"[1]R flush_numbers[0..12]: [137, 149, 149, 153, 153, 150, 153, 141, 149, 151, 152, 153]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172133199Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"downstairs","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"[1]R generation[0..12]: [69, 75, 75, 78, 78, 76, 78, 71, 75, 76, 77, 78]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172146778Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"downstairs","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172166048Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"downstairs","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"[2]R flush_numbers[0..12]: [137, 149, 149, 153, 153, 150, 153, 141, 149, 151, 152, 153]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172180287Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"downstairs","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"[2]R generation[0..12]: [69, 75, 75, 78, 78, 76, 78, 71, 75, 76, 77, 78]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172193976Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"downstairs","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172208046Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"downstairs","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"Max found gen is 80","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172220695Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"downstairs","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"Generation requested: 80 >= found:80","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172233764Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"downstairs","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {Wait for a query_work_queue command to finish before sending IO "msg":"Next flush: 156","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172250184Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"downstairs","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172263223Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"":"downstairs","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172276332Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172288332Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"0","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172301761Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"1","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172315141Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"client":"2","":"downstairs","session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.1723285Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172340679Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"bca29cbb-8cb2-49bb-a433-92599dc32361 is now active with session: 270572bc-ffb7-4e68-804b-d4c640300528","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172353559Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172365418Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502,"session_id":"270572bc-ffb7-4e68-804b-d4c640300528"} {"msg":"The guest has finished waiting for activation with:80","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:18.172380507Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 563 len 28672 data: 11 11 10 11 10 9 6 02/30 Write block 539 len 16384 data: 6 6 7 6 03/30 Read block 188 len 20480 04/30 Write block 493 len 36864 data: 4 4 5 6 9 6 6 6 7 05/30 Write block 49 len 8192 data: 6 5 06/30 Write block 105 len 28672 data: 7 4 3 4 4 4 4 07/30 Read block 260 len 36864 08/30 Write block 148 len 40960 data: 9 9 9 7 7 8 7 6 5 5 09/30 Read block 368 len 12288 10/30 Read block 24 len 36864 11/30 Read block 371 len 8192 12/30 Write block 143 len 16384 data: 6 6 6 7 13/30 Write block 313 len 20480 data: 9 11 10 8 7 14/30 Read block 223 len 12288 15/30 Write block 190 len 4096 data: 11 16/30 Read block 571 len 12288 17/30 Read block 373 len 24576 18/30 Read block 228 len 8192 19/30 Write block 442 len 32768 data: 7 6 6 9 7 5 7 8 20/30 Read block 526 len 4096 21/30 Read block 34 len 28672 22/30 Read block 99 len 36864 23/30 Read block 516 len 20480 24/30 Read block 96 len 32768 25/30 Read block 417 len 32768 26/30 Write block 169 len 20480 data: 7 8 8 8 8 27/30 Read block 424 len 28672 28/30 Write block 221 len 28672 data: 9 10 8 7 7 5 6 29/30 Write block 398 len 28672 data: 9 7 7 7 7 6 7 30/30 Write block 24 len 32768 data: 11 10 12 11 11 10 8 6 ---------------------------------------------------------------- Crucible gen:80 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 23 Acked 1022 Read 9 Done Done Done false 24 Acked 1023 Read 5 Done Done Done false 25 Acked 1024 Read 8 Done Done Done false 26 Acked 1025 Read 8 Done Done Done false 27 Acked 1026 Write 5 Done Done Done false 28 Acked 1027 Read 7 Done Done Done false 29 Acked 1028 Write 7 Done Done Done false 30 Acked 1029 Write 7 Done Done Done false 31 Acked 1030 Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 8 8 8 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1021 1021 1021 Downstairs last five completed: 1021 1020 1019 1018 1017 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:43:18.197848089Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1502} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 75 75 75 149 149 149 T T T 2 040-059 80 80 80 156 156 156 F F F 3 060-079 78 78 78 153 153 153 F F F 4 080-099 78 78 78 153 153 153 F F F 5 100-119 80 80 80 156 156 156 F F F 6 120-139 78 78 78 153 153 153 F F F 7 140-159 80 80 80 156 156 156 F F F 8 160-179 75 75 75 149 149 149 T T T 9 180-199 80 80 80 156 156 156 F F F 10 200-219 77 77 77 152 152 152 F F F 11 220-239 78 78 78 153 153 153 T T T 12 240-259 78 78 78 153 153 153 F F F 13 260-279 79 79 79 155 155 155 F F F 14 280-299 79 79 79 155 155 155 F F F 15 300-319 80 80 80 156 156 156 F F F 16 320-339 72 72 72 143 143 143 F F F 17 340-359 77 77 77 152 152 152 F F F 18 360-379 77 77 77 152 152 152 F F F 19 380-399 76 76 76 151 151 151 T T T 20 400-419 70 70 70 138 138 138 T T T 21 420-439 73 73 73 144 144 144 F F F 22 440-459 80 80 80 156 156 156 F F F 23 460-479 74 74 74 146 146 146 F F F 24 480-499 80 80 80 156 156 156 F F F 25 500-519 80 80 80 156 156 156 F F F 26 520-539 80 80 80 156 156 156 F F F 27 540-559 80 80 80 156 156 156 F F F 28 560-579 80 80 80 156 156 156 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 80, Max flush: 156 A Difference in extent metadata is expected here On loop 40 Verifying data now No change after verify Loop: 40 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 81 81 81 157 157 157 F F F 2 040-059 80 80 80 156 156 156 F F F 3 060-079 78 78 78 153 153 153 F F F 4 080-099 78 78 78 153 153 153 F F F 5 100-119 80 80 80 156 156 156 F F F 6 120-139 78 78 78 153 153 153 F F F 7 140-159 80 80 80 156 156 156 F F F 8 160-179 81 81 81 157 157 157 F F F 9 180-199 80 80 80 156 156 156 F F F 10 200-219 77 77 77 152 152 152 F F F 11 220-239 81 81 81 157 157 157 F F F 12 240-259 78 78 78 153 153 153 F F F 13 260-279 79 79 79 155 155 155 F F F 14 280-299 79 79 79 155 155 155 F F F 15 300-319 80 80 80 156 156 156 F F F 16 320-339 72 72 72 143 143 143 F F F 17 340-359 77 77 77 152 152 152 F F F 18 360-379 77 77 77 152 152 152 F F F 19 380-399 81 81 81 157 157 157 F F F 20 400-419 81 81 81 157 157 157 F F F 21 420-439 73 73 73 144 144 144 F F F 22 440-459 80 80 80 156 156 156 F F F 23 460-479 74 74 74 146 146 146 F F F 24 480-499 80 80 80 156 156 156 F F F 25 500-519 80 80 80 156 156 156 F F F 26 520-539 80 80 80 156 156 156 F F F 27 540-559 80 80 80 156 156 156 F F F 28 560-579 80 80 80 156 156 156 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 81, Max flush: 157 Begin loop 41 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.419553066Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.419849333Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.419866542Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510} {"msg":"Crucible 1c3fbe79-3280-4faf-9cd1-5c44222e692d has session id: b95ffddd-4abd-4453-85ef-39e5904749cb","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.420504262Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"Upstairs opts: Upstairs UUID: 1c3fbe79-3280-4faf-9cd1-5c44222e692d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.42054226Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"Crucible stats registered with UUID: 1c3fbe79-3280-4faf-9cd1-5c44222e692d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.420560719Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:82","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.420907343Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510} {"msg":"1c3fbe79-3280-4faf-9cd1-5c44222e692d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.420989999Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.421007948Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"0","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.421027967Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"0","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.421042267Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.421056006Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.421069365Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"2","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.421085345Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"2","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.421108434Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"io task","client":"2","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.421127623Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"io task","client":"0","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.421141502Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"io task","client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"ds_connection connected from Ok(127.0.0.1:45477)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.421354112Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"io task","client":"2","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:43:32.421486656Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"io task","client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:43:32.421528544Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"io task","client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"ds_connection connected from Ok(127.0.0.1:62348)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.421558123Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"io task","client":"0","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:43:32.421591681Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.42161072Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.42162485Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.421694566Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"io task","client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.422228771Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"2","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.42225654Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"2","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.422281819Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"0","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.422307308Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"0","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.422942208Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"2","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.422980786Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"0","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.42353803Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"0","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.423564139Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"0","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.423582388Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.423594928Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"2","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.423608477Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"2","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:32.423621786Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.422063291Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"io task","client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"ds_connection connected from Ok(127.0.0.1:57190)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.422327409Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"io task","client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.423413879Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.423447487Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.423999651Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424614843Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424637982Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"[0]R flush_numbers[0..12]: [137, 157, 156, 153, 153, 156, 153, 156, 157, 156, 152, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424656061Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"downstairs","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"[0]R generation[0..12]: [69, 81, 80, 78, 78, 80, 78, 80, 81, 80, 77, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.42467411Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"downstairs","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424692009Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"downstairs","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"[1]R flush_numbers[0..12]: [137, 157, 156, 153, 153, 156, 153, 156, 157, 156, 152, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424709128Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"downstairs","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"[1]R generation[0..12]: [69, 81, 80, 78, 78, 80, 78, 80, 81, 80, 77, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424730997Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"downstairs","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424748916Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"downstairs","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"[2]R flush_numbers[0..12]: [137, 157, 156, 153, 153, 156, 153, 156, 157, 156, 152, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424766025Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"downstairs","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"[2]R generation[0..12]: [69, 81, 80, 78, 78, 80, 78, 80, 81, 80, 77, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424782875Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"downstairs","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {Wait for a query_work_queue command to finish before sending IO "msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424803274Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"downstairs","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"Max found gen is 82","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424819763Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"downstairs","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"Generation requested: 82 >= found:82","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424836132Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"downstairs","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"Next flush: 158","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424852121Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"downstairs","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424869341Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"":"downstairs","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.42488631Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424901379Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"0","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424917918Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"1","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424934068Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"client":"2","":"downstairs","session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424950407Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424965426Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"1c3fbe79-3280-4faf-9cd1-5c44222e692d is now active with session: b95ffddd-4abd-4453-85ef-39e5904749cb","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.424985545Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.425000674Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510,"session_id":"b95ffddd-4abd-4453-85ef-39e5904749cb"} {"msg":"The guest has finished waiting for activation with:82","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:42.425015904Z","hostname":"Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 391 len 36864 data: 8 11 9 8 8 10 10 10 8 02/30 Flush 03/30 Write block 262 len 16384 data: 9 6 7 7 04/30 Read block 409 len 24576 05/30 Read block 269 len 8192 06/30 Read block 228 len 16384 07/30 Read block 255 len 32768 08/30 Write block 499 len 40960 data: 7 7 8 10 8 8 9 9 9 10 09/30 Write block 556 len 4096 data: 10 10/30 Write block 193 len 4096 data: 8 11/30 Read block 147 len 36864 12/30 Write block 383 len 40960 data: 7 8 5 6 7 5 5 8 9 12 13/30 Read block 24 len 32768 14/30 Read block 437 len 36864 15/30 Write block 187 len 28672 data: 13 13 10 12 8 7 9 16/30 Read block 262 len 16384 17/30 Write block 108 len 28672 data: 5 5 5 5 4 4 6 18/30 Flush 19/30 Write block 537 len 32768 data: 7 7 7 7 8 7 7 8 20/30 Read block 267 len 12288 21/30 Write block 276 len 8192 data: 8 8 22/30 Write block 467 len 36864 data: 4 4 4 5 7 7 7 7 8 23/30 Write block 405 len 36864 data: 8 9 8 6 5 4 4 4 4 24/30 Read block 123 len 40960 25/30 Write block 459 len 20480 data: 4 6 6 5 4 26/30 Read block 218 len 12288 27/30 Write block 310 len 36864 data: 7 8 7 10 12 11 9 8 9 28/30 Write block 178 len 28672 data: 4 5 4 6 7 9 9 29/30 Read block 62 len 16384 30/30 Read block 391 len 20480 ---------------------------------------------------------------- Crucible gen:82 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 19 Acked 1018 Write 8 Done Done Done false 20 Acked 1019 Read 3 Done Done Done false 21 Acked 1020 Write 2 Done Done Done false 22 Acked 1021 Write 9 Done Done Done false 23 Acked 1022 Write 9 Done Done Done false 24 Acked 1023 Read 10 Done Done Done false 25 Acked 1024 Write 5 Done Done Done false 26 Acked 1025 Read 3 Done Done Done false 27 Acked 1026 Write 9 Done Done Done false 28 Acked 1027 Write 7 Done Done Done false 29 Acked 1028 Read 4 Done Done Done false 30 Acked 1029 Read 5 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 0 1 Done 12 11 12 35 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1017 1017 1017 Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:43:42.450879667Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1510} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 81 81 81 157 157 157 F F F 2 040-059 80 80 80 156 156 156 F F F 3 060-079 78 78 78 153 153 153 F F F 4 080-099 78 78 78 153 153 153 F F F 5 100-119 82 82 82 159 159 159 F F F 6 120-139 78 78 78 153 153 153 F F F 7 140-159 80 80 80 156 156 156 F F F 8 160-179 81 81 81 157 157 157 T T T 9 180-199 82 82 82 159 159 159 T T T 10 200-219 77 77 77 152 152 152 F F F 11 220-239 81 81 81 157 157 157 F F F 12 240-259 78 78 78 153 153 153 F F F 13 260-279 82 82 82 159 159 159 T T T 14 280-299 79 79 79 155 155 155 F F F 15 300-319 80 80 80 156 156 156 T T T 16 320-339 72 72 72 143 143 143 F F F 17 340-359 77 77 77 152 152 152 F F F 18 360-379 77 77 77 152 152 152 F F F 19 380-399 82 82 82 159 159 159 F F F 20 400-419 81 81 81 157 157 157 T T T 21 420-439 73 73 73 144 144 144 F F F 22 440-459 80 80 80 156 156 156 T T T 23 460-479 74 74 74 146 146 146 T T T 24 480-499 82 82 82 159 159 159 F F F 25 500-519 82 82 82 159 159 159 F F F 26 520-539 80 80 80 156 156 156 T T T 27 540-559 82 82 82 159 159 159 T T T 28 560-579 80 80 80 156 156 156 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 82, Max flush: 159 A Difference in extent metadata is expected here On loop 41 Verifying data now No change after verify Loop: 41 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 81 81 81 157 157 157 F F F 2 040-059 80 80 80 156 156 156 F F F 3 060-079 78 78 78 153 153 153 F F F 4 080-099 78 78 78 153 153 153 F F F 5 100-119 82 82 82 159 159 159 F F F 6 120-139 78 78 78 153 153 153 F F F 7 140-159 80 80 80 156 156 156 F F F 8 160-179 83 83 83 160 160 160 F F F 9 180-199 83 83 83 160 160 160 F F F 10 200-219 77 77 77 152 152 152 F F F 11 220-239 81 81 81 157 157 157 F F F 12 240-259 78 78 78 153 153 153 F F F 13 260-279 83 83 83 160 160 160 F F F 14 280-299 79 79 79 155 155 155 F F F 15 300-319 83 83 83 160 160 160 F F F 16 320-339 72 72 72 143 143 143 F F F 17 340-359 77 77 77 152 152 152 F F F 18 360-379 77 77 77 152 152 152 F F F 19 380-399 82 82 82 159 159 159 F F F 20 400-419 83 83 83 160 160 160 F F F 21 420-439 73 73 73 144 144 144 F F F 22 440-459 83 83 83 160 160 160 F F F 23 460-479 83 83 83 160 160 160 F F F 24 480-499 82 82 82 159 159 159 F F F 25 500-519 82 82 82 159 159 159 F F F 26 520-539 83 83 83 160 160 160 F F F 27 540-559 83 83 83 160 160 160 F F F 28 560-579 80 80 80 156 156 156 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 83, Max flush: 160 Begin loop 42 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.721709392Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.721975519Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.722015497Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518} {"msg":"Crucible 676718ed-df2d-4680-a034-55c597a52016 has session id: e4599580-b856-4139-9011-2bf3d4d20c6f","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.722289415Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"Upstairs opts: Upstairs UUID: 676718ed-df2d-4680-a034-55c597a52016, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.722333382Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"Crucible stats registered with UUID: 676718ed-df2d-4680-a034-55c597a52016","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.722357141Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:84","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.722670457Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518} {"msg":"676718ed-df2d-4680-a034-55c597a52016 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.722762653Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.722794601Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"0","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.722829429Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"0","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.722854468Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"1","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.722879757Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"1","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.722904506Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.722971463Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.723000631Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"io task","client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.72302626Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"io task","client":"0","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.723057099Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"io task","client":"1","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:43:56.723090767Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"io task","client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:43:56.723119646Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"io task","client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:43:56.723147755Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.723258699Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.723293598Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.723317856Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"io task","client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"ds_connection connected from Ok(127.0.0.1:33794)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.723340226Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"io task","client":"1","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"ds_connection connected from Ok(127.0.0.1:53948)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.723355025Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"io task","client":"0","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.72409678Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"0","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.724124149Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"0","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.724138938Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"1","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.724152838Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"1","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.724727211Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"1","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.724866634Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"0","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.725197009Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"1","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.725235337Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"1","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.725254396Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.725374751Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"0","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.725401519Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"0","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:43:56.725417289Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.724591659Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"io task","client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"ds_connection connected from Ok(127.0.0.1:58419)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.724657186Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"io task","client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.725838391Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.72586846Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.726370666Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.726903881Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.72692659Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"[0]R flush_numbers[0..12]: [137, 157, 156, 153, 153, 159, 153, 156, 160, 160, 152, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.726944559Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"downstairs","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"[0]R generation[0..12]: [69, 81, 80, 78, 78, 82, 78, 80, 83, 83, 77, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.726962369Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"downstairs","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.726980588Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"downstairs","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"[1]R flush_numbers[0..12]: [137, 157, 156, 153, 153, 159, 153, 156, 160, 160, 152, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727003136Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"downstairs","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"[1]R generation[0..12]: [69, 81, 80, 78, 78, 82, 78, 80, 83, 83, 77, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727020946Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"downstairs","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727038665Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"downstairs","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"[2]R flush_numbers[0..12]: [137, 157, 156, 153, 153, 159, 153, 156, 160, 160, 152, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727059504Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"downstairs","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"[2]R generation[0..12]: [69, 81, 80, 78, 78, 82, 78, 80, 83, 83, 77, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727087803Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"downstairs","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:44:06.727118451Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"downstairs","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"Max found gen is 84","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.72714663Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"downstairs","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"Generation requested: 84 >= found:84","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727177099Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"downstairs","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"Next flush: 161","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727199617Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"downstairs","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727216297Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"":"downstairs","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727241445Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727268264Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"0","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727298533Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"1","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727327511Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"client":"2","":"downstairs","session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.72735009Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727377309Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"676718ed-df2d-4680-a034-55c597a52016 is now active with session: e4599580-b856-4139-9011-2bf3d4d20c6f","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727409248Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727436846Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518,"session_id":"e4599580-b856-4139-9011-2bf3d4d20c6f"} {"msg":"The guest has finished waiting for activation with:84","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:06.727456215Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 351 len 36864 data: 7 9 10 11 10 9 11 11 9 03/30 Read block 472 len 8192 04/30 Write block 507 len 40960 data: 10 11 8 8 8 7 6 6 5 5 05/30 Read block 58 len 4096 06/30 Flush 07/30 Write block 496 len 8192 data: 7 10 08/30 Flush 09/30 Write block 403 len 20480 data: 7 8 9 10 9 10/30 Write block 249 len 40960 data: 8 7 6 8 7 9 9 11 11 9 11/30 Write block 204 len 16384 data: 8 7 8 9 12/30 Read block 218 len 20480 13/30 Read block 445 len 36864 14/30 Flush 15/30 Write block 459 len 4096 data: 5 16/30 Write block 199 len 32768 data: 6 6 8 8 7 9 8 9 17/30 Write block 182 len 28672 data: 8 10 10 10 11 14 14 18/30 Read block 322 len 36864 19/30 Read block 148 len 32768 20/30 Write block 504 len 20480 data: 9 10 10 11 12 21/30 Write block 266 len 12288 data: 7 8 9 22/30 Write block 313 len 16384 data: 11 13 12 10 23/30 Flush 24/30 Write block 474 len 24576 data: 8 9 9 9 6 5 25/30 Write block 200 len 36864 data: 7 9 9 8 10 9 10 10 10 26/30 Read block 182 len 20480 27/30 Write block 444 len 36864 data: 7 10 8 6 8 9 7 6 6 28/30 Read block 304 len 4096 29/30 Write block 72 len 32768 data: 4 7 6 7 8 8 7 7 30/30 Write block 72 len 36864 data: 5 8 7 8 9 9 8 8 11 ---------------------------------------------------------------- Crucible gen:84 GIO:true work queues: Upstairs:0 downstairs:7 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 25 Acked 1024 Write 6 Done Done Done false 26 Acked 1025 Write 9 Done Done Done false 27 Acked 1026 Read 5 Done Done Done false 28 Acked 1027 Write 9 Done Done Done false 29 Acked 1028 Read 1 Done Done Done false 30 Acked 1029 Write 8 Done Done Done false 31 Acked 1030 Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 6 6 6 18 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1023 1023 1023 Downstairs last five completed: 1023 1022 1021 1020 1019 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:44:06.756398855Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1518} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 81 81 81 157 157 157 F F F 2 040-059 80 80 80 156 156 156 F F F 3 060-079 78 78 78 153 153 153 T T T 4 080-099 78 78 78 153 153 153 T T T 5 100-119 82 82 82 159 159 159 F F F 6 120-139 78 78 78 153 153 153 F F F 7 140-159 80 80 80 156 156 156 F F F 8 160-179 83 83 83 160 160 160 F F F 9 180-199 84 84 84 166 166 166 F F F 10 200-219 84 84 84 165 165 165 T T T 11 220-239 81 81 81 157 157 157 F F F 12 240-259 84 84 84 164 164 164 F F F 13 260-279 84 84 84 166 166 166 F F F 14 280-299 79 79 79 155 155 155 F F F 15 300-319 84 84 84 166 166 166 F F F 16 320-339 72 72 72 143 143 143 F F F 17 340-359 84 84 84 162 162 162 F F F 18 360-379 77 77 77 152 152 152 F F F 19 380-399 82 82 82 159 159 159 F F F 20 400-419 84 84 84 164 164 164 F F F 21 420-439 73 73 73 144 144 144 F F F 22 440-459 84 84 84 165 165 165 T T T 23 460-479 83 83 83 160 160 160 T T T 24 480-499 84 84 84 163 163 163 F F F 25 500-519 84 84 84 166 166 166 F F F 26 520-539 83 83 83 160 160 160 F F F 27 540-559 83 83 83 160 160 160 F F F 28 560-579 80 80 80 156 156 156 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 84, Max flush: 166 A Difference in extent metadata is expected here On loop 42 Verifying data now No change after verify Loop: 42 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 81 81 81 157 157 157 F F F 2 040-059 80 80 80 156 156 156 F F F 3 060-079 85 85 85 167 167 167 F F F 4 080-099 85 85 85 167 167 167 F F F 5 100-119 82 82 82 159 159 159 F F F 6 120-139 78 78 78 153 153 153 F F F 7 140-159 80 80 80 156 156 156 F F F 8 160-179 83 83 83 160 160 160 F F F 9 180-199 84 84 84 166 166 166 F F F 10 200-219 85 85 85 167 167 167 F F F 11 220-239 81 81 81 157 157 157 F F F 12 240-259 84 84 84 164 164 164 F F F 13 260-279 84 84 84 166 166 166 F F F 14 280-299 79 79 79 155 155 155 F F F 15 300-319 84 84 84 166 166 166 F F F 16 320-339 72 72 72 143 143 143 F F F 17 340-359 84 84 84 162 162 162 F F F 18 360-379 77 77 77 152 152 152 F F F 19 380-399 82 82 82 159 159 159 F F F 20 400-419 84 84 84 164 164 164 F F F 21 420-439 73 73 73 144 144 144 F F F 22 440-459 85 85 85 167 167 167 F F F 23 460-479 85 85 85 167 167 167 F F F 24 480-499 84 84 84 163 163 163 F F F 25 500-519 84 84 84 166 166 166 F F F 26 520-539 83 83 83 160 160 160 F F F 27 540-559 83 83 83 160 160 160 F F F 28 560-579 80 80 80 156 156 156 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 85, Max flush: 167 Begin loop 43 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.985972066Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.986335709Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.986366297Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526} {"msg":"Crucible 8089d86e-2b58-442d-ae1b-c772810f7d2c has session id: 70df3dd2-f2ca-41f8-ab1d-9b447f4881ef","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.98651945Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"Upstairs opts: Upstairs UUID: 8089d86e-2b58-442d-ae1b-c772810f7d2c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.986563218Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"Crucible stats registered with UUID: 8089d86e-2b58-442d-ae1b-c772810f7d2c","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.986589547Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:86","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.986908172Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526} {"msg":"8089d86e-2b58-442d-ae1b-c772810f7d2c active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.987046736Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.987065055Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"0","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.987084334Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"0","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.987098153Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"1","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.987111593Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"1","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.987124622Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.987137831Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.987197118Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"io task","client":"0","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.987216917Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"io task","client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.987235007Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"io task","client":"1","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:44:20.987455926Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"io task","client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:44:20.987568051Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"io task","client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"ds_connection connected from Ok(127.0.0.1:50618)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.98760094Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"io task","client":"1","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:44:20.987630918Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.987652607Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.987673876Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"ds_connection connected from Ok(127.0.0.1:58675)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.987697515Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"io task","client":"0","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.987723424Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"io task","client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.988083657Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"0","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.988111796Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"0","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.988131855Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"1","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.988156754Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"1","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.988688089Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"1","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.988716327Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"0","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.989209564Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"0","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.989236703Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"0","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.989251163Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.989276232Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"1","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.989290621Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"1","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:20.98930377Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.988502358Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"io task","client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"ds_connection connected from Ok(127.0.0.1:53948)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.988684439Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"io task","client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.990094913Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.990122242Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.990719524Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991253009Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991276008Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"[0]R flush_numbers[0..12]: [137, 157, 156, 167, 167, 159, 153, 156, 160, 166, 167, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991290757Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"downstairs","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"[0]R generation[0..12]: [69, 81, 80, 85, 85, 82, 78, 80, 83, 84, 85, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991304977Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"downstairs","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991323256Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"downstairs","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"[1]R flush_numbers[0..12]: [137, 157, 156, 167, 167, 159, 153, 156, 160, 166, 167, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991337585Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"downstairs","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"[1]R generation[0..12]: [69, 81, 80, 85, 85, 82, 78, 80, 83, 84, 85, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991351095Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"downstairs","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991365154Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"downstairs","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"[2]R flush_numbers[0..12]: [137, 157, 156, 167, 167, 159, 153, 156, 160, 166, 167, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991379104Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"downstairs","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"[2]R generation[0..12]: [69, 81, 80, 85, 85, 82, 78, 80, 83, 84, 85, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991392763Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"downstairs","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991406692Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"downstairs","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"Max found gen is 86","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991419671Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"downstairs","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"Generation requested: 86 >= found:86","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991432971Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"downstairs","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"Next flush: 168","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:44:30.99144568Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"downstairs","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991463609Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"":"downstairs","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991476989Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991488908Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"0","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991502228Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"1","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991517907Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"client":"2","":"downstairs","session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991531266Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991543516Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"8089d86e-2b58-442d-ae1b-c772810f7d2c is now active with session: 70df3dd2-f2ca-41f8-ab1d-9b447f4881ef","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991556435Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991568304Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526,"session_id":"70df3dd2-f2ca-41f8-ab1d-9b447f4881ef"} {"msg":"The guest has finished waiting for activation with:86","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:30.991580504Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 504 len 16384 data: 10 11 11 12 02/30 Read block 397 len 16384 03/30 Read block 281 len 4096 04/30 Flush 05/30 Write block 357 len 28672 data: 12 12 10 7 4 5 4 06/30 Write block 450 len 16384 data: 8 7 7 5 07/30 Write block 320 len 36864 data: 7 6 5 6 5 5 4 5 8 08/30 Read block 117 len 4096 09/30 Read block 162 len 28672 10/30 Read block 507 len 8192 11/30 Read block 290 len 40960 12/30 Write block 340 len 16384 data: 5 7 7 7 13/30 Flush 14/30 Write block 378 len 20480 data: 10 9 8 7 7 15/30 Read block 487 len 28672 16/30 Write block 50 len 36864 data: 6 3 5 6 5 5 7 6 6 17/30 Write block 384 len 20480 data: 9 6 7 8 6 18/30 Read block 580 len 20480 19/30 Write block 191 len 28672 data: 9 8 10 7 8 8 8 20/30 Read block 560 len 40960 21/30 Read block 285 len 4096 22/30 Flush 23/30 Write block 381 len 20480 data: 8 8 8 10 7 24/30 Flush 25/30 Write block 398 len 20480 data: 11 9 8 8 8 26/30 Flush 27/30 Write block 36 len 4096 data: 7 28/30 Write block 74 len 32768 data: 8 9 10 10 9 9 12 10 29/30 Read block 391 len 40960 30/30 Read block 271 len 24576 ---------------------------------------------------------------- Crucible gen:86 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 28 Acked 1027 Write 1 Done Done Done false 29 Acked 1028 Write 8 Done Done Done false 30 Acked 1029 Read 10 Done Done Done false 31 Acked 1030 Read 6 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 0 2 Done 3 3 4 10 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1026 1026 1026 Downstairs last five completed: 1026 1025 1024 1023 1022 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:44:31.022027413Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1526} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 81 81 81 157 157 157 T T T 2 040-059 86 86 86 170 170 170 F F F 3 060-079 85 85 85 167 167 167 T T T 4 080-099 85 85 85 167 167 167 T T T 5 100-119 82 82 82 159 159 159 F F F 6 120-139 78 78 78 153 153 153 F F F 7 140-159 80 80 80 156 156 156 F F F 8 160-179 83 83 83 160 160 160 F F F 9 180-199 86 86 86 171 171 171 F F F 10 200-219 85 85 85 167 167 167 F F F 11 220-239 81 81 81 157 157 157 F F F 12 240-259 84 84 84 164 164 164 F F F 13 260-279 84 84 84 166 166 166 F F F 14 280-299 79 79 79 155 155 155 F F F 15 300-319 84 84 84 166 166 166 F F F 16 320-339 86 86 86 169 169 169 F F F 17 340-359 86 86 86 169 169 169 F F F 18 360-379 86 86 86 170 170 170 F F F 19 380-399 86 86 86 173 173 173 F F F 20 400-419 86 86 86 173 173 173 F F F 21 420-439 73 73 73 144 144 144 F F F 22 440-459 86 86 86 169 169 169 F F F 23 460-479 85 85 85 167 167 167 F F F 24 480-499 84 84 84 163 163 163 F F F 25 500-519 86 86 86 168 168 168 F F F 26 520-539 83 83 83 160 160 160 F F F 27 540-559 83 83 83 160 160 160 F F F 28 560-579 80 80 80 156 156 156 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 86, Max flush: 173 A Difference in extent metadata is expected here On loop 43 Verifying data now No change after verify Loop: 43 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 87 87 87 174 174 174 F F F 2 040-059 86 86 86 170 170 170 F F F 3 060-079 87 87 87 174 174 174 F F F 4 080-099 87 87 87 174 174 174 F F F 5 100-119 82 82 82 159 159 159 F F F 6 120-139 78 78 78 153 153 153 F F F 7 140-159 80 80 80 156 156 156 F F F 8 160-179 83 83 83 160 160 160 F F F 9 180-199 86 86 86 171 171 171 F F F 10 200-219 85 85 85 167 167 167 F F F 11 220-239 81 81 81 157 157 157 F F F 12 240-259 84 84 84 164 164 164 F F F 13 260-279 84 84 84 166 166 166 F F F 14 280-299 79 79 79 155 155 155 F F F 15 300-319 84 84 84 166 166 166 F F F 16 320-339 86 86 86 169 169 169 F F F 17 340-359 86 86 86 169 169 169 F F F 18 360-379 86 86 86 170 170 170 F F F 19 380-399 86 86 86 173 173 173 F F F 20 400-419 86 86 86 173 173 173 F F F 21 420-439 73 73 73 144 144 144 F F F 22 440-459 86 86 86 169 169 169 F F F 23 460-479 85 85 85 167 167 167 F F F 24 480-499 84 84 84 163 163 163 F F F 25 500-519 86 86 86 168 168 168 F F F 26 520-539 83 83 83 160 160 160 F F F 27 540-559 83 83 83 160 160 160 F F F 28 560-579 80 80 80 156 156 156 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 87, Max flush: 174 Begin loop 44 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.218688448Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.219048162Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.21908011Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534} {"msg":"Crucible 8eb5d606-acb0-4a1f-8359-821728039e5a has session id: 178d4972-9470-41ff-a184-2faf1a256bc0","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.219277451Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"Upstairs opts: Upstairs UUID: 8eb5d606-acb0-4a1f-8359-821728039e5a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.219321109Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"Crucible stats registered with UUID: 8eb5d606-acb0-4a1f-8359-821728039e5a","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.219345068Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:88","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.219701861Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534} {"msg":"8eb5d606-acb0-4a1f-8359-821728039e5a active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.219773228Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.219792197Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"0","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.219811836Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"0","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.219825675Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"1","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.219838925Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"1","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.219852004Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.219865364Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.219880113Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"io task","client":"0","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.219903972Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"io task","client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.219929251Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"io task","client":"1","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"ds_connection connected from Ok(127.0.0.1:49773)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.220265995Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"io task","client":"1","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"ds_connection connected from Ok(127.0.0.1:62239)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.220303353Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"io task","client":"0","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:44:45.220337971Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"io task","client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:44:45.22036886Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"io task","client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:44:45.220399989Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.220424427Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.220439197Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.22058318Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"io task","client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.221146444Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"1","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.221181462Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"1","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.221209361Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"0","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.22123587Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"0","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.221802373Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"1","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.221829432Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"0","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.222339038Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"1","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.222364337Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"1","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.222378616Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.222403855Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"0","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.222417605Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"0","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:45.222430634Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.221608843Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"io task","client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"ds_connection connected from Ok(127.0.0.1:52035)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.221947388Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"io task","client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.223148431Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.22317728Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.223715315Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.22425414Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224278819Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"[0]R flush_numbers[0..12]: [137, 174, 170, 174, 174, 159, 153, 156, 160, 171, 167, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224296638Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"downstairs","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"[0]R generation[0..12]: [69, 87, 86, 87, 87, 82, 78, 80, 83, 86, 85, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224314587Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"downstairs","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224337026Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"downstairs","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"[1]R flush_numbers[0..12]: [137, 174, 170, 174, 174, 159, 153, 156, 160, 171, 167, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224354735Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"downstairs","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"[1]R generation[0..12]: [69, 87, 86, 87, 87, 82, 78, 80, 83, 86, 85, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224371845Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"downstairs","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224389554Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"downstairs","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"[2]R flush_numbers[0..12]: [137, 174, 170, 174, 174, 159, 153, 156, 160, 171, 167, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224410893Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"downstairs","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"[2]R generation[0..12]: [69, 87, 86, 87, 87, 82, 78, 80, 83, 86, 85, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224440571Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"downstairs","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"}Wait for a query_work_queue command to finish before sending IO {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.22447403Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"downstairs","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"Max found gen is 88","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224505038Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"downstairs","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"Generation requested: 88 >= found:88","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224531677Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"downstairs","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"Next flush: 175","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224548606Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"downstairs","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224568975Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"":"downstairs","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224597944Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224625053Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"0","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224656831Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"1","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.2246759Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"client":"2","":"downstairs","session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224705779Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224729818Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"8eb5d606-acb0-4a1f-8359-821728039e5a is now active with session: 178d4972-9470-41ff-a184-2faf1a256bc0","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224750487Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224765236Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534,"session_id":"178d4972-9470-41ff-a184-2faf1a256bc0"} {"msg":"The guest has finished waiting for activation with:88","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:44:55.224786085Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 90 len 40960 data: 6 7 5 4 4 4 3 4 5 4 02/30 Read block 230 len 16384 03/30 Read block 167 len 20480 04/30 Read block 240 len 28672 05/30 Flush 06/30 Write block 418 len 24576 data: 8 11 11 8 8 7 07/30 Read block 87 len 28672 08/30 Write block 417 len 28672 data: 7 9 12 12 9 9 8 09/30 Read block 75 len 16384 10/30 Read block 181 len 8192 11/30 Read block 343 len 20480 12/30 Read block 386 len 28672 13/30 Write block 137 len 40960 data: 7 6 5 4 3 5 7 7 7 8 14/30 Read block 112 len 20480 15/30 Write block 208 len 4096 data: 11 16/30 Read block 57 len 36864 17/30 Write block 519 len 16384 data: 5 5 4 5 18/30 Read block 587 len 16384 19/30 Write block 141 len 20480 data: 4 6 8 8 8 20/30 Write block 207 len 32768 data: 11 12 9 9 8 7 5 6 21/30 Flush 22/30 Write block 516 len 40960 data: 6 4 5 6 6 5 6 6 7 5 23/30 Read block 297 len 12288 24/30 Read block 531 len 40960 25/30 Flush 26/30 Write block 188 len 36864 data: 15 11 13 10 9 11 8 9 9 27/30 Read block 182 len 40960 28/30 Read block 468 len 40960 29/30 Write block 150 len 4096 data: 10 30/30 Read block 466 len 4096 ---------------------------------------------------------------- Crucible gen:88 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 9 Done Done Done false 28 Acked 1027 Read 10 Done Done Done false 29 Acked 1028 Read 10 Done Done Done false 30 Acked 1029 Write 1 Done Done Done false 31 Acked 1030 Read 1 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 5 5 5 15 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:44:55.253991212Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1534} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 87 87 87 174 174 174 F F F 2 040-059 86 86 86 170 170 170 F F F 3 060-079 87 87 87 174 174 174 F F F 4 080-099 88 88 88 175 175 175 F F F 5 100-119 82 82 82 159 159 159 F F F 6 120-139 88 88 88 176 176 176 F F F 7 140-159 88 88 88 176 176 176 T T T 8 160-179 83 83 83 160 160 160 F F F 9 180-199 86 86 86 171 171 171 T T T 10 200-219 88 88 88 177 177 177 F F F 11 220-239 81 81 81 157 157 157 F F F 12 240-259 84 84 84 164 164 164 F F F 13 260-279 84 84 84 166 166 166 F F F 14 280-299 79 79 79 155 155 155 F F F 15 300-319 84 84 84 166 166 166 F F F 16 320-339 86 86 86 169 169 169 F F F 17 340-359 86 86 86 169 169 169 F F F 18 360-379 86 86 86 170 170 170 F F F 19 380-399 86 86 86 173 173 173 F F F 20 400-419 88 88 88 176 176 176 F F F 21 420-439 88 88 88 176 176 176 F F F 22 440-459 86 86 86 169 169 169 F F F 23 460-479 85 85 85 167 167 167 F F F 24 480-499 84 84 84 163 163 163 F F F 25 500-519 88 88 88 178 178 178 F F F 26 520-539 88 88 88 178 178 178 F F F 27 540-559 83 83 83 160 160 160 F F F 28 560-579 80 80 80 156 156 156 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 88, Max flush: 178 A Difference in extent metadata is expected here On loop 44 Verifying data now No change after verify Loop: 44 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 69 69 69 137 137 137 F F F 1 020-039 87 87 87 174 174 174 F F F 2 040-059 86 86 86 170 170 170 F F F 3 060-079 87 87 87 174 174 174 F F F 4 080-099 88 88 88 175 175 175 F F F 5 100-119 82 82 82 159 159 159 F F F 6 120-139 88 88 88 176 176 176 F F F 7 140-159 89 89 89 179 179 179 F F F 8 160-179 83 83 83 160 160 160 F F F 9 180-199 89 89 89 179 179 179 F F F 10 200-219 88 88 88 177 177 177 F F F 11 220-239 81 81 81 157 157 157 F F F 12 240-259 84 84 84 164 164 164 F F F 13 260-279 84 84 84 166 166 166 F F F 14 280-299 79 79 79 155 155 155 F F F 15 300-319 84 84 84 166 166 166 F F F 16 320-339 86 86 86 169 169 169 F F F 17 340-359 86 86 86 169 169 169 F F F 18 360-379 86 86 86 170 170 170 F F F 19 380-399 86 86 86 173 173 173 F F F 20 400-419 88 88 88 176 176 176 F F F 21 420-439 88 88 88 176 176 176 F F F 22 440-459 86 86 86 169 169 169 F F F 23 460-479 85 85 85 167 167 167 F F F 24 480-499 84 84 84 163 163 163 F F F 25 500-519 88 88 88 178 178 178 F F F 26 520-539 88 88 88 178 178 178 F F F 27 540-559 83 83 83 160 160 160 F F F 28 560-579 80 80 80 156 156 156 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 89, Max flush: 179 Begin loop 45 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.441729693Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.442098716Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.442127465Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542} {"msg":"Crucible 1d324585-73ef-4727-83c7-2142c39ed798 has session id: 8dbd3e32-827e-4edf-bf87-f0dc321b7af9","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.442319886Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"Upstairs opts: Upstairs UUID: 1d324585-73ef-4727-83c7-2142c39ed798, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.442363214Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"Crucible stats registered with UUID: 1d324585-73ef-4727-83c7-2142c39ed798","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.442384333Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:90","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.442675909Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542} {"msg":"1d324585-73ef-4727-83c7-2142c39ed798 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.442706918Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.442728677Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"0","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.442755506Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"0","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.442777515Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.442799154Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.442819583Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"2","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.442840882Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"2","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.442862241Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"io task","client":"0","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.44288483Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"io task","client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.442912858Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"io task","client":"2","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"ds_connection connected from Ok(127.0.0.1:56022)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.443009924Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"io task","client":"0","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:45:09.443051072Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"io task","client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:45:09.44309605Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"io task","client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:45:09.443129598Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"ds_connection connected from Ok(127.0.0.1:60992)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.443243973Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"io task","client":"2","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.443272141Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.443294991Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.443326409Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"io task","client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.443848515Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"2","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.443875893Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"2","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.443891283Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"0","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.443905202Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"0","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.444502734Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"2","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.444530973Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"0","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.445050249Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"2","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.445076537Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"2","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.445091127Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.445103556Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"0","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.445116875Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"0","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:09.445130335Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.443802239Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"io task","client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"ds_connection connected from Ok(127.0.0.1:56042)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.443937903Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"io task","client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.445087209Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.445121858Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.445739389Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446307762Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446338711Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"[0]R flush_numbers[0..12]: [137, 174, 170, 174, 175, 159, 176, 179, 160, 179, 177, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446374469Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"downstairs","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"[0]R generation[0..12]: [69, 87, 86, 87, 88, 82, 88, 89, 83, 89, 88, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446393098Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"downstairs","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446411278Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"downstairs","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"[1]R flush_numbers[0..12]: [137, 174, 170, 174, 175, 159, 176, 179, 160, 179, 177, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446433576Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"downstairs","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"[1]R generation[0..12]: [69, 87, 86, 87, 88, 82, 88, 89, 83, 89, 88, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446450726Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"downstairs","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446468305Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"downstairs","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"[2]R flush_numbers[0..12]: [137, 174, 170, 174, 175, 159, 176, 179, 160, 179, 177, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446485234Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"downstairs","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg"Wait for a query_work_queue command to finish before sending IO :"[2]R generation[0..12]: [69, 87, 86, 87, 88, 82, 88, 89, 83, 89, 88, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446506193Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"downstairs","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446524042Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"downstairs","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"Max found gen is 90","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446540121Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"downstairs","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"Generation requested: 90 >= found:90","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446556481Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"downstairs","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"Next flush: 180","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.44657255Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"downstairs","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446588789Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"":"downstairs","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446605238Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446620018Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"0","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446636697Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"1","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446652966Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"client":"2","":"downstairs","session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446671835Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446687134Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"1d324585-73ef-4727-83c7-2142c39ed798 is now active with session: 8dbd3e32-827e-4edf-bf87-f0dc321b7af9","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446703044Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446717863Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542,"session_id":"8dbd3e32-827e-4edf-bf87-f0dc321b7af9"} {"msg":"The guest has finished waiting for activation with:90","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:19.446733032Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 412 len 16384 data: 5 5 3 5 02/30 Write block 105 len 36864 data: 8 5 4 6 6 6 6 5 5 03/30 Read block 7 len 24576 04/30 Read block 257 len 20480 05/30 Read block 122 len 32768 06/30 Flush 07/30 Flush 08/30 Write block 1 len 28672 data: 3 3 4 4 4 5 6 09/30 Read block 202 len 20480 10/30 Read block 0 len 16384 11/30 Read block 483 len 20480 12/30 Read block 296 len 4096 13/30 Read block 100 len 24576 14/30 Write block 395 len 12288 data: 9 11 11 15/30 Read block 587 len 32768 16/30 Write block 562 len 4096 data: 8 17/30 Read block 437 len 32768 18/30 Write block 285 len 32768 data: 8 6 6 7 8 9 9 9 19/30 Write block 497 len 16384 data: 11 7 8 8 20/30 Write block 166 len 20480 data: 7 7 7 8 9 21/30 Flush 22/30 Write block 470 len 28672 data: 6 8 8 8 9 10 10 23/30 Read block 200 len 8192 24/30 Flush 25/30 Write block 412 len 24576 data: 6 6 4 6 6 8 26/30 Write block 376 len 28672 data: 8 8 11 10 9 9 9 27/30 Write block 47 len 40960 data: 7 6 7 7 4 6 7 6 6 8 28/30 Read block 537 len 40960 29/30 Read block 172 len 4096 30/30 Write block 308 len 4096 data: 8 ---------------------------------------------------------------- Crucible gen:90 GIO:true work queues: Upstairs:0 downstairs:6 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 25 Acked 1024 Write 6 Done Done Done false 26 Acked 1025 Write 7 Done Done Done false 27 Acked 1026 Write 10 Done Done Done false 28 Acked 1027 Read 10 Done Done Done false 29 Acked 1028 Read 1 Done Done Done false 30 Acked 1029 Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 5 5 5 15 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1023 1023 1023 Downstairs last five completed: 1023 1022 1021 1020 1019 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:45:19.471956166Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1542} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 182 182 182 F F F 1 020-039 87 87 87 174 174 174 F F F 2 040-059 86 86 86 170 170 170 T T T 3 060-079 87 87 87 174 174 174 F F F 4 080-099 88 88 88 175 175 175 F F F 5 100-119 90 90 90 180 180 180 F F F 6 120-139 88 88 88 176 176 176 F F F 7 140-159 89 89 89 179 179 179 F F F 8 160-179 90 90 90 182 182 182 F F F 9 180-199 89 89 89 179 179 179 F F F 10 200-219 88 88 88 177 177 177 F F F 11 220-239 81 81 81 157 157 157 F F F 12 240-259 84 84 84 164 164 164 F F F 13 260-279 84 84 84 166 166 166 F F F 14 280-299 90 90 90 182 182 182 F F F 15 300-319 84 84 84 166 166 166 T T T 16 320-339 86 86 86 169 169 169 F F F 17 340-359 86 86 86 169 169 169 F F F 18 360-379 86 86 86 170 170 170 T T T 19 380-399 90 90 90 182 182 182 T T T 20 400-419 90 90 90 180 180 180 T T T 21 420-439 88 88 88 176 176 176 F F F 22 440-459 86 86 86 169 169 169 F F F 23 460-479 90 90 90 183 183 183 F F F 24 480-499 90 90 90 182 182 182 F F F 25 500-519 90 90 90 182 182 182 F F F 26 520-539 88 88 88 178 178 178 F F F 27 540-559 83 83 83 160 160 160 F F F 28 560-579 90 90 90 182 182 182 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 90, Max flush: 183 A Difference in extent metadata is expected here On loop 45 Verifying data now No change after verify Loop: 45 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 182 182 182 F F F 1 020-039 87 87 87 174 174 174 F F F 2 040-059 91 91 91 184 184 184 F F F 3 060-079 87 87 87 174 174 174 F F F 4 080-099 88 88 88 175 175 175 F F F 5 100-119 90 90 90 180 180 180 F F F 6 120-139 88 88 88 176 176 176 F F F 7 140-159 89 89 89 179 179 179 F F F 8 160-179 90 90 90 182 182 182 F F F 9 180-199 89 89 89 179 179 179 F F F 10 200-219 88 88 88 177 177 177 F F F 11 220-239 81 81 81 157 157 157 F F F 12 240-259 84 84 84 164 164 164 F F F 13 260-279 84 84 84 166 166 166 F F F 14 280-299 90 90 90 182 182 182 F F F 15 300-319 91 91 91 184 184 184 F F F 16 320-339 86 86 86 169 169 169 F F F 17 340-359 86 86 86 169 169 169 F F F 18 360-379 91 91 91 184 184 184 F F F 19 380-399 91 91 91 184 184 184 F F F 20 400-419 91 91 91 184 184 184 F F F 21 420-439 88 88 88 176 176 176 F F F 22 440-459 86 86 86 169 169 169 F F F 23 460-479 90 90 90 183 183 183 F F F 24 480-499 90 90 90 182 182 182 F F F 25 500-519 90 90 90 182 182 182 F F F 26 520-539 88 88 88 178 178 178 F F F 27 540-559 83 83 83 160 160 160 F F F 28 560-579 90 90 90 182 182 182 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 91, Max flush: 184 Begin loop 46 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.707235107Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.707539383Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.707567741Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550} {"msg":"Crucible 5bc2cfbc-4c7b-4982-a9fb-c9483be84c68 has session id: 38f6f2ef-bd65-4903-aaaa-12624d1ade35","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.707871957Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"Upstairs opts: Upstairs UUID: 5bc2cfbc-4c7b-4982-a9fb-c9483be84c68, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.707913535Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"Crucible stats registered with UUID: 5bc2cfbc-4c7b-4982-a9fb-c9483be84c68","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.707937474Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:92","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.708281358Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550} {"msg":"5bc2cfbc-4c7b-4982-a9fb-c9483be84c68 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.708352475Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.708379514Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"0","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.708411982Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"0","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.708438691Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.70846246Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.708490378Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"2","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.708513977Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"2","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.708536566Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"io task","client":"0","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.708561655Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"io task","client":"2","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.708585184Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"io task","client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:45:33.708681979Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"io task","client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:45:33.708719718Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"io task","client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:45:33.708747256Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"ds_connection connected from Ok(127.0.0.1:58044)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.708769255Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"io task","client":"2","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.708830473Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.708859991Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"ds_connection connected from Ok(127.0.0.1:38384)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.708929698Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"io task","client":"0","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.708961086Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"io task","client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.709483302Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"2","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.709510691Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"2","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.70952623Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"0","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.709540129Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"0","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.710120862Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"2","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.710148471Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"0","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.710665587Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"0","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.710692335Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"0","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.710708445Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.710733224Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"2","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.710747533Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"2","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:33.710760482Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.71087493Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"io task","client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"ds_connection connected from Ok(127.0.0.1:59230)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.711106089Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"io task","client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.712097033Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.712124142Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.712650337Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.71322688Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713254799Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"[0]R flush_numbers[0..12]: [182, 174, 184, 174, 175, 180, 176, 179, 182, 179, 177, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713273198Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"downstairs","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"[0]R generation[0..12]: [90, 87, 91, 87, 88, 90, 88, 89, 90, 89, 88, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713290467Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"downstairs","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713308057Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"downstairs","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"[1]R flush_numbers[0..12]: [182, 174, 184, 174, 175, 180, 176, 179, 182, 179, 177, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713330385Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"downstairs","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"[1]R generation[0..12]: [90, 87, 91, 87, 88, 90, 88, 89, 90, 89, 88, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713347804Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"downstairs","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713365244Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"downstairs","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"[2]R flush_numbers[0..12]: [182, 174, 184, 174, 175, 180, 176, 179, 182, 179, 177, 157]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713382543Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"downstairs","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"[2]R generation[0..12]: [90, 87, 91, 87, 88, 90, 88, 89, 90, 89, 88, 81]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713399282Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"downstairs","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713420671Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"downstairs","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"Max found gen is 92","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.71343823Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"downstairs","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"Generation requested: 92 >= found:92","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713454859Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"downstairs","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"Next flush: 185","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713470929Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"downstairs","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713486918Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"":"downstairs","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713503337Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713518017Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"0","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713534216Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"1","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713550265Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"client":"2","":"downstairs","session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713569354Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713584713Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"5bc2cfbc-4c7b-4982-a9fb-c9483be84c68 is now active with session: 38f6f2ef-bd65-4903-aaaa-12624d1ade35","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713600483Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:43.713615072Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550,"session_id":"38f6f2ef-bd65-4903-aaaa-12624d1ade35"} {"msg":"The guest has finished waiting for activation with:92","v":0,"name":"Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 crucible","level":30,"time":"2024-07-31T18:45:43.713637771Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 161 len 12288 data: 6 4 4 02/30 Write block 350 len 8192 data: 8 8 03/30 Read block 191 len 8192 04/30 Write block 55 len 28672 data: 7 9 7 7 6 5 3 05/30 Flush 06/30 Write block 448 len 32768 data: 9 10 9 8 8 6 4 6 07/30 Read block 442 len 12288 08/30 Read block 42 len 8192 09/30 Read block 310 len 28672 10/30 Flush 11/30 Write block 479 len 36864 data: 6 5 5 5 6 8 6 5 7 12/30 Read block 286 len 36864 13/30 Read block 93 len 20480 14/30 Write block 418 len 24576 data: 10 13 13 10 10 9 15/30 Write block 371 len 36864 data: 13 9 8 7 8 9 9 12 11 16/30 Flush 17/30 Write block 239 len 16384 data: 8 7 6 7 18/30 Write block 439 len 4096 data: 7 19/30 Write block 390 len 24576 data: 9 10 13 10 9 10 20/30 Read block 403 len 20480 21/30 Write block 168 len 4096 data: 8 22/30 Write block 44 len 36864 data: 6 6 6 8 7 8 8 5 7 23/30 Write block 97 len 24576 data: 5 6 5 6 6 5 24/30 Read block 484 len 32768 25/30 Write block 441 len 40960 data: 6 8 7 8 11 9 7 10 11 10 26/30 Read block 193 len 28672 27/30 Write block 350 len 4096 data: 9 28/30 Read block 62 len 8192 29/30 Write block 406 len 20480 data: 11 10 7 6 5 30/30 Read block 581 len 28672 ---------------------------------------------------------------- Crucible gen:92 GIO:true work queues: Upstairs:0 downstairs:14 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 17 Acked 1016 Write 4 Done Done Done false 18 Acked 1017 Write 1 Done Done Done false 19 Acked 1018 Write 6 Done Done Done false 20 Acked 1019 Read 5 Done Done Done false 21 Acked 1020 Write 1 Done Done Done false 22 Acked 1021 Write 9 Done Done Done false 23 Acked 1022 Write 6 Done Done Done false 24 Acked 1023 Read 8 Done Done Done false 25 Acked 1024 Write 10 Done Done Done false 26 Acked 1025 Read 7 Done Done Done false 27 Acked 1026 Write 1 Done Done Done false 28 Acked 1027 Read 2 Done Done Done false 29 Acked 1028 Write 5 Done Done Done false 30 Acked 1029 Read 7 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 14 14 14 42 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1015 1015 1015 Downstairs last five completed: 1015 1014 1013 1012 1011 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:45:43.740481869Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1550} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 182 182 182 F F F 1 020-039 87 87 87 174 174 174 F F F 2 040-059 92 92 92 185 185 185 T T T 3 060-079 92 92 92 185 185 185 F F F 4 080-099 88 88 88 175 175 175 T T T 5 100-119 90 90 90 180 180 180 T T T 6 120-139 88 88 88 176 176 176 F F F 7 140-159 89 89 89 179 179 179 F F F 8 160-179 92 92 92 185 185 185 T T T 9 180-199 89 89 89 179 179 179 F F F 10 200-219 88 88 88 177 177 177 F F F 11 220-239 81 81 81 157 157 157 T T T 12 240-259 84 84 84 164 164 164 T T T 13 260-279 84 84 84 166 166 166 F F F 14 280-299 90 90 90 182 182 182 F F F 15 300-319 91 91 91 184 184 184 F F F 16 320-339 86 86 86 169 169 169 F F F 17 340-359 92 92 92 185 185 185 T T T 18 360-379 92 92 92 187 187 187 F F F 19 380-399 91 91 91 184 184 184 T T T 20 400-419 92 92 92 187 187 187 T T T 21 420-439 92 92 92 187 187 187 T T T 22 440-459 92 92 92 186 186 186 T T T 23 460-479 92 92 92 187 187 187 F F F 24 480-499 92 92 92 187 187 187 F F F 25 500-519 90 90 90 182 182 182 F F F 26 520-539 88 88 88 178 178 178 F F F 27 540-559 83 83 83 160 160 160 F F F 28 560-579 90 90 90 182 182 182 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 92, Max flush: 187 A Difference in extent metadata is expected here On loop 46 Verifying data now No change after verify Loop: 46 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 182 182 182 F F F 1 020-039 87 87 87 174 174 174 F F F 2 040-059 93 93 93 188 188 188 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 93 93 93 188 188 188 F F F 5 100-119 93 93 93 188 188 188 F F F 6 120-139 88 88 88 176 176 176 F F F 7 140-159 89 89 89 179 179 179 F F F 8 160-179 93 93 93 188 188 188 F F F 9 180-199 89 89 89 179 179 179 F F F 10 200-219 88 88 88 177 177 177 F F F 11 220-239 93 93 93 188 188 188 F F F 12 240-259 93 93 93 188 188 188 F F F 13 260-279 84 84 84 166 166 166 F F F 14 280-299 90 90 90 182 182 182 F F F 15 300-319 91 91 91 184 184 184 F F F 16 320-339 86 86 86 169 169 169 F F F 17 340-359 93 93 93 188 188 188 F F F 18 360-379 92 92 92 187 187 187 F F F 19 380-399 93 93 93 188 188 188 F F F 20 400-419 93 93 93 188 188 188 F F F 21 420-439 93 93 93 188 188 188 F F F 22 440-459 93 93 93 188 188 188 F F F 23 460-479 92 92 92 187 187 187 F F F 24 480-499 92 92 92 187 187 187 F F F 25 500-519 90 90 90 182 182 182 F F F 26 520-539 88 88 88 178 178 178 F F F 27 540-559 83 83 83 160 160 160 F F F 28 560-579 90 90 90 182 182 182 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 93, Max flush: 188 Begin loop 47 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.041764449Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.042531463Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.042565752Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558} {"msg":"Crucible 6c8f49a9-c5d7-437b-8ab6-ffc548fae64b has session id: e369ec76-7b6b-42f0-9e57-18dc8c01d713","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.042589811Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"Upstairs opts: Upstairs UUID: 6c8f49a9-c5d7-437b-8ab6-ffc548fae64b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.042626749Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"Crucible stats registered with UUID: 6c8f49a9-c5d7-437b-8ab6-ffc548fae64b","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.042652358Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:94","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.04281379Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558} {"msg":"6c8f49a9-c5d7-437b-8ab6-ffc548fae64b active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.042880277Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.042906606Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"0","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.042935944Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"0","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.042963353Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"1","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.042988622Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"1","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.043013511Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.04303682Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.043060369Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"io task","client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.043088387Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"io task","client":"1","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.043111396Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"io task","client":"0","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"ds_connection connected from Ok(127.0.0.1:33100)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.043448551Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"io task","client":"1","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"ds_connection connected from Ok(127.0.0.1:60864)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.043488019Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"io task","client":"0","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:45:58.043531477Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"io task","client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:45:58.043561685Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"io task","client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:45:58.043590894Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.043613873Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.043638252Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.043753286Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"io task","client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.044223124Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"0","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.044249623Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"0","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.044264482Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"1","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.044278532Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"1","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.044852795Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"1","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.04496736Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"0","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.04539616Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"1","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.045423528Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"1","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.045439418Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.045466796Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"0","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.045481136Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"0","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:45:58.045494365Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.044689296Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"io task","client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"ds_connection connected from Ok(127.0.0.1:46538)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.044793172Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"io task","client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.04589906Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.045925629Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.046435985Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047093375Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047120253Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"[0]R flush_numbers[0..12]: [182, 174, 188, 185, 188, 188, 176, 179, 188, 179, 177, 188]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047135853Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"downstairs","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"[0]R generation[0..12]: [90, 87, 93, 92, 93, 93, 88, 89, 93, 89, 88, 93]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047155792Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"downstairs","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047170801Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"downstairs","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"[1]R flush_numbers[0..12]: [182, 174, 188, 185, 188, 188, 176, 179, 188, 179, 177, 188]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.04718491Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"downstairs","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"[1]R generation[0..12]: [90, 87, 93, 92, 93, 93, 88, 89, 93, 89, 88, 93]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047198939Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"downstairs","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047213059Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"downstairs","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"[2]R flush_numbers[0..12]: [182, 174, 188, 185, 188, 188, 176, 179, 188, 179, 177, 188]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047226948Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"downstairs","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"[2]R generation[0..12]: [90, 87, 93, 92, 93, 93, 88, 89, 93, 89, 88, 93]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047240528Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"downstairs","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047254527Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"downstairs","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"Max found gen is 94","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047267686Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"downstairs","":"downstairs","Wait for a query_work_queue command to finish before sending IO session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"Generation requested: 94 >= found:94","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047284755Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"downstairs","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"Next flush: 189","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047297725Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"downstairs","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047310754Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"":"downstairs","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047324044Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047336083Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"0","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047351512Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"1","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047365102Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"client":"2","":"downstairs","session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047378461Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.04739079Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"6c8f49a9-c5d7-437b-8ab6-ffc548fae64b is now active with session: e369ec76-7b6b-42f0-9e57-18dc8c01d713","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.04740401Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047415979Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558,"session_id":"e369ec76-7b6b-42f0-9e57-18dc8c01d713"} {"msg":"The guest has finished waiting for activation with:94","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:08.047428209Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 477 len 36864 data: 10 7 7 6 6 6 7 9 7 02/30 Read block 572 len 4096 03/30 Read block 73 len 16384 04/30 Read block 314 len 32768 05/30 Flush 06/30 Write block 373 len 4096 data: 9 07/30 Write block 572 len 16384 data: 12 14 13 11 08/30 Read block 109 len 20480 09/30 Flush 10/30 Write block 524 len 32768 data: 8 6 7 6 6 5 6 7 11/30 Write block 312 len 24576 data: 8 12 14 13 11 9 12/30 Flush 13/30 Write block 513 len 12288 data: 7 7 6 14/30 Read block 376 len 24576 15/30 Write block 143 len 16384 data: 9 9 9 9 16/30 Write block 332 len 32768 data: 8 9 9 9 8 8 6 4 17/30 Write block 90 len 32768 data: 7 8 6 5 5 5 4 6 18/30 Read block 451 len 12288 19/30 Read block 137 len 20480 20/30 Read block 533 len 12288 21/30 Write block 424 len 40960 data: 6 9 10 9 10 14 16 15 14 10 22/30 Write block 81 len 32768 data: 11 10 11 13 13 10 7 5 23/30 Read block 1 len 40960 24/30 Read block 359 len 4096 25/30 Read block 19 len 24576 26/30 Read block 564 len 28672 27/30 Write block 141 len 28672 data: 5 7 10 10 10 10 7 28/30 Read block 567 len 16384 29/30 Read block 104 len 28672 30/30 Write block 247 len 36864 data: 8 8 9 8 7 9 8 10 10 ---------------------------------------------------------------- Crucible gen:94 GIO:true work queues: Upstairs:0 downstairs:14 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 18 Acked 1017 Write 8 Done Done Done false 19 Acked 1018 Read 3 Done Done Done false 20 Acked 1019 Read 5 Done Done Done false 21 Acked 1020 Read 3 Done Done Done false 22 Acked 1021 Write 10 Done Done Done false 23 Acked 1022 Write 8 Done Done Done false 24 Acked 1023 Read 10 Done Done Done false 25 Acked 1024 Read 1 Done Done Done false 26 Acked 1025 Read 6 Done Done Done false 27 Acked 1026 Read 7 Done Done Done false 28 Acked 1027 Write 7 Done Done Done false 29 Acked 1028 Read 4 Done Done Done false 30 Acked 1029 Read 7 Done Done Done false 31 Acked 1030 Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 13 13 13 39 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1016 1016 1016 Downstairs last five completed: 1016 1015 1014 1013 1012 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:46:08.075136166Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1558} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 182 182 182 F F F 1 020-039 87 87 87 174 174 174 F F F 2 040-059 93 93 93 188 188 188 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 93 93 93 188 188 188 T T T 5 100-119 93 93 93 188 188 188 F F F 6 120-139 88 88 88 176 176 176 F F F 7 140-159 94 94 94 192 192 192 T T T 8 160-179 93 93 93 188 188 188 F F F 9 180-199 89 89 89 179 179 179 F F F 10 200-219 88 88 88 177 177 177 F F F 11 220-239 93 93 93 188 188 188 F F F 12 240-259 93 93 93 188 188 188 T T T 13 260-279 84 84 84 166 166 166 F F F 14 280-299 90 90 90 182 182 182 F F F 15 300-319 94 94 94 191 191 191 F F F 16 320-339 94 94 94 192 192 192 F F F 17 340-359 93 93 93 188 188 188 F F F 18 360-379 94 94 94 190 190 190 F F F 19 380-399 93 93 93 188 188 188 F F F 20 400-419 93 93 93 188 188 188 F F F 21 420-439 93 93 93 188 188 188 T T T 22 440-459 93 93 93 188 188 188 F F F 23 460-479 94 94 94 189 189 189 F F F 24 480-499 94 94 94 189 189 189 F F F 25 500-519 94 94 94 192 192 192 F F F 26 520-539 94 94 94 191 191 191 F F F 27 540-559 83 83 83 160 160 160 F F F 28 560-579 94 94 94 190 190 190 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 94, Max flush: 192 A Difference in extent metadata is expected here On loop 47 Verifying data now No change after verify Loop: 47 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 182 182 182 F F F 1 020-039 87 87 87 174 174 174 F F F 2 040-059 93 93 93 188 188 188 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 95 95 95 193 193 193 F F F 5 100-119 93 93 93 188 188 188 F F F 6 120-139 88 88 88 176 176 176 F F F 7 140-159 95 95 95 193 193 193 F F F 8 160-179 93 93 93 188 188 188 F F F 9 180-199 89 89 89 179 179 179 F F F 10 200-219 88 88 88 177 177 177 F F F 11 220-239 93 93 93 188 188 188 F F F 12 240-259 95 95 95 193 193 193 F F F 13 260-279 84 84 84 166 166 166 F F F 14 280-299 90 90 90 182 182 182 F F F 15 300-319 94 94 94 191 191 191 F F F 16 320-339 94 94 94 192 192 192 F F F 17 340-359 93 93 93 188 188 188 F F F 18 360-379 94 94 94 190 190 190 F F F 19 380-399 93 93 93 188 188 188 F F F 20 400-419 93 93 93 188 188 188 F F F 21 420-439 95 95 95 193 193 193 F F F 22 440-459 93 93 93 188 188 188 F F F 23 460-479 94 94 94 189 189 189 F F F 24 480-499 94 94 94 189 189 189 F F F 25 500-519 94 94 94 192 192 192 F F F 26 520-539 94 94 94 191 191 191 F F F 27 540-559 83 83 83 160 160 160 F F F 28 560-579 94 94 94 190 190 190 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 95, Max flush: 193 Begin loop 48 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.287961976Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.288343068Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.288369377Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566} {"msg":"Crucible 6b270a13-3e46-4ef3-9790-b1f133977a46 has session id: 8e3a424a-bf98-4c1c-b458-531d1489378a","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.288761389Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"Upstairs opts: Upstairs UUID: 6b270a13-3e46-4ef3-9790-b1f133977a46, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.288806477Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"Crucible stats registered with UUID: 6b270a13-3e46-4ef3-9790-b1f133977a46","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.288830406Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:96","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.289105663Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566} {"msg":"6b270a13-3e46-4ef3-9790-b1f133977a46 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.289138641Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.2891633Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"0","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.289192259Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"0","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.289216858Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.289241207Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.289270815Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"2","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.289298154Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"2","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.289326793Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"io task","client":"0","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.289353471Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"io task","client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.28938112Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"io task","client":"2","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"ds_connection connected from Ok(127.0.0.1:61822)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.289533993Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"io task","client":"0","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"ds_connection connected from Ok(127.0.0.1:53809)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.289660957Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"io task","client":"2","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:46:22.289719534Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"io task","client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:46:22.289760222Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"io task","client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:46:22.289780412Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.289793571Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.2898068Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.289940154Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"io task","client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.290393523Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"2","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.290420302Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"2","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.290439291Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"0","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.29045334Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"0","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.291065041Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"2","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.2910936Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"0","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.291618516Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"2","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.291644465Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"2","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.291659564Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.291672133Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"0","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.291685663Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"0","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:22.291699062Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.291299614Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"io task","client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"ds_connection connected from Ok(127.0.0.1:54930)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.291439837Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"io task","client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.292437111Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.29246218Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293027294Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293595447Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293626775Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"[0]R flush_numbers[0..12]: [182, 174, 188, 185, 193, 188, 176, 193, 188, 179, 177, 188]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293644995Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"downstairs","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"[0]R generation[0..12]: [90, 87, 93, 92, 95, 93, 88, 95, 93, 89, 88, 93]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293662514Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"downstairs","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293680203Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"downstairs","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"[1]R flush_numbers[0..12]: [182, 174, 188, 185, 193, 188, 176, 193, 188, 179, 177, 188]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293697582Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"downstairs","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"[1]R generation[0..12]: [90, 87, 93, 92, 95, 93, 88, 95, 93, 89, 88, 93]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293714821Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"downstairs","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.29373253Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"downstairs","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"[2]R flush_numbers[0..12]: [182, 174, 188, 185, 193, 188, 176, 193, 188, 179, 177, 188]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.29374985Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"downstairs","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"[2]R generation[0..12]: [90, 87, 93, 92, 95, 93, 88, 95, 93, 89, 88, 93]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293766729Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293788718Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"downstairs","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"Max found gen is 96","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293805517Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"downstairs","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"Generation requested: 96 >= found:96","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293822206Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"downstairs","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"Next flush: 194","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293838086Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"downstairs","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293856495Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"":"downstairs","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293873464Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293888233Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"0","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293904822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"1","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293920962Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"client":"2","":"downstairs","session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293937351Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.29395247Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"6b270a13-3e46-4ef3-9790-b1f133977a46 is now active with session: 8e3a424a-bf98-4c1c-b458-531d1489378a","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.29396849Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293983269Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566,"session_id":"8e3a424a-bf98-4c1c-b458-531d1489378a"} {"msg":"The guest has finished waiting for activation with:96","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:32.293998518Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 93 len 12288 data: 6 6 6 02/30 Flush 03/30 Write block 511 len 16384 data: 9 8 8 8 04/30 Write block 213 len 16384 data: 6 7 7 7 05/30 Write block 156 len 12288 data: 6 6 6 06/30 Read block 364 len 36864 07/30 Flush 08/30 Write block 230 len 36864 data: 9 9 11 8 9 11 11 9 10 09/30 Write block 511 len 40960 data: 10 9 9 9 7 7 5 6 7 7 10/30 Read block 394 len 4096 11/30 Write block 142 len 16384 data: 8 11 11 11 12/30 Read block 192 len 4096 13/30 Write block 228 len 24576 data: 7 8 10 10 12 9 14/30 Read block 206 len 40960 15/30 Read block 51 len 32768 16/30 Read block 323 len 24576 17/30 Write block 89 len 8192 data: 6 8 18/30 Read block 499 len 20480 19/30 Write block 361 len 12288 data: 5 6 5 20/30 Write block 314 len 36864 data: 15 14 12 10 10 8 8 7 6 21/30 Write block 157 len 12288 data: 7 7 5 22/30 Write block 337 len 16384 data: 9 7 5 6 23/30 Read block 43 len 28672 24/30 Read block 127 len 16384 25/30 Read block 67 len 12288 26/30 Read block 46 len 12288 27/30 Write block 257 len 28672 data: 12 10 10 9 7 10 7 28/30 Write block 393 len 4096 data: 11 29/30 Write block 558 len 20480 data: 4 6 6 8 9 30/30 Write block 114 len 36864 data: 7 7 7 7 6 7 8 6 6 ---------------------------------------------------------------- Crucible gen:96 GIO:true work queues: Upstairs:0 downstairs:7 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 25 Acked 1024 Read 4 Done Done Done false 26 Acked 1025 Read 3 Done Done Done false 27 Acked 1026 Read 3 Done Done Done false 28 Acked 1027 Write 7 Done Done Done false 29 Acked 1028 Write 1 Done Done Done false 30 Acked 1029 Write 5 Done Done Done false 31 Acked 1030 Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 6 6 6 18 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1023 1023 1023 Downstairs last five completed: 1023 1022 1021 1020 1019 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:46:32.319493799Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1566} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 182 182 182 F F F 1 020-039 87 87 87 174 174 174 F F F 2 040-059 93 93 93 188 188 188 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 96 96 96 196 196 196 F F F 5 100-119 93 93 93 188 188 188 T T T 6 120-139 88 88 88 176 176 176 T T T 7 140-159 96 96 96 196 196 196 F F F 8 160-179 93 93 93 188 188 188 F F F 9 180-199 89 89 89 179 179 179 F F F 10 200-219 96 96 96 195 195 195 F F F 11 220-239 96 96 96 196 196 196 F F F 12 240-259 95 95 95 193 193 193 T T T 13 260-279 84 84 84 166 166 166 T T T 14 280-299 90 90 90 182 182 182 F F F 15 300-319 96 96 96 196 196 196 F F F 16 320-339 96 96 96 196 196 196 F F F 17 340-359 96 96 96 196 196 196 F F F 18 360-379 96 96 96 196 196 196 F F F 19 380-399 93 93 93 188 188 188 T T T 20 400-419 93 93 93 188 188 188 F F F 21 420-439 95 95 95 193 193 193 F F F 22 440-459 93 93 93 188 188 188 F F F 23 460-479 94 94 94 189 189 189 F F F 24 480-499 94 94 94 189 189 189 F F F 25 500-519 96 96 96 196 196 196 F F F 26 520-539 96 96 96 196 196 196 F F F 27 540-559 83 83 83 160 160 160 T T T 28 560-579 94 94 94 190 190 190 T T T 29 580-599 78 78 78 154 154 154 F F F Max gen: 96, Max flush: 196 A Difference in extent metadata is expected here On loop 48 Verifying data now No change after verify Loop: 48 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 182 182 182 F F F 1 020-039 87 87 87 174 174 174 F F F 2 040-059 93 93 93 188 188 188 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 96 96 96 196 196 196 F F F 5 100-119 97 97 97 197 197 197 F F F 6 120-139 97 97 97 197 197 197 F F F 7 140-159 96 96 96 196 196 196 F F F 8 160-179 93 93 93 188 188 188 F F F 9 180-199 89 89 89 179 179 179 F F F 10 200-219 96 96 96 195 195 195 F F F 11 220-239 96 96 96 196 196 196 F F F 12 240-259 97 97 97 197 197 197 F F F 13 260-279 97 97 97 197 197 197 F F F 14 280-299 90 90 90 182 182 182 F F F 15 300-319 96 96 96 196 196 196 F F F 16 320-339 96 96 96 196 196 196 F F F 17 340-359 96 96 96 196 196 196 F F F 18 360-379 96 96 96 196 196 196 F F F 19 380-399 97 97 97 197 197 197 F F F 20 400-419 93 93 93 188 188 188 F F F 21 420-439 95 95 95 193 193 193 F F F 22 440-459 93 93 93 188 188 188 F F F 23 460-479 94 94 94 189 189 189 F F F 24 480-499 94 94 94 189 189 189 F F F 25 500-519 96 96 96 196 196 196 F F F 26 520-539 96 96 96 196 196 196 F F F 27 540-559 97 97 97 197 197 197 F F F 28 560-579 97 97 97 197 197 197 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 97, Max flush: 197 Begin loop 49 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.568849496Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.569216639Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.569241418Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574} {"msg":"Crucible 61d8c942-3a22-454c-86e8-088cdf680891 has session id: efc27e32-92e7-4a26-b0a8-3516ee613898","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.569567733Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"Upstairs opts: Upstairs UUID: 61d8c942-3a22-454c-86e8-088cdf680891, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.569602101Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"Crucible stats registered with UUID: 61d8c942-3a22-454c-86e8-088cdf680891","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.56961818Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:98","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.569929936Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574} {"msg":"61d8c942-3a22-454c-86e8-088cdf680891 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.57004584Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.570065299Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"0","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.570086149Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"0","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.570100138Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.570113497Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.570129896Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"2","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.570143796Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"2","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.570162975Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"io task","client":"2","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.570177124Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"io task","client":"0","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.570190714Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"io task","client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:46:46.570365225Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"io task","client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:46:46.570408774Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"io task","client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:46:46.570437882Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.570460601Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"ds_connection connected from Ok(127.0.0.1:56020)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.57048397Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"io task","client":"0","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.570507099Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"ds_connection connected from Ok(127.0.0.1:47717)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.570529918Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"io task","client":"2","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.570617904Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"io task","client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.571303412Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"2","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.57134315Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"2","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.571404647Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"0","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.571430826Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"0","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.571958201Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"2","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.57198205Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"0","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.572534254Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"2","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.572560743Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"2","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.572575272Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.572587682Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"0","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.572601281Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"0","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:46.57261426Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.572599624Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"io task","client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"ds_connection connected from Ok(127.0.0.1:42957)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.572971467Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"io task","client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.573942962Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.57397105Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.574494906Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575057Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575083798Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"[0]R flush_numbers[0..12]: [182, 174, 188, 185, 196, 197, 197, 196, 188, 179, 195, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575099768Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"downstairs","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"[0]R generation[0..12]: [90, 87, 93, 92, 96, 97, 97, 96, 93, 89, 96, 96]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575114527Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"downstairs","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575129226Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"downstairs","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"[1]R flush_numbers[0..12]: [182, 174, 188, 185, 196, 197, 197, 196, 188, 179, 195, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575143136Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"downstairs","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"[1]R generation[0..12]: [90, 87, 93, 92, 96, 97, 97, 96, 93, 89, 96, 96]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575156735Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"downstairs","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575170774Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"downstairs","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"[2]R flush_numbers[0..12]: [182, 174, 188, 185, 196, 197, 197, 196, 188, 179, 195, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575184534Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"downstairs","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"[2]R generation[0..12]: [90, 87, 93, 92, 96, 97, 97, 96, 93, 89, 96, 96]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575198153Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"downstairs","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575212172Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"downstairs","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"Max found gen is 98","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575224902Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"downstairs","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"Generation requested: 98 >= found:98","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575237881Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"downstairs","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"Next flush: 198","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575250901Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"downstairs","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"All extents match","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2024-07-31T18:46:56.575284949Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"":"downstairs","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575299439Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575311388Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"0","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575324777Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"1","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575337987Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"client":"2","":"downstairs","session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575351426Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575363565Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"61d8c942-3a22-454c-86e8-088cdf680891 is now active with session: efc27e32-92e7-4a26-b0a8-3516ee613898","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575376325Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575388254Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574,"session_id":"efc27e32-92e7-4a26-b0a8-3516ee613898"} {"msg":"The guest has finished waiting for activation with:98","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:46:56.575400404Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 537 len 24576 data: 8 8 8 8 9 8 02/30 Write block 440 len 12288 data: 6 7 9 03/30 Write block 21 len 36864 data: 9 9 10 12 11 13 12 12 11 04/30 Write block 248 len 32768 data: 9 10 9 8 10 9 11 11 05/30 Read block 395 len 16384 06/30 Read block 290 len 24576 07/30 Write block 398 len 32768 data: 12 10 9 9 9 8 9 10 08/30 Read block 394 len 36864 09/30 Read block 5 len 8192 10/30 Read block 348 len 40960 11/30 Write block 263 len 4096 data: 8 12/30 Write block 133 len 20480 data: 7 6 5 6 8 13/30 Write block 277 len 4096 data: 9 14/30 Read block 427 len 4096 15/30 Read block 528 len 12288 16/30 Read block 502 len 12288 17/30 Read block 326 len 20480 18/30 Read block 27 len 36864 19/30 Write block 540 len 12288 data: 9 10 9 20/30 Read block 240 len 8192 21/30 Flush 22/30 Write block 381 len 20480 data: 10 10 9 11 8 23/30 Read block 14 len 28672 24/30 Write block 324 len 32768 data: 6 6 5 6 9 8 8 8 25/30 Write block 250 len 40960 data: 10 9 11 10 12 12 12 13 11 11 26/30 Write block 27 len 32768 data: 13 13 12 9 7 4 4 5 27/30 Write block 465 len 20480 data: 4 4 5 5 5 28/30 Read block 171 len 12288 29/30 Read block 396 len 20480 30/30 Write block 573 len 24576 data: 15 14 12 11 10 9 ---------------------------------------------------------------- Crucible gen:98 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 5 Done Done Done false 23 Acked 1022 Read 7 Done Done Done false 24 Acked 1023 Write 8 Done Done Done false 25 Acked 1024 Write 10 Done Done Done false 26 Acked 1025 Write 8 Done Done Done false 27 Acked 1026 Write 5 Done Done Done false 28 Acked 1027 Read 3 Done Done Done false 29 Acked 1028 Read 5 Done Done Done false 30 Acked 1029 Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 8 8 8 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:46:56.599673671Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1574} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 182 182 182 F F F 1 020-039 98 98 98 198 198 198 T T T 2 040-059 93 93 93 188 188 188 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 96 96 96 196 196 196 F F F 5 100-119 97 97 97 197 197 197 F F F 6 120-139 98 98 98 198 198 198 F F F 7 140-159 96 96 96 196 196 196 F F F 8 160-179 93 93 93 188 188 188 F F F 9 180-199 89 89 89 179 179 179 F F F 10 200-219 96 96 96 195 195 195 F F F 11 220-239 96 96 96 196 196 196 F F F 12 240-259 98 98 98 198 198 198 T T T 13 260-279 98 98 98 198 198 198 F F F 14 280-299 90 90 90 182 182 182 F F F 15 300-319 96 96 96 196 196 196 F F F 16 320-339 96 96 96 196 196 196 T T T 17 340-359 96 96 96 196 196 196 F F F 18 360-379 96 96 96 196 196 196 F F F 19 380-399 98 98 98 198 198 198 T T T 20 400-419 98 98 98 198 198 198 F F F 21 420-439 95 95 95 193 193 193 F F F 22 440-459 98 98 98 198 198 198 F F F 23 460-479 94 94 94 189 189 189 T T T 24 480-499 94 94 94 189 189 189 F F F 25 500-519 96 96 96 196 196 196 F F F 26 520-539 98 98 98 198 198 198 F F F 27 540-559 98 98 98 198 198 198 F F F 28 560-579 97 97 97 197 197 197 T T T 29 580-599 78 78 78 154 154 154 F F F Max gen: 98, Max flush: 198 A Difference in extent metadata is expected here On loop 49 Verifying data now No change after verify Loop: 49 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 182 182 182 F F F 1 020-039 99 99 99 199 199 199 F F F 2 040-059 93 93 93 188 188 188 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 96 96 96 196 196 196 F F F 5 100-119 97 97 97 197 197 197 F F F 6 120-139 98 98 98 198 198 198 F F F 7 140-159 96 96 96 196 196 196 F F F 8 160-179 93 93 93 188 188 188 F F F 9 180-199 89 89 89 179 179 179 F F F 10 200-219 96 96 96 195 195 195 F F F 11 220-239 96 96 96 196 196 196 F F F 12 240-259 99 99 99 199 199 199 F F F 13 260-279 98 98 98 198 198 198 F F F 14 280-299 90 90 90 182 182 182 F F F 15 300-319 96 96 96 196 196 196 F F F 16 320-339 99 99 99 199 199 199 F F F 17 340-359 96 96 96 196 196 196 F F F 18 360-379 96 96 96 196 196 196 F F F 19 380-399 99 99 99 199 199 199 F F F 20 400-419 98 98 98 198 198 198 F F F 21 420-439 95 95 95 193 193 193 F F F 22 440-459 98 98 98 198 198 198 F F F 23 460-479 99 99 99 199 199 199 F F F 24 480-499 94 94 94 189 189 189 F F F 25 500-519 96 96 96 196 196 196 F F F 26 520-539 98 98 98 198 198 198 F F F 27 540-559 98 98 98 198 198 198 F F F 28 560-579 99 99 99 199 199 199 F F F 29 580-599 78 78 78 154 154 154 F F F Max gen: 99, Max flush: 199 Begin loop 50 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.832940237Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.833293351Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.833324679Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582} {"msg":"Crucible 0b0e2f1e-7489-4bd2-8842-9ada7d8f340c has session id: bdaa31fd-620a-49d1-907c-76ba435b1329","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.833637034Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"Upstairs opts: Upstairs UUID: 0b0e2f1e-7489-4bd2-8842-9ada7d8f340c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.833680552Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"Crucible stats registered with UUID: 0b0e2f1e-7489-4bd2-8842-9ada7d8f340c","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.833703411Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:100","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.834042296Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582} {"msg":"0b0e2f1e-7489-4bd2-8842-9ada7d8f340c active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.834144591Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.83416659Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"0","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.834202228Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"0","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.834239126Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.834269515Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.834293184Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"2","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.834314673Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"2","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.834336932Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"io task","client":"0","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.834360331Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"io task","client":"2","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.83438421Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"io task","client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"ds_connection connected from Ok(127.0.0.1:64103)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.834581601Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"io task","client":"0","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"ds_connection connected from Ok(127.0.0.1:54066)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.834850658Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"io task","client":"2","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:47:10.834943214Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"io task","client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:47:10.834979852Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"io task","client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:47:10.835007651Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.835030289Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.835053799Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.835075678Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"io task","client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.835302007Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"0","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.835329146Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"0","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.835432631Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"2","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.83545116Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"2","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.835810303Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"0","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.835895289Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"2","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.836389516Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"2","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.836428404Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"2","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.836458173Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.836475742Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"0","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.836489641Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"0","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:10.836503171Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.836283515Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"io task","client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"ds_connection connected from Ok(127.0.0.1:61706)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.836647707Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"io task","client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.837837282Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.837863811Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.838404065Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.838934371Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.838961279Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"[0]R flush_numbers[0..12]: [182, 199, 188, 185, 196, 197, 198, 196, 188, 179, 195, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.838977469Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"downstairs","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"[0]R generation[0..12]: [90, 99, 93, 92, 96, 97, 98, 96, 93, 89, 96, 96]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.838992038Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"downstairs","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839006297Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"downstairs","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"[1]R flush_numbers[0..12]: [182, 199, 188, 185, 196, 197, 198, 196, 188, 179, 195, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839020627Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"downstairs","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"[1]R generation[0..12]: [90, 99, 93, 92, 96, 97, 98, 96, 93, 89, 96, 96]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839034216Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"downstairs","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839048326Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"downstairs","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"[2]R flush_numbers[0..12]: [182, 199, 188, 185, 196, 197, 198, 196, 188, 179, 195, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839062315Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"downstairs","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"[2]R generation[0..12]: [90, 99, 93, 92, 96, 97, 98, 96, 93, 89, 96, 96]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839076124Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"downstairs","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839090344Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"downstairs","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"Max found gen is 100","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839103423Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"downstairs","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"Wait for a query_work_queue command to finish before sending IO Generation requested: 100 >= found:100","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839122722Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"downstairs","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"Next flush: 200","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839135731Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"downstairs","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839148521Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"":"downstairs","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.8391616Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.83917351Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"0","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839186999Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"1","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839200368Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"client":"2","":"downstairs","session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839213828Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839226017Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"0b0e2f1e-7489-4bd2-8842-9ada7d8f340c is now active with session: bdaa31fd-620a-49d1-907c-76ba435b1329","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839238647Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839250536Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582,"session_id":"bdaa31fd-620a-49d1-907c-76ba435b1329"} {"msg":"The guest has finished waiting for activation with:100","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:20.839262896Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 303 len 20480 data: 9 8 7 7 7 02/30 Read block 504 len 12288 03/30 Write block 48 len 24576 data: 8 9 9 6 8 8 04/30 Write block 274 len 40960 data: 6 8 9 10 8 10 11 7 7 6 05/30 Write block 314 len 28672 data: 16 15 13 11 11 9 9 06/30 Read block 173 len 12288 07/30 Write block 441 len 20480 data: 8 10 8 9 12 08/30 Read block 87 len 8192 09/30 Write block 375 len 12288 data: 9 10 10 10/30 Flush 11/30 Write block 430 len 24576 data: 17 16 15 11 8 10 12/30 Read block 284 len 8192 13/30 Read block 177 len 8192 14/30 Write block 565 len 36864 data: 11 12 11 10 7 10 12 13 16 15/30 Write block 578 len 28672 data: 10 8 8 8 10 8 9 16/30 Read block 386 len 20480 17/30 Write block 244 len 4096 data: 5 18/30 Read block 563 len 12288 19/30 Read block 158 len 16384 20/30 Flush 21/30 Write block 444 len 20480 data: 10 13 10 8 11 22/30 Read block 511 len 32768 23/30 Write block 11 len 12288 data: 7 5 3 24/30 Write block 592 len 4096 data: 6 25/30 Read block 187 len 32768 26/30 Flush 27/30 Write block 147 len 16384 data: 8 10 10 11 28/30 Read block 416 len 4096 29/30 Write block 544 len 28672 data: 9 7 7 7 8 7 9 30/30 Write block 510 len 8192 data: 9 11 ---------------------------------------------------------------- Crucible gen:100 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 4 Done Done Done false 28 Acked 1027 Read 1 Done Done Done false 29 Acked 1028 Write 7 Sent Sent Sent false 30 Acked 1029 Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 2 2 2 6 Done 2 2 2 6 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:47:20.866093473Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1582} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 100 100 100 202 202 202 F F F 1 020-039 99 99 99 199 199 199 F F F 2 040-059 100 100 100 200 200 200 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 96 96 96 196 196 196 F F F 5 100-119 97 97 97 197 197 197 F F F 6 120-139 98 98 98 198 198 198 F F F 7 140-159 96 96 96 196 196 196 T T T 8 160-179 93 93 93 188 188 188 F F F 9 180-199 89 89 89 179 179 179 F F F 10 200-219 96 96 96 195 195 195 F F F 11 220-239 96 96 96 196 196 196 F F F 12 240-259 100 100 100 201 201 201 F F F 13 260-279 100 100 100 200 200 200 F F F 14 280-299 100 100 100 200 200 200 F F F 15 300-319 100 100 100 200 200 200 F F F 16 320-339 100 100 100 200 200 200 F F F 17 340-359 96 96 96 196 196 196 F F F 18 360-379 100 100 100 200 200 200 F F F 19 380-399 99 99 99 199 199 199 F F F 20 400-419 98 98 98 198 198 198 F F F 21 420-439 100 100 100 201 201 201 F F F 22 440-459 100 100 100 202 202 202 F F F 23 460-479 99 99 99 199 199 199 F F F 24 480-499 94 94 94 189 189 189 F F F 25 500-519 96 96 96 196 196 196 T T T 26 520-539 98 98 98 198 198 198 F F F 27 540-559 98 98 98 198 198 198 T T T 28 560-579 100 100 100 201 201 201 F F F 29 580-599 100 100 100 202 202 202 F F F Max gen: 100, Max flush: 202 A Difference in extent metadata is expected here On loop 50 Verifying data now No change after verify Loop: 50 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 100 100 100 202 202 202 F F F 1 020-039 99 99 99 199 199 199 F F F 2 040-059 100 100 100 200 200 200 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 96 96 96 196 196 196 F F F 5 100-119 97 97 97 197 197 197 F F F 6 120-139 98 98 98 198 198 198 F F F 7 140-159 101 101 101 203 203 203 F F F 8 160-179 93 93 93 188 188 188 F F F 9 180-199 89 89 89 179 179 179 F F F 10 200-219 96 96 96 195 195 195 F F F 11 220-239 96 96 96 196 196 196 F F F 12 240-259 100 100 100 201 201 201 F F F 13 260-279 100 100 100 200 200 200 F F F 14 280-299 100 100 100 200 200 200 F F F 15 300-319 100 100 100 200 200 200 F F F 16 320-339 100 100 100 200 200 200 F F F 17 340-359 96 96 96 196 196 196 F F F 18 360-379 100 100 100 200 200 200 F F F 19 380-399 99 99 99 199 199 199 F F F 20 400-419 98 98 98 198 198 198 F F F 21 420-439 100 100 100 201 201 201 F F F 22 440-459 100 100 100 202 202 202 F F F 23 460-479 99 99 99 199 199 199 F F F 24 480-499 94 94 94 189 189 189 F F F 25 500-519 101 101 101 203 203 203 F F F 26 520-539 98 98 98 198 198 198 F F F 27 540-559 101 101 101 203 203 203 F F F 28 560-579 100 100 100 201 201 201 F F F 29 580-599 100 100 100 202 202 202 F F F Max gen: 101, Max flush: 203 Begin loop 51 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.065108328Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.065530128Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.065559377Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590} {"msg":"Crucible 54ae99d1-91ff-421b-bc4d-d56a2ac7812d has session id: 4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.065797726Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"Upstairs opts: Upstairs UUID: 54ae99d1-91ff-421b-bc4d-d56a2ac7812d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.065942039Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"Crucible stats registered with UUID: 54ae99d1-91ff-421b-bc4d-d56a2ac7812d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.065971678Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:102","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.066248945Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590} {"msg":"54ae99d1-91ff-421b-bc4d-d56a2ac7812d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.0663472Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.066377209Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"0","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.066396978Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"0","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.066411237Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"1","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.066429237Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"1","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.066442956Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.066456605Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.066469735Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"io task","client":"0","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.066483554Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"io task","client":"1","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.066496963Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"io task","client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"ds_connection connected from Ok(127.0.0.1:35628)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.066661926Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"io task","client":"0","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:47:35.066837087Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"io task","client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:47:35.066875796Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"io task","client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"ds_connection connected from Ok(127.0.0.1:54895)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.066898955Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"io task","client":"1","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:47:35.066915904Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.066929323Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.066942673Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.067122924Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"io task","client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.067650709Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"1","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.067675058Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"1","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.067689108Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"0","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.067702617Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"0","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.06827801Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"0","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.068306319Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"1","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.068854213Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"0","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.068878782Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"0","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.068892882Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.068905351Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"1","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.06891892Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"1","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:35.06893244Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.068453815Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"io task","client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"ds_connection connected from Ok(127.0.0.1:52310)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.068582779Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"io task","client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.069470108Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.069504846Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070042131Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070645503Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070674992Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"[0]R flush_numbers[0..12]: [202, 199, 200, 185, 196, 197, 198, 203, 188, 179, 195, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070693761Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"downstairs","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"[0]R generation[0..12]: [100, 99, 100, 92, 96, 97, 98, 101, 93, 89, 96, 96]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.07071121Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"downstairs","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070729089Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"downstairs","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"[1]R flush_numbers[0..12]: [202, 199, 200, 185, 196, 197, 198, 203, 188, 179, 195, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070746348Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"downstairs","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"[1]R generation[0..12]: [100, 99, 100, 92, 96, 97, 98, 101, 93, 89, 96, 96]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070763418Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"downstairs","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070780887Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"downstairs","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"[2]R flush_numbers[0..12]: [202, 199, 200, 185, 196, 197, 198, 203, 188, 179, 195, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070798026Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"downstairs","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"[2]R generation[0..12]: [100, 99, 100, 92, 96, 97, 98, 101, 93, 89, 96, 96]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070814905Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"downstairs","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070832454Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"downstairs","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"Max found gen is 102","v":0,"name":"crucible","level":30,Wait for a query_work_queue command to finish before sending IO "time":"2024-07-31T18:47:45.070848614Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"downstairs","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"Generation requested: 102 >= found:102","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070874772Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"downstairs","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"Next flush: 204","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070891622Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"downstairs","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070907781Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"":"downstairs","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.07092444Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070939399Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"0","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070955929Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"1","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070972298Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"client":"2","":"downstairs","session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.070988797Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.071003886Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"54ae99d1-91ff-421b-bc4d-d56a2ac7812d is now active with session: 4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.071020126Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.071034825Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590,"session_id":"4ea09a7f-e4e7-4dc3-96b9-c79b2dce6222"} {"msg":"The guest has finished waiting for activation with:102","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:45.071049974Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 263 len 12288 data: 9 8 8 02/30 Read block 388 len 20480 03/30 Flush 04/30 Write block 289 len 16384 data: 9 10 10 10 05/30 Write block 11 len 36864 data: 8 6 4 3 4 4 6 6 5 06/30 Read block 26 len 24576 07/30 Read block 391 len 4096 08/30 Read block 162 len 8192 09/30 Write block 96 len 36864 data: 5 7 7 6 7 7 6 6 6 10/30 Write block 142 len 4096 data: 9 11/30 Read block 171 len 16384 12/30 Flush 13/30 Write block 243 len 32768 data: 6 6 6 7 9 10 11 11 14/30 Flush 15/30 Write block 90 len 24576 data: 9 9 7 7 7 7 16/30 Read block 541 len 28672 17/30 Write block 518 len 8192 data: 7 8 18/30 Read block 507 len 12288 19/30 Write block 297 len 40960 data: 5 4 5 5 8 10 10 9 8 8 20/30 Read block 145 len 36864 21/30 Read block 423 len 28672 22/30 Read block 186 len 12288 23/30 Read block 109 len 20480 24/30 Write block 134 len 16384 data: 7 6 7 9 25/30 Write block 474 len 4096 data: 10 26/30 Flush 27/30 Write block 262 len 36864 data: 11 10 9 9 8 9 10 8 6 28/30 Write block 441 len 20480 data: 9 11 9 11 14 29/30 Write block 179 len 8192 data: 6 5 30/30 Write block 466 len 16384 data: 5 6 6 6 ---------------------------------------------------------------- Crucible gen:102 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 28 Acked 1027 Write 9 Done Done Done false 29 Acked 1028 Write 5 Done Done Done false 30 Acked 1029 Write 2 Done Done Sent false 31 Acked 1030 Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 2 4 Done 3 3 2 8 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1026 1026 1026 Downstairs last five completed: 1026 1025 1024 1023 1022 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:47:45.098680645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1590} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 205 205 205 F F F 1 020-039 99 99 99 199 199 199 F F F 2 040-059 100 100 100 200 200 200 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 102 102 102 207 207 207 F F F 5 100-119 102 102 102 205 205 205 F F F 6 120-139 102 102 102 208 208 208 F F F 7 140-159 102 102 102 205 205 205 F F F 8 160-179 93 93 93 188 188 188 T T T 9 180-199 89 89 89 179 179 179 T T T 10 200-219 96 96 96 195 195 195 F F F 11 220-239 96 96 96 196 196 196 F F F 12 240-259 102 102 102 206 206 206 F F F 13 260-279 102 102 102 204 204 204 T T T 14 280-299 102 102 102 208 208 208 F F F 15 300-319 102 102 102 208 208 208 F F F 16 320-339 100 100 100 200 200 200 F F F 17 340-359 96 96 96 196 196 196 F F F 18 360-379 100 100 100 200 200 200 F F F 19 380-399 99 99 99 199 199 199 F F F 20 400-419 98 98 98 198 198 198 F F F 21 420-439 100 100 100 201 201 201 F F F 22 440-459 100 100 100 202 202 202 T T T 23 460-479 102 102 102 208 208 208 T T T 24 480-499 94 94 94 189 189 189 F F F 25 500-519 102 102 102 207 207 207 F F F 26 520-539 98 98 98 198 198 198 F F F 27 540-559 101 101 101 203 203 203 F F F 28 560-579 100 100 100 201 201 201 F F F 29 580-599 100 100 100 202 202 202 F F F Max gen: 102, Max flush: 208 A Difference in extent metadata is expected here On loop 51 Verifying data now No change after verify Loop: 51 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 205 205 205 F F F 1 020-039 99 99 99 199 199 199 F F F 2 040-059 100 100 100 200 200 200 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 102 102 102 207 207 207 F F F 5 100-119 102 102 102 205 205 205 F F F 6 120-139 102 102 102 208 208 208 F F F 7 140-159 102 102 102 205 205 205 F F F 8 160-179 103 103 103 209 209 209 F F F 9 180-199 103 103 103 209 209 209 F F F 10 200-219 96 96 96 195 195 195 F F F 11 220-239 96 96 96 196 196 196 F F F 12 240-259 102 102 102 206 206 206 F F F 13 260-279 103 103 103 209 209 209 F F F 14 280-299 102 102 102 208 208 208 F F F 15 300-319 102 102 102 208 208 208 F F F 16 320-339 100 100 100 200 200 200 F F F 17 340-359 96 96 96 196 196 196 F F F 18 360-379 100 100 100 200 200 200 F F F 19 380-399 99 99 99 199 199 199 F F F 20 400-419 98 98 98 198 198 198 F F F 21 420-439 100 100 100 201 201 201 F F F 22 440-459 103 103 103 209 209 209 F F F 23 460-479 103 103 103 209 209 209 F F F 24 480-499 94 94 94 189 189 189 F F F 25 500-519 102 102 102 207 207 207 F F F 26 520-539 98 98 98 198 198 198 F F F 27 540-559 101 101 101 203 203 203 F F F 28 560-579 100 100 100 201 201 201 F F F 29 580-599 100 100 100 202 202 202 F F F Max gen: 103, Max flush: 209 Begin loop 52 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.321971907Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.322434806Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.322467064Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598} {"msg":"Crucible e3aa43ed-14de-44ce-b806-4bcab0d11737 has session id: 05cc8de7-406b-4bfe-b41b-7d685f7e432d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.322709123Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"Upstairs opts: Upstairs UUID: e3aa43ed-14de-44ce-b806-4bcab0d11737, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.322754231Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"Crucible stats registered with UUID: e3aa43ed-14de-44ce-b806-4bcab0d11737","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.32277795Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:104","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.323094275Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598} {"msg":"e3aa43ed-14de-44ce-b806-4bcab0d11737 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.323163432Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.323182331Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"0","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.32320308Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"0","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.323217009Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"1","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.323241108Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"1","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.323267137Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.323282736Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.323295795Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"io task","client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.323309375Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"io task","client":"0","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.323322864Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"io task","client":"1","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"ds_connection connected from Ok(127.0.0.1:47343)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.323613361Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"io task","client":"1","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"ds_connection connected from Ok(127.0.0.1:54891)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.323795052Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"io task","client":"0","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:47:59.32383976Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"io task","client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:47:59.323866759Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"io task","client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:47:59.323946715Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.323968694Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.323982523Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.324109688Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"io task","client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.324417723Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"0","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.324458331Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"0","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.324590085Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"1","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.324617914Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"1","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.324961938Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"0","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.325071812Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"1","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.325520342Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"0","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.325542781Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"0","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.32555687Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.325617967Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"1","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.325636126Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"1","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:47:59.325649336Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.325065636Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"io task","client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"ds_connection connected from Ok(127.0.0.1:65530)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.325285436Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"io task","client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.326551447Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.326585805Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327101671Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327680174Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327700983Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"[0]R flush_numbers[0..12]: [205, 199, 200, 185, 207, 205, 208, 205, 209, 209, 195, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327715983Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"downstairs","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"[0]R generation[0..12]: [102, 99, 100, 92, 102, 102, 102, 102, 103, 103, 96, 96]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327730302Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"downstairs","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327744801Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"downstairs","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"[1]R flush_numbers[0..12]: [205, 199, 200, 185, 207, 205, 208, 205, 209, 209, 195, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327758781Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"downstairs","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"[1]R generation[0..12]: [102, 99, 100, 92, 102, 102, 102, 102, 103, 103, 96, 96]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.32777257Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"downstairs","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327787199Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"downstairs","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"[2]R flush_numbers[0..12]: [205, 199, 200, 185, 207, 205, 208, 205, 209, 209, 195, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327800929Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"downstairs","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"[2]R generation[0..12]: [102, 99, 100, 92, 102, 102, 102, 102, 103, 103, 96, 96]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327814658Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"downstairs","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327828767Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"downstairs","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"Max found gen is 104","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327841717Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"downstairs","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"Generation requested: 104 >= found:104","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327854796Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"downstairs","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"Next flush: 210","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327871335Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"Wait for a query_work_queue command to finish before sending IO ":"downstairs","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327888255Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"":"downstairs","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327901584Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327913724Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"0","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327926933Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"1","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327939952Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"client":"2","":"downstairs","session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327953111Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.327965131Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"e3aa43ed-14de-44ce-b806-4bcab0d11737 is now active with session: 05cc8de7-406b-4bfe-b41b-7d685f7e432d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.32797783Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.32798977Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598,"session_id":"05cc8de7-406b-4bfe-b41b-7d685f7e432d"} {"msg":"The guest has finished waiting for activation with:104","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:09.328002109Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 557 len 40960 data: 4 5 7 7 9 10 12 12 12 13 02/30 Write block 503 len 20480 data: 9 11 12 12 13 03/30 Write block 414 len 4096 data: 5 04/30 Read block 552 len 12288 05/30 Flush 06/30 Write block 233 len 36864 data: 10 10 12 12 10 11 9 8 7 07/30 Read block 322 len 40960 08/30 Write block 505 len 4096 data: 13 09/30 Read block 147 len 20480 10/30 Read block 239 len 4096 11/30 Write block 369 len 16384 data: 13 12 14 10 12/30 Write block 225 len 24576 data: 8 6 7 8 9 11 13/30 Read block 464 len 8192 14/30 Write block 45 len 40960 data: 7 7 9 9 10 10 7 9 9 7 15/30 Write block 445 len 36864 data: 15 11 9 12 12 11 9 9 7 16/30 Write block 165 len 32768 data: 7 8 8 9 9 10 9 9 17/30 Flush 18/30 Write block 564 len 36864 data: 13 13 14 12 11 8 11 13 14 19/30 Write block 458 len 16384 data: 6 6 7 7 20/30 Write block 507 len 16384 data: 14 13 9 10 21/30 Write block 115 len 40960 data: 8 8 8 7 8 9 7 7 7 5 22/30 Write block 99 len 40960 data: 7 8 8 7 7 7 9 6 5 7 23/30 Write block 384 len 16384 data: 12 9 8 9 24/30 Flush 25/30 Flush 26/30 Write block 307 len 20480 data: 8 9 9 8 9 27/30 Write block 452 len 4096 data: 10 28/30 Read block 247 len 16384 29/30 Read block 11 len 40960 30/30 Read block 216 len 28672 ---------------------------------------------------------------- Crucible gen:104 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 5 Done Done Done false 28 Acked 1027 Write 1 Done Done Done false 29 Acked 1028 Read 4 Done Done Done false 30 Acked 1029 Read 10 Done Done Done false 31 Acked 1030 Read 7 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 1 2 Done 5 4 4 13 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:48:09.357787179Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1598} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 205 205 205 F F F 1 020-039 99 99 99 199 199 199 F F F 2 040-059 104 104 104 211 211 211 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 104 104 104 213 213 213 F F F 5 100-119 104 104 104 213 213 213 F F F 6 120-139 104 104 104 213 213 213 F F F 7 140-159 102 102 102 205 205 205 F F F 8 160-179 104 104 104 211 211 211 F F F 9 180-199 103 103 103 209 209 209 F F F 10 200-219 96 96 96 195 195 195 F F F 11 220-239 104 104 104 211 211 211 F F F 12 240-259 104 104 104 211 211 211 F F F 13 260-279 103 103 103 209 209 209 F F F 14 280-299 102 102 102 208 208 208 F F F 15 300-319 102 102 102 208 208 208 T T T 16 320-339 100 100 100 200 200 200 F F F 17 340-359 96 96 96 196 196 196 F F F 18 360-379 104 104 104 211 211 211 F F F 19 380-399 104 104 104 213 213 213 F F F 20 400-419 104 104 104 210 210 210 F F F 21 420-439 100 100 100 201 201 201 F F F 22 440-459 104 104 104 212 212 212 T T T 23 460-479 104 104 104 212 212 212 F F F 24 480-499 94 94 94 189 189 189 F F F 25 500-519 104 104 104 213 213 213 F F F 26 520-539 98 98 98 198 198 198 F F F 27 540-559 104 104 104 210 210 210 F F F 28 560-579 104 104 104 212 212 212 F F F 29 580-599 100 100 100 202 202 202 F F F Max gen: 104, Max flush: 213 A Difference in extent metadata is expected here On loop 52 Verifying data now No change after verify Loop: 52 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 205 205 205 F F F 1 020-039 99 99 99 199 199 199 F F F 2 040-059 104 104 104 211 211 211 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 104 104 104 213 213 213 F F F 5 100-119 104 104 104 213 213 213 F F F 6 120-139 104 104 104 213 213 213 F F F 7 140-159 102 102 102 205 205 205 F F F 8 160-179 104 104 104 211 211 211 F F F 9 180-199 103 103 103 209 209 209 F F F 10 200-219 96 96 96 195 195 195 F F F 11 220-239 104 104 104 211 211 211 F F F 12 240-259 104 104 104 211 211 211 F F F 13 260-279 103 103 103 209 209 209 F F F 14 280-299 102 102 102 208 208 208 F F F 15 300-319 105 105 105 214 214 214 F F F 16 320-339 100 100 100 200 200 200 F F F 17 340-359 96 96 96 196 196 196 F F F 18 360-379 104 104 104 211 211 211 F F F 19 380-399 104 104 104 213 213 213 F F F 20 400-419 104 104 104 210 210 210 F F F 21 420-439 100 100 100 201 201 201 F F F 22 440-459 105 105 105 214 214 214 F F F 23 460-479 104 104 104 212 212 212 F F F 24 480-499 94 94 94 189 189 189 F F F 25 500-519 104 104 104 213 213 213 F F F 26 520-539 98 98 98 198 198 198 F F F 27 540-559 104 104 104 210 210 210 F F F 28 560-579 104 104 104 212 212 212 F F F 29 580-599 100 100 100 202 202 202 F F F Max gen: 105, Max flush: 214 Begin loop 53 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.550253611Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.550677651Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.55070536Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606} {"msg":"Crucible 5b3ba593-55e4-49a3-9543-cd58874fb7b6 has session id: 74de83e8-96b8-4a10-9d6f-a3bcb0bc002e","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.550809905Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"Upstairs opts: Upstairs UUID: 5b3ba593-55e4-49a3-9543-cd58874fb7b6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.55091819Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"Crucible stats registered with UUID: 5b3ba593-55e4-49a3-9543-cd58874fb7b6","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.550943339Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:106","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.551359809Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606} {"msg":"5b3ba593-55e4-49a3-9543-cd58874fb7b6 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.551445156Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.551463665Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"0","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.551482774Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"0","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.551501233Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.551515232Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.551528342Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"2","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.551541921Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"2","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.55155624Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"io task","client":"2","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.55157033Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"io task","client":"0","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.551583829Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"io task","client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"ds_connection connected from Ok(127.0.0.1:63508)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.551839077Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"io task","client":"0","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"ds_connection connected from Ok(127.0.0.1:64102)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.551963671Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"io task","client":"2","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:48:23.55199588Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"io task","client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:48:23.552028038Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"io task","client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:48:23.552121484Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.552154123Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.552177581Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.552283777Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"io task","client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.552675088Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"0","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.552712526Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"0","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.552740995Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"2","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.552768454Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"2","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.553437213Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"2","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.553466281Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"0","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.554004596Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"2","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.554042504Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"2","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.554070223Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.55413111Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"0","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.554157669Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"0","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:23.554172858Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.55315269Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"io task","client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"ds_connection connected from Ok(127.0.0.1:65179)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.553240425Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"io task","client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.554522966Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.554551174Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555098219Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555653473Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555674752Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"[0]R flush_numbers[0..12]: [205, 199, 211, 185, 213, 213, 213, 205, 211, 209, 195, 211]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555690021Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"downstairs","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"[0]R generation[0..12]: [102, 99, 104, 92, 104, 104, 104, 102, 104, 103, 96, 104]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.55570668Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"downstairs","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.55572493Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"downstairs","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"[1]R flush_numbers[0..12]: [205, 199, 211, 185, 213, 213, 213, 205, 211, 209, 195, 211]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555742649Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"downstairs","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"[1]R generation[0..12]: [102, 99, 104, 92, 104, 104, 104, 102, 104, 103, 96, 104]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555759878Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"downstairs","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555777417Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"downstairs","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"[2]R flush_numbers[0..12]: [205, 199, 211, 185, 213, 213, 213, 205, 211, 209, 195, 211]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555794496Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"downstairs","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"[2]R generation[0..12]: [102, 99, 104, 92, 104, 104, 104, 102, 104, 103, 96, 104]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555811375Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"downstairs","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555828854Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"downstairs","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"Max found gen is 106","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555844924Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"Generation requested: 106 >= found:106","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555868693Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"downstairs","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"Next flush: 215","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555885252Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"downstairs","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555901261Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"":"downstairs","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.55591771Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.55593282Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"0","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555949409Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"1","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555965898Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"client":"2","":"downstairs","session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555982747Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.555998307Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"5b3ba593-55e4-49a3-9543-cd58874fb7b6 is now active with session: 74de83e8-96b8-4a10-9d6f-a3bcb0bc002e","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.556014346Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.556029055Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606,"session_id":"74de83e8-96b8-4a10-9d6f-a3bcb0bc002e"} {"msg":"The guest has finished waiting for activation with:106","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:33.556044104Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 191 len 12288 data: 11 10 12 03/30 Read block 481 len 32768 04/30 Read block 121 len 40960 05/30 Read block 579 len 40960 06/30 Write block 98 len 20480 data: 8 8 9 9 8 07/30 Write block 557 len 36864 data: 5 6 8 8 10 11 13 14 14 08/30 Read block 583 len 36864 09/30 Read block 457 len 28672 10/30 Read block 181 len 36864 11/30 Write block 240 len 16384 data: 9 8 8 7 12/30 Read block 33 len 12288 13/30 Write block 513 len 40960 data: 10 10 8 8 6 8 9 8 6 7 14/30 Read block 333 len 40960 15/30 Read block 494 len 4096 16/30 Flush 17/30 Flush 18/30 Write block 107 len 4096 data: 6 19/30 Read block 513 len 32768 20/30 Write block 341 len 24576 data: 8 8 8 6 6 5 21/30 Write block 285 len 12288 data: 9 7 7 22/30 Read block 491 len 4096 23/30 Write block 441 len 16384 data: 10 12 10 12 24/30 Write block 387 len 24576 data: 10 7 6 10 11 14 25/30 Read block 411 len 16384 26/30 Read block 419 len 24576 27/30 Write block 393 len 36864 data: 12 10 11 12 12 13 11 10 10 28/30 Write block 81 len 4096 data: 12 29/30 Write block 450 len 24576 data: 12 10 11 8 5 7 30/30 Read block 332 len 32768 ---------------------------------------------------------------- Crucible gen:106 GIO:true work queues: Upstairs:0 downstairs:11 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 21 Acked 1020 Write 6 Done Done Done false 22 Acked 1021 Write 3 Done Done Done false 23 Acked 1022 Read 1 Done Done Done false 24 Acked 1023 Write 4 Done Done Done false 25 Acked 1024 Write 6 Done Done Done false 26 Acked 1025 Read 4 Done Done Done false 27 Acked 1026 Read 6 Done Done Done false 28 Acked 1027 Write 9 Done Done Done false 29 Acked 1028 Write 1 Done Done Done false 30 Acked 1029 Write 6 Done Done Done false 31 Acked 1030 Read 8 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 0 1 Done 10 11 11 32 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1019 1019 1019 Downstairs last five completed: 1019 1018 1017 1016 1015 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:48:33.581067777Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1606} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 205 205 205 F F F 1 020-039 99 99 99 199 199 199 F F F 2 040-059 104 104 104 211 211 211 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 106 106 106 216 216 216 T T T 5 100-119 106 106 106 218 218 218 F F F 6 120-139 104 104 104 213 213 213 F F F 7 140-159 102 102 102 205 205 205 F F F 8 160-179 104 104 104 211 211 211 F F F 9 180-199 106 106 106 216 216 216 F F F 10 200-219 96 96 96 195 195 195 F F F 11 220-239 104 104 104 211 211 211 F F F 12 240-259 106 106 106 216 216 216 F F F 13 260-279 103 103 103 209 209 209 F F F 14 280-299 102 102 102 208 208 208 T T T 15 300-319 105 105 105 214 214 214 F F F 16 320-339 100 100 100 200 200 200 F F F 17 340-359 96 96 96 196 196 196 T T T 18 360-379 104 104 104 211 211 211 F F F 19 380-399 104 104 104 213 213 213 T T T 20 400-419 104 104 104 210 210 210 T T T 21 420-439 100 100 100 201 201 201 F F F 22 440-459 105 105 105 214 214 214 T T T 23 460-479 104 104 104 212 212 212 F F F 24 480-499 94 94 94 189 189 189 F F F 25 500-519 106 106 106 216 216 216 F F F 26 520-539 106 106 106 216 216 216 F F F 27 540-559 106 106 106 216 216 216 F F F 28 560-579 106 106 106 216 216 216 F F F 29 580-599 100 100 100 202 202 202 F F F Max gen: 106, Max flush: 218 A Difference in extent metadata is expected here On loop 53 Verifying data now No change after verify Loop: 53 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 205 205 205 F F F 1 020-039 99 99 99 199 199 199 F F F 2 040-059 104 104 104 211 211 211 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 107 107 107 219 219 219 F F F 5 100-119 106 106 106 218 218 218 F F F 6 120-139 104 104 104 213 213 213 F F F 7 140-159 102 102 102 205 205 205 F F F 8 160-179 104 104 104 211 211 211 F F F 9 180-199 106 106 106 216 216 216 F F F 10 200-219 96 96 96 195 195 195 F F F 11 220-239 104 104 104 211 211 211 F F F 12 240-259 106 106 106 216 216 216 F F F 13 260-279 103 103 103 209 209 209 F F F 14 280-299 107 107 107 219 219 219 F F F 15 300-319 105 105 105 214 214 214 F F F 16 320-339 100 100 100 200 200 200 F F F 17 340-359 107 107 107 219 219 219 F F F 18 360-379 104 104 104 211 211 211 F F F 19 380-399 107 107 107 219 219 219 F F F 20 400-419 107 107 107 219 219 219 F F F 21 420-439 100 100 100 201 201 201 F F F 22 440-459 107 107 107 219 219 219 F F F 23 460-479 104 104 104 212 212 212 F F F 24 480-499 94 94 94 189 189 189 F F F 25 500-519 106 106 106 216 216 216 F F F 26 520-539 106 106 106 216 216 216 F F F 27 540-559 106 106 106 216 216 216 F F F 28 560-579 106 106 106 216 216 216 F F F 29 580-599 100 100 100 202 202 202 F F F Max gen: 107, Max flush: 219 Begin loop 54 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.814796623Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.815321478Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.815355547Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614} {"msg":"Crucible 8d0616e6-5859-4a58-8a9d-3576773bb3f5 has session id: 809f7de5-0dec-4b1c-8019-19989decfa43","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.815420634Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"Upstairs opts: Upstairs UUID: 8d0616e6-5859-4a58-8a9d-3576773bb3f5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.815524859Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"Crucible stats registered with UUID: 8d0616e6-5859-4a58-8a9d-3576773bb3f5","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.815555357Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:108","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.8159281Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614} {"msg":"8d0616e6-5859-4a58-8a9d-3576773bb3f5 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.815964068Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.815989597Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"0","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.816044645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"0","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.816072593Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"1","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.816101542Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"1","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.816127461Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.81615418Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.816180398Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"io task","client":"1","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.816204637Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"io task","client":"0","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.816227506Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"io task","client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"ds_connection connected from Ok(127.0.0.1:47871)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.816251595Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"io task","client":"0","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:48:47.816299333Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"io task","client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:48:47.816422197Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"io task","client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:48:47.816548551Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"ds_connection connected from Ok(127.0.0.1:45815)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.81656739Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"io task","client":"1","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.816583419Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.816596809Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.816616078Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"io task","client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.817157682Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"0","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.817192011Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"0","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.81721876Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"1","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.817245418Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"1","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.81785181Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"0","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.817891788Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"1","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.818414364Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"1","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.818440463Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"1","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.818455042Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.818467472Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"0","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.818480661Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"0","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:47.81849374Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.818265792Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"io task","client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"ds_connection connected from Ok(127.0.0.1:43168)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.818633325Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"io task","client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.819555492Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.819584621Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820109466Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820703959Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820733397Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"[0]R flush_numbers[0..12]: [205, 199, 211, 185, 219, 218, 213, 205, 211, 216, 195, 211]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820752346Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"downstairs","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"[0]R generation[0..12]: [102, 99, 104, 92, 107, 106, 104, 102, 104, 106, 96, 104]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820770276Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"downstairs","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820788484Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"downstairs","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"[1]R flush_numbers[0..12]: [205, 199, 211, 185, 219, 218, 213, 205, 211, 216, 195, 211]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820805554Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"downstairs","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"[1]R generation[0..12]: [102, 99, 104, 92, 107, 106, 104, 102, 104, 106, 96, 104]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820819293Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"downstairs","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820833393Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"downstairs","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"[2]R flush_numbers[0..12]: [205, 199, 211, 185, 219, 218, 213, 205, 211, 216, 195, 211]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820847192Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"downstairs","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"[2]R generation[0..12]: [102, 99, 104, 92, 107, 106, 104, 102, 104, 106, 96, 104]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820860971Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"downstairs","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.82087526Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"downstairs","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} Wait for a query_work_queue command to finish before sending IO {"msg":"Max found gen is 108","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.82089579Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"downstairs","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"Generation requested: 108 >= found:108","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820909989Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"downstairs","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"Next flush: 220","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820923108Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"downstairs","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820936018Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"":"downstairs","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820949137Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820960927Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"0","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820974186Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"1","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.820987385Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"client":"2","":"downstairs","session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.821016124Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.821030053Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"8d0616e6-5859-4a58-8a9d-3576773bb3f5 is now active with session: 809f7de5-0dec-4b1c-8019-19989decfa43","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.821043003Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.821054702Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614,"session_id":"809f7de5-0dec-4b1c-8019-19989decfa43"} {"msg":"The guest has finished waiting for activation with:108","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:48:57.821066822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 98 len 24576 data: 9 9 10 10 9 8 02/30 Flush 03/30 Write block 128 len 36864 data: 7 6 7 7 7 8 8 7 8 04/30 Read block 232 len 32768 05/30 Read block 89 len 28672 06/30 Write block 86 len 28672 data: 11 8 6 7 10 10 8 07/30 Read block 98 len 20480 08/30 Read block 380 len 12288 09/30 Write block 232 len 40960 data: 13 11 11 13 13 11 12 10 10 9 10/30 Write block 47 len 24576 data: 10 10 11 11 8 10 11/30 Read block 199 len 24576 12/30 Read block 549 len 12288 13/30 Flush 14/30 Write block 307 len 20480 data: 9 10 10 9 10 15/30 Read block 218 len 4096 16/30 Write block 134 len 8192 data: 9 8 17/30 Write block 194 len 16384 data: 9 10 10 9 18/30 Read block 551 len 28672 19/30 Flush 20/30 Flush 21/30 Write block 185 len 28672 data: 11 12 15 16 12 14 12 22/30 Read block 236 len 28672 23/30 Write block 0 len 20480 data: 3 4 4 5 5 24/30 Write block 316 len 4096 data: 14 25/30 Read block 115 len 32768 26/30 Read block 181 len 24576 27/30 Write block 24 len 16384 data: 13 12 14 14 28/30 Write block 425 len 24576 data: 10 11 10 11 15 18 29/30 Read block 277 len 16384 30/30 Write block 175 len 8192 data: 6 5 ---------------------------------------------------------------- Crucible gen:108 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 7 Done Done Done false 23 Acked 1022 Read 7 Done Done Done false 24 Acked 1023 Write 5 Done Done Done false 25 Acked 1024 Write 1 Done Done Done false 26 Acked 1025 Read 8 Done Done Done false 27 Acked 1026 Read 6 Done Done Done false 28 Acked 1027 Write 4 Done Done Done false 29 Acked 1028 Write 6 Done Done Done false 30 Acked 1029 Read 4 Done Done Done false 31 Acked 1030 Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 9 9 9 27 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:48:57.84746918Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1614} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 205 205 205 T T T 1 020-039 99 99 99 199 199 199 T T T 2 040-059 108 108 108 221 221 221 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 108 108 108 221 221 221 F F F 5 100-119 108 108 108 220 220 220 F F F 6 120-139 108 108 108 222 222 222 F F F 7 140-159 102 102 102 205 205 205 F F F 8 160-179 104 104 104 211 211 211 T T T 9 180-199 108 108 108 222 222 222 T T T 10 200-219 96 96 96 195 195 195 F F F 11 220-239 108 108 108 221 221 221 F F F 12 240-259 108 108 108 221 221 221 F F F 13 260-279 103 103 103 209 209 209 F F F 14 280-299 107 107 107 219 219 219 F F F 15 300-319 108 108 108 222 222 222 T T T 16 320-339 100 100 100 200 200 200 F F F 17 340-359 107 107 107 219 219 219 F F F 18 360-379 104 104 104 211 211 211 F F F 19 380-399 107 107 107 219 219 219 F F F 20 400-419 107 107 107 219 219 219 F F F 21 420-439 100 100 100 201 201 201 T T T 22 440-459 107 107 107 219 219 219 F F F 23 460-479 104 104 104 212 212 212 F F F 24 480-499 94 94 94 189 189 189 F F F 25 500-519 106 106 106 216 216 216 F F F 26 520-539 106 106 106 216 216 216 F F F 27 540-559 106 106 106 216 216 216 F F F 28 560-579 106 106 106 216 216 216 F F F 29 580-599 100 100 100 202 202 202 F F F Max gen: 108, Max flush: 222 A Difference in extent metadata is expected here On loop 54 Verifying data now No change after verify Loop: 54 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 109 109 109 223 223 223 F F F 1 020-039 109 109 109 223 223 223 F F F 2 040-059 108 108 108 221 221 221 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 108 108 108 221 221 221 F F F 5 100-119 108 108 108 220 220 220 F F F 6 120-139 108 108 108 222 222 222 F F F 7 140-159 102 102 102 205 205 205 F F F 8 160-179 109 109 109 223 223 223 F F F 9 180-199 109 109 109 223 223 223 F F F 10 200-219 96 96 96 195 195 195 F F F 11 220-239 108 108 108 221 221 221 F F F 12 240-259 108 108 108 221 221 221 F F F 13 260-279 103 103 103 209 209 209 F F F 14 280-299 107 107 107 219 219 219 F F F 15 300-319 109 109 109 223 223 223 F F F 16 320-339 100 100 100 200 200 200 F F F 17 340-359 107 107 107 219 219 219 F F F 18 360-379 104 104 104 211 211 211 F F F 19 380-399 107 107 107 219 219 219 F F F 20 400-419 107 107 107 219 219 219 F F F 21 420-439 109 109 109 223 223 223 F F F 22 440-459 107 107 107 219 219 219 F F F 23 460-479 104 104 104 212 212 212 F F F 24 480-499 94 94 94 189 189 189 F F F 25 500-519 106 106 106 216 216 216 F F F 26 520-539 106 106 106 216 216 216 F F F 27 540-559 106 106 106 216 216 216 F F F 28 560-579 106 106 106 216 216 216 F F F 29 580-599 100 100 100 202 202 202 F F F Max gen: 109, Max flush: 223 Begin loop 55 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.080809402Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.081313098Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.081343097Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622} {"msg":"Crucible 36ffe9b6-e0ef-41c2-b356-e2fca2679fa1 has session id: 9ba82caa-6c94-4b18-ac87-57905eb106f5","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.081615504Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"Upstairs opts: Upstairs UUID: 36ffe9b6-e0ef-41c2-b356-e2fca2679fa1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.081654162Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"Crucible stats registered with UUID: 36ffe9b6-e0ef-41c2-b356-e2fca2679fa1","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.081671551Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:110","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.081949538Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622} {"msg":"36ffe9b6-e0ef-41c2-b356-e2fca2679fa1 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.082015865Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.082037924Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"0","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.082057333Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"0","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.082071163Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"1","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.082085652Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"1","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.082104551Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.08212893Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.082148489Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"io task","client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.082163468Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"io task","client":"0","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.082177008Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"io task","client":"1","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:47237)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.082351259Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"io task","client":"0","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:49:12.082390118Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"io task","client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:49:12.082418046Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"io task","client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:49:12.082446955Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.082471164Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.082494453Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:62144)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.082577729Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"io task","client":"1","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.082606208Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"io task","client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.083097355Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"0","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.083125023Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"0","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.083139443Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"1","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.083153142Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"1","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.083607211Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"1","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.083710156Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"0","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.084044051Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"1","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.084080039Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"1","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.084105978Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.084193333Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"0","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.084213983Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"0","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:12.084227742Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.083654723Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"io task","client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:34223)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.084002566Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"io task","client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.084944703Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.084971341Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.085541675Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086146906Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086173805Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"[0]R flush_numbers[0..12]: [223, 223, 221, 185, 221, 220, 222, 205, 223, 223, 195, 221]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086192344Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"downstairs","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"[0]R generation[0..12]: [109, 109, 108, 92, 108, 108, 108, 102, 109, 109, 96, 108]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086210193Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"downstairs","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086228063Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"downstairs","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"[1]R flush_numbers[0..12]: [223, 223, 221, 185, 221, 220, 222, 205, 223, 223, 195, 221]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086245432Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"downstairs","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"[1]R generation[0..12]: [109, 109, 108, 92, 108, 108, 108, 102, 109, 109, 96, 108]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086262601Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"downstairs","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.08628018Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"downstairs","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"[2]R flush_numbers[0..12]: [223, 223, 221, 185, 221, 220, 222, 205, 223, 223, 195, 221]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086297929Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"downstairs","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"[2]R generation[0..12]: [109, 109, 108, 92, 108, 108, 108, 102, 109, 109, 96, 108]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086314919Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"downstairs","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086338058Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"downstairs","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"Max found gen is 110","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086354917Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"downstairs","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"Generation requested: 110 >= found:110","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086375656Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"downstairs","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"Next flush: 224","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086391535Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"downstairs","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086407654Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"":"downstairs","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086424463Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086439273Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"0","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086456012Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"1","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086472781Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"client":"2","":"downstairs","session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.08648954Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.08650487Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"36ffe9b6-e0ef-41c2-b356-e2fca2679fa1 is now active with session: 9ba82caa-6c94-4b18-ac87-57905eb106f5","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086521059Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086535838Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622,"session_id":"9ba82caa-6c94-4b18-ac87-57905eb106f5"} {"msg":"The guest has finished waiting for activation with:110","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:22.086550917Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 35 len 16384 data: 5 8 7 8 02/30 Write block 413 len 36864 data: 7 6 7 7 9 11 14 14 11 03/30 Write block 590 len 40960 data: 6 5 7 8 8 7 5 5 5 2 04/30 Read block 369 len 12288 05/30 Flush 06/30 Write block 86 len 24576 data: 12 9 7 8 11 11 07/30 Write block 236 len 40960 data: 14 12 13 11 11 10 9 8 7 7 08/30 Write block 472 len 36864 data: 9 9 11 11 11 11 8 8 7 09/30 Read block 129 len 12288 10/30 Read block 8 len 12288 11/30 Read block 362 len 28672 12/30 Read block 294 len 8192 13/30 Read block 346 len 36864 14/30 Read block 58 len 28672 15/30 Read block 355 len 8192 16/30 Write block 189 len 12288 data: 13 15 13 17/30 Read block 443 len 20480 18/30 Read block 70 len 20480 19/30 Read block 74 len 4096 20/30 Read block 350 len 20480 21/30 Read block 201 len 8192 22/30 Flush 23/30 Write block 92 len 40960 data: 9 8 8 8 6 8 10 10 11 11 24/30 Read block 66 len 40960 25/30 Write block 430 len 4096 data: 19 26/30 Read block 43 len 12288 27/30 Read block 404 len 16384 28/30 Write block 498 len 24576 data: 8 9 9 9 11 10 29/30 Read block 311 len 20480 30/30 Write block 212 len 4096 data: 8 ---------------------------------------------------------------- Crucible gen:110 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 23 Acked 1022 Write 10 Done Done Done false 24 Acked 1023 Read 10 Done Done Done false 25 Acked 1024 Write 1 Done Done Done false 26 Acked 1025 Read 3 Done Done Done false 27 Acked 1026 Read 4 Done Done Done false 28 Acked 1027 Write 6 Done Done Sent false 29 Acked 1028 Read 5 Done Sent Sent false 30 Acked 1029 Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 2 3 6 Done 7 6 5 18 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1021 1021 1021 Downstairs last five completed: 1021 1020 1019 1018 1017 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:49:22.112383852Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1622} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 109 109 109 223 223 223 F F F 1 020-039 110 110 110 224 224 224 F F F 2 040-059 108 108 108 221 221 221 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 110 110 110 225 225 225 T T T 5 100-119 108 108 108 220 220 220 T T T 6 120-139 108 108 108 222 222 222 F F F 7 140-159 102 102 102 205 205 205 F F F 8 160-179 109 109 109 223 223 223 F F F 9 180-199 110 110 110 225 225 225 F F F 10 200-219 96 96 96 195 195 195 T T T 11 220-239 110 110 110 225 225 225 F F F 12 240-259 110 110 110 225 225 225 F F F 13 260-279 103 103 103 209 209 209 F F F 14 280-299 107 107 107 219 219 219 F F F 15 300-319 109 109 109 223 223 223 F F F 16 320-339 100 100 100 200 200 200 F F F 17 340-359 107 107 107 219 219 219 F F F 18 360-379 104 104 104 211 211 211 F F F 19 380-399 107 107 107 219 219 219 F F F 20 400-419 110 110 110 224 224 224 F F F 21 420-439 110 110 110 224 224 224 T T T 22 440-459 107 107 107 219 219 219 F F F 23 460-479 110 110 110 225 225 225 F F F 24 480-499 110 110 110 225 225 225 T T T 25 500-519 106 106 106 216 216 216 T T T 26 520-539 106 106 106 216 216 216 F F F 27 540-559 106 106 106 216 216 216 F F F 28 560-579 106 106 106 216 216 216 F F F 29 580-599 110 110 110 224 224 224 F F F Max gen: 110, Max flush: 225 A Difference in extent metadata is expected here On loop 55 Verifying data now No change after verify Loop: 55 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 109 109 109 223 223 223 F F F 1 020-039 110 110 110 224 224 224 F F F 2 040-059 108 108 108 221 221 221 F F F 3 060-079 92 92 92 185 185 185 F F F 4 080-099 111 111 111 226 226 226 F F F 5 100-119 111 111 111 226 226 226 F F F 6 120-139 108 108 108 222 222 222 F F F 7 140-159 102 102 102 205 205 205 F F F 8 160-179 109 109 109 223 223 223 F F F 9 180-199 110 110 110 225 225 225 F F F 10 200-219 111 111 111 226 226 226 F F F 11 220-239 110 110 110 225 225 225 F F F 12 240-259 110 110 110 225 225 225 F F F 13 260-279 103 103 103 209 209 209 F F F 14 280-299 107 107 107 219 219 219 F F F 15 300-319 109 109 109 223 223 223 F F F 16 320-339 100 100 100 200 200 200 F F F 17 340-359 107 107 107 219 219 219 F F F 18 360-379 104 104 104 211 211 211 F F F 19 380-399 107 107 107 219 219 219 F F F 20 400-419 110 110 110 224 224 224 F F F 21 420-439 111 111 111 226 226 226 F F F 22 440-459 107 107 107 219 219 219 F F F 23 460-479 110 110 110 225 225 225 F F F 24 480-499 111 111 111 226 226 226 F F F 25 500-519 111 111 111 226 226 226 F F F 26 520-539 106 106 106 216 216 216 F F F 27 540-559 106 106 106 216 216 216 F F F 28 560-579 106 106 106 216 216 216 F F F 29 580-599 110 110 110 224 224 224 F F F Max gen: 111, Max flush: 226 Begin loop 56 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.343377196Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.343853394Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.343880852Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630} {"msg":"Crucible 998413bf-75ad-47a9-a56f-32fbc4d7b1c6 has session id: 6e91da4c-1083-4644-8f5e-623be9eaceec","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.344102502Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"Upstairs opts: Upstairs UUID: 998413bf-75ad-47a9-a56f-32fbc4d7b1c6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.344132341Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"Crucible stats registered with UUID: 998413bf-75ad-47a9-a56f-32fbc4d7b1c6","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.34414675Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:112","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.344473595Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630} {"msg":"998413bf-75ad-47a9-a56f-32fbc4d7b1c6 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.344544812Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.34457013Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"0","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.344591119Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"0","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.344605089Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"1","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.344618848Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"1","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.344632237Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.344646677Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.344669186Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"io task","client":"1","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.344694455Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"io task","client":"0","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.34478295Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"io task","client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"ds_connection connected from Ok(127.0.0.1:51435)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.344995351Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"io task","client":"0","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"ds_connection connected from Ok(127.0.0.1:34389)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.345031439Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"io task","client":"1","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:49:36.345174172Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"io task","client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:49:36.345207521Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"io task","client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:49:36.345373353Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.345410191Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.34543746Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.345480628Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"io task","client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.345900848Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"0","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.345927737Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"0","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.345942036Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"1","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.345955716Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"1","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.346571847Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"0","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.346600306Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"1","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.347129371Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"0","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.34715573Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"0","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.347171019Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.347183298Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"1","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.347196548Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"1","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:36.347209417Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.347189691Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"io task","client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"ds_connection connected from Ok(127.0.0.1:55709)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.347343164Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"io task","client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.34828159Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.348317468Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.348809485Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349332701Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.34935648Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"[0]R flush_numbers[0..12]: [223, 224, 221, 185, 226, 226, 222, 205, 223, 225, 226, 225]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349371789Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"downstairs","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"[0]R generation[0..12]: [109, 110, 108, 92, 111, 111, 108, 102, 109, 110, 111, 110]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349386288Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"downstairs","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349400708Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"downstairs","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"[1]R flush_numbers[0..12]: [223, 224, 221, 185, 226, 226, 222, 205, 223, 225, 226, 225]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349414737Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"downstairs","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"[1]R generation[0..12]: [109, 110, 108, 92, 111, 111, 108, 102, 109, 110, 111, 110]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349428266Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"downstairs","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349442386Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"downstairs","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"[2]R flush_numbers[0..12]: [223, 224, 221, 185, 226, 226, 222, 205, 223, 225, 226, 225]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349456335Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"downstairs","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"[2]R generation[0..12]: [109, 110, 108, 92, 111, 111, 108, 102, 109, 110, 111, 110]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349470235Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"downstairs","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349489434Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"downstairs","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"Max found gen is 112","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349502923Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"downstairs","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"Generation requested: 112 >= found:112","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:49:46.349516272Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"downstairs","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"Next flush: 227","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349532812Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"downstairs","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349546011Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"":"downstairs","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.34955911Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.34957094Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"0","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349584129Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"1","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349597379Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"client":"2","":"downstairs","session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349610468Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349622408Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"998413bf-75ad-47a9-a56f-32fbc4d7b1c6 is now active with session: 6e91da4c-1083-4644-8f5e-623be9eaceec","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349636177Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349648296Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630,"session_id":"6e91da4c-1083-4644-8f5e-623be9eaceec"} {"msg":"The guest has finished waiting for activation with:112","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:49:46.349661556Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 133 len 36864 data: 9 10 9 9 10 7 6 5 6 02/30 Read block 83 len 40960 03/30 Read block 270 len 32768 04/30 Write block 518 len 32768 data: 9 10 9 7 8 7 9 7 05/30 Read block 419 len 24576 06/30 Write block 485 len 4096 data: 8 07/30 Read block 450 len 28672 08/30 Read block 438 len 28672 09/30 Write block 565 len 40960 data: 15 15 13 12 9 12 14 15 17 15 10/30 Write block 520 len 28672 data: 10 8 9 8 10 8 8 11/30 Write block 299 len 12288 data: 6 6 9 12/30 Read block 588 len 28672 13/30 Flush 14/30 Write block 407 len 32768 data: 11 8 7 6 5 7 8 7 15/30 Flush 16/30 Write block 377 len 16384 data: 11 13 12 10 17/30 Write block 36 len 16384 data: 9 8 9 9 18/30 Read block 380 len 16384 19/30 Flush 20/30 Write block 73 len 36864 data: 9 9 10 11 11 10 10 13 13 21/30 Read block 78 len 32768 22/30 Write block 245 len 4096 data: 8 23/30 Read block 150 len 8192 24/30 Write block 99 len 40960 data: 11 12 12 10 9 8 10 7 7 8 25/30 Write block 388 len 8192 data: 8 7 26/30 Read block 9 len 28672 27/30 Write block 467 len 36864 data: 7 7 7 7 9 10 10 12 12 28/30 Write block 260 len 16384 data: 10 8 12 11 29/30 Write block 370 len 40960 data: 13 15 11 10 8 10 11 12 14 13 30/30 Read block 29 len 8192 ---------------------------------------------------------------- Crucible gen:112 GIO:true work queues: Upstairs:0 downstairs:11 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 20 Acked 1019 Write 9 Done Done Done false 21 Acked 1020 Read 8 Done Done Done false 22 Acked 1021 Write 1 Done Done Done false 23 Acked 1022 Read 2 Done Done Done false 24 Acked 1023 Write 10 Done Done Done false 25 Acked 1024 Write 2 Done Done Done false 26 Acked 1025 Read 7 Done Done Done false 27 Acked 1026 Write 9 Done Done Done false 28 Acked 1027 Write 4 Done Done Done false 29 Acked 1028 Write 10 Done Done Done false 30 Acked 1029 Read 2 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 11 11 11 33 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1018 1018 Downstairs last five completed: 1018 1017 1016 1015 1014 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:49:46.375364206Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1630} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 109 109 109 223 223 223 F F F 1 020-039 112 112 112 229 229 229 F F F 2 040-059 108 108 108 221 221 221 F F F 3 060-079 92 92 92 185 185 185 T T T 4 080-099 111 111 111 226 226 226 T T T 5 100-119 111 111 111 226 226 226 T T T 6 120-139 112 112 112 227 227 227 F F F 7 140-159 112 112 112 227 227 227 F F F 8 160-179 109 109 109 223 223 223 F F F 9 180-199 110 110 110 225 225 225 F F F 10 200-219 111 111 111 226 226 226 F F F 11 220-239 110 110 110 225 225 225 F F F 12 240-259 110 110 110 225 225 225 T T T 13 260-279 103 103 103 209 209 209 T T T 14 280-299 112 112 112 227 227 227 F F F 15 300-319 112 112 112 227 227 227 F F F 16 320-339 100 100 100 200 200 200 F F F 17 340-359 107 107 107 219 219 219 F F F 18 360-379 112 112 112 229 229 229 T T T 19 380-399 112 112 112 229 229 229 T T T 20 400-419 112 112 112 228 228 228 F F F 21 420-439 111 111 111 226 226 226 F F F 22 440-459 107 107 107 219 219 219 F F F 23 460-479 110 110 110 225 225 225 T T T 24 480-499 112 112 112 227 227 227 F F F 25 500-519 112 112 112 227 227 227 F F F 26 520-539 112 112 112 227 227 227 F F F 27 540-559 106 106 106 216 216 216 F F F 28 560-579 112 112 112 227 227 227 F F F 29 580-599 110 110 110 224 224 224 F F F Max gen: 112, Max flush: 229 A Difference in extent metadata is expected here On loop 56 Verifying data now No change after verify Loop: 56 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 109 109 109 223 223 223 F F F 1 020-039 112 112 112 229 229 229 F F F 2 040-059 108 108 108 221 221 221 F F F 3 060-079 113 113 113 230 230 230 F F F 4 080-099 113 113 113 230 230 230 F F F 5 100-119 113 113 113 230 230 230 F F F 6 120-139 112 112 112 227 227 227 F F F 7 140-159 112 112 112 227 227 227 F F F 8 160-179 109 109 109 223 223 223 F F F 9 180-199 110 110 110 225 225 225 F F F 10 200-219 111 111 111 226 226 226 F F F 11 220-239 110 110 110 225 225 225 F F F 12 240-259 113 113 113 230 230 230 F F F 13 260-279 113 113 113 230 230 230 F F F 14 280-299 112 112 112 227 227 227 F F F 15 300-319 112 112 112 227 227 227 F F F 16 320-339 100 100 100 200 200 200 F F F 17 340-359 107 107 107 219 219 219 F F F 18 360-379 113 113 113 230 230 230 F F F 19 380-399 113 113 113 230 230 230 F F F 20 400-419 112 112 112 228 228 228 F F F 21 420-439 111 111 111 226 226 226 F F F 22 440-459 107 107 107 219 219 219 F F F 23 460-479 113 113 113 230 230 230 F F F 24 480-499 112 112 112 227 227 227 F F F 25 500-519 112 112 112 227 227 227 F F F 26 520-539 112 112 112 227 227 227 F F F 27 540-559 106 106 106 216 216 216 F F F 28 560-579 112 112 112 227 227 227 F F F 29 580-599 110 110 110 224 224 224 F F F Max gen: 113, Max flush: 230 Begin loop 57 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.634814882Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.635195314Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.635225153Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638} {"msg":"Crucible 691224d7-26c9-4ac9-b770-ca3616017934 has session id: e361cbb4-bf49-4202-9c72-f67934ab801f","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.635627054Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"Upstairs opts: Upstairs UUID: 691224d7-26c9-4ac9-b770-ca3616017934, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.635675152Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"Crucible stats registered with UUID: 691224d7-26c9-4ac9-b770-ca3616017934","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.63570421Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:114","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.635947029Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638} {"msg":"691224d7-26c9-4ac9-b770-ca3616017934 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.636011866Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.636038825Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.636059094Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.636073473Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"1","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.636087283Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"1","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.636100822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"2","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.636122331Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"2","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.63613674Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"io task","client":"2","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.63615099Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"io task","client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.636164679Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"io task","client":"1","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"ds_connection connected from Ok(127.0.0.1:65462)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.636438866Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"io task","client":"2","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"ds_connection connected from Ok(127.0.0.1:50350)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.636733542Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"io task","client":"1","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:50:00.636866446Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"io task","client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:50:00.636908624Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"io task","client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:50:00.636944323Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.636967011Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.636991181Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.637039828Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"io task","client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.637314495Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"1","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.637341914Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"1","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.637356573Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"2","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.637370283Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"2","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.637947186Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"1","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.638087979Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"2","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.638464252Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"1","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.638485371Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"1","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.63849972Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.638579436Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"2","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.638601245Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"2","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:00.638614985Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.637822594Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"io task","client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"ds_connection connected from Ok(127.0.0.1:55482)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.637983726Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"io task","client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.639207319Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.639241397Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.639808161Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640350286Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640377775Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"[0]R flush_numbers[0..12]: [223, 229, 221, 230, 230, 230, 227, 227, 223, 225, 226, 225]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640396983Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"downstairs","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"[0]R generation[0..12]: [109, 112, 108, 113, 113, 113, 112, 112, 109, 110, 111, 110]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640414663Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"downstairs","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640432482Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"downstairs","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"[1]R flush_numbers[0..12]: [223, 229, 221, 230, 230, 230, 227, 227, 223, 225, 226, 225]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640449981Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"downstairs","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"[1]R generation[0..12]: [109, 112, 108, 113, 113, 113, 112, 112, 109, 110, 111, 110]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.64046753Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"downstairs","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640485239Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"downstairs","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"[2]R flush_numbers[0..12]: [223, 229, 221, 230, 230, 230, 227, 227, 223, 225, 226, 225]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640506998Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"downstairs","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"[2]R generation[0..12]: [109, 112, 108, 113, 113, 113, 112, 112, 109, 110, 111, 110]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640535117Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"downstairs","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:50:10.640565036Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"downstairs","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"Max found gen is 114","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640594464Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"downstairs","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"Generation requested: 114 >= found:114","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640625213Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"downstairs","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"Next flush: 231","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640652132Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"downstairs","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.64067641Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"":"downstairs","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640704419Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640731028Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"0","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640762566Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"1","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640784996Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"client":"2","":"downstairs","session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640809044Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640836163Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"691224d7-26c9-4ac9-b770-ca3616017934 is now active with session: e361cbb4-bf49-4202-9c72-f67934ab801f","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640865862Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.64088948Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638,"session_id":"e361cbb4-bf49-4202-9c72-f67934ab801f"} {"msg":"The guest has finished waiting for activation with:114","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:10.640921959Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 154 len 36864 data: 8 7 7 8 8 6 5 7 5 03/30 Read block 451 len 24576 04/30 Write block 65 len 4096 data: 5 05/30 Flush 06/30 Write block 174 len 24576 data: 5 7 6 4 5 7 07/30 Read block 23 len 16384 08/30 Flush 09/30 Flush 10/30 Write block 114 len 8192 data: 8 9 11/30 Write block 342 len 12288 data: 9 9 7 12/30 Read block 18 len 28672 13/30 Write block 269 len 8192 data: 9 7 14/30 Write block 265 len 40960 data: 10 9 10 11 10 8 5 5 5 7 15/30 Read block 319 len 16384 16/30 Read block 591 len 8192 17/30 Read block 498 len 24576 18/30 Flush 19/30 Write block 201 len 32768 data: 10 10 9 11 10 11 12 13 20/30 Read block 36 len 20480 21/30 Flush 22/30 Write block 76 len 28672 data: 12 12 11 11 14 14 11 23/30 Write block 481 len 40960 data: 7 7 8 10 9 6 8 7 5 6 24/30 Write block 195 len 16384 data: 11 11 10 8 25/30 Read block 475 len 20480 26/30 Read block 66 len 4096 27/30 Write block 174 len 36864 data: 6 8 7 5 6 8 6 7 9 28/30 Write block 160 len 8192 data: 6 8 29/30 Write block 353 len 12288 data: 11 12 11 30/30 Read block 372 len 28672 ---------------------------------------------------------------- Crucible gen:114 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 7 Done Done Done false 23 Acked 1022 Write 10 Done Done Done false 24 Acked 1023 Write 4 Done Done Done false 25 Acked 1024 Read 5 Done Done Done false 26 Acked 1025 Read 1 Done Done Done false 27 Acked 1026 Write 9 Done Done Done false 28 Acked 1027 Write 2 Done Done Done false 29 Acked 1028 Write 3 Done Done Sent false 30 Acked 1029 Read 7 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 2 3 Done 9 8 7 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:50:10.671289992Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1638} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 109 109 109 223 223 223 F F F 1 020-039 112 112 112 229 229 229 F F F 2 040-059 108 108 108 221 221 221 F F F 3 060-079 114 114 114 232 232 232 T T T 4 080-099 113 113 113 230 230 230 T T T 5 100-119 114 114 114 235 235 235 F F F 6 120-139 112 112 112 227 227 227 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 114 114 114 233 233 233 T T T 9 180-199 110 110 110 225 225 225 T T T 10 200-219 114 114 114 236 236 236 F F F 11 220-239 110 110 110 225 225 225 F F F 12 240-259 113 113 113 230 230 230 F F F 13 260-279 114 114 114 235 235 235 F F F 14 280-299 112 112 112 227 227 227 F F F 15 300-319 112 112 112 227 227 227 F F F 16 320-339 100 100 100 200 200 200 F F F 17 340-359 114 114 114 235 235 235 T T T 18 360-379 113 113 113 230 230 230 F F F 19 380-399 113 113 113 230 230 230 F F F 20 400-419 112 112 112 228 228 228 F F F 21 420-439 111 111 111 226 226 226 F F F 22 440-459 107 107 107 219 219 219 F F F 23 460-479 113 113 113 230 230 230 F F F 24 480-499 112 112 112 227 227 227 T T T 25 500-519 112 112 112 227 227 227 F F F 26 520-539 112 112 112 227 227 227 F F F 27 540-559 106 106 106 216 216 216 F F F 28 560-579 112 112 112 227 227 227 F F F 29 580-599 110 110 110 224 224 224 F F F Max gen: 114, Max flush: 236 A Difference in extent metadata is expected here On loop 57 Verifying data now No change after verify Loop: 57 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 109 109 109 223 223 223 F F F 1 020-039 112 112 112 229 229 229 F F F 2 040-059 108 108 108 221 221 221 F F F 3 060-079 115 115 115 237 237 237 F F F 4 080-099 115 115 115 237 237 237 F F F 5 100-119 114 114 114 235 235 235 F F F 6 120-139 112 112 112 227 227 227 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 115 115 115 237 237 237 F F F 9 180-199 115 115 115 237 237 237 F F F 10 200-219 114 114 114 236 236 236 F F F 11 220-239 110 110 110 225 225 225 F F F 12 240-259 113 113 113 230 230 230 F F F 13 260-279 114 114 114 235 235 235 F F F 14 280-299 112 112 112 227 227 227 F F F 15 300-319 112 112 112 227 227 227 F F F 16 320-339 100 100 100 200 200 200 F F F 17 340-359 115 115 115 237 237 237 F F F 18 360-379 113 113 113 230 230 230 F F F 19 380-399 113 113 113 230 230 230 F F F 20 400-419 112 112 112 228 228 228 F F F 21 420-439 111 111 111 226 226 226 F F F 22 440-459 107 107 107 219 219 219 F F F 23 460-479 113 113 113 230 230 230 F F F 24 480-499 115 115 115 237 237 237 F F F 25 500-519 112 112 112 227 227 227 F F F 26 520-539 112 112 112 227 227 227 F F F 27 540-559 106 106 106 216 216 216 F F F 28 560-579 112 112 112 227 227 227 F F F 29 580-599 110 110 110 224 224 224 F F F Max gen: 115, Max flush: 237 Begin loop 58 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.908188799Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.908571901Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.90859793Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646} {"msg":"Crucible fb83bb4b-9785-40ed-a65e-55a82cee3f30 has session id: bdf1019c-7947-4112-af7c-b5c7f805bb85","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.908752093Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"Upstairs opts: Upstairs UUID: fb83bb4b-9785-40ed-a65e-55a82cee3f30, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.90880774Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"Crucible stats registered with UUID: fb83bb4b-9785-40ed-a65e-55a82cee3f30","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.908836059Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:116","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.909283168Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646} {"msg":"fb83bb4b-9785-40ed-a65e-55a82cee3f30 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.909380234Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.909406792Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.909427762Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.909442071Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"1","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.90946399Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"1","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.909488169Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"2","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.909503348Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"2","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.909516497Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"io task","client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.909530347Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"io task","client":"2","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.909543836Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"io task","client":"1","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:50:24.909651221Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"io task","client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:50:24.909691879Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"io task","client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:50:24.909725748Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.909749596Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.909775405Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"ds_connection connected from Ok(127.0.0.1:55002)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.909800734Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"io task","client":"1","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"ds_connection connected from Ok(127.0.0.1:50385)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.909825883Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"io task","client":"2","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.909848982Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"io task","client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.910592247Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"2","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.910618566Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"2","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.910634165Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"1","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.910657084Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"1","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.911220498Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"1","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.911260096Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"2","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.911832339Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"1","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.911868277Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"1","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.911891827Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.911911766Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"2","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.911937634Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"2","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:24.911959163Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.911143512Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"io task","client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"ds_connection connected from Ok(127.0.0.1:48244)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.911311425Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"io task","client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.912393884Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.912422383Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.912986386Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913583539Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913609967Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"[0]R flush_numbers[0..12]: [223, 229, 221, 237, 237, 235, 227, 232, 237, 237, 236, 225]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913625686Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"downstairs","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"[0]R generation[0..12]: [109, 112, 108, 115, 115, 114, 112, 114, 115, 115, 114, 110]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913640396Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"downstairs","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913654695Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"downstairs","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"[1]R flush_numbers[0..12]: [223, 229, 221, 237, 237, 235, 227, 232, 237, 237, 236, 225]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913668395Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"downstairs","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"[1]R generation[0..12]: [109, 112, 108, 115, 115, 114, 112, 114, 115, 115, 114, 110]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913682114Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"downstairs","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913700573Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"downstairs","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"[2]R flush_numbers[0..12]: [223, 229, 221, 237, 237, 235, 227, 232, 237, 237, 236, 225]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913714533Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"downstairs","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"[2]R generation[0..12]: [109, 112, 108, 115, 115, 114, 112, 114, 115, 115, 114, 110]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913729372Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"downstairs","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913743431Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"downstairs","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"Max found gen is 116","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913756211Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"downstairs","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"Generation requested: 116 >= found:116","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.91376934Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"downstairs","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"Next flush: 238","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913782379Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"downstairs","":"downstairs","session_id":Wait for a query_work_queue command to finish before sending IO "bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913799678Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"":"downstairs","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913812968Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913824847Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"0","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913838217Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"1","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913851536Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"client":"2","":"downstairs","session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913865185Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913877375Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"fb83bb4b-9785-40ed-a65e-55a82cee3f30 is now active with session: bdf1019c-7947-4112-af7c-b5c7f805bb85","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913890014Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913904324Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646,"session_id":"bdf1019c-7947-4112-af7c-b5c7f805bb85"} {"msg":"The guest has finished waiting for activation with:116","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:34.913916423Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 488 len 20480 data: 8 6 7 6 6 02/30 Write block 196 len 20480 data: 12 11 9 7 8 03/30 Read block 163 len 8192 04/30 Read block 381 len 40960 05/30 Write block 313 len 32768 data: 13 17 16 15 12 12 10 10 06/30 Write block 394 len 16384 data: 11 12 13 13 07/30 Write block 379 len 4096 data: 14 08/30 Read block 273 len 28672 09/30 Read block 462 len 12288 10/30 Write block 122 len 40960 data: 8 8 6 6 6 7 8 7 8 8 11/30 Read block 559 len 12288 12/30 Write block 78 len 28672 data: 12 12 15 15 12 12 14 13/30 Flush 14/30 Write block 381 len 32768 data: 11 11 10 13 10 9 11 9 15/30 Write block 287 len 4096 data: 8 16/30 Read block 474 len 36864 17/30 Flush 18/30 Write block 527 len 40960 data: 7 7 6 7 8 6 8 8 8 8 19/30 Read block 54 len 16384 20/30 Read block 341 len 28672 21/30 Read block 29 len 4096 22/30 Write block 415 len 28672 data: 8 8 10 12 15 15 12 23/30 Read block 362 len 8192 24/30 Write block 11 len 12288 data: 9 7 5 25/30 Read block 396 len 8192 26/30 Read block 466 len 40960 27/30 Write block 584 len 32768 data: 10 8 10 6 8 6 7 6 28/30 Write block 323 len 40960 data: 7 7 7 6 7 10 9 9 9 9 29/30 Write block 114 len 32768 data: 9 10 9 9 8 9 10 8 30/30 Read block 336 len 8192 ---------------------------------------------------------------- Crucible gen:116 GIO:true work queues: Upstairs:0 downstairs:13 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 18 Acked 1017 Write 10 Done Done Done false 19 Acked 1018 Read 4 Done Done Done false 20 Acked 1019 Read 7 Done Done Done false 21 Acked 1020 Read 1 Done Done Done false 22 Acked 1021 Write 7 Done Done Done false 23 Acked 1022 Read 2 Done Done Done false 24 Acked 1023 Write 3 Done Done Done false 25 Acked 1024 Read 2 Done Done Done false 26 Acked 1025 Read 10 Done Done Done false 27 Acked 1026 Write 8 Done Done Done false 28 Acked 1027 Write 10 Done Done Done false 29 Acked 1028 Write 8 Done Done Done false 30 Acked 1029 Read 2 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 13 13 13 39 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1016 1016 1016 Downstairs last five completed: 1016 1015 1014 1013 1012 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:50:34.939744988Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1646} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 109 109 109 223 223 223 T T T 1 020-039 112 112 112 229 229 229 F F F 2 040-059 108 108 108 221 221 221 F F F 3 060-079 116 116 116 238 238 238 F F F 4 080-099 116 116 116 238 238 238 F F F 5 100-119 114 114 114 235 235 235 T T T 6 120-139 116 116 116 238 238 238 T T T 7 140-159 114 114 114 232 232 232 F F F 8 160-179 115 115 115 237 237 237 F F F 9 180-199 116 116 116 238 238 238 F F F 10 200-219 116 116 116 238 238 238 F F F 11 220-239 110 110 110 225 225 225 F F F 12 240-259 113 113 113 230 230 230 F F F 13 260-279 114 114 114 235 235 235 F F F 14 280-299 116 116 116 239 239 239 F F F 15 300-319 116 116 116 238 238 238 F F F 16 320-339 116 116 116 238 238 238 T T T 17 340-359 115 115 115 237 237 237 F F F 18 360-379 116 116 116 238 238 238 F F F 19 380-399 116 116 116 239 239 239 F F F 20 400-419 112 112 112 228 228 228 T T T 21 420-439 111 111 111 226 226 226 T T T 22 440-459 107 107 107 219 219 219 F F F 23 460-479 113 113 113 230 230 230 F F F 24 480-499 116 116 116 238 238 238 F F F 25 500-519 112 112 112 227 227 227 F F F 26 520-539 112 112 112 227 227 227 T T T 27 540-559 106 106 106 216 216 216 F F F 28 560-579 112 112 112 227 227 227 F F F 29 580-599 110 110 110 224 224 224 T T T Max gen: 116, Max flush: 239 A Difference in extent metadata is expected here On loop 58 Verifying data now No change after verify Loop: 58 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 117 117 117 240 240 240 F F F 1 020-039 112 112 112 229 229 229 F F F 2 040-059 108 108 108 221 221 221 F F F 3 060-079 116 116 116 238 238 238 F F F 4 080-099 116 116 116 238 238 238 F F F 5 100-119 117 117 117 240 240 240 F F F 6 120-139 117 117 117 240 240 240 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 115 115 115 237 237 237 F F F 9 180-199 116 116 116 238 238 238 F F F 10 200-219 116 116 116 238 238 238 F F F 11 220-239 110 110 110 225 225 225 F F F 12 240-259 113 113 113 230 230 230 F F F 13 260-279 114 114 114 235 235 235 F F F 14 280-299 116 116 116 239 239 239 F F F 15 300-319 116 116 116 238 238 238 F F F 16 320-339 117 117 117 240 240 240 F F F 17 340-359 115 115 115 237 237 237 F F F 18 360-379 116 116 116 238 238 238 F F F 19 380-399 116 116 116 239 239 239 F F F 20 400-419 117 117 117 240 240 240 F F F 21 420-439 117 117 117 240 240 240 F F F 22 440-459 107 107 107 219 219 219 F F F 23 460-479 113 113 113 230 230 230 F F F 24 480-499 116 116 116 238 238 238 F F F 25 500-519 112 112 112 227 227 227 F F F 26 520-539 117 117 117 240 240 240 F F F 27 540-559 106 106 106 216 216 216 F F F 28 560-579 112 112 112 227 227 227 F F F 29 580-599 117 117 117 240 240 240 F F F Max gen: 117, Max flush: 240 Begin loop 59 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.200331041Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.200784409Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.200816948Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654} {"msg":"Crucible e00f74dd-5af4-482f-91a4-3fee10cb1332 has session id: 70a2b6d5-10ba-49c1-8809-e2fb88de3727","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.200964021Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"Upstairs opts: Upstairs UUID: e00f74dd-5af4-482f-91a4-3fee10cb1332, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.201087175Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"Crucible stats registered with UUID: e00f74dd-5af4-482f-91a4-3fee10cb1332","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.201119694Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:118","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.201464428Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654} {"msg":"e00f74dd-5af4-482f-91a4-3fee10cb1332 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.201497336Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.201511786Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.201542494Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.201568523Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"1","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.201593282Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"1","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.201615771Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"2","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.20163974Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"2","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.201662328Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"io task","client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.201682168Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"io task","client":"2","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.201703817Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"io task","client":"1","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"ds_connection connected from Ok(127.0.0.1:43595)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.201825721Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"io task","client":"2","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:50:49.201866369Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"io task","client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:50:49.201894538Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"io task","client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:50:49.201922216Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.201946165Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.201971564Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"ds_connection connected from Ok(127.0.0.1:49386)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.2020585Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"io task","client":"1","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.202094788Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"io task","client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.20269577Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"2","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.202720019Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"2","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.202733998Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"1","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.202747458Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"1","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.203251854Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"1","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.203366509Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"2","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.20378311Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"1","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.203826018Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"1","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.203845907Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.203861256Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"2","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.203878045Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"2","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:49.203894355Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.20316969Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"io task","client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"ds_connection connected from Ok(127.0.0.1:48873)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.203309123Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"io task","client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.204534066Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.204565375Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205130628Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205720271Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.20574405Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"[0]R flush_numbers[0..12]: [240, 229, 221, 238, 238, 240, 240, 232, 237, 238, 238, 225]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205759169Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"downstairs","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"[0]R generation[0..12]: [117, 112, 108, 116, 116, 117, 117, 114, 115, 116, 116, 110]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205773608Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"downstairs","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205787978Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"downstairs","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"[1]R flush_numbers[0..12]: [240, 229, 221, 238, 238, 240, 240, 232, 237, 238, 238, 225]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205806507Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"downstairs","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"[1]R generation[0..12]: [117, 112, 108, 116, 116, 117, 117, 114, 115, 116, 116, 110]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205820736Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"downstairs","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205834805Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"downstairs","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"[2]R flush_numbers[0..12]: [240, 229, 221, 238, 238, 240, 240, 232, 237, 238, 238, 225]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205848645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"downstairs","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"[2]R generation[0..12]: [117, 112, 108, 116, 116, 117, 117, 114, 115, 116, 116, 110]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205862374Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"downstairs","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205876534Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"downstairs","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"Max found gen is 118","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205889543Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"downstairs","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"Generation requested: 118 >= found:118","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:50:59.205904202Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"downstairs","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"Next flush: 241","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205918492Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"downstairs","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205931551Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"":"downstairs","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.2059447Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.20595663Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"0","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205970079Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"1","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205983638Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"client":"2","":"downstairs","session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.205997498Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.206011747Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"e00f74dd-5af4-482f-91a4-3fee10cb1332 is now active with session: 70a2b6d5-10ba-49c1-8809-e2fb88de3727","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.206025087Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.206037006Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654,"session_id":"70a2b6d5-10ba-49c1-8809-e2fb88de3727"} {"msg":"The guest has finished waiting for activation with:118","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:50:59.206049166Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 96 len 8192 data: 7 9 03/30 Write block 42 len 32768 data: 5 6 7 8 8 11 11 12 04/30 Write block 542 len 28672 data: 10 8 10 8 8 8 9 05/30 Write block 485 len 28672 data: 10 7 9 9 7 8 7 06/30 Read block 0 len 20480 07/30 Read block 21 len 36864 08/30 Write block 222 len 32768 data: 11 9 8 9 7 8 9 10 09/30 Read block 269 len 8192 10/30 Write block 54 len 16384 data: 8 8 10 8 11/30 Write block 261 len 28672 data: 9 13 12 10 11 10 11 12/30 Flush 13/30 Write block 573 len 24576 data: 18 16 13 12 11 11 14/30 Flush 15/30 Write block 163 len 32768 data: 5 6 8 9 9 10 10 11 16/30 Flush 17/30 Write block 544 len 8192 data: 11 9 18/30 Read block 121 len 24576 19/30 Read block 257 len 12288 20/30 Read block 122 len 32768 21/30 Write block 402 len 20480 data: 10 9 10 11 12 22/30 Write block 84 len 24576 data: 15 14 13 10 8 9 23/30 Read block 172 len 36864 24/30 Read block 550 len 32768 25/30 Read block 563 len 36864 26/30 Read block 87 len 12288 27/30 Read block 304 len 40960 28/30 Read block 524 len 36864 29/30 Write block 334 len 4096 data: 10 30/30 Read block 373 len 40960 ---------------------------------------------------------------- Crucible gen:118 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 20 Acked 1019 Read 3 Done Done Done false 21 Acked 1020 Read 8 Done Done Done false 22 Acked 1021 Write 5 Done Done Done false 23 Acked 1022 Write 6 Done Done Done false 24 Acked 1023 Read 9 Done Done Done false 25 Acked 1024 Read 8 Done Done Done false 26 Acked 1025 Read 9 Done Done Done false 27 Acked 1026 Read 3 Done Done Done false 28 Acked 1027 Read 10 Done Done Done false 29 Acked 1028 Read 9 Done Done Done false 30 Acked 1029 Write 1 Done Done Done false 31 Acked 1030 Read 10 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 1 2 Done 11 12 11 34 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1018 1018 Downstairs last five completed: 1018 1017 1016 1015 1014 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:50:59.232411275Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1654} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 117 117 117 240 240 240 F F F 1 020-039 112 112 112 229 229 229 F F F 2 040-059 118 118 118 242 242 242 F F F 3 060-079 116 116 116 238 238 238 F F F 4 080-099 118 118 118 242 242 242 T T T 5 100-119 117 117 117 240 240 240 F F F 6 120-139 117 117 117 240 240 240 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 118 118 118 244 244 244 F F F 9 180-199 116 116 116 238 238 238 F F F 10 200-219 116 116 116 238 238 238 F F F 11 220-239 118 118 118 242 242 242 F F F 12 240-259 113 113 113 230 230 230 F F F 13 260-279 118 118 118 242 242 242 F F F 14 280-299 116 116 116 239 239 239 F F F 15 300-319 116 116 116 238 238 238 F F F 16 320-339 117 117 117 240 240 240 T T T 17 340-359 115 115 115 237 237 237 F F F 18 360-379 116 116 116 238 238 238 F F F 19 380-399 116 116 116 239 239 239 F F F 20 400-419 117 117 117 240 240 240 T T T 21 420-439 117 117 117 240 240 240 F F F 22 440-459 107 107 107 219 219 219 F F F 23 460-479 113 113 113 230 230 230 F F F 24 480-499 118 118 118 242 242 242 F F F 25 500-519 112 112 112 227 227 227 F F F 26 520-539 117 117 117 240 240 240 F F F 27 540-559 118 118 118 245 245 245 F F F 28 560-579 118 118 118 243 243 243 F F F 29 580-599 117 117 117 240 240 240 F F F Max gen: 118, Max flush: 245 A Difference in extent metadata is expected here On loop 59 Verifying data now No change after verify Loop: 59 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 117 117 117 240 240 240 F F F 1 020-039 112 112 112 229 229 229 F F F 2 040-059 118 118 118 242 242 242 F F F 3 060-079 116 116 116 238 238 238 F F F 4 080-099 119 119 119 246 246 246 F F F 5 100-119 117 117 117 240 240 240 F F F 6 120-139 117 117 117 240 240 240 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 118 118 118 244 244 244 F F F 9 180-199 116 116 116 238 238 238 F F F 10 200-219 116 116 116 238 238 238 F F F 11 220-239 118 118 118 242 242 242 F F F 12 240-259 113 113 113 230 230 230 F F F 13 260-279 118 118 118 242 242 242 F F F 14 280-299 116 116 116 239 239 239 F F F 15 300-319 116 116 116 238 238 238 F F F 16 320-339 119 119 119 246 246 246 F F F 17 340-359 115 115 115 237 237 237 F F F 18 360-379 116 116 116 238 238 238 F F F 19 380-399 116 116 116 239 239 239 F F F 20 400-419 119 119 119 246 246 246 F F F 21 420-439 117 117 117 240 240 240 F F F 22 440-459 107 107 107 219 219 219 F F F 23 460-479 113 113 113 230 230 230 F F F 24 480-499 118 118 118 242 242 242 F F F 25 500-519 112 112 112 227 227 227 F F F 26 520-539 117 117 117 240 240 240 F F F 27 540-559 118 118 118 245 245 245 F F F 28 560-579 118 118 118 243 243 243 F F F 29 580-599 117 117 117 240 240 240 F F F Max gen: 119, Max flush: 246 Begin loop 60 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.433051877Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.433447108Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.433482886Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662} {"msg":"Crucible 407072aa-3107-47b2-9fe9-58cf2fc0632d has session id: aa7230b2-83ac-42ee-bf03-342d1e8a08a3","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.433703896Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"Upstairs opts: Upstairs UUID: 407072aa-3107-47b2-9fe9-58cf2fc0632d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.433735975Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"Crucible stats registered with UUID: 407072aa-3107-47b2-9fe9-58cf2fc0632d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.433750984Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:120","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.434195323Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662} {"msg":"407072aa-3107-47b2-9fe9-58cf2fc0632d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.434221192Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.434239791Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.43427158Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.434302888Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"1","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.434329117Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"1","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.434353476Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"2","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.434375615Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"2","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.434391244Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"io task","client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.434414813Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"io task","client":"1","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.434443322Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"io task","client":"2","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:51:13.43447273Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"io task","client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:51:13.434503709Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"io task","client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:51:13.434626043Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.434721769Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.434745518Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"ds_connection connected from Ok(127.0.0.1:52906)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.434760747Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"io task","client":"1","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.434774846Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"io task","client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"ds_connection connected from Ok(127.0.0.1:54320)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.434808755Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"io task","client":"2","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.435413547Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"1","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.435441145Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"1","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.435455594Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"2","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.435477433Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"2","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.436030258Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"2","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.436157062Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"1","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.436511845Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"2","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.436544044Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"2","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.436558903Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.436650469Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"1","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.436669018Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"1","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:13.436682347Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.435481367Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"io task","client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"ds_connection connected from Ok(127.0.0.1:47769)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.435635699Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"io task","client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.436596255Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.436624943Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.437392448Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.437948602Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.437977Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"[0]R flush_numbers[0..12]: [240, 229, 242, 238, 246, 240, 240, 232, 244, 238, 238, 242]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.437992829Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"downstairs","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"[0]R generation[0..12]: [117, 112, 118, 116, 119, 117, 117, 114, 118, 116, 116, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438007599Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"downstairs","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438027048Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"downstairs","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"[1]R flush_numbers[0..12]: [240, 229, 242, 238, 246, 240, 240, 232, 244, 238, 238, 242]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438041327Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"downstairs","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"[1]R generation[0..12]: [117, 112, 118, 116, 119, 117, 117, 114, 118, 116, 116, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438055486Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"downstairs","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438069746Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"downstairs","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"[2]R flush_numbers[0..12]: [240, 229, 242, 238, 246, 240, 240, 232, 244, 238, 238, 242]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438083625Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"downstairs","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"[2]R generation[0..12]: [117, 112, 118, 116, 119, 117, 117, 114, 118, 116, 116, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438097544Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"downstairs","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438111754Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"downstairs","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"Max found gen is 120","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438124923Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"Generation requested: 120 >= found:120","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438142553Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"downstairs","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"Next flush: 247","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438155572Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"downstairs","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438168651Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"":"downstairs","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438181741Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.43819381Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"0","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438208279Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"1","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438224229Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"client":"2","":"downstairs","session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438237728Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438249788Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"407072aa-3107-47b2-9fe9-58cf2fc0632d is now active with session: aa7230b2-83ac-42ee-bf03-342d1e8a08a3","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438262487Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438274346Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662,"session_id":"aa7230b2-83ac-42ee-bf03-342d1e8a08a3"} {"msg":"The guest has finished waiting for activation with:120","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:23.438286556Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 554 len 8192 data: 11 11 03/30 Write block 193 len 24576 data: 13 10 12 13 12 10 04/30 Read block 159 len 4096 05/30 Read block 144 len 36864 06/30 Flush 07/30 Write block 188 len 8192 data: 17 14 08/30 Read block 62 len 40960 09/30 Read block 406 len 28672 10/30 Read block 568 len 40960 11/30 Write block 452 len 40960 data: 12 9 6 8 7 6 7 7 8 8 12/30 Read block 318 len 4096 13/30 Read block 437 len 24576 14/30 Write block 484 len 36864 data: 11 11 8 10 10 8 9 8 7 15/30 Read block 216 len 12288 16/30 Read block 145 len 24576 17/30 Write block 278 len 16384 data: 9 11 12 8 18/30 Write block 107 len 28672 data: 8 9 7 7 7 6 6 19/30 Read block 191 len 16384 20/30 Read block 547 len 16384 21/30 Read block 217 len 36864 22/30 Read block 252 len 20480 23/30 Read block 379 len 4096 24/30 Read block 573 len 32768 25/30 Read block 139 len 40960 26/30 Read block 248 len 24576 27/30 Read block 87 len 40960 28/30 Read block 300 len 20480 29/30 Write block 122 len 16384 data: 9 9 7 7 30/30 Read block 228 len 28672 ---------------------------------------------------------------- Crucible gen:120 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Read 6 Done Done Done false 28 Acked 1027 Read 10 Done Done Done false 29 Acked 1028 Read 5 Done Done Done false 30 Acked 1029 Write 4 Done Done Sent false 31 Acked 1030 Read 7 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 2 3 Done 4 5 3 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 31 30 29 28 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:51:23.464252354Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1662} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 117 117 117 240 240 240 F F F 1 020-039 112 112 112 229 229 229 F F F 2 040-059 118 118 118 242 242 242 F F F 3 060-079 116 116 116 238 238 238 F F F 4 080-099 119 119 119 246 246 246 F F F 5 100-119 120 120 120 249 249 249 F F F 6 120-139 117 117 117 240 240 240 T T T 7 140-159 114 114 114 232 232 232 F F F 8 160-179 118 118 118 244 244 244 F F F 9 180-199 120 120 120 249 249 249 F F F 10 200-219 116 116 116 238 238 238 F F F 11 220-239 118 118 118 242 242 242 F F F 12 240-259 113 113 113 230 230 230 F F F 13 260-279 120 120 120 249 249 249 F F F 14 280-299 120 120 120 249 249 249 F F F 15 300-319 116 116 116 238 238 238 F F F 16 320-339 119 119 119 246 246 246 F F F 17 340-359 115 115 115 237 237 237 F F F 18 360-379 116 116 116 238 238 238 F F F 19 380-399 116 116 116 239 239 239 F F F 20 400-419 119 119 119 246 246 246 F F F 21 420-439 117 117 117 240 240 240 F F F 22 440-459 120 120 120 249 249 249 F F F 23 460-479 120 120 120 249 249 249 F F F 24 480-499 120 120 120 249 249 249 F F F 25 500-519 112 112 112 227 227 227 F F F 26 520-539 117 117 117 240 240 240 F F F 27 540-559 120 120 120 248 248 248 F F F 28 560-579 118 118 118 243 243 243 F F F 29 580-599 117 117 117 240 240 240 F F F Max gen: 120, Max flush: 249 A Difference in extent metadata is expected here On loop 60 Verifying data now No change after verify Loop: 60 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 117 117 117 240 240 240 F F F 1 020-039 112 112 112 229 229 229 F F F 2 040-059 118 118 118 242 242 242 F F F 3 060-079 116 116 116 238 238 238 F F F 4 080-099 119 119 119 246 246 246 F F F 5 100-119 120 120 120 249 249 249 F F F 6 120-139 121 121 121 250 250 250 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 118 118 118 244 244 244 F F F 9 180-199 120 120 120 249 249 249 F F F 10 200-219 116 116 116 238 238 238 F F F 11 220-239 118 118 118 242 242 242 F F F 12 240-259 113 113 113 230 230 230 F F F 13 260-279 120 120 120 249 249 249 F F F 14 280-299 120 120 120 249 249 249 F F F 15 300-319 116 116 116 238 238 238 F F F 16 320-339 119 119 119 246 246 246 F F F 17 340-359 115 115 115 237 237 237 F F F 18 360-379 116 116 116 238 238 238 F F F 19 380-399 116 116 116 239 239 239 F F F 20 400-419 119 119 119 246 246 246 F F F 21 420-439 117 117 117 240 240 240 F F F 22 440-459 120 120 120 249 249 249 F F F 23 460-479 120 120 120 249 249 249 F F F 24 480-499 120 120 120 249 249 249 F F F 25 500-519 112 112 112 227 227 227 F F F 26 520-539 117 117 117 240 240 240 F F F 27 540-559 120 120 120 248 248 248 F F F 28 560-579 118 118 118 243 243 243 F F F 29 580-599 117 117 117 240 240 240 F F F Max gen: 121, Max flush: 250 Begin loop 61 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.639818584Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.640635886Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.640662344Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670} {"msg":"Crucible b5cd5cff-1b02-4e5b-a07e-8f6fd8160544 has session id: 25662942-e29c-4a9d-928c-1e5fa2d115c2","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.640682663Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"Upstairs opts: Upstairs UUID: b5cd5cff-1b02-4e5b-a07e-8f6fd8160544, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.640729701Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"Crucible stats registered with UUID: b5cd5cff-1b02-4e5b-a07e-8f6fd8160544","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.64075217Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:122","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.641096944Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670} {"msg":"b5cd5cff-1b02-4e5b-a07e-8f6fd8160544 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.641120003Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.641133262Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"0","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.641152552Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"0","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.641166391Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.64117994Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.6411933Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"2","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.641206809Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"2","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.641281315Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"io task","client":"0","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.641300755Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"io task","client":"2","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.641314714Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"io task","client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"ds_connection connected from Ok(127.0.0.1:64247)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.641583812Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"io task","client":"0","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:51:33.641695176Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"io task","client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:51:33.641732205Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"io task","client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"ds_connection connected from Ok(127.0.0.1:61922)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.641762163Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"io task","client":"2","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:51:33.641782392Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.641795362Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.641809281Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.6418226Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"io task","client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.6422559Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"2","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.642290879Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"2","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.642318597Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"0","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.642346066Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"0","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.642924099Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"0","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.642953517Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"2","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.643498852Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"2","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.643525681Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"2","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.64354036Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.643580378Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"0","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.643597098Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"0","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:33.643610507Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.64340431Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"io task","client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"ds_connection connected from Ok(127.0.0.1:43408)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.643813561Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"io task","client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.644634793Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.644670291Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.64512567Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.645663795Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.645688063Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"[0]R flush_numbers[0..12]: [240, 229, 242, 238, 246, 249, 250, 232, 244, 249, 238, 242]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.645706063Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"downstairs","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"[0]R generation[0..12]: [117, 112, 118, 116, 119, 120, 121, 114, 118, 120, 116, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.645723452Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"downstairs","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.645747381Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"downstairs","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"[1]R flush_numbers[0..12]: [240, 229, 242, 238, 246, 249, 250, 232, 244, 249, 238, 242]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.64576539Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"downstairs","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"[1]R generation[0..12]: [117, 112, 118, 116, 119, 120, 121, 114, 118, 120, 116, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.645782819Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"downstairs","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.645800458Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"downstairs","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"[2]R flush_numbers[0..12]: [240, 229, 242, 238, 246, 249, 250, 232, 244, 249, 238, 242]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.645817657Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"downstairs","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"[2]R generation[0..12]: [117, 112, 118, 116, 119, 120, 121, 114, 118, 120, 116, 118]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.645834747Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"downstairs","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.645852096Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"downstairs","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"Max found gen is 122","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:51:43.645868335Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"downstairs","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"Generation requested: 122 >= found:122","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.645885034Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"downstairs","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"Next flush: 251","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.645901134Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"downstairs","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.645917333Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"":"downstairs","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.645933592Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.645953091Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"0","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.64598324Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"1","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.646011139Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"client":"2","":"downstairs","session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.646038097Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.646065886Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"b5cd5cff-1b02-4e5b-a07e-8f6fd8160544 is now active with session: 25662942-e29c-4a9d-928c-1e5fa2d115c2","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.646095644Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:43.646123253Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670,"session_id":"25662942-e29c-4a9d-928c-1e5fa2d115c2"} {"msg":"The guest has finished waiting for activation with:122","v":0,"name"Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :"crucible","level":30,"time":"2024-07-31T18:51:43.646159232Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 92 len 8192 data: 10 9 02/30 Read block 136 len 8192 03/30 Write block 167 len 28672 data: 10 11 11 12 10 10 9 04/30 Read block 22 len 32768 05/30 Write block 519 len 24576 data: 11 11 9 10 9 11 06/30 Read block 325 len 24576 07/30 Write block 496 len 40960 data: 8 12 9 10 10 10 12 11 12 14 08/30 Read block 137 len 16384 09/30 Flush 10/30 Write block 394 len 8192 data: 12 13 11/30 Read block 363 len 40960 12/30 Read block 282 len 16384 13/30 Read block 269 len 8192 14/30 Read block 433 len 28672 15/30 Write block 497 len 28672 data: 13 10 11 11 11 13 12 16/30 Write block 127 len 28672 data: 8 9 8 9 9 8 10 17/30 Read block 464 len 12288 18/30 Write block 376 len 20480 data: 12 13 15 15 11 19/30 Flush 20/30 Write block 479 len 36864 data: 9 8 8 8 9 12 12 9 11 21/30 Read block 526 len 20480 22/30 Read block 181 len 40960 23/30 Read block 192 len 36864 24/30 Read block 121 len 4096 25/30 Write block 215 len 32768 data: 8 8 6 6 8 8 10 12 26/30 Read block 526 len 28672 27/30 Read block 78 len 4096 28/30 Write block 8 len 40960 data: 6 7 7 10 8 6 4 5 5 7 29/30 Read block 148 len 16384 30/30 Read block 466 len 12288 ---------------------------------------------------------------- Crucible gen:122 GIO:true work queues: Upstairs:0 downstairs:11 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 20 Acked 1019 Write 9 Done Done Done false 21 Acked 1020 Read 5 Done Done Done false 22 Acked 1021 Read 10 Done Done Done false 23 Acked 1022 Read 9 Done Done Done false 24 Acked 1023 Read 1 Done Done Done false 25 Acked 1024 Write 8 Done Done Done false 26 Acked 1025 Read 7 Done Done Done false 27 Acked 1026 Read 1 Done Done Done false 28 Acked 1027 Write 10 Done Done Done false 29 Acked 1028 Read 4 Done Done Done false 30 Acked 1029 Read 3 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 1 2 Done 11 10 10 31 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1018 1018 Downstairs last five completed: 1018 1017 1016 1015 1014 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:51:43.672739361Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1670} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 117 117 117 240 240 240 T T T 1 020-039 112 112 112 229 229 229 F F F 2 040-059 118 118 118 242 242 242 F F F 3 060-079 116 116 116 238 238 238 F F F 4 080-099 122 122 122 251 251 251 F F F 5 100-119 120 120 120 249 249 249 F F F 6 120-139 122 122 122 252 252 252 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 122 122 122 251 251 251 F F F 9 180-199 120 120 120 249 249 249 F F F 10 200-219 116 116 116 238 238 238 T T T 11 220-239 118 118 118 242 242 242 T T T 12 240-259 113 113 113 230 230 230 F F F 13 260-279 120 120 120 249 249 249 F F F 14 280-299 120 120 120 249 249 249 F F F 15 300-319 116 116 116 238 238 238 F F F 16 320-339 119 119 119 246 246 246 F F F 17 340-359 115 115 115 237 237 237 F F F 18 360-379 122 122 122 252 252 252 F F F 19 380-399 122 122 122 252 252 252 F F F 20 400-419 119 119 119 246 246 246 F F F 21 420-439 117 117 117 240 240 240 F F F 22 440-459 120 120 120 249 249 249 F F F 23 460-479 120 120 120 249 249 249 T T T 24 480-499 122 122 122 252 252 252 T T T 25 500-519 122 122 122 252 252 252 F F F 26 520-539 122 122 122 251 251 251 F F F 27 540-559 120 120 120 248 248 248 F F F 28 560-579 118 118 118 243 243 243 F F F 29 580-599 117 117 117 240 240 240 F F F Max gen: 122, Max flush: 252 A Difference in extent metadata is expected here On loop 61 Verifying data now No change after verify Loop: 61 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 123 123 123 253 253 253 F F F 1 020-039 112 112 112 229 229 229 F F F 2 040-059 118 118 118 242 242 242 F F F 3 060-079 116 116 116 238 238 238 F F F 4 080-099 122 122 122 251 251 251 F F F 5 100-119 120 120 120 249 249 249 F F F 6 120-139 122 122 122 252 252 252 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 122 122 122 251 251 251 F F F 9 180-199 120 120 120 249 249 249 F F F 10 200-219 123 123 123 253 253 253 F F F 11 220-239 123 123 123 253 253 253 F F F 12 240-259 113 113 113 230 230 230 F F F 13 260-279 120 120 120 249 249 249 F F F 14 280-299 120 120 120 249 249 249 F F F 15 300-319 116 116 116 238 238 238 F F F 16 320-339 119 119 119 246 246 246 F F F 17 340-359 115 115 115 237 237 237 F F F 18 360-379 122 122 122 252 252 252 F F F 19 380-399 122 122 122 252 252 252 F F F 20 400-419 119 119 119 246 246 246 F F F 21 420-439 117 117 117 240 240 240 F F F 22 440-459 120 120 120 249 249 249 F F F 23 460-479 123 123 123 253 253 253 F F F 24 480-499 123 123 123 253 253 253 F F F 25 500-519 122 122 122 252 252 252 F F F 26 520-539 122 122 122 251 251 251 F F F 27 540-559 120 120 120 248 248 248 F F F 28 560-579 118 118 118 243 243 243 F F F 29 580-599 117 117 117 240 240 240 F F F Max gen: 123, Max flush: 253 Begin loop 62 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.906342742Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.906897786Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.906925775Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678} {"msg":"Crucible fb3daa94-4161-4c5c-8eee-10d398a73c52 has session id: 1920f730-0c88-4529-8ece-d078a4dddb00","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.90703796Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"Upstairs opts: Upstairs UUID: fb3daa94-4161-4c5c-8eee-10d398a73c52, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.907099637Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"Crucible stats registered with UUID: fb3daa94-4161-4c5c-8eee-10d398a73c52","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.907122885Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:124","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.907436681Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678} {"msg":"fb3daa94-4161-4c5c-8eee-10d398a73c52 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.907557365Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.907579434Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.907611453Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.907638561Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"1","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.90766271Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"1","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.907687509Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"2","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.907712588Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"2","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.907734327Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"io task","client":"2","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.907761816Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"io task","client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.907786005Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"io task","client":"1","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"ds_connection connected from Ok(127.0.0.1:35877)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.907873741Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"io task","client":"1","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:51:57.907912649Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"io task","client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:51:57.907940687Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"io task","client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:51:57.907966616Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.907988525Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.908012694Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"ds_connection connected from Ok(127.0.0.1:47852)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.908036293Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"io task","client":"2","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.908210215Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"io task","client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.908707271Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"1","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.90873473Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"1","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.90875Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"2","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.908769869Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"2","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.909313803Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"1","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.909429868Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"2","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.90980753Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"1","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.909835029Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"1","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.909849898Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.909924025Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"2","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.909943074Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"2","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:51:57.909956253Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.909828291Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"io task","client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"ds_connection connected from Ok(127.0.0.1:50262)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.909958556Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"io task","client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.911439866Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.911473875Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.911936963Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912492317Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912518056Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"[0]R flush_numbers[0..12]: [253, 229, 242, 238, 251, 249, 252, 232, 251, 249, 253, 253]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912553664Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"downstairs","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"[0]R generation[0..12]: [123, 112, 118, 116, 122, 120, 122, 114, 122, 120, 123, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912578023Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"downstairs","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912596382Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"downstairs","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"[1]R flush_numbers[0..12]: [253, 229, 242, 238, 251, 249, 252, 232, 251, 249, 253, 253]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912613801Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"downstairs","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"[1]R generation[0..12]: [123, 112, 118, 116, 122, 120, 122, 114, 122, 120, 123, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912630951Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"downstairs","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.9126487Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"downstairs","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"[2]R flush_numbers[0..12]: [253, 229, 242, 238, 251, 249, 252, 232, 251, 249, 253, 253]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912666199Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"downstairs","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"[2]R generation[0..12]: [123, 112, 118, 116, 122, 120, 122, 114, 122, 120, 123, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912683448Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"downstairs","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912701157Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"downstairs","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"Max found gen is 124","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:52:07.912717167Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"downstairs","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"Generation requested: 124 >= found:124","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912735966Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"downstairs","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"Next flush: 254","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912753775Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"downstairs","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912770864Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"":"downstairs","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912787584Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912802563Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"0","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912821492Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"1","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912838111Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"client":"2","":"downstairs","session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.91285481Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.91287046Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"fb3daa94-4161-4c5c-8eee-10d398a73c52 is now active with session: 1920f730-0c88-4529-8ece-d078a4dddb00","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912883419Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912895539Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678,"session_id":"1920f730-0c88-4529-8ece-d078a4dddb00"} {"msg":"The guest has finished waiting for activation with:124","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:07.912907908Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 273 len 24576 data: 6 8 9 10 11 10 02/30 Read block 571 len 40960 03/30 Read block 141 len 12288 04/30 Write block 467 len 28672 data: 8 8 8 8 10 11 11 05/30 Read block 465 len 32768 06/30 Write block 380 len 20480 data: 12 12 12 11 14 07/30 Read block 538 len 20480 08/30 Write block 33 len 28672 data: 5 6 6 10 9 10 10 09/30 Read block 333 len 28672 10/30 Write block 429 len 4096 data: 16 11/30 Read block 588 len 32768 12/30 Write block 395 len 20480 data: 14 14 14 14 12 13/30 Flush 14/30 Write block 443 len 24576 data: 11 13 16 12 10 13 15/30 Read block 243 len 4096 16/30 Read block 584 len 20480 17/30 Write block 591 len 4096 data: 7 18/30 Read block 187 len 4096 19/30 Read block 40 len 16384 20/30 Read block 352 len 32768 21/30 Read block 38 len 16384 22/30 Write block 476 len 16384 data: 12 12 9 10 23/30 Read block 175 len 20480 24/30 Read block 339 len 8192 25/30 Read block 593 len 16384 26/30 Write block 271 len 32768 data: 6 6 7 9 10 11 12 11 27/30 Write block 562 len 40960 data: 12 14 15 16 16 14 13 10 13 15 28/30 Read block 417 len 40960 29/30 Read block 82 len 16384 30/30 Read block 74 len 32768 ---------------------------------------------------------------- Crucible gen:124 GIO:true work queues: Upstairs:0 downstairs:13 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 19 Acked 1018 Read 1 Done Done Done false 20 Acked 1019 Read 4 Done Done Done false 21 Acked 1020 Read 8 Done Done Done false 22 Acked 1021 Read 4 Done Done Done false 23 Acked 1022 Write 4 Done Done Done false 24 Acked 1023 Read 5 Done Done Done false 25 Acked 1024 Read 2 Done Done Done false 26 Acked 1025 Read 4 Done Done Done false 27 Acked 1026 Write 8 Done Done Done false 28 Acked 1027 Write 10 Done Done Done false 29 Acked 1028 Read 10 Done Done Done false 30 Acked 1029 Read 4 Done Done Done false 31 Acked 1030 Read 8 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 0 2 Done 12 12 13 37 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1017 1017 1017 Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:52:07.939320945Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1678} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 123 123 123 253 253 253 F F F 1 020-039 124 124 124 254 254 254 F F F 2 040-059 118 118 118 242 242 242 F F F 3 060-079 116 116 116 238 238 238 F F F 4 080-099 122 122 122 251 251 251 F F F 5 100-119 120 120 120 249 249 249 F F F 6 120-139 122 122 122 252 252 252 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 122 122 122 251 251 251 F F F 9 180-199 120 120 120 249 249 249 F F F 10 200-219 123 123 123 253 253 253 F F F 11 220-239 123 123 123 253 253 253 F F F 12 240-259 113 113 113 230 230 230 F F F 13 260-279 124 124 124 254 254 254 T T T 14 280-299 120 120 120 249 249 249 F F F 15 300-319 116 116 116 238 238 238 F F F 16 320-339 119 119 119 246 246 246 F F F 17 340-359 115 115 115 237 237 237 F F F 18 360-379 122 122 122 252 252 252 F F F 19 380-399 124 124 124 254 254 254 F F F 20 400-419 119 119 119 246 246 246 F F F 21 420-439 124 124 124 254 254 254 F F F 22 440-459 124 124 124 255 255 255 F F F 23 460-479 124 124 124 254 254 254 T T T 24 480-499 123 123 123 253 253 253 F F F 25 500-519 122 122 122 252 252 252 F F F 26 520-539 122 122 122 251 251 251 F F F 27 540-559 120 120 120 248 248 248 F F F 28 560-579 118 118 118 243 243 243 T T T 29 580-599 124 124 124 255 255 255 F F F Max gen: 124, Max flush: 255 A Difference in extent metadata is expected here On loop 62 Verifying data now No change after verify Loop: 62 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 123 123 123 253 253 253 F F F 1 020-039 124 124 124 254 254 254 F F F 2 040-059 118 118 118 242 242 242 F F F 3 060-079 116 116 116 238 238 238 F F F 4 080-099 122 122 122 251 251 251 F F F 5 100-119 120 120 120 249 249 249 F F F 6 120-139 122 122 122 252 252 252 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 122 122 122 251 251 251 F F F 9 180-199 120 120 120 249 249 249 F F F 10 200-219 123 123 123 253 253 253 F F F 11 220-239 123 123 123 253 253 253 F F F 12 240-259 113 113 113 230 230 230 F F F 13 260-279 125 125 125 256 256 256 F F F 14 280-299 120 120 120 249 249 249 F F F 15 300-319 116 116 116 238 238 238 F F F 16 320-339 119 119 119 246 246 246 F F F 17 340-359 115 115 115 237 237 237 F F F 18 360-379 122 122 122 252 252 252 F F F 19 380-399 124 124 124 254 254 254 F F F 20 400-419 119 119 119 246 246 246 F F F 21 420-439 124 124 124 254 254 254 F F F 22 440-459 124 124 124 255 255 255 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 123 123 123 253 253 253 F F F 25 500-519 122 122 122 252 252 252 F F F 26 520-539 122 122 122 251 251 251 F F F 27 540-559 120 120 120 248 248 248 F F F 28 560-579 125 125 125 256 256 256 F F F 29 580-599 124 124 124 255 255 255 F F F Max gen: 125, Max flush: 256 Begin loop 63 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.140018821Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.140640462Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.140673941Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686} {"msg":"Crucible ca398551-1adc-4c47-a53d-585b65fe7be9 has session id: 1779098f-99e3-44fb-bb09-0fc3fa2efe70","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.140697319Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"Upstairs opts: Upstairs UUID: ca398551-1adc-4c47-a53d-585b65fe7be9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.140853712Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"Crucible stats registered with UUID: ca398551-1adc-4c47-a53d-585b65fe7be9","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.140888921Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:126","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.141173087Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686} {"msg":"ca398551-1adc-4c47-a53d-585b65fe7be9 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.141250094Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.141267803Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"0","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.141287672Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"0","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.141301591Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.141315161Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.14132847Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"2","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.141349349Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"2","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.141363268Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"io task","client":"0","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.141376958Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"io task","client":"2","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.141390707Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"io task","client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"ds_connection connected from Ok(127.0.0.1:63630)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.141578928Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"io task","client":"0","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:52:22.141628896Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"io task","client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:52:22.141663065Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"io task","client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:52:22.141717652Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.141740561Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.14176189Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"ds_connection connected from Ok(127.0.0.1:61135)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.141783769Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"io task","client":"2","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.141805308Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"io task","client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.1423919Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"0","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.142420399Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"0","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.142436228Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"2","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.142459367Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"2","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.142995882Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"0","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.143130186Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"2","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.1434825Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"0","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.143521798Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"0","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.143537167Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.143602934Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"2","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.143621723Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"2","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:22.143635242Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.14290106Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"io task","client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"ds_connection connected from Ok(127.0.0.1:50034)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.143075792Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"io task","client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.14418831Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.144234428Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.144720395Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145238991Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145266379Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"[0]R flush_numbers[0..12]: [253, 254, 242, 238, 251, 249, 252, 232, 251, 249, 253, 253]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145281889Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"downstairs","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"[0]R generation[0..12]: [123, 124, 118, 116, 122, 120, 122, 114, 122, 120, 123, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145296508Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"downstairs","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145310608Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"downstairs","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"[1]R flush_numbers[0..12]: [253, 254, 242, 238, 251, 249, 252, 232, 251, 249, 253, 253]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145324347Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"downstairs","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"[1]R generation[0..12]: [123, 124, 118, 116, 122, 120, 122, 114, 122, 120, 123, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145338076Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"downstairs","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145352035Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"downstairs","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"[2]R flush_numbers[0..12]: [253, 254, 242, 238, 251, 249, 252, 232, 251, 249, 253, 253]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145365995Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"downstairs","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"[2]R generation[0..12]: [123, 124, 118, 116, 122, 120, 122, 114, 122, 120, 123, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145379694Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"downstairs","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145393424Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"downstairs","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"Max found gen is 126","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145406233Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"downstairs","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"Generation requested: 126 >= found:126","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145419242Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"downstairs","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"Next flush: 257","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:52:32.145431982Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"downstairs","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145446391Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"":"downstairs","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.14546199Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.14547437Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"0","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145487609Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"1","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145500939Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"client":"2","":"downstairs","session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145514468Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145526497Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"ca398551-1adc-4c47-a53d-585b65fe7be9 is now active with session: 1779098f-99e3-44fb-bb09-0fc3fa2efe70","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145539347Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145551176Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686,"session_id":"1779098f-99e3-44fb-bb09-0fc3fa2efe70"} {"msg":"The guest has finished waiting for activation with:126","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:32.145563366Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 507 len 20480 data: 15 14 10 11 12 02/30 Read block 254 len 32768 03/30 Flush 04/30 Write block 118 len 36864 data: 9 10 11 9 10 10 8 8 7 05/30 Write block 398 len 28672 data: 15 13 11 11 11 10 11 06/30 Read block 325 len 24576 07/30 Write block 191 len 4096 data: 14 08/30 Read block 448 len 36864 09/30 Read block 346 len 20480 10/30 Read block 66 len 40960 11/30 Read block 386 len 20480 12/30 Read block 232 len 24576 13/30 Read block 367 len 32768 14/30 Write block 203 len 20480 data: 10 12 11 12 13 15/30 Write block 76 len 8192 data: 13 13 16/30 Write block 259 len 32768 data: 12 11 10 14 13 11 12 11 17/30 Read block 131 len 20480 18/30 Write block 118 len 8192 data: 10 11 19/30 Write block 78 len 40960 data: 13 13 16 16 13 13 16 15 14 11 20/30 Write block 174 len 8192 data: 7 9 21/30 Write block 416 len 4096 data: 9 22/30 Write block 378 len 36864 data: 16 16 13 13 13 12 15 11 10 23/30 Write block 135 len 8192 data: 10 10 24/30 Read block 570 len 20480 25/30 Flush 26/30 Write block 527 len 4096 data: 8 27/30 Write block 327 len 24576 data: 8 11 10 10 10 10 28/30 Write block 307 len 20480 data: 10 11 11 10 11 29/30 Read block 337 len 32768 30/30 Write block 372 len 28672 data: 12 11 9 11 13 14 17 ---------------------------------------------------------------- Crucible gen:126 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Write 1 Done Done Done false 28 Acked 1027 Write 6 Done Done Done false 29 Acked 1028 Write 5 Done Done Done false 30 Acked 1029 Read 8 Done Done Done false 31 Acked 1030 Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:52:32.172495189Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1686} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 123 123 123 253 253 253 F F F 1 020-039 124 124 124 254 254 254 F F F 2 040-059 118 118 118 242 242 242 F F F 3 060-079 126 126 126 258 258 258 F F F 4 080-099 126 126 126 258 258 258 F F F 5 100-119 126 126 126 258 258 258 F F F 6 120-139 126 126 126 259 259 259 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 126 126 126 258 258 258 F F F 9 180-199 126 126 126 258 258 258 F F F 10 200-219 126 126 126 258 258 258 F F F 11 220-239 123 123 123 253 253 253 F F F 12 240-259 126 126 126 258 258 258 F F F 13 260-279 126 126 126 258 258 258 F F F 14 280-299 120 120 120 249 249 249 F F F 15 300-319 116 116 116 238 238 238 T T T 16 320-339 119 119 119 246 246 246 T T T 17 340-359 115 115 115 237 237 237 F F F 18 360-379 126 126 126 259 259 259 T T T 19 380-399 126 126 126 259 259 259 F F F 20 400-419 126 126 126 258 258 258 F F F 21 420-439 124 124 124 254 254 254 F F F 22 440-459 124 124 124 255 255 255 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 123 123 123 253 253 253 F F F 25 500-519 126 126 126 257 257 257 F F F 26 520-539 122 122 122 251 251 251 T T T 27 540-559 120 120 120 248 248 248 F F F 28 560-579 125 125 125 256 256 256 F F F 29 580-599 124 124 124 255 255 255 F F F Max gen: 126, Max flush: 259 A Difference in extent metadata is expected here On loop 63 Verifying data now No change after verify Loop: 63 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 123 123 123 253 253 253 F F F 1 020-039 124 124 124 254 254 254 F F F 2 040-059 118 118 118 242 242 242 F F F 3 060-079 126 126 126 258 258 258 F F F 4 080-099 126 126 126 258 258 258 F F F 5 100-119 126 126 126 258 258 258 F F F 6 120-139 126 126 126 259 259 259 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 126 126 126 258 258 258 F F F 9 180-199 126 126 126 258 258 258 F F F 10 200-219 126 126 126 258 258 258 F F F 11 220-239 123 123 123 253 253 253 F F F 12 240-259 126 126 126 258 258 258 F F F 13 260-279 126 126 126 258 258 258 F F F 14 280-299 120 120 120 249 249 249 F F F 15 300-319 127 127 127 260 260 260 F F F 16 320-339 127 127 127 260 260 260 F F F 17 340-359 115 115 115 237 237 237 F F F 18 360-379 127 127 127 260 260 260 F F F 19 380-399 126 126 126 259 259 259 F F F 20 400-419 126 126 126 258 258 258 F F F 21 420-439 124 124 124 254 254 254 F F F 22 440-459 124 124 124 255 255 255 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 123 123 123 253 253 253 F F F 25 500-519 126 126 126 257 257 257 F F F 26 520-539 127 127 127 260 260 260 F F F 27 540-559 120 120 120 248 248 248 F F F 28 560-579 125 125 125 256 256 256 F F F 29 580-599 124 124 124 255 255 255 F F F Max gen: 127, Max flush: 260 Begin loop 64 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.392343424Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.393026032Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.393058721Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694} {"msg":"Crucible 3b96c772-fedd-4c31-8083-f8de5f86168e has session id: 41b280b5-24fc-4941-8656-55b3a6bb4e0c","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.39308297Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"Upstairs opts: Upstairs UUID: 3b96c772-fedd-4c31-8083-f8de5f86168e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.393121478Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"Crucible stats registered with UUID: 3b96c772-fedd-4c31-8083-f8de5f86168e","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.393147587Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:128","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.393382976Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694} {"msg":"3b96c772-fedd-4c31-8083-f8de5f86168e active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.39351266Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.393531439Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.393561287Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.393588946Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"1","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.393614465Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"1","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.393638684Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"2","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.393665983Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"2","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.393689221Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"io task","client":"1","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.39371131Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"io task","client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.393734689Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"io task","client":"2","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:52:46.393913941Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"io task","client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"ds_connection connected from Ok(127.0.0.1:51993)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.393949839Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"io task","client":"1","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"ds_connection connected from Ok(127.0.0.1:54172)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.393978248Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"io task","client":"2","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:52:46.394004727Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"io task","client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:52:46.394031935Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.394054164Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.394078983Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.394260145Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"io task","client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.394673256Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"1","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.394711314Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"1","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.394741102Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"2","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.394772581Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"2","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.395277797Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"2","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.395414271Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"1","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.395766254Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"2","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.395804303Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"2","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.395833271Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.39585889Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"1","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.395886829Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"1","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:46.395909128Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.395782887Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"io task","client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"ds_connection connected from Ok(127.0.0.1:38096)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.396143241Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"io task","client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.397058248Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.397091217Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.397682379Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398285411Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.39831304Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"[0]R flush_numbers[0..12]: [253, 254, 242, 258, 258, 258, 259, 232, 258, 258, 258, 253]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398331279Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"downstairs","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"[0]R generation[0..12]: [123, 124, 118, 126, 126, 126, 126, 114, 126, 126, 126, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398349108Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"downstairs","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398366727Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"downstairs","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"[1]R flush_numbers[0..12]: [253, 254, 242, 258, 258, 258, 259, 232, 258, 258, 258, 253]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398383856Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"downstairs","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"[1]R generation[0..12]: [123, 124, 118, 126, 126, 126, 126, 114, 126, 126, 126, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398400996Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"downstairs","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398418435Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"downstairs","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"[2]R flush_numbers[0..12]: [253, 254, 242, 258, 258, 258, 259, 232, 258, 258, 258, 253]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398435494Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"downstairs","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"[2]R generation[0..12]: [123, 124, 118, 126, 126, 126, 126, 114, 126, 126, 126, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398452513Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"downstairs","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398469642Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"downstairs","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {Wait for a query_work_queue command to finish before sending IO "msg":"Max found gen is 128","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398489321Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"downstairs","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"Generation requested: 128 >= found:128","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398505831Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"downstairs","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"Next flush: 261","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.39852175Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"downstairs","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398537959Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"":"downstairs","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398556728Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398571877Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"0","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398588507Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"1","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398604526Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"client":"2","":"downstairs","session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398621005Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398636034Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"3b96c772-fedd-4c31-8083-f8de5f86168e is now active with session: 41b280b5-24fc-4941-8656-55b3a6bb4e0c","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398651904Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398666613Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694,"session_id":"41b280b5-24fc-4941-8656-55b3a6bb4e0c"} {"Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 msg":"The guest has finished waiting for activation with:128","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:52:56.398689502Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 260 len 20480 data: 12 11 15 14 12 03/30 Read block 501 len 16384 04/30 Read block 434 len 4096 05/30 Read block 579 len 20480 06/30 Read block 388 len 4096 07/30 Read block 136 len 4096 08/30 Write block 45 len 4096 data: 9 09/30 Read block 156 len 36864 10/30 Write block 0 len 16384 data: 4 5 5 6 11/30 Read block 298 len 40960 12/30 Read block 195 len 40960 13/30 Flush 14/30 Write block 531 len 20480 data: 9 7 9 9 9 15/30 Write block 169 len 12288 data: 12 13 11 16/30 Write block 82 len 20480 data: 14 14 17 16 15 17/30 Read block 217 len 40960 18/30 Read block 167 len 28672 19/30 Read block 40 len 8192 20/30 Read block 392 len 20480 21/30 Read block 156 len 20480 22/30 Write block 334 len 12288 data: 11 10 9 23/30 Read block 398 len 16384 24/30 Write block 356 len 40960 data: 10 13 13 11 8 6 7 6 8 9 25/30 Read block 318 len 24576 26/30 Read block 327 len 4096 27/30 Write block 163 len 24576 data: 6 7 9 10 11 12 28/30 Write block 262 len 4096 data: 16 29/30 Read block 18 len 36864 30/30 Read block 320 len 8192 ---------------------------------------------------------------- Crucible gen:128 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 20 Acked 1019 Read 2 Done Done Done false 21 Acked 1020 Read 5 Done Done Done false 22 Acked 1021 Read 5 Done Done Done false 23 Acked 1022 Write 3 Done Done Done false 24 Acked 1023 Read 4 Done Done Done false 25 Acked 1024 Write 10 Done Done Done false 26 Acked 1025 Read 6 Done Done Done false 27 Acked 1026 Read 1 Done Done Done false 28 Acked 1027 Write 6 Done Done Done false 29 Acked 1028 Write 1 Done Done Done false 30 Acked 1029 Read 9 Done Done Done false 31 Acked 1030 Read 2 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 12 12 12 36 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1018 1018 Downstairs last five completed: 1018 1017 1016 1015 1014 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:52:56.422352898Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1694} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 262 262 262 F F F 1 020-039 124 124 124 254 254 254 F F F 2 040-059 128 128 128 262 262 262 F F F 3 060-079 126 126 126 258 258 258 F F F 4 080-099 128 128 128 263 263 263 F F F 5 100-119 126 126 126 258 258 258 F F F 6 120-139 126 126 126 259 259 259 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 128 128 128 263 263 263 T T T 9 180-199 126 126 126 258 258 258 F F F 10 200-219 126 126 126 258 258 258 F F F 11 220-239 123 123 123 253 253 253 F F F 12 240-259 126 126 126 258 258 258 F F F 13 260-279 128 128 128 262 262 262 T T T 14 280-299 120 120 120 249 249 249 F F F 15 300-319 127 127 127 260 260 260 F F F 16 320-339 127 127 127 260 260 260 T T T 17 340-359 115 115 115 237 237 237 T T T 18 360-379 127 127 127 260 260 260 T T T 19 380-399 126 126 126 259 259 259 F F F 20 400-419 126 126 126 258 258 258 F F F 21 420-439 124 124 124 254 254 254 F F F 22 440-459 124 124 124 255 255 255 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 123 123 123 253 253 253 F F F 25 500-519 126 126 126 257 257 257 F F F 26 520-539 128 128 128 263 263 263 F F F 27 540-559 120 120 120 248 248 248 F F F 28 560-579 125 125 125 256 256 256 F F F 29 580-599 124 124 124 255 255 255 F F F Max gen: 128, Max flush: 263 A Difference in extent metadata is expected here On loop 64 Verifying data now No change after verify Loop: 64 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 262 262 262 F F F 1 020-039 124 124 124 254 254 254 F F F 2 040-059 128 128 128 262 262 262 F F F 3 060-079 126 126 126 258 258 258 F F F 4 080-099 128 128 128 263 263 263 F F F 5 100-119 126 126 126 258 258 258 F F F 6 120-139 126 126 126 259 259 259 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 129 129 129 264 264 264 F F F 9 180-199 126 126 126 258 258 258 F F F 10 200-219 126 126 126 258 258 258 F F F 11 220-239 123 123 123 253 253 253 F F F 12 240-259 126 126 126 258 258 258 F F F 13 260-279 129 129 129 264 264 264 F F F 14 280-299 120 120 120 249 249 249 F F F 15 300-319 127 127 127 260 260 260 F F F 16 320-339 129 129 129 264 264 264 F F F 17 340-359 129 129 129 264 264 264 F F F 18 360-379 129 129 129 264 264 264 F F F 19 380-399 126 126 126 259 259 259 F F F 20 400-419 126 126 126 258 258 258 F F F 21 420-439 124 124 124 254 254 254 F F F 22 440-459 124 124 124 255 255 255 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 123 123 123 253 253 253 F F F 25 500-519 126 126 126 257 257 257 F F F 26 520-539 128 128 128 263 263 263 F F F 27 540-559 120 120 120 248 248 248 F F F 28 560-579 125 125 125 256 256 256 F F F 29 580-599 124 124 124 255 255 255 F F F Max gen: 129, Max flush: 264 Begin loop 65 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.654955043Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.655539396Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.655570695Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702} {"msg":"Crucible 24923207-fb6d-4151-a6c7-89615ada12c6 has session id: c2912251-31fa-4095-bc52-c40d9ff836c7","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.655709938Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"Upstairs opts: Upstairs UUID: 24923207-fb6d-4151-a6c7-89615ada12c6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.655743927Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"Crucible stats registered with UUID: 24923207-fb6d-4151-a6c7-89615ada12c6","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.655759416Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:130","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.656049002Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702} {"msg":"24923207-fb6d-4151-a6c7-89615ada12c6 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.656079791Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.65610076Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"0","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.656131358Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"0","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.656153587Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"1","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.656175526Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"1","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.656201635Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.656223654Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.656244053Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"io task","client":"0","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.656265992Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"io task","client":"1","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.656288561Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"io task","client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:53:10.656331709Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"io task","client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:53:10.656358658Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"io task","client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:53:10.656448784Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.656484542Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.656567088Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"ds_connection connected from Ok(127.0.0.1:51563)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.656595717Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"io task","client":"0","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"ds_connection connected from Ok(127.0.0.1:64654)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.656625845Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"io task","client":"1","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.656673653Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"io task","client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.657256036Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"1","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.657288735Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"1","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.657319923Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"0","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.657347382Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"0","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.657962743Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"0","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.657990242Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"1","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.658532186Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"1","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.658559425Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"1","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.658576744Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.658596673Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"0","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.658613383Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"0","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:10.658629832Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.657395891Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"io task","client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"ds_connection connected from Ok(127.0.0.1:50144)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.657701917Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"io task","client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.658952909Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.658981827Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.659497213Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.6600001Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660026268Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"[0]R flush_numbers[0..12]: [262, 254, 262, 258, 263, 258, 259, 232, 264, 258, 258, 253]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660044847Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"downstairs","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"[0]R generation[0..12]: [128, 124, 128, 126, 128, 126, 126, 114, 129, 126, 126, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660062527Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"downstairs","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660080146Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"downstairs","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"[1]R flush_numbers[0..12]: [262, 254, 262, 258, 263, 258, 259, 232, 264, 258, 258, 253]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660097095Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"downstairs","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"[1]R generation[0..12]: [128, 124, 128, 126, 128, 126, 126, 114, 129, 126, 126, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660114084Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"downstairs","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660131534Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"downstairs","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"[2]R flush_numbers[0..12]: [262, 254, 262, 258, 263, 258, 259, 232, 264, 258, 258, 253]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660148603Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"downstairs","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"[2]R generation[0..12]: [128, 124, 128, 126, 128, 126, 126, 114, 129, 126, 126, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660165722Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"downstairs","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:53:20.660183831Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"downstairs","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"Max found gen is 130","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.66019992Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"downstairs","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"Generation requested: 130 >= found:130","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.66021651Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"downstairs","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"Next flush: 265","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660232339Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"downstairs","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660251698Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"":"downstairs","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660268327Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660283196Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"0","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660299396Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"1","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660315855Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"client":"2","":"downstairs","session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660332194Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660347433Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"24923207-fb6d-4151-a6c7-89615ada12c6 is now active with session: c2912251-31fa-4095-bc52-c40d9ff836c7","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660363983Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.660389821Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702,"session_id":"c2912251-31fa-4095-bc52-c40d9ff836c7"} {"msg":"The guest has finished waiting for activation with:130","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:20.66041629Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 530 len 16384 data: 8 10 8 10 02/30 Read block 74 len 8192 03/30 Write block 574 len 40960 data: 17 14 13 12 12 9 9 9 11 9 04/30 Write block 417 len 40960 data: 11 13 16 16 13 11 10 7 11 12 05/30 Read block 412 len 12288 06/30 Write block 121 len 8192 data: 10 11 07/30 Write block 248 len 8192 data: 11 12 08/30 Flush 09/30 Write block 127 len 24576 data: 9 10 9 10 10 9 10/30 Read block 218 len 24576 11/30 Read block 580 len 4096 12/30 Write block 197 len 20480 data: 13 11 8 9 11 13/30 Read block 147 len 40960 14/30 Write block 161 len 4096 data: 9 15/30 Read block 494 len 12288 16/30 Write block 532 len 24576 data: 9 11 10 10 9 9 17/30 Read block 563 len 24576 18/30 Write block 383 len 36864 data: 13 16 12 11 12 10 8 11 12 19/30 Read block 440 len 20480 20/30 Write block 397 len 4096 data: 15 21/30 Read block 271 len 8192 22/30 Read block 245 len 36864 23/30 Read block 89 len 20480 24/30 Write block 310 len 28672 data: 11 12 9 14 18 17 16 25/30 Read block 551 len 24576 26/30 Read block 545 len 28672 27/30 Write block 492 len 8192 data: 8 5 28/30 Read block 271 len 4096 29/30 Read block 363 len 8192 30/30 Read block 524 len 28672 ---------------------------------------------------------------- Crucible gen:130 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 27 Acked 1026 Read 7 Done Done Done false 28 Acked 1027 Write 2 Done Done Done false 29 Acked 1028 Read 1 Done Done Done false 30 Acked 1029 Read 2 Done Done Done false 31 Acked 1030 Read 7 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 5 5 5 15 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1025 1025 1025 Downstairs last five completed: 1025 1024 1023 1022 1021 Upstairs last five completed: 31 30 29 28 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:53:20.686759411Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1702} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 262 262 262 F F F 1 020-039 124 124 124 254 254 254 F F F 2 040-059 128 128 128 262 262 262 F F F 3 060-079 126 126 126 258 258 258 F F F 4 080-099 128 128 128 263 263 263 F F F 5 100-119 126 126 126 258 258 258 F F F 6 120-139 130 130 130 266 266 266 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 130 130 130 266 266 266 F F F 9 180-199 130 130 130 266 266 266 F F F 10 200-219 130 130 130 266 266 266 F F F 11 220-239 123 123 123 253 253 253 F F F 12 240-259 130 130 130 265 265 265 F F F 13 260-279 129 129 129 264 264 264 F F F 14 280-299 120 120 120 249 249 249 F F F 15 300-319 130 130 130 266 266 266 F F F 16 320-339 129 129 129 264 264 264 F F F 17 340-359 129 129 129 264 264 264 F F F 18 360-379 129 129 129 264 264 264 F F F 19 380-399 130 130 130 266 266 266 F F F 20 400-419 130 130 130 265 265 265 F F F 21 420-439 130 130 130 265 265 265 F F F 22 440-459 124 124 124 255 255 255 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 123 123 123 253 253 253 T T T 25 500-519 126 126 126 257 257 257 F F F 26 520-539 130 130 130 266 266 266 F F F 27 540-559 120 120 120 248 248 248 F F F 28 560-579 130 130 130 265 265 265 F F F 29 580-599 130 130 130 265 265 265 F F F Max gen: 130, Max flush: 266 A Difference in extent metadata is expected here On loop 65 Verifying data now No change after verify Loop: 65 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 262 262 262 F F F 1 020-039 124 124 124 254 254 254 F F F 2 040-059 128 128 128 262 262 262 F F F 3 060-079 126 126 126 258 258 258 F F F 4 080-099 128 128 128 263 263 263 F F F 5 100-119 126 126 126 258 258 258 F F F 6 120-139 130 130 130 266 266 266 F F F 7 140-159 114 114 114 232 232 232 F F F 8 160-179 130 130 130 266 266 266 F F F 9 180-199 130 130 130 266 266 266 F F F 10 200-219 130 130 130 266 266 266 F F F 11 220-239 123 123 123 253 253 253 F F F 12 240-259 130 130 130 265 265 265 F F F 13 260-279 129 129 129 264 264 264 F F F 14 280-299 120 120 120 249 249 249 F F F 15 300-319 130 130 130 266 266 266 F F F 16 320-339 129 129 129 264 264 264 F F F 17 340-359 129 129 129 264 264 264 F F F 18 360-379 129 129 129 264 264 264 F F F 19 380-399 130 130 130 266 266 266 F F F 20 400-419 130 130 130 265 265 265 F F F 21 420-439 130 130 130 265 265 265 F F F 22 440-459 124 124 124 255 255 255 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 131 131 131 267 267 267 F F F 25 500-519 126 126 126 257 257 257 F F F 26 520-539 130 130 130 266 266 266 F F F 27 540-559 120 120 120 248 248 248 F F F 28 560-579 130 130 130 265 265 265 F F F 29 580-599 130 130 130 265 265 265 F F F Max gen: 131, Max flush: 267 Begin loop 66 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.860420409Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.860737514Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.860763963Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710} {"msg":"Crucible c8594696-fd39-4646-8f68-d3238f316131 has session id: a51cfa22-dff6-44ba-8c4b-822edcea9483","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.861096897Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"Upstairs opts: Upstairs UUID: c8594696-fd39-4646-8f68-d3238f316131, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.861131746Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"Crucible stats registered with UUID: c8594696-fd39-4646-8f68-d3238f316131","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.861147965Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:132","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.861536607Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710} {"msg":"c8594696-fd39-4646-8f68-d3238f316131 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.861573515Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.861587364Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"0","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.861613083Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"0","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.861639002Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"1","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.861664721Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"1","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.861692189Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.861715858Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.861741057Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"io task","client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.861765986Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"io task","client":"1","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.861790325Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"io task","client":"0","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"ds_connection connected from Ok(127.0.0.1:46166)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.862081331Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"io task","client":"1","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"ds_connection connected from Ok(127.0.0.1:34238)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.862159767Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"io task","client":"0","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:53:30.8623241Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"io task","client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:53:30.862354958Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"io task","client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:53:30.862411586Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.862443544Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.862466873Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.862488402Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"io task","client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.862816037Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"1","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.862849545Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"1","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.862872254Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"0","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.862893263Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"0","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.863440248Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"1","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.863582281Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"0","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.863902806Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"1","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.863925645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"1","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.863939544Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.864024711Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"0","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.86404383Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"0","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:30.864057239Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.863730358Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"io task","client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"ds_connection connected from Ok(127.0.0.1:58057)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.863841882Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"io task","client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.865118353Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.865152701Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.865685776Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866211552Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866232071Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"[0]R flush_numbers[0..12]: [262, 254, 262, 258, 263, 258, 266, 232, 266, 266, 266, 253]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.86624682Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"downstairs","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"[0]R generation[0..12]: [128, 124, 128, 126, 128, 126, 130, 114, 130, 130, 130, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866261389Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"downstairs","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866275349Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"downstairs","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"[1]R flush_numbers[0..12]: [262, 254, 262, 258, 263, 258, 266, 232, 266, 266, 266, 253]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866289178Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"downstairs","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"[1]R generation[0..12]: [128, 124, 128, 126, 128, 126, 130, 114, 130, 130, 130, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866302947Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"downstairs","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866317047Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"downstairs","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"[2]R flush_numbers[0..12]: [262, 254, 262, 258, 263, 258, 266, 232, 266, 266, 266, 253]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866330816Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"downstairs","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"[2]R generation[0..12]: [128, 124, 128, 126, 128, 126, 130, 114, 130, 130, 130, 123]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866344585Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"downstairs","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866358345Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"downstairs","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"Max found gen is 132","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866371074Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"downstairs","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"Generation requested: 132 >= found:132","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866384314Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"downstairs","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"Next flush: 268","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2024-07-31T18:53:40.866397173Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"downstairs","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866417392Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"":"downstairs","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866431092Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866443071Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"0","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.86645651Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"1","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.86646985Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"client":"2","":"downstairs","session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866482879Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866494889Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"c8594696-fd39-4646-8f68-d3238f316131 is now active with session: a51cfa22-dff6-44ba-8c4b-822edcea9483","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866507928Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866519927Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710,"session_id":"a51cfa22-dff6-44ba-8c4b-822edcea9483"} {"msg":"The guest has finished waiting for activation with:132","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:40.866531967Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 524 len 40960 data: 12 9 9 9 8 7 9 11 10 12 02/30 Write block 383 len 40960 data: 14 17 13 12 13 11 9 12 13 15 03/30 Read block 291 len 36864 04/30 Read block 353 len 32768 05/30 Flush 06/30 Write block 595 len 20480 data: 8 6 6 6 3 07/30 Read block 563 len 16384 08/30 Read block 127 len 12288 09/30 Write block 577 len 4096 data: 13 10/30 Read block 570 len 28672 11/30 Write block 71 len 4096 data: 4 12/30 Write block 116 len 4096 data: 10 13/30 Read block 548 len 12288 14/30 Write block 354 len 20480 data: 13 12 11 14 14 15/30 Read block 522 len 20480 16/30 Read block 267 len 16384 17/30 Flush 18/30 Write block 523 len 28672 data: 10 13 10 10 10 9 8 19/30 Read block 358 len 32768 20/30 Write block 174 len 24576 data: 8 10 8 6 7 9 21/30 Write block 143 len 20480 data: 12 12 12 11 9 22/30 Flush 23/30 Write block 318 len 24576 data: 13 11 11 8 7 8 24/30 Read block 27 len 20480 25/30 Read block 301 len 12288 26/30 Read block 534 len 8192 27/30 Read block 541 len 36864 28/30 Read block 448 len 40960 29/30 Read block 117 len 36864 30/30 Write block 227 len 12288 data: 9 10 11 ---------------------------------------------------------------- Crucible gen:132 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 23 Acked 1022 Write 6 Done Done Done false 24 Acked 1023 Read 5 Done Done Done false 25 Acked 1024 Read 3 Done Done Done false 26 Acked 1025 Read 2 Done Done Done false 27 Acked 1026 Read 9 Done Done Done false 28 Acked 1027 Read 10 Done Done Done false 29 Acked 1028 Read 9 Done Done Done false 30 Acked 1029 Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1021 1021 1021 Downstairs last five completed: 1021 1020 1019 1018 1017 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:53:40.894409576Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1710} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 262 262 262 F F F 1 020-039 124 124 124 254 254 254 F F F 2 040-059 128 128 128 262 262 262 F F F 3 060-079 132 132 132 269 269 269 F F F 4 080-099 128 128 128 263 263 263 F F F 5 100-119 132 132 132 269 269 269 F F F 6 120-139 130 130 130 266 266 266 F F F 7 140-159 132 132 132 270 270 270 F F F 8 160-179 132 132 132 270 270 270 F F F 9 180-199 130 130 130 266 266 266 F F F 10 200-219 130 130 130 266 266 266 F F F 11 220-239 123 123 123 253 253 253 T T T 12 240-259 130 130 130 265 265 265 F F F 13 260-279 129 129 129 264 264 264 F F F 14 280-299 120 120 120 249 249 249 F F F 15 300-319 130 130 130 266 266 266 T T T 16 320-339 129 129 129 264 264 264 T T T 17 340-359 132 132 132 269 269 269 F F F 18 360-379 129 129 129 264 264 264 F F F 19 380-399 132 132 132 268 268 268 F F F 20 400-419 130 130 130 265 265 265 F F F 21 420-439 130 130 130 265 265 265 F F F 22 440-459 124 124 124 255 255 255 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 131 131 131 267 267 267 F F F 25 500-519 126 126 126 257 257 257 F F F 26 520-539 132 132 132 270 270 270 F F F 27 540-559 120 120 120 248 248 248 F F F 28 560-579 132 132 132 269 269 269 F F F 29 580-599 132 132 132 269 269 269 F F F Max gen: 132, Max flush: 270 A Difference in extent metadata is expected here On loop 66 Verifying data now No change after verify Loop: 66 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 262 262 262 F F F 1 020-039 124 124 124 254 254 254 F F F 2 040-059 128 128 128 262 262 262 F F F 3 060-079 132 132 132 269 269 269 F F F 4 080-099 128 128 128 263 263 263 F F F 5 100-119 132 132 132 269 269 269 F F F 6 120-139 130 130 130 266 266 266 F F F 7 140-159 132 132 132 270 270 270 F F F 8 160-179 132 132 132 270 270 270 F F F 9 180-199 130 130 130 266 266 266 F F F 10 200-219 130 130 130 266 266 266 F F F 11 220-239 133 133 133 271 271 271 F F F 12 240-259 130 130 130 265 265 265 F F F 13 260-279 129 129 129 264 264 264 F F F 14 280-299 120 120 120 249 249 249 F F F 15 300-319 133 133 133 271 271 271 F F F 16 320-339 133 133 133 271 271 271 F F F 17 340-359 132 132 132 269 269 269 F F F 18 360-379 129 129 129 264 264 264 F F F 19 380-399 132 132 132 268 268 268 F F F 20 400-419 130 130 130 265 265 265 F F F 21 420-439 130 130 130 265 265 265 F F F 22 440-459 124 124 124 255 255 255 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 131 131 131 267 267 267 F F F 25 500-519 126 126 126 257 257 257 F F F 26 520-539 132 132 132 270 270 270 F F F 27 540-559 120 120 120 248 248 248 F F F 28 560-579 132 132 132 269 269 269 F F F 29 580-599 132 132 132 269 269 269 F F F Max gen: 133, Max flush: 271 Begin loop 67 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.100597555Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.101052784Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.101084222Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718} {"msg":"Crucible c173904c-9afd-4369-adc9-f04351aeda68 has session id: f079505e-95a9-4605-aa7f-ca26f43b65d2","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.101403617Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"Upstairs opts: Upstairs UUID: c173904c-9afd-4369-adc9-f04351aeda68, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.101458895Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"Crucible stats registered with UUID: c173904c-9afd-4369-adc9-f04351aeda68","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.101487793Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:134","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.101822988Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718} {"msg":"c173904c-9afd-4369-adc9-f04351aeda68 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.101901304Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.101927013Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"0","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.101949222Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"0","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.101963941Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.10198881Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.102015899Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"2","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.102035828Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"2","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.102050067Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"io task","client":"0","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.102065516Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"io task","client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.102087805Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"io task","client":"2","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:53:55.102264207Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"io task","client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:53:55.102314775Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"io task","client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"ds_connection connected from Ok(127.0.0.1:45171)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.102341933Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"io task","client":"0","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:53:55.102371312Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.102393371Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.10241408Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"ds_connection connected from Ok(127.0.0.1:35136)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.102437819Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"io task","client":"2","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.102582932Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"io task","client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.103086529Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"2","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.103117257Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"2","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.103134436Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"0","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.103158105Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"0","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.10370242Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"0","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.103825574Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"2","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.104262554Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"0","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.104286803Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"0","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.104300822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.104327691Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"2","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.10434162Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"2","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:53:55.10435489Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.103271571Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"io task","client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"ds_connection connected from Ok(127.0.0.1:44734)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.103436673Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"io task","client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.104678146Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.104727623Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.105370953Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.105972205Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.105994514Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"[0]R flush_numbers[0..12]: [262, 254, 262, 269, 263, 269, 266, 270, 270, 266, 266, 271]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106009773Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"downstairs","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"[0]R generation[0..12]: [128, 124, 128, 132, 128, 132, 130, 132, 132, 130, 130, 133]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106024203Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"downstairs","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106038542Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"downstairs","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"[1]R flush_numbers[0..12]: [262, 254, 262, 269, 263, 269, 266, 270, 270, 266, 266, 271]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106052362Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"downstairs","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"[1]R generation[0..12]: [128, 124, 128, 132, 128, 132, 130, 132, 132, 130, 130, 133]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106066351Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"downstairs","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.10608029Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"downstairs","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"[2]R flush_numbers[0..12]: [262, 254, 262, 269, 263, 269, 266, 270, 270, 266, 266, 271]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106094509Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"downstairs","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"[2]R generation[0..12]: [128, 124, 128, 132, 128, 132, 130, 132, 132, 130, 130, 133]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106108329Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"downstairs","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106122438Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"downstairs","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"Max found gen is 134","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106135358Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"downstairs","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"Generation requested: 134 >= found:134","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106148577Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":Wait for a query_work_queue command to finish before sending IO 1718,"":"downstairs","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"Next flush: 272","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106167896Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"downstairs","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106180965Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"":"downstairs","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106194105Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106206174Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"0","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106219354Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"1","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106232443Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"client":"2","":"downstairs","session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106245602Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106257802Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"c173904c-9afd-4369-adc9-f04351aeda68 is now active with session: f079505e-95a9-4605-aa7f-ca26f43b65d2","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106270471Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.106282381Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718,"session_id":"f079505e-95a9-4605-aa7f-ca26f43b65d2"} {"msg":"The guest has finished waiting for activation with:134","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:05.10629457Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 100 len 12288 data: 13 13 11 02/30 Read block 90 len 40960 03/30 Write block 589 len 40960 data: 7 8 8 8 9 9 9 7 7 7 04/30 Flush 05/30 Write block 566 len 16384 data: 17 15 14 11 06/30 Read block 507 len 20480 07/30 Read block 416 len 8192 08/30 Read block 262 len 20480 09/30 Read block 452 len 8192 10/30 Flush 11/30 Write block 372 len 16384 data: 13 12 10 12 12/30 Read block 520 len 4096 13/30 Write block 277 len 32768 data: 13 12 12 13 9 8 7 8 14/30 Read block 496 len 8192 15/30 Read block 548 len 20480 16/30 Read block 566 len 8192 17/30 Write block 584 len 4096 data: 11 18/30 Read block 234 len 20480 19/30 Write block 384 len 24576 data: 18 14 13 14 12 10 20/30 Read block 556 len 16384 21/30 Read block 466 len 4096 22/30 Read block 240 len 36864 23/30 Write block 345 len 12288 data: 7 6 5 24/30 Write block 15 len 8192 data: 6 6 25/30 Write block 538 len 20480 data: 9 9 10 11 11 26/30 Write block 185 len 12288 data: 12 13 16 27/30 Read block 179 len 12288 28/30 Write block 206 len 16384 data: 13 14 14 10 29/30 Read block 440 len 20480 30/30 Read block 72 len 4096 ---------------------------------------------------------------- Crucible gen:134 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Read 1 Done Done Done false 23 Acked 1022 Read 9 Done Done Done false 24 Acked 1023 Write 3 Done Done Done false 25 Acked 1024 Write 2 Done Done Done false 26 Acked 1025 Write 5 Done Done Done false 27 Acked 1026 Write 3 Done Done Done false 28 Acked 1027 Read 3 Done Done Done false 29 Acked 1028 Write 4 Done Done Done false 30 Acked 1029 Read 5 Done Done Done false 31 Acked 1030 Read 1 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 1 2 Done 10 9 9 28 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:54:05.131762122Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1718} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 128 128 128 262 262 262 T T T 1 020-039 124 124 124 254 254 254 F F F 2 040-059 128 128 128 262 262 262 F F F 3 060-079 132 132 132 269 269 269 F F F 4 080-099 128 128 128 263 263 263 F F F 5 100-119 134 134 134 272 272 272 F F F 6 120-139 130 130 130 266 266 266 F F F 7 140-159 132 132 132 270 270 270 F F F 8 160-179 132 132 132 270 270 270 F F F 9 180-199 130 130 130 266 266 266 T T T 10 200-219 130 130 130 266 266 266 T T T 11 220-239 133 133 133 271 271 271 F F F 12 240-259 130 130 130 265 265 265 F F F 13 260-279 134 134 134 274 274 274 F F F 14 280-299 134 134 134 274 274 274 F F F 15 300-319 133 133 133 271 271 271 F F F 16 320-339 133 133 133 271 271 271 F F F 17 340-359 132 132 132 269 269 269 T T T 18 360-379 134 134 134 274 274 274 F F F 19 380-399 134 134 134 274 274 274 F F F 20 400-419 130 130 130 265 265 265 F F F 21 420-439 130 130 130 265 265 265 F F F 22 440-459 124 124 124 255 255 255 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 131 131 131 267 267 267 F F F 25 500-519 126 126 126 257 257 257 F F F 26 520-539 132 132 132 270 270 270 T T T 27 540-559 120 120 120 248 248 248 T T T 28 560-579 134 134 134 273 273 273 F F F 29 580-599 134 134 134 274 274 274 F F F Max gen: 134, Max flush: 274 A Difference in extent metadata is expected here On loop 67 Verifying data now No change after verify Loop: 67 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 275 275 275 F F F 1 020-039 124 124 124 254 254 254 F F F 2 040-059 128 128 128 262 262 262 F F F 3 060-079 132 132 132 269 269 269 F F F 4 080-099 128 128 128 263 263 263 F F F 5 100-119 134 134 134 272 272 272 F F F 6 120-139 130 130 130 266 266 266 F F F 7 140-159 132 132 132 270 270 270 F F F 8 160-179 132 132 132 270 270 270 F F F 9 180-199 135 135 135 275 275 275 F F F 10 200-219 135 135 135 275 275 275 F F F 11 220-239 133 133 133 271 271 271 F F F 12 240-259 130 130 130 265 265 265 F F F 13 260-279 134 134 134 274 274 274 F F F 14 280-299 134 134 134 274 274 274 F F F 15 300-319 133 133 133 271 271 271 F F F 16 320-339 133 133 133 271 271 271 F F F 17 340-359 135 135 135 275 275 275 F F F 18 360-379 134 134 134 274 274 274 F F F 19 380-399 134 134 134 274 274 274 F F F 20 400-419 130 130 130 265 265 265 F F F 21 420-439 130 130 130 265 265 265 F F F 22 440-459 124 124 124 255 255 255 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 131 131 131 267 267 267 F F F 25 500-519 126 126 126 257 257 257 F F F 26 520-539 135 135 135 275 275 275 F F F 27 540-559 135 135 135 275 275 275 F F F 28 560-579 134 134 134 273 273 273 F F F 29 580-599 134 134 134 274 274 274 F F F Max gen: 135, Max flush: 275 Begin loop 68 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.382949533Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.383666639Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.383700608Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726} {"msg":"Crucible affebfaa-5bf0-45d6-b750-2ba99684b7d2 has session id: 912619c0-efac-4c20-bef0-db106d304278","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.383723667Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"Upstairs opts: Upstairs UUID: affebfaa-5bf0-45d6-b750-2ba99684b7d2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.383755195Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"Crucible stats registered with UUID: affebfaa-5bf0-45d6-b750-2ba99684b7d2","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.383777004Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"session_id":"912619c0-efac-4c20-bef0-db106d304278"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:136","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.38408452Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726} {"msg":"affebfaa-5bf0-45d6-b750-2ba99684b7d2 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.384129428Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.384153316Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"0","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.384181025Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"0","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.384200394Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.384214954Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.384228393Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"2","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.384241662Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"2","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.384254912Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"io task","client":"0","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.384268681Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"io task","client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.38428231Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"io task","client":"2","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:54:19.384522409Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"io task","client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"ds_connection connected from Ok(127.0.0.1:58706)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.384561917Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"io task","client":"2","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:54:19.384594266Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"io task","client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"ds_connection connected from Ok(127.0.0.1:55570)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.384668112Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"io task","client":"0","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:54:19.384700681Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.38472224Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.384744379Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.384922761Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"io task","client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.385393249Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"2","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.385420207Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"2","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.385435657Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"0","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.385449786Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"0","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.385932704Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"0","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.386036309Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"2","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.38644678Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"0","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.386473898Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"0","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.386489207Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.386547965Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"2","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.386567664Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"2","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:19.386581153Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.386156957Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"io task","client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"ds_connection connected from Ok(127.0.0.1:34674)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.386339188Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"io task","client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.387489715Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.387527663Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.388201101Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.388902559Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.388928858Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"[0]R flush_numbers[0..12]: [275, 254, 262, 269, 263, 272, 266, 270, 270, 275, 275, 271]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.388947887Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"downstairs","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"[0]R generation[0..12]: [135, 124, 128, 132, 128, 134, 130, 132, 132, 135, 135, 133]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.388965866Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"downstairs","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.388983625Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"downstairs","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"[1]R flush_numbers[0..12]: [275, 254, 262, 269, 263, 272, 266, 270, 270, 275, 275, 271]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389001024Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"downstairs","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"[1]R generation[0..12]: [135, 124, 128, 132, 128, 134, 130, 132, 132, 135, 135, 133]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389018873Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"downstairs","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389036462Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"downstairs","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"[2]R flush_numbers[0..12]: [275, 254, 262, 269, 263, 272, 266, 270, 270, 275, 275, 271]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389053672Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"downstairs","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"[2]R generation[0..12]: [135, 124, 128, 132, 128, 134, 130, 132, 132, 135, 135, 133]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389070731Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"downstairs","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"Wait for a query_work_queue command to finish before sending IO [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.38909138Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"downstairs","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"Max found gen is 136","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389108739Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"downstairs","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"Generation requested: 136 >= found:136","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389131378Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"downstairs","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"Next flush: 276","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389148247Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"downstairs","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389164347Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"":"downstairs","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389180676Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389195525Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"0","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389212134Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"1","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389228713Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"client":"2","":"downstairs","session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389245513Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389260772Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"affebfaa-5bf0-45d6-b750-2ba99684b7d2 is now active with session: 912619c0-efac-4c20-bef0-db106d304278","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389276921Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726,"session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.38929179Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 1726,"session_id":"912619c0-efac-4c20-bef0-db106d304278"} {"msg":"The guest has finished waiting for activation with:136","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:29.389314329Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 542 len 32768 data: 12 9 12 10 9 9 10 8 02/30 Read block 130 len 12288 03/30 Write block 435 len 36864 data: 11 9 7 6 8 7 11 13 12 04/30 Read block 510 len 4096 05/30 Write block 445 len 8192 data: 17 13 06/30 Read block 27 len 8192 07/30 Write block 142 len 24576 data: 10 13 13 13 12 10 08/30 Write block 194 len 4096 data: 11 09/30 Write block 272 len 4096 data: 7 10/30 Read block 360 len 40960 11/30 Read block 367 len 20480 12/30 Read block 94 len 16384 13/30 Read block 100 len 40960 14/30 Write block 150 len 36864 data: 12 8 8 9 9 8 8 9 9 15/30 Write block 0 len 16384 data: 5 6 6 7 16/30 Read block 255 len 4096 17/30 Write block 431 len 4096 data: 17 18/30 Flush 19/30 Flush 20/30 Write block 85 len 12288 data: 17 16 12 21/30 Write block 166 len 24576 data: 11 12 13 13 14 12 22/30 Read block 61 len 16384 23/30 Read block 220 len 20480 24/30 Write block 311 len 24576 data: 13 10 15 19 18 17 25/30 Write block 45 len 4096 data: 10 26/30 Write block 510 len 8192 data: 12 13 27/30 Read block 283 len 24576 28/30 Write block 14 len 32768 data: 5 7 7 8 7 6 8 10 29/30 Read block 132 len 12288 30/30 Write block 414 len 36864 data: 8 9 10 12 14 17 17 14 12 ---------------------------------------------------------------- Crucible gen:136 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Read 5 Done Done Done false 25 Acked 1024 Write 6 Done Done Done false 26 Acked 1025 Write 1 Done Done Done false 27 Acked 1026 Write 2 Done Done Done false 28 Acked 1027 Read 6 Done Done Done false 29 Acked 1028 Write 8 Done Done Done false 30 Acked 1029 Read 3 Done Done Done false 31 Acked 1030 Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:54:29.413976399Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1726} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 136 136 136 276 276 276 T T T 1 020-039 124 124 124 254 254 254 T T T 2 040-059 128 128 128 262 262 262 T T T 3 060-079 132 132 132 269 269 269 F F F 4 080-099 136 136 136 278 278 278 F F F 5 100-119 134 134 134 272 272 272 F F F 6 120-139 130 130 130 266 266 266 F F F 7 140-159 136 136 136 276 276 276 F F F 8 160-179 136 136 136 278 278 278 F F F 9 180-199 136 136 136 276 276 276 F F F 10 200-219 135 135 135 275 275 275 F F F 11 220-239 133 133 133 271 271 271 F F F 12 240-259 130 130 130 265 265 265 F F F 13 260-279 136 136 136 276 276 276 F F F 14 280-299 134 134 134 274 274 274 F F F 15 300-319 133 133 133 271 271 271 T T T 16 320-339 133 133 133 271 271 271 F F F 17 340-359 135 135 135 275 275 275 F F F 18 360-379 134 134 134 274 274 274 F F F 19 380-399 134 134 134 274 274 274 F F F 20 400-419 130 130 130 265 265 265 T T T 21 420-439 136 136 136 276 276 276 T T T 22 440-459 136 136 136 276 276 276 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 131 131 131 267 267 267 F F F 25 500-519 126 126 126 257 257 257 T T T 26 520-539 135 135 135 275 275 275 F F F 27 540-559 136 136 136 276 276 276 F F F 28 560-579 134 134 134 273 273 273 F F F 29 580-599 134 134 134 274 274 274 F F F Max gen: 136, Max flush: 278 A Difference in extent metadata is expected here On loop 68 Verifying data now No change after verify Loop: 68 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 137 137 137 279 279 279 F F F 1 020-039 137 137 137 279 279 279 F F F 2 040-059 137 137 137 279 279 279 F F F 3 060-079 132 132 132 269 269 269 F F F 4 080-099 136 136 136 278 278 278 F F F 5 100-119 134 134 134 272 272 272 F F F 6 120-139 130 130 130 266 266 266 F F F 7 140-159 136 136 136 276 276 276 F F F 8 160-179 136 136 136 278 278 278 F F F 9 180-199 136 136 136 276 276 276 F F F 10 200-219 135 135 135 275 275 275 F F F 11 220-239 133 133 133 271 271 271 F F F 12 240-259 130 130 130 265 265 265 F F F 13 260-279 136 136 136 276 276 276 F F F 14 280-299 134 134 134 274 274 274 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 133 133 133 271 271 271 F F F 17 340-359 135 135 135 275 275 275 F F F 18 360-379 134 134 134 274 274 274 F F F 19 380-399 134 134 134 274 274 274 F F F 20 400-419 137 137 137 279 279 279 F F F 21 420-439 137 137 137 279 279 279 F F F 22 440-459 136 136 136 276 276 276 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 131 131 131 267 267 267 F F F 25 500-519 137 137 137 279 279 279 F F F 26 520-539 135 135 135 275 275 275 F F F 27 540-559 136 136 136 276 276 276 F F F 28 560-579 134 134 134 273 273 273 F F F 29 580-599 134 134 134 274 274 274 F F F Max gen: 137, Max flush: 279 Begin loop 69 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.670056381Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.670450892Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.670483451Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734} {"msg":"Crucible 2fbdb7ff-4d2f-4771-b58f-b5d9405ef81b has session id: bc2d9a28-d9b7-453e-8200-2082080e6076","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.670776207Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"Upstairs opts: Upstairs UUID: 2fbdb7ff-4d2f-4771-b58f-b5d9405ef81b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.670820495Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"Crucible stats registered with UUID: 2fbdb7ff-4d2f-4771-b58f-b5d9405ef81b","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.670843704Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:138","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.671115161Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734} {"msg":"2fbdb7ff-4d2f-4771-b58f-b5d9405ef81b active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.671266404Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.671284913Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"0","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.671308852Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"0","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.671324251Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.671340811Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.67136423Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"2","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.671387589Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"2","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.671401698Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"io task","client":"2","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.671415587Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"io task","client":"0","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.671432137Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"io task","client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"ds_connection connected from Ok(127.0.0.1:58170)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.671583119Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"io task","client":"0","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"ds_connection connected from Ok(127.0.0.1:64632)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.671622138Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"io task","client":"2","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:54:43.671849127Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"io task","client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:54:43.671890565Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"io task","client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:54:43.672021679Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.672040088Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.672054068Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.672165462Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"io task","client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.67263267Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"0","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.672659039Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"0","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.672673849Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"2","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.672687628Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"2","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.673167625Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"2","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.673299839Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"0","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.673690861Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"2","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.673729029Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"2","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.673757208Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.673816815Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"0","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.673843684Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"0","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:43.673859123Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.673971021Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"io task","client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"ds_connection connected from Ok(127.0.0.1:47886)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.674335844Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"io task","client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.675359466Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.675395765Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.675831494Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676429946Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676451685Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"[0]R flush_numbers[0..12]: [279, 279, 279, 269, 278, 272, 266, 276, 278, 276, 275, 271]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676466444Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"downstairs","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"[0]R generation[0..12]: [137, 137, 137, 132, 136, 134, 130, 136, 136, 136, 135, 133]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676496413Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"downstairs","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676511473Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"downstairs","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"[1]R flush_numbers[0..12]: [279, 279, 279, 269, 278, 272, 266, 276, 278, 276, 275, 271]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676525502Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"downstairs","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"[1]R generation[0..12]: [137, 137, 137, 132, 136, 134, 130, 136, 136, 136, 135, 133]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676539251Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"downstairs","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676553011Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"downstairs","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"[2]R flush_numbers[0..12]: [279, 279, 279, 269, 278, 272, 266, 276, 278, 276, 275, 271]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.6765668Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"downstairs","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"[2]R generation[0..12]: [137, 137, 137, 132, 136, 134, 130, 136, 136, 136, 135, 133]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676580629Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"downstairs","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676594718Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"downstairs","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"Max found gen is 138","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676611348Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"downstairs","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"Generation requested: 138 >= found:138","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:54:53.676624757Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"downstairs","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"Next flush: 280","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676640436Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"downstairs","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676653376Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"":"downstairs","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676666485Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676678415Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"0","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676691704Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"1","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676704644Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"client":"2","":"downstairs","session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676717683Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676729992Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"2fbdb7ff-4d2f-4771-b58f-b5d9405ef81b is now active with session: bc2d9a28-d9b7-453e-8200-2082080e6076","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676742822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.676755491Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734,"session_id":"bc2d9a28-d9b7-453e-8200-2082080e6076"} {"msg":"The guest has finished waiting for activation with:138","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:54:53.67676757Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 17 len 28672 data: 9 8 7 9 11 10 11 02/30 Write block 357 len 32768 data: 15 15 12 9 7 8 7 9 03/30 Write block 212 len 36864 data: 9 7 8 9 9 7 7 9 9 04/30 Read block 585 len 24576 05/30 Read block 283 len 20480 06/30 Write block 334 len 12288 data: 12 11 10 07/30 Read block 529 len 36864 08/30 Write block 481 len 16384 data: 9 9 10 13 09/30 Read block 576 len 4096 10/30 Read block 391 len 40960 11/30 Read block 464 len 12288 12/30 Write block 576 len 24576 data: 14 14 13 10 10 10 13/30 Read block 307 len 24576 14/30 Read block 534 len 8192 15/30 Read block 193 len 16384 16/30 Read block 471 len 40960 17/30 Read block 164 len 16384 18/30 Read block 288 len 16384 19/30 Write block 212 len 40960 data: 10 8 9 10 10 8 8 10 10 11 20/30 Read block 359 len 28672 21/30 Flush 22/30 Write block 24 len 20480 data: 14 13 15 15 14 23/30 Read block 321 len 8192 24/30 Read block 38 len 28672 25/30 Read block 438 len 8192 26/30 Read block 304 len 24576 27/30 Write block 322 len 24576 data: 8 9 8 8 7 9 28/30 Write block 563 len 36864 data: 15 16 17 18 16 15 12 14 16 29/30 Write block 75 len 24576 data: 11 14 14 14 14 17 30/30 Read block 235 len 4096 ---------------------------------------------------------------- Crucible gen:138 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 5 Done Done Done false 23 Acked 1022 Read 2 Done Done Done false 24 Acked 1023 Read 7 Done Done Done false 25 Acked 1024 Read 2 Done Done Done false 26 Acked 1025 Read 6 Done Done Done false 27 Acked 1026 Write 6 Done Done Done false 28 Acked 1027 Write 9 Done Done Done false 29 Acked 1028 Write 6 Done Done Done false 30 Acked 1029 Read 1 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 9 9 9 27 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:54:53.704261207Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1734} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 280 280 280 F F F 1 020-039 138 138 138 280 280 280 T T T 2 040-059 137 137 137 279 279 279 F F F 3 060-079 132 132 132 269 269 269 T T T 4 080-099 136 136 136 278 278 278 T T T 5 100-119 134 134 134 272 272 272 F F F 6 120-139 130 130 130 266 266 266 F F F 7 140-159 136 136 136 276 276 276 F F F 8 160-179 136 136 136 278 278 278 F F F 9 180-199 136 136 136 276 276 276 F F F 10 200-219 138 138 138 280 280 280 F F F 11 220-239 138 138 138 280 280 280 F F F 12 240-259 130 130 130 265 265 265 F F F 13 260-279 136 136 136 276 276 276 F F F 14 280-299 134 134 134 274 274 274 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 138 138 138 280 280 280 T T T 17 340-359 138 138 138 280 280 280 F F F 18 360-379 138 138 138 280 280 280 F F F 19 380-399 134 134 134 274 274 274 F F F 20 400-419 137 137 137 279 279 279 F F F 21 420-439 137 137 137 279 279 279 F F F 22 440-459 136 136 136 276 276 276 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 138 138 138 280 280 280 F F F 25 500-519 137 137 137 279 279 279 F F F 26 520-539 135 135 135 275 275 275 F F F 27 540-559 136 136 136 276 276 276 F F F 28 560-579 138 138 138 280 280 280 T T T 29 580-599 138 138 138 280 280 280 F F F Max gen: 138, Max flush: 280 A Difference in extent metadata is expected here On loop 69 Verifying data now No change after verify Loop: 69 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 280 280 280 F F F 1 020-039 139 139 139 281 281 281 F F F 2 040-059 137 137 137 279 279 279 F F F 3 060-079 139 139 139 281 281 281 F F F 4 080-099 139 139 139 281 281 281 F F F 5 100-119 134 134 134 272 272 272 F F F 6 120-139 130 130 130 266 266 266 F F F 7 140-159 136 136 136 276 276 276 F F F 8 160-179 136 136 136 278 278 278 F F F 9 180-199 136 136 136 276 276 276 F F F 10 200-219 138 138 138 280 280 280 F F F 11 220-239 138 138 138 280 280 280 F F F 12 240-259 130 130 130 265 265 265 F F F 13 260-279 136 136 136 276 276 276 F F F 14 280-299 134 134 134 274 274 274 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 139 139 139 281 281 281 F F F 17 340-359 138 138 138 280 280 280 F F F 18 360-379 138 138 138 280 280 280 F F F 19 380-399 134 134 134 274 274 274 F F F 20 400-419 137 137 137 279 279 279 F F F 21 420-439 137 137 137 279 279 279 F F F 22 440-459 136 136 136 276 276 276 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 138 138 138 280 280 280 F F F 25 500-519 137 137 137 279 279 279 F F F 26 520-539 135 135 135 275 275 275 F F F 27 540-559 136 136 136 276 276 276 F F F 28 560-579 139 139 139 281 281 281 F F F 29 580-599 138 138 138 280 280 280 F F F Max gen: 139, Max flush: 281 Begin loop 70 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.932434112Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.932876571Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.93290203Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742} {"msg":"Crucible 0d34f14d-eb48-410f-ba5c-371576b4824d has session id: 6827c832-f53d-403d-93d4-708965cd74d7","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.93311508Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"Upstairs opts: Upstairs UUID: 0d34f14d-eb48-410f-ba5c-371576b4824d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.933173427Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"Crucible stats registered with UUID: 0d34f14d-eb48-410f-ba5c-371576b4824d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.933198266Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:140","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.933529981Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742} {"msg":"0d34f14d-eb48-410f-ba5c-371576b4824d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.933592948Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.933627856Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"0","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.933662894Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"0","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.933688983Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.933713892Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.933739041Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"2","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.93375976Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"2","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.933780639Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"io task","client":"2","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.933804898Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"io task","client":"0","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.933830656Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"io task","client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:55:07.933954571Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"io task","client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:55:07.933993519Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"io task","client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:55:07.934024607Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.934048506Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.934074495Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"ds_connection connected from Ok(127.0.0.1:48086)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.934101774Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"io task","client":"2","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"ds_connection connected from Ok(127.0.0.1:43005)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.934265766Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"io task","client":"0","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.934305245Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"io task","client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.934798582Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"2","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.93482616Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"2","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.934844719Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"0","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.934869048Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"0","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.935439632Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"0","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.93547102Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"2","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.936028464Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"2","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.936055663Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"2","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.936071152Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.936083742Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"0","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.936097211Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"0","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:07.93611098Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.934775596Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"io task","client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"ds_connection connected from Ok(127.0.0.1:44497)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.934960577Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"io task","client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.936069995Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.936107123Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.936579871Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937099087Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937123366Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"[0]R flush_numbers[0..12]: [280, 281, 279, 281, 281, 272, 266, 276, 278, 276, 280, 280]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937141645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"downstairs","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"[0]R generation[0..12]: [138, 139, 137, 139, 139, 134, 130, 136, 136, 136, 138, 138]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937159624Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"downstairs","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937177323Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"downstairs","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"[1]R flush_numbers[0..12]: [280, 281, 279, 281, 281, 272, 266, 276, 278, 276, 280, 280]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937194663Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"downstairs","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"[1]R generation[0..12]: [138, 139, 137, 139, 139, 134, 130, 136, 136, 136, 138, 138]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937211882Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"downstairs","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937229361Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"downstairs","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"[2]R flush_numbers[0..12]: [280, 281, 279, 281, 281, 272, 266, 276, 278, 276, 280, 280]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.93724645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"downstairs","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"[2]R generation[0..12]: [138, 139, 137, 139, 139, 134, 130, 136, 136, 136, 138, 138]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937264039Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"downstairs","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"Wait for a query_work_queue command to finish before sending IO [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937288458Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"downstairs","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"Max found gen is 140","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937305798Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"downstairs","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"Generation requested: 140 >= found:140","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937322517Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"downstairs","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"Next flush: 282","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937338646Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"downstairs","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937354785Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"":"downstairs","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937371254Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937386484Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"0","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937403013Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"1","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937419232Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"client":"2","":"downstairs","session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937435711Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937450961Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"0d34f14d-eb48-410f-ba5c-371576b4824d is now active with session: 6827c832-f53d-403d-93d4-708965cd74d7","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.93746701Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937481939Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742,"session_id":"6827c832-f53d-403d-93d4-708965cd74d7"} {"msg":"The guest has finished waiting for activation with:140","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:17.937497198Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 99 len 16384 data: 12 14 14 12 02/30 Read block 150 len 8192 03/30 Read block 319 len 8192 04/30 Write block 333 len 28672 data: 10 13 12 11 10 8 6 05/30 Read block 350 len 24576 06/30 Flush 07/30 Write block 520 len 12288 data: 12 10 11 08/30 Write block 276 len 24576 data: 12 14 13 13 14 10 09/30 Flush 10/30 Write block 357 len 32768 data: 16 16 13 10 8 9 8 10 11/30 Write block 352 len 8192 data: 10 12 12/30 Write block 434 len 36864 data: 9 12 10 8 7 9 8 12 14 13/30 Read block 454 len 36864 14/30 Read block 472 len 36864 15/30 Write block 337 len 8192 data: 11 9 16/30 Write block 407 len 20480 data: 12 9 8 7 6 17/30 Read block 465 len 8192 18/30 Write block 414 len 32768 data: 9 10 11 13 15 18 18 15 19/30 Read block 535 len 24576 20/30 Read block 411 len 16384 21/30 Read block 109 len 4096 22/30 Write block 244 len 32768 data: 8 9 8 10 12 13 12 10 23/30 Read block 570 len 8192 24/30 Write block 404 len 32768 data: 12 12 13 13 10 9 8 7 25/30 Read block 386 len 12288 26/30 Read block 542 len 16384 27/30 Write block 358 len 28672 data: 17 14 11 9 10 9 11 28/30 Write block 529 len 40960 data: 9 10 12 11 13 11 11 10 10 10 29/30 Read block 452 len 36864 30/30 Read block 38 len 20480 ---------------------------------------------------------------- Crucible gen:140 GIO:true work queues: Upstairs:0 downstairs:7 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 25 Acked 1024 Write 8 Done Done Done false 26 Acked 1025 Read 3 Done Done Done false 27 Acked 1026 Read 4 Done Done Done false 28 Acked 1027 Write 7 Done Done Done false 29 Acked 1028 Write 10 Done Done Done false 30 Acked 1029 Read 9 Done Done Done false 31 Acked 1030 Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 1 2 Done 7 6 6 19 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1023 1023 1023 Downstairs last five completed: 1023 1022 1021 1020 1019 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:55:17.964775396Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1742} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 280 280 280 F F F 1 020-039 139 139 139 281 281 281 F F F 2 040-059 137 137 137 279 279 279 F F F 3 060-079 139 139 139 281 281 281 F F F 4 080-099 140 140 140 282 282 282 F F F 5 100-119 140 140 140 282 282 282 F F F 6 120-139 130 130 130 266 266 266 F F F 7 140-159 136 136 136 276 276 276 F F F 8 160-179 136 136 136 278 278 278 F F F 9 180-199 136 136 136 276 276 276 F F F 10 200-219 138 138 138 280 280 280 F F F 11 220-239 138 138 138 280 280 280 F F F 12 240-259 140 140 140 284 284 284 F F F 13 260-279 140 140 140 283 283 283 F F F 14 280-299 140 140 140 283 283 283 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 140 140 140 284 284 284 F F F 17 340-359 140 140 140 284 284 284 T T T 18 360-379 140 140 140 284 284 284 T T T 19 380-399 134 134 134 274 274 274 F F F 20 400-419 140 140 140 284 284 284 T T T 21 420-439 140 140 140 284 284 284 F F F 22 440-459 140 140 140 284 284 284 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 138 138 138 280 280 280 F F F 25 500-519 137 137 137 279 279 279 F F F 26 520-539 140 140 140 283 283 283 T T T 27 540-559 136 136 136 276 276 276 F F F 28 560-579 139 139 139 281 281 281 F F F 29 580-599 138 138 138 280 280 280 F F F Max gen: 140, Max flush: 284 A Difference in extent metadata is expected here On loop 70 Verifying data now No change after verify Loop: 70 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 280 280 280 F F F 1 020-039 139 139 139 281 281 281 F F F 2 040-059 137 137 137 279 279 279 F F F 3 060-079 139 139 139 281 281 281 F F F 4 080-099 140 140 140 282 282 282 F F F 5 100-119 140 140 140 282 282 282 F F F 6 120-139 130 130 130 266 266 266 F F F 7 140-159 136 136 136 276 276 276 F F F 8 160-179 136 136 136 278 278 278 F F F 9 180-199 136 136 136 276 276 276 F F F 10 200-219 138 138 138 280 280 280 F F F 11 220-239 138 138 138 280 280 280 F F F 12 240-259 140 140 140 284 284 284 F F F 13 260-279 140 140 140 283 283 283 F F F 14 280-299 140 140 140 283 283 283 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 140 140 140 284 284 284 F F F 17 340-359 141 141 141 285 285 285 F F F 18 360-379 141 141 141 285 285 285 F F F 19 380-399 134 134 134 274 274 274 F F F 20 400-419 141 141 141 285 285 285 F F F 21 420-439 140 140 140 284 284 284 F F F 22 440-459 140 140 140 284 284 284 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 138 138 138 280 280 280 F F F 25 500-519 137 137 137 279 279 279 F F F 26 520-539 141 141 141 285 285 285 F F F 27 540-559 136 136 136 276 276 276 F F F 28 560-579 139 139 139 281 281 281 F F F 29 580-599 138 138 138 280 280 280 F F F Max gen: 141, Max flush: 285 Begin loop 71 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.182213752Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.182549837Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.182572446Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750} {"msg":"Crucible f8147b57-f628-49a8-a4e0-c73afe38f4eb has session id: 3c85ea4f-8005-4ed5-b9c5-64d349076738","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.182872952Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"Upstairs opts: Upstairs UUID: f8147b57-f628-49a8-a4e0-c73afe38f4eb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.18290948Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"Crucible stats registered with UUID: f8147b57-f628-49a8-a4e0-c73afe38f4eb","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.182928419Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:142","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.183196396Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750} {"msg":"f8147b57-f628-49a8-a4e0-c73afe38f4eb active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.183285242Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.183304051Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"0","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.183323741Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"0","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.18333763Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"1","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.183351229Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"1","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.183364799Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.183378198Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.183391207Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"io task","client":"0","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.183405137Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"io task","client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.183418656Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"io task","client":"1","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"ds_connection connected from Ok(127.0.0.1:62588)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.183652275Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"io task","client":"1","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:55:32.183787249Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"io task","client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:55:32.183828247Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"io task","client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:55:32.183857406Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.183880495Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.183909663Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"ds_connection connected from Ok(127.0.0.1:34398)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.183933162Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"io task","client":"0","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.184029408Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"io task","client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.184509845Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"1","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.184532754Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"1","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.184546764Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"0","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.184560353Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"0","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.185122227Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"1","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.185250741Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"0","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.185680661Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"1","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.18570295Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"1","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.185716949Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.185742368Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"0","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.185756037Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"0","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:32.185769267Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.186042837Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"io task","client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"ds_connection connected from Ok(127.0.0.1:59190)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.186173281Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"io task","client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.187201373Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.187227612Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.187756687Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188279613Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188302692Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"[0]R flush_numbers[0..12]: [280, 281, 279, 281, 282, 282, 266, 276, 278, 276, 280, 280]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188317831Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"downstairs","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"[0]R generation[0..12]: [138, 139, 137, 139, 140, 140, 130, 136, 136, 136, 138, 138]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.18833247Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"downstairs","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188347089Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"downstairs","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"[1]R flush_numbers[0..12]: [280, 281, 279, 281, 282, 282, 266, 276, 278, 276, 280, 280]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188361089Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"downstairs","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"[1]R generation[0..12]: [138, 139, 137, 139, 140, 140, 130, 136, 136, 136, 138, 138]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188374908Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"downstairs","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188388998Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"downstairs","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"[2]R flush_numbers[0..12]: [280, 281, 279, 281, 282, 282, 266, 276, 278, 276, 280, 280]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188403057Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"downstairs","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"[2]R generation[0..12]: [138, 139, 137, 139, 140, 140, 130, 136, 136, 136, 138, 138]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188416736Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"downstairs","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188434276Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"downstairs","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"Max found gen is 142","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188447675Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"downstairs","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"Generation requested: 142 >= found:142","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:55:42.188461004Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"downstairs","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"Next flush: 286","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188477503Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"downstairs","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188490653Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"":"downstairs","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188503962Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188515882Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"0","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188529031Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"1","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.18854222Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"client":"2","":"downstairs","session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.18855558Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188567729Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"f8147b57-f628-49a8-a4e0-c73afe38f4eb is now active with session: 3c85ea4f-8005-4ed5-b9c5-64d349076738","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188580569Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188592428Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750,"session_id":"3c85ea4f-8005-4ed5-b9c5-64d349076738"} {"msg":"The guest has finished waiting for activation with:142","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:42.188604588Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 214 len 16384 data: 10 11 11 9 03/30 Write block 400 len 32768 data: 12 12 12 11 13 13 14 14 04/30 Read block 214 len 8192 05/30 Read block 209 len 40960 06/30 Write block 404 len 28672 data: 14 14 15 15 11 10 9 07/30 Write block 387 len 20480 data: 15 13 11 13 14 08/30 Read block 325 len 12288 09/30 Write block 582 len 16384 data: 12 10 12 9 10/30 Write block 279 len 32768 data: 14 15 11 9 8 9 10 8 11/30 Read block 208 len 32768 12/30 Write block 394 len 40960 data: 13 15 15 16 16 14 13 13 13 12 13/30 Write block 63 len 20480 data: 3 4 6 5 5 14/30 Read block 385 len 4096 15/30 Read block 250 len 24576 16/30 Read block 297 len 28672 17/30 Read block 324 len 4096 18/30 Write block 142 len 12288 data: 11 14 14 19/30 Flush 20/30 Write block 590 len 16384 data: 9 9 9 10 21/30 Flush 22/30 Write block 280 len 24576 data: 16 12 10 9 10 11 23/30 Read block 411 len 32768 24/30 Read block 185 len 40960 25/30 Read block 388 len 28672 26/30 Read block 439 len 24576 27/30 Read block 235 len 16384 28/30 Read block 58 len 36864 29/30 Read block 423 len 24576 30/30 Write block 546 len 24576 data: 10 10 11 9 10 10 ---------------------------------------------------------------- Crucible gen:142 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 6 Done Done Done false 23 Acked 1022 Read 8 Done Done Done false 24 Acked 1023 Read 10 Done Done Done false 25 Acked 1024 Read 7 Done Done Done false 26 Acked 1025 Read 6 Done Done Done false 27 Acked 1026 Read 4 Done Done Done false 28 Acked 1027 Read 9 Done Done Done false 29 Acked 1028 Read 6 Done Done Done false 30 Acked 1029 Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 8 8 8 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:55:42.215407837Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1750} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 280 280 280 F F F 1 020-039 139 139 139 281 281 281 F F F 2 040-059 137 137 137 279 279 279 F F F 3 060-079 142 142 142 287 287 287 F F F 4 080-099 140 140 140 282 282 282 F F F 5 100-119 140 140 140 282 282 282 F F F 6 120-139 130 130 130 266 266 266 F F F 7 140-159 142 142 142 287 287 287 F F F 8 160-179 136 136 136 278 278 278 F F F 9 180-199 136 136 136 276 276 276 F F F 10 200-219 142 142 142 287 287 287 F F F 11 220-239 138 138 138 280 280 280 F F F 12 240-259 140 140 140 284 284 284 F F F 13 260-279 142 142 142 287 287 287 F F F 14 280-299 142 142 142 287 287 287 T T T 15 300-319 137 137 137 279 279 279 F F F 16 320-339 140 140 140 284 284 284 F F F 17 340-359 141 141 141 285 285 285 F F F 18 360-379 141 141 141 285 285 285 F F F 19 380-399 142 142 142 287 287 287 F F F 20 400-419 142 142 142 287 287 287 F F F 21 420-439 140 140 140 284 284 284 F F F 22 440-459 140 140 140 284 284 284 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 138 138 138 280 280 280 F F F 25 500-519 137 137 137 279 279 279 F F F 26 520-539 141 141 141 285 285 285 F F F 27 540-559 136 136 136 276 276 276 T T T 28 560-579 139 139 139 281 281 281 F F F 29 580-599 142 142 142 288 288 288 F F F Max gen: 142, Max flush: 288 A Difference in extent metadata is expected here On loop 71 Verifying data now No change after verify Loop: 71 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 280 280 280 F F F 1 020-039 139 139 139 281 281 281 F F F 2 040-059 137 137 137 279 279 279 F F F 3 060-079 142 142 142 287 287 287 F F F 4 080-099 140 140 140 282 282 282 F F F 5 100-119 140 140 140 282 282 282 F F F 6 120-139 130 130 130 266 266 266 F F F 7 140-159 142 142 142 287 287 287 F F F 8 160-179 136 136 136 278 278 278 F F F 9 180-199 136 136 136 276 276 276 F F F 10 200-219 142 142 142 287 287 287 F F F 11 220-239 138 138 138 280 280 280 F F F 12 240-259 140 140 140 284 284 284 F F F 13 260-279 142 142 142 287 287 287 F F F 14 280-299 143 143 143 289 289 289 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 140 140 140 284 284 284 F F F 17 340-359 141 141 141 285 285 285 F F F 18 360-379 141 141 141 285 285 285 F F F 19 380-399 142 142 142 287 287 287 F F F 20 400-419 142 142 142 287 287 287 F F F 21 420-439 140 140 140 284 284 284 F F F 22 440-459 140 140 140 284 284 284 F F F 23 460-479 125 125 125 256 256 256 F F F 24 480-499 138 138 138 280 280 280 F F F 25 500-519 137 137 137 279 279 279 F F F 26 520-539 141 141 141 285 285 285 F F F 27 540-559 143 143 143 289 289 289 F F F 28 560-579 139 139 139 281 281 281 F F F 29 580-599 142 142 142 288 288 288 F F F Max gen: 143, Max flush: 289 Begin loop 72 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.403380488Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.403718833Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.403748141Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758} {"msg":"Crucible 2beb6201-e340-4f6d-9b4e-4d5336730b06 has session id: 095c40ba-d47c-40e8-953e-588ca4aa1b21","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.403876855Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"Upstairs opts: Upstairs UUID: 2beb6201-e340-4f6d-9b4e-4d5336730b06, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.40398735Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"Crucible stats registered with UUID: 2beb6201-e340-4f6d-9b4e-4d5336730b06","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.404023518Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:144","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.404325394Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758} {"msg":"2beb6201-e340-4f6d-9b4e-4d5336730b06 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.40442602Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.404451628Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"0","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.404473567Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"0","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.404487947Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"1","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.404503316Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"1","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.404527345Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.404552304Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.404575842Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"io task","client":"0","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.404590372Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"io task","client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.404604181Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"io task","client":"1","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"ds_connection connected from Ok(127.0.0.1:35725)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.404779763Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"io task","client":"1","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"ds_connection connected from Ok(127.0.0.1:44172)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.404814762Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"io task","client":"0","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:55:56.405009352Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"io task","client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:55:56.405049891Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"io task","client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:55:56.405078369Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.405102638Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.405128547Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.405153486Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"io task","client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.40569984Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"0","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.405727959Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"0","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.405743048Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"1","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.405760727Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"1","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.406242895Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"1","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.406356639Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"0","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.406737202Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"1","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.40677544Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"1","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.406803629Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.406873795Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"0","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.406906764Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"0","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:55:56.406925383Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.405648506Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"io task","client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"ds_connection connected from Ok(127.0.0.1:43510)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.40577231Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"io task","client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.406828341Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.406863269Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.407369166Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.407955189Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.407982797Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"[0]R flush_numbers[0..12]: [280, 281, 279, 287, 282, 282, 266, 287, 278, 276, 287, 280]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.407999196Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"downstairs","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"[0]R generation[0..12]: [138, 139, 137, 142, 140, 140, 130, 142, 136, 136, 142, 138]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408013706Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"downstairs","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408028035Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"downstairs","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"[1]R flush_numbers[0..12]: [280, 281, 279, 287, 282, 282, 266, 287, 278, 276, 287, 280]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408043225Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"downstairs","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"[1]R generation[0..12]: [138, 139, 137, 142, 140, 140, 130, 142, 136, 136, 142, 138]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408057284Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"downstairs","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408071443Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"downstairs","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"[2]R flush_numbers[0..12]: [280, 281, 279, 287, 282, 282, 266, 287, 278, 276, 287, 280]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408085263Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"downstairs","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"[2]R generation[0..12]: [138, 139, 137, 142, 140, 140, 130, 142, 136, 136, 142, 138]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408107991Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"downstairs","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.40812601Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"downstairs","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"Max found gen is 144","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.40814243Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"downstairs","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"Generation requested: 144 >= found:144","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408159029Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"downstairs","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"Next flush: 290","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:56:06.408175198Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"downstairs","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408194137Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"":"downstairs","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408211326Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408226326Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"0","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408243015Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"1","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408259834Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"client":"2","":"downstairs","session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408276563Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408292323Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"2beb6201-e340-4f6d-9b4e-4d5336730b06 is now active with session: 095c40ba-d47c-40e8-953e-588ca4aa1b21","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408308872Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408324071Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758,"session_id":"095c40ba-d47c-40e8-953e-588ca4aa1b21"} {"msg":"The guest has finished waiting for activation with:144","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:06.408339261Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 368 len 24576 data: 13 14 14 16 14 13 02/30 Read block 176 len 4096 03/30 Write block 590 len 4096 data: 10 04/30 Read block 340 len 24576 05/30 Read block 295 len 28672 06/30 Read block 230 len 12288 07/30 Read block 296 len 40960 08/30 Write block 89 len 40960 data: 10 12 12 11 10 9 9 8 10 11 09/30 Flush 10/30 Write block 581 len 8192 data: 11 13 11/30 Write block 238 len 36864 data: 14 12 12 11 10 9 9 10 9 12/30 Read block 297 len 12288 13/30 Read block 139 len 8192 14/30 Read block 459 len 24576 15/30 Read block 209 len 20480 16/30 Write block 69 len 28672 data: 4 3 5 6 10 10 12 17/30 Flush 18/30 Write block 337 len 28672 data: 12 10 7 7 9 10 10 19/30 Write block 562 len 28672 data: 13 16 17 18 19 17 16 20/30 Write block 551 len 28672 data: 11 9 11 12 12 11 6 21/30 Write block 118 len 28672 data: 11 12 12 11 12 11 9 22/30 Read block 352 len 12288 23/30 Flush 24/30 Write block 138 len 16384 data: 8 7 6 7 25/30 Read block 357 len 32768 26/30 Write block 164 len 40960 data: 8 10 12 13 14 14 15 13 11 10 27/30 Write block 496 len 20480 data: 9 14 11 12 12 28/30 Write block 480 len 40960 data: 9 10 10 11 14 13 10 12 11 9 29/30 Read block 405 len 24576 30/30 Write block 454 len 36864 data: 7 9 8 7 8 8 9 9 6 ---------------------------------------------------------------- Crucible gen:144 GIO:true work queues: Upstairs:0 downstairs:7 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 25 Acked 1024 Write 4 Done Done Done false 26 Acked 1025 Read 8 Done Done Done false 27 Acked 1026 Write 10 Done Done Done false 28 Acked 1027 Write 5 Done Done Done false 29 Acked 1028 Write 10 Done Done Done false 30 Acked 1029 Read 6 Done Done Done false 31 Acked 1030 Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 6 6 6 18 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1023 1023 1023 Downstairs last five completed: 1023 1022 1021 1020 1019 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:56:06.438556831Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1758} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 280 280 280 F F F 1 020-039 139 139 139 281 281 281 F F F 2 040-059 137 137 137 279 279 279 F F F 3 060-079 144 144 144 291 291 291 F F F 4 080-099 144 144 144 290 290 290 F F F 5 100-119 144 144 144 293 293 293 F F F 6 120-139 144 144 144 293 293 293 T T T 7 140-159 142 142 142 287 287 287 T T T 8 160-179 136 136 136 278 278 278 T T T 9 180-199 136 136 136 276 276 276 F F F 10 200-219 142 142 142 287 287 287 F F F 11 220-239 144 144 144 291 291 291 F F F 12 240-259 144 144 144 291 291 291 F F F 13 260-279 142 142 142 287 287 287 F F F 14 280-299 143 143 143 289 289 289 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 144 144 144 292 292 292 F F F 17 340-359 144 144 144 292 292 292 F F F 18 360-379 144 144 144 290 290 290 F F F 19 380-399 142 142 142 287 287 287 F F F 20 400-419 142 142 142 287 287 287 F F F 21 420-439 140 140 140 284 284 284 F F F 22 440-459 140 140 140 284 284 284 T T T 23 460-479 125 125 125 256 256 256 T T T 24 480-499 138 138 138 280 280 280 T T T 25 500-519 137 137 137 279 279 279 T T T 26 520-539 141 141 141 285 285 285 F F F 27 540-559 144 144 144 292 292 292 F F F 28 560-579 144 144 144 292 292 292 F F F 29 580-599 144 144 144 291 291 291 F F F Max gen: 144, Max flush: 293 A Difference in extent metadata is expected here On loop 72 Verifying data now No change after verify Loop: 72 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 280 280 280 F F F 1 020-039 139 139 139 281 281 281 F F F 2 040-059 137 137 137 279 279 279 F F F 3 060-079 144 144 144 291 291 291 F F F 4 080-099 144 144 144 290 290 290 F F F 5 100-119 144 144 144 293 293 293 F F F 6 120-139 145 145 145 294 294 294 F F F 7 140-159 145 145 145 294 294 294 F F F 8 160-179 145 145 145 294 294 294 F F F 9 180-199 136 136 136 276 276 276 F F F 10 200-219 142 142 142 287 287 287 F F F 11 220-239 144 144 144 291 291 291 F F F 12 240-259 144 144 144 291 291 291 F F F 13 260-279 142 142 142 287 287 287 F F F 14 280-299 143 143 143 289 289 289 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 144 144 144 292 292 292 F F F 17 340-359 144 144 144 292 292 292 F F F 18 360-379 144 144 144 290 290 290 F F F 19 380-399 142 142 142 287 287 287 F F F 20 400-419 142 142 142 287 287 287 F F F 21 420-439 140 140 140 284 284 284 F F F 22 440-459 145 145 145 294 294 294 F F F 23 460-479 145 145 145 294 294 294 F F F 24 480-499 145 145 145 294 294 294 F F F 25 500-519 145 145 145 294 294 294 F F F 26 520-539 141 141 141 285 285 285 F F F 27 540-559 144 144 144 292 292 292 F F F 28 560-579 144 144 144 292 292 292 F F F 29 580-599 144 144 144 291 291 291 F F F Max gen: 145, Max flush: 294 Begin loop 73 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.693670798Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694114187Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694155175Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766} {"msg":"Crucible ab662a91-22a1-457e-8b9f-eab3e7fc8431 has session id: 98d069f9-4ce2-4384-8c84-c57c2ba68873","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694395774Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"Upstairs opts: Upstairs UUID: ab662a91-22a1-457e-8b9f-eab3e7fc8431, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694439292Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"Crucible stats registered with UUID: ab662a91-22a1-457e-8b9f-eab3e7fc8431","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694466301Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:146","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694756237Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766} {"msg":"ab662a91-22a1-457e-8b9f-eab3e7fc8431 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694779526Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694792905Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"0","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694811824Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"0","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694825494Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694838863Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694851912Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"2","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694865522Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"2","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694894281Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"io task","client":"0","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694925439Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"io task","client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.694942928Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"io task","client":"2","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"ds_connection connected from Ok(127.0.0.1:45046)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.695141709Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"io task","client":"2","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"ds_connection connected from Ok(127.0.0.1:34580)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.695233425Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"io task","client":"0","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:56:20.695462974Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"io task","client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:56:20.695505582Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"io task","client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:56:20.69554238Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.695570499Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.695596248Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.695694123Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"io task","client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.696111304Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"2","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.696138892Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"2","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.696154512Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"0","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.696168361Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"0","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.696803741Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"2","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.69683416Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"0","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.697443942Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"2","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.69748215Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"2","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.697511178Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.697529658Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"0","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.697544067Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"0","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:20.697561036Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.696579254Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"io task","client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"ds_connection connected from Ok(127.0.0.1:50070)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.696730267Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"io task","client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.697643154Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.697675173Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.698234607Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.698766492Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.698794271Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"[0]R flush_numbers[0..12]: [280, 281, 279, 291, 290, 293, 294, 294, 294, 276, 287, 291]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.69881227Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"downstairs","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"[0]R generation[0..12]: [138, 139, 137, 144, 144, 144, 145, 145, 145, 136, 142, 144]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.698830249Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"downstairs","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.698847728Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"downstairs","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"[1]R flush_numbers[0..12]: [280, 281, 279, 291, 290, 293, 294, 294, 294, 276, 287, 291]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.698865007Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"downstairs","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"[1]R generation[0..12]: [138, 139, 137, 144, 144, 144, 145, 145, 145, 136, 142, 144]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.698882076Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"downstairs","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.698899636Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"downstairs","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"[2]R flush_numbers[0..12]: [280, 281, 279, 291, 290, 293, 294, 294, 294, 276, 287, 291]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.698923185Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"downstairs","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"[2]R generation[0..12]: [138, 139, 137, 144, 144, 144, 145, 145, 145, 136, 142, 144]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.698940734Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"downstairs","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0Wait for a query_work_queue command to finish before sending IO ,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.698961513Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"downstairs","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"Max found gen is 146","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.698977782Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"downstairs","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"Generation requested: 146 >= found:146","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.698994101Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"downstairs","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"Next flush: 295","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.699010101Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"downstairs","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.69902597Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"":"downstairs","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.699042099Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.699056859Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"0","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.699073058Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"1","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.699089277Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"client":"2","":"downstairs","session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.699105626Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.699120725Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"ab662a91-22a1-457e-8b9f-eab3e7fc8431 is now active with session: 98d069f9-4ce2-4384-8c84-c57c2ba68873","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.699136665Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:30.699151524Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766,"session_id":"98d069f9-4ce2-4384-8c84-c57c2ba68873"} {"msg":"The guest has finished waiting for activation with:146","v":0,"name":"crucible","level":30Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2024-07-31T18:56:30.699169043Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 478 len 32768 data: 10 11 10 11 11 12 15 14 02/30 Read block 473 len 20480 03/30 Read block 152 len 28672 04/30 Flush 05/30 Write block 70 len 12288 data: 4 6 7 06/30 Read block 251 len 16384 07/30 Read block 318 len 4096 08/30 Write block 514 len 32768 data: 11 9 9 7 10 12 13 11 09/30 Read block 10 len 32768 10/30 Read block 257 len 4096 11/30 Read block 21 len 4096 12/30 Flush 13/30 Write block 557 len 12288 data: 7 7 9 14/30 Flush 15/30 Write block 577 len 16384 data: 15 14 11 11 16/30 Read block 504 len 16384 17/30 Read block 103 len 20480 18/30 Write block 141 len 4096 data: 8 19/30 Read block 291 len 40960 20/30 Write block 147 len 8192 data: 11 11 21/30 Flush 22/30 Write block 502 len 24576 data: 14 13 13 15 13 16 23/30 Read block 56 len 28672 24/30 Read block 449 len 12288 25/30 Read block 186 len 16384 26/30 Write block 186 len 24576 data: 14 17 18 15 16 15 27/30 Read block 36 len 20480 28/30 Write block 5 len 24576 data: 5 6 7 7 8 8 29/30 Read block 29 len 40960 30/30 Read block 576 len 4096 ---------------------------------------------------------------- Crucible gen:146 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 6 Done Done Done false 23 Acked 1022 Read 7 Done Done Done false 24 Acked 1023 Read 3 Done Done Done false 25 Acked 1024 Read 4 Done Done Done false 26 Acked 1025 Write 6 Done Done Done false 27 Acked 1026 Read 5 Done Done Done false 28 Acked 1027 Write 6 Done Done Done false 29 Acked 1028 Read 10 Done Done Done false 30 Acked 1029 Read 1 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 1 2 Done 9 8 8 25 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:56:30.727684363Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1766} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 138 138 138 280 280 280 T T T 1 020-039 139 139 139 281 281 281 F F F 2 040-059 137 137 137 279 279 279 F F F 3 060-079 146 146 146 296 296 296 F F F 4 080-099 144 144 144 290 290 290 F F F 5 100-119 144 144 144 293 293 293 F F F 6 120-139 145 145 145 294 294 294 F F F 7 140-159 146 146 146 298 298 298 F F F 8 160-179 145 145 145 294 294 294 F F F 9 180-199 136 136 136 276 276 276 T T T 10 200-219 142 142 142 287 287 287 F F F 11 220-239 144 144 144 291 291 291 F F F 12 240-259 144 144 144 291 291 291 F F F 13 260-279 142 142 142 287 287 287 F F F 14 280-299 143 143 143 289 289 289 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 144 144 144 292 292 292 F F F 17 340-359 144 144 144 292 292 292 F F F 18 360-379 144 144 144 290 290 290 F F F 19 380-399 142 142 142 287 287 287 F F F 20 400-419 142 142 142 287 287 287 F F F 21 420-439 140 140 140 284 284 284 F F F 22 440-459 145 145 145 294 294 294 F F F 23 460-479 146 146 146 295 295 295 F F F 24 480-499 146 146 146 295 295 295 F F F 25 500-519 146 146 146 296 296 296 T T T 26 520-539 146 146 146 296 296 296 F F F 27 540-559 146 146 146 297 297 297 F F F 28 560-579 146 146 146 298 298 298 F F F 29 580-599 146 146 146 298 298 298 F F F Max gen: 146, Max flush: 298 A Difference in extent metadata is expected here On loop 73 Verifying data now No change after verify Loop: 73 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 147 147 147 299 299 299 F F F 1 020-039 139 139 139 281 281 281 F F F 2 040-059 137 137 137 279 279 279 F F F 3 060-079 146 146 146 296 296 296 F F F 4 080-099 144 144 144 290 290 290 F F F 5 100-119 144 144 144 293 293 293 F F F 6 120-139 145 145 145 294 294 294 F F F 7 140-159 146 146 146 298 298 298 F F F 8 160-179 145 145 145 294 294 294 F F F 9 180-199 147 147 147 299 299 299 F F F 10 200-219 142 142 142 287 287 287 F F F 11 220-239 144 144 144 291 291 291 F F F 12 240-259 144 144 144 291 291 291 F F F 13 260-279 142 142 142 287 287 287 F F F 14 280-299 143 143 143 289 289 289 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 144 144 144 292 292 292 F F F 17 340-359 144 144 144 292 292 292 F F F 18 360-379 144 144 144 290 290 290 F F F 19 380-399 142 142 142 287 287 287 F F F 20 400-419 142 142 142 287 287 287 F F F 21 420-439 140 140 140 284 284 284 F F F 22 440-459 145 145 145 294 294 294 F F F 23 460-479 146 146 146 295 295 295 F F F 24 480-499 146 146 146 295 295 295 F F F 25 500-519 147 147 147 299 299 299 F F F 26 520-539 146 146 146 296 296 296 F F F 27 540-559 146 146 146 297 297 297 F F F 28 560-579 146 146 146 298 298 298 F F F 29 580-599 146 146 146 298 298 298 F F F Max gen: 147, Max flush: 299 Begin loop 74 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.931753594Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.932202943Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.932229182Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774} {"msg":"Crucible 26895802-1ebf-4990-9617-879601861f7f has session id: d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.932455511Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"Upstairs opts: Upstairs UUID: 26895802-1ebf-4990-9617-879601861f7f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.932497999Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"Crucible stats registered with UUID: 26895802-1ebf-4990-9617-879601861f7f","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.932525608Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:148","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.932850423Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774} {"msg":"26895802-1ebf-4990-9617-879601861f7f active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.932875482Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.932899211Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.932932729Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.932955998Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"1","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.932979097Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"1","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.933003466Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"2","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.933026245Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"2","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.933049464Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"io task","client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.933074163Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"io task","client":"1","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.933098411Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"io task","client":"2","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:56:44.933215556Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"io task","client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:56:44.933254144Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"io task","client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:56:44.933283533Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.933309932Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"ds_connection connected from Ok(127.0.0.1:45802)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.93333534Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"io task","client":"2","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.933368369Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"ds_connection connected from Ok(127.0.0.1:49866)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.933395768Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"io task","client":"1","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.933504682Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"io task","client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.933998659Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"1","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.934023578Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"1","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.934052297Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"2","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.934066536Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"2","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.934581382Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"2","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.934691567Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"1","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.93506001Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"2","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.935094578Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"2","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.935117927Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.935137806Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"1","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.935163605Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"1","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:44.935185374Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.933987164Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"io task","client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"ds_connection connected from Ok(127.0.0.1:44314)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.934213324Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"io task","client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.93535824Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.935396689Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.936039528Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.9366422Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.936665389Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"[0]R flush_numbers[0..12]: [299, 281, 279, 296, 290, 293, 294, 298, 294, 299, 287, 291]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.936685558Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"downstairs","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"[0]R generation[0..12]: [147, 139, 137, 146, 144, 144, 145, 146, 145, 147, 142, 144]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.936703677Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"downstairs","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.936721517Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"downstairs","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"[1]R flush_numbers[0..12]: [299, 281, 279, 296, 290, 293, 294, 298, 294, 299, 287, 291]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.936738536Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"downstairs","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"[1]R generation[0..12]: [147, 139, 137, 146, 144, 144, 145, 146, 145, 147, 142, 144]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.936755665Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"downstairs","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.93685457Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"downstairs","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"[2]R flush_numbers[0..12]: [299, 281, 279, 296, 290, 293, 294, 298, 294, 299, 287, 291]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.936882549Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"downstairs","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"[2]R generation[0..12]: [147, 139, 137, 146, 144, 144, 145, 146, 145, 147, 142, 144]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.936900468Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"downstairs","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.936917947Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"downstairs","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"Max found gen is 148","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.936935507Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"downstairs","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"Generation requested: 148 >= found:148","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.936963085Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"downstairs","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg"Wait for a query_work_queue command to finish before sending IO :"Next flush: 300","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.936994414Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"downstairs","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.937024873Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"":"downstairs","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.937043692Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.937058891Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"0","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.93707534Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"1","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.937091789Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"client":"2","":"downstairs","session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.937108309Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.937123478Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"26895802-1ebf-4990-9617-879601861f7f is now active with session: d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.937139487Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.937154227Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774,"session_id":"d7a4fa11-c1f4-4101-a6bf-1ff7a1e27e37"} {"msg":"The guest has finished waiting for activation with:148","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:56:54.937172256Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 12 len 36864 data: 9 7 6 8 8 10 9 8 10 02/30 Write block 514 len 8192 data: 12 10 03/30 Flush 04/30 Write block 383 len 40960 data: 15 19 15 14 16 14 12 14 15 16 05/30 Flush 06/30 Write block 197 len 16384 data: 14 12 9 10 07/30 Write block 482 len 16384 data: 12 13 16 15 08/30 Read block 461 len 32768 09/30 Write block 531 len 16384 data: 13 12 14 12 10/30 Write block 503 len 36864 data: 14 14 16 14 17 15 11 13 14 11/30 Write block 20 len 24576 data: 11 12 11 12 15 14 12/30 Write block 439 len 4096 data: 10 13/30 Write block 124 len 24576 data: 10 9 8 10 11 10 14/30 Read block 74 len 8192 15/30 Read block 8 len 32768 16/30 Read block 129 len 4096 17/30 Write block 427 len 28672 data: 11 12 17 20 18 16 12 18/30 Flush 19/30 Write block 332 len 24576 data: 11 11 14 13 12 13 20/30 Flush 21/30 Write block 183 len 28672 data: 11 11 13 15 18 19 16 22/30 Write block 498 len 8192 data: 12 13 23/30 Read block 537 len 40960 24/30 Write block 234 len 28672 data: 12 14 15 13 15 13 13 25/30 Flush 26/30 Write block 551 len 16384 data: 12 10 12 13 27/30 Write block 224 len 16384 data: 9 10 8 10 28/30 Write block 99 len 28672 data: 13 15 15 13 10 9 11 29/30 Write block 321 len 12288 data: 9 9 10 30/30 Write block 472 len 32768 data: 12 12 13 13 13 13 11 12 ---------------------------------------------------------------- Crucible gen:148 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 26 Acked 1025 Write 4 Done Done Done false 27 Acked 1026 Write 4 Done Done Done false 28 Acked 1027 Write 7 Done Done Done false 29 Acked 1028 Write 3 Done Done Done false 30 Acked 1029 Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1024 1024 1024 Downstairs last five completed: 1024 1023 1022 1021 1020 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:56:54.968595729Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1774} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 148 148 148 300 300 300 F F F 1 020-039 148 148 148 302 302 302 F F F 2 040-059 137 137 137 279 279 279 F F F 3 060-079 146 146 146 296 296 296 F F F 4 080-099 144 144 144 290 290 290 T T T 5 100-119 144 144 144 293 293 293 T T T 6 120-139 148 148 148 302 302 302 F F F 7 140-159 146 146 146 298 298 298 F F F 8 160-179 145 145 145 294 294 294 F F F 9 180-199 148 148 148 304 304 304 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 148 148 148 304 304 304 T T T 12 240-259 148 148 148 304 304 304 F F F 13 260-279 142 142 142 287 287 287 F F F 14 280-299 143 143 143 289 289 289 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 148 148 148 303 303 303 T T T 17 340-359 144 144 144 292 292 292 F F F 18 360-379 144 144 144 290 290 290 F F F 19 380-399 148 148 148 301 301 301 F F F 20 400-419 142 142 142 287 287 287 F F F 21 420-439 148 148 148 302 302 302 F F F 22 440-459 145 145 145 294 294 294 F F F 23 460-479 146 146 146 295 295 295 T T T 24 480-499 148 148 148 304 304 304 F F F 25 500-519 148 148 148 302 302 302 F F F 26 520-539 148 148 148 302 302 302 F F F 27 540-559 146 146 146 297 297 297 T T T 28 560-579 146 146 146 298 298 298 F F F 29 580-599 146 146 146 298 298 298 F F F Max gen: 148, Max flush: 304 A Difference in extent metadata is expected here On loop 74 Verifying data now No change after verify Loop: 74 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 148 148 148 300 300 300 F F F 1 020-039 148 148 148 302 302 302 F F F 2 040-059 137 137 137 279 279 279 F F F 3 060-079 146 146 146 296 296 296 F F F 4 080-099 149 149 149 305 305 305 F F F 5 100-119 149 149 149 305 305 305 F F F 6 120-139 148 148 148 302 302 302 F F F 7 140-159 146 146 146 298 298 298 F F F 8 160-179 145 145 145 294 294 294 F F F 9 180-199 148 148 148 304 304 304 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 149 149 149 305 305 305 F F F 12 240-259 148 148 148 304 304 304 F F F 13 260-279 142 142 142 287 287 287 F F F 14 280-299 143 143 143 289 289 289 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 149 149 149 305 305 305 F F F 17 340-359 144 144 144 292 292 292 F F F 18 360-379 144 144 144 290 290 290 F F F 19 380-399 148 148 148 301 301 301 F F F 20 400-419 142 142 142 287 287 287 F F F 21 420-439 148 148 148 302 302 302 F F F 22 440-459 145 145 145 294 294 294 F F F 23 460-479 149 149 149 305 305 305 F F F 24 480-499 148 148 148 304 304 304 F F F 25 500-519 148 148 148 302 302 302 F F F 26 520-539 148 148 148 302 302 302 F F F 27 540-559 149 149 149 305 305 305 F F F 28 560-579 146 146 146 298 298 298 F F F 29 580-599 146 146 146 298 298 298 F F F Max gen: 149, Max flush: 305 Begin loop 75 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.214611681Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.214958974Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.214981644Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782} {"msg":"Crucible d199769e-9978-44af-863a-3adb574b16a6 has session id: b1700697-9fd5-4baa-a101-d16273901612","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.21548154Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"Upstairs opts: Upstairs UUID: d199769e-9978-44af-863a-3adb574b16a6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.215526088Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"Crucible stats registered with UUID: d199769e-9978-44af-863a-3adb574b16a6","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.215554017Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"session_id":"b1700697-9fd5-4baa-a101-d16273901612"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:150","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.215817885Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782} {"msg":"d199769e-9978-44af-863a-3adb574b16a6 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.215933979Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.215953118Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"0","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.215974197Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"0","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.215988177Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"1","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.216001676Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"1","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.216022475Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.216047504Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.216071003Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"io task","client":"0","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.216095032Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"io task","client":"1","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.21611868Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"io task","client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:57:09.216300072Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"io task","client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:57:09.21634042Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"io task","client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:57:09.216453355Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.216483183Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.216506822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"ds_connection connected from Ok(127.0.0.1:56421)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.216522872Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"io task","client":"0","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"ds_connection connected from Ok(127.0.0.1:55435)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.216537331Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"io task","client":"1","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.216569339Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"io task","client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.217144632Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"1","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.217170621Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"1","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.217185071Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"0","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.21719866Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"0","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.217759414Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"0","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.217876538Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"1","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.218314968Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"0","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.218337727Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"0","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.218351726Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.218408684Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"1","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.218431352Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"1","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:09.218445402Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.217806983Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"io task","client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"ds_connection connected from Ok(127.0.0.1:59447)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.217976655Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"io task","client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.219131551Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.219158829Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.219886115Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220502337Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220535825Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"[0]R flush_numbers[0..12]: [300, 302, 279, 296, 305, 305, 302, 298, 294, 304, 302, 305]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220556514Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"downstairs","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"[0]R generation[0..12]: [148, 148, 137, 146, 149, 149, 148, 146, 145, 148, 148, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220575023Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"downstairs","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220594642Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"downstairs","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"[1]R flush_numbers[0..12]: [300, 302, 279, 296, 305, 305, 302, 298, 294, 304, 302, 305]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220612642Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"downstairs","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"[1]R generation[0..12]: [148, 148, 137, 146, 149, 149, 148, 146, 145, 148, 148, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220630101Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"downstairs","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.22065353Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"downstairs","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"[2]R flush_numbers[0..12]: [300, 302, 279, 296, 305, 305, 302, 298, 294, 304, 302, 305]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220671609Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"downstairs","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"[2]R generation[0..12]: [148, 148, 137, 146, 149, 149, 148, 146, 145, 148, 148, 149]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220688748Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"downstairs","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T18:57:19.220706297Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"downstairs","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"Max found gen is 150","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220724936Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"downstairs","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"Generation requested: 150 >= found:150","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220741406Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"downstairs","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"Next flush: 306","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220757245Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"downstairs","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220773324Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"":"downstairs","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220789993Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220804763Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"0","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220821662Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"1","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220838151Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"client":"2","":"downstairs","session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.22085471Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.22086989Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"d199769e-9978-44af-863a-3adb574b16a6 is now active with session: b1700697-9fd5-4baa-a101-d16273901612","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220885979Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220904268Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782,"session_id":"b1700697-9fd5-4baa-a101-d16273901612"} {"msg":"The guest has finished waiting for activation with:150","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:19.220919877Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 536 len 24576 data: 11 11 11 10 11 12 02/30 Read block 511 len 36864 03/30 Write block 62 len 40960 data: 2 4 5 7 6 6 5 5 5 7 04/30 Read block 245 len 36864 05/30 Read block 1 len 36864 06/30 Read block 291 len 32768 07/30 Write block 287 len 40960 data: 9 8 10 11 11 11 8 8 8 7 08/30 Write block 421 len 12288 data: 16 13 11 09/30 Read block 480 len 36864 10/30 Flush 11/30 Write block 530 len 24576 data: 11 14 13 15 13 12 12/30 Read block 381 len 4096 13/30 Write block 134 len 12288 data: 11 11 11 14/30 Write block 582 len 8192 data: 14 11 15/30 Write block 327 len 20480 data: 10 12 11 11 11 16/30 Write block 550 len 12288 data: 11 13 11 17/30 Write block 222 len 24576 data: 13 10 10 11 9 11 18/30 Read block 197 len 32768 19/30 Read block 187 len 28672 20/30 Read block 357 len 16384 21/30 Read block 102 len 40960 22/30 Read block 227 len 8192 23/30 Read block 248 len 32768 24/30 Read block 398 len 4096 25/30 Write block 19 len 12288 data: 9 12 13 26/30 Read block 371 len 28672 27/30 Read block 500 len 12288 28/30 Read block 250 len 32768 29/30 Read block 339 len 28672 30/30 Write block 75 len 36864 data: 13 15 15 15 15 18 17 15 15 ---------------------------------------------------------------- Crucible gen:150 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Read 10 Done Done Done false 23 Acked 1022 Read 2 Done Done Done false 24 Acked 1023 Read 8 Done Done Done false 25 Acked 1024 Read 1 Done Done Done false 26 Acked 1025 Write 3 Done Done Done false 27 Acked 1026 Read 7 Done Done Done false 28 Acked 1027 Read 3 Done Done Done false 29 Acked 1028 Read 8 Done Done Done false 30 Acked 1029 Read 7 Done Done Done false 31 Acked 1030 Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 9 9 9 27 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:57:19.247277377Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1782} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 148 148 148 300 300 300 T T T 1 020-039 148 148 148 302 302 302 T T T 2 040-059 137 137 137 279 279 279 F F F 3 060-079 150 150 150 306 306 306 T T T 4 080-099 149 149 149 305 305 305 T T T 5 100-119 149 149 149 305 305 305 F F F 6 120-139 150 150 150 307 307 307 F F F 7 140-159 146 146 146 298 298 298 F F F 8 160-179 145 145 145 294 294 294 F F F 9 180-199 148 148 148 304 304 304 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 150 150 150 307 307 307 F F F 12 240-259 148 148 148 304 304 304 F F F 13 260-279 142 142 142 287 287 287 F F F 14 280-299 150 150 150 306 306 306 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 150 150 150 307 307 307 F F F 17 340-359 144 144 144 292 292 292 F F F 18 360-379 144 144 144 290 290 290 F F F 19 380-399 148 148 148 301 301 301 F F F 20 400-419 142 142 142 287 287 287 F F F 21 420-439 150 150 150 306 306 306 F F F 22 440-459 145 145 145 294 294 294 F F F 23 460-479 149 149 149 305 305 305 F F F 24 480-499 148 148 148 304 304 304 F F F 25 500-519 148 148 148 302 302 302 F F F 26 520-539 150 150 150 307 307 307 F F F 27 540-559 150 150 150 307 307 307 F F F 28 560-579 146 146 146 298 298 298 F F F 29 580-599 150 150 150 307 307 307 F F F Max gen: 150, Max flush: 307 A Difference in extent metadata is expected here On loop 75 Verifying data now No change after verify Loop: 75 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 151 151 151 308 308 308 F F F 1 020-039 151 151 151 308 308 308 F F F 2 040-059 137 137 137 279 279 279 F F F 3 060-079 151 151 151 308 308 308 F F F 4 080-099 151 151 151 308 308 308 F F F 5 100-119 149 149 149 305 305 305 F F F 6 120-139 150 150 150 307 307 307 F F F 7 140-159 146 146 146 298 298 298 F F F 8 160-179 145 145 145 294 294 294 F F F 9 180-199 148 148 148 304 304 304 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 150 150 150 307 307 307 F F F 12 240-259 148 148 148 304 304 304 F F F 13 260-279 142 142 142 287 287 287 F F F 14 280-299 150 150 150 306 306 306 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 150 150 150 307 307 307 F F F 17 340-359 144 144 144 292 292 292 F F F 18 360-379 144 144 144 290 290 290 F F F 19 380-399 148 148 148 301 301 301 F F F 20 400-419 142 142 142 287 287 287 F F F 21 420-439 150 150 150 306 306 306 F F F 22 440-459 145 145 145 294 294 294 F F F 23 460-479 149 149 149 305 305 305 F F F 24 480-499 148 148 148 304 304 304 F F F 25 500-519 148 148 148 302 302 302 F F F 26 520-539 150 150 150 307 307 307 F F F 27 540-559 150 150 150 307 307 307 F F F 28 560-579 146 146 146 298 298 298 F F F 29 580-599 150 150 150 307 307 307 F F F Max gen: 151, Max flush: 308 Begin loop 76 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.465429801Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.465833802Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.465865121Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790} {"msg":"Crucible 4a6f41aa-a849-4ed0-a976-039c5dbd7b31 has session id: 8d84085f-2e26-48e7-b1d2-e4d39af32142","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.466121299Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"Upstairs opts: Upstairs UUID: 4a6f41aa-a849-4ed0-a976-039c5dbd7b31, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.466171666Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"Crucible stats registered with UUID: 4a6f41aa-a849-4ed0-a976-039c5dbd7b31","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.466196575Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:152","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.466539689Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790} {"msg":"4a6f41aa-a849-4ed0-a976-039c5dbd7b31 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.466563028Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.466576518Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.466596117Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.466610036Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"1","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.466623535Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"1","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.466636395Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"2","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.466653874Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"2","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.466677323Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"io task","client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.466697002Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"io task","client":"2","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.466710771Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"io task","client":"1","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:57:33.466996218Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"io task","client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:57:33.467094343Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"io task","client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"ds_connection connected from Ok(127.0.0.1:36984)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.467127702Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"io task","client":"2","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:57:33.46715875Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"ds_connection connected from Ok(127.0.0.1:58801)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.467184649Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"io task","client":"1","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.467200129Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.467213228Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.467292544Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"io task","client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.467760322Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"2","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.467786231Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"2","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.467800541Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"1","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.46781386Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"1","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.468354124Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"1","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.468472039Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"2","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.46887838Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"1","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.468916078Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"1","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.468949977Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.468976026Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"2","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.469004444Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"2","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:33.469029483Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.468548229Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"io task","client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"ds_connection connected from Ok(127.0.0.1:33820)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.468708001Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"io task","client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.470148644Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.470181352Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.470712798Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471228274Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471249643Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"[0]R flush_numbers[0..12]: [308, 308, 279, 308, 308, 305, 307, 298, 294, 304, 302, 307]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471264282Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"downstairs","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"[0]R generation[0..12]: [151, 151, 137, 151, 151, 149, 150, 146, 145, 148, 148, 150]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471278571Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"downstairs","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471292801Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"downstairs","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"[1]R flush_numbers[0..12]: [308, 308, 279, 308, 308, 305, 307, 298, 294, 304, 302, 307]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.47130669Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"downstairs","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"[1]R generation[0..12]: [151, 151, 137, 151, 151, 149, 150, 146, 145, 148, 148, 150]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471325139Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"downstairs","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471339329Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"downstairs","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"[2]R flush_numbers[0..12]: [308, 308, 279, 308, 308, 305, 307, 298, 294, 304, 302, 307]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471353228Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"downstairs","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"[2]R generation[0..12]: [151, 151, 137, 151, 151, 149, 150, 146, 145, 148, 148, 150]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471367117Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"downstairs","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471381036Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"downstairs","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"Max found gen is 152","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471394066Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"downstairs","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"Generation requested: 152 >= found:152","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471407155Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"downstairs","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"Next flush: 309","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471420105Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"downstairs","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471447223Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"":"downstairs","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471460933Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471473052Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"0","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471486282Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"1","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471499321Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"client":"2","":"downstairs","session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.47151244Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"Wait for a query_work_queue command to finish before sending IO Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.47152842Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"4a6f41aa-a849-4ed0-a976-039c5dbd7b31 is now active with session: 8d84085f-2e26-48e7-b1d2-e4d39af32142","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471543669Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471555798Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790,"session_id":"8d84085f-2e26-48e7-b1d2-e4d39af32142"} {"msg":"The guest has finished waiting for activation with:152","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:43.471567818Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 189 len 36864 data: 17 17 16 11 14 12 13 14 15 02/30 Read block 182 len 32768 03/30 Read block 526 len 32768 04/30 Write block 389 len 4096 data: 13 05/30 Read block 230 len 32768 06/30 Read block 353 len 36864 07/30 Write block 433 len 24576 data: 13 10 13 11 9 8 08/30 Write block 450 len 40960 data: 13 11 13 10 8 10 9 8 9 9 09/30 Read block 313 len 24576 10/30 Write block 391 len 8192 data: 16 17 11/30 Read block 22 len 28672 12/30 Write block 372 len 16384 data: 15 14 11 13 13/30 Flush 14/30 Write block 8 len 12288 data: 8 9 9 15/30 Read block 289 len 24576 16/30 Read block 390 len 16384 17/30 Read block 23 len 20480 18/30 Write block 125 len 24576 data: 10 9 11 12 11 11 19/30 Write block 463 len 36864 data: 5 5 5 6 9 9 9 9 11 20/30 Read block 266 len 4096 21/30 Write block 135 len 8192 data: 12 12 22/30 Read block 49 len 32768 23/30 Write block 154 len 20480 data: 10 9 9 10 10 24/30 Read block 156 len 28672 25/30 Read block 231 len 28672 26/30 Read block 281 len 28672 27/30 Write block 375 len 36864 data: 14 14 15 18 17 14 14 14 16 28/30 Write block 47 len 32768 data: 12 12 13 12 9 11 10 9 29/30 Write block 405 len 40960 data: 15 16 16 12 11 10 8 8 9 10 30/30 Write block 138 len 20480 data: 9 8 7 9 12 ---------------------------------------------------------------- Crucible gen:152 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 2 Done Done Done false 23 Acked 1022 Read 8 Done Done Done false 24 Acked 1023 Write 5 Done Done Done false 25 Acked 1024 Read 7 Done Done Done false 26 Acked 1025 Read 7 Done Done Done false 27 Acked 1026 Read 7 Done Done Done false 28 Acked 1027 Write 9 Done Done Done false 29 Acked 1028 Write 8 Done Done Done false 30 Acked 1029 Write 10 Sent Done Done false 31 Acked 1030 Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 2 1 1 4 Done 8 9 9 26 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:57:43.497379863Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1790} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 152 152 152 310 310 310 F F F 1 020-039 151 151 151 308 308 308 F F F 2 040-059 137 137 137 279 279 279 T T T 3 060-079 151 151 151 308 308 308 F F F 4 080-099 151 151 151 308 308 308 F F F 5 100-119 149 149 149 305 305 305 F F F 6 120-139 152 152 152 310 310 310 T T T 7 140-159 146 146 146 298 298 298 T T T 8 160-179 145 145 145 294 294 294 F F F 9 180-199 152 152 152 309 309 309 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 150 150 150 307 307 307 F F F 12 240-259 148 148 148 304 304 304 F F F 13 260-279 142 142 142 287 287 287 F F F 14 280-299 150 150 150 306 306 306 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 150 150 150 307 307 307 F F F 17 340-359 144 144 144 292 292 292 F F F 18 360-379 152 152 152 309 309 309 T T T 19 380-399 152 152 152 309 309 309 T T T 20 400-419 142 142 142 287 287 287 T T T 21 420-439 152 152 152 309 309 309 F F F 22 440-459 152 152 152 309 309 309 F F F 23 460-479 152 152 152 310 310 310 F F F 24 480-499 148 148 148 304 304 304 F F F 25 500-519 148 148 148 302 302 302 F F F 26 520-539 150 150 150 307 307 307 F F F 27 540-559 150 150 150 307 307 307 F F F 28 560-579 146 146 146 298 298 298 F F F 29 580-599 150 150 150 307 307 307 F F F Max gen: 152, Max flush: 310 A Difference in extent metadata is expected here On loop 76 Verifying data now No change after verify Loop: 76 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 152 152 152 310 310 310 F F F 1 020-039 151 151 151 308 308 308 F F F 2 040-059 153 153 153 311 311 311 F F F 3 060-079 151 151 151 308 308 308 F F F 4 080-099 151 151 151 308 308 308 F F F 5 100-119 149 149 149 305 305 305 F F F 6 120-139 153 153 153 311 311 311 F F F 7 140-159 153 153 153 311 311 311 F F F 8 160-179 145 145 145 294 294 294 F F F 9 180-199 152 152 152 309 309 309 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 150 150 150 307 307 307 F F F 12 240-259 148 148 148 304 304 304 F F F 13 260-279 142 142 142 287 287 287 F F F 14 280-299 150 150 150 306 306 306 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 150 150 150 307 307 307 F F F 17 340-359 144 144 144 292 292 292 F F F 18 360-379 153 153 153 311 311 311 F F F 19 380-399 153 153 153 311 311 311 F F F 20 400-419 153 153 153 311 311 311 F F F 21 420-439 152 152 152 309 309 309 F F F 22 440-459 152 152 152 309 309 309 F F F 23 460-479 152 152 152 310 310 310 F F F 24 480-499 148 148 148 304 304 304 F F F 25 500-519 148 148 148 302 302 302 F F F 26 520-539 150 150 150 307 307 307 F F F 27 540-559 150 150 150 307 307 307 F F F 28 560-579 146 146 146 298 298 298 F F F 29 580-599 150 150 150 307 307 307 F F F Max gen: 153, Max flush: 311 Begin loop 77 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.735223358Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.735780322Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.73581622Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798} {"msg":"Crucible d915515a-4b21-420d-8d60-fd38918438d1 has session id: 59612c3f-a527-4bc1-b623-21b794e63f99","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.73602367Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"Upstairs opts: Upstairs UUID: d915515a-4b21-420d-8d60-fd38918438d1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.736068178Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"Crucible stats registered with UUID: d915515a-4b21-420d-8d60-fd38918438d1","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.736092057Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:154","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.73646439Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798} {"msg":"d915515a-4b21-420d-8d60-fd38918438d1 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.736513278Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.736529177Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"0","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.736548866Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"0","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.736562655Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"1","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.736576415Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"1","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.736589324Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.736602613Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.736619622Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"io task","client":"0","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.736633492Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"io task","client":"1","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.736647561Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"io task","client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:57:57.736933048Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"io task","client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:57:57.737057002Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"io task","client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"ds_connection connected from Ok(127.0.0.1:49511)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.737082341Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"io task","client":"0","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:57:57.73710859Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.737128639Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.737149048Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"ds_connection connected from Ok(127.0.0.1:59513)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.737170097Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"io task","client":"1","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.737191696Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"io task","client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.737665164Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"1","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.737697232Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"1","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.737723461Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"0","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.73775079Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"0","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.738266176Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"0","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.73839747Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"1","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.738756733Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"0","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.738782801Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"0","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.738797211Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.738888537Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"1","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.738907576Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"1","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:57:57.738920905Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.738560285Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"io task","client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"ds_connection connected from Ok(127.0.0.1:37351)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.738809934Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"io task","client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.73973895Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.739765999Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.740363951Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.740911636Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.740939394Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"[0]R flush_numbers[0..12]: [310, 308, 311, 308, 308, 305, 311, 311, 294, 309, 302, 307]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.740957853Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"downstairs","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"[0]R generation[0..12]: [152, 151, 153, 151, 151, 149, 153, 153, 145, 152, 148, 150]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.740975663Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"downstairs","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.740993722Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"downstairs","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"[1]R flush_numbers[0..12]: [310, 308, 311, 308, 308, 305, 311, 311, 294, 309, 302, 307]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741018241Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"downstairs","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"[1]R generation[0..12]: [152, 151, 153, 151, 151, 149, 153, 153, 145, 152, 148, 150]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.74103591Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"downstairs","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741053619Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"downstairs","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"[2]R flush_numbers[0..12]: [310, 308, 311, 308, 308, 305, 311, 311, 294, 309, 302, 307]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741070518Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"downstairs","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"[2]R generation[0..12]: [152, 151, 153, 151, 151, 149, 153, 153, 145, 152, 148, 150]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741087538Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"downstairs","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741104987Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"downstairs","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"Max found gen is 154","v":0Wait for a query_work_queue command to finish before sending IO ,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741125276Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"downstairs","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"Generation requested: 154 >= found:154","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741142035Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"downstairs","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"Next flush: 312","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741158004Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"downstairs","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741173923Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"":"downstairs","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741190123Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741205062Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"0","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741221621Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"1","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.74123799Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"client":"2","":"downstairs","session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.74125663Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741271939Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"d915515a-4b21-420d-8d60-fd38918438d1 is now active with session: 59612c3f-a527-4bc1-b623-21b794e63f99","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741287768Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741302607Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798,"session_id":"59612c3f-a527-4bc1-b623-21b794e63f99"} {"msg":"The guest has finished waiting for activation with:154","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:07.741317577Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 470 len 8192 data: 10 12 02/30 Write block 42 len 24576 data: 6 7 8 11 9 13 03/30 Write block 241 len 28672 data: 12 11 10 10 11 10 11 04/30 Write block 81 len 36864 data: 18 16 16 18 18 17 13 9 11 05/30 Write block 35 len 8192 data: 7 11 06/30 Read block 209 len 8192 07/30 Read block 303 len 40960 08/30 Read block 297 len 24576 09/30 Write block 274 len 4096 data: 10 10/30 Read block 294 len 36864 11/30 Read block 377 len 12288 12/30 Read block 151 len 12288 13/30 Write block 32 len 4096 data: 5 14/30 Write block 402 len 12288 data: 14 13 15 15/30 Read block 41 len 4096 16/30 Write block 92 len 24576 data: 12 11 10 10 9 11 17/30 Read block 293 len 40960 18/30 Read block 412 len 32768 19/30 Write block 81 len 20480 data: 19 17 17 19 19 20/30 Read block 502 len 40960 21/30 Write block 110 len 32768 data: 8 8 7 7 10 11 11 10 22/30 Write block 255 len 4096 data: 13 23/30 Read block 127 len 20480 24/30 Read block 154 len 8192 25/30 Read block 521 len 8192 26/30 Read block 222 len 4096 27/30 Write block 489 len 20480 data: 10 10 9 9 6 28/30 Write block 72 len 28672 data: 8 11 11 14 16 16 16 29/30 Write block 145 len 40960 data: 14 13 12 12 11 13 9 9 10 11 30/30 Read block 301 len 20480 ---------------------------------------------------------------- Crucible gen:154 GIO:true work queues: Upstairs:0 downstairs:6 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 26 Acked 1025 Read 2 Done Done Done false 27 Acked 1026 Read 1 Done Done Done false 28 Acked 1027 Write 5 Done Done Done false 29 Acked 1028 Write 7 Done Done Done false 30 Acked 1029 Write 10 Done Done Done false 31 Acked 1030 Read 5 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 1 2 Done 5 6 5 16 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1024 1024 1024 Downstairs last five completed: 1024 1023 1022 1021 1020 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:58:07.764937945Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1798} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 152 152 152 310 310 310 F F F 1 020-039 154 154 154 312 312 312 F F F 2 040-059 154 154 154 312 312 312 F F F 3 060-079 151 151 151 308 308 308 T T T 4 080-099 154 154 154 312 312 312 F F F 5 100-119 154 154 154 312 312 312 F F F 6 120-139 153 153 153 311 311 311 F F F 7 140-159 153 153 153 311 311 311 T T T 8 160-179 145 145 145 294 294 294 F F F 9 180-199 152 152 152 309 309 309 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 150 150 150 307 307 307 F F F 12 240-259 154 154 154 312 312 312 F F F 13 260-279 154 154 154 312 312 312 F F F 14 280-299 150 150 150 306 306 306 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 150 150 150 307 307 307 F F F 17 340-359 144 144 144 292 292 292 F F F 18 360-379 153 153 153 311 311 311 F F F 19 380-399 153 153 153 311 311 311 F F F 20 400-419 154 154 154 312 312 312 F F F 21 420-439 152 152 152 309 309 309 F F F 22 440-459 152 152 152 309 309 309 F F F 23 460-479 154 154 154 312 312 312 F F F 24 480-499 148 148 148 304 304 304 T T T 25 500-519 148 148 148 302 302 302 F F F 26 520-539 150 150 150 307 307 307 F F F 27 540-559 150 150 150 307 307 307 F F F 28 560-579 146 146 146 298 298 298 F F F 29 580-599 150 150 150 307 307 307 F F F Max gen: 154, Max flush: 312 A Difference in extent metadata is expected here On loop 77 Verifying data now No change after verify Loop: 77 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 152 152 152 310 310 310 F F F 1 020-039 154 154 154 312 312 312 F F F 2 040-059 154 154 154 312 312 312 F F F 3 060-079 155 155 155 313 313 313 F F F 4 080-099 154 154 154 312 312 312 F F F 5 100-119 154 154 154 312 312 312 F F F 6 120-139 153 153 153 311 311 311 F F F 7 140-159 155 155 155 313 313 313 F F F 8 160-179 145 145 145 294 294 294 F F F 9 180-199 152 152 152 309 309 309 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 150 150 150 307 307 307 F F F 12 240-259 154 154 154 312 312 312 F F F 13 260-279 154 154 154 312 312 312 F F F 14 280-299 150 150 150 306 306 306 F F F 15 300-319 137 137 137 279 279 279 F F F 16 320-339 150 150 150 307 307 307 F F F 17 340-359 144 144 144 292 292 292 F F F 18 360-379 153 153 153 311 311 311 F F F 19 380-399 153 153 153 311 311 311 F F F 20 400-419 154 154 154 312 312 312 F F F 21 420-439 152 152 152 309 309 309 F F F 22 440-459 152 152 152 309 309 309 F F F 23 460-479 154 154 154 312 312 312 F F F 24 480-499 155 155 155 313 313 313 F F F 25 500-519 148 148 148 302 302 302 F F F 26 520-539 150 150 150 307 307 307 F F F 27 540-559 150 150 150 307 307 307 F F F 28 560-579 146 146 146 298 298 298 F F F 29 580-599 150 150 150 307 307 307 F F F Max gen: 155, Max flush: 313 Begin loop 78 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.97167986Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.971973237Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.971992176Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806} {"msg":"Crucible 94f56d68-1182-46ad-9de2-32e8edf8d421 has session id: 3e4d37c5-28ee-4a30-adf1-c1e261fdca4c","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.972430045Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"Upstairs opts: Upstairs UUID: 94f56d68-1182-46ad-9de2-32e8edf8d421, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.972472153Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"Crucible stats registered with UUID: 94f56d68-1182-46ad-9de2-32e8edf8d421","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.972488763Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:156","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.972881444Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806} {"msg":"94f56d68-1182-46ad-9de2-32e8edf8d421 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.972947911Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.97297888Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.973013218Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.973039077Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"1","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.973062876Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"1","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.973087455Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"2","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.973102934Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"2","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.973116323Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"io task","client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.973130163Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"io task","client":"2","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.97318074Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"io task","client":"1","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"ds_connection connected from Ok(127.0.0.1:38290)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.973520135Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"io task","client":"1","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"ds_connection connected from Ok(127.0.0.1:46765)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.973558643Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"io task","client":"2","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:58:21.973590931Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"io task","client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:58:21.97361844Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"io task","client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:58:21.973648489Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.973663198Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.973676707Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.973813711Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"io task","client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.974359715Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"1","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.974391434Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"1","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.974418602Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"2","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.974446751Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"2","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.97491121Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"2","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.975029054Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"1","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.975421376Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"2","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.975458934Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"2","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.975485883Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.97554301Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"1","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.975570629Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"1","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:21.975586358Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.975282088Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"io task","client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"ds_connection connected from Ok(127.0.0.1:49616)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.975693359Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"io task","client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.976634985Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.976662344Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.97715515Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977694575Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977718254Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"[0]R flush_numbers[0..12]: [310, 312, 312, 313, 312, 312, 311, 313, 294, 309, 302, 307]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977733084Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"downstairs","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"[0]R generation[0..12]: [152, 154, 154, 155, 154, 154, 153, 155, 145, 152, 148, 150]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977747473Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"downstairs","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977767972Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"downstairs","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"[1]R flush_numbers[0..12]: [310, 312, 312, 313, 312, 312, 311, 313, 294, 309, 302, 307]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977782481Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"downstairs","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"[1]R generation[0..12]: [152, 154, 154, 155, 154, 154, 153, 155, 145, 152, 148, 150]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977796081Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"downstairs","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.97781008Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"downstairs","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"[2]R flush_numbers[0..12]: [310, 312, 312, 313, 312, 312, 311, 313, 294, 309, 302, 307]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977823909Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"downstairs","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"[2]R generation[0..12]: [152, 154, 154, 155, 154, 154, 153, 155, 145, 152, 148, 150]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977837649Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"downstairs","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977851648Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"downstairs","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"Max found gen is 156","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977864768Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"downstairs","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"Generation requested: 156 >= found:156","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977877927Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"downstairs","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"Next flush: 314","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977890796Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"downstairs","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977903576Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"":"downstairs","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977920925Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977932884Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"0","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977946474Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"1","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977959773Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"client":"2","":"downstairs","session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977975202Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.977987742Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"94f56d68-1182-46ad-9de2-32e8edf8d421 is now active with session: 3e4d37c5-28ee-4a30-adf1-c1e261fdca4c","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.978000711Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.978012541Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806,"session_id":"3e4d37c5-28ee-4a30-adf1-c1e261fdca4c"} {"msg":"The guest has finished waiting for activation with:156","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:31.97802486Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 550 len 8192 data: 12 14 02/30 Write block 254 len 8192 data: 13 14 03/30 Flush 04/30 Write block 76 len 24576 data: 17 17 17 16 19 20 05/30 Read block 389 len 12288 06/30 Read block 396 len 32768 07/30 Flush 08/30 Flush 09/30 Write block 313 len 12288 data: 16 20 19 10/30 Flush 11/30 Write block 388 len 24576 data: 15 14 15 17 18 13 12/30 Write block 125 len 4096 data: 11 13/30 Write block 416 len 28672 data: 12 14 16 19 19 17 14 14/30 Flush 15/30 Write block 447 len 36864 data: 11 14 13 14 12 14 11 9 11 16/30 Flush 17/30 Write block 406 len 12288 data: 17 17 13 18/30 Write block 459 len 20480 data: 10 10 10 7 6 19/30 Write block 32 len 32768 data: 6 6 7 8 12 10 11 11 20/30 Write block 587 len 16384 data: 7 9 8 11 21/30 Write block 260 len 4096 data: 13 22/30 Read block 382 len 24576 23/30 Write block 268 len 20480 data: 12 11 9 7 8 24/30 Read block 175 len 24576 25/30 Write block 165 len 40960 data: 11 13 14 15 15 16 14 12 11 9 26/30 Write block 387 len 24576 data: 17 16 15 16 18 19 27/30 Write block 71 len 20480 data: 8 9 12 12 15 28/30 Write block 416 len 28672 data: 13 15 17 20 20 18 15 29/30 Read block 180 len 12288 30/30 Write block 108 len 24576 data: 10 8 9 9 8 8 ---------------------------------------------------------------- Crucible gen:156 GIO:true work queues: Upstairs:0 downstairs:14 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 17 Acked 1016 Write 3 Done Done Done false 18 Acked 1017 Write 5 Done Done Done false 19 Acked 1018 Write 8 Done Done Done false 20 Acked 1019 Write 4 Done Done Done false 21 Acked 1020 Write 1 Done Done Done false 22 Acked 1021 Read 6 Done Done Done false 23 Acked 1022 Write 5 Done Done Done false 24 Acked 1023 Read 6 Done Done Done false 25 Acked 1024 Write 10 Done Done Done false 26 Acked 1025 Write 6 Done Done Done false 27 Acked 1026 Write 5 Done Done Done false 28 Acked 1027 Write 7 Done Done Done false 29 Acked 1028 Read 3 Done Done Done false 30 Acked 1029 Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 13 13 13 39 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1015 1015 1015 Downstairs last five completed: 1015 1014 1013 1012 1011 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:58:32.004710145Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1806} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 152 152 152 310 310 310 F F F 1 020-039 154 154 154 312 312 312 T T T 2 040-059 154 154 154 312 312 312 F F F 3 060-079 156 156 156 315 315 315 T T T 4 080-099 156 156 156 315 315 315 F F F 5 100-119 154 154 154 312 312 312 T T T 6 120-139 156 156 156 318 318 318 F F F 7 140-159 155 155 155 313 313 313 F F F 8 160-179 145 145 145 294 294 294 T T T 9 180-199 152 152 152 309 309 309 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 150 150 150 307 307 307 F F F 12 240-259 156 156 156 314 314 314 F F F 13 260-279 154 154 154 312 312 312 T T T 14 280-299 150 150 150 306 306 306 F F F 15 300-319 156 156 156 317 317 317 F F F 16 320-339 150 150 150 307 307 307 F F F 17 340-359 144 144 144 292 292 292 F F F 18 360-379 153 153 153 311 311 311 F F F 19 380-399 156 156 156 318 318 318 T T T 20 400-419 156 156 156 318 318 318 T T T 21 420-439 156 156 156 318 318 318 T T T 22 440-459 156 156 156 319 319 319 T T T 23 460-479 154 154 154 312 312 312 T T T 24 480-499 155 155 155 313 313 313 F F F 25 500-519 148 148 148 302 302 302 F F F 26 520-539 150 150 150 307 307 307 F F F 27 540-559 156 156 156 314 314 314 F F F 28 560-579 146 146 146 298 298 298 F F F 29 580-599 150 150 150 307 307 307 T T T Max gen: 156, Max flush: 319 A Difference in extent metadata is expected here On loop 78 Verifying data now No change after verify Loop: 78 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 152 152 152 310 310 310 F F F 1 020-039 157 157 157 320 320 320 F F F 2 040-059 154 154 154 312 312 312 F F F 3 060-079 157 157 157 320 320 320 F F F 4 080-099 156 156 156 315 315 315 F F F 5 100-119 157 157 157 320 320 320 F F F 6 120-139 156 156 156 318 318 318 F F F 7 140-159 155 155 155 313 313 313 F F F 8 160-179 157 157 157 320 320 320 F F F 9 180-199 152 152 152 309 309 309 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 150 150 150 307 307 307 F F F 12 240-259 156 156 156 314 314 314 F F F 13 260-279 157 157 157 320 320 320 F F F 14 280-299 150 150 150 306 306 306 F F F 15 300-319 156 156 156 317 317 317 F F F 16 320-339 150 150 150 307 307 307 F F F 17 340-359 144 144 144 292 292 292 F F F 18 360-379 153 153 153 311 311 311 F F F 19 380-399 157 157 157 320 320 320 F F F 20 400-419 157 157 157 320 320 320 F F F 21 420-439 157 157 157 320 320 320 F F F 22 440-459 157 157 157 320 320 320 F F F 23 460-479 157 157 157 320 320 320 F F F 24 480-499 155 155 155 313 313 313 F F F 25 500-519 148 148 148 302 302 302 F F F 26 520-539 150 150 150 307 307 307 F F F 27 540-559 156 156 156 314 314 314 F F F 28 560-579 146 146 146 298 298 298 F F F 29 580-599 157 157 157 320 320 320 F F F Max gen: 157, Max flush: 320 Begin loop 79 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.295076548Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.295469919Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.295491729Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814} {"msg":"Crucible e20079c2-0ebf-4db9-9289-57e4a09fa881 has session id: 2a256760-31b8-4609-9d41-08323908cac3","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.295781645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"Upstairs opts: Upstairs UUID: e20079c2-0ebf-4db9-9289-57e4a09fa881, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.295829283Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"Crucible stats registered with UUID: e20079c2-0ebf-4db9-9289-57e4a09fa881","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.295858252Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"session_id":"2a256760-31b8-4609-9d41-08323908cac3"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:158","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.296162277Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814} {"msg":"e20079c2-0ebf-4db9-9289-57e4a09fa881 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.29631544Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.296339709Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.296357358Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.296371137Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"1","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.296384967Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"1","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.296405666Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"2","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.296430375Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"2","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.296449164Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"io task","client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.296463233Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"io task","client":"1","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.296483742Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"io task","client":"2","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"ds_connection connected from Ok(127.0.0.1:61297)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.296667654Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"io task","client":"2","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"ds_connection connected from Ok(127.0.0.1:61183)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.296703152Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"io task","client":"1","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:58:46.296730551Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"io task","client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:58:46.296755539Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"io task","client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:58:46.296780248Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.296856185Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.296884424Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.297099223Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"io task","client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.297616919Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"2","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.297654648Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"2","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.297684126Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"1","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.297711215Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"1","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.298324696Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"1","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.298349005Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"2","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.298903919Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"2","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.298926738Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"2","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.298940608Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.298952797Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"1","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.298966226Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"1","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:46.298979176Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.297815393Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"io task","client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"ds_connection connected from Ok(127.0.0.1:47059)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.297953086Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"io task","client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.299411478Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.299442726Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.299932524Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300556674Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300583573Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"[0]R flush_numbers[0..12]: [310, 320, 312, 320, 315, 320, 318, 313, 320, 309, 302, 307]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300606542Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"downstairs","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"[0]R generation[0..12]: [152, 157, 154, 157, 156, 157, 156, 155, 157, 152, 148, 150]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300624661Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"downstairs","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.30064235Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"downstairs","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"[1]R flush_numbers[0..12]: [310, 320, 312, 320, 315, 320, 318, 313, 320, 309, 302, 307]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.3006598Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"downstairs","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"[1]R generation[0..12]: [152, 157, 154, 157, 156, 157, 156, 155, 157, 152, 148, 150]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300677049Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"downstairs","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300694448Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"downstairs","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"[2]R flush_numbers[0..12]: [310, 320, 312, 320, 315, 320, 318, 313, 320, 309, 302, 307]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300711477Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"downstairs","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"[2]R generation[0..12]: [152, 157, 154, 157, 156, 157, 156, 155, 157, 152, 148, 150]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300728497Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"downstairs","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300745816Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"downstairs","":"downstairs"Wait for a query_work_queue command to finish before sending IO ,"session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"Max found gen is 158","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300766135Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"downstairs","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"Generation requested: 158 >= found:158","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300782604Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"downstairs","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"Next flush: 321","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300798533Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"downstairs","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300814433Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"":"downstairs","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300830882Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300848661Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"0","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.30086251Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"1","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.30087589Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"client":"2","":"downstairs","session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300889219Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300901509Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"e20079c2-0ebf-4db9-9289-57e4a09fa881 is now active with session: 2a256760-31b8-4609-9d41-08323908cac3","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300914228Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300926147Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814,"session_id":"2a256760-31b8-4609-9d41-08323908cac3"} {"msg":"The guest has finished waiting for activation with:158","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:58:56.300938327Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 228 len 36864 data: 11 12 12 11 14 12 13 15 16 02/30 Read block 222 len 20480 03/30 Write block 345 len 8192 data: 8 7 04/30 Read block 551 len 8192 05/30 Read block 460 len 32768 06/30 Write block 551 len 20480 data: 15 12 13 14 13 07/30 Flush 08/30 Write block 119 len 24576 data: 13 13 12 13 12 11 09/30 Read block 297 len 32768 10/30 Write block 178 len 12288 data: 8 10 7 11/30 Flush 12/30 Write block 514 len 8192 data: 13 11 13/30 Read block 337 len 20480 14/30 Write block 66 len 8192 data: 7 7 15/30 Read block 53 len 24576 16/30 Flush 17/30 Write block 447 len 24576 data: 12 15 14 15 13 15 18/30 Read block 338 len 40960 19/30 Write block 553 len 8192 data: 14 15 20/30 Write block 463 len 16384 data: 7 6 6 7 21/30 Write block 471 len 4096 data: 13 22/30 Read block 462 len 4096 23/30 Write block 112 len 8192 data: 9 9 24/30 Write block 590 len 28672 data: 12 10 10 11 10 10 8 25/30 Read block 361 len 20480 26/30 Read block 429 len 4096 27/30 Read block 83 len 20480 28/30 Read block 393 len 16384 29/30 Read block 178 len 28672 30/30 Write block 333 len 40960 data: 12 15 14 13 14 11 8 8 10 11 ---------------------------------------------------------------- Crucible gen:158 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 1 Done Done Done false 23 Acked 1022 Read 1 Done Done Done false 24 Acked 1023 Write 2 Done Done Done false 25 Acked 1024 Write 7 Done Done Done false 26 Acked 1025 Read 5 Done Done Done false 27 Acked 1026 Read 1 Done Done Done false 28 Acked 1027 Read 5 Done Done Done false 29 Acked 1028 Read 4 Done Done Done false 30 Acked 1029 Read 7 Done Done Done false 31 Acked 1030 Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 9 9 9 27 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:58:56.327022899Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1814} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 152 152 152 310 310 310 F F F 1 020-039 157 157 157 320 320 320 F F F 2 040-059 154 154 154 312 312 312 F F F 3 060-079 158 158 158 323 323 323 F F F 4 080-099 156 156 156 315 315 315 F F F 5 100-119 158 158 158 322 322 322 T T T 6 120-139 158 158 158 322 322 322 F F F 7 140-159 155 155 155 313 313 313 F F F 8 160-179 158 158 158 322 322 322 F F F 9 180-199 158 158 158 322 322 322 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 158 158 158 321 321 321 F F F 12 240-259 156 156 156 314 314 314 F F F 13 260-279 157 157 157 320 320 320 F F F 14 280-299 150 150 150 306 306 306 F F F 15 300-319 156 156 156 317 317 317 F F F 16 320-339 150 150 150 307 307 307 T T T 17 340-359 158 158 158 321 321 321 T T T 18 360-379 153 153 153 311 311 311 F F F 19 380-399 157 157 157 320 320 320 F F F 20 400-419 157 157 157 320 320 320 F F F 21 420-439 157 157 157 320 320 320 F F F 22 440-459 158 158 158 324 324 324 F F F 23 460-479 158 158 158 324 324 324 T T T 24 480-499 155 155 155 313 313 313 F F F 25 500-519 158 158 158 323 323 323 F F F 26 520-539 150 150 150 307 307 307 F F F 27 540-559 158 158 158 324 324 324 F F F 28 560-579 146 146 146 298 298 298 F F F 29 580-599 157 157 157 320 320 320 T T T Max gen: 158, Max flush: 324 A Difference in extent metadata is expected here On loop 79 Verifying data now No change after verify Loop: 79 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 152 152 152 310 310 310 F F F 1 020-039 157 157 157 320 320 320 F F F 2 040-059 154 154 154 312 312 312 F F F 3 060-079 158 158 158 323 323 323 F F F 4 080-099 156 156 156 315 315 315 F F F 5 100-119 159 159 159 325 325 325 F F F 6 120-139 158 158 158 322 322 322 F F F 7 140-159 155 155 155 313 313 313 F F F 8 160-179 158 158 158 322 322 322 F F F 9 180-199 158 158 158 322 322 322 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 158 158 158 321 321 321 F F F 12 240-259 156 156 156 314 314 314 F F F 13 260-279 157 157 157 320 320 320 F F F 14 280-299 150 150 150 306 306 306 F F F 15 300-319 156 156 156 317 317 317 F F F 16 320-339 159 159 159 325 325 325 F F F 17 340-359 159 159 159 325 325 325 F F F 18 360-379 153 153 153 311 311 311 F F F 19 380-399 157 157 157 320 320 320 F F F 20 400-419 157 157 157 320 320 320 F F F 21 420-439 157 157 157 320 320 320 F F F 22 440-459 158 158 158 324 324 324 F F F 23 460-479 159 159 159 325 325 325 F F F 24 480-499 155 155 155 313 313 313 F F F 25 500-519 158 158 158 323 323 323 F F F 26 520-539 150 150 150 307 307 307 F F F 27 540-559 158 158 158 324 324 324 F F F 28 560-579 146 146 146 298 298 298 F F F 29 580-599 159 159 159 325 325 325 F F F Max gen: 159, Max flush: 325 Begin loop 80 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.544926102Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.545343662Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.545372821Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822} {"msg":"Crucible 025142c8-df9c-4baa-8a68-f905b1ebcf2d has session id: 69abe55e-d797-4bc5-b891-ddb435d38350","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.545407469Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"Upstairs opts: Upstairs UUID: 025142c8-df9c-4baa-8a68-f905b1ebcf2d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.545440508Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"Crucible stats registered with UUID: 025142c8-df9c-4baa-8a68-f905b1ebcf2d","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.545463477Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:160","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.545962963Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822} {"msg":"025142c8-df9c-4baa-8a68-f905b1ebcf2d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.546049139Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.546076148Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"0","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.546098217Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"0","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.546112696Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"1","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.546126426Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"1","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.546139675Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.546157064Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.546170324Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"io task","client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.546184083Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"io task","client":"0","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.546201002Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"io task","client":"1","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"ds_connection connected from Ok(127.0.0.1:65359)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.546498808Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"io task","client":"0","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"ds_connection connected from Ok(127.0.0.1:60805)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.546687169Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"io task","client":"1","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:59:10.546780895Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"io task","client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:59:10.546803884Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"io task","client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:59:10.546833943Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.546846942Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.546860081Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.547017244Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"io task","client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.547243394Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"0","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.547271302Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"0","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.547372217Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"1","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.547432595Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"1","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.547721071Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"0","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.547828226Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"1","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.548226308Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"0","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.548252867Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"0","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.548267576Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.548323953Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"1","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.548342812Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"1","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:10.548356312Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.548118645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"io task","client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"ds_connection connected from Ok(127.0.0.1:52624)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.548244799Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"io task","client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.549345117Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.549378966Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.549828875Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550411758Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550445986Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"[0]R flush_numbers[0..12]: [310, 320, 312, 323, 315, 325, 322, 313, 322, 322, 302, 321]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550465815Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"downstairs","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"[0]R generation[0..12]: [152, 157, 154, 158, 156, 159, 158, 155, 158, 158, 148, 158]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550483524Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"downstairs","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550501474Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"downstairs","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"[1]R flush_numbers[0..12]: [310, 320, 312, 323, 315, 325, 322, 313, 322, 322, 302, 321]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550519093Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"downstairs","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"[1]R generation[0..12]: [152, 157, 154, 158, 156, 159, 158, 155, 158, 158, 148, 158]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550536512Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"downstairs","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550553981Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"downstairs","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"[2]R flush_numbers[0..12]: [310, 320, 312, 323, 315, 325, 322, 313, 322, 322, 302, 321]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.55057122Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"downstairs","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"[2]R generation[0..12]: [152, 157, 154, 158, 156, 159, 158, 155, 158, 158, 148, 158]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550596029Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pidWait for a query_work_queue command to finish before sending IO ":1822,"":"downstairs","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550618348Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"downstairs","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"Max found gen is 160","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550634437Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"downstairs","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"Generation requested: 160 >= found:160","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550650857Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"downstairs","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"Next flush: 326","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550666776Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"downstairs","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550682715Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"":"downstairs","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550701334Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550716444Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"0","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550733173Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"1","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550749832Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"client":"2","":"downstairs","session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550766791Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550783521Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"025142c8-df9c-4baa-8a68-f905b1ebcf2d is now active with session: 69abe55e-d797-4bc5-b891-ddb435d38350","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.55079986Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550814709Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822,"session_id":"69abe55e-d797-4bc5-b891-ddb435d38350"} {"msg":"The guest has finished waiting for activation with:160","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:20.550830528Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 337 len 8192 data: 15 12 02/30 Write block 558 len 28672 data: 8 10 9 11 14 17 18 03/30 Flush 04/30 Write block 19 len 40960 data: 10 13 14 12 13 16 15 16 16 15 05/30 Read block 35 len 4096 06/30 Flush 07/30 Flush 08/30 Write block 439 len 36864 data: 11 9 13 15 13 14 18 14 13 09/30 Write block 542 len 32768 data: 13 10 13 11 11 11 12 10 10/30 Read block 64 len 32768 11/30 Write block 172 len 24576 data: 13 12 10 11 9 7 12/30 Write block 291 len 28672 data: 12 12 9 9 9 8 6 13/30 Write block 448 len 28672 data: 16 15 16 14 16 12 10 14/30 Read block 198 len 40960 15/30 Write block 137 len 20480 data: 11 10 9 8 10 16/30 Read block 358 len 24576 17/30 Write block 486 len 32768 data: 11 13 12 11 11 10 10 7 18/30 Write block 117 len 4096 data: 11 19/30 Write block 350 len 32768 data: 10 9 11 13 14 13 12 17 20/30 Read block 553 len 32768 21/30 Write block 4 len 32768 data: 6 6 7 8 9 10 10 11 22/30 Read block 569 len 40960 23/30 Write block 34 len 4096 data: 8 24/30 Write block 381 len 32768 data: 15 15 17 20 16 15 18 17 25/30 Write block 291 len 4096 data: 13 26/30 Read block 592 len 8192 27/30 Read block 536 len 4096 28/30 Write block 24 len 32768 data: 17 16 17 17 16 13 10 8 29/30 Read block 240 len 28672 30/30 Write block 506 len 36864 data: 15 18 16 12 14 15 10 11 14 ---------------------------------------------------------------- Crucible gen:160 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 20 Acked 1019 Write 8 Done Done Done false 21 Acked 1020 Read 8 Done Done Done false 22 Acked 1021 Write 8 Done Done Done false 23 Acked 1022 Read 10 Done Done Done false 24 Acked 1023 Write 1 Done Done Done false 25 Acked 1024 Write 8 Done Done Done false 26 Acked 1025 Write 1 Done Done Done false 27 Acked 1026 Read 2 Done Done Done false 28 Acked 1027 Read 1 Done Done Done false 29 Acked 1028 Write 8 Done Done Done false 30 Acked 1029 Read 7 Done Done Done false 31 Acked 1030 Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 11 11 11 33 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1018 1018 1018 Downstairs last five completed: 1018 1017 1016 1015 1014 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:59:20.577859807Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1822} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 160 160 160 327 327 327 T T T 1 020-039 160 160 160 327 327 327 T T T 2 040-059 154 154 154 312 312 312 F F F 3 060-079 158 158 158 323 323 323 F F F 4 080-099 156 156 156 315 315 315 F F F 5 100-119 160 160 160 329 329 329 F F F 6 120-139 160 160 160 329 329 329 F F F 7 140-159 160 160 160 329 329 329 F F F 8 160-179 160 160 160 329 329 329 F F F 9 180-199 158 158 158 322 322 322 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 158 158 158 321 321 321 F F F 12 240-259 156 156 156 314 314 314 F F F 13 260-279 157 157 157 320 320 320 F F F 14 280-299 160 160 160 329 329 329 T T T 15 300-319 156 156 156 317 317 317 F F F 16 320-339 160 160 160 326 326 326 F F F 17 340-359 159 159 159 325 325 325 T T T 18 360-379 153 153 153 311 311 311 F F F 19 380-399 157 157 157 320 320 320 T T T 20 400-419 157 157 157 320 320 320 F F F 21 420-439 160 160 160 329 329 329 F F F 22 440-459 160 160 160 329 329 329 F F F 23 460-479 159 159 159 325 325 325 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 158 158 158 323 323 323 T T T 26 520-539 150 150 150 307 307 307 F F F 27 540-559 160 160 160 329 329 329 F F F 28 560-579 160 160 160 326 326 326 F F F 29 580-599 159 159 159 325 325 325 F F F Max gen: 160, Max flush: 329 A Difference in extent metadata is expected here On loop 80 Verifying data now No change after verify Loop: 80 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 161 161 161 330 330 330 F F F 1 020-039 161 161 161 330 330 330 F F F 2 040-059 154 154 154 312 312 312 F F F 3 060-079 158 158 158 323 323 323 F F F 4 080-099 156 156 156 315 315 315 F F F 5 100-119 160 160 160 329 329 329 F F F 6 120-139 160 160 160 329 329 329 F F F 7 140-159 160 160 160 329 329 329 F F F 8 160-179 160 160 160 329 329 329 F F F 9 180-199 158 158 158 322 322 322 F F F 10 200-219 148 148 148 302 302 302 F F F 11 220-239 158 158 158 321 321 321 F F F 12 240-259 156 156 156 314 314 314 F F F 13 260-279 157 157 157 320 320 320 F F F 14 280-299 161 161 161 330 330 330 F F F 15 300-319 156 156 156 317 317 317 F F F 16 320-339 160 160 160 326 326 326 F F F 17 340-359 161 161 161 330 330 330 F F F 18 360-379 153 153 153 311 311 311 F F F 19 380-399 161 161 161 330 330 330 F F F 20 400-419 157 157 157 320 320 320 F F F 21 420-439 160 160 160 329 329 329 F F F 22 440-459 160 160 160 329 329 329 F F F 23 460-479 159 159 159 325 325 325 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 161 161 161 330 330 330 F F F 26 520-539 150 150 150 307 307 307 F F F 27 540-559 160 160 160 329 329 329 F F F 28 560-579 160 160 160 326 326 326 F F F 29 580-599 159 159 159 325 325 325 F F F Max gen: 161, Max flush: 330 Begin loop 81 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.820565374Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.821000854Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.821031973Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830} {"msg":"Crucible ed157d13-5449-4d24-9a73-6b3ac682a085 has session id: d5f5159c-c0a2-4603-823d-903e7dc05226","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.821172466Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"Upstairs opts: Upstairs UUID: ed157d13-5449-4d24-9a73-6b3ac682a085, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.821212004Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"Crucible stats registered with UUID: ed157d13-5449-4d24-9a73-6b3ac682a085","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.821235003Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:162","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.821564548Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830} {"msg":"ed157d13-5449-4d24-9a73-6b3ac682a085 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.821695462Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.82172706Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"0","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.821757919Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"0","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.821782218Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.821812326Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.821836775Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"2","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.821851464Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"2","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.821872263Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"io task","client":"0","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.821896832Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"io task","client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.821922241Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"io task","client":"2","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:59:34.822019096Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"io task","client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:59:34.822057125Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"io task","client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:59:34.822087693Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"ds_connection connected from Ok(127.0.0.1:63847)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.822109362Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"io task","client":"0","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.822133451Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.82215561Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"ds_connection connected from Ok(127.0.0.1:61524)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.822284064Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"io task","client":"2","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.822310443Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"io task","client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.822855588Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"2","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.822876676Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"2","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.822890526Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"0","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.822903885Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"0","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.823544905Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"2","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.823576604Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"0","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.824069151Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"0","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.82409632Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"0","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.824112389Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.824187645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"2","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.824206825Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"2","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:34.824221044Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.823103438Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"io task","client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"ds_connection connected from Ok(127.0.0.1:48713)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.823315648Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"io task","client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.824717763Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.824745892Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.825257558Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.825792413Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.825812502Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"[0]R flush_numbers[0..12]: [330, 330, 312, 323, 315, 329, 329, 329, 329, 322, 302, 321]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.825826961Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"downstairs","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"[0]R generation[0..12]: [161, 161, 154, 158, 156, 160, 160, 160, 160, 158, 148, 158]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.825841121Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"downstairs","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.82585526Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"downstairs","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"[1]R flush_numbers[0..12]: [330, 330, 312, 323, 315, 329, 329, 329, 329, 322, 302, 321]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.825869419Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"downstairs","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"[1]R generation[0..12]: [161, 161, 154, 158, 156, 160, 160, 160, 160, 158, 148, 158]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.825883309Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"downstairs","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.825897178Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"downstairs","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"[2]R flush_numbers[0..12]: [330, 330, 312, 323, 315, 329, 329, 329, 329, 322, 302, 321]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.825910917Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"downstairs","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"[2]R generation[0..12]: [161, 161, 154, 158, 156, 160, 160, 160, 160, 158, 148, 158]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.825924867Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"downstairs","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.825938866Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"downstairs","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"Max found gen is 162","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.825951676Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"downstairs","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"Generation requested: 162 >= found:162","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.825964835Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"downstairs","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} Wait for a query_work_queue command to finish before sending IO {"msg":"Next flush: 331","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.825985334Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"downstairs","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.825999083Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"":"downstairs","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.826012233Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.826024252Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"0","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.826037532Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"1","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.826050561Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"client":"2","":"downstairs","session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.82606386Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.82607618Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"ed157d13-5449-4d24-9a73-6b3ac682a085 is now active with session: d5f5159c-c0a2-4603-823d-903e7dc05226","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.826089089Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.826100989Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830,"session_id":"d5f5159c-c0a2-4603-823d-903e7dc05226"} {"msg":"The guest has finished waiting for activation with:162","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:44.826113198Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 450 len 16384 data: 17 15 17 13 02/30 Write block 477 len 4096 data: 14 03/30 Write block 587 len 32768 data: 8 10 9 13 11 11 12 11 04/30 Read block 87 len 24576 05/30 Write block 360 len 20480 data: 12 10 11 10 12 06/30 Write block 197 len 24576 data: 16 13 10 11 12 11 07/30 Read block 158 len 12288 08/30 Write block 571 len 16384 data: 17 16 19 18 09/30 Read block 435 len 24576 10/30 Write block 358 len 16384 data: 18 15 13 11 11/30 Write block 257 len 40960 data: 14 12 13 14 12 17 15 13 13 12 12/30 Read block 13 len 4096 13/30 Read block 273 len 32768 14/30 Read block 486 len 16384 15/30 Write block 366 len 24576 data: 9 12 14 15 15 17 16/30 Read block 554 len 36864 17/30 Write block 437 len 40960 data: 10 9 12 10 14 16 14 15 19 15 18/30 Flush 19/30 Write block 163 len 8192 data: 7 9 20/30 Read block 101 len 40960 21/30 Write block 365 len 36864 data: 10 10 13 15 16 16 18 16 15 22/30 Read block 94 len 12288 23/30 Read block 584 len 20480 24/30 Read block 37 len 32768 25/30 Read block 478 len 36864 26/30 Write block 249 len 20480 data: 14 13 11 12 11 27/30 Write block 528 len 24576 data: 10 10 12 15 14 16 28/30 Read block 574 len 8192 29/30 Read block 132 len 36864 30/30 Read block 411 len 24576 ---------------------------------------------------------------- Crucible gen:162 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 19 Acked 1018 Write 2 Done Done Done false 20 Acked 1019 Read 10 Done Done Done false 21 Acked 1020 Write 9 Done Done Done false 22 Acked 1021 Read 3 Done Done Done false 23 Acked 1022 Read 5 Done Done Done false 24 Acked 1023 Read 8 Done Done Done false 25 Acked 1024 Read 9 Done Done Done false 26 Acked 1025 Write 5 Done Done Done false 27 Acked 1026 Write 6 Done Done Done false 28 Acked 1027 Read 2 Done Done Done false 29 Acked 1028 Read 9 Done Done Done false 30 Acked 1029 Read 6 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 12 12 12 36 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1017 1017 1017 Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:59:44.851935623Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1830} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 161 161 161 330 330 330 F F F 1 020-039 161 161 161 330 330 330 F F F 2 040-059 154 154 154 312 312 312 F F F 3 060-079 158 158 158 323 323 323 F F F 4 080-099 156 156 156 315 315 315 F F F 5 100-119 160 160 160 329 329 329 F F F 6 120-139 160 160 160 329 329 329 F F F 7 140-159 160 160 160 329 329 329 F F F 8 160-179 160 160 160 329 329 329 T T T 9 180-199 162 162 162 331 331 331 F F F 10 200-219 162 162 162 331 331 331 F F F 11 220-239 158 158 158 321 321 321 F F F 12 240-259 162 162 162 331 331 331 T T T 13 260-279 162 162 162 331 331 331 F F F 14 280-299 161 161 161 330 330 330 F F F 15 300-319 156 156 156 317 317 317 F F F 16 320-339 160 160 160 326 326 326 F F F 17 340-359 162 162 162 331 331 331 F F F 18 360-379 162 162 162 331 331 331 T T T 19 380-399 161 161 161 330 330 330 F F F 20 400-419 157 157 157 320 320 320 F F F 21 420-439 162 162 162 331 331 331 F F F 22 440-459 162 162 162 331 331 331 F F F 23 460-479 162 162 162 331 331 331 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 161 161 161 330 330 330 F F F 26 520-539 150 150 150 307 307 307 T T T 27 540-559 160 160 160 329 329 329 F F F 28 560-579 162 162 162 331 331 331 F F F 29 580-599 162 162 162 331 331 331 F F F Max gen: 162, Max flush: 331 A Difference in extent metadata is expected here On loop 81 Verifying data now No change after verify Loop: 81 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 161 161 161 330 330 330 F F F 1 020-039 161 161 161 330 330 330 F F F 2 040-059 154 154 154 312 312 312 F F F 3 060-079 158 158 158 323 323 323 F F F 4 080-099 156 156 156 315 315 315 F F F 5 100-119 160 160 160 329 329 329 F F F 6 120-139 160 160 160 329 329 329 F F F 7 140-159 160 160 160 329 329 329 F F F 8 160-179 163 163 163 332 332 332 F F F 9 180-199 162 162 162 331 331 331 F F F 10 200-219 162 162 162 331 331 331 F F F 11 220-239 158 158 158 321 321 321 F F F 12 240-259 163 163 163 332 332 332 F F F 13 260-279 162 162 162 331 331 331 F F F 14 280-299 161 161 161 330 330 330 F F F 15 300-319 156 156 156 317 317 317 F F F 16 320-339 160 160 160 326 326 326 F F F 17 340-359 162 162 162 331 331 331 F F F 18 360-379 163 163 163 332 332 332 F F F 19 380-399 161 161 161 330 330 330 F F F 20 400-419 157 157 157 320 320 320 F F F 21 420-439 162 162 162 331 331 331 F F F 22 440-459 162 162 162 331 331 331 F F F 23 460-479 162 162 162 331 331 331 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 161 161 161 330 330 330 F F F 26 520-539 163 163 163 332 332 332 F F F 27 540-559 160 160 160 329 329 329 F F F 28 560-579 162 162 162 331 331 331 F F F 29 580-599 162 162 162 331 331 331 F F F Max gen: 163, Max flush: 332 Begin loop 82 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.070153111Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.070594051Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.0706248Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838} {"msg":"Crucible d858e91a-9452-4621-afc6-87c9488dd8e2 has session id: be9590fb-ba69-4b86-bbc6-87c6133bef52","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.070923475Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"Upstairs opts: Upstairs UUID: d858e91a-9452-4621-afc6-87c9488dd8e2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.070970363Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"Crucible stats registered with UUID: d858e91a-9452-4621-afc6-87c9488dd8e2","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.070995082Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:164","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.071393284Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838} {"msg":"d858e91a-9452-4621-afc6-87c9488dd8e2 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.07146508Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.071483609Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"0","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.071511678Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"0","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.071537097Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"1","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.071565125Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"1","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.071590074Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.071615313Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.071639592Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"io task","client":"0","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.071665361Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"io task","client":"1","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.07168902Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"io task","client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"ds_connection connected from Ok(127.0.0.1:50270)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.071726778Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"io task","client":"1","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"ds_connection connected from Ok(127.0.0.1:62177)","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.071755367Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"io task","client":"0","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:59:59.071984776Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"io task","client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:59:59.072041333Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"io task","client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T18:59:59.072149308Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.072167028Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.072183737Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.072286822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"io task","client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.072773039Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"0","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.072806538Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"0","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.072833486Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"1","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.072859895Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"1","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.073307964Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"1","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.073419279Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"0","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.073780092Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"1","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.07381762Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"1","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.07383827Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.073851389Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"0","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.073865218Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"0","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T18:59:59.073878418Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.073738015Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"io task","client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"ds_connection connected from Ok(127.0.0.1:59986)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.073883268Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"io task","client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.074790776Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.074823764Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.075280173Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.075809329Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.075832517Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"[0]R flush_numbers[0..12]: [330, 330, 312, 323, 315, 329, 329, 329, 332, 331, 331, 321]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.075848037Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"downstairs","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"[0]R generation[0..12]: [161, 161, 154, 158, 156, 160, 160, 160, 163, 162, 162, 158]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.075862336Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"downstairs","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.075876855Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"downstairs","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"[1]R flush_numbers[0..12]: [330, 330, 312, 323, 315, 329, 329, 329, 332, 331, 331, 321]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.075891275Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"downstairs","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"[1]R generation[0..12]: [161, 161, 154, 158, 156, 160, 160, 160, 163, 162, 162, 158]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.075905374Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"downstairs","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.075919224Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"downstairs","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"[2]R flush_numbers[0..12]: [330, 330, 312, 323, 315, 329, 329, 329, 332, 331, 331, 321]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.075933113Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"downstairs","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"[2]R generation[0..12]: [161, 161, 154, 158, 156, 160, 160, 160, 163, 162, 162, 158]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.075946902Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"downstairs","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.075961031Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"downstairs","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"Max found gen is 164","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.075977571Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"downstairs","":"downstairs","Wait for a query_work_queue command to finish before sending IO session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"Generation requested: 164 >= found:164","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.07599569Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"downstairs","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"Next flush: 333","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.076009289Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"downstairs","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.076022459Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"":"downstairs","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.076035988Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.076048038Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"0","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.076061527Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"1","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.076075196Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"client":"2","":"downstairs","session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.076088576Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.076100965Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"d858e91a-9452-4621-afc6-87c9488dd8e2 is now active with session: be9590fb-ba69-4b86-bbc6-87c6133bef52","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.076114014Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.076125934Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838,"session_id":"be9590fb-ba69-4b86-bbc6-87c6133bef52"} {"msg":"The guest has finished waiting for activation with:164","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:09.076138083Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 269 len 16384 data: 12 10 8 9 02/30 Read block 514 len 24576 03/30 Write block 171 len 20480 data: 15 14 13 11 12 04/30 Read block 76 len 20480 05/30 Read block 318 len 8192 06/30 Write block 413 len 28672 data: 10 11 11 14 16 18 21 07/30 Write block 325 len 36864 data: 9 8 11 13 12 12 12 12 13 08/30 Flush 09/30 Flush 10/30 Write block 422 len 20480 data: 16 12 8 12 13 11/30 Read block 446 len 28672 12/30 Read block 407 len 20480 13/30 Read block 528 len 32768 14/30 Read block 416 len 36864 15/30 Read block 521 len 12288 16/30 Read block 395 len 32768 17/30 Flush 18/30 Write block 432 len 20480 data: 17 14 11 14 12 19/30 Write block 419 len 28672 data: 22 21 19 17 13 9 13 20/30 Read block 314 len 20480 21/30 Read block 377 len 36864 22/30 Write block 512 len 40960 data: 11 12 15 12 10 8 11 13 14 12 23/30 Write block 6 len 4096 data: 8 24/30 Read block 50 len 4096 25/30 Read block 303 len 40960 26/30 Write block 348 len 20480 data: 5 7 11 10 12 27/30 Read block 97 len 16384 28/30 Read block 15 len 8192 29/30 Write block 392 len 4096 data: 20 30/30 Write block 280 len 32768 data: 17 13 11 10 11 12 9 10 ---------------------------------------------------------------- Crucible gen:164 GIO:true work queues: Upstairs:0 downstairs:13 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 18 Acked 1017 Write 5 Done Done Done false 19 Acked 1018 Write 7 Done Done Done false 20 Acked 1019 Read 5 Done Done Done false 21 Acked 1020 Read 9 Done Done Done false 22 Acked 1021 Write 10 Done Done Done false 23 Acked 1022 Write 1 Done Done Done false 24 Acked 1023 Read 1 Done Done Done false 25 Acked 1024 Read 10 Done Done Done false 26 Acked 1025 Write 5 Done Done Done false 27 Acked 1026 Read 4 Done Done Done false 28 Acked 1027 Read 2 Done Done Done false 29 Acked 1028 Write 1 Done Done Done false 30 Acked 1029 Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 12 12 12 36 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1016 1016 1016 Downstairs last five completed: 1016 1015 1014 1013 1012 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:00:09.101626784Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1838} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 161 161 161 330 330 330 T T T 1 020-039 161 161 161 330 330 330 F F F 2 040-059 154 154 154 312 312 312 F F F 3 060-079 158 158 158 323 323 323 F F F 4 080-099 156 156 156 315 315 315 F F F 5 100-119 160 160 160 329 329 329 F F F 6 120-139 160 160 160 329 329 329 F F F 7 140-159 160 160 160 329 329 329 F F F 8 160-179 164 164 164 333 333 333 F F F 9 180-199 162 162 162 331 331 331 F F F 10 200-219 162 162 162 331 331 331 F F F 11 220-239 158 158 158 321 321 321 F F F 12 240-259 163 163 163 332 332 332 F F F 13 260-279 164 164 164 333 333 333 F F F 14 280-299 161 161 161 330 330 330 T T T 15 300-319 156 156 156 317 317 317 F F F 16 320-339 164 164 164 333 333 333 F F F 17 340-359 162 162 162 331 331 331 T T T 18 360-379 163 163 163 332 332 332 F F F 19 380-399 161 161 161 330 330 330 T T T 20 400-419 164 164 164 333 333 333 T T T 21 420-439 164 164 164 335 335 335 T T T 22 440-459 162 162 162 331 331 331 F F F 23 460-479 162 162 162 331 331 331 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 161 161 161 330 330 330 T T T 26 520-539 163 163 163 332 332 332 T T T 27 540-559 160 160 160 329 329 329 F F F 28 560-579 162 162 162 331 331 331 F F F 29 580-599 162 162 162 331 331 331 F F F Max gen: 164, Max flush: 335 A Difference in extent metadata is expected here On loop 82 Verifying data now No change after verify Loop: 82 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 165 165 165 336 336 336 F F F 1 020-039 161 161 161 330 330 330 F F F 2 040-059 154 154 154 312 312 312 F F F 3 060-079 158 158 158 323 323 323 F F F 4 080-099 156 156 156 315 315 315 F F F 5 100-119 160 160 160 329 329 329 F F F 6 120-139 160 160 160 329 329 329 F F F 7 140-159 160 160 160 329 329 329 F F F 8 160-179 164 164 164 333 333 333 F F F 9 180-199 162 162 162 331 331 331 F F F 10 200-219 162 162 162 331 331 331 F F F 11 220-239 158 158 158 321 321 321 F F F 12 240-259 163 163 163 332 332 332 F F F 13 260-279 164 164 164 333 333 333 F F F 14 280-299 165 165 165 336 336 336 F F F 15 300-319 156 156 156 317 317 317 F F F 16 320-339 164 164 164 333 333 333 F F F 17 340-359 165 165 165 336 336 336 F F F 18 360-379 163 163 163 332 332 332 F F F 19 380-399 165 165 165 336 336 336 F F F 20 400-419 165 165 165 336 336 336 F F F 21 420-439 165 165 165 336 336 336 F F F 22 440-459 162 162 162 331 331 331 F F F 23 460-479 162 162 162 331 331 331 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 165 165 165 336 336 336 F F F 26 520-539 165 165 165 336 336 336 F F F 27 540-559 160 160 160 329 329 329 F F F 28 560-579 162 162 162 331 331 331 F F F 29 580-599 162 162 162 331 331 331 F F F Max gen: 165, Max flush: 336 Begin loop 83 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.365460213Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.365897472Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.365929251Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846} {"msg":"Crucible e22c839c-4675-408b-844e-e05d7b92b177 has session id: e503b03f-89a5-4d64-9655-4cc046b12815","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.36615552Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"Upstairs opts: Upstairs UUID: e22c839c-4675-408b-844e-e05d7b92b177, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.366203068Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"Crucible stats registered with UUID: e22c839c-4675-408b-844e-e05d7b92b177","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.366227027Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:166","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.366522863Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846} {"msg":"e22c839c-4675-408b-844e-e05d7b92b177 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.366556972Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.36658217Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"0","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.366611709Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"0","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.366633998Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"1","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.366658087Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"1","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.366681936Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.366707715Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.366733233Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"io task","client":"0","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.366756742Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"io task","client":"1","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.366779041Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"io task","client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"ds_connection connected from Ok(127.0.0.1:61309)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.366953763Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"io task","client":"1","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"ds_connection connected from Ok(127.0.0.1:60066)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.367092027Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"io task","client":"0","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:00:23.367151204Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"io task","client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:00:23.367186802Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"io task","client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:00:23.367215731Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.36724168Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.367266789Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.367372364Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"io task","client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.367653971Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"1","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.367688039Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"1","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.367714008Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"0","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.367739187Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"0","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.368207485Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"0","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.368319519Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"1","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.368668623Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"0","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.368698942Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"0","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.368717571Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.368841415Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"1","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.368865384Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"1","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:23.368882693Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.368231865Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"io task","client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"ds_connection connected from Ok(127.0.0.1:34719)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.368442406Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"io task","client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.369872679Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.369906307Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.370410244Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.370935279Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.370960538Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"[0]R flush_numbers[0..12]: [336, 330, 312, 323, 315, 329, 329, 329, 333, 331, 331, 321]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.370980277Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"downstairs","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"[0]R generation[0..12]: [165, 161, 154, 158, 156, 160, 160, 160, 164, 162, 162, 158]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371012946Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"downstairs","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371032585Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"downstairs","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"[1]R flush_numbers[0..12]: [336, 330, 312, 323, 315, 329, 329, 329, 333, 331, 331, 321]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371051884Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"downstairs","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"[1]R generation[0..12]: [165, 161, 154, 158, 156, 160, 160, 160, 164, 162, 162, 158]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371069833Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"downstairs","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371087582Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"downstairs","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"[2]R flush_numbers[0..12]: [336, 330, 312, 323, 315, 329, 329, 329, 333, 331, 331, 321]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371104921Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"downstairs","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"Wait for a query_work_queue command to finish before sending IO } {"msg":"[2]R generation[0..12]: [165, 161, 154, 158, 156, 160, 160, 160, 164, 162, 162, 158]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.37112925Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"downstairs","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371147049Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"downstairs","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"Max found gen is 166","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371163199Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"downstairs","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"Generation requested: 166 >= found:166","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371179678Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"downstairs","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"Next flush: 337","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371195827Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"downstairs","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371211916Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"":"downstairs","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371228506Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371243615Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"0","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371260964Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"1","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371277933Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"client":"2","":"downstairs","session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371296422Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371322131Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"e22c839c-4675-408b-844e-e05d7b92b177 is now active with session: e503b03f-89a5-4d64-9655-4cc046b12815","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.3713433Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.37135688Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846,"session_id":"e503b03f-89a5-4d64-9655-4cc046b12815"} {"msg":"The guest has finished waiting for activation with:166","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:33.371369459Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 444 len 12288 data: 16 20 16 02/30 Write block 363 len 16384 data: 11 13 11 11 03/30 Read block 486 len 28672 04/30 Write block 371 len 20480 data: 19 17 16 12 15 05/30 Read block 93 len 4096 06/30 Flush 07/30 Write block 548 len 32768 data: 13 11 13 16 13 15 16 14 08/30 Read block 447 len 4096 09/30 Read block 134 len 32768 10/30 Write block 519 len 36864 data: 14 15 13 12 11 14 11 11 11 11/30 Read block 96 len 24576 12/30 Read block 32 len 28672 13/30 Read block 592 len 32768 14/30 Read block 283 len 20480 15/30 Read block 6 len 20480 16/30 Write block 420 len 36864 data: 22 20 18 14 10 14 14 12 13 17/30 Read block 125 len 20480 18/30 Write block 248 len 4096 data: 13 19/30 Read block 382 len 40960 20/30 Flush 21/30 Write block 545 len 4096 data: 12 22/30 Write block 459 len 4096 data: 11 23/30 Write block 379 len 8192 data: 18 15 24/30 Read block 417 len 20480 25/30 Read block 562 len 32768 26/30 Read block 428 len 32768 27/30 Read block 298 len 20480 28/30 Write block 100 len 8192 data: 16 16 29/30 Write block 350 len 12288 data: 12 11 13 30/30 Read block 559 len 28672 ---------------------------------------------------------------- Crucible gen:166 GIO:true work queues: Upstairs:0 downstairs:10 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 22 Acked 1021 Write 1 Done Done Done false 23 Acked 1022 Write 1 Done Done Done false 24 Acked 1023 Write 2 Done Done Done false 25 Acked 1024 Read 5 Done Done Done false 26 Acked 1025 Read 8 Done Done Done false 27 Acked 1026 Read 8 Done Done Done false 28 Acked 1027 Read 5 Done Done Done false 29 Acked 1028 Write 2 Done Done Done false 30 Acked 1029 Write 3 Done Done Done false 31 Acked 1030 Read 7 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 10 10 10 30 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1020 1020 1020 Downstairs last five completed: 1020 1019 1018 1017 1016 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:00:33.396450419Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1846} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 165 165 165 336 336 336 F F F 1 020-039 161 161 161 330 330 330 F F F 2 040-059 154 154 154 312 312 312 F F F 3 060-079 158 158 158 323 323 323 F F F 4 080-099 156 156 156 315 315 315 F F F 5 100-119 160 160 160 329 329 329 T T T 6 120-139 160 160 160 329 329 329 F F F 7 140-159 160 160 160 329 329 329 F F F 8 160-179 164 164 164 333 333 333 F F F 9 180-199 162 162 162 331 331 331 F F F 10 200-219 162 162 162 331 331 331 F F F 11 220-239 158 158 158 321 321 321 F F F 12 240-259 166 166 166 338 338 338 F F F 13 260-279 164 164 164 333 333 333 F F F 14 280-299 165 165 165 336 336 336 F F F 15 300-319 156 156 156 317 317 317 F F F 16 320-339 164 164 164 333 333 333 F F F 17 340-359 165 165 165 336 336 336 T T T 18 360-379 166 166 166 337 337 337 T T T 19 380-399 165 165 165 336 336 336 T T T 20 400-419 165 165 165 336 336 336 F F F 21 420-439 166 166 166 338 338 338 F F F 22 440-459 166 166 166 337 337 337 T T T 23 460-479 162 162 162 331 331 331 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 166 166 166 338 338 338 F F F 26 520-539 166 166 166 338 338 338 F F F 27 540-559 166 166 166 338 338 338 T T T 28 560-579 162 162 162 331 331 331 F F F 29 580-599 162 162 162 331 331 331 F F F Max gen: 166, Max flush: 338 A Difference in extent metadata is expected here On loop 83 Verifying data now No change after verify Loop: 83 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 165 165 165 336 336 336 F F F 1 020-039 161 161 161 330 330 330 F F F 2 040-059 154 154 154 312 312 312 F F F 3 060-079 158 158 158 323 323 323 F F F 4 080-099 156 156 156 315 315 315 F F F 5 100-119 167 167 167 339 339 339 F F F 6 120-139 160 160 160 329 329 329 F F F 7 140-159 160 160 160 329 329 329 F F F 8 160-179 164 164 164 333 333 333 F F F 9 180-199 162 162 162 331 331 331 F F F 10 200-219 162 162 162 331 331 331 F F F 11 220-239 158 158 158 321 321 321 F F F 12 240-259 166 166 166 338 338 338 F F F 13 260-279 164 164 164 333 333 333 F F F 14 280-299 165 165 165 336 336 336 F F F 15 300-319 156 156 156 317 317 317 F F F 16 320-339 164 164 164 333 333 333 F F F 17 340-359 167 167 167 339 339 339 F F F 18 360-379 167 167 167 339 339 339 F F F 19 380-399 167 167 167 339 339 339 F F F 20 400-419 165 165 165 336 336 336 F F F 21 420-439 166 166 166 338 338 338 F F F 22 440-459 167 167 167 339 339 339 F F F 23 460-479 162 162 162 331 331 331 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 166 166 166 338 338 338 F F F 26 520-539 166 166 166 338 338 338 F F F 27 540-559 167 167 167 339 339 339 F F F 28 560-579 162 162 162 331 331 331 F F F 29 580-599 162 162 162 331 331 331 F F F Max gen: 167, Max flush: 339 Begin loop 84 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.640179409Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.640566901Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.64059479Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854} {"msg":"Crucible 171c3651-098f-4014-acb3-6613bf8ed126 has session id: 82364294-b5e2-4b89-8e60-f3558408bb40","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.640716354Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"Upstairs opts: Upstairs UUID: 171c3651-098f-4014-acb3-6613bf8ed126, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.640755472Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"Crucible stats registered with UUID: 171c3651-098f-4014-acb3-6613bf8ed126","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.640779061Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:168","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.641112415Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854} {"msg":"171c3651-098f-4014-acb3-6613bf8ed126 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.641135404Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.641148834Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"0","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.641168233Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"0","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.641181882Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.641195242Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.641208201Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"2","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.64122151Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"2","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.64123866Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"io task","client":"0","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.641252789Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"io task","client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.641279068Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"io task","client":"2","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"ds_connection connected from Ok(127.0.0.1:58874)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.641515277Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"io task","client":"2","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"ds_connection connected from Ok(127.0.0.1:50962)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.641636641Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"io task","client":"0","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:00:47.641678649Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"io task","client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:00:47.641711547Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"io task","client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:00:47.641862171Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.641886169Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.641910668Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.642019143Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"io task","client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.642469822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"2","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.642496261Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"2","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.64251079Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"0","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.642524309Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"0","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.643168339Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"2","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.643199728Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"0","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.643735503Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"2","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.643763052Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"2","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.643777771Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.643790901Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"0","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.64380451Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"0","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:47.643820819Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.64256784Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"io task","client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"ds_connection connected from Ok(127.0.0.1:44481)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.642727673Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"io task","client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.644123928Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.644164056Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.644768338Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.64535205Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645378489Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"[0]R flush_numbers[0..12]: [336, 330, 312, 323, 315, 339, 329, 329, 333, 331, 331, 321]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645394979Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"downstairs","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"[0]R generation[0..12]: [165, 161, 154, 158, 156, 167, 160, 160, 164, 162, 162, 158]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645409568Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"downstairs","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645423977Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"downstairs","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"[1]R flush_numbers[0..12]: [336, 330, 312, 323, 315, 339, 329, 329, 333, 331, 331, 321]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645439466Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"downstairs","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"[1]R generation[0..12]: [165, 161, 154, 158, 156, 167, 160, 160, 164, 162, 162, 158]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645453806Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"downstairs","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645468125Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"downstairs","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"[2]R flush_numbers[0..12]: [336, 330, 312, 323, 315, 339, 329, 329, 333, 331, 331, 321]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645482214Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"downstairs","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"[2]R generation[0..12]: [165, 161, 154, 158, 156, 167, 160, 160, 164, 162, 162, 158]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645501114Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"downstairs","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645515333Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"downstairs","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"Max found gen is 168","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T19:00:57.645528122Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"downstairs","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"Generation requested: 168 >= found:168","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645543632Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"downstairs","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"Next flush: 340","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645557001Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"downstairs","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.64557014Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"":"downstairs","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.64558339Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645595229Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"0","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645608478Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"1","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645621708Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"client":"2","":"downstairs","session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645635017Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645647307Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"171c3651-098f-4014-acb3-6613bf8ed126 is now active with session: 82364294-b5e2-4b89-8e60-f3558408bb40","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645660146Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645672106Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854,"session_id":"82364294-b5e2-4b89-8e60-f3558408bb40"} {"msg":"The guest has finished waiting for activation with:168","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:00:57.645684195Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 247 len 40960 data: 12 14 15 14 12 13 12 14 15 13 03/30 Read block 229 len 36864 04/30 Write block 69 len 28672 data: 6 6 9 10 13 13 16 05/30 Read block 264 len 8192 06/30 Read block 252 len 40960 07/30 Read block 284 len 36864 08/30 Write block 470 len 32768 data: 11 14 13 13 14 14 14 15 09/30 Read block 470 len 36864 10/30 Read block 57 len 4096 11/30 Write block 336 len 36864 data: 14 16 13 9 9 11 12 11 8 12/30 Write block 81 len 20480 data: 21 18 18 20 20 13/30 Read block 424 len 4096 14/30 Read block 217 len 24576 15/30 Read block 369 len 8192 16/30 Flush 17/30 Flush 18/30 Write block 239 len 16384 data: 14 14 13 12 19/30 Read block 108 len 36864 20/30 Write block 450 len 32768 data: 18 16 18 14 11 12 10 9 21/30 Read block 273 len 32768 22/30 Write block 70 len 36864 data: 7 10 11 14 14 17 18 18 18 23/30 Read block 67 len 24576 24/30 Read block 426 len 4096 25/30 Read block 114 len 40960 26/30 Read block 287 len 32768 27/30 Read block 273 len 12288 28/30 Write block 234 len 20480 data: 14 16 17 14 16 29/30 Write block 330 len 24576 data: 13 13 13 14 16 15 30/30 Read block 483 len 12288 ---------------------------------------------------------------- Crucible gen:168 GIO:true work queues: Upstairs:0 downstairs:13 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 18 Acked 1017 Write 4 Done Done Done false 19 Acked 1018 Read 9 Done Done Done false 20 Acked 1019 Write 8 Done Done Done false 21 Acked 1020 Read 8 Done Done Done false 22 Acked 1021 Write 9 Done Done Done false 23 Acked 1022 Read 6 Done Done Done false 24 Acked 1023 Read 1 Done Done Done false 25 Acked 1024 Read 10 Done Done Done false 26 Acked 1025 Read 8 Done Done Done false 27 Acked 1026 Read 3 Done Done Done false 28 Acked 1027 Write 5 Done Done Done false 29 Acked 1028 Write 6 Done Done Done false 30 Acked 1029 Read 3 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 13 13 13 39 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1016 1016 1016 Downstairs last five completed: 1016 1015 1014 1013 1012 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:00:57.672196698Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1854} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 165 165 165 336 336 336 F F F 1 020-039 161 161 161 330 330 330 F F F 2 040-059 154 154 154 312 312 312 F F F 3 060-079 168 168 168 341 341 341 T T T 4 080-099 168 168 168 341 341 341 F F F 5 100-119 167 167 167 339 339 339 F F F 6 120-139 160 160 160 329 329 329 F F F 7 140-159 160 160 160 329 329 329 F F F 8 160-179 164 164 164 333 333 333 F F F 9 180-199 162 162 162 331 331 331 F F F 10 200-219 162 162 162 331 331 331 F F F 11 220-239 158 158 158 321 321 321 T T T 12 240-259 168 168 168 341 341 341 T T T 13 260-279 164 164 164 333 333 333 F F F 14 280-299 165 165 165 336 336 336 F F F 15 300-319 156 156 156 317 317 317 F F F 16 320-339 168 168 168 341 341 341 T T T 17 340-359 168 168 168 341 341 341 F F F 18 360-379 167 167 167 339 339 339 F F F 19 380-399 167 167 167 339 339 339 F F F 20 400-419 165 165 165 336 336 336 F F F 21 420-439 166 166 166 338 338 338 F F F 22 440-459 167 167 167 339 339 339 T T T 23 460-479 168 168 168 341 341 341 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 166 166 166 338 338 338 F F F 26 520-539 166 166 166 338 338 338 F F F 27 540-559 167 167 167 339 339 339 F F F 28 560-579 162 162 162 331 331 331 F F F 29 580-599 162 162 162 331 331 331 F F F Max gen: 168, Max flush: 341 A Difference in extent metadata is expected here On loop 84 Verifying data now No change after verify Loop: 84 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 165 165 165 336 336 336 F F F 1 020-039 161 161 161 330 330 330 F F F 2 040-059 154 154 154 312 312 312 F F F 3 060-079 169 169 169 342 342 342 F F F 4 080-099 168 168 168 341 341 341 F F F 5 100-119 167 167 167 339 339 339 F F F 6 120-139 160 160 160 329 329 329 F F F 7 140-159 160 160 160 329 329 329 F F F 8 160-179 164 164 164 333 333 333 F F F 9 180-199 162 162 162 331 331 331 F F F 10 200-219 162 162 162 331 331 331 F F F 11 220-239 169 169 169 342 342 342 F F F 12 240-259 169 169 169 342 342 342 F F F 13 260-279 164 164 164 333 333 333 F F F 14 280-299 165 165 165 336 336 336 F F F 15 300-319 156 156 156 317 317 317 F F F 16 320-339 169 169 169 342 342 342 F F F 17 340-359 168 168 168 341 341 341 F F F 18 360-379 167 167 167 339 339 339 F F F 19 380-399 167 167 167 339 339 339 F F F 20 400-419 165 165 165 336 336 336 F F F 21 420-439 166 166 166 338 338 338 F F F 22 440-459 169 169 169 342 342 342 F F F 23 460-479 168 168 168 341 341 341 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 166 166 166 338 338 338 F F F 26 520-539 166 166 166 338 338 338 F F F 27 540-559 167 167 167 339 339 339 F F F 28 560-579 162 162 162 331 331 331 F F F 29 580-599 162 162 162 331 331 331 F F F Max gen: 169, Max flush: 342 Begin loop 85 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.901531389Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.901949179Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.901976008Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862} {"msg":"Crucible a42fc75e-b5b1-4c59-af43-f3e8a14b1f6b has session id: 6a2fbfe2-25da-4179-bee8-ebc5489d3b43","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.90215042Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"Upstairs opts: Upstairs UUID: a42fc75e-b5b1-4c59-af43-f3e8a14b1f6b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.902197308Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"Crucible stats registered with UUID: a42fc75e-b5b1-4c59-af43-f3e8a14b1f6b","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.902221066Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:170","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.902524422Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862} {"msg":"a42fc75e-b5b1-4c59-af43-f3e8a14b1f6b active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.902618988Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.902636767Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.902657226Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.902681155Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"1","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.902707064Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"1","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.902732433Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"2","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.902757572Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"2","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.902779191Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"io task","client":"2","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.902804459Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"io task","client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.902830388Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"io task","client":"1","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:01:11.902902695Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"io task","client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:01:11.902929213Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"io task","client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:01:11.902960892Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.902983661Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.90300551Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"ds_connection connected from Ok(127.0.0.1:53130)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.903133074Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"io task","client":"1","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.903165982Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"io task","client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"ds_connection connected from Ok(127.0.0.1:35420)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.903260768Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"io task","client":"2","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.903802143Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"2","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.903829622Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"2","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.903846041Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"1","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.90385979Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"1","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.904477991Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"1","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.90450433Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"2","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.905040205Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"2","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.905087933Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"2","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.905123991Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.90515399Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"1","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.905172159Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"1","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:11.905188818Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.903784066Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"io task","client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"ds_connection connected from Ok(127.0.0.1:34469)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.903845423Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"io task","client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.905050107Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.905100095Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.905564483Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906111428Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906139846Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"[0]R flush_numbers[0..12]: [336, 330, 312, 342, 341, 339, 329, 329, 333, 331, 331, 342]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906159066Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"downstairs","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"[0]R generation[0..12]: [165, 161, 154, 169, 168, 167, 160, 160, 164, 162, 162, 169]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906176965Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"downstairs","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906194574Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"downstairs","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"[1]R flush_numbers[0..12]: [336, 330, 312, 342, 341, 339, 329, 329, 333, 331, 331, 342]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906211963Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"downstairs","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"[1]R generation[0..12]: [165, 161, 154, 169, 168, 167, 160, 160, 164, 162, 162, 169]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906229292Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"downstairs","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906253231Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"downstairs","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"[2]R flush_numbers[0..12]: [336, 330, 312, 342, 341, 339, 329, 329, 333, 331, 331, 342]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.90627098Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"downstairs","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"[2]R generation[0..12]: [165, 161, 154, 169, 168, 167, 160, 160, 164, 162, 162, 169]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906288469Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"downstairs","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {Wait for a query_work_queue command to finish before sending IO "msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906309718Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"downstairs","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"Max found gen is 170","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906327058Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"downstairs","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"Generation requested: 170 >= found:170","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906343787Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"downstairs","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"Next flush: 343","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906360096Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"downstairs","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906376195Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"":"downstairs","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906392515Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906407454Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"0","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906423753Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"1","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906440102Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"client":"2","":"downstairs","session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906456532Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906471671Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"a42fc75e-b5b1-4c59-af43-f3e8a14b1f6b is now active with session: 6a2fbfe2-25da-4179-bee8-ebc5489d3b43","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.90648773Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:21.906504899Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862,"session_id":"6a2fbfe2-25da-4179-bee8-ebc5489d3b43"} {"msg":"The guest has finished waiting for activation with:170","v":0,"name":"crucible","level":30Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2024-07-31T19:01:21.906520189Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 8 len 8192 data: 10 11 02/30 Write block 59 len 28672 data: 7 6 4 3 5 6 8 03/30 Read block 580 len 4096 04/30 Read block 278 len 40960 05/30 Flush 06/30 Write block 136 len 36864 data: 13 12 11 10 9 11 13 15 15 07/30 Flush 08/30 Write block 589 len 40960 data: 10 14 12 12 13 12 11 9 8 8 09/30 Read block 14 len 20480 10/30 Read block 520 len 24576 11/30 Write block 300 len 24576 data: 7 10 11 11 10 9 12/30 Write block 191 len 28672 data: 17 12 15 13 14 15 17 13/30 Write block 72 len 8192 data: 12 15 14/30 Read block 226 len 24576 15/30 Write block 80 len 8192 data: 20 22 16/30 Read block 28 len 4096 17/30 Read block 295 len 16384 18/30 Flush 19/30 Write block 143 len 32768 data: 16 16 15 14 13 13 12 14 20/30 Write block 317 len 12288 data: 13 14 12 21/30 Write block 51 len 36864 data: 10 12 11 10 9 11 9 8 8 22/30 Read block 268 len 20480 23/30 Read block 254 len 8192 24/30 Write block 224 len 36864 data: 11 12 10 12 12 13 13 12 15 25/30 Read block 290 len 40960 26/30 Read block 399 len 36864 27/30 Read block 0 len 28672 28/30 Read block 173 len 16384 29/30 Write block 63 len 4096 data: 6 30/30 Write block 289 len 4096 data: 11 ---------------------------------------------------------------- Crucible gen:170 GIO:true work queues: Upstairs:0 downstairs:12 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 19 Acked 1018 Write 8 Done Done Done false 20 Acked 1019 Write 3 Done Done Done false 21 Acked 1020 Write 9 Done Done Done false 22 Acked 1021 Read 5 Done Done Done false 23 Acked 1022 Read 2 Done Done Done false 24 Acked 1023 Write 9 Done Done Done false 25 Acked 1024 Read 10 Done Done Done false 26 Acked 1025 Read 9 Done Done Done false 27 Acked 1026 Read 7 Done Done Done false 28 Acked 1027 Read 4 Done Done Done false 29 Acked 1028 Write 1 Sent Sent Sent false 30 Acked 1029 Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 2 2 2 6 Done 10 10 10 30 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1017 1017 1017 Downstairs last five completed: 1017 1016 1015 1014 1013 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:01:21.934089092Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1862} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 170 170 343 343 343 F F F 1 020-039 161 161 161 330 330 330 F F F 2 040-059 170 170 170 343 343 343 T T T 3 060-079 170 170 170 345 345 345 T T T 4 080-099 170 170 170 345 345 345 F F F 5 100-119 167 167 167 339 339 339 F F F 6 120-139 170 170 170 344 344 344 F F F 7 140-159 170 170 170 344 344 344 T T T 8 160-179 164 164 164 333 333 333 F F F 9 180-199 170 170 170 345 345 345 F F F 10 200-219 162 162 162 331 331 331 F F F 11 220-239 169 169 169 342 342 342 T T T 12 240-259 169 169 169 342 342 342 F F F 13 260-279 164 164 164 333 333 333 F F F 14 280-299 165 165 165 336 336 336 T T T 15 300-319 170 170 170 345 345 345 T T T 16 320-339 169 169 169 342 342 342 F F F 17 340-359 168 168 168 341 341 341 F F F 18 360-379 167 167 167 339 339 339 F F F 19 380-399 167 167 167 339 339 339 F F F 20 400-419 165 165 165 336 336 336 F F F 21 420-439 166 166 166 338 338 338 F F F 22 440-459 169 169 169 342 342 342 F F F 23 460-479 168 168 168 341 341 341 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 166 166 166 338 338 338 F F F 26 520-539 166 166 166 338 338 338 F F F 27 540-559 167 167 167 339 339 339 F F F 28 560-579 162 162 162 331 331 331 F F F 29 580-599 170 170 170 345 345 345 F F F Max gen: 170, Max flush: 345 A Difference in extent metadata is expected here On loop 85 Verifying data now No change after verify Loop: 85 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 170 170 343 343 343 F F F 1 020-039 161 161 161 330 330 330 F F F 2 040-059 171 171 171 346 346 346 F F F 3 060-079 171 171 171 346 346 346 F F F 4 080-099 170 170 170 345 345 345 F F F 5 100-119 167 167 167 339 339 339 F F F 6 120-139 170 170 170 344 344 344 F F F 7 140-159 171 171 171 346 346 346 F F F 8 160-179 164 164 164 333 333 333 F F F 9 180-199 170 170 170 345 345 345 F F F 10 200-219 162 162 162 331 331 331 F F F 11 220-239 171 171 171 346 346 346 F F F 12 240-259 169 169 169 342 342 342 F F F 13 260-279 164 164 164 333 333 333 F F F 14 280-299 171 171 171 346 346 346 F F F 15 300-319 171 171 171 346 346 346 F F F 16 320-339 169 169 169 342 342 342 F F F 17 340-359 168 168 168 341 341 341 F F F 18 360-379 167 167 167 339 339 339 F F F 19 380-399 167 167 167 339 339 339 F F F 20 400-419 165 165 165 336 336 336 F F F 21 420-439 166 166 166 338 338 338 F F F 22 440-459 169 169 169 342 342 342 F F F 23 460-479 168 168 168 341 341 341 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 166 166 166 338 338 338 F F F 26 520-539 166 166 166 338 338 338 F F F 27 540-559 167 167 167 339 339 339 F F F 28 560-579 162 162 162 331 331 331 F F F 29 580-599 170 170 170 345 345 345 F F F Max gen: 171, Max flush: 346 Begin loop 86 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.187194204Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.187580766Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.187603905Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870} {"msg":"Crucible 8caa4395-a8bd-4d1d-b57e-6dd820eb874e has session id: 71254593-55f6-4327-955a-50fccd4e1200","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.187798466Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"Upstairs opts: Upstairs UUID: 8caa4395-a8bd-4d1d-b57e-6dd820eb874e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.187840894Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"Crucible stats registered with UUID: 8caa4395-a8bd-4d1d-b57e-6dd820eb874e","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.187864013Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"session_id":"71254593-55f6-4327-955a-50fccd4e1200"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:172","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188198647Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870} {"msg":"8caa4395-a8bd-4d1d-b57e-6dd820eb874e active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188232586Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188256534Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"0","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188290213Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"0","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188316492Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188342101Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188367159Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"2","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188390388Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"2","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188414207Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"io task","client":"0","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188438706Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188461895Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"io task","client":"2","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"ds_connection connected from Ok(127.0.0.1:33614)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188553101Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"io task","client":"0","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:01:36.188592949Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"ds_connection connected from Ok(127.0.0.1:36816)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188749531Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"io task","client":"2","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:01:36.18877918Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:01:36.188809469Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188830158Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188854177Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.188913574Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.189345734Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"2","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.189370642Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"2","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.189505386Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"0","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.189524335Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"0","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.189898648Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"2","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.190043411Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"0","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.190432813Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"2","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.190463862Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"2","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.190481401Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.190555557Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"0","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.190577666Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"0","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:36.190594345Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.19034686Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"ds_connection connected from Ok(127.0.0.1:62827)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.190458185Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.191675918Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.191713816Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.192260261Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.192800246Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.192826794Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"[0]R flush_numbers[0..12]: [343, 330, 346, 346, 345, 339, 344, 346, 333, 345, 331, 346]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.192845753Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"downstairs","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"[0]R generation[0..12]: [170, 161, 171, 171, 170, 167, 170, 171, 164, 170, 162, 171]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.192863863Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"downstairs","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.192882032Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"downstairs","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"[1]R flush_numbers[0..12]: [343, 330, 346, 346, 345, 339, 344, 346, 333, 345, 331, 346]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.192899141Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"downstairs","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"[1]R generation[0..12]: [170, 161, 171, 171, 170, 167, 170, 171, 164, 170, 162, 171]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.19292181Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"downstairs","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.192941489Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"downstairs","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"[2]R flush_numbers[0..12]: [343, 330, 346, 346, 345, 339, 344, 346, 333, 345, 331, 346]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.192958928Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"downstairs","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"[2]R generation[0..12]: [170, 161, 171, 171, 170, 167, 170, 171, 164, 170, 162, 171]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T19:01:46.192976107Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"downstairs","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.192996437Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"downstairs","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"Max found gen is 172","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.193025215Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"downstairs","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"Generation requested: 172 >= found:172","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.193044604Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"downstairs","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"Next flush: 347","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.193061034Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"downstairs","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.193077223Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"":"downstairs","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.193093912Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.193108771Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"0","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.193125291Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"1","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.19314182Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"client":"2","":"downstairs","session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.193158489Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.193176548Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"8caa4395-a8bd-4d1d-b57e-6dd820eb874e is now active with session: 71254593-55f6-4327-955a-50fccd4e1200","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.193192837Z","hostname":"Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.193214866Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870,"session_id":"71254593-55f6-4327-955a-50fccd4e1200"} {"msg":"The guest has finished waiting for activation with:172","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:46.193230246Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870} Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 259 len 12288 data: 14 15 13 02/30 Write block 184 len 12288 data: 12 14 16 03/30 Read block 579 len 32768 04/30 Write block 358 len 16384 data: 19 16 14 12 05/30 Read block 171 len 12288 06/30 Flush 07/30 Flush 08/30 Write block 424 len 24576 data: 11 15 15 13 14 18 09/30 Flush 10/30 Write block 552 len 4096 data: 14 11/30 Read block 116 len 28672 12/30 Read block 566 len 8192 13/30 Read block 91 len 4096 14/30 Read block 366 len 28672 15/30 Flush 16/30 Write block 436 len 12288 data: 13 11 10 17/30 Write block 345 len 4096 data: 9 18/30 Write block 37 len 12288 data: 11 12 12 19/30 Read block 585 len 28672 20/30 Write block 462 len 36864 data: 8 8 7 7 8 10 10 10 12 21/30 Write block 219 len 24576 data: 11 11 12 14 11 12 22/30 Flush 23/30 Write block 404 len 32768 data: 16 16 18 18 14 12 11 9 24/30 Write block 58 len 24576 data: 9 9 7 5 4 7 25/30 Flush 26/30 Flush 27/30 Write block 90 len 16384 data: 13 13 13 12 28/30 Read block 247 len 40960 29/30 Read block 509 len 28672 30/30 Read block 104 len 12288 ---------------------------------------------------------------- Crucible gen:172 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 28 Acked 1027 Write 4 Done Done Done false 29 Acked 1028 Read 10 Done Done Done false 30 Acked 1029 Read 7 Done Done Done false 31 Acked 1030 Read 3 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 1 2 Done 4 3 3 10 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1026 1026 1026 Downstairs last five completed: 1026 1025 1024 1023 1022 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:01:46.223591189Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1870} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 170 170 343 343 343 F F F 1 020-039 172 172 172 352 352 352 F F F 2 040-059 172 172 172 353 353 353 F F F 3 060-079 172 172 172 353 353 353 F F F 4 080-099 170 170 170 345 345 345 T T T 5 100-119 167 167 167 339 339 339 F F F 6 120-139 170 170 170 344 344 344 F F F 7 140-159 171 171 171 346 346 346 F F F 8 160-179 164 164 164 333 333 333 F F F 9 180-199 172 172 172 347 347 347 F F F 10 200-219 172 172 172 352 352 352 F F F 11 220-239 172 172 172 352 352 352 F F F 12 240-259 172 172 172 347 347 347 F F F 13 260-279 172 172 172 347 347 347 F F F 14 280-299 171 171 171 346 346 346 F F F 15 300-319 171 171 171 346 346 346 F F F 16 320-339 169 169 169 342 342 342 F F F 17 340-359 172 172 172 351 351 351 F F F 18 360-379 172 172 172 347 347 347 F F F 19 380-399 167 167 167 339 339 339 F F F 20 400-419 172 172 172 353 353 353 F F F 21 420-439 172 172 172 351 351 351 F F F 22 440-459 169 169 169 342 342 342 F F F 23 460-479 172 172 172 352 352 352 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 166 166 166 338 338 338 F F F 26 520-539 166 166 166 338 338 338 F F F 27 540-559 172 172 172 350 350 350 F F F 28 560-579 162 162 162 331 331 331 F F F 29 580-599 170 170 170 345 345 345 F F F Max gen: 172, Max flush: 353 A Difference in extent metadata is expected here On loop 86 Verifying data now No change after verify Loop: 86 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 170 170 343 343 343 F F F 1 020-039 172 172 172 352 352 352 F F F 2 040-059 172 172 172 353 353 353 F F F 3 060-079 172 172 172 353 353 353 F F F 4 080-099 173 173 173 354 354 354 F F F 5 100-119 167 167 167 339 339 339 F F F 6 120-139 170 170 170 344 344 344 F F F 7 140-159 171 171 171 346 346 346 F F F 8 160-179 164 164 164 333 333 333 F F F 9 180-199 172 172 172 347 347 347 F F F 10 200-219 172 172 172 352 352 352 F F F 11 220-239 172 172 172 352 352 352 F F F 12 240-259 172 172 172 347 347 347 F F F 13 260-279 172 172 172 347 347 347 F F F 14 280-299 171 171 171 346 346 346 F F F 15 300-319 171 171 171 346 346 346 F F F 16 320-339 169 169 169 342 342 342 F F F 17 340-359 172 172 172 351 351 351 F F F 18 360-379 172 172 172 347 347 347 F F F 19 380-399 167 167 167 339 339 339 F F F 20 400-419 172 172 172 353 353 353 F F F 21 420-439 172 172 172 351 351 351 F F F 22 440-459 169 169 169 342 342 342 F F F 23 460-479 172 172 172 352 352 352 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 166 166 166 338 338 338 F F F 26 520-539 166 166 166 338 338 338 F F F 27 540-559 172 172 172 350 350 350 F F F 28 560-579 162 162 162 331 331 331 F F F 29 580-599 170 170 170 345 345 345 F F F Max gen: 173, Max flush: 354 Begin loop 87 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.397412822Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.397819723Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.397849372Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878} {"msg":"Crucible ecbb9bde-276b-4d94-b2b7-534133829ed2 has session id: 46eee72e-6b8e-4ad5-bb0b-319bc97c5844","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.398139378Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"Upstairs opts: Upstairs UUID: ecbb9bde-276b-4d94-b2b7-534133829ed2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.398188066Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"Crucible stats registered with UUID: ecbb9bde-276b-4d94-b2b7-534133829ed2","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.398211615Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:174","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.398468343Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878} {"msg":"ecbb9bde-276b-4d94-b2b7-534133829ed2 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.398548999Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.398569048Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"0","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.398589807Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"0","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.398611716Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"1","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.398628045Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"1","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.398652134Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.398672363Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.398692232Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"io task","client":"0","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.398708681Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"io task","client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.39872604Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"io task","client":"1","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"ds_connection connected from Ok(127.0.0.1:58761)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.398936191Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"io task","client":"1","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"ds_connection connected from Ok(127.0.0.1:61972)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.398972859Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"io task","client":"0","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:01:56.399187089Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"io task","client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:01:56.399226707Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"io task","client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:01:56.399259076Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.399284825Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.399308173Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.399329712Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"io task","client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.3998143Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"1","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.399850508Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"1","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.399911835Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"0","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.399946623Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"0","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.400486888Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"1","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.400518517Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"0","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.401015754Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"0","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.401042872Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"0","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.401061491Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.40108955Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"1","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.40110434Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"1","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:01:56.401118099Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.400028101Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"io task","client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"ds_connection connected from Ok(127.0.0.1:60944)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.400288049Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"io task","client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.401448185Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.401475223Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.401938902Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402487016Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402509995Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"[0]R flush_numbers[0..12]: [343, 352, 353, 353, 354, 339, 344, 346, 333, 347, 352, 352]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402525685Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"downstairs","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"[0]R generation[0..12]: [170, 172, 172, 172, 173, 167, 170, 171, 164, 172, 172, 172]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402540464Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"downstairs","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402554793Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"downstairs","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"[1]R flush_numbers[0..12]: [343, 352, 353, 353, 354, 339, 344, 346, 333, 347, 352, 352]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402569092Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"downstairs","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"[1]R generation[0..12]: [170, 172, 172, 172, 173, 167, 170, 171, 164, 172, 172, 172]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402588142Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"downstairs","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402602441Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"downstairs","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"[2]R flush_numbers[0..12]: [343, 352, 353, 353, 354, 339, 344, 346, 333, 347, 352, 352]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.40261653Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"downstairs","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"[2]R generation[0..12]: [170, 172, 172, 172, 173, 167, 170, 171, 164, 172, 172, 172]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402630489Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"downstairs","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402644769Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"downstairs","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"Max found gen is 174","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402657968Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"downstairs","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"Generation requested: 174 >= found:174","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402671068Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"downstairs","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"Next flush: 355","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402687747Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"downstairs","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402700756Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"":"downstairs","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402714326Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402726385Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"0","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402739864Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"1","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402753484Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"client":"2","":"downstairs","session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402766893Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402779262Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"ecbb9bde-276b-4d94-b2b7-534133829ed2 is now active with session: 46eee72e-6b8e-4ad5-bb0b-319bc97c5844","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402794262Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402806471Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878,"session_id":"46eee72e-6b8e-4ad5-bb0b-319bc97c5844"} {"msg":"The guest has finished waiting for activation with:174","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:06.402818751Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 36 len 16384 data: 13 12 13 13 02/30 Write block 106 len 20480 data: 8 9 11 9 10 03/30 Read block 71 len 8192 04/30 Read block 103 len 36864 05/30 Flush 06/30 Write block 115 len 16384 data: 12 12 12 12 07/30 Write block 107 len 28672 data: 10 12 10 11 10 10 10 08/30 Read block 122 len 8192 09/30 Read block 533 len 20480 10/30 Read block 389 len 24576 11/30 Read block 359 len 20480 12/30 Write block 448 len 32768 data: 17 16 19 17 19 15 12 13 13/30 Write block 232 len 36864 data: 16 13 15 17 18 15 17 15 15 14/30 Flush 15/30 Flush 16/30 Flush 17/30 Write block 225 len 16384 data: 13 11 13 13 18/30 Read block 128 len 36864 19/30 Read block 347 len 4096 20/30 Flush 21/30 Write block 558 len 4096 data: 9 22/30 Write block 214 len 4096 data: 11 23/30 Flush 24/30 Write block 444 len 16384 data: 17 21 17 14 25/30 Write block 469 len 40960 data: 11 13 15 14 14 15 15 15 16 12 26/30 Write block 63 len 12288 data: 8 7 9 27/30 Write block 450 len 28672 data: 20 18 20 16 13 14 11 28/30 Read block 376 len 40960 29/30 Write block 574 len 40960 data: 19 15 15 16 15 12 12 12 15 12 30/30 Write block 53 len 4096 data: 12 ---------------------------------------------------------------- Crucible gen:174 GIO:true work queues: Upstairs:0 downstairs:7 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 4 Done Done Done false 25 Acked 1024 Write 10 Done Done Done false 26 Acked 1025 Write 3 Done Done Done false 27 Acked 1026 Write 7 Done Done Done false 28 Acked 1027 Read 10 Done Done Done false 29 Acked 1028 Write 10 Sent Sent Sent false 30 Acked 1029 Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 2 2 2 6 Done 5 5 5 15 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:02:06.431376698Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1878} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 170 170 343 343 343 F F F 1 020-039 174 174 174 355 355 355 F F F 2 040-059 172 172 172 353 353 353 T T T 3 060-079 172 172 172 353 353 353 T T T 4 080-099 173 173 173 354 354 354 F F F 5 100-119 174 174 174 356 356 356 F F F 6 120-139 170 170 170 344 344 344 F F F 7 140-159 171 171 171 346 346 346 F F F 8 160-179 164 164 164 333 333 333 F F F 9 180-199 172 172 172 347 347 347 F F F 10 200-219 174 174 174 360 360 360 F F F 11 220-239 174 174 174 359 359 359 F F F 12 240-259 174 174 174 356 356 356 F F F 13 260-279 172 172 172 347 347 347 F F F 14 280-299 171 171 171 346 346 346 F F F 15 300-319 171 171 171 346 346 346 F F F 16 320-339 169 169 169 342 342 342 F F F 17 340-359 172 172 172 351 351 351 F F F 18 360-379 172 172 172 347 347 347 F F F 19 380-399 167 167 167 339 339 339 F F F 20 400-419 172 172 172 353 353 353 F F F 21 420-439 172 172 172 351 351 351 F F F 22 440-459 174 174 174 356 356 356 T T T 23 460-479 172 172 172 352 352 352 T T T 24 480-499 160 160 160 329 329 329 F F F 25 500-519 166 166 166 338 338 338 F F F 26 520-539 166 166 166 338 338 338 F F F 27 540-559 174 174 174 360 360 360 F F F 28 560-579 162 162 162 331 331 331 T T T 29 580-599 170 170 170 345 345 345 T T T Max gen: 174, Max flush: 360 A Difference in extent metadata is expected here On loop 87 Verifying data now No change after verify Loop: 87 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 170 170 170 343 343 343 F F F 1 020-039 174 174 174 355 355 355 F F F 2 040-059 175 175 175 361 361 361 F F F 3 060-079 175 175 175 361 361 361 F F F 4 080-099 173 173 173 354 354 354 F F F 5 100-119 174 174 174 356 356 356 F F F 6 120-139 170 170 170 344 344 344 F F F 7 140-159 171 171 171 346 346 346 F F F 8 160-179 164 164 164 333 333 333 F F F 9 180-199 172 172 172 347 347 347 F F F 10 200-219 174 174 174 360 360 360 F F F 11 220-239 174 174 174 359 359 359 F F F 12 240-259 174 174 174 356 356 356 F F F 13 260-279 172 172 172 347 347 347 F F F 14 280-299 171 171 171 346 346 346 F F F 15 300-319 171 171 171 346 346 346 F F F 16 320-339 169 169 169 342 342 342 F F F 17 340-359 172 172 172 351 351 351 F F F 18 360-379 172 172 172 347 347 347 F F F 19 380-399 167 167 167 339 339 339 F F F 20 400-419 172 172 172 353 353 353 F F F 21 420-439 172 172 172 351 351 351 F F F 22 440-459 175 175 175 361 361 361 F F F 23 460-479 175 175 175 361 361 361 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 166 166 166 338 338 338 F F F 26 520-539 166 166 166 338 338 338 F F F 27 540-559 174 174 174 360 360 360 F F F 28 560-579 175 175 175 361 361 361 F F F 29 580-599 175 175 175 361 361 361 F F F Max gen: 175, Max flush: 361 Begin loop 88 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.678055348Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.678445689Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.678467778Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886} {"msg":"Crucible 5270fe38-1215-43de-a2ae-c749d11dad1d has session id: bd82655a-91b9-482b-8aa4-d12d1498c4a1","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.678675819Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"Upstairs opts: Upstairs UUID: 5270fe38-1215-43de-a2ae-c749d11dad1d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.678721966Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"Crucible stats registered with UUID: 5270fe38-1215-43de-a2ae-c749d11dad1d","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.678744965Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:176","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.679046641Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886} {"msg":"5270fe38-1215-43de-a2ae-c749d11dad1d active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.67907307Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.679093879Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"0","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.679116638Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"0","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.679141767Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"1","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.679169645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"1","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.679193194Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.679214313Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.679232922Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"io task","client":"0","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.679252132Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"io task","client":"1","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.679271911Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"io task","client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:02:20.679405174Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"io task","client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:02:20.679450222Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"io task","client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"ds_connection connected from Ok(127.0.0.1:56405)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.679477821Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"io task","client":"1","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"ds_connection connected from Ok(127.0.0.1:55035)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.67950234Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"io task","client":"0","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:02:20.679569147Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.679596376Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.679611215Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.679790407Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"io task","client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.680339121Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"1","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.68036533Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"1","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.680380529Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"0","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.680394658Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"0","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.680989091Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"0","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.681105675Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"1","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.681484657Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"0","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.681510876Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"0","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.681529625Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.681569404Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"1","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.681586333Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"1","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:20.681599802Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.680143531Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"io task","client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"ds_connection connected from Ok(127.0.0.1:49145)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.680345612Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"io task","client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.681761115Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.681790404Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.6823097Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.682795537Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.682816506Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"[0]R flush_numbers[0..12]: [343, 355, 361, 361, 354, 356, 344, 346, 333, 347, 360, 359]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.682831565Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"downstairs","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"[0]R generation[0..12]: [170, 174, 175, 175, 173, 174, 170, 171, 164, 172, 174, 174]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.682846145Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"downstairs","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.682860374Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"downstairs","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"[1]R flush_numbers[0..12]: [343, 355, 361, 361, 354, 356, 344, 346, 333, 347, 360, 359]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.682874454Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"downstairs","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"[1]R generation[0..12]: [170, 174, 175, 175, 173, 174, 170, 171, 164, 172, 174, 174]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.682891983Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"downstairs","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.682906312Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"downstairs","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"[2]R flush_numbers[0..12]: [343, 355, 361, 361, 354, 356, 344, 346, 333, 347, 360, 359]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.682920081Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"downstairs","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"[2]R generation[0..12]: [170, 174, 175, 175, 173, 174, 170, 171, 164, 172, 174, 174]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.682933911Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"downstairs","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.68294789Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"downstairs","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"Max found gen is 176","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.682960799Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"downstairs","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"Generation requested: 176 >= found:176","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.682974509Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"downstairs","":"downstairs"Wait for a query_work_queue command to finish before sending IO ,"session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"Next flush: 362","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.682992058Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"downstairs","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.683005037Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"":"downstairs","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.683018207Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.683030216Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"0","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.683043466Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"1","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.683056575Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"client":"2","":"downstairs","session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.683069845Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.683081974Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"5270fe38-1215-43de-a2ae-c749d11dad1d is now active with session: bd82655a-91b9-482b-8aa4-d12d1498c4a1","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.683096863Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.683108803Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886,"session_id":"bd82655a-91b9-482b-8aa4-d12d1498c4a1"} {"msg":"The guest has finished waiting for activation with:176","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:30.683121022Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 2 len 40960 data: 7 8 7 7 9 9 11 12 11 12 02/30 Read block 158 len 28672 03/30 Write block 431 len 40960 data: 19 18 15 12 15 14 12 11 13 11 04/30 Write block 423 len 28672 data: 15 12 16 16 14 15 19 05/30 Write block 412 len 12288 data: 9 11 12 06/30 Write block 372 len 20480 data: 18 17 13 16 15 07/30 Read block 384 len 4096 08/30 Read block 487 len 24576 09/30 Write block 273 len 24576 data: 8 11 11 13 15 14 10/30 Write block 529 len 16384 data: 11 13 16 15 11/30 Write block 396 len 24576 data: 16 17 17 15 14 14 12/30 Read block 243 len 32768 13/30 Write block 60 len 12288 data: 8 6 5 14/30 Read block 510 len 32768 15/30 Read block 400 len 32768 16/30 Read block 463 len 36864 17/30 Write block 292 len 36864 data: 13 10 10 10 9 7 5 7 8 18/30 Write block 60 len 28672 data: 9 7 6 9 8 10 8 19/30 Read block 35 len 40960 20/30 Read block 43 len 40960 21/30 Read block 78 len 8192 22/30 Read block 134 len 40960 23/30 Write block 540 len 8192 data: 12 13 24/30 Read block 313 len 32768 25/30 Write block 464 len 24576 data: 8 8 9 11 11 12 26/30 Read block 2 len 16384 27/30 Read block 359 len 24576 28/30 Read block 107 len 4096 29/30 Read block 41 len 20480 30/30 Write block 526 len 24576 data: 12 12 11 12 14 17 ---------------------------------------------------------------- Crucible gen:176 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 2 Done Done Done false 25 Acked 1024 Read 8 Done Done Done false 26 Acked 1025 Write 6 Done Done Done false 27 Acked 1026 Read 4 Done Done Done false 28 Acked 1027 Read 6 Done Done Done false 29 Acked 1028 Read 1 Done Done Done false 30 Acked 1029 Read 5 Done Done Done false 31 Acked 1030 Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:02:30.709014404Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1886} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 176 176 176 362 362 362 F F F 1 020-039 174 174 174 355 355 355 F F F 2 040-059 175 175 175 361 361 361 F F F 3 060-079 176 176 176 362 362 362 F F F 4 080-099 173 173 173 354 354 354 F F F 5 100-119 174 174 174 356 356 356 F F F 6 120-139 170 170 170 344 344 344 F F F 7 140-159 171 171 171 346 346 346 F F F 8 160-179 164 164 164 333 333 333 F F F 9 180-199 172 172 172 347 347 347 F F F 10 200-219 174 174 174 360 360 360 F F F 11 220-239 174 174 174 359 359 359 F F F 12 240-259 174 174 174 356 356 356 F F F 13 260-279 176 176 176 362 362 362 F F F 14 280-299 176 176 176 362 362 362 F F F 15 300-319 176 176 176 362 362 362 F F F 16 320-339 169 169 169 342 342 342 F F F 17 340-359 172 172 172 351 351 351 F F F 18 360-379 176 176 176 362 362 362 F F F 19 380-399 176 176 176 362 362 362 F F F 20 400-419 176 176 176 362 362 362 F F F 21 420-439 176 176 176 362 362 362 F F F 22 440-459 176 176 176 362 362 362 F F F 23 460-479 175 175 175 361 361 361 T T T 24 480-499 160 160 160 329 329 329 F F F 25 500-519 166 166 166 338 338 338 F F F 26 520-539 176 176 176 362 362 362 T T T 27 540-559 174 174 174 360 360 360 T T T 28 560-579 175 175 175 361 361 361 F F F 29 580-599 175 175 175 361 361 361 F F F Max gen: 176, Max flush: 362 A Difference in extent metadata is expected here On loop 88 Verifying data now No change after verify Loop: 88 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 176 176 176 362 362 362 F F F 1 020-039 174 174 174 355 355 355 F F F 2 040-059 175 175 175 361 361 361 F F F 3 060-079 176 176 176 362 362 362 F F F 4 080-099 173 173 173 354 354 354 F F F 5 100-119 174 174 174 356 356 356 F F F 6 120-139 170 170 170 344 344 344 F F F 7 140-159 171 171 171 346 346 346 F F F 8 160-179 164 164 164 333 333 333 F F F 9 180-199 172 172 172 347 347 347 F F F 10 200-219 174 174 174 360 360 360 F F F 11 220-239 174 174 174 359 359 359 F F F 12 240-259 174 174 174 356 356 356 F F F 13 260-279 176 176 176 362 362 362 F F F 14 280-299 176 176 176 362 362 362 F F F 15 300-319 176 176 176 362 362 362 F F F 16 320-339 169 169 169 342 342 342 F F F 17 340-359 172 172 172 351 351 351 F F F 18 360-379 176 176 176 362 362 362 F F F 19 380-399 176 176 176 362 362 362 F F F 20 400-419 176 176 176 362 362 362 F F F 21 420-439 176 176 176 362 362 362 F F F 22 440-459 176 176 176 362 362 362 F F F 23 460-479 177 177 177 363 363 363 F F F 24 480-499 160 160 160 329 329 329 F F F 25 500-519 166 166 166 338 338 338 F F F 26 520-539 177 177 177 363 363 363 F F F 27 540-559 177 177 177 363 363 363 F F F 28 560-579 175 175 175 361 361 361 F F F 29 580-599 175 175 175 361 361 361 F F F Max gen: 177, Max flush: 363 Begin loop 89 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.913456125Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.913894145Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.913925194Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894} {"msg":"Crucible 347cfa64-e946-45fb-808d-07677dadede0 has session id: 22a71751-5638-48d5-8451-494fa07690fa","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.914157833Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"Upstairs opts: Upstairs UUID: 347cfa64-e946-45fb-808d-07677dadede0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.91421308Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"Crucible stats registered with UUID: 347cfa64-e946-45fb-808d-07677dadede0","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.914241479Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"session_id":"22a71751-5638-48d5-8451-494fa07690fa"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:178","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.914530445Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894} {"msg":"347cfa64-e946-45fb-808d-07677dadede0 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.914560894Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.914582363Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"0","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.914609732Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"0","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.914632281Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.91465442Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.914675359Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"2","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.914696947Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"2","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.914718086Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"io task","client":"0","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.914739965Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.914762424Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"io task","client":"2","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"ds_connection connected from Ok(127.0.0.1:57323)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.914797063Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"io task","client":"0","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"ds_connection connected from Ok(127.0.0.1:65251)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.914887719Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"io task","client":"2","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:02:44.915038592Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:02:44.91508334Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:02:44.915186765Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.915213114Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.915228813Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.91529801Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.915645883Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"0","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.915684361Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"0","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.91571403Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"2","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.915740739Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"2","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.916280734Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"2","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.916399578Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"0","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.916748162Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"2","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.916775401Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"2","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.91679015Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.916919904Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"0","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.916939383Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"0","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:44.916952942Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.916592662Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"ds_connection connected from Ok(127.0.0.1:49091)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.91706854Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.91792293Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.917949879Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.918426656Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.918957841Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.918976471Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"[0]R flush_numbers[0..12]: [362, 355, 361, 362, 354, 356, 344, 346, 333, 347, 360, 359]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.91899074Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"downstairs","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"[0]R generation[0..12]: [176, 174, 175, 176, 173, 174, 170, 171, 164, 172, 174, 174]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919046107Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"downstairs","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919065627Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"downstairs","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"[1]R flush_numbers[0..12]: [362, 355, 361, 362, 354, 356, 344, 346, 333, 347, 360, 359]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919084575Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"downstairs","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"[1]R generation[0..12]: [176, 174, 175, 176, 173, 174, 170, 171, 164, 172, 174, 174]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919099995Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"downstairs","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919117994Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"downstairs","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"[2]R flush_numbers[0..12]: [362, 355, 361, 362, 354, 356, 344, 346, 333, 347, 360, 359]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919135423Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"downstairs","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"[2]R generation[0..12]: [176, 174, 175, 176, 173, 174, 170, 171, 164, 172, 174, 174]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919152512Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"downstairs","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919170071Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"downstairs","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"Max found gen is 178","v":0,"name":"crucible","level":30,Wait for a query_work_queue command to finish before sending IO "time":"2024-07-31T19:02:54.919186181Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"downstairs","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"Generation requested: 178 >= found:178","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.91920698Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"downstairs","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"Next flush: 364","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919223179Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"downstairs","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919239258Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"":"downstairs","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919255927Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919270947Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"0","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919287386Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"1","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919303895Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"client":"2","":"downstairs","session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919322804Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919338514Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"347cfa64-e946-45fb-808d-07677dadede0 is now active with session: 22a71751-5638-48d5-8451-494fa07690fa","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919361323Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919383882Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894,"session_id":"22a71751-5638-48d5-8451-494fa07690fa"} {"msg":"The guest has finished waiting for activation with:178","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:02:54.919399581Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 330 len 8192 data: 14 14 02/30 Read block 470 len 32768 03/30 Read block 594 len 8192 04/30 Write block 23 len 8192 data: 14 18 05/30 Read block 195 len 32768 06/30 Read block 557 len 36864 07/30 Read block 312 len 12288 08/30 Read block 219 len 20480 09/30 Flush 10/30 Write block 518 len 24576 data: 12 15 16 14 13 12 11/30 Read block 7 len 8192 12/30 Write block 170 len 36864 data: 17 16 15 14 12 13 10 8 9 13/30 Read block 258 len 24576 14/30 Flush 15/30 Flush 16/30 Write block 181 len 28672 data: 8 10 12 13 15 17 19 17/30 Read block 421 len 4096 18/30 Read block 448 len 28672 19/30 Write block 484 len 28672 data: 17 16 12 14 13 12 12 20/30 Write block 290 len 16384 data: 12 14 14 11 21/30 Read block 4 len 40960 22/30 Flush 23/30 Write block 362 len 28672 data: 12 12 14 12 12 14 16 24/30 Write block 186 len 16384 data: 18 20 20 18 25/30 Read block 104 len 40960 26/30 Read block 181 len 16384 27/30 Write block 10 len 16384 data: 12 13 10 8 28/30 Write block 198 len 32768 data: 14 11 12 13 12 11 13 12 29/30 Read block 206 len 32768 30/30 Read block 322 len 24576 ---------------------------------------------------------------- Crucible gen:178 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 7 Done Done Done false 25 Acked 1024 Write 4 Done Done Done false 26 Acked 1025 Read 10 Done Done Done false 27 Acked 1026 Read 4 Done Done Done false 28 Acked 1027 Write 4 Done Done Done false 29 Acked 1028 Write 8 Done Done Done false 30 Acked 1029 Read 8 Done Done Done false 31 Acked 1030 Read 6 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 8 8 8 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:02:54.947769897Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1894} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 176 176 176 362 362 362 T T T 1 020-039 178 178 178 364 364 364 F F F 2 040-059 175 175 175 361 361 361 F F F 3 060-079 176 176 176 362 362 362 F F F 4 080-099 173 173 173 354 354 354 F F F 5 100-119 174 174 174 356 356 356 F F F 6 120-139 170 170 170 344 344 344 F F F 7 140-159 171 171 171 346 346 346 F F F 8 160-179 178 178 178 365 365 365 F F F 9 180-199 178 178 178 367 367 367 T T T 10 200-219 174 174 174 360 360 360 T T T 11 220-239 174 174 174 359 359 359 F F F 12 240-259 174 174 174 356 356 356 F F F 13 260-279 176 176 176 362 362 362 F F F 14 280-299 178 178 178 368 368 368 F F F 15 300-319 176 176 176 362 362 362 F F F 16 320-339 178 178 178 364 364 364 F F F 17 340-359 172 172 172 351 351 351 F F F 18 360-379 176 176 176 362 362 362 T T T 19 380-399 176 176 176 362 362 362 F F F 20 400-419 176 176 176 362 362 362 F F F 21 420-439 176 176 176 362 362 362 F F F 22 440-459 176 176 176 362 362 362 F F F 23 460-479 177 177 177 363 363 363 F F F 24 480-499 178 178 178 368 368 368 F F F 25 500-519 178 178 178 365 365 365 F F F 26 520-539 178 178 178 365 365 365 F F F 27 540-559 177 177 177 363 363 363 F F F 28 560-579 175 175 175 361 361 361 F F F 29 580-599 175 175 175 361 361 361 F F F Max gen: 178, Max flush: 368 A Difference in extent metadata is expected here On loop 89 Verifying data now No change after verify Loop: 89 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 179 179 179 369 369 369 F F F 1 020-039 178 178 178 364 364 364 F F F 2 040-059 175 175 175 361 361 361 F F F 3 060-079 176 176 176 362 362 362 F F F 4 080-099 173 173 173 354 354 354 F F F 5 100-119 174 174 174 356 356 356 F F F 6 120-139 170 170 170 344 344 344 F F F 7 140-159 171 171 171 346 346 346 F F F 8 160-179 178 178 178 365 365 365 F F F 9 180-199 179 179 179 369 369 369 F F F 10 200-219 179 179 179 369 369 369 F F F 11 220-239 174 174 174 359 359 359 F F F 12 240-259 174 174 174 356 356 356 F F F 13 260-279 176 176 176 362 362 362 F F F 14 280-299 178 178 178 368 368 368 F F F 15 300-319 176 176 176 362 362 362 F F F 16 320-339 178 178 178 364 364 364 F F F 17 340-359 172 172 172 351 351 351 F F F 18 360-379 179 179 179 369 369 369 F F F 19 380-399 176 176 176 362 362 362 F F F 20 400-419 176 176 176 362 362 362 F F F 21 420-439 176 176 176 362 362 362 F F F 22 440-459 176 176 176 362 362 362 F F F 23 460-479 177 177 177 363 363 363 F F F 24 480-499 178 178 178 368 368 368 F F F 25 500-519 178 178 178 365 365 365 F F F 26 520-539 178 178 178 365 365 365 F F F 27 540-559 177 177 177 363 363 363 F F F 28 560-579 175 175 175 361 361 361 F F F 29 580-599 175 175 175 361 361 361 F F F Max gen: 179, Max flush: 369 Begin loop 90 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.171994297Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.172324912Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.172359271Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902} {"msg":"Crucible ed84d29e-b0f0-4cb4-9870-fc9aefe4db34 has session id: dcd7478b-ec50-4a54-9636-13a500856e4b","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.172629128Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"Upstairs opts: Upstairs UUID: ed84d29e-b0f0-4cb4-9870-fc9aefe4db34, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.172673746Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"Crucible stats registered with UUID: ed84d29e-b0f0-4cb4-9870-fc9aefe4db34","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.172690895Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:180","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.172985411Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902} {"msg":"ed84d29e-b0f0-4cb4-9870-fc9aefe4db34 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.17300848Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.173027489Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"0","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.173044799Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"0","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.173058718Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.173079997Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.173105446Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"2","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.173132454Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"2","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.173158223Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"io task","client":"0","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.173183102Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"io task","client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.173207261Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"io task","client":"2","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"ds_connection connected from Ok(127.0.0.1:43731)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.173287257Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"io task","client":"2","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"ds_connection connected from Ok(127.0.0.1:54960)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.173505227Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"io task","client":"0","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:03:09.173548915Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"io task","client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:03:09.173581023Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"io task","client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:03:09.173611702Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.173635631Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.17365107Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.173699188Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"io task","client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.174238003Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"2","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.174265462Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"2","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.174281341Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"0","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.17430071Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"0","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.174800446Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"0","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.17494167Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"2","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.175309083Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"0","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.175336561Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"0","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.175351471Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.17536389Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"2","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.17537766Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"2","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:09.175391019Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.17411888Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"io task","client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"ds_connection connected from Ok(127.0.0.1:37978)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.174282272Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"io task","client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.175517265Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.175564593Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176088568Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176645512Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176673971Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"[0]R flush_numbers[0..12]: [369, 364, 361, 362, 354, 356, 344, 346, 365, 369, 369, 359]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.17669327Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"downstairs","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"[0]R generation[0..12]: [179, 178, 175, 176, 173, 174, 170, 171, 178, 179, 179, 174]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176711899Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"downstairs","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176729738Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"downstairs","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"[1]R flush_numbers[0..12]: [369, 364, 361, 362, 354, 356, 344, 346, 365, 369, 369, 359]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176752117Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"downstairs","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"[1]R generation[0..12]: [179, 178, 175, 176, 173, 174, 170, 171, 178, 179, 179, 174]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176769826Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"downstairs","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176787385Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"downstairs","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"[2]R flush_numbers[0..12]: [369, 364, 361, 362, 354, 356, 344, 346, 365, 369, 369, 359]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176804725Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"downstairs","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"[2]R generation[0..12]: [179, 178, 175, 176, 173, 174, 170, 171, 178, 179, 179, 174]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176821894Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"downstairs","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} Wait for a query_work_queue command to finish before sending IO {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176843333Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"downstairs","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"Max found gen is 180","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176860302Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"downstairs","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"Generation requested: 180 >= found:180","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176876771Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"downstairs","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"Next flush: 370","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.17689292Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"downstairs","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.1769089Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"":"downstairs","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176925619Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176940618Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"0","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176957428Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"1","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176973847Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"client":"2","":"downstairs","session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.176992896Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.177008445Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"ed84d29e-b0f0-4cb4-9870-fc9aefe4db34 is now active with session: dcd7478b-ec50-4a54-9636-13a500856e4b","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.177024784Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.177039444Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902,"session_id":"dcd7478b-ec50-4a54-9636-13a500856e4b"} {"msg":"The guest has finished waiting for activation with:180","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:19.177054533Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 164 len 40960 data: 10 12 14 15 16 16 18 17 16 15 03/30 Write block 560 len 4096 data: 10 04/30 Read block 40 len 20480 05/30 Write block 467 len 24576 data: 12 12 13 14 16 15 06/30 Read block 279 len 12288 07/30 Flush 08/30 Write block 456 len 40960 data: 12 10 10 12 11 11 9 9 9 9 09/30 Write block 320 len 12288 data: 12 10 10 10/30 Write block 62 len 12288 data: 7 10 9 11/30 Write block 511 len 16384 data: 16 12 13 16 12/30 Read block 134 len 8192 13/30 Write block 183 len 20480 data: 13 14 16 19 21 14/30 Read block 503 len 16384 15/30 Write block 304 len 4096 data: 11 16/30 Write block 462 len 16384 data: 10 10 10 10 17/30 Write block 167 len 20480 data: 16 17 17 19 18 18/30 Read block 275 len 40960 19/30 Write block 81 len 16384 data: 23 19 19 21 20/30 Write block 68 len 8192 data: 6 7 21/30 Read block 406 len 28672 22/30 Read block 134 len 36864 23/30 Write block 215 len 28672 data: 12 12 10 9 12 12 13 24/30 Read block 221 len 8192 25/30 Read block 298 len 16384 26/30 Flush 27/30 Write block 386 len 28672 data: 16 19 18 16 17 19 21 28/30 Read block 194 len 24576 29/30 Write block 174 len 16384 data: 13 14 11 9 30/30 Read block 430 len 40960 ---------------------------------------------------------------- Crucible gen:180 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 28 Acked 1027 Write 7 Done Done Done false 29 Acked 1028 Read 6 Done Done Done false 30 Acked 1029 Write 4 Done Done Done false 31 Acked 1030 Read 10 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 0 2 Done 3 3 4 10 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1026 1026 1026 Downstairs last five completed: 1026 1025 1024 1023 1022 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:03:19.204306461Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1902} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 179 179 179 369 369 369 F F F 1 020-039 178 178 178 364 364 364 F F F 2 040-059 175 175 175 361 361 361 F F F 3 060-079 180 180 180 372 372 372 F F F 4 080-099 180 180 180 372 372 372 F F F 5 100-119 174 174 174 356 356 356 F F F 6 120-139 170 170 170 344 344 344 F F F 7 140-159 171 171 171 346 346 346 F F F 8 160-179 180 180 180 372 372 372 T T T 9 180-199 180 180 180 372 372 372 F F F 10 200-219 180 180 180 372 372 372 F F F 11 220-239 180 180 180 372 372 372 F F F 12 240-259 174 174 174 356 356 356 F F F 13 260-279 176 176 176 362 362 362 F F F 14 280-299 178 178 178 368 368 368 F F F 15 300-319 180 180 180 372 372 372 F F F 16 320-339 180 180 180 372 372 372 F F F 17 340-359 172 172 172 351 351 351 F F F 18 360-379 179 179 179 369 369 369 F F F 19 380-399 176 176 176 362 362 362 T T T 20 400-419 176 176 176 362 362 362 F F F 21 420-439 176 176 176 362 362 362 F F F 22 440-459 180 180 180 372 372 372 F F F 23 460-479 180 180 180 372 372 372 F F F 24 480-499 178 178 178 368 368 368 F F F 25 500-519 180 180 180 372 372 372 F F F 26 520-539 178 178 178 365 365 365 F F F 27 540-559 177 177 177 363 363 363 F F F 28 560-579 180 180 180 371 371 371 F F F 29 580-599 175 175 175 361 361 361 F F F Max gen: 180, Max flush: 372 A Difference in extent metadata is expected here On loop 90 Verifying data now No change after verify Loop: 90 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 179 179 179 369 369 369 F F F 1 020-039 178 178 178 364 364 364 F F F 2 040-059 175 175 175 361 361 361 F F F 3 060-079 180 180 180 372 372 372 F F F 4 080-099 180 180 180 372 372 372 F F F 5 100-119 174 174 174 356 356 356 F F F 6 120-139 170 170 170 344 344 344 F F F 7 140-159 171 171 171 346 346 346 F F F 8 160-179 181 181 181 373 373 373 F F F 9 180-199 180 180 180 372 372 372 F F F 10 200-219 180 180 180 372 372 372 F F F 11 220-239 180 180 180 372 372 372 F F F 12 240-259 174 174 174 356 356 356 F F F 13 260-279 176 176 176 362 362 362 F F F 14 280-299 178 178 178 368 368 368 F F F 15 300-319 180 180 180 372 372 372 F F F 16 320-339 180 180 180 372 372 372 F F F 17 340-359 172 172 172 351 351 351 F F F 18 360-379 179 179 179 369 369 369 F F F 19 380-399 181 181 181 373 373 373 F F F 20 400-419 176 176 176 362 362 362 F F F 21 420-439 176 176 176 362 362 362 F F F 22 440-459 180 180 180 372 372 372 F F F 23 460-479 180 180 180 372 372 372 F F F 24 480-499 178 178 178 368 368 368 F F F 25 500-519 180 180 180 372 372 372 F F F 26 520-539 178 178 178 365 365 365 F F F 27 540-559 177 177 177 363 363 363 F F F 28 560-579 180 180 180 371 371 371 F F F 29 580-599 175 175 175 361 361 361 F F F Max gen: 181, Max flush: 373 Begin loop 91 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.400216341Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.400512667Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.400532146Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910} {"msg":"Crucible 950d7fb1-86c8-40f7-8527-bbc329bfbfe4 has session id: f5b11a16-b7e8-488f-b3fa-03c5fa18378e","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.400966126Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"Upstairs opts: Upstairs UUID: 950d7fb1-86c8-40f7-8527-bbc329bfbfe4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.401000184Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"Crucible stats registered with UUID: 950d7fb1-86c8-40f7-8527-bbc329bfbfe4","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.401015744Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:182","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.401340208Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910} {"msg":"950d7fb1-86c8-40f7-8527-bbc329bfbfe4 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.401410765Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.401436514Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"0","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.401467393Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"0","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.401495461Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.40152172Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.401551819Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"2","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.401577937Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"2","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.401601026Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"io task","client":"2","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.401623955Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"io task","client":"0","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.401646004Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"io task","client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"ds_connection connected from Ok(127.0.0.1:49634)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.401691022Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"io task","client":"2","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"ds_connection connected from Ok(127.0.0.1:48687)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.401722341Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"io task","client":"0","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:03:33.401752969Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"io task","client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:03:33.401919292Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"io task","client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:03:33.40195315Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.401979659Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.402006248Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.402033136Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"io task","client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.402550272Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"2","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.402574731Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"2","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.40258881Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"0","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.40260248Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"0","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.403208681Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"2","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.40323766Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"0","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.403788974Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"2","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.403816093Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"2","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.403832342Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.403853611Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"0","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.40387477Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"0","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:33.40388901Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.402872718Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"io task","client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"ds_connection connected from Ok(127.0.0.1:55640)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.403078718Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"io task","client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.404248194Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.404289112Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.404789458Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405321474Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405345072Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"[0]R flush_numbers[0..12]: [369, 364, 361, 372, 372, 356, 344, 346, 373, 372, 372, 372]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405365342Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"downstairs","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"[0]R generation[0..12]: [179, 178, 175, 180, 180, 174, 170, 171, 181, 180, 180, 180]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405383421Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"downstairs","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.40540147Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"downstairs","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"[1]R flush_numbers[0..12]: [369, 364, 361, 372, 372, 356, 344, 346, 373, 372, 372, 372]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405424399Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"downstairs","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"[1]R generation[0..12]: [179, 178, 175, 180, 180, 174, 170, 171, 181, 180, 180, 180]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405442158Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"downstairs","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405459847Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"downstairs","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"[2]R flush_numbers[0..12]: [369, 364, 361, 372, 372, 356, 344, 346, 373, 372, 372, 372]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405477096Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"downstairs","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"[2]R generation[0..12]: [179, 178, 175, 180, 180, 174, 170, 171, 181, 180, 180, 180]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405494005Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"downstairs","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"Wait for a query_work_queue command to finish before sending IO [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405514435Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"downstairs","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"Max found gen is 182","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405531694Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"downstairs","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"Generation requested: 182 >= found:182","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405548273Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"downstairs","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"Next flush: 374","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405564492Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"downstairs","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405580541Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"":"downstairs","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405597001Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.40561195Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"0","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405628629Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"1","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405644929Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"client":"2","":"downstairs","session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405663458Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405678927Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"950d7fb1-86c8-40f7-8527-bbc329bfbfe4 is now active with session: f5b11a16-b7e8-488f-b3fa-03c5fa18378e","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405695056Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405709775Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910,"session_id":"f5b11a16-b7e8-488f-b3fa-03c5fa18378e"} {"msg":"The guest has finished waiting for activation with:182","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:43.405725195Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 527 len 4096 data: 13 02/30 Read block 333 len 16384 03/30 Write block 122 len 16384 data: 14 13 12 12 04/30 Write block 36 len 36864 data: 14 13 14 14 7 6 7 8 9 05/30 Read block 278 len 40960 06/30 Read block 72 len 4096 07/30 Read block 258 len 20480 08/30 Write block 209 len 20480 data: 11 10 9 11 9 09/30 Flush 10/30 Write block 219 len 4096 data: 13 11/30 Read block 276 len 28672 12/30 Read block 214 len 28672 13/30 Flush 14/30 Write block 145 len 20480 data: 16 15 14 14 13 15/30 Read block 428 len 40960 16/30 Write block 371 len 20480 data: 20 19 18 14 17 17/30 Flush 18/30 Write block 429 len 16384 data: 20 21 20 19 19/30 Read block 550 len 24576 20/30 Read block 385 len 32768 21/30 Write block 6 len 16384 data: 10 10 12 13 22/30 Write block 431 len 28672 data: 21 20 16 13 16 15 13 23/30 Write block 296 len 8192 data: 10 8 24/30 Flush 25/30 Flush 26/30 Write block 3 len 20480 data: 9 8 8 11 11 27/30 Write block 340 len 40960 data: 10 12 13 12 9 10 8 6 6 8 28/30 Read block 467 len 16384 29/30 Read block 44 len 24576 30/30 Write block 25 len 40960 data: 17 18 18 17 14 11 9 7 7 9 ---------------------------------------------------------------- Crucible gen:182 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 26 Acked 1025 Write 5 Done Done Done false 27 Acked 1026 Write 10 Done Done Done false 28 Acked 1027 Read 4 Done Done Done false 29 Acked 1028 Read 6 Done Done Done false 30 Acked 1029 Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 4 4 4 12 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1024 1024 1024 Downstairs last five completed: 1024 1023 1022 1021 1020 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:03:43.434432155Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1910} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 182 182 182 377 377 377 T T T 1 020-039 182 182 182 374 374 374 T T T 2 040-059 182 182 182 374 374 374 F F F 3 060-079 180 180 180 372 372 372 F F F 4 080-099 180 180 180 372 372 372 F F F 5 100-119 174 174 174 356 356 356 F F F 6 120-139 182 182 182 374 374 374 F F F 7 140-159 182 182 182 376 376 376 F F F 8 160-179 181 181 181 373 373 373 F F F 9 180-199 180 180 180 372 372 372 F F F 10 200-219 182 182 182 375 375 375 F F F 11 220-239 180 180 180 372 372 372 F F F 12 240-259 174 174 174 356 356 356 F F F 13 260-279 176 176 176 362 362 362 F F F 14 280-299 182 182 182 377 377 377 F F F 15 300-319 180 180 180 372 372 372 F F F 16 320-339 180 180 180 372 372 372 F F F 17 340-359 172 172 172 351 351 351 T T T 18 360-379 182 182 182 376 376 376 F F F 19 380-399 181 181 181 373 373 373 F F F 20 400-419 176 176 176 362 362 362 F F F 21 420-439 182 182 182 377 377 377 F F F 22 440-459 180 180 180 372 372 372 F F F 23 460-479 180 180 180 372 372 372 F F F 24 480-499 178 178 178 368 368 368 F F F 25 500-519 180 180 180 372 372 372 F F F 26 520-539 182 182 182 374 374 374 F F F 27 540-559 177 177 177 363 363 363 F F F 28 560-579 180 180 180 371 371 371 F F F 29 580-599 175 175 175 361 361 361 F F F Max gen: 182, Max flush: 377 A Difference in extent metadata is expected here On loop 91 Verifying data now No change after verify Loop: 91 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 183 183 183 378 378 378 F F F 1 020-039 183 183 183 378 378 378 F F F 2 040-059 182 182 182 374 374 374 F F F 3 060-079 180 180 180 372 372 372 F F F 4 080-099 180 180 180 372 372 372 F F F 5 100-119 174 174 174 356 356 356 F F F 6 120-139 182 182 182 374 374 374 F F F 7 140-159 182 182 182 376 376 376 F F F 8 160-179 181 181 181 373 373 373 F F F 9 180-199 180 180 180 372 372 372 F F F 10 200-219 182 182 182 375 375 375 F F F 11 220-239 180 180 180 372 372 372 F F F 12 240-259 174 174 174 356 356 356 F F F 13 260-279 176 176 176 362 362 362 F F F 14 280-299 182 182 182 377 377 377 F F F 15 300-319 180 180 180 372 372 372 F F F 16 320-339 180 180 180 372 372 372 F F F 17 340-359 183 183 183 378 378 378 F F F 18 360-379 182 182 182 376 376 376 F F F 19 380-399 181 181 181 373 373 373 F F F 20 400-419 176 176 176 362 362 362 F F F 21 420-439 182 182 182 377 377 377 F F F 22 440-459 180 180 180 372 372 372 F F F 23 460-479 180 180 180 372 372 372 F F F 24 480-499 178 178 178 368 368 368 F F F 25 500-519 180 180 180 372 372 372 F F F 26 520-539 182 182 182 374 374 374 F F F 27 540-559 177 177 177 363 363 363 F F F 28 560-579 180 180 180 371 371 371 F F F 29 580-599 175 175 175 361 361 361 F F F Max gen: 183, Max flush: 378 Begin loop 92 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.635977253Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.636605283Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.636639492Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918} {"msg":"Crucible 8f4a1d24-1d41-447d-a1a0-0ef67c8293af has session id: c0e3f020-a1b3-49a7-984b-60d0c3e5043a","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.636716078Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"Upstairs opts: Upstairs UUID: 8f4a1d24-1d41-447d-a1a0-0ef67c8293af, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.636826643Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"Crucible stats registered with UUID: 8f4a1d24-1d41-447d-a1a0-0ef67c8293af","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.636859452Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:184","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.637160068Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918} {"msg":"8f4a1d24-1d41-447d-a1a0-0ef67c8293af active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.637278202Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.637297661Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"0","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.6373177Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"0","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.637333689Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"1","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.637351199Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"1","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.637375498Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.637394207Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.637407516Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"io task","client":"0","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.637421276Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"io task","client":"1","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.637434755Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"io task","client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:03:57.637569198Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"io task","client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:03:57.637608417Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"io task","client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:03:57.637636875Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.637658344Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.637779679Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"ds_connection connected from Ok(127.0.0.1:35327)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.637814427Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"io task","client":"1","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.637840126Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"io task","client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"ds_connection connected from Ok(127.0.0.1:43584)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.637873154Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"io task","client":"0","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.638327223Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"1","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.638354842Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"1","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.638495535Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"0","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.638514585Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"0","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.638870028Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"1","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.638983212Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"0","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.639369365Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"1","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.639411872Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"1","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.639441521Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.639498299Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"0","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.639524837Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"0","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:03:57.639539866Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.638126155Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"io task","client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"ds_connection connected from Ok(127.0.0.1:51589)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.638289597Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"io task","client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.639701701Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.63973084Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.640444907Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.640978122Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.640999641Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"[0]R flush_numbers[0..12]: [378, 378, 374, 372, 372, 356, 374, 376, 373, 372, 375, 372]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.64101738Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"downstairs","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"[0]R generation[0..12]: [183, 183, 182, 180, 180, 174, 182, 182, 181, 180, 182, 180]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641032219Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"downstairs","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641046828Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"downstairs","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"[1]R flush_numbers[0..12]: [378, 378, 374, 372, 372, 356, 374, 376, 373, 372, 375, 372]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641065827Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"downstairs","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"[1]R generation[0..12]: [183, 183, 182, 180, 180, 174, 182, 182, 181, 180, 182, 180]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641080137Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"downstairs","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641094296Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"downstairs","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"[2]R flush_numbers[0..12]: [378, 378, 374, 372, 372, 356, 374, 376, 373, 372, 375, 372]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641108085Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"downstairs","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"[2]R generation[0..12]: [183, 183, 182, 180, 180, 174, 182, 182, 181, 180, 182, 180]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641121675Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"downstairs","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641135694Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"downstairs","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"Max found gen is 184","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641148674Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"downstairs","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"Generation requested: 184 >= found:184","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641161893Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"downstairs","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"Next flush: 379","v":0,"name":"crucible","level":30,"Wait for a query_work_queue command to finish before sending IO time":"2024-07-31T19:04:07.641174892Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"downstairs","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641191361Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"":"downstairs","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641204721Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.64121668Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"0","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.64123013Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"1","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641243549Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"client":"2","":"downstairs","session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641258849Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641271428Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"8f4a1d24-1d41-447d-a1a0-0ef67c8293af is now active with session: c0e3f020-a1b3-49a7-984b-60d0c3e5043a","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641284207Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641296107Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918,"session_id":"c0e3f020-a1b3-49a7-984b-60d0c3e5043a"} {"msg":"The guest has finished waiting for activation with:184","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:07.641308296Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 15 len 28672 data: 9 9 11 10 11 14 15 02/30 Read block 430 len 40960 03/30 Read block 414 len 40960 04/30 Write block 5 len 12288 data: 9 12 12 05/30 Write block 585 len 32768 data: 10 11 9 11 11 15 13 13 06/30 Write block 357 len 40960 data: 18 20 17 15 13 13 13 15 13 13 07/30 Write block 448 len 20480 data: 18 17 21 19 21 08/30 Read block 134 len 32768 09/30 Read block 537 len 20480 10/30 Read block 168 len 32768 11/30 Read block 174 len 20480 12/30 Read block 389 len 40960 13/30 Write block 85 len 12288 data: 21 18 14 14/30 Read block 462 len 36864 15/30 Read block 581 len 16384 16/30 Read block 112 len 36864 17/30 Write block 414 len 8192 data: 13 12 18/30 Read block 222 len 8192 19/30 Write block 377 len 40960 data: 16 19 19 16 16 16 18 21 17 17 20/30 Read block 27 len 24576 21/30 Read block 43 len 32768 22/30 Read block 272 len 40960 23/30 Write block 478 len 36864 data: 13 13 11 12 13 14 18 17 13 24/30 Read block 532 len 16384 25/30 Read block 340 len 12288 26/30 Write block 211 len 28672 data: 10 12 10 12 13 13 11 27/30 Write block 502 len 32768 data: 15 15 15 17 16 19 17 13 28/30 Write block 67 len 4096 data: 8 29/30 Write block 184 len 36864 data: 15 17 20 22 21 19 18 18 13 30/30 Write block 36 len 36864 data: 15 14 15 15 8 7 8 9 10 ---------------------------------------------------------------- Crucible gen:184 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 9 Done Done Done false 25 Acked 1024 Read 4 Done Done Done false 26 Acked 1025 Read 3 Done Done Done false 27 Acked 1026 Write 7 Done Done Done false 28 Acked 1027 Write 8 Done Done Done false 29 Acked 1028 Write 1 Done Done Done false 30 Acked 1029 Write 9 Done Done Done false 31 Acked 1030 Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:04:07.666763228Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1918} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 184 184 184 379 379 379 F F F 1 020-039 184 184 184 379 379 379 T T T 2 040-059 182 182 182 374 374 374 T T T 3 060-079 180 180 180 372 372 372 T T T 4 080-099 184 184 184 379 379 379 F F F 5 100-119 174 174 174 356 356 356 F F F 6 120-139 182 182 182 374 374 374 F F F 7 140-159 182 182 182 376 376 376 F F F 8 160-179 181 181 181 373 373 373 F F F 9 180-199 180 180 180 372 372 372 T T T 10 200-219 182 182 182 375 375 375 T T T 11 220-239 180 180 180 372 372 372 F F F 12 240-259 174 174 174 356 356 356 F F F 13 260-279 176 176 176 362 362 362 F F F 14 280-299 182 182 182 377 377 377 F F F 15 300-319 180 180 180 372 372 372 F F F 16 320-339 180 180 180 372 372 372 F F F 17 340-359 184 184 184 379 379 379 F F F 18 360-379 184 184 184 379 379 379 F F F 19 380-399 184 184 184 379 379 379 F F F 20 400-419 184 184 184 379 379 379 F F F 21 420-439 182 182 182 377 377 377 F F F 22 440-459 184 184 184 379 379 379 F F F 23 460-479 180 180 180 372 372 372 T T T 24 480-499 178 178 178 368 368 368 T T T 25 500-519 180 180 180 372 372 372 T T T 26 520-539 182 182 182 374 374 374 F F F 27 540-559 177 177 177 363 363 363 F F F 28 560-579 180 180 180 371 371 371 F F F 29 580-599 184 184 184 379 379 379 F F F Max gen: 184, Max flush: 379 A Difference in extent metadata is expected here On loop 92 Verifying data now No change after verify Loop: 92 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 184 184 184 379 379 379 F F F 1 020-039 185 185 185 380 380 380 F F F 2 040-059 185 185 185 380 380 380 F F F 3 060-079 185 185 185 380 380 380 F F F 4 080-099 184 184 184 379 379 379 F F F 5 100-119 174 174 174 356 356 356 F F F 6 120-139 182 182 182 374 374 374 F F F 7 140-159 182 182 182 376 376 376 F F F 8 160-179 181 181 181 373 373 373 F F F 9 180-199 185 185 185 380 380 380 F F F 10 200-219 185 185 185 380 380 380 F F F 11 220-239 180 180 180 372 372 372 F F F 12 240-259 174 174 174 356 356 356 F F F 13 260-279 176 176 176 362 362 362 F F F 14 280-299 182 182 182 377 377 377 F F F 15 300-319 180 180 180 372 372 372 F F F 16 320-339 180 180 180 372 372 372 F F F 17 340-359 184 184 184 379 379 379 F F F 18 360-379 184 184 184 379 379 379 F F F 19 380-399 184 184 184 379 379 379 F F F 20 400-419 184 184 184 379 379 379 F F F 21 420-439 182 182 182 377 377 377 F F F 22 440-459 184 184 184 379 379 379 F F F 23 460-479 185 185 185 380 380 380 F F F 24 480-499 185 185 185 380 380 380 F F F 25 500-519 185 185 185 380 380 380 F F F 26 520-539 182 182 182 374 374 374 F F F 27 540-559 177 177 177 363 363 363 F F F 28 560-579 180 180 180 371 371 371 F F F 29 580-599 184 184 184 379 379 379 F F F Max gen: 185, Max flush: 380 Begin loop 93 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.943197101Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.943775374Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.943810072Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926} {"msg":"Crucible 12ea52f6-7ac6-4dfc-9b46-4194684df5af has session id: 8dce9ee5-09b5-4ad9-ab03-9fd463438f9c","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.943891038Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"Upstairs opts: Upstairs UUID: 12ea52f6-7ac6-4dfc-9b46-4194684df5af, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.943927896Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"Crucible stats registered with UUID: 12ea52f6-7ac6-4dfc-9b46-4194684df5af","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.943951525Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:186","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.9442814Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926} {"msg":"12ea52f6-7ac6-4dfc-9b46-4194684df5af active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.944324938Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.944340687Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.944357996Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.944371886Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"1","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.944385385Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"1","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.944398485Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"2","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.944417264Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"2","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.944441672Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"io task","client":"2","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.944458932Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"io task","client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.944475971Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"io task","client":"1","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"ds_connection connected from Ok(127.0.0.1:57951)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.944628304Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"io task","client":"1","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"ds_connection connected from Ok(127.0.0.1:60638)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.944665352Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"io task","client":"2","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:04:21.944693591Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"io task","client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:04:21.944720559Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"io task","client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:04:21.944749768Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.944774517Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.944798486Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.944822385Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"io task","client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.945382799Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"2","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.945409967Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"2","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.945424657Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"1","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.945438136Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"1","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.945933883Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"1","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.946037048Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"2","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.946406401Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"1","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.946445819Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"1","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.946474608Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.946536255Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"2","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.946562184Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"2","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:21.946576643Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.945552704Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"io task","client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"ds_connection connected from Ok(127.0.0.1:48795)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.94565361Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"io task","client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.946879532Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.946907371Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.947449826Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.947964392Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.947983791Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"[0]R flush_numbers[0..12]: [379, 380, 380, 380, 379, 356, 374, 376, 373, 380, 380, 372]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.94799809Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"downstairs","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"[0]R generation[0..12]: [184, 185, 185, 185, 184, 174, 182, 182, 181, 185, 185, 180]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.94801233Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"downstairs","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948030869Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"downstairs","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"[1]R flush_numbers[0..12]: [379, 380, 380, 380, 379, 356, 374, 376, 373, 380, 380, 372]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948045088Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"downstairs","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"[1]R generation[0..12]: [184, 185, 185, 185, 184, 174, 182, 182, 181, 185, 185, 180]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948059017Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"downstairs","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948073057Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"downstairs","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"[2]R flush_numbers[0..12]: [379, 380, 380, 380, 379, 356, 374, 376, 373, 380, 380, 372]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948086716Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"downstairs","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"[2]R generation[0..12]: [184, 185, 185, 185, 184, 174, 182, 182, 181, 185, 185, 180]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948100575Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"downstairs","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948155643Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"downstairs","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"Max found gen is 186","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948173852Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"downstairs","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"Generation requested: 186 >= found:186","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948187322Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"downstairs","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"Next flush: 381","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948200271Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"downstairs","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.94821311Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"":"downstairs","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.94822627Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948238159Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"0","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2024-07-31T19:04:31.948251468Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"1","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948270947Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"client":"2","":"downstairs","session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948284327Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948296416Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"12ea52f6-7ac6-4dfc-9b46-4194684df5af is now active with session: 8dce9ee5-09b5-4ad9-ab03-9fd463438f9c","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948309186Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948321015Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926,"session_id":"8dce9ee5-09b5-4ad9-ab03-9fd463438f9c"} {"msg":"The guest has finished waiting for activation with:186","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:31.948333135Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 550 len 16384 data: 14 17 15 16 02/30 Write block 509 len 24576 data: 14 15 17 13 14 17 03/30 Write block 567 len 16384 data: 18 17 13 15 04/30 Write block 561 len 36864 data: 12 15 18 19 19 20 19 18 14 05/30 Read block 90 len 16384 06/30 Write block 102 len 8192 data: 14 11 07/30 Flush 08/30 Write block 157 len 32768 data: 11 11 7 7 10 6 8 11 09/30 Read block 100 len 32768 10/30 Read block 478 len 32768 11/30 Flush 12/30 Write block 408 len 32768 data: 15 13 12 10 10 12 14 13 13/30 Write block 433 len 12288 data: 17 14 17 14/30 Flush 15/30 Flush 16/30 Write block 92 len 16384 data: 14 13 11 11 17/30 Write block 47 len 28672 data: 14 13 14 13 11 13 13 18/30 Read block 58 len 4096 19/30 Read block 455 len 28672 20/30 Write block 398 len 28672 data: 18 16 15 15 15 14 17 21/30 Read block 65 len 36864 22/30 Read block 554 len 20480 23/30 Flush 24/30 Write block 200 len 40960 data: 13 14 13 12 14 13 14 15 15 12 25/30 Write block 559 len 20480 data: 11 11 13 16 19 26/30 Flush 27/30 Write block 329 len 40960 data: 13 15 15 14 15 17 16 15 17 14 28/30 Read block 251 len 36864 29/30 Write block 236 len 40960 data: 19 16 18 16 16 14 13 11 11 12 30/30 Write block 547 len 8192 data: 12 14 ---------------------------------------------------------------- Crucible gen:186 GIO:true work queues: Upstairs:0 downstairs:4 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 28 Acked 1027 Write 10 Done Done Done false 29 Acked 1028 Read 9 Done Done Done false 30 Acked 1029 Write 10 Sent Sent Sent false 31 Acked 1030 Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 2 2 2 6 Done 2 2 2 6 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1026 1026 1026 Downstairs last five completed: 1026 1025 1024 1023 1022 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:04:31.979978558Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1926} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 184 184 184 379 379 379 F F F 1 020-039 185 185 185 380 380 380 F F F 2 040-059 186 186 186 385 385 385 F F F 3 060-079 185 185 185 380 380 380 F F F 4 080-099 186 186 186 385 385 385 F F F 5 100-119 186 186 186 381 381 381 F F F 6 120-139 182 182 182 374 374 374 F F F 7 140-159 186 186 186 382 382 382 F F F 8 160-179 186 186 186 382 382 382 F F F 9 180-199 185 185 185 380 380 380 F F F 10 200-219 186 186 186 387 387 387 F F F 11 220-239 180 180 180 372 372 372 T T T 12 240-259 174 174 174 356 356 356 T T T 13 260-279 176 176 176 362 362 362 F F F 14 280-299 182 182 182 377 377 377 F F F 15 300-319 180 180 180 372 372 372 F F F 16 320-339 180 180 180 372 372 372 T T T 17 340-359 184 184 184 379 379 379 F F F 18 360-379 184 184 184 379 379 379 F F F 19 380-399 186 186 186 385 385 385 F F F 20 400-419 186 186 186 385 385 385 F F F 21 420-439 186 186 186 383 383 383 F F F 22 440-459 184 184 184 379 379 379 F F F 23 460-479 185 185 185 380 380 380 F F F 24 480-499 185 185 185 380 380 380 F F F 25 500-519 186 186 186 381 381 381 F F F 26 520-539 182 182 182 374 374 374 F F F 27 540-559 186 186 186 387 387 387 T T T 28 560-579 186 186 186 387 387 387 F F F 29 580-599 184 184 184 379 379 379 F F F Max gen: 186, Max flush: 387 A Difference in extent metadata is expected here On loop 93 Verifying data now No change after verify Loop: 93 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 184 184 184 379 379 379 F F F 1 020-039 185 185 185 380 380 380 F F F 2 040-059 186 186 186 385 385 385 F F F 3 060-079 185 185 185 380 380 380 F F F 4 080-099 186 186 186 385 385 385 F F F 5 100-119 186 186 186 381 381 381 F F F 6 120-139 182 182 182 374 374 374 F F F 7 140-159 186 186 186 382 382 382 F F F 8 160-179 186 186 186 382 382 382 F F F 9 180-199 185 185 185 380 380 380 F F F 10 200-219 186 186 186 387 387 387 F F F 11 220-239 187 187 187 388 388 388 F F F 12 240-259 187 187 187 388 388 388 F F F 13 260-279 176 176 176 362 362 362 F F F 14 280-299 182 182 182 377 377 377 F F F 15 300-319 180 180 180 372 372 372 F F F 16 320-339 187 187 187 388 388 388 F F F 17 340-359 184 184 184 379 379 379 F F F 18 360-379 184 184 184 379 379 379 F F F 19 380-399 186 186 186 385 385 385 F F F 20 400-419 186 186 186 385 385 385 F F F 21 420-439 186 186 186 383 383 383 F F F 22 440-459 184 184 184 379 379 379 F F F 23 460-479 185 185 185 380 380 380 F F F 24 480-499 185 185 185 380 380 380 F F F 25 500-519 186 186 186 381 381 381 F F F 26 520-539 182 182 182 374 374 374 F F F 27 540-559 187 187 187 388 388 388 F F F 28 560-579 186 186 186 387 387 387 F F F 29 580-599 184 184 184 379 379 379 F F F Max gen: 187, Max flush: 388 Begin loop 94 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.196436251Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.196809204Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.196837762Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934} {"msg":"Crucible e0cc738e-71cc-40db-accb-12173cf3a58e has session id: 622a0589-511c-4416-a435-2d4534cdfb92","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.197214695Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"Upstairs opts: Upstairs UUID: e0cc738e-71cc-40db-accb-12173cf3a58e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.197255003Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"Crucible stats registered with UUID: e0cc738e-71cc-40db-accb-12173cf3a58e","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.197272752Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:188","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.197565588Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934} {"msg":"e0cc738e-71cc-40db-accb-12173cf3a58e active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.197649784Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.197668124Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"0","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.197687903Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"0","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.197701622Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"1","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.197715191Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"1","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.19773157Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.197760439Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.197777388Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"io task","client":"0","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.197791348Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"io task","client":"1","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.197809457Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"io task","client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"ds_connection connected from Ok(127.0.0.1:46565)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.198025037Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"io task","client":"1","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:04:46.198108503Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"io task","client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"ds_connection connected from Ok(127.0.0.1:45924)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.198139571Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"io task","client":"0","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:04:46.19816619Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"io task","client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:04:46.198191619Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.198211708Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.198232767Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.198403349Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"io task","client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.19882042Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"1","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.198847809Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"1","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.198863338Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"0","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.198877517Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"0","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.199478859Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"1","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.199512018Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"0","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.199992475Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"0","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.200055922Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"0","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.200079651Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.2000993Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"1","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.200120949Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"1","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:46.200142738Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.199445084Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"io task","client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"ds_connection connected from Ok(127.0.0.1:48867)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.199538009Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"io task","client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.200756433Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.200793481Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.201286068Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.201827123Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.201848691Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"[0]R flush_numbers[0..12]: [379, 380, 385, 380, 385, 381, 374, 382, 382, 380, 387, 388]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.201864151Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"downstairs","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"[0]R generation[0..12]: [184, 185, 186, 185, 186, 186, 182, 186, 186, 185, 186, 187]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.20187884Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"downstairs","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.201898899Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"downstairs","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"[1]R flush_numbers[0..12]: [379, 380, 385, 380, 385, 381, 374, 382, 382, 380, 387, 388]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.201913388Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"downstairs","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"[1]R generation[0..12]: [184, 185, 186, 185, 186, 186, 182, 186, 186, 185, 186, 187]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.201927328Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"downstairs","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.201941487Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"downstairs","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"[2]R flush_numbers[0..12]: [379, 380, 385, 380, 385, 381, 374, 382, 382, 380, 387, 388]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.201955297Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"downstairs","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"[2]R generation[0..12]: [184, 185, 186, 185, 186, 186, 182, 186, 186, 185, 186, 187]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.201969116Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"downstairs","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.201983115Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"downstairs","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"Max found gen is 188","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.201996075Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"downstairs","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"Generation requested: 188 >= found:188","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.202009354Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"downstairs","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"Next flush: 389","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.202022243Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"downstairs","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.202035103Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"":"downstairs","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T19:04:56.202048252Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.202062972Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"0","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.202076371Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"1","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.2020916Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"client":"2","":"downstairs","session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.20210517Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.202117639Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"e0cc738e-71cc-40db-accb-12173cf3a58e is now active with session: 622a0589-511c-4416-a435-2d4534cdfb92","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.202130528Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.202142318Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934,"session_id":"622a0589-511c-4416-a435-2d4534cdfb92"} {"msg":"The guest has finished waiting for activation with:188","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:04:56.202154387Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 365 len 20480 data: 14 14 15 17 17 02/30 Read block 462 len 36864 03/30 Write block 61 len 4096 data: 8 04/30 Flush 05/30 Write block 282 len 24576 data: 12 11 12 13 10 11 06/30 Write block 475 len 40960 data: 16 16 17 14 14 12 13 14 15 19 07/30 Flush 08/30 Write block 119 len 8192 data: 14 14 09/30 Write block 468 len 32768 data: 13 14 15 17 16 15 16 17 10/30 Write block 14 len 8192 data: 7 10 11/30 Read block 405 len 16384 12/30 Write block 227 len 32768 data: 14 14 14 14 13 17 14 16 13/30 Write block 234 len 12288 data: 17 18 20 14/30 Read block 256 len 8192 15/30 Read block 414 len 20480 16/30 Write block 200 len 36864 data: 14 15 14 13 15 14 15 16 16 17/30 Flush 18/30 Write block 110 len 40960 data: 12 11 11 11 11 13 13 13 13 15 19/30 Write block 164 len 32768 data: 12 13 15 17 18 18 20 19 20/30 Write block 459 len 4096 data: 13 21/30 Write block 518 len 4096 data: 13 22/30 Read block 551 len 24576 23/30 Read block 286 len 8192 24/30 Read block 163 len 28672 25/30 Write block 327 len 24576 data: 12 14 14 16 16 15 26/30 Read block 467 len 28672 27/30 Write block 284 len 20480 data: 13 14 11 12 9 28/30 Write block 262 len 32768 data: 18 16 14 14 13 12 13 13 29/30 Read block 302 len 24576 30/30 Write block 224 len 40960 data: 13 14 12 15 15 15 15 14 18 15 ---------------------------------------------------------------- Crucible gen:188 GIO:true work queues: Upstairs:0 downstairs:13 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 18 Acked 1017 Write 10 Done Done Done false 19 Acked 1018 Write 8 Done Done Done false 20 Acked 1019 Write 1 Done Done Done false 21 Acked 1020 Write 1 Done Done Done false 22 Acked 1021 Read 6 Done Done Done false 23 Acked 1022 Read 2 Done Done Done false 24 Acked 1023 Read 7 Done Done Done false 25 Acked 1024 Write 6 Done Done Done false 26 Acked 1025 Read 7 Done Done Done false 27 Acked 1026 Write 5 Done Done Done false 28 Acked 1027 Write 8 Done Done Done false 29 Acked 1028 Read 6 Done Done Done false 30 Acked 1029 Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 12 12 12 36 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1016 1016 1016 Downstairs last five completed: 1016 1015 1014 1013 1012 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:04:56.229632095Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1934} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 391 391 391 F F F 1 020-039 185 185 185 380 380 380 F F F 2 040-059 186 186 186 385 385 385 F F F 3 060-079 188 188 188 389 389 389 F F F 4 080-099 186 186 186 385 385 385 F F F 5 100-119 188 188 188 391 391 391 T T T 6 120-139 188 188 188 391 391 391 F F F 7 140-159 186 186 186 382 382 382 F F F 8 160-179 186 186 186 382 382 382 T T T 9 180-199 185 185 185 380 380 380 F F F 10 200-219 188 188 188 391 391 391 F F F 11 220-239 188 188 188 391 391 391 T T T 12 240-259 187 187 187 388 388 388 F F F 13 260-279 176 176 176 362 362 362 T T T 14 280-299 188 188 188 390 390 390 T T T 15 300-319 180 180 180 372 372 372 F F F 16 320-339 187 187 187 388 388 388 T T T 17 340-359 184 184 184 379 379 379 F F F 18 360-379 188 188 188 389 389 389 F F F 19 380-399 186 186 186 385 385 385 F F F 20 400-419 186 186 186 385 385 385 F F F 21 420-439 186 186 186 383 383 383 F F F 22 440-459 184 184 184 379 379 379 T T T 23 460-479 188 188 188 391 391 391 F F F 24 480-499 188 188 188 390 390 390 F F F 25 500-519 186 186 186 381 381 381 T T T 26 520-539 182 182 182 374 374 374 F F F 27 540-559 187 187 187 388 388 388 F F F 28 560-579 186 186 186 387 387 387 F F F 29 580-599 184 184 184 379 379 379 F F F Max gen: 188, Max flush: 391 A Difference in extent metadata is expected here On loop 94 Verifying data now No change after verify Loop: 94 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 391 391 391 F F F 1 020-039 185 185 185 380 380 380 F F F 2 040-059 186 186 186 385 385 385 F F F 3 060-079 188 188 188 389 389 389 F F F 4 080-099 186 186 186 385 385 385 F F F 5 100-119 189 189 189 392 392 392 F F F 6 120-139 188 188 188 391 391 391 F F F 7 140-159 186 186 186 382 382 382 F F F 8 160-179 189 189 189 392 392 392 F F F 9 180-199 185 185 185 380 380 380 F F F 10 200-219 188 188 188 391 391 391 F F F 11 220-239 189 189 189 392 392 392 F F F 12 240-259 187 187 187 388 388 388 F F F 13 260-279 189 189 189 392 392 392 F F F 14 280-299 189 189 189 392 392 392 F F F 15 300-319 180 180 180 372 372 372 F F F 16 320-339 189 189 189 392 392 392 F F F 17 340-359 184 184 184 379 379 379 F F F 18 360-379 188 188 188 389 389 389 F F F 19 380-399 186 186 186 385 385 385 F F F 20 400-419 186 186 186 385 385 385 F F F 21 420-439 186 186 186 383 383 383 F F F 22 440-459 189 189 189 392 392 392 F F F 23 460-479 188 188 188 391 391 391 F F F 24 480-499 188 188 188 390 390 390 F F F 25 500-519 189 189 189 392 392 392 F F F 26 520-539 182 182 182 374 374 374 F F F 27 540-559 187 187 187 388 388 388 F F F 28 560-579 186 186 186 387 387 387 F F F 29 580-599 184 184 184 379 379 379 F F F Max gen: 189, Max flush: 392 Begin loop 95 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.500531686Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501023633Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501049812Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942} {"msg":"Crucible a0d9db19-301f-4cf9-bb81-02f5c69272b8 has session id: 29f51411-27c0-46cd-b6c5-34e7b01e42b3","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501208234Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"Upstairs opts: Upstairs UUID: a0d9db19-301f-4cf9-bb81-02f5c69272b8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501245213Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"Crucible stats registered with UUID: a0d9db19-301f-4cf9-bb81-02f5c69272b8","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501267041Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:190","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501650054Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942} {"msg":"a0d9db19-301f-4cf9-bb81-02f5c69272b8 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501715591Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501747739Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"0","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501781757Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"0","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501806876Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501831395Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501853404Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"2","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501880063Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"2","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501904952Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"io task","client":"0","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501931061Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"io task","client":"2","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.501956699Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"ds_connection connected from Ok(127.0.0.1:35533)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.502092443Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"io task","client":"0","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"ds_connection connected from Ok(127.0.0.1:51942)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.502131071Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"io task","client":"2","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:05:10.502303733Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:05:10.502343951Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:05:10.50236797Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.502386109Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.502399789Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.502475155Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.502980492Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"2","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.50302086Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"2","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.503036949Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"0","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.503056548Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"0","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.503607512Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"0","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.503730077Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"2","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.50409377Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"0","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.504129568Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"0","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.504153947Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.504207384Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"2","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.504235513Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"2","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:10.504252632Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.503781616Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"ds_connection connected from Ok(127.0.0.1:43405)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.50392017Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.505443759Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.505487497Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506015142Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506571996Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506603715Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"[0]R flush_numbers[0..12]: [391, 380, 385, 389, 385, 392, 391, 382, 392, 380, 391, 392]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506623354Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"downstairs","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"[0]R generation[0..12]: [188, 185, 186, 188, 186, 189, 188, 186, 189, 185, 188, 189]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506641463Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"downstairs","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506659162Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"downstairs","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"[1]R flush_numbers[0..12]: [391, 380, 385, 389, 385, 392, 391, 382, 392, 380, 391, 392]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506676741Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"downstairs","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"[1]R generation[0..12]: [188, 185, 186, 188, 186, 189, 188, 186, 189, 185, 188, 189]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.50669412Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"downstairs","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.5067117Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"downstairs","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"[2]R flush_numbers[0..12]: [391, 380, 385, 389, 385, 392, 391, 382, 392, 380, 391, 392]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506729189Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"downstairs","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"[2]R generation[0..12]: [188, 185, 186, 188, 186, 189, 188, 186, 189, 185, 188, 189]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506746338Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"downstairs","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506763737Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"downstairs","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"Max found gen is 190","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T19:05:20.506779947Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"downstairs","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"Generation requested: 190 >= found:190","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506798556Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"downstairs","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"Next flush: 393","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506811655Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"downstairs","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506824685Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"":"downstairs","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506839954Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506852003Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"0","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506872412Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"1","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506886022Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"client":"2","":"downstairs","session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506899661Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.50691206Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"a0d9db19-301f-4cf9-bb81-02f5c69272b8 is now active with session: 29f51411-27c0-46cd-b6c5-34e7b01e42b3","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.50692523Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506937319Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942,"session_id":"29f51411-27c0-46cd-b6c5-34e7b01e42b3"} {"msg":"The guest has finished waiting for activation with:190","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:20.506949588Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 543 len 20480 data: 11 14 13 12 13 02/30 Read block 585 len 36864 03/30 Write block 408 len 16384 data: 16 14 13 11 04/30 Write block 185 len 32768 data: 18 21 23 22 20 19 19 14 05/30 Write block 224 len 40960 data: 14 15 13 16 16 16 16 15 19 16 06/30 Read block 295 len 32768 07/30 Write block 158 len 28672 data: 12 8 8 11 7 9 13 08/30 Read block 235 len 20480 09/30 Read block 21 len 16384 10/30 Write block 107 len 32768 data: 11 13 11 13 12 12 12 12 11/30 Write block 249 len 16384 data: 16 15 13 14 12/30 Write block 245 len 36864 data: 13 11 13 15 17 16 14 15 13 13/30 Read block 561 len 28672 14/30 Read block 484 len 8192 15/30 Read block 586 len 36864 16/30 Read block 266 len 36864 17/30 Read block 578 len 32768 18/30 Write block 160 len 12288 data: 9 12 8 19/30 Write block 300 len 4096 data: 9 20/30 Read block 262 len 40960 21/30 Flush 22/30 Write block 338 len 12288 data: 15 10 11 23/30 Read block 151 len 4096 24/30 Read block 328 len 32768 25/30 Write block 566 len 12288 data: 21 20 19 26/30 Read block 30 len 20480 27/30 Write block 509 len 24576 data: 15 16 18 14 15 18 28/30 Read block 380 len 40960 29/30 Read block 226 len 16384 30/30 Read block 393 len 24576 ---------------------------------------------------------------- Crucible gen:190 GIO:true work queues: Upstairs:0 downstairs:9 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 23 Acked 1022 Write 3 Done Done Done false 24 Acked 1023 Read 1 Done Done Done false 25 Acked 1024 Read 8 Done Done Done false 26 Acked 1025 Write 3 Done Done Done false 27 Acked 1026 Read 5 Done Done Done false 28 Acked 1027 Write 6 Done Done Done false 29 Acked 1028 Read 10 Done Done Done false 30 Acked 1029 Read 4 Done Done Done false 31 Acked 1030 Read 6 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 1 0 1 Done 9 8 9 26 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1021 1021 1021 Downstairs last five completed: 1021 1020 1019 1018 1017 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:05:20.533042581Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1942} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 391 391 391 F F F 1 020-039 185 185 185 380 380 380 F F F 2 040-059 186 186 186 385 385 385 F F F 3 060-079 188 188 188 389 389 389 F F F 4 080-099 186 186 186 385 385 385 F F F 5 100-119 190 190 190 393 393 393 F F F 6 120-139 188 188 188 391 391 391 F F F 7 140-159 190 190 190 393 393 393 F F F 8 160-179 190 190 190 394 394 394 F F F 9 180-199 190 190 190 393 393 393 F F F 10 200-219 188 188 188 391 391 391 F F F 11 220-239 190 190 190 393 393 393 F F F 12 240-259 190 190 190 393 393 393 F F F 13 260-279 189 189 189 392 392 392 F F F 14 280-299 189 189 189 392 392 392 F F F 15 300-319 190 190 190 394 394 394 F F F 16 320-339 189 189 189 392 392 392 T T T 17 340-359 184 184 184 379 379 379 T T T 18 360-379 188 188 188 389 389 389 F F F 19 380-399 186 186 186 385 385 385 F F F 20 400-419 190 190 190 393 393 393 F F F 21 420-439 186 186 186 383 383 383 F F F 22 440-459 189 189 189 392 392 392 F F F 23 460-479 188 188 188 391 391 391 F F F 24 480-499 188 188 188 390 390 390 F F F 25 500-519 189 189 189 392 392 392 T T T 26 520-539 182 182 182 374 374 374 F F F 27 540-559 190 190 190 393 393 393 F F F 28 560-579 186 186 186 387 387 387 T T T 29 580-599 184 184 184 379 379 379 F F F Max gen: 190, Max flush: 394 A Difference in extent metadata is expected here On loop 95 Verifying data now No change after verify Loop: 95 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 391 391 391 F F F 1 020-039 185 185 185 380 380 380 F F F 2 040-059 186 186 186 385 385 385 F F F 3 060-079 188 188 188 389 389 389 F F F 4 080-099 186 186 186 385 385 385 F F F 5 100-119 190 190 190 393 393 393 F F F 6 120-139 188 188 188 391 391 391 F F F 7 140-159 190 190 190 393 393 393 F F F 8 160-179 190 190 190 394 394 394 F F F 9 180-199 190 190 190 393 393 393 F F F 10 200-219 188 188 188 391 391 391 F F F 11 220-239 190 190 190 393 393 393 F F F 12 240-259 190 190 190 393 393 393 F F F 13 260-279 189 189 189 392 392 392 F F F 14 280-299 189 189 189 392 392 392 F F F 15 300-319 190 190 190 394 394 394 F F F 16 320-339 191 191 191 395 395 395 F F F 17 340-359 191 191 191 395 395 395 F F F 18 360-379 188 188 188 389 389 389 F F F 19 380-399 186 186 186 385 385 385 F F F 20 400-419 190 190 190 393 393 393 F F F 21 420-439 186 186 186 383 383 383 F F F 22 440-459 189 189 189 392 392 392 F F F 23 460-479 188 188 188 391 391 391 F F F 24 480-499 188 188 188 390 390 390 F F F 25 500-519 191 191 191 395 395 395 F F F 26 520-539 182 182 182 374 374 374 F F F 27 540-559 190 190 190 393 393 393 F F F 28 560-579 191 191 191 395 395 395 F F F 29 580-599 184 184 184 379 379 379 F F F Max gen: 191, Max flush: 395 Begin loop 96 Downstairs to restart: 0 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.753500956Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.753909367Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.753944705Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950} {"msg":"Crucible d4b4369b-39ee-437b-8f7e-0bb349177042 has session id: eec046b9-f092-47d8-9925-e8eca5d81d5d","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.754177544Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"Upstairs opts: Upstairs UUID: d4b4369b-39ee-437b-8f7e-0bb349177042, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.754221992Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"Crucible stats registered with UUID: d4b4369b-39ee-437b-8f7e-0bb349177042","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.754245551Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:192","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.754573686Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950} {"msg":"d4b4369b-39ee-437b-8f7e-0bb349177042 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.754603764Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.754617694Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.754642163Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.754667641Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"1","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.75469318Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"1","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.754722309Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"2","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.754746638Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"2","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.754769577Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"io task","client":"2","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.754794045Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"io task","client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.754810625Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"io task","client":"1","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:05:34.755000756Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"io task","client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"ds_connection connected from Ok(127.0.0.1:44045)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.755040454Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"io task","client":"1","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:05:34.755069913Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"io task","client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:05:34.755097251Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.755161108Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.755189027Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.755214276Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"io task","client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"ds_connection connected from Ok(127.0.0.1:46625)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.755311891Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"io task","client":"2","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.755726882Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"1","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.755751301Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"1","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.755859966Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"2","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.755889784Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"2","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.756262297Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"1","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.756373712Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"2","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.756702427Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"1","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.756724466Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"1","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.756738065Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.756915077Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"2","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.756934196Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"2","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:34.756947475Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.756920389Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"io task","client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"ds_connection connected from Ok(127.0.0.1:48709)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.757082801Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"io task","client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.758155201Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.75818103Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.758642278Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759177533Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759204752Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"[0]R flush_numbers[0..12]: [391, 380, 385, 389, 385, 393, 391, 393, 394, 393, 391, 393]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759223141Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"downstairs","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"[0]R generation[0..12]: [188, 185, 186, 188, 186, 190, 188, 190, 190, 190, 188, 190]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.75924077Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"downstairs","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.75925868Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"downstairs","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"[1]R flush_numbers[0..12]: [391, 380, 385, 389, 385, 393, 391, 393, 394, 393, 391, 393]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759275969Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"downstairs","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"[1]R generation[0..12]: [188, 185, 186, 188, 186, 190, 188, 190, 190, 190, 188, 190]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759293248Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"downstairs","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759310667Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"downstairs","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"[2]R flush_numbers[0..12]: [391, 380, 385, 389, 385, 393, 391, 393, 394, 393, 391, 393]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759327716Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"downstairs","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"[2]R generation[0..12]: [188, 185, 186, 188, 186, 190, 188, 190, 190, 190, 188, 190]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759344765Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"downstairs","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759362355Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"downstairs","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"Max found gen is 192","v"Wait for a query_work_queue command to finish before sending IO :0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759382194Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"downstairs","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"Generation requested: 192 >= found:192","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759398813Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"downstairs","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"Next flush: 396","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759415152Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"downstairs","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759435101Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"":"downstairs","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.75945172Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.75946648Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"0","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759482839Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"1","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759499198Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"client":"2","":"downstairs","session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759515508Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759530637Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"d4b4369b-39ee-437b-8f7e-0bb349177042 is now active with session: eec046b9-f092-47d8-9925-e8eca5d81d5d","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759546806Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759561465Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950,"session_id":"eec046b9-f092-47d8-9925-e8eca5d81d5d"} {"msg":"The guest has finished waiting for activation with:192","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:44.759576775Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 57 len 12288 data: 10 10 10 02/30 Write block 539 len 40960 data: 11 13 14 14 12 15 14 13 14 15 03/30 Read block 579 len 36864 04/30 Read block 420 len 16384 05/30 Read block 397 len 24576 06/30 Write block 338 len 32768 data: 16 11 12 13 14 13 10 11 07/30 Write block 488 len 4096 data: 14 08/30 Read block 209 len 12288 09/30 Read block 124 len 12288 10/30 Read block 498 len 4096 11/30 Read block 171 len 40960 12/30 Read block 499 len 32768 13/30 Write block 501 len 8192 data: 12 16 14/30 Read block 418 len 20480 15/30 Read block 581 len 28672 16/30 Read block 144 len 28672 17/30 Flush 18/30 Write block 222 len 40960 data: 15 12 15 16 14 17 17 17 17 16 19/30 Write block 88 len 8192 data: 10 12 20/30 Read block 103 len 32768 21/30 Write block 39 len 36864 data: 16 9 8 9 10 11 12 10 15 22/30 Flush 23/30 Write block 282 len 28672 data: 13 12 14 15 12 13 10 24/30 Read block 10 len 8192 25/30 Read block 177 len 4096 26/30 Read block 425 len 16384 27/30 Read block 158 len 36864 28/30 Read block 487 len 16384 29/30 Write block 396 len 12288 data: 17 18 19 30/30 Read block 318 len 32768 ---------------------------------------------------------------- Crucible gen:192 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 23 Acked 1022 Write 7 Done Done Done false 24 Acked 1023 Read 2 Done Done Done false 25 Acked 1024 Read 1 Done Done Done false 26 Acked 1025 Read 4 Done Done Done false 27 Acked 1026 Read 9 Done Done Done false 28 Acked 1027 Read 4 Done Done Done false 29 Acked 1028 Write 3 Done Done Done false 30 Acked 1029 Read 8 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 8 8 8 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1021 1021 1021 Downstairs last five completed: 1021 1020 1019 1018 1017 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:05:44.786266009Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1950} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 391 391 391 F F F 1 020-039 192 192 192 397 397 397 F F F 2 040-059 192 192 192 397 397 397 F F F 3 060-079 188 188 188 389 389 389 F F F 4 080-099 192 192 192 397 397 397 F F F 5 100-119 190 190 190 393 393 393 F F F 6 120-139 188 188 188 391 391 391 F F F 7 140-159 190 190 190 393 393 393 F F F 8 160-179 190 190 190 394 394 394 F F F 9 180-199 190 190 190 393 393 393 F F F 10 200-219 188 188 188 391 391 391 F F F 11 220-239 192 192 192 397 397 397 F F F 12 240-259 190 190 190 393 393 393 F F F 13 260-279 189 189 189 392 392 392 F F F 14 280-299 189 189 189 392 392 392 T T T 15 300-319 190 190 190 394 394 394 F F F 16 320-339 192 192 192 396 396 396 F F F 17 340-359 192 192 192 396 396 396 F F F 18 360-379 188 188 188 389 389 389 F F F 19 380-399 186 186 186 385 385 385 T T T 20 400-419 190 190 190 393 393 393 F F F 21 420-439 186 186 186 383 383 383 F F F 22 440-459 189 189 189 392 392 392 F F F 23 460-479 188 188 188 391 391 391 F F F 24 480-499 192 192 192 396 396 396 F F F 25 500-519 192 192 192 396 396 396 F F F 26 520-539 192 192 192 396 396 396 F F F 27 540-559 192 192 192 396 396 396 F F F 28 560-579 191 191 191 395 395 395 F F F 29 580-599 184 184 184 379 379 379 F F F Max gen: 192, Max flush: 397 A Difference in extent metadata is expected here On loop 96 Verifying data now No change after verify Loop: 96 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 391 391 391 F F F 1 020-039 192 192 192 397 397 397 F F F 2 040-059 192 192 192 397 397 397 F F F 3 060-079 188 188 188 389 389 389 F F F 4 080-099 192 192 192 397 397 397 F F F 5 100-119 190 190 190 393 393 393 F F F 6 120-139 188 188 188 391 391 391 F F F 7 140-159 190 190 190 393 393 393 F F F 8 160-179 190 190 190 394 394 394 F F F 9 180-199 190 190 190 393 393 393 F F F 10 200-219 188 188 188 391 391 391 F F F 11 220-239 192 192 192 397 397 397 F F F 12 240-259 190 190 190 393 393 393 F F F 13 260-279 189 189 189 392 392 392 F F F 14 280-299 193 193 193 398 398 398 F F F 15 300-319 190 190 190 394 394 394 F F F 16 320-339 192 192 192 396 396 396 F F F 17 340-359 192 192 192 396 396 396 F F F 18 360-379 188 188 188 389 389 389 F F F 19 380-399 193 193 193 398 398 398 F F F 20 400-419 190 190 190 393 393 393 F F F 21 420-439 186 186 186 383 383 383 F F F 22 440-459 189 189 189 392 392 392 F F F 23 460-479 188 188 188 391 391 391 F F F 24 480-499 192 192 192 396 396 396 F F F 25 500-519 192 192 192 396 396 396 F F F 26 520-539 192 192 192 396 396 396 F F F 27 540-559 192 192 192 396 396 396 F F F 28 560-579 191 191 191 395 395 395 F F F 29 580-599 184 184 184 379 379 379 F F F Max gen: 193, Max flush: 398 Begin loop 97 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.978390566Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.978900753Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.978928791Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958} {"msg":"Crucible e75e49c7-ae9f-479f-973d-bbf592d9aed9 has session id: b373a78f-a687-4c10-9302-037b8e16a583","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979058515Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"Upstairs opts: Upstairs UUID: e75e49c7-ae9f-479f-973d-bbf592d9aed9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979107723Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"Crucible stats registered with UUID: e75e49c7-ae9f-479f-973d-bbf592d9aed9","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979131462Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:194","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979440708Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958} {"msg":"e75e49c7-ae9f-479f-973d-bbf592d9aed9 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979471686Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979495445Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"0","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979526264Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"0","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979549732Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"1","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979574541Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"1","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.97960354Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979634619Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979661327Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"io task","client":"0","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979687116Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"io task","client":"1","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979710955Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"io task","client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:05:58.97981044Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"io task","client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:05:58.979849668Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"io task","client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:05:58.979880587Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979905676Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979929835Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"ds_connection connected from Ok(127.0.0.1:45231)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.979952504Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"io task","client":"0","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"ds_connection connected from Ok(127.0.0.1:65394)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.980049509Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"io task","client":"1","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.980084127Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"io task","client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.980855302Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"1","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.98088538Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"1","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.980900609Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"0","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.980914399Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"0","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.98153731Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"1","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.981610996Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"0","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.982109423Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"1","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.982136552Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"1","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.982152131Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.98216464Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"0","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.98217838Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"0","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:05:58.982191699Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.981333592Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"io task","client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"ds_connection connected from Ok(127.0.0.1:48760)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.981675176Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"io task","client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.982637431Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.98266479Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.983188366Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.983709911Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.98373709Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"[0]R flush_numbers[0..12]: [391, 397, 397, 389, 397, 393, 391, 393, 394, 393, 391, 397]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.983756679Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"downstairs","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"[0]R generation[0..12]: [188, 192, 192, 188, 192, 190, 188, 190, 190, 190, 188, 192]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.983774878Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"downstairs","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.983793457Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"downstairs","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"[1]R flush_numbers[0..12]: [391, 397, 397, 389, 397, 393, 391, 393, 394, 393, 391, 397]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.983811306Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"downstairs","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"[1]R generation[0..12]: [188, 192, 192, 188, 192, 190, 188, 190, 190, 190, 188, 192]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.983828845Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"downstairs","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.983846645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"downstairs","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"[2]R flush_numbers[0..12]: [391, 397, 397, 389, 397, 393, 391, 393, 394, 393, 391, 397]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.983863934Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"downstairs","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"[2]R generation[0..12]: [188, 192, 192, 188, 192, 190, 188, 190, 190, 190, 188, 192]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.983881143Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"downstairs","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.983898672Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"downstairs","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"Max found gen is 194","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2024-07-31T19:06:08.983914992Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"downstairs","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"Generation requested: 194 >= found:194","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.983931971Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"downstairs","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"Next flush: 399","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.98395137Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"downstairs","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.983967859Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"":"downstairs","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.983984598Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.983999038Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"0","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.984012617Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"1","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.984026286Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"client":"2","":"downstairs","session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.984039726Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.984052115Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"e75e49c7-ae9f-479f-973d-bbf592d9aed9 is now active with session: b373a78f-a687-4c10-9302-037b8e16a583","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.984064914Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.984076924Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958,"session_id":"b373a78f-a687-4c10-9302-037b8e16a583"} {"msg":"The guest has finished waiting for activation with:194","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:08.984088993Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 240 len 8192 data: 17 15 02/30 Read block 100 len 20480 03/30 Read block 394 len 28672 04/30 Write block 327 len 24576 data: 13 15 15 17 17 16 05/30 Write block 398 len 20480 data: 20 17 16 16 16 06/30 Write block 278 len 32768 data: 15 15 18 14 14 13 15 16 07/30 Read block 310 len 24576 08/30 Write block 481 len 8192 data: 14 15 09/30 Read block 424 len 32768 10/30 Write block 133 len 40960 data: 11 12 13 14 13 12 11 10 12 14 11/30 Write block 360 len 36864 data: 16 14 14 14 16 15 15 16 18 12/30 Flush 13/30 Write block 521 len 12288 data: 15 14 13 14/30 Read block 50 len 16384 15/30 Read block 43 len 16384 16/30 Flush 17/30 Write block 369 len 28672 data: 18 17 21 20 19 15 18 18/30 Read block 437 len 32768 19/30 Read block 150 len 8192 20/30 Read block 90 len 40960 21/30 Write block 103 len 28672 data: 12 10 12 9 12 14 12 22/30 Read block 456 len 16384 23/30 Read block 111 len 24576 24/30 Write block 239 len 40960 data: 17 18 16 14 12 12 14 12 14 16 25/30 Read block 407 len 36864 26/30 Read block 573 len 24576 27/30 Read block 219 len 32768 28/30 Read block 364 len 12288 29/30 Write block 302 len 40960 data: 12 12 12 10 9 11 12 12 12 14 30/30 Write block 160 len 20480 data: 10 13 9 10 14 ---------------------------------------------------------------- Crucible gen:194 GIO:true work queues: Upstairs:0 downstairs:14 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 17 Acked 1016 Write 7 Done Done Done false 18 Acked 1017 Read 8 Done Done Done false 19 Acked 1018 Read 2 Done Done Done false 20 Acked 1019 Read 10 Done Done Done false 21 Acked 1020 Write 7 Done Done Done false 22 Acked 1021 Read 4 Done Done Done false 23 Acked 1022 Read 6 Done Done Done false 24 Acked 1023 Write 10 Done Done Done false 25 Acked 1024 Read 9 Done Done Done false 26 Acked 1025 Read 6 Done Done Done false 27 Acked 1026 Read 8 Done Done Done false 28 Acked 1027 Read 3 Done Done Done false 29 Acked 1028 Write 10 Done Done Done false 30 Acked 1029 Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 1 1 3 Done 13 13 13 39 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1015 1015 1015 Downstairs last five completed: 1015 1014 1013 1012 1011 Upstairs last five completed: 30 29 28 27 26 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:06:09.012963446Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1958} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 391 391 391 F F F 1 020-039 192 192 192 397 397 397 F F F 2 040-059 192 192 192 397 397 397 F F F 3 060-079 188 188 188 389 389 389 F F F 4 080-099 192 192 192 397 397 397 F F F 5 100-119 190 190 190 393 393 393 T T T 6 120-139 194 194 194 399 399 399 F F F 7 140-159 194 194 194 399 399 399 F F F 8 160-179 190 190 190 394 394 394 T T T 9 180-199 190 190 190 393 393 393 F F F 10 200-219 188 188 188 391 391 391 F F F 11 220-239 192 192 192 397 397 397 T T T 12 240-259 194 194 194 399 399 399 T T T 13 260-279 194 194 194 399 399 399 F F F 14 280-299 194 194 194 399 399 399 F F F 15 300-319 190 190 190 394 394 394 T T T 16 320-339 194 194 194 399 399 399 F F F 17 340-359 192 192 192 396 396 396 F F F 18 360-379 194 194 194 399 399 399 T T T 19 380-399 194 194 194 399 399 399 F F F 20 400-419 194 194 194 399 399 399 F F F 21 420-439 186 186 186 383 383 383 F F F 22 440-459 189 189 189 392 392 392 F F F 23 460-479 188 188 188 391 391 391 F F F 24 480-499 194 194 194 399 399 399 F F F 25 500-519 192 192 192 396 396 396 F F F 26 520-539 194 194 194 400 400 400 F F F 27 540-559 192 192 192 396 396 396 F F F 28 560-579 191 191 191 395 395 395 F F F 29 580-599 184 184 184 379 379 379 F F F Max gen: 194, Max flush: 400 A Difference in extent metadata is expected here On loop 97 Verifying data now No change after verify Loop: 97 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 391 391 391 F F F 1 020-039 192 192 192 397 397 397 F F F 2 040-059 192 192 192 397 397 397 F F F 3 060-079 188 188 188 389 389 389 F F F 4 080-099 192 192 192 397 397 397 F F F 5 100-119 195 195 195 401 401 401 F F F 6 120-139 194 194 194 399 399 399 F F F 7 140-159 194 194 194 399 399 399 F F F 8 160-179 195 195 195 401 401 401 F F F 9 180-199 190 190 190 393 393 393 F F F 10 200-219 188 188 188 391 391 391 F F F 11 220-239 195 195 195 401 401 401 F F F 12 240-259 195 195 195 401 401 401 F F F 13 260-279 194 194 194 399 399 399 F F F 14 280-299 194 194 194 399 399 399 F F F 15 300-319 195 195 195 401 401 401 F F F 16 320-339 194 194 194 399 399 399 F F F 17 340-359 192 192 192 396 396 396 F F F 18 360-379 195 195 195 401 401 401 F F F 19 380-399 194 194 194 399 399 399 F F F 20 400-419 194 194 194 399 399 399 F F F 21 420-439 186 186 186 383 383 383 F F F 22 440-459 189 189 189 392 392 392 F F F 23 460-479 188 188 188 391 391 391 F F F 24 480-499 194 194 194 399 399 399 F F F 25 500-519 192 192 192 396 396 396 F F F 26 520-539 194 194 194 400 400 400 F F F 27 540-559 192 192 192 396 396 396 F F F 28 560-579 191 191 191 395 395 395 F F F 29 580-599 184 184 184 379 379 379 F F F Max gen: 195, Max flush: 401 Begin loop 98 Downstairs to restart: 1 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.259162651Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.259582901Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.25960557Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966} {"msg":"Crucible 93a07de6-0acf-4675-ac16-7372373f1395 has session id: 0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.259981363Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"Upstairs opts: Upstairs UUID: 93a07de6-0acf-4675-ac16-7372373f1395, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.260014451Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"Crucible stats registered with UUID: 93a07de6-0acf-4675-ac16-7372373f1395","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.26003001Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:196","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.260448401Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966} {"msg":"93a07de6-0acf-4675-ac16-7372373f1395 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.26046978Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.260482439Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"0","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.260505828Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"0","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.260520588Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.260534377Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.260547586Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"2","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.260561036Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"2","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.260574305Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"io task","client":"0","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.260587994Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"io task","client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.260601614Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"io task","client":"2","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"ds_connection connected from Ok(127.0.0.1:45276)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.260789365Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"io task","client":"0","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"ds_connection connected from Ok(127.0.0.1:44487)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.260831003Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"io task","client":"2","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:06:23.261032424Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"io task","client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:06:23.261072162Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"io task","client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:06:23.261168877Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.261187176Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.261200746Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.261279742Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"io task","client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.261640105Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"2","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.261662634Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"2","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.261803298Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"0","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.261821277Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"0","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.262268096Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"2","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.26240197Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"0","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.262901197Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"2","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.262925035Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"2","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.262938705Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.262950974Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"0","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.262964343Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"0","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:23.262977573Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.263245043Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"io task","client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"ds_connection connected from Ok(127.0.0.1:54046)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.263677373Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"io task","client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.264727414Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.264767662Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.265367234Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.265924278Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.265949437Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"[0]R flush_numbers[0..12]: [391, 397, 397, 389, 397, 401, 399, 399, 401, 393, 391, 401]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.265967556Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"downstairs","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"[0]R generation[0..12]: [188, 192, 192, 188, 192, 195, 194, 194, 195, 190, 188, 195]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.265985215Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"downstairs","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266003325Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"downstairs","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"[1]R flush_numbers[0..12]: [391, 397, 397, 389, 397, 401, 399, 399, 401, 393, 391, 401]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266020584Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"downstairs","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"[1]R generation[0..12]: [188, 192, 192, 188, 192, 195, 194, 194, 195, 190, 188, 195]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266037783Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"downstairs","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266055292Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"downstairs","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"[2]R flush_numbers[0..12]: [391, 397, 397, 389, 397, 401, 399, 399, 401, 393, 391, 401]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266072491Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"downstairs","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"[2]R generation[0..12]: [188, 192, 192, 188, 192, 195, 194, 194, 195, 190, 188, 195]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.26608989Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"downstairs","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.26610767Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"downstairs","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} Wait for a query_work_queue command to finish before sending IO {"msg":"Max found gen is 196","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266128819Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"downstairs","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"Generation requested: 196 >= found:196","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266142968Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"downstairs","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"Next flush: 402","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266156017Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"downstairs","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266168707Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"":"downstairs","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266182156Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266194545Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"0","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266207975Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"1","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266221334Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"client":"2","":"downstairs","session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266235034Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266247383Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"93a07de6-0acf-4675-ac16-7372373f1395 is now active with session: 0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266260222Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266272002Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966,"session_id":"0b7ddb2f-43a7-40ae-8573-8d48e6aaaa53"} {"msg":"The guest has finished waiting for activation with:196","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:33.266284191Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 318 len 16384 data: 15 13 13 11 02/30 Write block 168 len 40960 data: 19 19 21 20 17 16 14 15 12 10 03/30 Read block 248 len 32768 04/30 Write block 267 len 12288 data: 13 14 14 05/30 Write block 52 len 28672 data: 14 14 11 10 12 11 11 06/30 Write block 220 len 20480 data: 13 14 16 13 16 07/30 Read block 141 len 28672 08/30 Write block 159 len 32768 data: 9 11 14 10 11 15 14 16 09/30 Read block 135 len 28672 10/30 Read block 88 len 36864 11/30 Write block 207 len 16384 data: 17 17 13 11 12/30 Flush 13/30 Write block 81 len 8192 data: 24 20 14/30 Read block 91 len 24576 15/30 Read block 481 len 16384 16/30 Write block 326 len 4096 data: 9 17/30 Read block 489 len 12288 18/30 Write block 567 len 28672 data: 21 20 15 16 18 17 20 19/30 Write block 565 len 12288 data: 20 22 22 20/30 Write block 450 len 24576 data: 22 20 22 17 14 15 21/30 Write block 423 len 12288 data: 16 13 17 22/30 Flush 23/30 Write block 421 len 24576 data: 21 19 17 14 18 17 24/30 Write block 28 len 20480 data: 18 15 12 10 8 25/30 Read block 578 len 8192 26/30 Write block 442 len 4096 data: 17 27/30 Read block 68 len 8192 28/30 Write block 456 len 12288 data: 13 11 11 29/30 Write block 95 len 32768 data: 12 10 12 12 14 17 17 15 30/30 Read block 284 len 8192 ---------------------------------------------------------------- Crucible gen:196 GIO:true work queues: Upstairs:0 downstairs:8 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 24 Acked 1023 Write 6 Done Done Done false 25 Acked 1024 Write 5 Done Done Done false 26 Acked 1025 Read 2 Done Done Done false 27 Acked 1026 Write 1 Done Done Done false 28 Acked 1027 Read 2 Done Done Done false 29 Acked 1028 Write 3 Done Done Done false 30 Acked 1029 Write 8 Done Done Done false 31 Acked 1030 Read 2 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 8 8 8 24 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1022 1022 1022 Downstairs last five completed: 1022 1021 1020 1019 1018 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:06:33.292079258Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1966} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 391 391 391 F F F 1 020-039 192 192 192 397 397 397 T T T 2 040-059 196 196 196 402 402 402 F F F 3 060-079 188 188 188 389 389 389 F F F 4 080-099 196 196 196 403 403 403 T T T 5 100-119 195 195 195 401 401 401 T T T 6 120-139 194 194 194 399 399 399 F F F 7 140-159 196 196 196 402 402 402 F F F 8 160-179 196 196 196 402 402 402 F F F 9 180-199 190 190 190 393 393 393 F F F 10 200-219 196 196 196 402 402 402 F F F 11 220-239 196 196 196 402 402 402 F F F 12 240-259 195 195 195 401 401 401 F F F 13 260-279 196 196 196 402 402 402 F F F 14 280-299 194 194 194 399 399 399 F F F 15 300-319 196 196 196 402 402 402 F F F 16 320-339 196 196 196 404 404 404 F F F 17 340-359 192 192 192 396 396 396 F F F 18 360-379 195 195 195 401 401 401 F F F 19 380-399 194 194 194 399 399 399 F F F 20 400-419 194 194 194 399 399 399 F F F 21 420-439 196 196 196 404 404 404 T T T 22 440-459 196 196 196 404 404 404 T T T 23 460-479 188 188 188 391 391 391 F F F 24 480-499 194 194 194 399 399 399 F F F 25 500-519 192 192 192 396 396 396 F F F 26 520-539 194 194 194 400 400 400 F F F 27 540-559 192 192 192 396 396 396 F F F 28 560-579 196 196 196 404 404 404 F F F 29 580-599 184 184 184 379 379 379 F F F Max gen: 196, Max flush: 404 A Difference in extent metadata is expected here On loop 98 Verifying data now No change after verify Loop: 98 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 188 188 188 391 391 391 F F F 1 020-039 197 197 197 405 405 405 F F F 2 040-059 196 196 196 402 402 402 F F F 3 060-079 188 188 188 389 389 389 F F F 4 080-099 197 197 197 405 405 405 F F F 5 100-119 197 197 197 405 405 405 F F F 6 120-139 194 194 194 399 399 399 F F F 7 140-159 196 196 196 402 402 402 F F F 8 160-179 196 196 196 402 402 402 F F F 9 180-199 190 190 190 393 393 393 F F F 10 200-219 196 196 196 402 402 402 F F F 11 220-239 196 196 196 402 402 402 F F F 12 240-259 195 195 195 401 401 401 F F F 13 260-279 196 196 196 402 402 402 F F F 14 280-299 194 194 194 399 399 399 F F F 15 300-319 196 196 196 402 402 402 F F F 16 320-339 196 196 196 404 404 404 F F F 17 340-359 192 192 192 396 396 396 F F F 18 360-379 195 195 195 401 401 401 F F F 19 380-399 194 194 194 399 399 399 F F F 20 400-419 194 194 194 399 399 399 F F F 21 420-439 197 197 197 405 405 405 F F F 22 440-459 197 197 197 405 405 405 F F F 23 460-479 188 188 188 391 391 391 F F F 24 480-499 194 194 194 399 399 399 F F F 25 500-519 192 192 192 396 396 396 F F F 26 520-539 194 194 194 400 400 400 F F F 27 540-559 192 192 192 396 396 396 F F F 28 560-579 196 196 196 404 404 404 F F F 29 580-599 184 184 184 379 379 379 F F F Max gen: 197, Max flush: 405 Begin loop 99 Downstairs to restart: 2 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.524064385Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5ae6bd0137b92ba7ccd4c1802a9f2f433ead0b4c\",\n git_commit_timestamp: \"2024-07-31T18:11:08.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.52458841Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.524620529Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974} {"msg":"Crucible 6ed7ea16-e12c-4feb-bf61-708ba516bb37 has session id: 75d2f2a7-cc15-4e7c-9b56-feb877702ac9","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.524714624Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"Upstairs opts: Upstairs UUID: 6ed7ea16-e12c-4feb-bf61-708ba516bb37, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.524818419Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"Crucible stats registered with UUID: 6ed7ea16-e12c-4feb-bf61-708ba516bb37","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.524857138Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:198","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.525176483Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974} {"msg":"6ed7ea16-e12c-4feb-bf61-708ba516bb37 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.525271208Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.525293467Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"0","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.525313866Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"0","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.525334195Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"1","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.525359734Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"1","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.525383913Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.525406072Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.525430861Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"io task","client":"0","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.52545365Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"io task","client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.525471659Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"io task","client":"1","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:06:47.525612622Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"io task","client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:06:47.52565667Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"io task","client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:06:47.525684719Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.525706168Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.525727807Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"ds_connection connected from Ok(127.0.0.1:49680)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.525751816Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"io task","client":"0","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"ds_connection connected from Ok(127.0.0.1:49585)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.525783634Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"io task","client":"1","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.525908029Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"io task","client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.526690762Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"0","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.526717261Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"0","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.52673198Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"1","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.526745749Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"1","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.52737234Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"0","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.527413458Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"1","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.527977202Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"1","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.528010321Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"1","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.52802907Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.528044739Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"0","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.528061428Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"0","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:47.528078307Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.527331685Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"io task","client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"ds_connection connected from Ok(127.0.0.1:57562)","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.527471568Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"io task","client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.528905591Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.5289339Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.529418937Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.529935693Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.529958752Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"[0]R flush_numbers[0..12]: [391, 405, 402, 389, 405, 405, 399, 402, 402, 393, 402, 402]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.529973671Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"downstairs","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"[0]R generation[0..12]: [188, 197, 196, 188, 197, 197, 194, 196, 196, 190, 196, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.529988221Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"downstairs","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530015549Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"downstairs","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"[1]R flush_numbers[0..12]: [391, 405, 402, 389, 405, 405, 399, 402, 402, 393, 402, 402]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530031568Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"downstairs","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"[1]R generation[0..12]: [188, 197, 196, 188, 197, 197, 194, 196, 196, 190, 196, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530045468Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"downstairs","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530059567Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"downstairs","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"[2]R flush_numbers[0..12]: [391, 405, 402, 389, 405, 405, 399, 402, 402, 393, 402, 402]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530073177Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"downstairs","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"[2]R generation[0..12]: [188, 197, 196, 188, 197, 197, 194, 196, 196, 190, 196, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530086886Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"downstairs","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530104645Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"downstairs","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"Max found gen is 198","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530118115Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"downstairs","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"Generation requested: 198 >= found:198","v":0,"name":"crucible","level":Wait for a query_work_queue command to finish before sending IO 30,"time":"2024-07-31T19:06:57.530135024Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"downstairs","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"Next flush: 406","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530148193Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"downstairs","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530161153Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"":"downstairs","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530174392Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530186351Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"0","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530199581Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"1","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.5302129Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"client":"2","":"downstairs","session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530226779Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530238999Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"6ed7ea16-e12c-4feb-bf61-708ba516bb37 is now active with session: 75d2f2a7-cc15-4e7c-9b56-feb877702ac9","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530251678Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530263688Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974,"session_id":"75d2f2a7-cc15-4e7c-9b56-feb877702ac9"} {"msg":"The guest has finished waiting for activation with:198","v":0,"name":"crucible","level":30,"time":"2024-07-31T19:06:57.530275837Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Run Repair workload 01/30 Write block 200 len 36864 data: 15 16 15 14 16 15 16 18 18 02/30 Read block 302 len 8192 03/30 Write block 380 len 24576 data: 17 17 17 19 22 18 04/30 Read block 340 len 20480 05/30 Read block 506 len 12288 06/30 Flush 07/30 Write block 432 len 28672 data: 21 18 15 18 16 14 12 08/30 Write block 12 len 16384 data: 11 9 8 11 09/30 Read block 507 len 28672 10/30 Write block 174 len 32768 data: 15 16 13 11 10 11 8 9 11/30 Read block 189 len 4096 12/30 Read block 591 len 28672 13/30 Flush 14/30 Write block 316 len 28672 data: 18 14 16 14 14 12 11 15/30 Read block 471 len 16384 16/30 Write block 46 len 12288 data: 11 16 14 17/30 Read block 524 len 16384 18/30 Read block 498 len 36864 19/30 Write block 429 len 36864 data: 21 22 22 22 19 16 19 17 15 20/30 Read block 547 len 8192 21/30 Write block 353 len 24576 data: 14 15 14 13 19 21 22/30 Read block 97 len 24576 23/30 Read block 195 len 20480 24/30 Write block 552 len 12288 data: 16 17 17 25/30 Read block 224 len 32768 26/30 Read block 440 len 8192 27/30 Read block 163 len 24576 28/30 Write block 89 len 8192 data: 13 14 29/30 Write block 289 len 24576 data: 12 13 15 15 12 11 30/30 Read block 587 len 32768 ---------------------------------------------------------------- Crucible gen:198 GIO:true work queues: Upstairs:0 downstairs:15 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 17 Acked 1016 Write 3 Done Done Done false 18 Acked 1017 Read 4 Done Done Done false 19 Acked 1018 Read 9 Done Done Done false 20 Acked 1019 Write 9 Done Done Done false 21 Acked 1020 Read 2 Done Done Done false 22 Acked 1021 Write 6 Done Done Done false 23 Acked 1022 Read 6 Done Done Done false 24 Acked 1023 Read 5 Done Done Done false 25 Acked 1024 Write 3 Done Done Done false 26 Acked 1025 Read 8 Done Done Done false 27 Acked 1026 Read 2 Done Done Done false 28 Acked 1027 Read 6 Done Done Done false 29 Acked 1028 Write 2 Done Done Done false 30 Acked 1029 Write 6 Done Done Done false 31 Acked 1030 Read 8 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 1 0 1 2 Done 14 15 14 43 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1015 1015 1015 Downstairs last five completed: 1015 1014 1013 1012 1011 Upstairs last five completed: 31 30 29 28 27 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-07-31T19:06:57.557321505Z","hostname":"w-01J450MQWJJ6THYPZVP3CB5SWT","pid":1974} Wrote out file "/tmp/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 198 198 198 407 407 407 F F F 1 020-039 197 197 197 405 405 405 F F F 2 040-059 196 196 196 402 402 402 T T T 3 060-079 188 188 188 389 389 389 F F F 4 080-099 197 197 197 405 405 405 T T T 5 100-119 197 197 197 405 405 405 F F F 6 120-139 194 194 194 399 399 399 F F F 7 140-159 196 196 196 402 402 402 F F F 8 160-179 198 198 198 407 407 407 F F F 9 180-199 198 198 198 407 407 407 F F F 10 200-219 198 198 198 406 406 406 F F F 11 220-239 196 196 196 402 402 402 F F F 12 240-259 195 195 195 401 401 401 F F F 13 260-279 196 196 196 402 402 402 F F F 14 280-299 194 194 194 399 399 399 T T T 15 300-319 198 198 198 408 408 408 F F F 16 320-339 198 198 198 408 408 408 F F F 17 340-359 192 192 192 396 396 396 T T T 18 360-379 195 195 195 401 401 401 F F F 19 380-399 198 198 198 406 406 406 F F F 20 400-419 194 194 194 399 399 399 F F F 21 420-439 198 198 198 407 407 407 T T T 22 440-459 197 197 197 405 405 405 F F F 23 460-479 188 188 188 391 391 391 F F F 24 480-499 194 194 194 399 399 399 F F F 25 500-519 192 192 192 396 396 396 F F F 26 520-539 194 194 194 400 400 400 F F F 27 540-559 192 192 192 396 396 396 T T T 28 560-579 196 196 196 404 404 404 F F F 29 580-599 184 184 184 379 379 379 F F F Max gen: 198, Max flush: 408 A Difference in extent metadata is expected here On loop 99 Verifying data now No change after verify Loop: 99 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 198 198 198 407 407 407 F F F 1 020-039 197 197 197 405 405 405 F F F 2 040-059 199 199 199 409 409 409 F F F 3 060-079 188 188 188 389 389 389 F F F 4 080-099 199 199 199 409 409 409 F F F 5 100-119 197 197 197 405 405 405 F F F 6 120-139 194 194 194 399 399 399 F F F 7 140-159 196 196 196 402 402 402 F F F 8 160-179 198 198 198 407 407 407 F F F 9 180-199 198 198 198 407 407 407 F F F 10 200-219 198 198 198 406 406 406 F F F 11 220-239 196 196 196 402 402 402 F F F 12 240-259 195 195 195 401 401 401 F F F 13 260-279 196 196 196 402 402 402 F F F 14 280-299 199 199 199 409 409 409 F F F 15 300-319 198 198 198 408 408 408 F F F 16 320-339 198 198 198 408 408 408 F F F 17 340-359 199 199 199 409 409 409 F F F 18 360-379 195 195 195 401 401 401 F F F 19 380-399 198 198 198 406 406 406 F F F 20 400-419 194 194 194 399 399 399 F F F 21 420-439 199 199 199 409 409 409 F F F 22 440-459 197 197 197 405 405 405 F F F 23 460-479 188 188 188 391 391 391 F F F 24 480-499 194 194 194 399 399 399 F F F 25 500-519 192 192 192 396 396 396 F F F 26 520-539 194 194 194 400 400 400 F F F 27 540-559 199 199 199 409 409 409 F F F 28 560-579 196 196 196 404 404 404 F F F 29 580-599 184 184 184 379 379 379 F F F Max gen: 199, Max flush: 409 40:21 Test duration Test completed real 40:21.187435410 user 41.133452432 sys 19.456320910 trap 0.051844684 tflt 0.147837493 dflt 0.141389013 kflt 0.006423796 lock 64:10:17.040901971 slp 3:19:54.800794570 lat 8.060075325 stop 0.064361092