01KZSFYW94WFWHGJVPT5HJNQ1C: test-up-2region-encrypted

BasicConfig {
    output_rules: [
        "%/tmp/test_up*/*.txt",
        "%/tmp/test_up*/dsc/*.txt",
        "%/tmp/debug/*",
        "/tmp/core.*",
    ],
    rust_toolchain: None,
    target: Some(
        "helios-2.0",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01KZSFZKWKJKK54921EW16FDS0

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12026-08-11T22:56:57.563Zjob dependencies complete; ready to run (waiting for 12 m 18 s)
22026-08-11T22:57:45.980Zjob assigned to worker 01KZSGPBC51FTS1QWF7FE5A8CR [factory aws, i-05f2c3fbd4be90e51] (queued for 48 s)
32026-08-11T22:57:52.874Zdownloading input: /input/build/work/bins/crucible-downstairs.gz
42026-08-11T22:57:55.900Zdownloaded input: /input/build/work/bins/crucible-downstairs.gz
52026-08-11T22:57:55.908Zdownloading input: /input/build/work/bins/crucible-hammer.gz
62026-08-11T22:58:03.515Zdownloaded input: /input/build/work/bins/crucible-hammer.gz
72026-08-11T22:58:03.530Zdownloading input: /input/build/work/bins/crutest.gz
82026-08-11T22:58:06.832Zdownloaded input: /input/build/work/bins/crutest.gz
92026-08-11T22:58:06.843Zdownloading input: /input/build/work/bins/dsc.gz
102026-08-11T22:58:17.664Zdownloaded input: /input/build/work/bins/dsc.gz
112026-08-11T22:58:17.675Zdownloading input: /input/build/work/scripts/perf-downstairs-tick.d
122026-08-11T22:58:17.696Zdownloaded input: /input/build/work/scripts/perf-downstairs-tick.d
132026-08-11T22:58:17.704Zdownloading input: /input/build/work/scripts/test_ds.sh
142026-08-11T22:58:17.752Zdownloaded input: /input/build/work/scripts/test_ds.sh
152026-08-11T22:58:17.752Zdownloading input: /input/build/work/scripts/test_live_repair.sh
162026-08-11T22:58:17.792Zdownloaded input: /input/build/work/scripts/test_live_repair.sh
172026-08-11T22:58:17.792Zdownloading input: /input/build/work/scripts/test_repair.sh
182026-08-11T22:58:17.827Zdownloaded input: /input/build/work/scripts/test_repair.sh
192026-08-11T22:58:17.827Zdownloading input: /input/build/work/scripts/test_replay.sh
202026-08-11T22:58:17.865Zdownloaded input: /input/build/work/scripts/test_replay.sh
212026-08-11T22:58:17.871Zdownloading input: /input/build/work/scripts/test_up.sh
222026-08-11T22:58:17.923Zdownloaded input: /input/build/work/scripts/test_up.sh
232026-08-11T22:58:17.924Zdownloading input: /input/build/work/scripts/upstairs_info.d
242026-08-11T22:58:17.957Zdownloaded input: /input/build/work/scripts/upstairs_info.d
252026-08-11T22:58:17.971Zdownloading input: /input/build/tmp/cargo-test-out.log
262026-08-11T22:58:18.754Zdownloaded input: /input/build/tmp/cargo-test-out.log
 
272026-08-11T22:58:18.754Zstarting task 0: "setup"
282026-08-11T22:58:18.766Z++ uname -s
292026-08-11T22:58:18.779Z+ kern=SunOS
302026-08-11T22:58:18.779Z+ build_user=build
312026-08-11T22:58:18.779Z+ build_uid=12345
322026-08-11T22:58:18.780Z+ work_dir=/work
332026-08-11T22:58:18.780Z+ input_dir=/input
342026-08-11T22:58:18.780Z+ [[ 0 == 12345 ]]
352026-08-11T22:58:18.780Z+ case "$kern" in
362026-08-11T22:58:18.780Z+ groupadd -g 12345 build
372026-08-11T22:58:18.780Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
382026-08-11T22:58:20.769Z+ zfs create -o mountpoint=/work rpool/work
392026-08-11T22:58:20.915Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
402026-08-11T22:58:20.919Z+ home_fs=zfs
412026-08-11T22:58:20.919Z+ [[ zfs == autofs ]]
422026-08-11T22:58:20.919Z+ mkdir -p /home/build
432026-08-11T22:58:20.919Z+ chown build:build /home/build /work
442026-08-11T22:58:21.920Z+ chmod 0700 /home/build /work
452026-08-11T22:58:21.924Zprocess exited: duration 3167 ms, exit code 0
 
462026-08-11T22:58:21.933Zstarting task 1: "authentication"
472026-08-11T22:58:21.947ZWARNING: job store has no value for "GITHUB_TOKEN"; waiting for a value...
482026-08-11T22:58:29.966Zprocess exited: duration 8032 ms, exit code 0
 
492026-08-11T22:58:29.972Zstarting task 2: "build"
502026-08-11T22:58:29.976Z+ banner cores
512026-08-11T22:58:29.979Z
522026-08-11T22:58:29.979Z #### #### ##### ###### ####
532026-08-11T22:58:29.979Z # # # # # # # #
542026-08-11T22:58:29.979Z # # # # # ##### ####
552026-08-11T22:58:29.979Z # # # ##### # #
562026-08-11T22:58:29.979Z # # # # # # # # #
572026-08-11T22:58:29.979Z #### #### # # ###### ####
582026-08-11T22:58:29.979Z
592026-08-11T22:58:29.979Z+ pfexec coreadm -i /tmp/core.%f.%p -g /tmp/core.%f.%p -e global -e log -e proc-setid -e global-setid
602026-08-11T22:58:29.984Z+ banner unpack
612026-08-11T22:58:29.987Z
622026-08-11T22:58:29.987Z # # # # ##### ## #### # #
632026-08-11T22:58:29.987Z # # ## # # # # # # # # #
642026-08-11T22:58:29.987Z # # # # # # # # # # ####
652026-08-11T22:58:29.987Z # # # # # ##### ###### # # #
662026-08-11T22:58:29.987Z # # # ## # # # # # # #
672026-08-11T22:58:29.987Z #### # # # # # #### # #
682026-08-11T22:58:29.987Z
692026-08-11T22:58:29.987Z+ mkdir -p /var/tmp/bins
702026-08-11T22:58:29.987Z+ for t in "$input/bins/"*.gz
712026-08-11T22:58:29.990Z++ basename /input/build/work/bins/crucible-downstairs.gz
722026-08-11T22:58:29.990Z+ b=crucible-downstairs.gz
732026-08-11T22:58:29.990Z+ b=crucible-downstairs
742026-08-11T22:58:29.990Z+ gunzip
752026-08-11T22:58:32.525Z+ chmod +x /var/tmp/bins/crucible-downstairs
762026-08-11T22:58:32.529Z+ for t in "$input/bins/"*.gz
772026-08-11T22:58:32.529Z++ basename /input/build/work/bins/crucible-hammer.gz
782026-08-11T22:58:32.529Z+ b=crucible-hammer.gz
792026-08-11T22:58:32.529Z+ b=crucible-hammer
802026-08-11T22:58:32.529Z+ gunzip
812026-08-11T22:58:34.941Z+ chmod +x /var/tmp/bins/crucible-hammer
822026-08-11T22:58:34.944Z+ for t in "$input/bins/"*.gz
832026-08-11T22:58:34.944Z++ basename /input/build/work/bins/crutest.gz
842026-08-11T22:58:34.944Z+ b=crutest.gz
852026-08-11T22:58:34.944Z+ b=crutest
862026-08-11T22:58:34.944Z+ gunzip
872026-08-11T22:58:37.473Z+ chmod +x /var/tmp/bins/crutest
882026-08-11T22:58:37.479Z+ for t in "$input/bins/"*.gz
892026-08-11T22:58:37.479Z++ basename /input/build/work/bins/dsc.gz
902026-08-11T22:58:37.479Z+ b=dsc.gz
912026-08-11T22:58:37.479Z+ b=dsc
922026-08-11T22:58:37.479Z+ gunzip
932026-08-11T22:58:38.586Z+ chmod +x /var/tmp/bins/dsc
942026-08-11T22:58:38.589Z+ export BINDIR=/var/tmp/bins
952026-08-11T22:58:38.589Z+ BINDIR=/var/tmp/bins
962026-08-11T22:58:38.589Z+ jobpid=1088
972026-08-11T22:58:38.589Z+ echo 'Setup debug logging'
982026-08-11T22:58:38.589ZSetup debug logging
992026-08-11T22:58:38.589Z+ mkdir /tmp/debug
1002026-08-11T22:58:38.589Z+ sleep 3600
1012026-08-11T22:58:38.593Z+ psrinfo -v
1022026-08-11T22:58:38.596Z+ df -h
1032026-08-11T22:58:38.599Z+ prstat -d d -mLc 1
1042026-08-11T22:58:38.601Z+ iostat -T d -xn 1
1052026-08-11T22:58:38.602Z+ mpstat -T d 1
1062026-08-11T22:58:38.602Z+ vmstat -T d -p 1
1072026-08-11T22:58:38.602Z+ pfexec dtrace -Z -s /input/build/work/scripts/perf-downstairs-tick.d
1082026-08-11T22:58:38.602Z+ banner test_up_2r_encrypted
1092026-08-11T22:58:38.602Z+ pfexec dtrace -Z -s /input/build/work/scripts/upstairs_info.d
1102026-08-11T22:58:38.605Z #####
1112026-08-11T22:58:38.605Z ##### ###### #### ##### # # ##### # # #####
1122026-08-11T22:58:38.605Z # # # # # # # # # # #
1132026-08-11T22:58:38.605Z # ##### #### # # # # # ##### # #
1142026-08-11T22:58:38.605Z # # # # # # ##### # #####
1152026-08-11T22:58:38.605Z # # # # # # # # # # #
1162026-08-11T22:58:38.605Z # ###### #### # ####### #### # ####### ####### # #
1172026-08-11T22:58:38.605Z
1182026-08-11T22:58:38.605Z+ ptime -m bash /input/build/work/scripts/test_up.sh -r 2 -N encrypted
1192026-08-11T22:58:38.611Z/input/build/work
1202026-08-11T22:58:38.619ZUsing 2 region sets
1212026-08-11T22:58:38.619ZTurn off color for downstairs dump
1222026-08-11T22:58:38.631ZUpstairs using key: GrUTHX1kzQjaDvu0ajWJsunlf244tzyYbPQ+zJmQJVw=
1232026-08-11T22:58:38.635Zdsc output goes to /tmp/test_up-build/dsc-out.txt
1242026-08-11T22:58:38.635ZCreating 6 downstairs regions
1252026-08-11T22:58:38.889ZStarting 6 downstairs
1262026-08-11T22:58:38.892Zdsc started at PID: 1136
1272026-08-11T22:58:43.978ZDisable automatic restart on all downstairs
1282026-08-11T22:58:43.990Z
1292026-08-11T22:58:43.990ZBegin tests, output goes to /tmp/test_up-build/test_up_out.txt
1302026-08-11T22:58:43.993ZRunning test: span
1312026-08-11T22:58:48.059ZCompleted test: span
1322026-08-11T22:58:53.065ZRunning test: big
1332026-08-11T22:58:57.369ZCompleted test: big
1342026-08-11T22:59:02.378ZRunning test: dep
1352026-08-11T22:59:21.859ZCompleted test: dep
1362026-08-11T22:59:26.866ZRunning test: balloon
1372026-08-11T23:01:53.966ZCompleted test: balloon
1382026-08-11T23:01:58.973ZRunning test: deactivate
1392026-08-11T23:02:03.342ZCompleted test: deactivate
1402026-08-11T23:02:08.348ZRunning hammer
1412026-08-11T23:02:15.835ZRun repair tests
1422026-08-11T23:02:15.838Z/var/tmp/bins/crutest fill -g 31 -q --key GrUTHX1kzQjaDvu0ajWJsunlf244tzyYbPQ+zJmQJVw= --dsc 127.0.0.1:9998 --verify-out /var/tmp/test_up-build/verify_file
1432026-08-11T23:02:15.856Z{"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 10, extent_count: 5 }","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.006106246Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"task":"crutest"}
1442026-08-11T23:02:15.860Z{"msg":"dsc has 6 regions. This means 2 sub_volumes","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.006305264Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"task":"crutest"}
1452026-08-11T23:02:15.861Z{"msg":"SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.006927687Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"task":"crutest"}
1462026-08-11T23:02:15.861Z{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.007161991Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179}
1472026-08-11T23:02:15.861Z{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"3d9051b3f6e9b5b4034790013a3d1fc35523d0f6\",\n git_commit_timestamp: \"2026-08-11T22:43:07.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.007198148Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179}
1482026-08-11T23:02:15.861Z{"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.007207746Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179}
1492026-08-11T23:02:15.861Z{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 10, shift: 12 }, extent_count: 5, uuid: 00000000-0000-0000-0000-000000000000, encrypted: true, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.007220285Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179}
1502026-08-11T23:02:15.861Z{"msg":"Crucible 20930fde-2b67-4e8d-bae6-e634cc19d757 has session id: 47e124fe-014e-476c-9f83-5e01d1f184db","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.007622762Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1512026-08-11T23:02:15.861Z{"msg":"Upstairs opts: Upstairs UUID: 20930fde-2b67-4e8d-bae6-e634cc19d757, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: true, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.007643929Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1522026-08-11T23:02:15.861Z{"msg":"Crucible stats registered with UUID: 20930fde-2b67-4e8d-bae6-e634cc19d757","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.007654648Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1532026-08-11T23:02:15.861Z{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-08-11T23:02:14.007667897Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1542026-08-11T23:02:15.861Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.007809301Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"1","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1552026-08-11T23:02:15.861Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.007842988Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"2","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1562026-08-11T23:02:15.861Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.007861686Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"0","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1572026-08-11T23:02:15.861Z{"msg":"SV 1 has targets: [127.0.0.1:8840, 127.0.0.1:8850, 127.0.0.1:8860]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.008643931Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"task":"crutest"}
1582026-08-11T23:02:15.861Z{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.008700885Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179}
1592026-08-11T23:02:15.861Z{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"3d9051b3f6e9b5b4034790013a3d1fc35523d0f6\",\n git_commit_timestamp: \"2026-08-11T22:43:07.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.008721483Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179}
1602026-08-11T23:02:15.861Z{"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.008730812Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179}
1612026-08-11T23:02:15.861Z{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 10, shift: 12 }, extent_count: 5, uuid: 00000000-0000-0000-0000-000000000000, encrypted: true, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.00874266Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179}
1622026-08-11T23:02:15.861Z{"msg":"Crucible 20930fde-2b67-4e8d-bae6-e634cc19d757 has session id: 18988b58-9b81-4f50-8a30-0fb18dd4ca53","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.008941459Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
1632026-08-11T23:02:15.861Z{"msg":"Upstairs opts: Upstairs UUID: 20930fde-2b67-4e8d-bae6-e634cc19d757, Targets: [127.0.0.1:8840, 127.0.0.1:8850, 127.0.0.1:8860], lossy: false, flush_timeout: None, key populated: true, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.008961127Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
1642026-08-11T23:02:15.861Z{"msg":"Crucible stats registered with UUID: 20930fde-2b67-4e8d-bae6-e634cc19d757","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.008972065Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
1652026-08-11T23:02:15.861Z{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-08-11T23:02:14.008981154Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
1662026-08-11T23:02:15.861Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.009004162Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"0","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
1672026-08-11T23:02:15.861Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.00901974Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"2","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
1682026-08-11T23:02:15.861Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.009030209Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"1","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
1692026-08-11T23:02:15.861Z{"msg":"Downstairs in volume = 6","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.009100312Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"task":"crutest"}
1702026-08-11T23:02:15.861Z{"msg":"The guest has requested activation with gen:31","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.0091143Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179}
1712026-08-11T23:02:15.861Z{"msg":"20930fde-2b67-4e8d-bae6-e634cc19d757 active request set","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.009139727Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1722026-08-11T23:02:15.861Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.009153796Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"0","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1732026-08-11T23:02:15.861Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.009164045Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"1","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1742026-08-11T23:02:15.861Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.009174403Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"2","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1752026-08-11T23:02:15.861Z{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.00920916Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"0","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1762026-08-11T23:02:15.861Z{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.009239606Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"1","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1772026-08-11T23:02:15.861Z{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.009262504Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"2","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1782026-08-11T23:02:15.861Z{"msg":"ds_connection connected from Ok(127.0.0.1:42996)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.009274183Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"0","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1792026-08-11T23:02:15.866Z{"msg":"ds_connection connected from Ok(127.0.0.1:48365)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.009285032Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"2","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1802026-08-11T23:02:15.866Z{"msg":"ds_connection connected from Ok(127.0.0.1:57258)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.00930234Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"1","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1812026-08-11T23:02:15.866Z{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.00994427Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"2","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1822026-08-11T23:02:15.866Z{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.009977806Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"0","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1832026-08-11T23:02:15.866Z{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.009989895Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"1","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1842026-08-11T23:02:15.866Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010124931Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1852026-08-11T23:02:15.866Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010167276Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1862026-08-11T23:02:15.866Z{"msg":"[0]R flush_numbers: [4000, 4000, 4000, 4000, 4000]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010189604Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1872026-08-11T23:02:15.866Z{"msg":"[0]R generation: [25, 25, 25, 25, 25]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010200972Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1882026-08-11T23:02:15.866Z{"msg":"[0]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010211651Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1892026-08-11T23:02:15.866Z{"msg":"[1]R flush_numbers: [4000, 4000, 4000, 4000, 4000]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.01022245Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1902026-08-11T23:02:15.866Z{"msg":"[1]R generation: [25, 25, 25, 25, 25]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010233169Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1912026-08-11T23:02:15.866Z{"msg":"[1]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010243578Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1922026-08-11T23:02:15.866Z{"msg":"[2]R flush_numbers: [4000, 4000, 4000, 4000, 4000]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010254317Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1932026-08-11T23:02:15.866Z{"msg":"[2]R generation: [25, 25, 25, 25, 25]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010264465Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1942026-08-11T23:02:15.866Z{"msg":"[2]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010274914Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1952026-08-11T23:02:15.866Z{"msg":"Max found gen is 26","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010284743Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1962026-08-11T23:02:15.866Z{"msg":"Generation requested: 31 >= found:26","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010295342Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1972026-08-11T23:02:15.866Z{"msg":"Next flush: 4001","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010305211Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1982026-08-11T23:02:15.866Z{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.01031543Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
1992026-08-11T23:02:15.866Z{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010326139Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
2002026-08-11T23:02:15.866Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010335718Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"0","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
2012026-08-11T23:02:15.866Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010345977Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"1","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
2022026-08-11T23:02:15.866Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010355646Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"2","":"downstairs","session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
2032026-08-11T23:02:15.866Z{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010366034Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
2042026-08-11T23:02:15.867Z{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010375623Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
2052026-08-11T23:02:15.867Z{"msg":"20930fde-2b67-4e8d-bae6-e634cc19d757 is now active with session: 47e124fe-014e-476c-9f83-5e01d1f184db","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010386302Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
2062026-08-11T23:02:15.867Z{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010401801Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"47e124fe-014e-476c-9f83-5e01d1f184db"}
2072026-08-11T23:02:15.867Z{"msg":"The guest has finished waiting for activation with:31","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.01041165Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179}
2082026-08-11T23:02:15.867Z{"msg":"Activated sub_volume 20930fde-2b67-4e8d-bae6-e634cc19d757","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010424308Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179}
2092026-08-11T23:02:15.867Z{"msg":"The guest has requested activation with gen:31","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010432497Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179}
2102026-08-11T23:02:15.867Z{"msg":"20930fde-2b67-4e8d-bae6-e634cc19d757 active request set","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010524007Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2112026-08-11T23:02:15.867Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010535146Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"0","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2122026-08-11T23:02:15.867Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010545505Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"1","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2132026-08-11T23:02:15.867Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010555394Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"2","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2142026-08-11T23:02:15.867Z{"msg":"connecting to 127.0.0.1:8860","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010569732Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"2","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2152026-08-11T23:02:15.867Z{"msg":"connecting to 127.0.0.1:8840","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010580201Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"0","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2162026-08-11T23:02:15.867Z{"msg":"connecting to 127.0.0.1:8850","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.01059077Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"1","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2172026-08-11T23:02:15.867Z{"msg":"ds_connection connected from Ok(127.0.0.1:51085)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010602189Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"2","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2182026-08-11T23:02:15.867Z{"msg":"ds_connection connected from Ok(127.0.0.1:43312)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010613088Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"1","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2192026-08-11T23:02:15.867Z{"msg":"ds_connection connected from Ok(127.0.0.1:54745)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010628126Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"io task","client":"0","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2202026-08-11T23:02:15.867Z{"msg":"downstairs client at Some(127.0.0.1:8860) has region UUID 12345678-0000-0000-0000-000000008860","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010806797Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"2","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2212026-08-11T23:02:15.867Z{"msg":"downstairs client at Some(127.0.0.1:8850) has region UUID 12345678-0000-0000-0000-000000008850","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010861161Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"1","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2222026-08-11T23:02:15.867Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.01095712Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2232026-08-11T23:02:15.867Z{"msg":"downstairs client at Some(127.0.0.1:8840) has region UUID 12345678-0000-0000-0000-000000008840","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.010990047Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"0","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2242026-08-11T23:02:15.867Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011030772Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2252026-08-11T23:02:15.867Z{"msg":"[0]R flush_numbers: [5320, 5306, 5321, 5321, 5303]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011131682Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2262026-08-11T23:02:15.867Z{"msg":"[0]R generation: [14, 9, 14, 14, 7]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.01114272Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2272026-08-11T23:02:15.867Z{"msg":"[0]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011153719Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2282026-08-11T23:02:15.867Z{"msg":"[1]R flush_numbers: [5320, 5306, 5321, 5321, 5303]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011168198Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2292026-08-11T23:02:15.867Z{"msg":"[1]R generation: [14, 9, 14, 14, 7]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011178456Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2302026-08-11T23:02:15.867Z{"msg":"[1]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011188865Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2312026-08-11T23:02:15.867Z{"msg":"[2]R flush_numbers: [5320, 5306, 5321, 5321, 5303]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011202964Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2322026-08-11T23:02:15.867Z{"msg":"[2]R generation: [14, 9, 14, 14, 7]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011212653Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2332026-08-11T23:02:15.867Z{"msg":"[2]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011223282Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2342026-08-11T23:02:15.867Z{"msg":"Max found gen is 15","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011233131Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2352026-08-11T23:02:15.867Z{"msg":"Generation requested: 31 >= found:15","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011243829Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2362026-08-11T23:02:15.867Z{"msg":"Next flush: 5322","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011253598Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2372026-08-11T23:02:15.867Z{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011271186Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"":"downstairs","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2382026-08-11T23:02:15.867Z{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011281395Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2392026-08-11T23:02:15.867Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011290434Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"0","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2402026-08-11T23:02:15.867Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011300863Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"1","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2412026-08-11T23:02:15.867Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011311002Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"client":"2","":"downstairs","session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2422026-08-11T23:02:15.867Z{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011320851Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2432026-08-11T23:02:15.868Z{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.01133063Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2442026-08-11T23:02:15.868Z{"msg":"20930fde-2b67-4e8d-bae6-e634cc19d757 is now active with session: 18988b58-9b81-4f50-8a30-0fb18dd4ca53","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011341359Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2452026-08-11T23:02:15.868Z{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011357787Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"session_id":"18988b58-9b81-4f50-8a30-0fb18dd4ca53"}
2462026-08-11T23:02:15.868Z{"msg":"The guest has finished waiting for activation with:31","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011367476Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179}
2472026-08-11T23:02:15.868Z{"msg":"Activated sub_volume 20930fde-2b67-4e8d-bae6-e634cc19d757","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011376365Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179}
2482026-08-11T23:02:15.868Z{"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011385014Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"task":"crutest"}
2492026-08-11T23:02:15.868ZDisk: sv:2 bs:4096 ts:409600 tb:100 max_io:100 or 409600
2502026-08-11T23:02:15.868Z{"msg":"Fill test","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.011459936Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"task":"crutest"}
2512026-08-11T23:02:15.887ZRead and Verify all blocks (0..100 range:false)
2522026-08-11T23:02:15.935Z{"msg":"Wrote out file \"/var/tmp/test_up-build/verify_file\"","v":----------------------------------------------------------------
2532026-08-11T23:02:15.935Z0,"name":"crucible","level": Crucible gen:31 GIO:true work queues: Upstairs:0 downstairs:1
2542026-08-11T23:02:15.935Z30States: Active Active Active
2552026-08-11T23:02:15.935ZJOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2562026-08-11T23:02:15.935Z 1002 Acked Read 50 Done, Done" Donetime" false:
2572026-08-11T23:02:15.935Z"2026-08-11T23:02:14.084430193Z" STATES DS:0 DS:1 DS:2 TOTAL
2582026-08-11T23:02:15.935Z, Sent " 0 hostname":" 0 0 0
2592026-08-11T23:02:15.935Zw-01KZSGPBC51FTS1QWF7FE5A8CR" Done 1 , 1 " 1 pid 3
2602026-08-11T23:02:15.938Z": Skipped 0 0 0 0
2612026-08-11T23:02:15.938Z1179 Error 0 0 0 0
2622026-08-11T23:02:15.939Z,"Last Flush: task":"Some(JobId(1001)) crutestSome(JobId(1001)) "Some(JobId(1001))
2632026-08-11T23:02:15.939Z}Downstairs last five completed:
2642026-08-11T23:02:15.939Z 1001 1000
2652026-08-11T23:02:15.939Z{"msgUpstairs last five completed: ":" 1002 1001 1000
2662026-08-11T23:02:15.939ZCLIENT: Tests done. All submitted work has been ACK'd","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.084467769Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"task":"crutest"}
2672026-08-11T23:02:15.939Z----------------------------------------------------------------
2682026-08-11T23:02:15.939Z Crucible gen:31 GIO:true work queues: Upstairs:0 downstairs:1
2692026-08-11T23:02:15.939ZStates: Active Active Active
2702026-08-11T23:02:15.939ZJOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2712026-08-11T23:02:15.939Z 1002 Acked Read 50 Done Done Done false
2722026-08-11T23:02:15.939Z STATES DS:0 DS:1 DS:2 TOTAL
2732026-08-11T23:02:15.939Z Sent 0 0 0 0
2742026-08-11T23:02:15.939Z Done 1 1 1 3
2752026-08-11T23:02:15.939Z Skipped 0 0 0 0
2762026-08-11T23:02:15.939Z Error 0 0 0 0
2772026-08-11T23:02:15.939ZLast Flush: Some(JobId(1001)) Some(JobId(1001)) Some(JobId(1001))
2782026-08-11T23:02:15.939ZDownstairs last five completed: 1001 1000
2792026-08-11T23:02:15.939ZUpstairs last five completed: 1002 1001 1000
2802026-08-11T23:02:15.939Z{"msg":"CLIENT: Up:0 ds:2 act:6","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:14.084542781Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"task":"crutest"}
2812026-08-11T23:02:19.937Z----------------------------------------------------------------
2822026-08-11T23:02:19.937Z Crucible gen:31 GIO:true work queues: Upstairs:0 downstairs:0
2832026-08-11T23:02:19.937ZDownstairs last five completed: 1003 1002 1001 1000
2842026-08-11T23:02:19.937ZUpstairs last five completed: 1003 1002 1001 1000
2852026-08-11T23:02:19.937Z----------------------------------------------------------------
2862026-08-11T23:02:19.937Z Crucible gen:31 GIO:true work queues: Upstairs:0 downstairs:0
2872026-08-11T23:02:19.937ZDownstairs last five completed: 1003 1002 1001 1000
2882026-08-11T23:02:19.937ZUpstairs last five completed: 1003 1002 1001 1000
2892026-08-11T23:02:19.946Z{"msg":"CLIENT: Up:0 ds:0 act:6","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.085257102Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"task":"crutest"}
2902026-08-11T23:02:19.946Z{"msg":"CLIENT: All crucible jobs finished, exiting program","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.085308716Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179,"task":"crutest"}
2912026-08-11T23:02:19.946Z{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-08-11T23:02:18.085324415Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1179}
2922026-08-11T23:02:19.951ZRepair setup passed
2932026-08-11T23:02:19.951ZCopy the region for /var/tmp/test_up-build/8830
2942026-08-11T23:02:19.951Zcp -r /var/tmp/test_up-build/8830 /var/tmp/test_up-build/previous
2952026-08-11T23:02:19.958Z/var/tmp/bins/crutest fill -g 32 -q --key GrUTHX1kzQjaDvu0ajWJsunlf244tzyYbPQ+zJmQJVw= --dsc 127.0.0.1:9998 --verify-out /var/tmp/test_up-build/verify_file --verify-in /var/tmp/test_up-build/verify_file
2962026-08-11T23:02:19.971Z{"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 10, extent_count: 5 }","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.118475594Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"task":"crutest"}
2972026-08-11T23:02:19.976Z{"msg":"dsc has 6 regions. This means 2 sub_volumes","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.118771142Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"task":"crutest"}
2982026-08-11T23:02:19.976Z{"msg":"SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.119511682Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"task":"crutest"}
2992026-08-11T23:02:19.976Z{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.119737888Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185}
3002026-08-11T23:02:19.976Z{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"3d9051b3f6e9b5b4034790013a3d1fc35523d0f6\",\n git_commit_timestamp: \"2026-08-11T22:43:07.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.119763445Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185}
3012026-08-11T23:02:19.976Z{"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.119776623Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185}
3022026-08-11T23:02:19.976Z{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 10, shift: 12 }, extent_count: 5, uuid: 00000000-0000-0000-0000-000000000000, encrypted: true, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.119788952Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185}
3032026-08-11T23:02:19.976Z{"msg":"Crucible ddbcb2e3-95b4-436c-ad07-a8ed16556d9c has session id: 9e983874-62fd-4aa8-952a-968c19290a69","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.120169141Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3042026-08-11T23:02:19.977Z{"msg":"Upstairs opts: Upstairs UUID: ddbcb2e3-95b4-436c-ad07-a8ed16556d9c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: true, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.120197498Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3052026-08-11T23:02:19.977Z{"msg":"Crucible stats registered with UUID: ddbcb2e3-95b4-436c-ad07-a8ed16556d9c","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.120208217Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3062026-08-11T23:02:19.977Z{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-08-11T23:02:18.120221955Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3072026-08-11T23:02:19.977Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.120329593Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3082026-08-11T23:02:19.977Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.120352981Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3092026-08-11T23:02:19.977Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.12036341Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3102026-08-11T23:02:19.977Z{"msg":"SV 1 has targets: [127.0.0.1:8840, 127.0.0.1:8850, 127.0.0.1:8860]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121256933Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"task":"crutest"}
3112026-08-11T23:02:19.977Z{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121308527Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185}
3122026-08-11T23:02:19.977Z{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"3d9051b3f6e9b5b4034790013a3d1fc35523d0f6\",\n git_commit_timestamp: \"2026-08-11T22:43:07.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121333285Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185}
3132026-08-11T23:02:19.977Z{"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121346033Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185}
3142026-08-11T23:02:19.977Z{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 10, shift: 12 }, extent_count: 5, uuid: 00000000-0000-0000-0000-000000000000, encrypted: true, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121357682Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185}
3152026-08-11T23:02:19.977Z{"msg":"Crucible ddbcb2e3-95b4-436c-ad07-a8ed16556d9c has session id: 9b028c21-aa6f-4d43-b514-73333eb701d8","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.12156386Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3162026-08-11T23:02:19.977Z{"msg":"Upstairs opts: Upstairs UUID: ddbcb2e3-95b4-436c-ad07-a8ed16556d9c, Targets: [127.0.0.1:8840, 127.0.0.1:8850, 127.0.0.1:8860], lossy: false, flush_timeout: None, key populated: true, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121597076Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3172026-08-11T23:02:19.977Z{"msg":"Crucible stats registered with UUID: ddbcb2e3-95b4-436c-ad07-a8ed16556d9c","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121615694Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3182026-08-11T23:02:19.977Z{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-08-11T23:02:18.121625723Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3192026-08-11T23:02:19.977Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121634792Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3202026-08-11T23:02:19.977Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.12165355Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3212026-08-11T23:02:19.977Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121663909Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3222026-08-11T23:02:19.977Z{"msg":"Downstairs in volume = 6","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121792525Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"task":"crutest"}
3232026-08-11T23:02:19.977Z{"msg":"The guest has requested activation with gen:32","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121807543Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185}
3242026-08-11T23:02:19.977Z{"msg":"ddbcb2e3-95b4-436c-ad07-a8ed16556d9c active request set","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121854008Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3252026-08-11T23:02:19.977Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121868027Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"0","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3262026-08-11T23:02:19.977Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121878336Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"1","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3272026-08-11T23:02:19.977Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121888775Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"2","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3282026-08-11T23:02:19.977Z{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121899103Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3292026-08-11T23:02:19.977Z{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121910082Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3302026-08-11T23:02:19.977Z{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121920921Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3312026-08-11T23:02:19.977Z{"msg":"ds_connection connected from Ok(127.0.0.1:48377)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121968876Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3322026-08-11T23:02:19.978Z{"msg":"ds_connection connected from Ok(127.0.0.1:63013)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.121988324Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3332026-08-11T23:02:19.978Z{"msg":"ds_connection connected from Ok(127.0.0.1:54231)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.122007372Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3342026-08-11T23:02:19.978Z{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.122659431Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"0","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3352026-08-11T23:02:19.978Z{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.122682209Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"1","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3362026-08-11T23:02:19.978Z{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.122785687Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"2","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3372026-08-11T23:02:19.978Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.12285224Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3382026-08-11T23:02:19.978Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.122867539Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3392026-08-11T23:02:19.978Z{"msg":"[0]R flush_numbers: [4001, 4001, 4001, 4001, 4001]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.122918473Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3402026-08-11T23:02:19.978Z{"msg":"[0]R generation: [31, 31, 31, 31, 31]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.122933381Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3412026-08-11T23:02:19.978Z{"msg":"[0]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.12294381Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3422026-08-11T23:02:19.978Z{"msg":"[1]R flush_numbers: [4001, 4001, 4001, 4001, 4001]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.122958309Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3432026-08-11T23:02:19.978Z{"msg":"[1]R generation: [31, 31, 31, 31, 31]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.122968708Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3442026-08-11T23:02:19.978Z{"msg":"[1]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.122982856Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3452026-08-11T23:02:19.978Z{"msg":"[2]R flush_numbers: [4001, 4001, 4001, 4001, 4001]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123002084Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3462026-08-11T23:02:19.978Z{"msg":"[2]R generation: [31, 31, 31, 31, 31]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123011953Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3472026-08-11T23:02:19.978Z{"msg":"[2]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123026571Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3482026-08-11T23:02:19.978Z{"msg":"Max found gen is 32","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123043819Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3492026-08-11T23:02:19.978Z{"msg":"Generation requested: 32 >= found:32","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123053778Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3502026-08-11T23:02:19.978Z{"msg":"Next flush: 4002","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123067297Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3512026-08-11T23:02:19.978Z{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123077156Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3522026-08-11T23:02:19.978Z{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123091274Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3532026-08-11T23:02:19.978Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123100693Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"0","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3542026-08-11T23:02:19.979Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123114782Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"1","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3552026-08-11T23:02:19.979Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123124691Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"2","":"downstairs","session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3562026-08-11T23:02:19.979Z{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123138979Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3572026-08-11T23:02:19.979Z{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123148728Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3582026-08-11T23:02:19.979Z{"msg":"ddbcb2e3-95b4-436c-ad07-a8ed16556d9c is now active with session: 9e983874-62fd-4aa8-952a-968c19290a69","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123163597Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3592026-08-11T23:02:19.979Z{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123172796Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9e983874-62fd-4aa8-952a-968c19290a69"}
3602026-08-11T23:02:19.979Z{"msg":"The guest has finished waiting for activation with:32","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123186554Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185}
3612026-08-11T23:02:19.979Z{"msg":"Activated sub_volume ddbcb2e3-95b4-436c-ad07-a8ed16556d9c","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123194823Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185}
3622026-08-11T23:02:19.979Z{"msg":"The guest has requested activation with gen:32","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123208062Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185}
3632026-08-11T23:02:19.979Z{"msg":"ddbcb2e3-95b4-436c-ad07-a8ed16556d9c active request set","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123216021Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3642026-08-11T23:02:19.979Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.12322514Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"0","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3652026-08-11T23:02:19.979Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123242258Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"1","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3662026-08-11T23:02:19.980Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123252247Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"2","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3672026-08-11T23:02:19.980Z{"msg":"connecting to 127.0.0.1:8860","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123261966Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3682026-08-11T23:02:19.980Z{"msg":"connecting to 127.0.0.1:8840","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123272325Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3692026-08-11T23:02:19.980Z{"msg":"connecting to 127.0.0.1:8850","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123310381Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3702026-08-11T23:02:19.980Z{"msg":"ds_connection connected from Ok(127.0.0.1:48186)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123321559Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3712026-08-11T23:02:19.980Z{"msg":"ds_connection connected from Ok(127.0.0.1:62177)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123332488Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3722026-08-11T23:02:19.980Z{"msg":"ds_connection connected from Ok(127.0.0.1:62613)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123364425Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3732026-08-11T23:02:19.984Z{"msg":"downstairs client at Some(127.0.0.1:8860) has region UUID 12345678-0000-0000-0000-000000008860","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123634336Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"2","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3742026-08-11T23:02:19.985Z{"msg":"downstairs client at Some(127.0.0.1:8850) has region UUID 12345678-0000-0000-0000-000000008850","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.12368178Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"1","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3752026-08-11T23:02:19.985Z{"msg":"downstairs client at Some(127.0.0.1:8840) has region UUID 12345678-0000-0000-0000-000000008840","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123705858Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"0","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3762026-08-11T23:02:19.985Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123763462Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3772026-08-11T23:02:19.985Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.12377745Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3782026-08-11T23:02:19.985Z{"msg":"[0]R flush_numbers: [5322, 5322, 5322, 5322, 5322]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123914905Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3792026-08-11T23:02:19.985Z{"msg":"[0]R generation: [31, 31, 31, 31, 31]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123930074Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3802026-08-11T23:02:19.985Z{"msg":"[0]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123940582Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3812026-08-11T23:02:19.985Z{"msg":"[1]R flush_numbers: [5322, 5322, 5322, 5322, 5322]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123950671Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3822026-08-11T23:02:19.985Z{"msg":"[1]R generation: [31, 31, 31, 31, 31]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.12396052Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3832026-08-11T23:02:19.985Z{"msg":"[1]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123975319Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3842026-08-11T23:02:19.985Z{"msg":"[2]R flush_numbers: [5322, 5322, 5322, 5322, 5322]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123985428Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3852026-08-11T23:02:19.985Z{"msg":"[2]R generation: [31, 31, 31, 31, 31]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.123999526Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3862026-08-11T23:02:19.985Z{"msg":"[2]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.124009735Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3872026-08-11T23:02:19.985Z{"msg":"Max found gen is 32","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.124023713Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3882026-08-11T23:02:19.985Z{"msg":"Generation requested: 32 >= found:32","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.124033412Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3892026-08-11T23:02:19.985Z{"msg":"Next flush: 5323","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.124046911Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3902026-08-11T23:02:19.985Z{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.12405618Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"":"downstairs","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3912026-08-11T23:02:19.985Z{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.124070138Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3922026-08-11T23:02:19.985Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.124079107Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"0","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3932026-08-11T23:02:19.985Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.124093146Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"1","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3942026-08-11T23:02:19.985Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.124102795Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"client":"2","":"downstairs","session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3952026-08-11T23:02:19.986Z{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.124116793Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3962026-08-11T23:02:19.986Z{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.124126102Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3972026-08-11T23:02:19.986Z{"msg":"ddbcb2e3-95b4-436c-ad07-a8ed16556d9c is now active with session: 9b028c21-aa6f-4d43-b514-73333eb701d8","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.124140601Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3982026-08-11T23:02:19.986Z{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.12414934Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"session_id":"9b028c21-aa6f-4d43-b514-73333eb701d8"}
3992026-08-11T23:02:19.986Z{"msg":"The guest has finished waiting for activation with:32","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.124162938Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185}
4002026-08-11T23:02:19.986Z{"msg":"Activated sub_volume ddbcb2e3-95b4-436c-ad07-a8ed16556d9c","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.124171567Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185}
4012026-08-11T23:02:19.986Z{"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.124184166Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"task":"crutest"}
4022026-08-11T23:02:19.986ZDisk: sv:2 bs:4096 ts:409600 tb:100 max_io:100 or 409600
4032026-08-11T23:02:19.986ZLoading write count information from file "/var/tmp/test_up-build/verify_file"
4042026-08-11T23:02:19.986Z{"msg":"Fill test","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.124690721Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"task":"crutest"}
4052026-08-11T23:02:20.003ZRead and Verify all blocks (0..100 range:false)
4062026-08-11T23:02:20.064Z{"msg":"Wrote out file \"/var/tmp/test_up-build/verify_file\"","v":0,"name":"crucible","level":30,----------------------------------------------------------------
4072026-08-11T23:02:20.064Z"time":"2026-08-11T23:02:18.196837708Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid" Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:1
4082026-08-11T23:02:20.065Z:1185,"States:task":"crutest" Active} Active
4092026-08-11T23:02:20.065Z Active
4102026-08-11T23:02:20.065ZJOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4112026-08-11T23:02:20.074Z{"msg":" 1002 Acked Read 50CLIENT: Tests done. All submitted work has been ACK'd"," Donev Done" Done: false0
4122026-08-11T23:02:20.074Z,"name":"crucible","level": STATES DS:0 DS:1 DS:2 TOTAL
4132026-08-11T23:02:20.074Z30 Sent 0 0 0 0
4142026-08-11T23:02:20.074Z Done 1 1 1 3
4152026-08-11T23:02:20.074Z, Skipped " 0 time 0 " 0 : 0
4162026-08-11T23:02:20.074Z" Error 0 0 0 0
4172026-08-11T23:02:20.074Z2026-08-11T23:02:18.196872384Z"Last Flush: ,"Some(JobId(1001)) hostname"Some(JobId(1001)) :Some(JobId(1001)) "
4182026-08-11T23:02:20.074Zw-01KZSGPBC51FTS1QWF7FE5A8CR"Downstairs last five completed:,"pid":1185,"task 1001": 1000"
4192026-08-11T23:02:20.074Zcrutest"}
4202026-08-11T23:02:20.074ZUpstairs last five completed: 1002 1001 1000
4212026-08-11T23:02:20.074Z----------------------------------------------------------------
4222026-08-11T23:02:20.074Z Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:1
4232026-08-11T23:02:20.074ZStates: Active Active Active
4242026-08-11T23:02:20.074ZJOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4252026-08-11T23:02:20.074Z 1002 Acked Read 50 Done Done Done false
4262026-08-11T23:02:20.074Z STATES DS:0 DS:1 DS:2 TOTAL
4272026-08-11T23:02:20.074Z Sent 0 0 0 0
4282026-08-11T23:02:20.074Z Done 1 1 1 3
4292026-08-11T23:02:20.074Z Skipped 0 0 0 0
4302026-08-11T23:02:20.074Z Error 0 0 0 0
4312026-08-11T23:02:20.074ZLast Flush: Some(JobId(1001)) Some(JobId(1001)) Some(JobId(1001))
4322026-08-11T23:02:20.074ZDownstairs last five completed: 1001 1000
4332026-08-11T23:02:20.074ZUpstairs last five completed: 1002 1001 1000
4342026-08-11T23:02:20.074Z{"msg":"CLIENT: Up:0 ds:2 act:6","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:18.196950805Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"task":"crutest"}
4352026-08-11T23:02:24.049Z----------------------------------------------------------------
4362026-08-11T23:02:24.049Z Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:0
4372026-08-11T23:02:24.049ZDownstairs last five completed: 1003 1002 1001 1000
4382026-08-11T23:02:24.049ZUpstairs last five completed: 1003 1002 1001 1000
4392026-08-11T23:02:24.052Z----------------------------------------------------------------
4402026-08-11T23:02:24.052Z Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:0
4412026-08-11T23:02:24.052ZDownstairs last five completed: 1003 1002 1001 1000
4422026-08-11T23:02:24.052ZUpstairs last five completed: 1003 1002 1001 1000
4432026-08-11T23:02:24.052Z{"msg":"CLIENT: Up:0 ds:0 act:6","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.198000391Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"task":"crutest"}
4442026-08-11T23:02:24.052Z{"msg":"CLIENT: All crucible jobs finished, exiting program","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.198062154Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185,"task":"crutest"}
4452026-08-11T23:02:24.052Z{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-08-11T23:02:22.198078532Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1185}
4462026-08-11T23:02:24.056ZRepair part 1 passed
4472026-08-11T23:02:24.056Z
4482026-08-11T23:02:24.056ZKill the current downstairs
4492026-08-11T23:02:24.098ZDownstairs 2 stopped
4502026-08-11T23:02:24.098Zmv /var/tmp/test_up-build/8830 /var/tmp/test_up-build/new
4512026-08-11T23:02:24.103ZNow put back the original so we have a mismatch
4522026-08-11T23:02:24.103Zmv /var/tmp/test_up-build/previous /var/tmp/test_up-build/8830
4532026-08-11T23:02:24.103ZRestart downstairs with old directory
4542026-08-11T23:02:24.114Z/var/tmp/bins/crucible-downstairs dump --no-color -d /var/tmp/test_up-build/8810 -d /var/tmp/test_up-build/8820 -d /var/tmp/test_up-build/8830
4552026-08-11T23:02:24.127ZEXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
4562026-08-11T23:02:24.131Z 0 000-009 32 32 31 4002 4002 4001 F F F <---
4572026-08-11T23:02:24.131Z 1 010-019 32 32 31 4002 4002 4001 F F F <---
4582026-08-11T23:02:24.131Z 2 020-029 32 32 31 4002 4002 4001 F F F <---
4592026-08-11T23:02:24.131Z 3 030-039 32 32 31 4002 4002 4001 F F F <---
4602026-08-11T23:02:24.131Z 4 040-049 32 32 31 4002 4002 4001 F F F <---
4612026-08-11T23:02:24.131ZMax gen: 32, Max flush: 4002
4622026-08-11T23:02:24.132ZError: Difference in extent metadata found!
4632026-08-11T23:02:24.132Zdump test found error as expected
4642026-08-11T23:02:24.132Z
4652026-08-11T23:02:24.132Z
4662026-08-11T23:02:24.132Z/var/tmp/bins/crutest verify --range -g 33 -q --key GrUTHX1kzQjaDvu0ajWJsunlf244tzyYbPQ+zJmQJVw= --dsc 127.0.0.1:9998 --verify-out /var/tmp/test_up-build/verify_file --verify-in /var/tmp/test_up-build/verify_file
4672026-08-11T23:02:24.147Z{"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 10, extent_count: 5 }","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.296116065Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"task":"crutest"}
4682026-08-11T23:02:24.150Z{"msg":"dsc has 6 regions. This means 2 sub_volumes","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.296515262Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"task":"crutest"}
4692026-08-11T23:02:24.150Z{"msg":"SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.297529382Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"task":"crutest"}
4702026-08-11T23:02:24.150Z{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.297760127Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193}
4712026-08-11T23:02:24.150Z{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"3d9051b3f6e9b5b4034790013a3d1fc35523d0f6\",\n git_commit_timestamp: \"2026-08-11T22:43:07.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.297785504Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193}
4722026-08-11T23:02:24.150Z{"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.297794693Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193}
4732026-08-11T23:02:24.150Z{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 10, shift: 12 }, extent_count: 5, uuid: 00000000-0000-0000-0000-000000000000, encrypted: true, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.297807602Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193}
4742026-08-11T23:02:24.151Z{"msg":"Crucible 56fd8091-3a23-4170-bdb4-3e3d80d42b56 has session id: 528357a6-befa-46ab-92aa-de88ee14cdcc","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.298215267Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
4752026-08-11T23:02:24.151Z{"msg":"Upstairs opts: Upstairs UUID: 56fd8091-3a23-4170-bdb4-3e3d80d42b56, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: true, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.298239015Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
4762026-08-11T23:02:24.151Z{"msg":"Crucible stats registered with UUID: 56fd8091-3a23-4170-bdb4-3e3d80d42b56","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.298249554Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
4772026-08-11T23:02:24.151Z{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-08-11T23:02:22.298258753Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
4782026-08-11T23:02:24.151Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.298396178Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"1","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
4792026-08-11T23:02:24.151Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.298429154Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
4802026-08-11T23:02:24.151Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.298440953Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"0","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
4812026-08-11T23:02:24.154Z{"msg":"SV 1 has targets: [127.0.0.1:8840, 127.0.0.1:8850, 127.0.0.1:8860]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299318568Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"task":"crutest"}
4822026-08-11T23:02:24.154Z{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299368273Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193}
4832026-08-11T23:02:24.154Z{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"3d9051b3f6e9b5b4034790013a3d1fc35523d0f6\",\n git_commit_timestamp: \"2026-08-11T22:43:07.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.29938986Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193}
4842026-08-11T23:02:24.154Z{"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299399189Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193}
4852026-08-11T23:02:24.154Z{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 10, shift: 12 }, extent_count: 5, uuid: 00000000-0000-0000-0000-000000000000, encrypted: true, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299411398Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193}
4862026-08-11T23:02:24.154Z{"msg":"Crucible 56fd8091-3a23-4170-bdb4-3e3d80d42b56 has session id: a7899bf5-8818-4404-b3d4-8b10edfc5545","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299594358Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
4872026-08-11T23:02:24.154Z{"msg":"Upstairs opts: Upstairs UUID: 56fd8091-3a23-4170-bdb4-3e3d80d42b56, Targets: [127.0.0.1:8840, 127.0.0.1:8850, 127.0.0.1:8860], lossy: false, flush_timeout: None, key populated: true, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299617796Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
4882026-08-11T23:02:24.154Z{"msg":"Crucible stats registered with UUID: 56fd8091-3a23-4170-bdb4-3e3d80d42b56","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299627904Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
4892026-08-11T23:02:24.154Z{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-08-11T23:02:22.299641583Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
4902026-08-11T23:02:24.154Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299655112Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"1","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
4912026-08-11T23:02:24.155Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.29967009Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"0","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
4922026-08-11T23:02:24.155Z{"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299680729Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"2","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
4932026-08-11T23:02:24.155Z{"msg":"Downstairs in volume = 6","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299792617Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"task":"crutest"}
4942026-08-11T23:02:24.155Z{"msg":"The guest has requested activation with gen:33","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299811845Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193}
4952026-08-11T23:02:24.155Z{"msg":"56fd8091-3a23-4170-bdb4-3e3d80d42b56 active request set","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299824603Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
4962026-08-11T23:02:24.155Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299833632Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"0","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
4972026-08-11T23:02:24.155Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299848061Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"1","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
4982026-08-11T23:02:24.155Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.29985792Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
4992026-08-11T23:02:24.155Z{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299867738Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5002026-08-11T23:02:24.155Z{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299882167Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"1","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5012026-08-11T23:02:24.155Z{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299892596Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"0","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5022026-08-11T23:02:24.155Z{"msg":"ds_connection connected from Ok(127.0.0.1:35415)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299921763Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"1","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5032026-08-11T23:02:24.155Z{"msg":"ds_connection connected from Ok(127.0.0.1:40133)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.299933241Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"0","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5042026-08-11T23:02:24.155Z{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2026-08-11T23:02:22.300011243Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5052026-08-11T23:02:24.155Z{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2026-08-11T23:02:22.30004234Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5062026-08-11T23:02:24.155Z{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2026-08-11T23:02:22.300055898Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5072026-08-11T23:02:24.155Z{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.300070906Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5082026-08-11T23:02:24.155Z{"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.30012948Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5092026-08-11T23:02:24.155Z{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.300539006Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"0","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5102026-08-11T23:02:24.155Z{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.300578671Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"1","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5112026-08-11T23:02:24.155Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.300698339Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5122026-08-11T23:02:24.155Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:22.300712957Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5132026-08-11T23:02:34.153Z{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.300468322Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5142026-08-11T23:02:34.153Z{"msg":"ds_connection connected from Ok(127.0.0.1:57627)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.300533415Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5152026-08-11T23:02:34.158Z{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.301644625Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5162026-08-11T23:02:34.158Z{"msg":"[0]R flush_numbers: [4002, 4002, 4002, 4002, 4002]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.301889519Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5172026-08-11T23:02:34.158Z{"msg":"[0]R generation: [32, 32, 32, 32, 32]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.301911956Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5182026-08-11T23:02:34.158Z{"msg":"[0]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.301927764Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5192026-08-11T23:02:34.158Z{"msg":"[1]R flush_numbers: [4002, 4002, 4002, 4002, 4002]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.301948922Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5202026-08-11T23:02:34.158Z{"msg":"[1]R generation: [32, 32, 32, 32, 32]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.301963521Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5212026-08-11T23:02:34.158Z{"msg":"[1]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.301983848Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5222026-08-11T23:02:34.159Z{"msg":"[2]R flush_numbers: [4001, 4001, 4001, 4001, 4001]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.301998837Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5232026-08-11T23:02:34.159Z{"msg":"[2]R generation: [31, 31, 31, 31, 31]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302012835Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5242026-08-11T23:02:34.159Z{"msg":"[2]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302033533Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5252026-08-11T23:02:34.159Z{"msg":"Max found gen is 33","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302047501Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5262026-08-11T23:02:34.159Z{"msg":"Generation requested: 33 >= found:33","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302068109Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5272026-08-11T23:02:34.159Z{"msg":"Next flush: 4003","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302082218Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5282026-08-11T23:02:34.159Z{"msg":"extent 0 needs reconciliation: ClientMap(ClientData([Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 31, flush: 4001, dirty: false })]))","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302103055Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"mend","":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5292026-08-11T23:02:34.159Z{"msg":"extent:0 ClientMap(ClientData([Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 31, flush: 4001, dirty: false })])) => 0","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302130432Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"mend","":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5302026-08-11T23:02:34.159Z{"msg":"found dest for source 0 for extent at index 0 => [ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302145921Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"mend","":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5312026-08-11T23:02:34.159Z{"msg":"extent 1 needs reconciliation: ClientMap(ClientData([Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 31, flush: 4001, dirty: false })]))","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302172378Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"mend","":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5322026-08-11T23:02:34.159Z{"msg":"extent:1 ClientMap(ClientData([Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 31, flush: 4001, dirty: false })])) => 0","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302204054Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"mend","":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5332026-08-11T23:02:34.159Z{"msg":"found dest for source 0 for extent at index 1 => [ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302225792Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"mend","":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5342026-08-11T23:02:34.159Z{"msg":"extent 2 needs reconciliation: ClientMap(ClientData([Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 31, flush: 4001, dirty: false })]))","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.30224616Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"mend","":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5352026-08-11T23:02:34.159Z{"msg":"extent:2 ClientMap(ClientData([Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 31, flush: 4001, dirty: false })])) => 0","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302272287Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"mend","":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5362026-08-11T23:02:34.159Z{"msg":"found dest for source 0 for extent at index 2 => [ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302287685Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"mend","":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5372026-08-11T23:02:34.159Z{"msg":"extent 3 needs reconciliation: ClientMap(ClientData([Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 31, flush: 4001, dirty: false })]))","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302313743Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"mend","":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5382026-08-11T23:02:34.159Z{"msg":"extent:3 ClientMap(ClientData([Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 31, flush: 4001, dirty: false })])) => 0","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302345479Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"mend","":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5392026-08-11T23:02:34.159Z{"msg":"found dest for source 0 for extent at index 3 => [ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302366697Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"mend","":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5402026-08-11T23:02:34.159Z{"msg":"extent 4 needs reconciliation: ClientMap(ClientData([Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 31, flush: 4001, dirty: false })]))","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302386895Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"mend","":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5412026-08-11T23:02:34.159Z{"msg":"extent:4 ClientMap(ClientData([Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 32, flush: 4002, dirty: false }), Some(ExtentMetadata { generation: 31, flush: 4001, dirty: false })])) => 0","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302412602Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"mend","":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5422026-08-11T23:02:34.159Z{"msg":"found dest for source 0 for extent at index 4 => [ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302438889Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"mend","":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5432026-08-11T23:02:34.159Z{"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302462906Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"0","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5442026-08-11T23:02:34.159Z{"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302478555Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"1","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5452026-08-11T23:02:34.159Z{"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302504722Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5462026-08-11T23:02:34.159Z{"msg":"Full repair list: {ExtentId(1): ExtentFix { source: ClientId(0), dest: [ClientId(2)] }, ExtentId(4): ExtentFix { source: ClientId(0), dest: [ClientId(2)] }, ExtentId(0): ExtentFix { source: ClientId(0), dest: [ClientId(2)] }, ExtentId(2): ExtentFix { source: ClientId(0), dest: [ClientId(2)] }, ExtentId(3): ExtentFix { source: ClientId(0), dest: [ClientId(2)] }}","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302533649Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5472026-08-11T23:02:34.159Z{"msg":"Task list: [ReconcileIO { id: ReconciliationId(0), op: ExtentFlush { repair_id: ReconciliationId(0), extent_id: ExtentId(1), client_id: ClientId(0), flush_number: 4003, gen_number: 33 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(1), op: ExtentClose { repair_id: ReconciliationId(1), extent_id: ExtentId(1) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(2), op: ExtentRepair { repair_id: ReconciliationId(2), extent_id: ExtentId(1), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(3), op: ExtentReopen { repair_id: ReconciliationId(3), extent_id: ExtentId(1) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(4), op: ExtentFlush { repair_id: ReconciliationId(4), extent_id: ExtentId(4), client_id: ClientId(0), flush_number: 4003, gen_number: 33 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(5), op: ExtentClose { repair_id: ReconciliationId(5), extent_id: ExtentId(4) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(6), op: ExtentRepair { repair_id: ReconciliationId(6), extent_id: ExtentId(4), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(7), op: ExtentReopen { repair_id: ReconciliationId(7), extent_id: ExtentId(4) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(8), op: ExtentFlush { repair_id: ReconciliationId(8), extent_id: ExtentId(0), client_id: ClientId(0), flush_number: 4003, gen_number: 33 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(9), op: ExtentClose { repair_id: ReconciliationId(9), extent_id: ExtentId(0) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(10), op: ExtentRepair { repair_id: ReconciliationId(10), extent_id: ExtentId(0), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(11), op: ExtentReopen { repair_id: ReconciliationId(11), extent_id: ExtentId(0) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(12), op: ExtentFlush { repair_id: ReconciliationId(12), extent_id: ExtentId(2), client_id: ClientId(0), flush_number: 4003, gen_number: 33 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(13), op: ExtentClose { repair_id: ReconciliationId(13), extent_id: ExtentId(2) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(14), op: ExtentRepair { repair_id: ReconciliationId(14), extent_id: ExtentId(2), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(15), op: ExtentReopen { repair_id: ReconciliationId(15), extent_id: ExtentId(2) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(16), op: ExtentFlush { repair_id: ReconciliationId(16), extent_id: ExtentId(3), client_id: ClientId(0), flush_number: 4003, gen_number: 33 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(17), op: ExtentClose { repair_id: ReconciliationId(17), extent_id: ExtentId(3) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(18), op: ExtentRepair { repair_id: ReconciliationId(18), extent_id: ExtentId(3), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(19), op: ExtentReopen { repair_id: ReconciliationId(19), extent_id: ExtentId(3) }, state: ClientData([New, New, New]) }]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302667334Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5482026-08-11T23:02:34.159Z{"msg":"starting reconciliation b7f2d38b-f921-448f-9b28-e305fa442ed6: found 20 extents that need repair","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.302683803Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5492026-08-11T23:02:34.159Z{"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.30270414Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5502026-08-11T23:02:34.167Z{"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.314484874Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5512026-08-11T23:02:34.170Z{"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.316624163Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5522026-08-11T23:02:34.173Z{"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.318513438Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5532026-08-11T23:02:34.173Z{"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.320420862Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5542026-08-11T23:02:34.177Z{"msg":"done with reconciliation","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322122937Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5552026-08-11T23:02:34.177Z{"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322142995Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"0","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5562026-08-11T23:02:34.177Z{"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322153614Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"1","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5572026-08-11T23:02:34.177Z{"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322167892Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"2","":"downstairs","session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5582026-08-11T23:02:34.177Z{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322178141Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5592026-08-11T23:02:34.177Z{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.32219213Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5602026-08-11T23:02:34.177Z{"msg":"56fd8091-3a23-4170-bdb4-3e3d80d42b56 is now active with session: 528357a6-befa-46ab-92aa-de88ee14cdcc","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322202729Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"528357a6-befa-46ab-92aa-de88ee14cdcc"}
5612026-08-11T23:02:34.177Z{"msg":"The guest has finished waiting for activation with:33","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322216327Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193}
5622026-08-11T23:02:34.177Z{"msg":"Activated sub_volume 56fd8091-3a23-4170-bdb4-3e3d80d42b56","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322224686Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193}
5632026-08-11T23:02:34.177Z{"msg":"The guest has requested activation with gen:33","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322289659Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193}
5642026-08-11T23:02:34.177Z{"msg":"56fd8091-3a23-4170-bdb4-3e3d80d42b56 active request set","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322302858Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5652026-08-11T23:02:34.177Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322312287Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"0","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5662026-08-11T23:02:34.177Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322330235Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"1","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5672026-08-11T23:02:34.177Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322344183Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"2","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5682026-08-11T23:02:34.177Z{"msg":"connecting to 127.0.0.1:8860","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322353702Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"2","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5692026-08-11T23:02:34.177Z{"msg":"connecting to 127.0.0.1:8840","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322368111Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"0","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5702026-08-11T23:02:34.177Z{"msg":"connecting to 127.0.0.1:8850","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.32237822Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"1","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5712026-08-11T23:02:34.177Z{"msg":"ds_connection connected from Ok(127.0.0.1:56516)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322391748Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"2","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5722026-08-11T23:02:34.177Z{"msg":"ds_connection connected from Ok(127.0.0.1:56812)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322402437Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"0","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5732026-08-11T23:02:34.177Z{"msg":"ds_connection connected from Ok(127.0.0.1:47751)","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322412856Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"io task","client":"1","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5742026-08-11T23:02:34.177Z{"msg":"downstairs client at Some(127.0.0.1:8860) has region UUID 12345678-0000-0000-0000-000000008860","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322712253Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"2","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5752026-08-11T23:02:34.177Z{"msg":"downstairs client at Some(127.0.0.1:8850) has region UUID 12345678-0000-0000-0000-000000008850","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.32274556Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"1","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5762026-08-11T23:02:34.178Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322881265Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5772026-08-11T23:02:34.178Z{"msg":"downstairs client at Some(127.0.0.1:8840) has region UUID 12345678-0000-0000-0000-000000008840","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322910282Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"0","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5782026-08-11T23:02:34.178Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.322921501Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5792026-08-11T23:02:34.178Z{"msg":"[0]R flush_numbers: [5323, 5323, 5323, 5323, 5323]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323123629Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5802026-08-11T23:02:34.178Z{"msg":"[0]R generation: [32, 32, 32, 32, 32]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323139597Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5812026-08-11T23:02:34.178Z{"msg":"[0]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323149696Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5822026-08-11T23:02:34.178Z{"msg":"[1]R flush_numbers: [5323, 5323, 5323, 5323, 5323]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323164404Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5832026-08-11T23:02:34.178Z{"msg":"[1]R generation: [32, 32, 32, 32, 32]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323174403Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5842026-08-11T23:02:34.178Z{"msg":"[1]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323188642Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5852026-08-11T23:02:34.178Z{"msg":"[2]R flush_numbers: [5323, 5323, 5323, 5323, 5323]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323198611Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5862026-08-11T23:02:34.178Z{"msg":"[2]R generation: [32, 32, 32, 32, 32]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323212769Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5872026-08-11T23:02:34.178Z{"msg":"[2]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323222618Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5882026-08-11T23:02:34.178Z{"msg":"Max found gen is 33","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323232627Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5892026-08-11T23:02:34.178Z{"msg":"Generation requested: 33 >= found:33","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323246946Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5902026-08-11T23:02:34.178Z{"msg":"Next flush: 5324","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323256265Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5912026-08-11T23:02:34.178Z{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323270313Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"":"downstairs","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5922026-08-11T23:02:34.178Z{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323280152Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5932026-08-11T23:02:34.178Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323293341Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"0","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5942026-08-11T23:02:34.178Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323302909Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"1","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5952026-08-11T23:02:34.178Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323316878Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"client":"2","":"downstairs","session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5962026-08-11T23:02:34.178Z{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323326677Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5972026-08-11T23:02:34.178Z{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323340325Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5982026-08-11T23:02:34.178Z{"msg":"56fd8091-3a23-4170-bdb4-3e3d80d42b56 is now active with session: a7899bf5-8818-4404-b3d4-8b10edfc5545","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323350484Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
5992026-08-11T23:02:34.178Z{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323363353Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"session_id":"a7899bf5-8818-4404-b3d4-8b10edfc5545"}
6002026-08-11T23:02:34.178Z{"msg":"The guest has finished waiting for activation with:33","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323372672Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193}
6012026-08-11T23:02:34.178Z{"msg":"Activated sub_volume 56fd8091-3a23-4170-bdb4-3e3d80d42b56","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.323385131Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193}
6022026-08-11T23:02:34.178Z{"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.32339351Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"task":"crutest"}
6032026-08-11T23:02:34.178ZDisk: sv:2 bs:4096 ts:409600 tb:100 max_io:100 or 409600
6042026-08-11T23:02:34.178ZLoading write count information from file "/var/tmp/test_up-build/verify_file"
6052026-08-11T23:02:34.178ZRead and Verify all blocks (0..100 range:true)
6062026-08-11T23:02:34.224Z{"msg":"Wrote out file \"/var/tmp/test_up-build/verify_file\"","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.37205261Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"task":"crutest"}
6072026-08-11T23:02:34.224Z{"msg":"Verify test completed","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.372077557Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"task":"crutest"}
6082026-08-11T23:02:34.224Z{"msg":"Wrote out file \"/var/tmp/test_up-build/verify_file\"","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.372091316Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"task":"crutest"}
6092026-08-11T23:02:34.228Z{"msg":"CLIENT: Tests done. All submitted work has been ACK'd","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.372099925Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"task":"crutest"}
6102026-08-11T23:02:34.228Z----------------------------------------------------------------
6112026-08-11T23:02:34.228Z Crucible gen:33 GIO:true work queues: Upstairs:0 downstairs:1
6122026-08-11T23:02:34.228ZStates: Active Active Active
6132026-08-11T23:02:34.228ZJOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6142026-08-11T23:02:34.228Z 1000 Acked Read 50 Done Done Done false
6152026-08-11T23:02:34.228Z STATES DS:0 DS:1 DS:2 TOTAL
6162026-08-11T23:02:34.228Z Sent 0 0 0 0
6172026-08-11T23:02:34.228Z Done 1 1 1 3
6182026-08-11T23:02:34.228Z Skipped 0 0 0 0
6192026-08-11T23:02:34.228Z Error 0 0 0 0
6202026-08-11T23:02:34.228ZLast Flush: None None None
6212026-08-11T23:02:34.228ZDownstairs last five completed:
6222026-08-11T23:02:34.228ZUpstairs last five completed: 1000
6232026-08-11T23:02:34.228Z----------------------------------------------------------------
6242026-08-11T23:02:34.228Z Crucible gen:33 GIO:true work queues: Upstairs:0 downstairs:1
6252026-08-11T23:02:34.228ZStates: Active Active Active
6262026-08-11T23:02:34.228ZJOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6272026-08-11T23:02:34.228Z 1000 Acked Read 50 Done Done Done false
6282026-08-11T23:02:34.228Z STATES DS:0 DS:1 DS:2 TOTAL
6292026-08-11T23:02:34.228Z Sent 0 0 0 0
6302026-08-11T23:02:34.228Z Done 1 1 1 3
6312026-08-11T23:02:34.228Z Skipped 0 0 0 0
6322026-08-11T23:02:34.228Z Error 0 0 0 0
6332026-08-11T23:02:34.229ZLast Flush: None None None
6342026-08-11T23:02:34.229ZDownstairs last five completed:
6352026-08-11T23:02:34.229ZUpstairs last five completed: 1000
6362026-08-11T23:02:34.229Z{"msg":"CLIENT: Up:0 ds:2 act:6","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:32.372279465Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"task":"crutest"}
6372026-08-11T23:02:38.227Z----------------------------------------------------------------
6382026-08-11T23:02:38.227Z Crucible gen:33 GIO:true work queues: Upstairs:0 downstairs:0
6392026-08-11T23:02:38.227ZDownstairs last five completed: 1001 1000
6402026-08-11T23:02:38.227ZUpstairs last five completed: 1001 1000
6412026-08-11T23:02:38.230Z----------------------------------------------------------------
6422026-08-11T23:02:38.230Z Crucible gen:33 GIO:true work queues: Upstairs:0 downstairs:0
6432026-08-11T23:02:38.230ZDownstairs last five completed: 1001 1000
6442026-08-11T23:02:38.230ZUpstairs last five completed: 1001 1000
6452026-08-11T23:02:38.230Z{"msg":"CLIENT: Up:0 ds:0 act:6","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:36.373889511Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"task":"crutest"}
6462026-08-11T23:02:38.230Z{"msg":"CLIENT: All crucible jobs finished, exiting program","v":0,"name":"crucible","level":30,"time":"2026-08-11T23:02:36.373943835Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193,"task":"crutest"}
6472026-08-11T23:02:38.230Z{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-08-11T23:02:36.373958813Z","hostname":"w-01KZSGPBC51FTS1QWF7FE5A8CR","pid":1193}
6482026-08-11T23:02:38.233ZRepair part 2 passed
6492026-08-11T23:02:38.233Z/var/tmp/bins/crucible-downstairs dump --no-color -d /var/tmp/test_up-build/8810 -d /var/tmp/test_up-build/8820 -d /var/tmp/test_up-build/8830
6502026-08-11T23:02:38.247ZEXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
6512026-08-11T23:02:38.247Z 0 000-009 32 32 32 4002 4002 4002 F F F
6522026-08-11T23:02:38.247Z 1 010-019 32 32 32 4002 4002 4002 F F F
6532026-08-11T23:02:38.250Z 2 020-029 32 32 32 4002 4002 4002 F F F
6542026-08-11T23:02:38.250Z 3 030-039 32 32 32 4002 4002 4002 F F F
6552026-08-11T23:02:38.250Z 4 040-049 32 32 32 4002 4002 4002 F F F
6562026-08-11T23:02:38.250ZMax gen: 32, Max flush: 4002
6572026-08-11T23:02:38.250Zdump test passed
6582026-08-11T23:02:38.251Z/var/tmp/bins/crucible-downstairs dump --no-color -d /var/tmp/test_up-build/8810 -d /var/tmp/test_up-build/8820 -d /var/tmp/test_up-build/8830 -e 1
6592026-08-11T23:02:38.262Z Extent 1
6602026-08-11T23:02:38.265ZGEN 32 32 32
6612026-08-11T23:02:38.265ZFLUSH_ID 4002 4002 4002
6622026-08-11T23:02:38.265ZDIRTY
6632026-08-11T23:02:38.265Z
6642026-08-11T23:02:38.265ZBLOCK D0 D1 D2 C0 C1 C2 DIFF
6652026-08-11T23:02:38.265Z010 A A A A A A
6662026-08-11T23:02:38.265Z011 A A A A A A
6672026-08-11T23:02:38.265Z012 A A A A A A
6682026-08-11T23:02:38.265Z013 A A A A A A
6692026-08-11T23:02:38.268Z014 A A A A A A
6702026-08-11T23:02:38.268Z015 A A A A A A
6712026-08-11T23:02:38.268Z016 A A A A A A
6722026-08-11T23:02:38.268Z017 A A A A A A
6732026-08-11T23:02:38.268Z018 A A A A A A
6742026-08-11T23:02:38.271Z019 A A A A A A
6752026-08-11T23:02:38.271Zdump extent test passed
6762026-08-11T23:02:38.271Z/var/tmp/bins/crucible-downstairs dump --no-color -d /var/tmp/test_up-build/8810 -d /var/tmp/test_up-build/8820 -d /var/tmp/test_up-build/8830 -b 20
6772026-08-11T23:02:38.282ZExtent 2 Block in extent 0 Actual block 20
6782026-08-11T23:02:38.282Z
6792026-08-11T23:02:38.285Z DATA SHA256 VER
6802026-08-11T23:02:38.285Z------ ---------------------------------------------------------------- ---
6812026-08-11T23:02:38.285Z 0 d232dfef95e1a549edd73f301143bb8738f0e3871a53528ebe8e16ce9cda5ff5 A
6822026-08-11T23:02:38.285Z 1 d232dfef95e1a549edd73f301143bb8738f0e3871a53528ebe8e16ce9cda5ff5 A
6832026-08-11T23:02:38.285Z 2 d232dfef95e1a549edd73f301143bb8738f0e3871a53528ebe8e16ce9cda5ff5 A
6842026-08-11T23:02:38.285Z
6852026-08-11T23:02:38.285ZNONCES 0 1 2 DIFF
6862026-08-11T23:02:38.285Z------ ------------------------ ------------------------ ------------------------ -----
6872026-08-11T23:02:38.285Z 0 f8b64aff5fb78f49fba38dcc f8b64aff5fb78f49fba38dcc f8b64aff5fb78f49fba38dcc
6882026-08-11T23:02:38.285Z
6892026-08-11T23:02:38.285Z TAGS 0 1 2 DIFF
6902026-08-11T23:02:38.285Z------ -------------------------------- -------------------------------- -------------------------------- -----
6912026-08-11T23:02:38.285Z 0 5ad361b7e3709b808fb79d566fb30a28 5ad361b7e3709b808fb79d566fb30a28 5ad361b7e3709b808fb79d566fb30a28
6922026-08-11T23:02:38.285Z
6932026-08-11T23:02:38.285ZHASHES 0 1 2 DIFF
6942026-08-11T23:02:38.285Z------ ---------------- ---------------- ---------------- -----
6952026-08-11T23:02:38.285Z
6962026-08-11T23:02:38.289Zdump block test passed
6972026-08-11T23:02:38.289ZInitial upstairs tests have completed, stopping all downstairs
6982026-08-11T23:02:43.307ZCreating 7 larger downstairs regions
6992026-08-11T23:02:43.507ZStarting 7 downstairs
7002026-08-11T23:02:48.513Zdsc restarted at PID: 1209
7012026-08-11T23:02:48.612ZNow do the replace-reconcile test
7022026-08-11T23:02:48.624ZUsing 8870 for the replacement port
7032026-08-11T23:06:25.332ZCompleted test: replace-reconcile
7042026-08-11T23:06:25.335ZNow do the replace-before-active test
7052026-08-11T23:10:20.172ZCompleted test: replace-before-active
7062026-08-11T23:10:20.172ZAll tests have completed, stopping all downstairs
7072026-08-11T23:10:20.177Z
7082026-08-11T23:10:20.177ZAll Tests have passed
7092026-08-11T23:10:20.177Z11:42 Test duration
7102026-08-11T23:10:20.178Z
7112026-08-11T23:10:20.178Zreal 11:41.440066855
7122026-08-11T23:10:20.178Zuser 16:58.374003344
7132026-08-11T23:10:20.178Zsys 11.756480373
7142026-08-11T23:10:20.178Ztrap 0.427665338
7152026-08-11T23:10:20.178Ztflt 0.043402899
7162026-08-11T23:10:20.178Zdflt 0.014395597
7172026-08-11T23:10:20.178Zkflt 0.001133964
7182026-08-11T23:10:20.178Zlock 16:00:36.446539209
7192026-08-11T23:10:20.178Zslp 50:30.277496064
7202026-08-11T23:10:20.178Zlat 1:49.731328175
7212026-08-11T23:10:20.178Zstop 0.051951718
7222026-08-11T23:10:20.178Z+ echo 'test-up-2region-encrypted ends'
7232026-08-11T23:10:20.178Ztest-up-2region-encrypted ends
7242026-08-11T23:10:25.133Zprocess exited: duration 710073 ms, exit code 0
7252026-08-11T23:10:25.133Zexec warning: : stdio descriptors remain open after task exit; waiting 60 seconds for them to close
7262026-08-11T23:11:25.176Zexec warning: : stdout descriptor may be held open by a background process; giving up!
7272026-08-11T23:11:25.176Zexec warning: : stderr descriptor may be held open by a background process; giving up!
 
7282026-08-11T23:11:25.189Zfound 17 output files
7292026-08-11T23:11:25.189Zuploading: /tmp/test_up-build/dsc-out.txt (58386 bytes)
7302026-08-11T23:11:26.211Zuploaded: /tmp/test_up-build/dsc-out.txt
7312026-08-11T23:11:26.211Zuploading: /tmp/test_up-build/test_up_out.txt (23106831 bytes)
7322026-08-11T23:11:27.373Zuploaded: /tmp/test_up-build/test_up_out.txt
7332026-08-11T23:11:27.373Zuploading: /tmp/test_up-build/dsc/downstairs-8810.txt (551000 bytes)
7342026-08-11T23:11:28.393Zuploaded: /tmp/test_up-build/dsc/downstairs-8810.txt
7352026-08-11T23:11:28.393Zuploading: /tmp/test_up-build/dsc/downstairs-8820.txt (548348 bytes)
7362026-08-11T23:11:29.410Zuploaded: /tmp/test_up-build/dsc/downstairs-8820.txt
7372026-08-11T23:11:29.410Zuploading: /tmp/test_up-build/dsc/downstairs-8830.txt (978685 bytes)
7382026-08-11T23:11:30.431Zuploaded: /tmp/test_up-build/dsc/downstairs-8830.txt
7392026-08-11T23:11:30.431Zuploading: /tmp/test_up-build/dsc/downstairs-8840.txt (543066 bytes)
7402026-08-11T23:11:31.448Zuploaded: /tmp/test_up-build/dsc/downstairs-8840.txt
7412026-08-11T23:11:31.448Zuploading: /tmp/test_up-build/dsc/downstairs-8850.txt (1343627 bytes)
7422026-08-11T23:11:32.468Zuploaded: /tmp/test_up-build/dsc/downstairs-8850.txt
7432026-08-11T23:11:32.469Zuploading: /tmp/test_up-build/dsc/downstairs-8860.txt (47348 bytes)
7442026-08-11T23:11:33.480Zuploaded: /tmp/test_up-build/dsc/downstairs-8860.txt
7452026-08-11T23:11:33.480Zuploading: /tmp/test_up-build/dsc/downstairs-8870.txt (3586245 bytes)
7462026-08-11T23:11:34.516Zuploaded: /tmp/test_up-build/dsc/downstairs-8870.txt
7472026-08-11T23:11:34.516Zuploading: /tmp/debug/df.txt (1270 bytes)
7482026-08-11T23:11:35.524Zuploaded: /tmp/debug/df.txt
7492026-08-11T23:11:35.524Zuploading: /tmp/debug/dtrace.txt (1029938 bytes)
7502026-08-11T23:11:36.546Zuploaded: /tmp/debug/dtrace.txt
7512026-08-11T23:11:36.546Zuploading: /tmp/debug/iostat.txt (229604 bytes)
7522026-08-11T23:11:36.557Zupload warning: file "/tmp/debug/iostat.txt" changed size mid upload: 229604 -> 232893
7532026-08-11T23:11:37.564Zuploaded: /tmp/debug/iostat.txt
7542026-08-11T23:11:37.564Zuploading: /tmp/debug/mpstat.txt (558543 bytes)
7552026-08-11T23:11:37.575Zupload warning: file "/tmp/debug/mpstat.txt" changed size mid upload: 558543 -> 567279
7562026-08-11T23:11:38.584Zuploaded: /tmp/debug/mpstat.txt
7572026-08-11T23:11:38.584Zuploading: /tmp/debug/paging.txt (99604 bytes)
7582026-08-11T23:11:38.591Zupload warning: file "/tmp/debug/paging.txt" changed size mid upload: 99604 -> 101244
7592026-08-11T23:11:39.598Zuploaded: /tmp/debug/paging.txt
7602026-08-11T23:11:39.598Zuploading: /tmp/debug/prstat.txt (1068896 bytes)
7612026-08-11T23:11:39.615Zupload warning: file "/tmp/debug/prstat.txt" changed size mid upload: 1068896 -> 1087015
7622026-08-11T23:11:40.622Zuploaded: /tmp/debug/prstat.txt
7632026-08-11T23:11:40.622Zuploading: /tmp/debug/psrinfo.txt (1528 bytes)
7642026-08-11T23:11:41.633Zuploaded: /tmp/debug/psrinfo.txt
7652026-08-11T23:11:41.633Zuploading: /tmp/debug/upstairs-info.txt (165291 bytes)
7662026-08-11T23:11:42.652Zuploaded: /tmp/debug/upstairs-info.txt