01JE8WYCFXQQYBDSKKRVVQ97X2: test-up-encrypted

BasicConfig {
    output_rules: [
        "%/tmp/test_up*/*.txt",
        "%/tmp/test_up*/dsc/*.txt",
        "%/tmp/debug/*",
        "/tmp/core.*",
    ],
    rust_toolchain: None,
    target: Some(
        "helios-2.0",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01JE8WYNC1HAE32XEMSYG8Y3PG

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12024-12-04T13:29:16.891Zjob dependencies complete; ready to run (waiting for 14 m 12 s)
22024-12-04T13:30:19.056Zjob assigned to worker 01JE8XRW9K8QEGY0XVHXEW37RX [factory aws, i-028bd50f9649eea8f] (queued for 1 m 2 s)
32024-12-04T13:30:26.485Zdownloading input: /input/build/work/bins/crucible-downstairs.gz
42024-12-04T13:30:27.617Zdownloaded input: /input/build/work/bins/crucible-downstairs.gz
52024-12-04T13:30:27.617Zdownloading input: /input/build/work/bins/crucible-hammer.gz
62024-12-04T13:30:28.670Zdownloaded input: /input/build/work/bins/crucible-hammer.gz
72024-12-04T13:30:28.670Zdownloading input: /input/build/work/bins/crutest.gz
82024-12-04T13:30:29.773Zdownloaded input: /input/build/work/bins/crutest.gz
92024-12-04T13:30:29.773Zdownloading input: /input/build/work/bins/dsc.gz
102024-12-04T13:30:30.445Zdownloaded input: /input/build/work/bins/dsc.gz
112024-12-04T13:30:30.449Zdownloading input: /input/build/work/scripts/perf-downstairs-tick.d
122024-12-04T13:30:30.461Zdownloaded input: /input/build/work/scripts/perf-downstairs-tick.d
132024-12-04T13:30:30.461Zdownloading input: /input/build/work/scripts/test_ds.sh
142024-12-04T13:30:30.586Zdownloaded input: /input/build/work/scripts/test_ds.sh
152024-12-04T13:30:30.586Zdownloading input: /input/build/work/scripts/test_live_repair.sh
162024-12-04T13:30:30.606Zdownloaded input: /input/build/work/scripts/test_live_repair.sh
172024-12-04T13:30:30.606Zdownloading input: /input/build/work/scripts/test_repair.sh
182024-12-04T13:30:30.627Zdownloaded input: /input/build/work/scripts/test_repair.sh
192024-12-04T13:30:30.627Zdownloading input: /input/build/work/scripts/test_replay.sh
202024-12-04T13:30:30.643Zdownloaded input: /input/build/work/scripts/test_replay.sh
212024-12-04T13:30:30.643Zdownloading input: /input/build/work/scripts/test_up.sh
222024-12-04T13:30:30.663Zdownloaded input: /input/build/work/scripts/test_up.sh
232024-12-04T13:30:30.663Zdownloading input: /input/build/work/scripts/upstairs_info.d
242024-12-04T13:30:30.681Zdownloaded input: /input/build/work/scripts/upstairs_info.d
252024-12-04T13:30:30.686Zdownloading input: /input/build/tmp/cargo-test-out.log
262024-12-04T13:30:31.463Zdownloaded input: /input/build/tmp/cargo-test-out.log
 
272024-12-04T13:30:31.464Zstarting task 0: "setup"
282024-12-04T13:30:31.470Z++ uname -s
292024-12-04T13:30:31.473Z+ kern=SunOS
302024-12-04T13:30:31.473Z+ case "$kern" in
312024-12-04T13:30:31.473Z+ groupadd -g 12345 build
322024-12-04T13:30:31.477Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
332024-12-04T13:30:33.482Z+ zfs create -o mountpoint=/work rpool/work
342024-12-04T13:30:33.621Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
352024-12-04T13:30:33.624Z+ home_fs=zfs
362024-12-04T13:30:33.624Z+ [[ zfs == autofs ]]
372024-12-04T13:30:33.624Z+ mkdir -p /home/build
382024-12-04T13:30:33.627Z+ chown build:build /home/build /work
392024-12-04T13:30:35.627Z+ chmod 0700 /home/build /work
402024-12-04T13:30:35.630Zprocess exited: duration 4170 ms, exit code 0
 
412024-12-04T13:30:35.635Zstarting task 1: "authentication"
422024-12-04T13:30:35.651Zprocess exited: duration 16 ms, exit code 0
 
432024-12-04T13:30:35.657Zstarting task 2: "build"
442024-12-04T13:30:35.661Z+ banner cores
452024-12-04T13:30:35.665Z
462024-12-04T13:30:35.665Z #### #### ##### ###### ####
472024-12-04T13:30:35.665Z # # # # # # # #
482024-12-04T13:30:35.665Z # # # # # ##### ####
492024-12-04T13:30:35.665Z # # # ##### # #
502024-12-04T13:30:35.665Z # # # # # # # # #
512024-12-04T13:30:35.665Z #### #### # # ###### ####
522024-12-04T13:30:35.665Z
532024-12-04T13:30:35.668Z+ pfexec coreadm -i /tmp/core.%f.%p -g /tmp/core.%f.%p -e global -e log -e proc-setid -e global-setid
542024-12-04T13:30:35.673Z+ echo 'input bins dir contains:'
552024-12-04T13:30:35.674Zinput bins dir contains:
562024-12-04T13:30:35.676Z+ ls -ltr /input/build/work/bins
572024-12-04T13:30:35.676Ztotal 761868
582024-12-04T13:30:35.676Z-rw-r--r-- 1 root root 109324428 Dec 4 13:30 crucible-downstairs.gz
592024-12-04T13:30:35.676Z-rw-r--r-- 1 root root 104619017 Dec 4 13:30 crucible-hammer.gz
602024-12-04T13:30:35.676Z-rw-r--r-- 1 root root 109319980 Dec 4 13:30 crutest.gz
612024-12-04T13:30:35.676Z-rw-r--r-- 1 root root 66562355 Dec 4 13:30 dsc.gz
622024-12-04T13:30:35.676Z+ banner unpack
632024-12-04T13:30:35.679Z
642024-12-04T13:30:35.679Z # # # # ##### ## #### # #
652024-12-04T13:30:35.679Z # # ## # # # # # # # # #
662024-12-04T13:30:35.679Z # # # # # # # # # # ####
672024-12-04T13:30:35.679Z # # # # # ##### ###### # # #
682024-12-04T13:30:35.679Z # # # ## # # # # # # #
692024-12-04T13:30:35.679Z #### # # # # # #### # #
702024-12-04T13:30:35.679Z
712024-12-04T13:30:35.679Z+ mkdir -p /var/tmp/bins
722024-12-04T13:30:35.682Z+ for t in "$input/bins/"*.gz
732024-12-04T13:30:35.682Z++ basename /input/build/work/bins/crucible-downstairs.gz
742024-12-04T13:30:35.684Z+ b=crucible-downstairs.gz
752024-12-04T13:30:35.684Z+ b=crucible-downstairs
762024-12-04T13:30:35.684Z+ gunzip
772024-12-04T13:30:37.950Z+ chmod +x /var/tmp/bins/crucible-downstairs
782024-12-04T13:30:38.134Z+ for t in "$input/bins/"*.gz
792024-12-04T13:30:38.134Z++ basename /input/build/work/bins/crucible-hammer.gz
802024-12-04T13:30:38.134Z+ b=crucible-hammer.gz
812024-12-04T13:30:38.134Z+ b=crucible-hammer
822024-12-04T13:30:38.134Z+ gunzip
832024-12-04T13:30:40.102Z+ chmod +x /var/tmp/bins/crucible-hammer
842024-12-04T13:30:40.104Z+ for t in "$input/bins/"*.gz
852024-12-04T13:30:40.106Z++ basename /input/build/work/bins/crutest.gz
862024-12-04T13:30:40.106Z+ b=crutest.gz
872024-12-04T13:30:40.106Z+ b=crutest
882024-12-04T13:30:40.106Z+ gunzip
892024-12-04T13:30:42.419Z+ chmod +x /var/tmp/bins/crutest
902024-12-04T13:30:42.421Z+ for t in "$input/bins/"*.gz
912024-12-04T13:30:42.424Z++ basename /input/build/work/bins/dsc.gz
922024-12-04T13:30:42.424Z+ b=dsc.gz
932024-12-04T13:30:42.424Z+ b=dsc
942024-12-04T13:30:42.424Z+ gunzip
952024-12-04T13:30:43.808Z+ chmod +x /var/tmp/bins/dsc
962024-12-04T13:30:43.810Z+ export BINDIR=/var/tmp/bins
972024-12-04T13:30:43.810Z+ BINDIR=/var/tmp/bins
982024-12-04T13:30:43.810Z+ jobpid=1139
992024-12-04T13:30:43.813Z+ echo 'Setup debug logging'
1002024-12-04T13:30:43.813Z+ mkdir /tmp/debug
1012024-12-04T13:30:43.813ZSetup debug logging
1022024-12-04T13:30:43.813Z+ sleep 3600
1032024-12-04T13:30:43.813Z+ psrinfo -v
1042024-12-04T13:30:43.816Z+ df -h
1052024-12-04T13:30:43.825Z+ prstat -d d -mLc 1
1062024-12-04T13:30:43.827Z+ iostat -T d -xn 1
1072024-12-04T13:30:43.827Z+ mpstat -T d 1
1082024-12-04T13:30:43.827Z+ vmstat -T d -p 1
1092024-12-04T13:30:43.830Z+ pfexec dtrace -Z -s /input/build/work/scripts/perf-downstairs-tick.d
1102024-12-04T13:30:43.830Z+ banner test_up_encrypted
1112024-12-04T13:30:43.830Z+ pfexec dtrace -Z -s /input/build/work/scripts/upstairs_info.d
1122024-12-04T13:30:43.833Z
1132024-12-04T13:30:43.833Z ##### ###### #### ##### # # ##### ###### # #
1142024-12-04T13:30:43.833Z # # # # # # # # # ## #
1152024-12-04T13:30:43.833Z # ##### #### # # # # # ##### # # #
1162024-12-04T13:30:43.833Z # # # # # # ##### # # # #
1172024-12-04T13:30:43.833Z # # # # # # # # # # ##
1182024-12-04T13:30:43.833Z # ###### #### # ####### #### # ####### ###### # #
1192024-12-04T13:30:43.833Z
1202024-12-04T13:30:43.833Z+ ptime -m bash /input/build/work/scripts/test_up.sh -N encrypted
1212024-12-04T13:30:43.842Z/input/build/work
1222024-12-04T13:30:43.853ZTurn off color for downstairs dump
1232024-12-04T13:30:43.869ZUpstairs using key: pmItIDKXOseRmD3x9PAAoq30thNk1eZDKIPYRIorcus=
1242024-12-04T13:30:43.872Zdsc output goes to /tmp/test_up-build/dsc-out.txt
1252024-12-04T13:30:43.872ZCreating 3 downstairs regions
1262024-12-04T13:30:44.178ZStarting 3 downstairs
1272024-12-04T13:30:44.181Zdsc started at PID: 1185
1282024-12-04T13:30:49.354ZDisable automatic restart on all downstairs
1292024-12-04T13:30:49.406Z
1302024-12-04T13:30:49.406ZBegin tests, output goes to /tmp/test_up-build/test_up_out.txt
1312024-12-04T13:30:49.409ZRunning test: span
1322024-12-04T13:30:53.576ZCompleted test: span
1332024-12-04T13:30:58.583ZRunning test: big
1342024-12-04T13:31:02.922ZCompleted test: big
1352024-12-04T13:31:07.929ZRunning test: dep
1362024-12-04T13:31:59.569ZCompleted test: dep
1372024-12-04T13:32:04.576ZRunning test: balloon
1382024-12-04T13:32:44.357ZCompleted test: balloon
1392024-12-04T13:32:49.364ZRunning test: deactivate
1402024-12-04T13:33:44.038ZCompleted test: deactivate
1412024-12-04T13:33:49.045ZRunning hammer
1422024-12-04T13:34:01.656ZRun repair tests
1432024-12-04T13:34:01.659Z/var/tmp/bins/crutest fill -g 31 -q --key pmItIDKXOseRmD3x9PAAoq30thNk1eZDKIPYRIorcus= --dsc 127.0.0.1:9998 --verify-out /var/tmp/test_up-build/verify_file
1442024-12-04T13:34:01.718Z{"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 10, extent_count: 5 }","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.340264893Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"task":"crutest"}
1452024-12-04T13:34:01.721Z{"msg":"dsc has 3 regions. This means 1 sub_volumes","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.340880564Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"task":"crutest"}
1462024-12-04T13:34:01.721Z{"msg":"SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.342717719Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"task":"crutest"}
1472024-12-04T13:34:01.721Z{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.342951937Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222}
1482024-12-04T13:34:01.721Z{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"0e1f7f60aff9f26b0d737c97120445edbded1ad3\",\n git_commit_timestamp: \"2024-12-04T13:14:36.000000000Z\",\n git_branch: \"renovate/tokio-util-0.x-lockfile\",\n rustc_semver: \"1.82.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"f6e511eec7342f59a25f7c0534f1dbea00d01b14\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.342992183Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222}
1492024-12-04T13:34:01.721Z{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.343017221Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222}
1502024-12-04T13:34:01.721Z{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 10, shift: 12 }, extent_count: 5, uuid: 00000000-0000-0000-0000-000000000000, encrypted: true, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.343035879Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222}
1512024-12-04T13:34:01.723Z{"msg":"Crucible 0945fff8-f882-4ae3-bda6-5dfc21ee6773 has session id: a6b50ef1-398b-4110-8cf1-261cda0f8b90","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.343600715Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1522024-12-04T13:34:01.723Z{"msg":"Upstairs opts: Upstairs UUID: 0945fff8-f882-4ae3-bda6-5dfc21ee6773, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: true, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.34365947Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1532024-12-04T13:34:01.723Z{"msg":"Crucible stats registered with UUID: 0945fff8-f882-4ae3-bda6-5dfc21ee6773","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.343688787Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1542024-12-04T13:34:01.755Z{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.37752077Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"task":"crutest"}
1552024-12-04T13:34:01.755Z{"msg":"The guest has requested activation with gen:31","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.377581844Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222}
1562024-12-04T13:34:01.755Z{"msg":"0945fff8-f882-4ae3-bda6-5dfc21ee6773 active request set","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.377598582Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1572024-12-04T13:34:01.755Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.377615681Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"0","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1582024-12-04T13:34:01.755Z{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.377652037Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"0","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1592024-12-04T13:34:01.755Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.377679365Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"1","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1602024-12-04T13:34:01.759Z{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.377705612Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"1","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1612024-12-04T13:34:01.759Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.377767446Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"2","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1622024-12-04T13:34:01.759Z{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.377809892Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"2","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1632024-12-04T13:34:01.759Z{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.377841319Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"io task","client":"1","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1642024-12-04T13:34:01.759Z{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.377871176Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"io task","client":"0","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1652024-12-04T13:34:01.759Z{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.377922442Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"io task","client":"2","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1662024-12-04T13:34:01.759Z{"msg":"ds_connection connected from Ok(127.0.0.1:44519)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.377945319Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"io task","client":"2","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1672024-12-04T13:34:01.759Z{"msg":"ds_connection connected from Ok(127.0.0.1:38445)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.377966457Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"io task","client":"0","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1682024-12-04T13:34:01.759Z{"msg":"ds_connection connected from Ok(127.0.0.1:63819)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.378246731Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"io task","client":"1","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1692024-12-04T13:34:01.759Z{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.378765551Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"0","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1702024-12-04T13:34:01.759Z{"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.378810767Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"0","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1712024-12-04T13:34:01.759Z{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.378829815Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"1","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1722024-12-04T13:34:01.759Z{"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.378850293Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"1","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1732024-12-04T13:34:01.759Z{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.378878391Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"2","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1742024-12-04T13:34:01.759Z{"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.378906178Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"2","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1752024-12-04T13:34:01.759Z{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.379459465Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"1","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1762024-12-04T13:34:01.759Z{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.379666655Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"0","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1772024-12-04T13:34:01.759Z{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.37982292Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"2","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1782024-12-04T13:34:01.759Z{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.3799319Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"1","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1792024-12-04T13:34:01.759Z{"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.379965377Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"1","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1802024-12-04T13:34:01.759Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.379981275Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1812024-12-04T13:34:01.759Z{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380052659Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"0","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1822024-12-04T13:34:01.759Z{"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380082996Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"0","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1832024-12-04T13:34:01.759Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380100274Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1842024-12-04T13:34:01.760Z{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380127181Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"2","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1852024-12-04T13:34:01.760Z{"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380149209Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"2","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1862024-12-04T13:34:01.760Z{"msg":"[0]R flush_numbers: [1494, 1494, 1494, 1494, 1494]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380164248Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"downstairs","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1872024-12-04T13:34:01.760Z{"msg":"[0]R generation: [25, 25, 25, 25, 25]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380178896Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"downstairs","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1882024-12-04T13:34:01.760Z{"msg":"[0]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380200074Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"downstairs","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1892024-12-04T13:34:01.760Z{"msg":"[1]R flush_numbers: [1494, 1494, 1494, 1494, 1494]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380214153Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"downstairs","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1902024-12-04T13:34:01.760Z{"msg":"[1]R generation: [25, 25, 25, 25, 25]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380227712Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"downstairs","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1912024-12-04T13:34:01.760Z{"msg":"[1]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.38024874Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"downstairs","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1922024-12-04T13:34:01.760Z{"msg":"[2]R flush_numbers: [1494, 1494, 1494, 1494, 1494]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380262858Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"downstairs","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1932024-12-04T13:34:01.760Z{"msg":"[2]R generation: [25, 25, 25, 25, 25]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380276087Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"downstairs","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1942024-12-04T13:34:01.760Z{"msg":"[2]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380289356Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"downstairs","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1952024-12-04T13:34:01.760Z{"msg":"Max found gen is 26","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380302535Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"downstairs","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1962024-12-04T13:34:01.760Z{"msg":"Generation requested: 31 >= found:26","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380315663Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"downstairs","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1972024-12-04T13:34:01.760Z{"msg":"Next flush: 1495","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380329302Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"downstairs","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1982024-12-04T13:34:01.760Z{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380342381Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"":"downstairs","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
1992024-12-04T13:34:01.760Z{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380355679Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
2002024-12-04T13:34:01.760Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380374938Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"0","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
2012024-12-04T13:34:01.760Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380388596Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"1","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
2022024-12-04T13:34:01.760Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380401695Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"client":"2","":"downstairs","session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
2032024-12-04T13:34:01.760Z{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380422243Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
2042024-12-04T13:34:01.760Z{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380435222Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
2052024-12-04T13:34:01.760Z{"msg":"0945fff8-f882-4ae3-bda6-5dfc21ee6773 is now active with session: a6b50ef1-398b-4110-8cf1-261cda0f8b90","v":0,"name":"crucible","Wait for a query_work_queue command to finish before sending IO
2062024-12-04T13:34:01.760Zlevel":30,"time":"2024-12-04T13:34:00.380463499Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
2072024-12-04T13:34:01.760Z{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380475918Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222,"session_id":"a6b50ef1-398b-4110-8cf1-261cda0f8b90"}
2082024-12-04T13:34:01.760Z{"msg":"The guest has finished waiting for activation with:31","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:00.380488027Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1222}
2092024-12-04T13:34:01.760ZDisk: sv:1 bs:4096 ts:204800 tb:50 max_io:50 or 204800
2102024-12-04T13:34:01.760ZFill test
2112024-12-04T13:34:01.797ZRead and Verify all blocks (0..50 range:false)
2122024-12-04T13:34:01.840ZWrote out file "/var/tmp/test_up-build/verify_file"
2132024-12-04T13:34:01.841ZCLIENT: Tests done. All submitted work has been ACK'd
2142024-12-04T13:34:01.843Z----------------------------------------------------------------
2152024-12-04T13:34:01.843Z Crucible gen:31 GIO:true work queues: Upstairs:0 downstairs:1
2162024-12-04T13:34:01.843ZStates: Active Active Active
2172024-12-04T13:34:01.843ZJOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2182024-12-04T13:34:01.843Z 1002 Acked Read 50 Done Done Done false
2192024-12-04T13:34:01.843Z STATES DS:0 DS:1 DS:2 TOTAL
2202024-12-04T13:34:01.843Z Sent 0 0 0 0
2212024-12-04T13:34:01.843Z Done 1 1 1 3
2222024-12-04T13:34:01.843Z Skipped 0 0 0 0
2232024-12-04T13:34:01.843Z Error 0 0 0 0
2242024-12-04T13:34:01.843ZLast Flush: 1001 1001 1001
2252024-12-04T13:34:01.843ZDownstairs last five completed: 1001 1000
2262024-12-04T13:34:01.843ZUpstairs last five completed: 1002 1001 1000
2272024-12-04T13:34:01.843ZCLIENT: Up:0 ds:1 act:3
2282024-12-04T13:34:05.843Z----------------------------------------------------------------
2292024-12-04T13:34:05.843Z Crucible gen:31 GIO:true work queues: Upstairs:0 downstairs:0
2302024-12-04T13:34:05.843ZDownstairs last five completed: 1003 1002 1001 1000
2312024-12-04T13:34:05.843ZUpstairs last five completed: 1003 1002 1001 1000
2322024-12-04T13:34:05.843ZCLIENT: Up:0 ds:0 act:3
2332024-12-04T13:34:05.843ZCLIENT: All crucible jobs finished, exiting program
2342024-12-04T13:34:05.854ZRepair setup passed
2352024-12-04T13:34:05.857ZCopy the region for /var/tmp/test_up-build/8830
2362024-12-04T13:34:05.860Zcp -r /var/tmp/test_up-build/8830 /var/tmp/test_up-build/previous
2372024-12-04T13:34:05.863Z/var/tmp/bins/crutest fill -g 32 -q --key pmItIDKXOseRmD3x9PAAoq30thNk1eZDKIPYRIorcus= --dsc 127.0.0.1:9998 --verify-out /var/tmp/test_up-build/verify_file --verify-in /var/tmp/test_up-build/verify_file
2382024-12-04T13:34:05.919Z{"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 10, extent_count: 5 }","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.540724952Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"task":"crutest"}
2392024-12-04T13:34:05.921Z{"msg":"dsc has 3 regions. This means 1 sub_volumes","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.541282489Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"task":"crutest"}
2402024-12-04T13:34:05.922Z{"msg":"SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.542908283Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"task":"crutest"}
2412024-12-04T13:34:05.922Z{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.543079437Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228}
2422024-12-04T13:34:05.922Z{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"0e1f7f60aff9f26b0d737c97120445edbded1ad3\",\n git_commit_timestamp: \"2024-12-04T13:14:36.000000000Z\",\n git_branch: \"renovate/tokio-util-0.x-lockfile\",\n rustc_semver: \"1.82.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"f6e511eec7342f59a25f7c0534f1dbea00d01b14\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.543121753Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228}
2432024-12-04T13:34:05.922Z{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.543160689Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228}
2442024-12-04T13:34:05.922Z{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 10, shift: 12 }, extent_count: 5, uuid: 00000000-0000-0000-0000-000000000000, encrypted: true, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.543193016Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228}
2452024-12-04T13:34:05.922Z{"msg":"Crucible c0e9a40a-7db0-4018-8a19-c8f22d9b6bc3 has session id: 9d89290b-b152-4bc8-a40c-83597faeb093","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.543654582Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2462024-12-04T13:34:05.922Z{"msg":"Upstairs opts: Upstairs UUID: c0e9a40a-7db0-4018-8a19-c8f22d9b6bc3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: true, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.543703118Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2472024-12-04T13:34:05.922Z{"msg":"Crucible stats registered with UUID: c0e9a40a-7db0-4018-8a19-c8f22d9b6bc3","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.543727125Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2482024-12-04T13:34:05.954Z{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.576558294Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"task":"crutest"}
2492024-12-04T13:34:05.954Z{"msg":"The guest has requested activation with gen:32","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.576613599Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228}
2502024-12-04T13:34:05.954Z{"msg":"c0e9a40a-7db0-4018-8a19-c8f22d9b6bc3 active request set","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.576643156Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2512024-12-04T13:34:05.958Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.576660694Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"0","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2522024-12-04T13:34:05.958Z{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.5766992Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"0","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2532024-12-04T13:34:05.958Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.576728228Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"1","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2542024-12-04T13:34:05.958Z{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.576775783Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"1","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2552024-12-04T13:34:05.958Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.576813Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"2","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2562024-12-04T13:34:05.958Z{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.576873584Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"2","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2572024-12-04T13:34:05.958Z{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.576929749Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"io task","client":"2","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2582024-12-04T13:34:05.958Z{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.577156027Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"io task","client":"0","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2592024-12-04T13:34:05.958Z{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.577219181Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"io task","client":"1","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2602024-12-04T13:34:05.958Z{"msg":"ds_connection connected from Ok(127.0.0.1:54036)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.577269576Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"io task","client":"0","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2612024-12-04T13:34:05.958Z{"msg":"ds_connection connected from Ok(127.0.0.1:58622)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.577326231Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"io task","client":"2","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2622024-12-04T13:34:05.958Z{"msg":"ds_connection connected from Ok(127.0.0.1:33120)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.577348849Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"io task","client":"1","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2632024-12-04T13:34:05.958Z{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.57764469Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"2","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2642024-12-04T13:34:05.958Z{"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.577683517Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"2","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2652024-12-04T13:34:05.958Z{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.577727262Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"1","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2662024-12-04T13:34:05.958Z{"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.577769758Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"1","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2672024-12-04T13:34:05.958Z{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.577793036Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"0","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2682024-12-04T13:34:05.958Z{"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.577826203Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"0","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2692024-12-04T13:34:05.958Z{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.578269971Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"2","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2702024-12-04T13:34:05.958Z{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.578416407Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"1","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2712024-12-04T13:34:05.958Z{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.578526176Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"0","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2722024-12-04T13:34:05.958Z{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.578585681Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"2","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2732024-12-04T13:34:05.958Z{"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.578635046Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"2","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2742024-12-04T13:34:05.958Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.578662893Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2752024-12-04T13:34:05.958Z{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.578918529Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"1","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2762024-12-04T13:34:05.958Z{"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.578969844Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"1","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2772024-12-04T13:34:05.958Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.578997941Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2782024-12-04T13:34:05.958Z{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579019759Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"0","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2792024-12-04T13:34:05.958Z{"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579056286Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"0","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2802024-12-04T13:34:05.959Z{"msg":"[0]R flush_numbers: [1495, 1495, 1495, 1495, 1495]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579094332Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"downstairs","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2812024-12-04T13:34:05.959Z{"msg":"[0]R generation: [31, 31, 31, 31, 31]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579131089Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"downstairs","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2822024-12-04T13:34:05.959Z{"msg":"[0]R dirty: [false, false, false, false, false]","v":0,"name":"Wait for a query_work_queue command to finish before sending IO
2832024-12-04T13:34:05.959Zcrucible","level":30,"time":"2024-12-04T13:34:04.579187373Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"downstairs","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2842024-12-04T13:34:05.959Z{"msg":"[1]R flush_numbers: [1495, 1495, 1495, 1495, 1495]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579228029Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"downstairs","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2852024-12-04T13:34:05.959Z{"msg":"[1]R generation: [31, 31, 31, 31, 31]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579268335Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"downstairs","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2862024-12-04T13:34:05.959Z{"msg":"[1]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579307682Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"downstairs","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2872024-12-04T13:34:05.959Z{"msg":"[2]R flush_numbers: [1495, 1495, 1495, 1495, 1495]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579349528Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"downstairs","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2882024-12-04T13:34:05.959Z{"msg":"[2]R generation: [31, 31, 31, 31, 31]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579388984Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"downstairs","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2892024-12-04T13:34:05.959Z{"msg":"Disk: sv:1 bs:4096 ts:204800 tb:50 max_io:50 or 204800
2902024-12-04T13:34:05.959Z[2]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.57943215Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"downstairs","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2912024-12-04T13:34:05.959Z{"msg":"Max found gen is 32","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579465427Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"downstairs","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2922024-12-04T13:34:05.959Z{"msg":"Generation requested: 32 >= found:32","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579480645Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"downstairs","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2932024-12-04T13:34:05.959Z{"msg":"Next flush: 1496","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579501873Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"downstairs","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2942024-12-04T13:34:05.959Z{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579515992Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"":"downstairs","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2952024-12-04T13:34:05.959Z{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.57953694Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
2962024-12-04T13:34:05.962Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579580716Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"0","":"downstairs","session_id":"Loading write count information from file "/var/tmp/test_up-build/verify_file"
2972024-12-04T13:34:05.962Z9d89290b-b152-4bc8-a40c-83597faeb093"}
2982024-12-04T13:34:05.962Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Fill test
2992024-12-04T13:34:05.962Z,"time":"2024-12-04T13:34:04.57963654Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"1","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
3002024-12-04T13:34:05.962Z{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579669127Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"client":"2","":"downstairs","session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
3012024-12-04T13:34:05.962Z{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579695375Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
3022024-12-04T13:34:05.962Z{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579709504Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
3032024-12-04T13:34:05.962Z{"msg":"c0e9a40a-7db0-4018-8a19-c8f22d9b6bc3 is now active with session: 9d89290b-b152-4bc8-a40c-83597faeb093","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579722722Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
3042024-12-04T13:34:05.962Z{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.57974354Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228,"session_id":"9d89290b-b152-4bc8-a40c-83597faeb093"}
3052024-12-04T13:34:05.962Z{"msg":"The guest has finished waiting for activation with:32","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:04.579756919Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228}
3062024-12-04T13:34:05.995ZRead and Verify all blocks (0..50 range:false)
3072024-12-04T13:34:06.044ZWrote out file "/var/tmp/test_up-build/verify_file"
3082024-12-04T13:34:06.044ZCLIENT: Tests done. All submitted work has been ACK'd
3092024-12-04T13:34:06.046Z----------------------------------------------------------------
3102024-12-04T13:34:06.046Z Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:1
3112024-12-04T13:34:06.046ZStates: Active Active Active
3122024-12-04T13:34:06.046ZJOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3132024-12-04T13:34:06.046Z 1002 Acked Read 50 Done Done Done false
3142024-12-04T13:34:06.046Z STATES DS:0 DS:1 DS:2 TOTAL
3152024-12-04T13:34:06.046Z Sent 0 0 0 0
3162024-12-04T13:34:06.046Z Done 1 1 1 3
3172024-12-04T13:34:06.046Z Skipped 0 0 0 0
3182024-12-04T13:34:06.046Z Error 0 0 0 0
3192024-12-04T13:34:06.046ZLast Flush: 1001 1001 1001
3202024-12-04T13:34:06.046ZDownstairs last five completed: 1001 1000
3212024-12-04T13:34:06.046ZUpstairs last five completed: 1002 1001 1000
3222024-12-04T13:34:06.046ZCLIENT: Up:0 ds:1 act:3
3232024-12-04T13:34:10.045Z----------------------------------------------------------------
3242024-12-04T13:34:10.045Z Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:0
3252024-12-04T13:34:10.045ZDownstairs last five completed: 1003 1002 1001 1000
3262024-12-04T13:34:10.045ZUpstairs last five completed: 1003 1002 1001 1000
3272024-12-04T13:34:10.045ZCLIENT: Up:0 ds:0 act:3
3282024-12-04T13:34:10.045ZCLIENT: All crucible jobs finished, exiting program
3292024-12-04T13:34:10.048Z{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-12-04T13:34:08.66706088Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1228}
3302024-12-04T13:34:10.052ZRepair part 1 passed
3312024-12-04T13:34:10.055Z
3322024-12-04T13:34:10.055ZKill the current downstairs
3332024-12-04T13:34:10.220ZDownstairs 2 stopped
3342024-12-04T13:34:10.220Zmv /var/tmp/test_up-build/8830 /var/tmp/test_up-build/new
3352024-12-04T13:34:10.222ZNow put back the original so we have a mismatch
3362024-12-04T13:34:10.222Zmv /var/tmp/test_up-build/previous /var/tmp/test_up-build/8830
3372024-12-04T13:34:10.225ZRestart downstairs with old directory
3382024-12-04T13:34:10.278Z/var/tmp/bins/crucible-downstairs dump --no-color -d /var/tmp/test_up-build/8810 -d /var/tmp/test_up-build/8820 -d /var/tmp/test_up-build/8830
3392024-12-04T13:34:10.295ZEXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
3402024-12-04T13:34:10.297Z 0 000-009 32 32 31 1496 1496 1495 F F F <---
3412024-12-04T13:34:10.297Z 1 010-019 32 32 31 1496 1496 1495 F F F <---
3422024-12-04T13:34:10.297Z 2 020-029 32 32 31 1496 1496 1495 F F F <---
3432024-12-04T13:34:10.297Z 3 030-039 32 32 31 1496 1496 1495 F F F <---
3442024-12-04T13:34:10.297Z 4 040-049 32 32 31 1496 1496 1495 F F F <---
3452024-12-04T13:34:10.297ZMax gen: 32, Max flush: 1496
3462024-12-04T13:34:10.297ZError: Difference in extent metadata found!
3472024-12-04T13:34:10.299Zdump test found error as expected
3482024-12-04T13:34:10.299Z
3492024-12-04T13:34:10.299Z
3502024-12-04T13:34:10.299Z/var/tmp/bins/crutest verify --range -g 33 -q --key pmItIDKXOseRmD3x9PAAoq30thNk1eZDKIPYRIorcus= --dsc 127.0.0.1:9998 --verify-out /var/tmp/test_up-build/verify_file --verify-in /var/tmp/test_up-build/verify_file
3512024-12-04T13:34:10.356Z{"msg":"Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 10, extent_count: 5 }","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:08.977439037Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"task":"crutest"}
3522024-12-04T13:34:10.359Z{"msg":"dsc has 3 regions. This means 1 sub_volumes","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:08.978009812Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"task":"crutest"}
3532024-12-04T13:34:10.359Z{"msg":"SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:08.979659985Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"task":"crutest"}
3542024-12-04T13:34:10.359Z{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:08.979848467Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236}
3552024-12-04T13:34:10.359Z{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"0e1f7f60aff9f26b0d737c97120445edbded1ad3\",\n git_commit_timestamp: \"2024-12-04T13:14:36.000000000Z\",\n git_branch: \"renovate/tokio-util-0.x-lockfile\",\n rustc_semver: \"1.82.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"f6e511eec7342f59a25f7c0534f1dbea00d01b14\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:08.979888023Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236}
3562024-12-04T13:34:10.359Z{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:08.979902862Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236}
3572024-12-04T13:34:10.359Z{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 10, shift: 12 }, extent_count: 5, uuid: 00000000-0000-0000-0000-000000000000, encrypted: true, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:08.979926149Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236}
3582024-12-04T13:34:10.362Z{"msg":"Crucible 52f3d4ef-a61b-46a4-9dde-7efc6f0b6c0d has session id: 4c1b1095-af65-4484-9984-e42e9bfaea1d","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:08.980475737Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3592024-12-04T13:34:10.362Z{"msg":"Upstairs opts: Upstairs UUID: 52f3d4ef-a61b-46a4-9dde-7efc6f0b6c0d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: true, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:08.980536061Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3602024-12-04T13:34:10.362Z{"msg":"Crucible stats registered with UUID: 52f3d4ef-a61b-46a4-9dde-7efc6f0b6c0d","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:08.980571498Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3612024-12-04T13:34:10.392Z{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.013847964Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"task":"crutest"}
3622024-12-04T13:34:10.392Z{"msg":"The guest has requested activation with gen:33","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.013904199Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236}
3632024-12-04T13:34:10.395Z{"msg":"52f3d4ef-a61b-46a4-9dde-7efc6f0b6c0d active request set","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.013928846Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3642024-12-04T13:34:10.395Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.013965443Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"0","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3652024-12-04T13:34:10.395Z{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.014044525Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"0","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3662024-12-04T13:34:10.395Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.014078312Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"1","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3672024-12-04T13:34:10.395Z{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.014102369Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"1","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3682024-12-04T13:34:10.395Z{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.014126507Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3692024-12-04T13:34:10.395Z{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.014154115Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3702024-12-04T13:34:10.395Z{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.0141984Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"io task","client":"0","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3712024-12-04T13:34:10.395Z{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.014230167Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"io task","client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3722024-12-04T13:34:10.395Z{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.014291542Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"io task","client":"1","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3732024-12-04T13:34:10.395Z{"msg":"ds_connection connected from Ok(127.0.0.1:34102)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.014323279Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"io task","client":"0","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3742024-12-04T13:34:10.395Z{"msg":"ds_connection connected from Ok(127.0.0.1:61811)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.014356045Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"io task","client":"1","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3752024-12-04T13:34:10.395Z{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-12-04T13:34:09.014387442Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"io task","client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3762024-12-04T13:34:10.395Z{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-12-04T13:34:09.014444817Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"io task","client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3772024-12-04T13:34:10.395Z{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-12-04T13:34:09.01451233Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3782024-12-04T13:34:10.395Z{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.014608681Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"io task","client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3792024-12-04T13:34:10.395Z{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.015071397Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"1","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3802024-12-04T13:34:10.395Z{"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.015113633Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"1","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3812024-12-04T13:34:10.395Z{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.015132911Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"0","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3822024-12-04T13:34:10.395Z{"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.015206314Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"0","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3832024-12-04T13:34:10.395Z{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.015733344Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"0","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3842024-12-04T13:34:10.395Z{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.015784349Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"1","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3852024-12-04T13:34:10.395Z{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.016103159Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"1","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3862024-12-04T13:34:10.395Z{"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.016137455Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"1","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3872024-12-04T13:34:10.395Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.016153794Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3882024-12-04T13:34:10.395Z{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.016185931Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"0","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3892024-12-04T13:34:10.395Z{"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.016219868Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"0","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3902024-12-04T13:34:10.395Z{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:09.016254144Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3912024-12-04T13:34:20.396Z{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.015845253Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"io task","client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3922024-12-04T13:34:20.396Z{"msg":"ds_connection connected from Ok(127.0.0.1:38654)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.015928985Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"io task","client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3932024-12-04T13:34:20.399Z{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.017048139Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3942024-12-04T13:34:20.399Z{"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.017110573Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3952024-12-04T13:34:20.399Z{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.01765954Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3962024-12-04T13:34:20.399Z{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018044224Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3972024-12-04T13:34:20.399Z{"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018076581Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3982024-12-04T13:34:20.399Z{"msg":"[0]R flush_numbers: [1496, 1496, 1496, 1496, 1496]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018093029Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
3992024-12-04T13:34:20.399Z{"msg":"[0]R generation: [32, 32, 32, 32, 32]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018115417Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4002024-12-04T13:34:20.399Z{"msg":"[0]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018137965Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4012024-12-04T13:34:20.399Z{"msg":"[1]R flush_numbers: [1496, 1496, 1496, 1496, 1496]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018159453Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4022024-12-04T13:34:20.399Z{"msg":"[1]R generation: [32, 32, 32, 32, 32]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018174251Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4032024-12-04T13:34:20.399Z{"msg":"[1]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018195899Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4042024-12-04T13:34:20.399Z{"msg":"[2]R flush_numbers: [1495, 1495, 1495, 1495, 1495]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018210328Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4052024-12-04T13:34:20.399Z{"msg":"[2]R generation: [31, 31, 31, 31, 31]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018230936Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4062024-12-04T13:34:20.399Z{"msg":"[2]R dirty: [false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018245164Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4072024-12-04T13:34:20.399Z{"msg":"Max found gen is 33","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018258443Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4082024-12-04T13:34:20.399Z{"msg":"Generation requested: 33 >= found:33","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018278991Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4092024-12-04T13:34:20.399Z{"msg":"Next flush: 1497","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.01829229Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4102024-12-04T13:34:20.399Z{"msg":"Extent 0 has flush number mismatch","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018313098Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4112024-12-04T13:34:20.399Z{"msg":"First source client ID for extent 0","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018327757Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4122024-12-04T13:34:20.399Z{"msg":"extent:0 gens: 32 32 31","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018349324Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4132024-12-04T13:34:20.399Z{"msg":"extent:0 flush: 1496 1496 1495 scs: [ClientId(0), ClientId(1)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018364573Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4142024-12-04T13:34:20.399Z{"msg":"max_flush now has: [ClientId(0), ClientId(1)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018387551Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4152024-12-04T13:34:20.399Z{"msg":"extent:0 dirty: false false false","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018402539Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4162024-12-04T13:34:20.400Z{"msg":"No maxes found, left with: [ClientId(0), ClientId(1)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018424087Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4172024-12-04T13:34:20.400Z{"msg":"find dest for source 0 for extent at index 0","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018439066Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4182024-12-04T13:34:20.400Z{"msg":"source 0, add dest 2 gen","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018460684Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4192024-12-04T13:34:20.400Z{"msg":"Extent 1 has flush number mismatch","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018486031Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4202024-12-04T13:34:20.400Z{"msg":"First source client ID for extent 1","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018509639Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4212024-12-04T13:34:20.400Z{"msg":"extent:1 gens: 32 32 31","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018535017Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4222024-12-04T13:34:20.400Z{"msg":"extent:1 flush: 1496 1496 1495 scs: [ClientId(0), ClientId(1)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018569103Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4232024-12-04T13:34:20.403Z{"msg":"max_flush now has: [ClientId(0), ClientId(1)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018587202Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4242024-12-04T13:34:20.403Z{"msg":"extent:1 dirty: false false false","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.01860185Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4252024-12-04T13:34:20.403Z{"msg":"No maxes found, left with: [ClientId(0), ClientId(1)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018616759Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4262024-12-04T13:34:20.403Z{"msg":"find dest for source 0 for extent at index 1","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018638967Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4272024-12-04T13:34:20.403Z{"msg":"source 0, add dest 2 gen","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018654005Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4282024-12-04T13:34:20.403Z{"msg":"Extent 2 has flush number mismatch","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018676973Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4292024-12-04T13:34:20.403Z{"msg":"First source client ID for extent 2","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018691832Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4302024-12-04T13:34:20.404Z{"msg":"extent:2 gens: 32 32 31","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.01871348Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4312024-12-04T13:34:20.404Z{"msg":"extent:2 flush: 1496 1496 1495 scs: [ClientId(0), ClientId(1)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018728828Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4322024-12-04T13:34:20.404Z{"msg":"max_flush now has: [ClientId(0), ClientId(1)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018750386Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4332024-12-04T13:34:20.404Z{"msg":"extent:2 dirty: false false false","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018765245Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4342024-12-04T13:34:20.404Z{"msg":"No maxes found, left with: [ClientId(0), ClientId(1)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018787653Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4352024-12-04T13:34:20.404Z{"msg":"find dest for source 0 for extent at index 2","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018804931Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4362024-12-04T13:34:20.404Z{"msg":"source 0, add dest 2 gen","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018835058Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4372024-12-04T13:34:20.404Z{"msg":"Extent 3 has flush number mismatch","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018852596Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4382024-12-04T13:34:20.404Z{"msg":"First source client ID for extent 3","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018874614Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4392024-12-04T13:34:20.404Z{"msg":"extent:3 gens: 32 32 31","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018889653Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4402024-12-04T13:34:20.404Z{"msg":"extent:3 flush: 1496 1496 1495 scs: [ClientId(0), ClientId(1)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018912681Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4412024-12-04T13:34:20.404Z{"msg":"max_flush now has: [ClientId(0), ClientId(1)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018927829Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4422024-12-04T13:34:20.404Z{"msg":"extent:3 dirty: false false false","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018949187Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4432024-12-04T13:34:20.404Z{"msg":"No maxes found, left with: [ClientId(0), ClientId(1)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018965366Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4442024-12-04T13:34:20.404Z{"msg":"find dest for source 0 for extent at index 3","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.018987323Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4452024-12-04T13:34:20.404Z{"msg":"source 0, add dest 2 gen","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.019002672Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4462024-12-04T13:34:20.404Z{"msg":"Extent 4 has flush number mismatch","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.0190244Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4472024-12-04T13:34:20.404Z{"msg":"First source client ID for extent 4","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.019039279Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4482024-12-04T13:34:20.404Z{"msg":"extent:4 gens: 32 32 31","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.019054677Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4492024-12-04T13:34:20.404Z{"msg":"extent:4 flush: 1496 1496 1495 scs: [ClientId(0), ClientId(1)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.019077055Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4502024-12-04T13:34:20.404Z{"msg":"max_flush now has: [ClientId(0), ClientId(1)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.019092244Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4512024-12-04T13:34:20.404Z{"msg":"extent:4 dirty: false false false","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.019113611Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4522024-12-04T13:34:20.404Z{"msg":"No maxes found, left with: [ClientId(0), ClientId(1)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.01912868Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4532024-12-04T13:34:20.404Z{"msg":"find dest for source 0 for extent at index 4","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.019150318Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4542024-12-04T13:34:20.404Z{"msg":"source 0, add dest 2 gen","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.019165147Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"mrl":"flush_mismatch","":"mend","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4552024-12-04T13:34:20.404Z{"msg":"Transition from WaitQuorum to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.019195144Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"0","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4562024-12-04T13:34:20.404Z{"msg":"Transition from WaitQuorum to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.019213872Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"1","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4572024-12-04T13:34:20.404Z{"msg":"Transition from WaitQuorum to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.01924087Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4582024-12-04T13:34:20.404Z{"msg":"Full repair list: {ExtentId(0): ExtentFix { source: ClientId(0), dest: [ClientId(2)] }, ExtentId(3): ExtentFix { source: ClientId(0), dest: [ClientId(2)] }, ExtentId(2): ExtentFix { source: ClientId(0), dest: [ClientId(2)] }, ExtentId(1): ExtentFix { source: ClientId(0), dest: [ClientId(2)] }, ExtentId(4): ExtentFix { source: ClientId(0), dest: [ClientId(2)] }}","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.019261488Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4592024-12-04T13:34:20.404Z{"msg":"Task list: [ReconcileIO { id: ReconciliationId(0), op: ExtentFlush { repair_id: ReconciliationId(0), extent_id: ExtentId(0), client_id: ClientId(0), flush_number: 1497, gen_number: 33 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(1), op: ExtentClose { repair_id: ReconciliationId(1), extent_id: ExtentId(0) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(2), op: ExtentRepair { repair_id: ReconciliationId(2), extent_id: ExtentId(0), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(3), op: ExtentReopen { repair_id: ReconciliationId(3), extent_id: ExtentId(0) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(4), op: ExtentFlush { repair_id: ReconciliationId(4), extent_id: ExtentId(3), client_id: ClientId(0), flush_number: 1497, gen_number: 33 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(5), op: ExtentClose { repair_id: ReconciliationId(5), extent_id: ExtentId(3) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(6), op: ExtentRepair { repair_id: ReconciliationId(6), extent_id: ExtentId(3), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(7), op: ExtentReopen { repair_id: ReconciliationId(7), extent_id: ExtentId(3) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(8), op: ExtentFlush { repair_id: ReconciliationId(8), extent_id: ExtentId(2), client_id: ClientId(0), flush_number: 1497, gen_number: 33 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(9), op: ExtentClose { repair_id: ReconciliationId(9), extent_id: ExtentId(2) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(10), op: ExtentRepair { repair_id: ReconciliationId(10), extent_id: ExtentId(2), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(11), op: ExtentReopen { repair_id: ReconciliationId(11), extent_id: ExtentId(2) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(12), op: ExtentFlush { repair_id: ReconciliationId(12), extent_id: ExtentId(1), client_id: ClientId(0), flush_number: 1497, gen_number: 33 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(13), op: ExtentClose { repair_id: ReconciliationId(13), extent_id: ExtentId(1) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(14), op: ExtentRepair { repair_id: ReconciliationId(14), extent_id: ExtentId(1), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(15), op: ExtentReopen { repair_id: ReconciliationId(15), extent_id: ExtentId(1) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(16), op: ExtentFlush { repair_id: ReconciliationId(16), extent_id: ExtentId(4), client_id: ClientId(0), flush_number: 1497, gen_number: 33 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(17), op: ExtentClose { repair_id: ReconciliationId(17), extent_id: ExtentId(4) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(18), op: ExtentRepair { repair_id: ReconciliationId(18), extent_id: ExtentId(4), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(19), op: ExtentReopen { repair_id: ReconciliationId(19), extent_id: ExtentId(4) }, state: ClientData([New, New, New]) }]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.019373527Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4602024-12-04T13:34:20.404Z{"msg":"starting reconciliation 3e5ca855-a496-4780-8006-3b19a5e31986: found 20 extents that need repair","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.019420202Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4612024-12-04T13:34:20.404Z{"msg":"no Nexus client from DNS, aborting start notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.019455419Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4622024-12-04T13:34:20.404Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.019488906Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4632024-12-04T13:34:20.404Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.019523172Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4642024-12-04T13:34:20.404Z{"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.0195495Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4652024-12-04T13:34:20.404Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.019630682Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4662024-12-04T13:34:20.408Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.024344073Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4672024-12-04T13:34:20.408Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.024753704Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4682024-12-04T13:34:20.408Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.025020968Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4692024-12-04T13:34:20.408Z{"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.025335728Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4702024-12-04T13:34:20.408Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.025740299Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4712024-12-04T13:34:20.411Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.028192985Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4722024-12-04T13:34:20.411Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.028637343Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4732024-12-04T13:34:20.411Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.028853033Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4742024-12-04T13:34:20.411Z{"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.029173242Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4752024-12-04T13:34:20.411Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.029220938Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4762024-12-04T13:34:20.413Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.032097123Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4772024-12-04T13:34:20.413Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.032489876Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4782024-12-04T13:34:20.413Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.032766239Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4792024-12-04T13:34:20.413Z{"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.033030004Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4802024-12-04T13:34:20.413Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.033081879Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4812024-12-04T13:34:20.416Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.035843536Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4822024-12-04T13:34:20.416Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.036231119Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4832024-12-04T13:34:20.418Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.036517142Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4842024-12-04T13:34:20.418Z{"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.036770947Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4852024-12-04T13:34:20.418Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.036814763Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4862024-12-04T13:34:20.421Z{"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.039569031Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4872024-12-04T13:34:20.421Z{"msg":"done with reconciliation","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.039929196Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4882024-12-04T13:34:20.421Z{"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.039966143Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"0","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4892024-12-04T13:34:20.421Z{"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.039984191Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"1","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4902024-12-04T13:34:20.421Z{"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.040003709Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"client":"2","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4912024-12-04T13:34:20.421Z{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.040034966Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4922024-12-04T13:34:20.421Z{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.040050165Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4932024-12-04T13:34:20.421Z{"msg":"52f3d4ef-a61b-46a4-9dde-7efc6f0b6c0d is now active with session: 4c1b1095-af65-4484-9984-e42e9bfaea1d","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.040064693Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4942024-12-04T13:34:20.421Z{"msg":"The guest has finished waiting for activation with:33","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:34:19.040078002Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236}
4952024-12-04T13:34:20.421Z{"msg":"no Nexus client from DNS, aborting finish notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:34:19.040089871Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236,"reconcile":"3e5ca855-a496-4780-8006-3b19a5e31986","":"downstairs","":"downstairs","session_id":"4c1b1095-af65-4484-9984-e42e9bfaea1d"}
4962024-12-04T13:34:20.421ZWait for a query_work_queue command to finish before sending IO
4972024-12-04T13:34:20.421ZDisk: sv:1 bs:4096 ts:204800 tb:50 max_io:50 or 204800
4982024-12-04T13:34:20.421ZLoading write count information from file "/var/tmp/test_up-build/verify_file"
4992024-12-04T13:34:20.421ZRead and Verify all blocks (0..50 range:true)
5002024-12-04T13:34:20.458ZWrote out file "/var/tmp/test_up-build/verify_file"
5012024-12-04T13:34:20.459ZVerify test completed
5022024-12-04T13:34:20.461ZWrote out file "/var/tmp/test_up-build/verify_file"
5032024-12-04T13:34:20.461ZCLIENT: Tests done. All submitted work has been ACK'd
5042024-12-04T13:34:20.461Z----------------------------------------------------------------
5052024-12-04T13:34:20.461Z Crucible gen:33 GIO:true work queues: Upstairs:0 downstairs:1
5062024-12-04T13:34:20.461ZStates: Active Active Active
5072024-12-04T13:34:20.461ZJOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5082024-12-04T13:34:20.461Z 1000 Acked Read 50 Done Done Done false
5092024-12-04T13:34:20.461Z STATES DS:0 DS:1 DS:2 TOTAL
5102024-12-04T13:34:20.461Z Sent 0 0 0 0
5112024-12-04T13:34:20.461Z Done 1 1 1 3
5122024-12-04T13:34:20.461Z Skipped 0 0 0 0
5132024-12-04T13:34:20.461Z Error 0 0 0 0
5142024-12-04T13:34:20.461ZLast Flush: 0 0 0
5152024-12-04T13:34:20.461ZDownstairs last five completed:
5162024-12-04T13:34:20.461ZUpstairs last five completed: 1000
5172024-12-04T13:34:20.461ZCLIENT: Up:0 ds:1 act:3
5182024-12-04T13:34:24.462Z----------------------------------------------------------------
5192024-12-04T13:34:24.462Z Crucible gen:33 GIO:true work queues: Upstairs:0 downstairs:0
5202024-12-04T13:34:24.462ZDownstairs last five completed: 1001 1000
5212024-12-04T13:34:24.462ZUpstairs last five completed: 1001 1000
5222024-12-04T13:34:24.462ZCLIENT: Up:0 ds:0 act:3
5232024-12-04T13:34:24.462ZCLIENT: All crucible jobs finished, exiting program
5242024-12-04T13:34:24.464Z{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-12-04T13:34:23.081680272Z","hostname":"w-01JE8XRW9K8QEGY0XVHXEW37RX","pid":1236}
5252024-12-04T13:34:24.470ZRepair part 2 passed
5262024-12-04T13:34:24.472Z/var/tmp/bins/crucible-downstairs dump --no-color -d /var/tmp/test_up-build/8810 -d /var/tmp/test_up-build/8820 -d /var/tmp/test_up-build/8830
5272024-12-04T13:34:24.487ZEXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
5282024-12-04T13:34:24.487Z 0 000-009 32 32 32 1496 1496 1496 F F F
5292024-12-04T13:34:24.487Z 1 010-019 32 32 32 1496 1496 1496 F F F
5302024-12-04T13:34:24.487Z 2 020-029 32 32 32 1496 1496 1496 F F F
5312024-12-04T13:34:24.489Z 3 030-039 32 32 32 1496 1496 1496 F F F
5322024-12-04T13:34:24.489Z 4 040-049 32 32 32 1496 1496 1496 F F F
5332024-12-04T13:34:24.490ZMax gen: 32, Max flush: 1496
5342024-12-04T13:34:24.492Zdump test passed
5352024-12-04T13:34:24.492Z/var/tmp/bins/crucible-downstairs dump --no-color -d /var/tmp/test_up-build/8810 -d /var/tmp/test_up-build/8820 -d /var/tmp/test_up-build/8830 -e 1
5362024-12-04T13:34:24.507Z Extent 1
5372024-12-04T13:34:24.507ZGEN 32 32 32
5382024-12-04T13:34:24.507ZFLUSH_ID 1496 1496 1496
5392024-12-04T13:34:24.507ZDIRTY
5402024-12-04T13:34:24.507Z
5412024-12-04T13:34:24.509ZBLOCK D0 D1 D2 C0 C1 C2 DIFF
5422024-12-04T13:34:24.509Z010 A A A A A A
5432024-12-04T13:34:24.511Z011 A A A A A A
5442024-12-04T13:34:24.511Z012 A A A A A A
5452024-12-04T13:34:24.513Z013 A A A A A A
5462024-12-04T13:34:24.513Z014 A A A A A A
5472024-12-04T13:34:24.516Z015 A A A A A A
5482024-12-04T13:34:24.516Z016 A A A A A A
5492024-12-04T13:34:24.519Z017 A A A A A A
5502024-12-04T13:34:24.519Z018 A A A A A A
5512024-12-04T13:34:24.521Z019 A A A A A A
5522024-12-04T13:34:24.524Zdump extent test passed
5532024-12-04T13:34:24.524Z/var/tmp/bins/crucible-downstairs dump --no-color -d /var/tmp/test_up-build/8810 -d /var/tmp/test_up-build/8820 -d /var/tmp/test_up-build/8830 -b 20
5542024-12-04T13:34:24.538ZExtent 2 Block in extent 0 Actual block 20
5552024-12-04T13:34:24.538Z
5562024-12-04T13:34:24.541Z DATA SHA256 VER
5572024-12-04T13:34:24.541Z------ ---------------------------------------------------------------- ---
5582024-12-04T13:34:24.541Z 0 da9078a104f553bcfbccdb185967dbe518b8828c20403c7ac38d98373c310c38 A
5592024-12-04T13:34:24.541Z 1 da9078a104f553bcfbccdb185967dbe518b8828c20403c7ac38d98373c310c38 A
5602024-12-04T13:34:24.543Z 2 da9078a104f553bcfbccdb185967dbe518b8828c20403c7ac38d98373c310c38 A
5612024-12-04T13:34:24.543Z
5622024-12-04T13:34:24.543ZNONCES 0 1 2 DIFF
5632024-12-04T13:34:24.543Z------ ------------------------ ------------------------ ------------------------ -----
5642024-12-04T13:34:24.543Z 0 a44dfa527939e37349a3d375 a44dfa527939e37349a3d375 a44dfa527939e37349a3d375
5652024-12-04T13:34:24.543Z
5662024-12-04T13:34:24.543Z TAGS 0 1 2 DIFF
5672024-12-04T13:34:24.543Z------ -------------------------------- -------------------------------- -------------------------------- -----
5682024-12-04T13:34:24.543Z 0 9d9624d1be2eca13656e256f0f903adc 9d9624d1be2eca13656e256f0f903adc 9d9624d1be2eca13656e256f0f903adc
5692024-12-04T13:34:24.543Z
5702024-12-04T13:34:24.543ZHASHES 0 1 2 DIFF
5712024-12-04T13:34:24.543Z------ ---------------- ---------------- ---------------- -----
5722024-12-04T13:34:24.543Z
5732024-12-04T13:34:24.546Zdump block test passed
5742024-12-04T13:34:24.546ZInitial upstairs tests have completed, stopping all downstairs
5752024-12-04T13:34:29.607ZCreating 4 larger downstairs regions
5762024-12-04T13:34:29.717ZStarting 4 downstairs
5772024-12-04T13:34:34.724Zdsc restarted at PID: 1249
5782024-12-04T13:34:34.952ZNow do the replace-reconcile test
5792024-12-04T13:34:35.007ZUsing 8840 for the replacement port
5802024-12-04T13:38:05.550ZCompleted test: replace-reconcile
5812024-12-04T13:38:05.552ZNow do the replace-before-active test
5822024-12-04T13:41:12.129ZCompleted test: replace-before-active
5832024-12-04T13:41:12.132ZAll tests have completed, stopping all downstairs
5842024-12-04T13:41:12.185Z
5852024-12-04T13:41:12.185ZAll Tests have passed
5862024-12-04T13:41:12.188Z10:28 Test duration
5872024-12-04T13:41:12.188Z
5882024-12-04T13:41:12.188Zreal 10:28.278161908
5892024-12-04T13:41:12.188Zuser 6:27.548066113
5902024-12-04T13:41:12.188Zsys 11.724996871
5912024-12-04T13:41:12.188Ztrap 0.194698727
5922024-12-04T13:41:12.188Ztflt 0.012300983
5932024-12-04T13:41:12.188Zdflt 0.015684530
5942024-12-04T13:41:12.188Zkflt 0.001058840
5952024-12-04T13:41:12.188Zlock 8:36:10.295054941
5962024-12-04T13:41:12.188Zslp 34:59.495298233
5972024-12-04T13:41:12.188Zlat 24.757928933
5982024-12-04T13:41:12.188Zstop 0.123371255
5992024-12-04T13:41:12.188Z+ echo 'test-up-encrypted ends'
6002024-12-04T13:41:12.188Ztest-up-encrypted ends
6012024-12-04T13:41:17.191Zprocess exited: duration 636456 ms, exit code 0
6022024-12-04T13:41:17.191Zexec warning: : stdio descriptors remain open after task exit; waiting 60 seconds for them to close
6032024-12-04T13:42:17.222Zexec warning: : stdout descriptor may be held open by a background process; giving up!
6042024-12-04T13:42:17.222Zexec warning: : stderr descriptor may be held open by a background process; giving up!
 
6052024-12-04T13:42:17.232Zfound 14 output files
6062024-12-04T13:42:17.232Zuploading: /tmp/test_up-build/dsc-out.txt (45656 bytes)
6072024-12-04T13:42:18.244Zuploaded: /tmp/test_up-build/dsc-out.txt
6082024-12-04T13:42:18.253Zuploading: /tmp/test_up-build/test_up_out.txt (11339679 bytes)
6092024-12-04T13:42:19.351Zuploaded: /tmp/test_up-build/test_up_out.txt
6102024-12-04T13:42:19.354Zuploading: /tmp/test_up-build/dsc/downstairs-8810.txt (2509 bytes)
6112024-12-04T13:42:20.361Zuploaded: /tmp/test_up-build/dsc/downstairs-8810.txt
6122024-12-04T13:42:20.363Zuploading: /tmp/test_up-build/dsc/downstairs-8820.txt (144647 bytes)
6132024-12-04T13:42:21.375Zuploaded: /tmp/test_up-build/dsc/downstairs-8820.txt
6142024-12-04T13:42:21.377Zuploading: /tmp/test_up-build/dsc/downstairs-8830.txt (251110 bytes)
6152024-12-04T13:42:22.390Zuploaded: /tmp/test_up-build/dsc/downstairs-8830.txt
6162024-12-04T13:42:22.393Zuploading: /tmp/test_up-build/dsc/downstairs-8840.txt (139425 bytes)
6172024-12-04T13:42:23.404Zuploaded: /tmp/test_up-build/dsc/downstairs-8840.txt
6182024-12-04T13:42:23.407Zuploading: /tmp/debug/df.txt (1270 bytes)
6192024-12-04T13:42:24.413Zuploaded: /tmp/debug/df.txt
6202024-12-04T13:42:24.416Zuploading: /tmp/debug/dtrace.txt (589730 bytes)
6212024-12-04T13:42:25.439Zuploaded: /tmp/debug/dtrace.txt
6222024-12-04T13:42:25.442Zuploading: /tmp/debug/iostat.txt (208933 bytes)
6232024-12-04T13:42:25.458Zupload warning: file "/tmp/debug/iostat.txt" changed size mid upload: 208933 -> 211341
6242024-12-04T13:42:26.464Zuploaded: /tmp/debug/iostat.txt
6252024-12-04T13:42:26.466Zuploading: /tmp/debug/mpstat.txt (506624 bytes)
6262024-12-04T13:42:26.476Zupload warning: file "/tmp/debug/mpstat.txt" changed size mid upload: 506624 -> 513194
6272024-12-04T13:42:27.483Zuploaded: /tmp/debug/mpstat.txt
6282024-12-04T13:42:27.492Zuploading: /tmp/debug/paging.txt (91564 bytes)
6292024-12-04T13:42:27.495Zupload warning: file "/tmp/debug/paging.txt" changed size mid upload: 91564 -> 92882
6302024-12-04T13:42:28.501Zuploaded: /tmp/debug/paging.txt
6312024-12-04T13:42:28.504Zuploading: /tmp/debug/prstat.txt (968165 bytes)
6322024-12-04T13:42:28.525Zupload warning: file "/tmp/debug/prstat.txt" changed size mid upload: 968165 -> 982399
6332024-12-04T13:42:29.533Zuploaded: /tmp/debug/prstat.txt
6342024-12-04T13:42:29.533Zuploading: /tmp/debug/psrinfo.txt (1528 bytes)
6352024-12-04T13:42:30.540Zuploaded: /tmp/debug/psrinfo.txt
6362024-12-04T13:42:30.540Zuploading: /tmp/debug/upstairs-info.txt (100320 bytes)
6372024-12-04T13:42:31.556Zuploaded: /tmp/debug/upstairs-info.txt