01JPKS08HVCS6706FTPFNZQT9D: test-replay

BasicConfig {
    output_rules: [
        "/tmp/*.txt",
        "/tmp/*.log",
        "%/tmp/debug/*.txt",
        "%/tmp/dsc/*.txt",
        "/tmp/core.*",
    ],
    rust_toolchain: None,
    target: Some(
        "helios-2.0",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01JPKS1BWPGWYAQEYYC1ZWAQXC

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12025-03-18T04:57:15.743Zjob dependencies complete; ready to run (waiting for 13 m 23 s)
22025-03-18T04:58:27.972Zjob assigned to worker 01JPKSTFD0JBBPWE1J9H0R0QH2 [factory aws, i-0252500af95ef470d] (queued for 1 m 12 s)
32025-03-18T04:58:35.990Zdownloading input: /input/build/work/bins/crucible-downstairs.gz
42025-03-18T04:58:37.728Zdownloaded input: /input/build/work/bins/crucible-downstairs.gz
52025-03-18T04:58:37.728Zdownloading input: /input/build/work/bins/crucible-hammer.gz
62025-03-18T04:58:39.354Zdownloaded input: /input/build/work/bins/crucible-hammer.gz
72025-03-18T04:58:39.354Zdownloading input: /input/build/work/bins/crutest.gz
82025-03-18T04:58:41.486Zdownloaded input: /input/build/work/bins/crutest.gz
92025-03-18T04:58:41.486Zdownloading input: /input/build/work/bins/dsc.gz
102025-03-18T04:58:42.619Zdownloaded input: /input/build/work/bins/dsc.gz
112025-03-18T04:58:42.638Zdownloading input: /input/build/work/scripts/perf-downstairs-tick.d
122025-03-18T04:58:42.662Zdownloaded input: /input/build/work/scripts/perf-downstairs-tick.d
132025-03-18T04:58:42.662Zdownloading input: /input/build/work/scripts/test_ds.sh
142025-03-18T04:58:42.688Zdownloaded input: /input/build/work/scripts/test_ds.sh
152025-03-18T04:58:42.688Zdownloading input: /input/build/work/scripts/test_live_repair.sh
162025-03-18T04:58:42.713Zdownloaded input: /input/build/work/scripts/test_live_repair.sh
172025-03-18T04:58:42.713Zdownloading input: /input/build/work/scripts/test_repair.sh
182025-03-18T04:58:42.741Zdownloaded input: /input/build/work/scripts/test_repair.sh
192025-03-18T04:58:42.741Zdownloading input: /input/build/work/scripts/test_replay.sh
202025-03-18T04:58:42.776Zdownloaded input: /input/build/work/scripts/test_replay.sh
212025-03-18T04:58:42.777Zdownloading input: /input/build/work/scripts/test_up.sh
222025-03-18T04:58:42.809Zdownloaded input: /input/build/work/scripts/test_up.sh
232025-03-18T04:58:42.809Zdownloading input: /input/build/work/scripts/upstairs_info.d
242025-03-18T04:58:42.845Zdownloaded input: /input/build/work/scripts/upstairs_info.d
252025-03-18T04:58:42.865Zdownloading input: /input/build/tmp/cargo-test-out.log
262025-03-18T04:58:44.035Zdownloaded input: /input/build/tmp/cargo-test-out.log
 
272025-03-18T04:58:44.035Zstarting task 0: "setup"
282025-03-18T04:58:44.059Z++ uname -s
292025-03-18T04:58:44.059Z+ kern=SunOS
302025-03-18T04:58:44.059Z+ build_user=build
312025-03-18T04:58:44.060Z+ build_uid=12345
322025-03-18T04:58:44.060Z+ work_dir=/work
332025-03-18T04:58:44.060Z+ input_dir=/input
342025-03-18T04:58:44.060Z+ [[ 0 == 12345 ]]
352025-03-18T04:58:44.060Z+ case "$kern" in
362025-03-18T04:58:44.060Z+ groupadd -g 12345 build
372025-03-18T04:58:44.060Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
382025-03-18T04:58:46.042Z+ zfs create -o mountpoint=/work rpool/work
392025-03-18T04:58:46.280Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
402025-03-18T04:58:46.288Z+ home_fs=zfs
412025-03-18T04:58:46.288Z+ [[ zfs == autofs ]]
422025-03-18T04:58:46.288Z+ mkdir -p /home/build
432025-03-18T04:58:46.288Z+ chown build:build /home/build /work
442025-03-18T04:58:48.284Z+ chmod 0700 /home/build /work
452025-03-18T04:58:48.297Zprocess exited: duration 4257 ms, exit code 0
 
462025-03-18T04:58:48.335Zstarting task 1: "authentication"
472025-03-18T04:58:48.351ZWARNING: job store has no value for "GITHUB_TOKEN"; waiting for a value...
482025-03-18T04:58:52.415Zprocess exited: duration 4085 ms, exit code 0
 
492025-03-18T04:58:52.465Zstarting task 2: "build"
502025-03-18T04:58:52.502Z+ banner cores
512025-03-18T04:58:52.502Z
522025-03-18T04:58:52.502Z #### #### ##### ###### ####
532025-03-18T04:58:52.502Z # # # # # # # #
542025-03-18T04:58:52.502Z # # # # # ##### ####
552025-03-18T04:58:52.502Z # # # ##### # #
562025-03-18T04:58:52.502Z # # # # # # # # #
572025-03-18T04:58:52.502Z #### #### # # ###### ####
582025-03-18T04:58:52.502Z
592025-03-18T04:58:52.503Z+ pfexec coreadm -i /tmp/core.%f.%p -g /tmp/core.%f.%p -e global -e log -e proc-setid -e global-setid
602025-03-18T04:58:52.503Z+ pfexec chmod +x /input/build/work/scripts/perf-downstairs-tick.d /input/build/work/scripts/test_ds.sh /input/build/work/scripts/test_live_repair.sh /input/build/work/scripts/test_repair.sh /input/build/work/scripts/test_replay.sh /input/build/work/scripts/test_up.sh /input/build/work/scripts/upstairs_info.d
612025-03-18T04:58:52.503Z+ echo ' chmod input script dir contains:'
622025-03-18T04:58:52.503Z chmod input script dir contains:
632025-03-18T04:58:52.503Z+ ls -ltr /input/build/work/scripts
642025-03-18T04:58:52.503Ztotal 46
652025-03-18T04:58:52.503Z-rwxr-xr-x 1 root root 3056 Mar 18 04:58 perf-downstairs-tick.d
662025-03-18T04:58:52.503Z-rwxr-xr-x 1 root root 2127 Mar 18 04:58 test_ds.sh
672025-03-18T04:58:52.503Z-rwxr-xr-x 1 root root 4890 Mar 18 04:58 test_live_repair.sh
682025-03-18T04:58:52.503Z-rwxr-xr-x 1 root root 7419 Mar 18 04:58 test_repair.sh
692025-03-18T04:58:52.503Z-rwxr-xr-x 1 root root 4249 Mar 18 04:58 test_replay.sh
702025-03-18T04:58:52.503Z-rwxr-xr-x 1 root root 13887 Mar 18 04:58 test_up.sh
712025-03-18T04:58:52.503Z-rwxr-xr-x 1 root root 2996 Mar 18 04:58 upstairs_info.d
722025-03-18T04:58:52.503Z+ banner unpack
732025-03-18T04:58:52.503Z
742025-03-18T04:58:52.503Z # # # # ##### ## #### # #
752025-03-18T04:58:52.503Z # # ## # # # # # # # # #
762025-03-18T04:58:52.503Z # # # # # # # # # # ####
772025-03-18T04:58:52.503Z # # # # # ##### ###### # # #
782025-03-18T04:58:52.503Z # # # ## # # # # # # #
792025-03-18T04:58:52.503Z #### # # # # # #### # #
802025-03-18T04:58:52.503Z
812025-03-18T04:58:52.503Z+ mkdir -p /var/tmp/bins
822025-03-18T04:58:52.503Z+ for t in "$input/bins/"*.gz
832025-03-18T04:58:52.503Z++ basename /input/build/work/bins/crucible-downstairs.gz
842025-03-18T04:58:52.503Z+ b=crucible-downstairs.gz
852025-03-18T04:58:52.503Z+ b=crucible-downstairs
862025-03-18T04:58:52.503Z+ gunzip
872025-03-18T04:58:54.914Z+ chmod +x /var/tmp/bins/crucible-downstairs
882025-03-18T04:58:54.933Z+ for t in "$input/bins/"*.gz
892025-03-18T04:58:54.933Z++ basename /input/build/work/bins/crucible-hammer.gz
902025-03-18T04:58:54.933Z+ b=crucible-hammer.gz
912025-03-18T04:58:54.933Z+ b=crucible-hammer
922025-03-18T04:58:54.933Z+ gunzip
932025-03-18T04:58:57.130Z+ chmod +x /var/tmp/bins/crucible-hammer
942025-03-18T04:58:57.144Z+ for t in "$input/bins/"*.gz
952025-03-18T04:58:57.144Z++ basename /input/build/work/bins/crutest.gz
962025-03-18T04:58:57.144Z+ b=crutest.gz
972025-03-18T04:58:57.145Z+ b=crutest
982025-03-18T04:58:57.145Z+ gunzip
992025-03-18T04:58:59.480Z+ chmod +x /var/tmp/bins/crutest
1002025-03-18T04:58:59.483Z+ for t in "$input/bins/"*.gz
1012025-03-18T04:58:59.484Z++ basename /input/build/work/bins/dsc.gz
1022025-03-18T04:58:59.484Z+ b=dsc.gz
1032025-03-18T04:58:59.484Z+ b=dsc
1042025-03-18T04:58:59.484Z+ gunzip
1052025-03-18T04:59:00.872Z+ chmod +x /var/tmp/bins/dsc
1062025-03-18T04:59:00.875Z+ export BINDIR=/var/tmp/bins
1072025-03-18T04:59:00.875Z+ BINDIR=/var/tmp/bins
1082025-03-18T04:59:00.875Z+ export RUST_BACKTRACE=1
1092025-03-18T04:59:00.875Z+ RUST_BACKTRACE=1
1102025-03-18T04:59:00.875Z+ banner setup
1112025-03-18T04:59:00.878Z
1122025-03-18T04:59:00.878Z #### ###### ##### # # #####
1132025-03-18T04:59:00.878Z # # # # # # #
1142025-03-18T04:59:00.878Z #### ##### # # # # #
1152025-03-18T04:59:00.878Z # # # # # #####
1162025-03-18T04:59:00.878Z # # # # # # #
1172025-03-18T04:59:00.878Z #### ###### # #### #
1182025-03-18T04:59:00.878Z
1192025-03-18T04:59:00.878Z+ echo 'Setup self timeout'
1202025-03-18T04:59:00.878ZSetup self timeout
1212025-03-18T04:59:00.878Z+ jobpid=1145
1222025-03-18T04:59:00.878Z+ echo 'Setup debug logging'
1232025-03-18T04:59:00.878ZSetup debug logging
1242025-03-18T04:59:00.878Z+ mkdir /tmp/debug
1252025-03-18T04:59:00.878Z+ sleep 10800
1262025-03-18T04:59:00.881Z+ psrinfo -v
1272025-03-18T04:59:00.886Z+ df -h
1282025-03-18T04:59:00.889Z+ prstat -d d -mLc 1
1292025-03-18T04:59:00.892Z+ iostat -T d -xn 1
1302025-03-18T04:59:00.892Z+ mpstat -T d 1
1312025-03-18T04:59:00.892Z+ vmstat -T d -p 1
1322025-03-18T04:59:00.895Z+ pfexec dtrace -Z -s /input/build/work/scripts/perf-downstairs-tick.d
1332025-03-18T04:59:00.896Z+ banner replay
1342025-03-18T04:59:00.896Z+ pfexec dtrace -Z -s /input/build/work/scripts/upstairs_info.d
1352025-03-18T04:59:00.896Z
1362025-03-18T04:59:00.896Z ##### ###### ##### # ## # #
1372025-03-18T04:59:00.896Z # # # # # # # # # #
1382025-03-18T04:59:00.896Z # # ##### # # # # # #
1392025-03-18T04:59:00.896Z ##### # ##### # ###### #
1402025-03-18T04:59:00.896Z # # # # # # # #
1412025-03-18T04:59:00.896Z # # ###### # ###### # # #
1422025-03-18T04:59:00.896Z
1432025-03-18T04:59:00.896Z+ ptime -m bash /input/build/work/scripts/test_replay.sh
1442025-03-18T04:59:00.915Zstarting March 18, 2025 at 04:58:59 AM UTC
1452025-03-18T04:59:00.918ZTail /tmp/test_replay/test_replay.log for test output
1462025-03-18T04:59:00.921ZCreating 3 downstairs regions
1472025-03-18T04:59:01.259ZStarting 3 downstairs
1482025-03-18T04:59:06.267ZRunning initial fill
1492025-03-18T04:59:10.915ZReplay loop starts now March 18, 2025 at 04:59:09 AM UTC
1502025-03-18T05:07:02.779ZReplays:30 time: 7:52
1512025-03-18T05:07:02.783ZDo final verify
1522025-03-18T05:07:12.304ZMarch 18, 2025 at 05:07:11 AM UTC Test ends with 0
1532025-03-18T05:07:12.313Z
1542025-03-18T05:07:12.316Zreal 8:11.656993324
1552025-03-18T05:07:12.316Zuser 23.993370042
1562025-03-18T05:07:12.316Zsys 6.013408469
1572025-03-18T05:07:12.316Ztrap 0.049289787
1582025-03-18T05:07:12.316Ztflt 0.014967916
1592025-03-18T05:07:12.316Zdflt 0.014839456
1602025-03-18T05:07:12.316Zkflt 0.001130449
1612025-03-18T05:07:12.316Zlock 13:02:03.314654890
1622025-03-18T05:07:12.316Zslp 44:37.763521994
1632025-03-18T05:07:12.316Zlat 3.443480725
1642025-03-18T05:07:12.316Zstop 0.565179310
1652025-03-18T05:07:17.315Zprocess exited: duration 500117 ms, exit code 0
1662025-03-18T05:07:17.315Zexec warning: : stdio descriptors remain open after task exit; waiting 60 seconds for them to close
1672025-03-18T05:08:17.322Zexec warning: : stdout descriptor may be held open by a background process; giving up!
1682025-03-18T05:08:17.322Zexec warning: : stderr descriptor may be held open by a background process; giving up!
 
1692025-03-18T05:08:17.335Zfound 8 output files
1702025-03-18T05:08:17.335Zuploading: /tmp/debug/df.txt (1270 bytes)
1712025-03-18T05:08:18.345Zuploaded: /tmp/debug/df.txt
1722025-03-18T05:08:18.345Zuploading: /tmp/debug/iostat.txt (165996 bytes)
1732025-03-18T05:08:18.360Zupload warning: file "/tmp/debug/iostat.txt" changed size mid upload: 165996 -> 166294
1742025-03-18T05:08:19.368Zuploaded: /tmp/debug/iostat.txt
1752025-03-18T05:08:19.372Zuploading: /tmp/debug/mpstat.txt (404941 bytes)
1762025-03-18T05:08:19.386Zupload warning: file "/tmp/debug/mpstat.txt" changed size mid upload: 404941 -> 406395
1772025-03-18T05:08:20.396Zuploaded: /tmp/debug/mpstat.txt
1782025-03-18T05:08:20.399Zuploading: /tmp/debug/paging.txt (71789 bytes)
1792025-03-18T05:08:20.406Zupload warning: file "/tmp/debug/paging.txt" changed size mid upload: 71789 -> 72128
1802025-03-18T05:08:21.414Zuploaded: /tmp/debug/paging.txt
1812025-03-18T05:08:21.417Zuploading: /tmp/debug/perf.txt (1052495 bytes)
1822025-03-18T05:08:22.444Zuploaded: /tmp/debug/perf.txt
1832025-03-18T05:08:22.447Zuploading: /tmp/debug/prstat.txt (758477 bytes)
1842025-03-18T05:08:22.476Zupload warning: file "/tmp/debug/prstat.txt" changed size mid upload: 758477 -> 764936
1852025-03-18T05:08:23.484Zuploaded: /tmp/debug/prstat.txt
1862025-03-18T05:08:23.484Zuploading: /tmp/debug/psrinfo.txt (1528 bytes)
1872025-03-18T05:08:24.497Zuploaded: /tmp/debug/psrinfo.txt
1882025-03-18T05:08:24.500Zuploading: /tmp/debug/upinfo.txt (55220 bytes)
1892025-03-18T05:08:25.516Zuploaded: /tmp/debug/upinfo.txt