01JPKS2SJZ3F7MBKS5H62BD4YP: test-replay

BasicConfig {
    output_rules: [
        "/tmp/*.txt",
        "/tmp/*.log",
        "%/tmp/debug/*.txt",
        "%/tmp/dsc/*.txt",
        "/tmp/core.*",
    ],
    rust_toolchain: None,
    target: Some(
        "helios-2.0",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01JPKS3GY5Y2VVH4G4RR1SCG8G

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12025-03-18T04:58:36.073Zjob dependencies complete; ready to run (waiting for 13 m 33 s)
22025-03-18T04:59:46.016Zjob assigned to worker 01JPKSWNJ0SQWEEYX74PZDFQED [factory aws, i-08c899d62ff7f961c] (queued for 1 m 9 s)
32025-03-18T04:59:54.353Zdownloading input: /input/build/work/bins/crucible-downstairs.gz
42025-03-18T04:59:55.705Zdownloaded input: /input/build/work/bins/crucible-downstairs.gz
52025-03-18T04:59:55.705Zdownloading input: /input/build/work/bins/crucible-hammer.gz
62025-03-18T04:59:56.827Zdownloaded input: /input/build/work/bins/crucible-hammer.gz
72025-03-18T04:59:56.827Zdownloading input: /input/build/work/bins/crutest.gz
82025-03-18T04:59:58.078Zdownloaded input: /input/build/work/bins/crutest.gz
92025-03-18T04:59:58.078Zdownloading input: /input/build/work/bins/dsc.gz
102025-03-18T04:59:58.792Zdownloaded input: /input/build/work/bins/dsc.gz
112025-03-18T04:59:58.796Zdownloading input: /input/build/work/scripts/perf-downstairs-tick.d
122025-03-18T04:59:58.813Zdownloaded input: /input/build/work/scripts/perf-downstairs-tick.d
132025-03-18T04:59:58.814Zdownloading input: /input/build/work/scripts/test_ds.sh
142025-03-18T04:59:58.843Zdownloaded input: /input/build/work/scripts/test_ds.sh
152025-03-18T04:59:58.843Zdownloading input: /input/build/work/scripts/test_live_repair.sh
162025-03-18T04:59:58.862Zdownloaded input: /input/build/work/scripts/test_live_repair.sh
172025-03-18T04:59:58.862Zdownloading input: /input/build/work/scripts/test_repair.sh
182025-03-18T04:59:58.953Zdownloaded input: /input/build/work/scripts/test_repair.sh
192025-03-18T04:59:58.953Zdownloading input: /input/build/work/scripts/test_replay.sh
202025-03-18T04:59:58.973Zdownloaded input: /input/build/work/scripts/test_replay.sh
212025-03-18T04:59:58.973Zdownloading input: /input/build/work/scripts/test_up.sh
222025-03-18T04:59:58.989Zdownloaded input: /input/build/work/scripts/test_up.sh
232025-03-18T04:59:58.989Zdownloading input: /input/build/work/scripts/upstairs_info.d
242025-03-18T04:59:59.006Zdownloaded input: /input/build/work/scripts/upstairs_info.d
252025-03-18T04:59:59.009Zdownloading input: /input/build/tmp/cargo-test-out.log
262025-03-18T04:59:59.676Zdownloaded input: /input/build/tmp/cargo-test-out.log
 
272025-03-18T04:59:59.676Zstarting task 0: "setup"
282025-03-18T04:59:59.679Z++ uname -s
292025-03-18T04:59:59.682Z+ kern=SunOS
302025-03-18T04:59:59.682Z+ build_user=build
312025-03-18T04:59:59.682Z+ build_uid=12345
322025-03-18T04:59:59.682Z+ work_dir=/work
332025-03-18T04:59:59.683Z+ input_dir=/input
342025-03-18T04:59:59.683Z+ [[ 0 == 12345 ]]
352025-03-18T04:59:59.683Z+ case "$kern" in
362025-03-18T04:59:59.683Z+ groupadd -g 12345 build
372025-03-18T04:59:59.686Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
382025-03-18T05:00:01.692Z+ zfs create -o mountpoint=/work rpool/work
392025-03-18T05:00:01.920Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
402025-03-18T05:00:01.923Z+ home_fs=zfs
412025-03-18T05:00:01.923Z+ [[ zfs == autofs ]]
422025-03-18T05:00:01.923Z+ mkdir -p /home/build
432025-03-18T05:00:01.926Z+ chown build:build /home/build /work
442025-03-18T05:00:03.926Z+ chmod 0700 /home/build /work
452025-03-18T05:00:03.929Zprocess exited: duration 4256 ms, exit code 0
 
462025-03-18T05:00:03.935Zstarting task 1: "authentication"
472025-03-18T05:00:03.949ZWARNING: job store has no value for "GITHUB_TOKEN"; waiting for a value...
482025-03-18T05:00:05.958Zprocess exited: duration 2022 ms, exit code 0
 
492025-03-18T05:00:05.965Zstarting task 2: "build"
502025-03-18T05:00:05.970Z+ banner cores
512025-03-18T05:00:05.973Z
522025-03-18T05:00:05.973Z #### #### ##### ###### ####
532025-03-18T05:00:05.973Z # # # # # # # #
542025-03-18T05:00:05.973Z # # # # # ##### ####
552025-03-18T05:00:05.973Z # # # ##### # #
562025-03-18T05:00:05.973Z # # # # # # # # #
572025-03-18T05:00:05.973Z #### #### # # ###### ####
582025-03-18T05:00:05.973Z
592025-03-18T05:00:05.973Z+ pfexec coreadm -i /tmp/core.%f.%p -g /tmp/core.%f.%p -e global -e log -e proc-setid -e global-setid
602025-03-18T05:00:05.982Z+ pfexec chmod +x /input/build/work/scripts/perf-downstairs-tick.d /input/build/work/scripts/test_ds.sh /input/build/work/scripts/test_live_repair.sh /input/build/work/scripts/test_repair.sh /input/build/work/scripts/test_replay.sh /input/build/work/scripts/test_up.sh /input/build/work/scripts/upstairs_info.d
612025-03-18T05:00:05.985Z+ echo ' chmod input script dir contains:'
622025-03-18T05:00:05.985Z chmod input script dir contains:
632025-03-18T05:00:05.985Z+ ls -ltr /input/build/work/scripts
642025-03-18T05:00:05.988Ztotal 46
652025-03-18T05:00:05.988Z-rwxr-xr-x 1 root root 3056 Mar 18 04:59 perf-downstairs-tick.d
662025-03-18T05:00:05.988Z-rwxr-xr-x 1 root root 2127 Mar 18 04:59 test_ds.sh
672025-03-18T05:00:05.988Z-rwxr-xr-x 1 root root 4890 Mar 18 04:59 test_live_repair.sh
682025-03-18T05:00:05.988Z-rwxr-xr-x 1 root root 7419 Mar 18 04:59 test_repair.sh
692025-03-18T05:00:05.988Z-rwxr-xr-x 1 root root 4249 Mar 18 04:59 test_replay.sh
702025-03-18T05:00:05.988Z-rwxr-xr-x 1 root root 13887 Mar 18 04:59 test_up.sh
712025-03-18T05:00:05.988Z-rwxr-xr-x 1 root root 2996 Mar 18 04:59 upstairs_info.d
722025-03-18T05:00:05.988Z+ banner unpack
732025-03-18T05:00:05.991Z
742025-03-18T05:00:05.991Z # # # # ##### ## #### # #
752025-03-18T05:00:05.991Z # # ## # # # # # # # # #
762025-03-18T05:00:05.991Z # # # # # # # # # # ####
772025-03-18T05:00:05.992Z # # # # # ##### ###### # # #
782025-03-18T05:00:05.992Z # # # ## # # # # # # #
792025-03-18T05:00:05.992Z #### # # # # # #### # #
802025-03-18T05:00:05.992Z
812025-03-18T05:00:05.992Z+ mkdir -p /var/tmp/bins
822025-03-18T05:00:05.992Z+ for t in "$input/bins/"*.gz
832025-03-18T05:00:05.995Z++ basename /input/build/work/bins/crucible-downstairs.gz
842025-03-18T05:00:05.995Z+ b=crucible-downstairs.gz
852025-03-18T05:00:05.995Z+ b=crucible-downstairs
862025-03-18T05:00:05.995Z+ gunzip
872025-03-18T05:00:08.422Z+ chmod +x /var/tmp/bins/crucible-downstairs
882025-03-18T05:00:08.425Z+ for t in "$input/bins/"*.gz
892025-03-18T05:00:08.428Z++ basename /input/build/work/bins/crucible-hammer.gz
902025-03-18T05:00:08.428Z+ b=crucible-hammer.gz
912025-03-18T05:00:08.428Z+ b=crucible-hammer
922025-03-18T05:00:08.428Z+ gunzip
932025-03-18T05:00:10.604Z+ chmod +x /var/tmp/bins/crucible-hammer
942025-03-18T05:00:10.607Z+ for t in "$input/bins/"*.gz
952025-03-18T05:00:10.610Z++ basename /input/build/work/bins/crutest.gz
962025-03-18T05:00:10.610Z+ b=crutest.gz
972025-03-18T05:00:10.610Z+ b=crutest
982025-03-18T05:00:10.610Z+ gunzip
992025-03-18T05:00:12.955Z+ chmod +x /var/tmp/bins/crutest
1002025-03-18T05:00:12.959Z+ for t in "$input/bins/"*.gz
1012025-03-18T05:00:12.959Z++ basename /input/build/work/bins/dsc.gz
1022025-03-18T05:00:12.962Z+ b=dsc.gz
1032025-03-18T05:00:12.962Z+ b=dsc
1042025-03-18T05:00:12.962Z+ gunzip
1052025-03-18T05:00:14.326Z+ chmod +x /var/tmp/bins/dsc
1062025-03-18T05:00:14.329Z+ export BINDIR=/var/tmp/bins
1072025-03-18T05:00:14.329Z+ BINDIR=/var/tmp/bins
1082025-03-18T05:00:14.329Z+ export RUST_BACKTRACE=1
1092025-03-18T05:00:14.329Z+ RUST_BACKTRACE=1
1102025-03-18T05:00:14.329Z+ banner setup
1112025-03-18T05:00:14.330Z
1122025-03-18T05:00:14.330Z #### ###### ##### # # #####
1132025-03-18T05:00:14.330Z # # # # # # #
1142025-03-18T05:00:14.330Z #### ##### # # # # #
1152025-03-18T05:00:14.330Z # # # # # #####
1162025-03-18T05:00:14.330Z # # # # # # #
1172025-03-18T05:00:14.330Z #### ###### # #### #
1182025-03-18T05:00:14.330Z
1192025-03-18T05:00:14.330Z+ echo 'Setup self timeout'
1202025-03-18T05:00:14.330ZSetup self timeout
1212025-03-18T05:00:14.330Z+ jobpid=1148
1222025-03-18T05:00:14.330Z+ echo 'Setup debug logging'
1232025-03-18T05:00:14.330ZSetup debug logging
1242025-03-18T05:00:14.330Z+ mkdir /tmp/debug
1252025-03-18T05:00:14.330Z+ sleep 10800
1262025-03-18T05:00:14.333Z+ psrinfo -v
1272025-03-18T05:00:14.336Z+ df -h
1282025-03-18T05:00:14.340Z+ prstat -d d -mLc 1
1292025-03-18T05:00:14.344Z+ iostat -T d -xn 1
1302025-03-18T05:00:14.344Z+ mpstat -T d 1
1312025-03-18T05:00:14.344Z+ vmstat -T d -p 1
1322025-03-18T05:00:14.344Z+ pfexec dtrace -Z -s /input/build/work/scripts/perf-downstairs-tick.d
1332025-03-18T05:00:14.347Z+ banner replay
1342025-03-18T05:00:14.347Z+ pfexec dtrace -Z -s /input/build/work/scripts/upstairs_info.d
1352025-03-18T05:00:14.347Z
1362025-03-18T05:00:14.347Z ##### ###### ##### # ## # #
1372025-03-18T05:00:14.347Z # # # # # # # # # #
1382025-03-18T05:00:14.347Z # # ##### # # # # # #
1392025-03-18T05:00:14.347Z ##### # ##### # ###### #
1402025-03-18T05:00:14.347Z # # # # # # # #
1412025-03-18T05:00:14.347Z # # ###### # ###### # # #
1422025-03-18T05:00:14.347Z
1432025-03-18T05:00:14.348Z+ ptime -m bash /input/build/work/scripts/test_replay.sh
1442025-03-18T05:00:14.367Zstarting March 18, 2025 at 05:00:13 AM UTC
1452025-03-18T05:00:14.370ZTail /tmp/test_replay/test_replay.log for test output
1462025-03-18T05:00:14.373ZCreating 3 downstairs regions
1472025-03-18T05:00:14.772ZStarting 3 downstairs
1482025-03-18T05:00:19.784ZRunning initial fill
1492025-03-18T05:00:24.448ZReplay loop starts now March 18, 2025 at 05:00:23 AM UTC
1502025-03-18T05:07:32.588ZReplays:30 time: 7:08
1512025-03-18T05:07:32.591ZDo final verify
1522025-03-18T05:07:42.125ZMarch 18, 2025 at 05:07:41 AM UTC Test ends with 0
1532025-03-18T05:07:42.134Z
1542025-03-18T05:07:42.137Zreal 7:27.705699025
1552025-03-18T05:07:42.137Zuser 23.939829354
1562025-03-18T05:07:42.137Zsys 6.019957343
1572025-03-18T05:07:42.137Ztrap 0.048554573
1582025-03-18T05:07:42.137Ztflt 0.025099183
1592025-03-18T05:07:42.137Zdflt 0.090624990
1602025-03-18T05:07:42.137Zkflt 0.001132398
1612025-03-18T05:07:42.137Zlock 11:48:06.766088083
1622025-03-18T05:07:42.137Zslp 40:28.308459629
1632025-03-18T05:07:42.137Zlat 5.970440490
1642025-03-18T05:07:42.137Zstop 0.667527065
1652025-03-18T05:07:47.141Zprocess exited: duration 456090 ms, exit code 0
1662025-03-18T05:07:47.141Zexec warning: : stdio descriptors remain open after task exit; waiting 60 seconds for them to close
1672025-03-18T05:08:47.154Zexec warning: : stdout descriptor may be held open by a background process; giving up!
1682025-03-18T05:08:47.154Zexec warning: : stderr descriptor may be held open by a background process; giving up!
 
1692025-03-18T05:08:47.168Zfound 8 output files
1702025-03-18T05:08:47.168Zuploading: /tmp/debug/df.txt (1270 bytes)
1712025-03-18T05:08:48.182Zuploaded: /tmp/debug/df.txt
1722025-03-18T05:08:48.182Zuploading: /tmp/debug/iostat.txt (152885 bytes)
1732025-03-18T05:08:48.193Zupload warning: file "/tmp/debug/iostat.txt" changed size mid upload: 152885 -> 153183
1742025-03-18T05:08:49.202Zuploaded: /tmp/debug/iostat.txt
1752025-03-18T05:08:49.202Zuploading: /tmp/debug/mpstat.txt (372954 bytes)
1762025-03-18T05:08:49.215Zupload warning: file "/tmp/debug/mpstat.txt" changed size mid upload: 372954 -> 374408
1772025-03-18T05:08:50.223Zuploaded: /tmp/debug/mpstat.txt
1782025-03-18T05:08:50.223Zuploading: /tmp/debug/paging.txt (66185 bytes)
1792025-03-18T05:08:50.231Zupload warning: file "/tmp/debug/paging.txt" changed size mid upload: 66185 -> 66524
1802025-03-18T05:08:51.237Zuploaded: /tmp/debug/paging.txt
1812025-03-18T05:08:51.241Zuploading: /tmp/debug/perf.txt (886874 bytes)
1822025-03-18T05:08:52.261Zuploaded: /tmp/debug/perf.txt
1832025-03-18T05:08:52.261Zuploading: /tmp/debug/prstat.txt (699650 bytes)
1842025-03-18T05:08:52.278Zupload warning: file "/tmp/debug/prstat.txt" changed size mid upload: 699650 -> 706118
1852025-03-18T05:08:53.285Zuploaded: /tmp/debug/prstat.txt
1862025-03-18T05:08:53.289Zuploading: /tmp/debug/psrinfo.txt (1528 bytes)
1872025-03-18T05:08:54.296Zuploaded: /tmp/debug/psrinfo.txt
1882025-03-18T05:08:54.296Zuploading: /tmp/debug/upinfo.txt (50160 bytes)
1892025-03-18T05:08:55.308Zuploaded: /tmp/debug/upinfo.txt