01JN5HDP8QXHJYMJNBW6PH275H: test-repair

BasicConfig {
    output_rules: [
        "%/tmp/*.txt",
        "%/tmp/debug/*",
        "/tmp/core.*",
    ],
    rust_toolchain: None,
    target: Some(
        "helios-2.0",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01JN5HE4WTXRCN8B26EWAYM6BP

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12025-02-28T05:59:24.364Zjob dependencies complete; ready to run (waiting for 13 m 25 s)
22025-02-28T06:00:20.768Zjob assigned to worker 01JN5J6JHJCCZW36EJ7P887KV1 [factory aws, i-0d502b750769c6661] (queued for 56 s)
32025-02-28T06:00:28.824Zdownloading input: /input/build/work/bins/crucible-downstairs.gz
42025-02-28T06:00:31.697Zdownloaded input: /input/build/work/bins/crucible-downstairs.gz
52025-02-28T06:00:31.697Zdownloading input: /input/build/work/bins/crucible-hammer.gz
62025-02-28T06:00:33.374Zdownloaded input: /input/build/work/bins/crucible-hammer.gz
72025-02-28T06:00:33.375Zdownloading input: /input/build/work/bins/crutest.gz
82025-02-28T06:00:35.434Zdownloaded input: /input/build/work/bins/crutest.gz
92025-02-28T06:00:35.434Zdownloading input: /input/build/work/bins/dsc.gz
102025-02-28T06:00:36.376Zdownloaded input: /input/build/work/bins/dsc.gz
112025-02-28T06:00:36.387Zdownloading input: /input/build/work/scripts/perf-downstairs-tick.d
122025-02-28T06:00:36.400Zdownloaded input: /input/build/work/scripts/perf-downstairs-tick.d
132025-02-28T06:00:36.401Zdownloading input: /input/build/work/scripts/test_ds.sh
142025-02-28T06:00:36.430Zdownloaded input: /input/build/work/scripts/test_ds.sh
152025-02-28T06:00:36.430Zdownloading input: /input/build/work/scripts/test_live_repair.sh
162025-02-28T06:00:36.455Zdownloaded input: /input/build/work/scripts/test_live_repair.sh
172025-02-28T06:00:36.455Zdownloading input: /input/build/work/scripts/test_repair.sh
182025-02-28T06:00:36.485Zdownloaded input: /input/build/work/scripts/test_repair.sh
192025-02-28T06:00:36.485Zdownloading input: /input/build/work/scripts/test_replay.sh
202025-02-28T06:00:36.513Zdownloaded input: /input/build/work/scripts/test_replay.sh
212025-02-28T06:00:36.513Zdownloading input: /input/build/work/scripts/test_up.sh
222025-02-28T06:00:36.542Zdownloaded input: /input/build/work/scripts/test_up.sh
232025-02-28T06:00:36.542Zdownloading input: /input/build/work/scripts/upstairs_info.d
242025-02-28T06:00:36.564Zdownloaded input: /input/build/work/scripts/upstairs_info.d
252025-02-28T06:00:36.568Zdownloading input: /input/build/tmp/cargo-test-out.log
262025-02-28T06:00:37.444Zdownloaded input: /input/build/tmp/cargo-test-out.log
 
272025-02-28T06:00:37.444Zstarting task 0: "setup"
282025-02-28T06:00:37.461Z++ uname -s
292025-02-28T06:00:37.461Z+ kern=SunOS
302025-02-28T06:00:37.461Z+ build_user=build
312025-02-28T06:00:37.461Z+ build_uid=12345
322025-02-28T06:00:37.461Z+ work_dir=/work
332025-02-28T06:00:37.461Z+ input_dir=/input
342025-02-28T06:00:37.462Z+ [[ 0 == 12345 ]]
352025-02-28T06:00:37.462Z+ case "$kern" in
362025-02-28T06:00:37.462Z+ groupadd -g 12345 build
372025-02-28T06:00:37.462Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
382025-02-28T06:00:39.451Z+ zfs create -o mountpoint=/work rpool/work
392025-02-28T06:00:39.777Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
402025-02-28T06:00:39.787Z+ home_fs=zfs
412025-02-28T06:00:39.788Z+ [[ zfs == autofs ]]
422025-02-28T06:00:39.788Z+ mkdir -p /home/build
432025-02-28T06:00:39.788Z+ chown build:build /home/build /work
442025-02-28T06:00:41.784Z+ chmod 0700 /home/build /work
452025-02-28T06:00:41.792Zprocess exited: duration 4357 ms, exit code 0
 
462025-02-28T06:00:41.807Zstarting task 1: "authentication"
472025-02-28T06:00:41.819ZWARNING: job store has no value for "GITHUB_TOKEN"; waiting for a value...
482025-02-28T06:00:45.841Zprocess exited: duration 4037 ms, exit code 0
 
492025-02-28T06:00:45.848Zstarting task 2: "build"
502025-02-28T06:00:45.854Z+ banner cores
512025-02-28T06:00:45.860Z
522025-02-28T06:00:45.860Z #### #### ##### ###### ####
532025-02-28T06:00:45.860Z # # # # # # # #
542025-02-28T06:00:45.860Z # # # # # ##### ####
552025-02-28T06:00:45.860Z # # # ##### # #
562025-02-28T06:00:45.860Z # # # # # # # # #
572025-02-28T06:00:45.860Z #### #### # # ###### ####
582025-02-28T06:00:45.860Z
592025-02-28T06:00:45.860Z+ pfexec coreadm -i /tmp/core.%f.%p -g /tmp/core.%f.%p -e global -e log -e proc-setid -e global-setid
602025-02-28T06:00:45.867Z+ echo 'input bins dir contains:'
612025-02-28T06:00:45.867Zinput bins dir contains:
622025-02-28T06:00:45.871Z+ ls -ltr /input/build/work/bins
632025-02-28T06:00:45.872Ztotal 768616
642025-02-28T06:00:45.872Z-rw-r--r-- 1 root root 110137183 Feb 28 06:00 crucible-downstairs.gz
652025-02-28T06:00:45.872Z-rw-r--r-- 1 root root 103831748 Feb 28 06:00 crucible-hammer.gz
662025-02-28T06:00:45.872Z-rw-r--r-- 1 root root 113051968 Feb 28 06:00 crutest.gz
672025-02-28T06:00:45.872Z-rw-r--r-- 1 root root 66258622 Feb 28 06:00 dsc.gz
682025-02-28T06:00:45.872Z+ echo 'input script dir contains:'
692025-02-28T06:00:45.872Zinput script dir contains:
702025-02-28T06:00:45.872Z+ ls -ltr /input/build/work/scripts
712025-02-28T06:00:45.872Ztotal 43
722025-02-28T06:00:45.873Z-rw-r--r-- 1 root root 3056 Feb 28 06:00 perf-downstairs-tick.d
732025-02-28T06:00:45.873Z-rw-r--r-- 1 root root 2127 Feb 28 06:00 test_ds.sh
742025-02-28T06:00:45.873Z-rw-r--r-- 1 root root 4272 Feb 28 06:00 test_live_repair.sh
752025-02-28T06:00:45.873Z-rw-r--r-- 1 root root 6724 Feb 28 06:00 test_repair.sh
762025-02-28T06:00:45.873Z-rw-r--r-- 1 root root 3494 Feb 28 06:00 test_replay.sh
772025-02-28T06:00:45.873Z-rw-r--r-- 1 root root 13887 Feb 28 06:00 test_up.sh
782025-02-28T06:00:45.873Z-rw-r--r-- 1 root root 2996 Feb 28 06:00 upstairs_info.d
792025-02-28T06:00:45.873Z+ pfexec chmod +x /input/build/work/scripts/perf-downstairs-tick.d /input/build/work/scripts/test_ds.sh /input/build/work/scripts/test_live_repair.sh /input/build/work/scripts/test_repair.sh /input/build/work/scripts/test_replay.sh /input/build/work/scripts/test_up.sh /input/build/work/scripts/upstairs_info.d
802025-02-28T06:00:45.877Z+ echo ' chmod input script dir contains:'
812025-02-28T06:00:45.877Z+ ls -ltr /input/build/work/scripts
822025-02-28T06:00:45.877Z chmod input script dir contains:
832025-02-28T06:00:45.877Ztotal 43
842025-02-28T06:00:45.877Z-rwxr-xr-x 1 root root 3056 Feb 28 06:00 perf-downstairs-tick.d
852025-02-28T06:00:45.877Z-rwxr-xr-x 1 root root 2127 Feb 28 06:00 test_ds.sh
862025-02-28T06:00:45.877Z-rwxr-xr-x 1 root root 4272 Feb 28 06:00 test_live_repair.sh
872025-02-28T06:00:45.877Z-rwxr-xr-x 1 root root 6724 Feb 28 06:00 test_repair.sh
882025-02-28T06:00:45.878Z-rwxr-xr-x 1 root root 3494 Feb 28 06:00 test_replay.sh
892025-02-28T06:00:45.878Z-rwxr-xr-x 1 root root 13887 Feb 28 06:00 test_up.sh
902025-02-28T06:00:45.878Z-rwxr-xr-x 1 root root 2996 Feb 28 06:00 upstairs_info.d
912025-02-28T06:00:45.878Z+ banner unpack
922025-02-28T06:00:45.882Z
932025-02-28T06:00:45.882Z # # # # ##### ## #### # #
942025-02-28T06:00:45.882Z # # ## # # # # # # # # #
952025-02-28T06:00:45.882Z # # # # # # # # # # ####
962025-02-28T06:00:45.882Z # # # # # ##### ###### # # #
972025-02-28T06:00:45.882Z # # # ## # # # # # # #
982025-02-28T06:00:45.882Z #### # # # # # #### # #
992025-02-28T06:00:45.882Z
1002025-02-28T06:00:45.882Z+ mkdir -p /var/tmp/bins
1012025-02-28T06:00:45.882Z+ for t in "$input/bins/"*.gz
1022025-02-28T06:00:45.883Z++ basename /input/build/work/bins/crucible-downstairs.gz
1032025-02-28T06:00:45.883Z+ b=crucible-downstairs.gz
1042025-02-28T06:00:45.883Z+ b=crucible-downstairs
1052025-02-28T06:00:45.883Z+ gunzip
1062025-02-28T06:00:48.173Z+ chmod +x /var/tmp/bins/crucible-downstairs
1072025-02-28T06:00:48.182Z+ for t in "$input/bins/"*.gz
1082025-02-28T06:00:48.182Z++ basename /input/build/work/bins/crucible-hammer.gz
1092025-02-28T06:00:48.182Z+ b=crucible-hammer.gz
1102025-02-28T06:00:48.182Z+ b=crucible-hammer
1112025-02-28T06:00:48.182Z+ gunzip
1122025-02-28T06:00:50.344Z+ chmod +x /var/tmp/bins/crucible-hammer
1132025-02-28T06:00:50.347Z+ for t in "$input/bins/"*.gz
1142025-02-28T06:00:50.347Z++ basename /input/build/work/bins/crutest.gz
1152025-02-28T06:00:50.349Z+ b=crutest.gz
1162025-02-28T06:00:50.350Z+ b=crutest
1172025-02-28T06:00:50.350Z+ gunzip
1182025-02-28T06:00:52.641Z+ chmod +x /var/tmp/bins/crutest
1192025-02-28T06:00:52.646Z+ for t in "$input/bins/"*.gz
1202025-02-28T06:00:52.646Z++ basename /input/build/work/bins/dsc.gz
1212025-02-28T06:00:52.646Z+ b=dsc.gz
1222025-02-28T06:00:52.646Z+ b=dsc
1232025-02-28T06:00:52.646Z+ gunzip
1242025-02-28T06:00:54.046Z+ chmod +x /var/tmp/bins/dsc
1252025-02-28T06:00:54.050Z+ export BINDIR=/var/tmp/bins
1262025-02-28T06:00:54.050Z+ BINDIR=/var/tmp/bins
1272025-02-28T06:00:54.050Z+ export RUST_BACKTRACE=1
1282025-02-28T06:00:54.050Z+ RUST_BACKTRACE=1
1292025-02-28T06:00:54.050Z+ echo 'Setup self timeout'
1302025-02-28T06:00:54.050ZSetup self timeout
1312025-02-28T06:00:54.050Z+ jobpid=1150
1322025-02-28T06:00:54.050Z+ echo 'Setup debug logging'
1332025-02-28T06:00:54.050ZSetup debug logging
1342025-02-28T06:00:54.050Z+ mkdir /tmp/debug
1352025-02-28T06:00:54.050Z+ sleep 7200
1362025-02-28T06:00:54.053Z+ psrinfo -v
1372025-02-28T06:00:54.053Z+ df -h
1382025-02-28T06:00:54.062Z+ prstat -d d -mLc 1
1392025-02-28T06:00:54.065Z+ iostat -T d -xn 1
1402025-02-28T06:00:54.066Z+ mpstat -T d 1
1412025-02-28T06:00:54.066Z+ vmstat -T d -p 1
1422025-02-28T06:00:54.066Z+ pfexec dtrace -Z -s /input/build/work/scripts/perf-downstairs-tick.d
1432025-02-28T06:00:54.066Z+ banner repair
1442025-02-28T06:00:54.066Z+ pfexec dtrace -Z -s /input/build/work/scripts/upstairs_info.d
1452025-02-28T06:00:54.070Z
1462025-02-28T06:00:54.070Z ##### ###### ##### ## # #####
1472025-02-28T06:00:54.070Z # # # # # # # # # #
1482025-02-28T06:00:54.070Z # # ##### # # # # # # #
1492025-02-28T06:00:54.071Z ##### # ##### ###### # #####
1502025-02-28T06:00:54.071Z # # # # # # # # #
1512025-02-28T06:00:54.071Z # # ###### # # # # # #
1522025-02-28T06:00:54.071Z
1532025-02-28T06:00:54.071Z+ ptime -m bash /input/build/work/scripts/test_repair.sh -N
1542025-02-28T06:08:28.197Z+ echo 'Test repair finished with 0'
1552025-02-28T06:08:28.197ZTest repair finished with 0
1562025-02-28T06:08:33.202Zprocess exited: duration 462271 ms, exit code 0
1572025-02-28T06:08:33.202Zexec warning: : stdio descriptors remain open after task exit; waiting 60 seconds for them to close
1582025-02-28T06:09:33.256Zexec warning: : stdout descriptor may be held open by a background process; giving up!
1592025-02-28T06:09:33.256Zexec warning: : stderr descriptor may be held open by a background process; giving up!
 
1602025-02-28T06:09:33.269Zfound 13 output files
1612025-02-28T06:09:33.269Zuploading: /tmp/test-repair-out.txt (2232212 bytes)
1622025-02-28T06:09:34.330Zuploaded: /tmp/test-repair-out.txt
1632025-02-28T06:09:34.333Zuploading: /tmp/test_repair_ds8810.txt (124435 bytes)
1642025-02-28T06:09:35.349Zuploaded: /tmp/test_repair_ds8810.txt
1652025-02-28T06:09:35.352Zuploading: /tmp/test_repair_ds8820.txt (12714 bytes)
1662025-02-28T06:09:36.359Zuploaded: /tmp/test_repair_ds8820.txt
1672025-02-28T06:09:36.362Zuploading: /tmp/test_repair_ds8830.txt (34692 bytes)
1682025-02-28T06:09:37.372Zuploaded: /tmp/test_repair_ds8830.txt
1692025-02-28T06:09:37.376Zuploading: /tmp/test_repair_out.txt (31184 bytes)
1702025-02-28T06:09:38.385Zuploaded: /tmp/test_repair_out.txt
1712025-02-28T06:09:38.387Zuploading: /tmp/debug/df.txt (1270 bytes)
1722025-02-28T06:09:39.395Zuploaded: /tmp/debug/df.txt
1732025-02-28T06:09:39.396Zuploading: /tmp/debug/dtrace.txt (3826448 bytes)
1742025-02-28T06:09:40.503Zuploaded: /tmp/debug/dtrace.txt
1752025-02-28T06:09:40.505Zuploading: /tmp/debug/iostat.txt (156531 bytes)
1762025-02-28T06:09:40.515Zupload warning: file "/tmp/debug/iostat.txt" changed size mid upload: 156531 -> 158638
1772025-02-28T06:09:41.522Zuploaded: /tmp/debug/iostat.txt
1782025-02-28T06:09:41.527Zuploading: /tmp/debug/mpstat.txt (379652 bytes)
1792025-02-28T06:09:41.540Zupload warning: file "/tmp/debug/mpstat.txt" changed size mid upload: 379652 -> 385492
1802025-02-28T06:09:42.552Zuploaded: /tmp/debug/mpstat.txt
1812025-02-28T06:09:42.554Zuploading: /tmp/debug/paging.txt (68536 bytes)
1822025-02-28T06:09:42.559Zupload warning: file "/tmp/debug/paging.txt" changed size mid upload: 68536 -> 69738
1832025-02-28T06:09:43.572Zuploaded: /tmp/debug/paging.txt
1842025-02-28T06:09:43.572Zuploading: /tmp/debug/prstat.txt (726009 bytes)
1852025-02-28T06:09:43.607Zupload warning: file "/tmp/debug/prstat.txt" changed size mid upload: 726009 -> 738945
1862025-02-28T06:09:44.611Zuploaded: /tmp/debug/prstat.txt
1872025-02-28T06:09:44.614Zuploading: /tmp/debug/psrinfo.txt (1528 bytes)
1882025-02-28T06:09:45.621Zuploaded: /tmp/debug/psrinfo.txt
1892025-02-28T06:09:45.624Zuploading: /tmp/debug/upstairs-info.txt (48510 bytes)
1902025-02-28T06:09:46.632Zuploaded: /tmp/debug/upstairs-info.txt