01JN5HDP8ES2AYBQQM4Q6SVC3G: test-live-repair

BasicConfig {
    output_rules: [
        "/tmp/*.txt",
        "/tmp/*.log",
        "%/tmp/debug/*",
        "/tmp/core.*",
        "/tmp/dsc/*.txt",
        "/tmp/dsc.tar",
    ],
    rust_toolchain: None,
    target: Some(
        "helios-2.0",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01JN5HE4RRD9ZFVYT0WRB7RF7A

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12025-02-28T05:59:24.360Zjob dependencies complete; ready to run (waiting for 13 m 26 s)
22025-02-28T06:00:20.764Zjob assigned to worker 01JN5J6Z1H5GZ2ZRHNT6GZ5GES [factory aws, i-01c7d781dcef95f99] (queued for 56 s)
32025-02-28T06:00:28.568Zdownloading input: /input/build/work/bins/crucible-downstairs.gz
42025-02-28T06:00:31.547Zdownloaded input: /input/build/work/bins/crucible-downstairs.gz
52025-02-28T06:00:31.547Zdownloading input: /input/build/work/bins/crucible-hammer.gz
62025-02-28T06:00:33.266Zdownloaded input: /input/build/work/bins/crucible-hammer.gz
72025-02-28T06:00:33.266Zdownloading input: /input/build/work/bins/crutest.gz
82025-02-28T06:00:34.836Zdownloaded input: /input/build/work/bins/crutest.gz
92025-02-28T06:00:34.836Zdownloading input: /input/build/work/bins/dsc.gz
102025-02-28T06:00:35.856Zdownloaded input: /input/build/work/bins/dsc.gz
112025-02-28T06:00:35.871Zdownloading input: /input/build/work/scripts/perf-downstairs-tick.d
122025-02-28T06:00:35.897Zdownloaded input: /input/build/work/scripts/perf-downstairs-tick.d
132025-02-28T06:00:35.897Zdownloading input: /input/build/work/scripts/test_ds.sh
142025-02-28T06:00:35.937Zdownloaded input: /input/build/work/scripts/test_ds.sh
152025-02-28T06:00:35.937Zdownloading input: /input/build/work/scripts/test_live_repair.sh
162025-02-28T06:00:35.958Zdownloaded input: /input/build/work/scripts/test_live_repair.sh
172025-02-28T06:00:35.958Zdownloading input: /input/build/work/scripts/test_repair.sh
182025-02-28T06:00:35.998Zdownloaded input: /input/build/work/scripts/test_repair.sh
192025-02-28T06:00:35.998Zdownloading input: /input/build/work/scripts/test_replay.sh
202025-02-28T06:00:36.029Zdownloaded input: /input/build/work/scripts/test_replay.sh
212025-02-28T06:00:36.029Zdownloading input: /input/build/work/scripts/test_up.sh
222025-02-28T06:00:36.064Zdownloaded input: /input/build/work/scripts/test_up.sh
232025-02-28T06:00:36.064Zdownloading input: /input/build/work/scripts/upstairs_info.d
242025-02-28T06:00:36.091Zdownloaded input: /input/build/work/scripts/upstairs_info.d
252025-02-28T06:00:36.103Zdownloading input: /input/build/tmp/cargo-test-out.log
262025-02-28T06:00:36.946Zdownloaded input: /input/build/tmp/cargo-test-out.log
 
272025-02-28T06:00:36.947Zstarting task 0: "setup"
282025-02-28T06:00:36.956Z++ uname -s
292025-02-28T06:00:36.956Z+ kern=SunOS
302025-02-28T06:00:36.956Z+ build_user=build
312025-02-28T06:00:36.956Z+ build_uid=12345
322025-02-28T06:00:36.956Z+ work_dir=/work
332025-02-28T06:00:36.956Z+ input_dir=/input
342025-02-28T06:00:36.956Z+ [[ 0 == 12345 ]]
352025-02-28T06:00:36.956Z+ case "$kern" in
362025-02-28T06:00:36.956Z+ groupadd -g 12345 build
372025-02-28T06:00:36.957Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
382025-02-28T06:00:38.957Z+ zfs create -o mountpoint=/work rpool/work
392025-02-28T06:00:39.226Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
402025-02-28T06:00:39.230Z+ home_fs=zfs
412025-02-28T06:00:39.231Z+ [[ zfs == autofs ]]
422025-02-28T06:00:39.231Z+ mkdir -p /home/build
432025-02-28T06:00:39.231Z+ chown build:build /home/build /work
442025-02-28T06:00:41.234Z+ chmod 0700 /home/build /work
452025-02-28T06:00:41.240Zprocess exited: duration 4298 ms, exit code 0
 
462025-02-28T06:00:41.254Zstarting task 1: "authentication"
472025-02-28T06:00:41.270ZWARNING: job store has no value for "GITHUB_TOKEN"; waiting for a value...
482025-02-28T06:00:45.283Zprocess exited: duration 4029 ms, exit code 0
 
492025-02-28T06:00:45.290Zstarting task 2: "build"
502025-02-28T06:00:45.295Z+ banner cores
512025-02-28T06:00:45.298Z
522025-02-28T06:00:45.298Z #### #### ##### ###### ####
532025-02-28T06:00:45.298Z # # # # # # # #
542025-02-28T06:00:45.298Z # # # # # ##### ####
552025-02-28T06:00:45.298Z # # # ##### # #
562025-02-28T06:00:45.298Z # # # # # # # # #
572025-02-28T06:00:45.298Z #### #### # # ###### ####
582025-02-28T06:00:45.299Z
592025-02-28T06:00:45.299Z+ pfexec coreadm -i /tmp/core.%f.%p -g /tmp/core.%f.%p -e global -e log -e proc-setid -e global-setid
602025-02-28T06:00:45.307Z+ pfexec chmod +x /input/build/work/scripts/perf-downstairs-tick.d /input/build/work/scripts/test_ds.sh /input/build/work/scripts/test_live_repair.sh /input/build/work/scripts/test_repair.sh /input/build/work/scripts/test_replay.sh /input/build/work/scripts/test_up.sh /input/build/work/scripts/upstairs_info.d
612025-02-28T06:00:45.311Z+ echo 'input bins dir contains:'
622025-02-28T06:00:45.311Zinput bins dir contains:
632025-02-28T06:00:45.311Z+ ls -ltr /input/build/work/bins
642025-02-28T06:00:45.314Ztotal 768616
652025-02-28T06:00:45.314Z-rw-r--r-- 1 root root 110137183 Feb 28 06:00 crucible-downstairs.gz
662025-02-28T06:00:45.314Z-rw-r--r-- 1 root root 103831748 Feb 28 06:00 crucible-hammer.gz
672025-02-28T06:00:45.314Z-rw-r--r-- 1 root root 113051968 Feb 28 06:00 crutest.gz
682025-02-28T06:00:45.314Z-rw-r--r-- 1 root root 66258622 Feb 28 06:00 dsc.gz
692025-02-28T06:00:45.314Z+ echo 'input script dir contains:'
702025-02-28T06:00:45.314Zinput script dir contains:
712025-02-28T06:00:45.314Z+ ls -ltr /input/build/work/scripts
722025-02-28T06:00:45.318Ztotal 43
732025-02-28T06:00:45.318Z-rwxr-xr-x 1 root root 3056 Feb 28 06:00 perf-downstairs-tick.d
742025-02-28T06:00:45.318Z-rwxr-xr-x 1 root root 2127 Feb 28 06:00 test_ds.sh
752025-02-28T06:00:45.318Z-rwxr-xr-x 1 root root 4272 Feb 28 06:00 test_live_repair.sh
762025-02-28T06:00:45.318Z-rwxr-xr-x 1 root root 6724 Feb 28 06:00 test_repair.sh
772025-02-28T06:00:45.318Z-rwxr-xr-x 1 root root 3494 Feb 28 06:00 test_replay.sh
782025-02-28T06:00:45.318Z-rwxr-xr-x 1 root root 13887 Feb 28 06:00 test_up.sh
792025-02-28T06:00:45.318Z-rwxr-xr-x 1 root root 2996 Feb 28 06:00 upstairs_info.d
802025-02-28T06:00:45.318Z+ banner Unpack
812025-02-28T06:00:45.318Z# #
822025-02-28T06:00:45.318Z# # # # ##### ## #### # #
832025-02-28T06:00:45.318Z# # ## # # # # # # # # #
842025-02-28T06:00:45.318Z# # # # # # # # # # ####
852025-02-28T06:00:45.318Z# # # # # ##### ###### # # #
862025-02-28T06:00:45.318Z# # # ## # # # # # # #
872025-02-28T06:00:45.318Z ##### # # # # # #### # #
882025-02-28T06:00:45.318Z
892025-02-28T06:00:45.318Z+ mkdir -p /var/tmp/bins
902025-02-28T06:00:45.321Z+ for t in "$input/bins/"*.gz
912025-02-28T06:00:45.321Z++ basename /input/build/work/bins/crucible-downstairs.gz
922025-02-28T06:00:45.322Z+ b=crucible-downstairs.gz
932025-02-28T06:00:45.322Z+ b=crucible-downstairs
942025-02-28T06:00:45.322Z+ gunzip
952025-02-28T06:00:47.582Z+ chmod +x /var/tmp/bins/crucible-downstairs
962025-02-28T06:00:47.601Z+ for t in "$input/bins/"*.gz
972025-02-28T06:00:47.601Z++ basename /input/build/work/bins/crucible-hammer.gz
982025-02-28T06:00:47.601Z+ b=crucible-hammer.gz
992025-02-28T06:00:47.601Z+ b=crucible-hammer
1002025-02-28T06:00:47.601Z+ gunzip
1012025-02-28T06:00:49.755Z+ chmod +x /var/tmp/bins/crucible-hammer
1022025-02-28T06:00:49.760Z+ for t in "$input/bins/"*.gz
1032025-02-28T06:00:49.760Z++ basename /input/build/work/bins/crutest.gz
1042025-02-28T06:00:49.760Z+ b=crutest.gz
1052025-02-28T06:00:49.760Z+ b=crutest
1062025-02-28T06:00:49.760Z+ gunzip
1072025-02-28T06:00:52.155Z+ chmod +x /var/tmp/bins/crutest
1082025-02-28T06:00:52.249Z+ for t in "$input/bins/"*.gz
1092025-02-28T06:00:52.249Z++ basename /input/build/work/bins/dsc.gz
1102025-02-28T06:00:52.249Z+ b=dsc.gz
1112025-02-28T06:00:52.249Z+ b=dsc
1122025-02-28T06:00:52.249Z+ gunzip
1132025-02-28T06:00:53.451Z+ chmod +x /var/tmp/bins/dsc
1142025-02-28T06:00:53.457Z+ export BINDIR=/var/tmp/bins
1152025-02-28T06:00:53.457Z+ BINDIR=/var/tmp/bins
1162025-02-28T06:00:53.457Z+ export RUST_BACKTRACE=1
1172025-02-28T06:00:53.457Z+ RUST_BACKTRACE=1
1182025-02-28T06:00:53.457Z+ echo 'BINDIR is /var/tmp/bins'
1192025-02-28T06:00:53.457ZBINDIR is /var/tmp/bins
1202025-02-28T06:00:53.457Zbindir contains:
1212025-02-28T06:00:53.458Z+ echo 'bindir contains:'
1222025-02-28T06:00:53.458Z+ ls -ltr /var/tmp/bins
1232025-02-28T06:00:53.458Ztotal 695201
1242025-02-28T06:00:53.458Z-rwxr-xr-x 1 build build 405415320 Feb 28 06:00 crucible-downstairs
1252025-02-28T06:00:53.458Z-rwxr-xr-x 1 build build 380516800 Feb 28 06:00 crucible-hammer
1262025-02-28T06:00:53.458Z-rwxr-xr-x 1 build build 415671000 Feb 28 06:00 crutest
1272025-02-28T06:00:53.458Z-rwxr-xr-x 1 build build 239424496 Feb 28 06:00 dsc
1282025-02-28T06:00:53.459Z+ banner CreateDS
1292025-02-28T06:00:53.463Z ##### ###### #####
1302025-02-28T06:00:53.463Z# # ##### ###### ## ##### ###### # # # #
1312025-02-28T06:00:53.463Z# # # # # # # # # # #
1322025-02-28T06:00:53.463Z# # # ##### # # # ##### # # #####
1332025-02-28T06:00:53.463Z# ##### # ###### # # # # #
1342025-02-28T06:00:53.463Z# # # # # # # # # # # # #
1352025-02-28T06:00:53.463Z ##### # # ###### # # # ###### ###### #####
1362025-02-28T06:00:53.463Z
1372025-02-28T06:00:53.463Z+ echo /var/tmp/bins/dsc create --ds-bin /var/tmp/bins/crucible-downstairs --extent-size 4000 --extent-count 200 --region-count 4 --cleanup
1382025-02-28T06:00:53.463Z/var/tmp/bins/dsc create --ds-bin /var/tmp/bins/crucible-downstairs --extent-size 4000 --extent-count 200 --region-count 4 --cleanup
1392025-02-28T06:00:53.463Z+ /var/tmp/bins/dsc create --ds-bin /var/tmp/bins/crucible-downstairs --extent-size 4000 --extent-count 200 --region-count 4 --cleanup
1402025-02-28T06:00:53.474ZCreating region directory at: /var/tmp/dsc/region
1412025-02-28T06:00:53.504ZCreating dsc directory at: /tmp/dsc
1422025-02-28T06:00:53.638ZDownstairs region 0 created at /var/tmp/dsc/region/8810 in 0.16166304
1432025-02-28T06:00:53.754ZDownstairs region 1 created at /var/tmp/dsc/region/8820 in 0.11238355
1442025-02-28T06:00:53.861ZDownstairs region 2 created at /var/tmp/dsc/region/8830 in 0.11177651
1452025-02-28T06:00:53.974ZDownstairs region 3 created at /var/tmp/dsc/region/8840 in 0.113434
1462025-02-28T06:00:53.974ZCreated 4 regions
1472025-02-28T06:00:53.978Z+ banner StartDS
1482025-02-28T06:00:53.981Z ##### ###### #####
1492025-02-28T06:00:53.981Z# # ##### ## ##### ##### # # # #
1502025-02-28T06:00:53.981Z# # # # # # # # # #
1512025-02-28T06:00:53.982Z ##### # # # # # # # # #####
1522025-02-28T06:00:53.982Z # # ###### ##### # # # #
1532025-02-28T06:00:53.982Z# # # # # # # # # # # #
1542025-02-28T06:00:53.982Z ##### # # # # # # ###### #####
1552025-02-28T06:00:53.982Z
1562025-02-28T06:00:53.982Z+ sleep 5
1572025-02-28T06:00:53.982Z+ /var/tmp/bins/dsc start --ds-bin /var/tmp/bins/crucible-downstairs --region-count 4
1582025-02-28T06:01:03.435Z++ pgrep dsc
1592025-02-28T06:01:03.497Z+ dsc_pid=1174
1602025-02-28T06:01:03.497Z+ [[ 1174 -eq 0 ]]
1612025-02-28T06:01:03.497Z+ ps -p 1174
1622025-02-28T06:01:03.498Z PID TTY TIME CMD
1632025-02-28T06:01:03.498Z 1174 ? 0:00 dsc
1642025-02-28T06:01:03.498Z+ echo 'Found dsc running, continue tests'
1652025-02-28T06:01:03.498ZFound dsc running, continue tests
1662025-02-28T06:01:03.498Z+ echo 'Setup self timeout'
1672025-02-28T06:01:03.498ZSetup self timeout
1682025-02-28T06:01:03.498Z+ jobpid=1145
1692025-02-28T06:01:03.498Z+ echo 'Setup debug logging'
1702025-02-28T06:01:03.498ZSetup debug logging
1712025-02-28T06:01:03.498Z+ mkdir /tmp/debug
1722025-02-28T06:01:03.498Z+ sleep 7200
1732025-02-28T06:01:03.498Z+ psrinfo -v
1742025-02-28T06:01:03.498Z+ df -h
1752025-02-28T06:01:03.498Z+ prstat -d d -mLc 1
1762025-02-28T06:01:03.498Z+ iostat -T d -xn 1
1772025-02-28T06:01:03.498Z+ mpstat -T d 1
1782025-02-28T06:01:03.498Z+ vmstat -T d -p 1
1792025-02-28T06:01:03.498Z+ pfexec dtrace -Z -s /input/build/work/scripts/perf-downstairs-tick.d
1802025-02-28T06:01:03.498Z+ banner LR
1812025-02-28T06:01:03.498Z+ pfexec dtrace -Z -s /input/build/work/scripts/upstairs_info.d
1822025-02-28T06:01:03.498Z# ######
1832025-02-28T06:01:03.498Z# # #
1842025-02-28T06:01:03.498Z# # #
1852025-02-28T06:01:03.498Z# ######
1862025-02-28T06:01:03.498Z# # #
1872025-02-28T06:01:03.498Z# # #
1882025-02-28T06:01:03.498Z####### # #
1892025-02-28T06:01:03.498Z
1902025-02-28T06:01:03.498Z+ ptime -m /var/tmp/bins/crutest replace -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 --replacement 127.0.0.1:8840 -g 1 -c 10 --stable
1912025-02-28T06:09:03.605Z+ banner StopDSC
1922025-02-28T06:09:03.608Z ##### ###### ##### #####
1932025-02-28T06:09:03.609Z# # ##### #### ##### # # # # # #
1942025-02-28T06:09:03.609Z# # # # # # # # # #
1952025-02-28T06:09:03.609Z ##### # # # # # # # ##### #
1962025-02-28T06:09:03.609Z # # # # ##### # # # #
1972025-02-28T06:09:03.609Z# # # # # # # # # # # #
1982025-02-28T06:09:03.609Z ##### # #### # ###### ##### #####
1992025-02-28T06:09:03.609Z
2002025-02-28T06:09:03.609Z+ /var/tmp/bins/dsc cmd shutdown
2012025-02-28T06:09:03.687Z+ banner WaitStop
2022025-02-28T06:09:03.691Z# # #####
2032025-02-28T06:09:03.691Z# # # ## # ##### # # ##### #### #####
2042025-02-28T06:09:03.691Z# # # # # # # # # # # # #
2052025-02-28T06:09:03.691Z# # # # # # # ##### # # # # #
2062025-02-28T06:09:03.691Z# # # ###### # # # # # # #####
2072025-02-28T06:09:03.691Z# # # # # # # # # # # # #
2082025-02-28T06:09:03.691Z ## ## # # # # ##### # #### #
2092025-02-28T06:09:03.691Z
2102025-02-28T06:09:03.691Z+ wait 1174
2112025-02-28T06:09:04.686Z+ _exit_trap
2122025-02-28T06:09:04.686Z+ local status=0
2132025-02-28T06:09:04.688Z+ [[ 0 -eq 0 ]]
2142025-02-28T06:09:04.688Z+ exit 0
2152025-02-28T06:09:09.692Zprocess exited: duration 499326 ms, exit code 0
2162025-02-28T06:09:09.692Zexec warning: : stdio descriptors remain open after task exit; waiting 60 seconds for them to close
2172025-02-28T06:10:09.742Zexec warning: : stdout descriptor may be held open by a background process; giving up!
2182025-02-28T06:10:09.742Zexec warning: : stderr descriptor may be held open by a background process; giving up!
 
2192025-02-28T06:10:09.755Zfound 14 output files
2202025-02-28T06:10:09.755Zuploading: /tmp/crutest-replace.log (18723598 bytes)
2212025-02-28T06:10:11.051Zuploaded: /tmp/crutest-replace.log
2222025-02-28T06:10:11.052Zuploading: /tmp/dsc.log (2287 bytes)
2232025-02-28T06:10:12.061Zuploaded: /tmp/dsc.log
2242025-02-28T06:10:12.061Zuploading: /tmp/debug/df.txt (1270 bytes)
2252025-02-28T06:10:13.070Zuploaded: /tmp/debug/df.txt
2262025-02-28T06:10:13.070Zuploading: /tmp/debug/dtrace.txt (310961 bytes)
2272025-02-28T06:10:14.089Zuploaded: /tmp/debug/dtrace.txt
2282025-02-28T06:10:14.089Zuploading: /tmp/debug/iostat.txt (165915 bytes)
2292025-02-28T06:10:14.102Zupload warning: file "/tmp/debug/iostat.txt" changed size mid upload: 165915 -> 167420
2302025-02-28T06:10:15.109Zuploaded: /tmp/debug/iostat.txt
2312025-02-28T06:10:15.112Zuploading: /tmp/debug/mpstat.txt (403557 bytes)
2322025-02-28T06:10:15.127Zupload warning: file "/tmp/debug/mpstat.txt" changed size mid upload: 403557 -> 407937
2332025-02-28T06:10:16.135Zuploaded: /tmp/debug/mpstat.txt
2342025-02-28T06:10:16.138Zuploading: /tmp/debug/paging.txt (72764 bytes)
2352025-02-28T06:10:16.143Zupload warning: file "/tmp/debug/paging.txt" changed size mid upload: 72764 -> 73576
2362025-02-28T06:10:17.149Zuploaded: /tmp/debug/paging.txt
2372025-02-28T06:10:17.149Zuploading: /tmp/debug/prstat.txt (882341 bytes)
2382025-02-28T06:10:17.177Zupload warning: file "/tmp/debug/prstat.txt" changed size mid upload: 882341 -> 891395
2392025-02-28T06:10:18.184Zuploaded: /tmp/debug/prstat.txt
2402025-02-28T06:10:18.184Zuploading: /tmp/debug/psrinfo.txt (1528 bytes)
2412025-02-28T06:10:19.194Zuploaded: /tmp/debug/psrinfo.txt
2422025-02-28T06:10:19.194Zuploading: /tmp/debug/upstairs-info.txt (55770 bytes)
2432025-02-28T06:10:20.208Zuploaded: /tmp/debug/upstairs-info.txt
2442025-02-28T06:10:20.210Zuploading: /tmp/dsc/downstairs-8810.txt (1463953 bytes)
2452025-02-28T06:10:21.251Zuploaded: /tmp/dsc/downstairs-8810.txt
2462025-02-28T06:10:21.251Zuploading: /tmp/dsc/downstairs-8820.txt (978999 bytes)
2472025-02-28T06:10:22.287Zuploaded: /tmp/dsc/downstairs-8820.txt
2482025-02-28T06:10:22.289Zuploading: /tmp/dsc/downstairs-8830.txt (1197389 bytes)
2492025-02-28T06:10:23.329Zuploaded: /tmp/dsc/downstairs-8830.txt
2502025-02-28T06:10:23.329Zuploading: /tmp/dsc/downstairs-8840.txt (1242284 bytes)
2512025-02-28T06:10:24.371Zuploaded: /tmp/dsc/downstairs-8840.txt