01HFJZMPGAC9RB35Q8FTN7DDZS: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HFJZNVJ3A78ASSGGN4QT0BPA

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-19T05:59:38.398Zjob dependencies complete; ready to run (waiting for 51 m 9 s)
22023-11-19T06:06:01.667Zjob assigned to worker 01HFK2M0HXVEKNQBQVM91K782W (queued for 6 m 23 s)
32023-11-19T06:06:01.669Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-11-19T06:06:01.723Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-11-19T06:06:01.770Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-11-19T06:06:01.813Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-19T06:06:01.858Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-11-19T06:06:01.905ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-19T06:06:01.954Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-19T06:06:02.014ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-19T06:06:02.068ZConfiguring devices.
122023-11-19T06:06:02.135Z#################################################
132023-11-19T06:06:02.195Z#### ############# ##
142023-11-19T06:06:02.237Z### ### ############ ##
152023-11-19T06:06:02.277Z## ### # ## ### ## Oxide Computer Company ##
162023-11-19T06:06:02.319Z## ## ## ### # ### ##
172023-11-19T06:06:02.364Z## # ### #### #### This Station Under ##
182023-11-19T06:06:02.410Z### ### #### # ### Computer Control ##
192023-11-19T06:06:02.469Z#### #### ### ## ##
202023-11-19T06:06:02.532Z##################################################
212023-11-19T06:06:02.603ZHostname: sock
222023-11-19T06:06:02.710ZLoading smf(7) service descriptions:
232023-11-19T06:06:03.655Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-714f127c345e1afe.gz
242023-11-19T06:06:05.228Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-714f127c345e1afe.gz
252023-11-19T06:06:05.257Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-19T06:06:36.384Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-19T06:06:36.423Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2a5f0f6021ab3ba7.gz
282023-11-19T06:06:55.040Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2a5f0f6021ab3ba7.gz
292023-11-19T06:06:55.135Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-19T06:06:56.662Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-19T06:06:56.701Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-19T06:06:59.734Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-19T06:06:59.758Zdownloading input: /input/package/work/version.txt
342023-11-19T06:06:59.783Zdownloaded input: /input/package/work/version.txt
352023-11-19T06:06:59.806Zdownloading input: /input/package/work/package.tar.gz
362023-11-19T06:07:40.698Zdownloaded input: /input/package/work/package.tar.gz
372023-11-19T06:07:40.738Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-19T06:08:11.263Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-19T06:08:11.294Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-19T06:08:29.028Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-19T06:08:29.077Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-19T06:09:18.604Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-19T06:09:18.659Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-19T06:10:06.575Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-19T06:10:06.620Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-19T06:10:34.210Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-19T06:10:35.329Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-19T06:10:44.280Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-19T06:10:44.322Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-19T06:10:54.383Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-19T06:10:54.406Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-19T06:11:02.993Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-19T06:11:03.040Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-19T06:11:09.196Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-19T06:11:09.222Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-19T06:11:09.274Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-19T06:11:09.313Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-19T06:11:12.686Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-19T06:11:12.710Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-19T06:11:18.245Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-19T06:11:18.292Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-19T06:11:32.364Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-19T06:11:32.402Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-19T06:11:46.368Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-19T06:11:46.406Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-19T06:11:46.447Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-19T06:11:46.486Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-19T06:11:52.831Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-19T06:11:52.855Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-19T06:12:12.289Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-19T06:12:12.318Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-19T06:13:34.176Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-19T06:13:34.215Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-19T06:14:24.927Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-19T06:14:24.984Zstarting task 0: "setup"
762023-11-19T06:14:25.038Z++ uname -s
772023-11-19T06:14:25.086Z+ kern=SunOS
782023-11-19T06:14:25.258Z+ case "$kern" in
792023-11-19T06:14:25.346Z+ groupadd -g 12345 build
802023-11-19T06:14:25.415Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-19T06:14:27.042Z+ zfs create -o mountpoint=/work rpool/work
822023-11-19T06:14:27.092Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-19T06:14:27.131Z+ home_fs=zfs
842023-11-19T06:14:27.169Z+ [[ zfs == autofs ]]
852023-11-19T06:14:27.207Z+ mkdir -p /home/build
862023-11-19T06:14:27.246Z+ chown build:build /home/build /work
872023-11-19T06:14:29.066Z+ chmod 0700 /home/build /work
882023-11-19T06:14:29.105Zprocess exited: duration 4045 ms, exit code 0
 
892023-11-19T06:14:29.181Zstarting task 1: "authentication"
902023-11-19T06:14:29.279Zprocess exited: duration 58 ms, exit code 0
 
912023-11-19T06:14:29.356Zstarting task 2: "build"
922023-11-19T06:14:29.425Z+ trap _exit_trap EXIT
932023-11-19T06:14:29.465Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-19T06:14:29.602Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-19T06:14:29.647Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-19T06:14:29.688Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-19T06:14:29.727Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-19T06:14:29.766Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-19T06:14:29.805Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-19T06:14:29.845Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-19T06:14:29.886Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-19T06:14:29.924Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-19T06:14:29.963Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-19T06:14:30.003Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-19T06:14:30.042Z+ [[ -d /opt/oxide ]]
1062023-11-19T06:14:30.082Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-19T06:14:30.121Z+ pfexec mkdir /opt/oxide
1082023-11-19T06:14:30.161Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-19T06:14:30.200Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-19T06:14:30.239Z+ cd /opt/oxide-underneath
1112023-11-19T06:14:30.279Z+ pfexec tar ceEp@/f - .
1122023-11-19T06:14:30.318Z+ cd /opt/oxide
1132023-11-19T06:14:30.358Z+ pfexec tar xveEp@/f -
1142023-11-19T06:14:30.396Zx ., 0 bytes, 0 tape blocks
1152023-11-19T06:14:30.460Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-19T06:14:30.501Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-19T06:14:30.539Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-19T06:14:30.577Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-19T06:14:30.615Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-19T06:14:30.654Z+ pfexec mkdir /opt/oxide/work
1212023-11-19T06:14:30.692Z+ pfexec chown build:build /opt/oxide/work
1222023-11-19T06:14:30.731Z+ cd /opt/oxide/work
1232023-11-19T06:14:30.770Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-19T06:14:31.306ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-19T06:14:31.345Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-11-19T06:14:31.383Zx out/omicron-sled-agent.tar, 111605248 bytes, 217979 tape blocks
1272023-11-19T06:14:31.422Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-19T06:14:31.460Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-11-19T06:14:31.509Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-11-19T06:14:31.549Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-11-19T06:14:31.587Zx target/release/omicron-package, 42514992 bytes, 83038 tape blocks
1322023-11-19T06:14:31.625Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-11-19T06:14:31.712Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-19T06:14:31.751Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-19T06:14:31.789Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-19T06:14:31.828Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-19T06:14:31.868Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-19T06:14:31.907Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-19T06:14:31.945Z
1402023-11-19T06:14:31.984Zreal 1.540142402
1412023-11-19T06:14:32.026Zuser 1.385986626
1422023-11-19T06:14:32.064Zsys 0.270518637
1432023-11-19T06:14:32.103Ztrap 0.000063362
1442023-11-19T06:14:32.141Ztflt 0.000000000
1452023-11-19T06:14:32.180Zdflt 0.000000000
1462023-11-19T06:14:32.219Zkflt 0.000000000
1472023-11-19T06:14:32.257Zlock 0.000000000
1482023-11-19T06:14:32.296Zslp 1.388916280
1492023-11-19T06:14:32.335Zlat 0.032930044
1502023-11-19T06:14:32.374Zstop 0.000116221
1512023-11-19T06:14:32.413Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-19T06:14:32.451Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-19T06:14:32.489Z+ mkdir tests
1542023-11-19T06:14:32.543Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-19T06:14:32.582Z+ ptime -m gunzip
1562023-11-19T06:14:32.620Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-714f127c345e1afe
1572023-11-19T06:14:32.660Z
1582023-11-19T06:14:32.722Zreal 0.139866599
1592023-11-19T06:14:32.761Zuser 0.127494702
1602023-11-19T06:14:32.800Zsys 0.012141101
1612023-11-19T06:14:32.839Ztrap 0.000007359
1622023-11-19T06:14:32.878Ztflt 0.000000000
1632023-11-19T06:14:32.917Zdflt 0.000000000
1642023-11-19T06:14:32.955Zkflt 0.000000000
1652023-11-19T06:14:32.998Zlock 0.000000000
1662023-11-19T06:14:33.037Zslp 0.000022440
1672023-11-19T06:14:33.077Zlat 0.000015370
1682023-11-19T06:14:33.115Zstop 0.000119932
1692023-11-19T06:14:33.577Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-714f127c345e1afe
1702023-11-19T06:14:33.615Z+ chmod a+x tests/bootstrap-714f127c345e1afe
1712023-11-19T06:14:33.653Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-19T06:14:33.692Z+ ptime -m gunzip
1732023-11-19T06:14:33.730Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-19T06:14:34.545Z
1752023-11-19T06:14:34.609Zreal 2.001862527
1762023-11-19T06:14:34.647Zuser 1.858209089
1772023-11-19T06:14:34.686Zsys 0.143256629
1782023-11-19T06:14:34.728Ztrap 0.000162803
1792023-11-19T06:14:34.766Ztflt 0.000000000
1802023-11-19T06:14:34.804Zdflt 0.000000000
1812023-11-19T06:14:34.843Zkflt 0.000000000
1822023-11-19T06:14:34.881Zlock 0.000000000
1832023-11-19T06:14:34.920Zslp 0.000023770
1842023-11-19T06:14:34.959Zlat 0.000024311
1852023-11-19T06:14:34.997Zstop 0.000124132
1862023-11-19T06:14:35.036Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-19T06:14:35.075Z+ chmod a+x tests/bootstrap
1882023-11-19T06:14:35.114Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-19T06:14:35.153Z+ ptime -m gunzip
1902023-11-19T06:14:35.192Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2a5f0f6021ab3ba7
1912023-11-19T06:14:36.823Z
1922023-11-19T06:14:36.862Zreal 2.267286524
1932023-11-19T06:14:36.900Zuser 2.124016511
1942023-11-19T06:14:36.939Zsys 0.142871206
1952023-11-19T06:14:36.977Ztrap 0.000172352
1962023-11-19T06:14:37.016Ztflt 0.000000000
1972023-11-19T06:14:37.055Zdflt 0.000000000
1982023-11-19T06:14:37.094Zkflt 0.000000000
1992023-11-19T06:14:37.132Zlock 0.000000000
2002023-11-19T06:14:37.171Zslp 0.000022560
2012023-11-19T06:14:37.211Zlat 0.000024531
2022023-11-19T06:14:37.249Zstop 0.000121922
2032023-11-19T06:14:37.287Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2a5f0f6021ab3ba7
2042023-11-19T06:14:37.326Z+ chmod a+x tests/end_to_end_tests-2a5f0f6021ab3ba7
2052023-11-19T06:14:37.366Z+ bmat address ls
2062023-11-19T06:14:37.406ZNAME CIDR FIRST LAST COUNT
2072023-11-19T06:14:37.456Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-11-19T06:14:37.495Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-19T06:14:37.534Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-11-19T06:14:37.572Z+ EXTRA_IP_START=10.151.2.164
2112023-11-19T06:14:37.612Z+ EXTRA_IP_END=10.151.2.227
2122023-11-19T06:14:37.651Z+ EXTRA_IP_COUNT=64
2132023-11-19T06:14:37.710Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-19T06:14:37.750Z+ EXTRA_IP_BASE=10.151.2
2152023-11-19T06:14:37.789Z+ EXTRA_IP_FOCTET=164
2162023-11-19T06:14:37.828Z+ EXTRA_IP_LOCTET=227
2172023-11-19T06:14:37.867Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-11-19T06:14:37.906Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-11-19T06:14:37.945Z+ DNS_IP1=10.151.2.164
2202023-11-19T06:14:37.984Z+ DNS_IP2=10.151.2.165
2212023-11-19T06:14:38.023Z+ UPLINK_IP=10.151.2.174
2222023-11-19T06:14:38.062Z+ IPPOOL_START=10.151.2.175
2232023-11-19T06:14:38.101Z+ IPPOOL_END=10.151.2.227
2242023-11-19T06:14:38.139Z++ ipadm show-addr -po type,addr
2252023-11-19T06:14:38.179Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-19T06:14:38.217Z+ GATEWAY_IP=10.151.2.100
2272023-11-19T06:14:38.257Z+ [[ -n 10.151.2.100 ]]
2282023-11-19T06:14:38.320Z+ ping -s 10.151.2.100 56 1
2292023-11-19T06:14:38.361ZPING 10.151.2.100: 56 data bytes
2302023-11-19T06:14:38.405Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2312023-11-19T06:14:38.444Z
2322023-11-19T06:14:38.483Z----10.151.2.100 PING Statistics----
2332023-11-19T06:14:38.523Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-19T06:14:38.562Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-11-19T06:14:38.600Z++ arp -an
2362023-11-19T06:14:38.684Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-11-19T06:14:38.761Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-11-19T06:14:38.802Z+ routeadm -e ipv4-forwarding -u
2392023-11-19T06:14:38.841Z+ PXA_START=10.151.2.164
2402023-11-19T06:14:38.880Z+ PXA_END=10.151.2.227
2412023-11-19T06:14:38.918Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-19T06:14:38.957Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-19T06:14:38.995Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-19T06:14:39.033Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-19T06:14:39.071Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-19T06:14:39.512Z++ cd ./tools
2472023-11-19T06:14:39.551Z++ pwd
2482023-11-19T06:14:39.589Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-19T06:14:39.627Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-19T06:14:39.665Z+ SOFTNPU_MODE=zone
2512023-11-19T06:14:39.704Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-19T06:14:39.742Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-19T06:14:39.801Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-19T06:14:39.839Z++ dladm show-phys -p -o LINK
2552023-11-19T06:14:39.879Z++ head -1
2562023-11-19T06:14:39.917Z+ PHYSICAL_LINK=igb0
2572023-11-19T06:14:39.956Z+ echo 'Using igb0 as physical link'
2582023-11-19T06:14:39.995ZUsing igb0 as physical link
2592023-11-19T06:14:40.034Z+ ensure_run_as_root
2602023-11-19T06:14:40.073Z++ id -u
2612023-11-19T06:14:40.112Z+ [[ 0 -ne 0 ]]
2622023-11-19T06:14:40.150Z+ ensure_zpools
2632023-11-19T06:14:40.190Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-19T06:14:40.229Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-11-19T06:14:40.268Z+ readarray -t ZPOOLS
2662023-11-19T06:14:40.307Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-11-19T06:14:40.350Z++ sed 's/[ ",]//g'
2682023-11-19T06:14:40.396Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-11-19T06:14:40.439Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-11-19T06:14:40.477ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-11-19T06:14:40.515Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-11-19T06:14:40.554ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2732023-11-19T06:14:40.592Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2742023-11-19T06:14:40.630ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2752023-11-19T06:14:40.668Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2762023-11-19T06:14:40.706ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2772023-11-19T06:14:40.745Z0+0 records in
2782023-11-19T06:14:40.783ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2792023-11-19T06:14:40.847Z0+0 records out
2802023-11-19T06:14:40.890ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2812023-11-19T06:14:40.932Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2822023-11-19T06:14:40.970ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2832023-11-19T06:14:41.009Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2842023-11-19T06:14:41.048ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2852023-11-19T06:14:41.086Z+ set +x
2862023-11-19T06:14:41.124ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2872023-11-19T06:14:41.163Z++ zpool list -o name
2882023-11-19T06:14:41.201ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2892023-11-19T06:14:41.241Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2902023-11-19T06:14:41.286ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2912023-11-19T06:14:41.334Z+ [[ -z '' ]]
2922023-11-19T06:14:41.377ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2932023-11-19T06:14:41.419Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2942023-11-19T06:14:41.466ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2952023-11-19T06:14:41.505Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2962023-11-19T06:14:41.543ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2972023-11-19T06:14:41.582Z+ set +x
2982023-11-19T06:14:41.622ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2992023-11-19T06:14:41.661Z+ for ZPOOL in "${ZPOOLS[@]}"
3002023-11-19T06:14:41.700ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3012023-11-19T06:14:41.741Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3022023-11-19T06:14:41.786ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3032023-11-19T06:14:41.825Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3042023-11-19T06:14:41.880ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3052023-11-19T06:14:41.918Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3062023-11-19T06:14:41.956ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3072023-11-19T06:14:41.995Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3082023-11-19T06:14:42.034ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3092023-11-19T06:14:42.072Z0+0 records in
3102023-11-19T06:14:42.111ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3112023-11-19T06:14:42.567Z0+0 records out
3122023-11-19T06:14:43.421ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3132023-11-19T06:14:43.461Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3142023-11-19T06:14:43.499ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3152023-11-19T06:14:43.537Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-11-19T06:14:43.689ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3172023-11-19T06:14:43.728Z+ set +x
3182023-11-19T06:14:43.767ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3192023-11-19T06:14:43.806Z++ zpool list -o name
3202023-11-19T06:14:44.252ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3212023-11-19T06:14:44.291Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3222023-11-19T06:14:44.334ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3232023-11-19T06:14:44.380Z+ [[ -z '' ]]
3242023-11-19T06:14:44.419ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3252023-11-19T06:14:44.457Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3262023-11-19T06:14:44.496ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3272023-11-19T06:14:44.534Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3282023-11-19T06:14:44.573ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3292023-11-19T06:14:44.612Z+ set +x
3302023-11-19T06:14:44.651ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3312023-11-19T06:14:44.689Z+ for ZPOOL in "${ZPOOLS[@]}"
3322023-11-19T06:14:44.729ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3332023-11-19T06:14:44.769Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3342023-11-19T06:14:44.809ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3352023-11-19T06:14:44.848Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3362023-11-19T06:14:44.886ZSimnet net0/sc0_0 exists
3372023-11-19T06:14:44.925Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3382023-11-19T06:14:44.997ZSimnet net1/sc1_0 exists
3392023-11-19T06:14:45.036Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3402023-11-19T06:14:45.075ZVnic sc0_1 exists
3412023-11-19T06:14:45.113Z0+0 records in
3422023-11-19T06:14:45.562Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
3432023-11-19T06:14:45.604Z0+0 records out
3442023-11-19T06:14:45.653Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3452023-11-19T06:14:45.693Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3462023-11-19T06:14:45.731Z+ set +x
3472023-11-19T06:14:45.769Z++ zpool list -o name
3482023-11-19T06:14:45.807Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3492023-11-19T06:14:45.845Z+ [[ -z '' ]]
3502023-11-19T06:14:45.883Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3512023-11-19T06:14:45.923Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3522023-11-19T06:14:45.961Z+ set +x
3532023-11-19T06:14:46.030Z+ for ZPOOL in "${ZPOOLS[@]}"
3542023-11-19T06:14:46.069Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552023-11-19T06:14:46.108Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562023-11-19T06:14:46.188Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572023-11-19T06:14:46.233Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582023-11-19T06:14:46.272Z0+0 records in
3592023-11-19T06:14:46.312Z0+0 records out
3602023-11-19T06:14:46.362Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3612023-11-19T06:14:46.401Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622023-11-19T06:14:46.439Z+ set +x
3632023-11-19T06:14:46.480Z++ zpool list -o name
3642023-11-19T06:14:46.519Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3652023-11-19T06:14:46.558Z+ [[ -z '' ]]
3662023-11-19T06:14:46.596Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3672023-11-19T06:14:46.635Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3682023-11-19T06:14:46.673Z+ set +x
3692023-11-19T06:14:46.712Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-11-19T06:14:46.752Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3712023-11-19T06:14:46.805Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3722023-11-19T06:14:46.843Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3732023-11-19T06:14:46.882Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3742023-11-19T06:14:46.921Z0+0 records in
3752023-11-19T06:14:46.961Z0+0 records out
3762023-11-19T06:14:47.004Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3772023-11-19T06:14:47.084Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3782023-11-19T06:14:47.122Z+ set +x
3792023-11-19T06:14:47.160Z++ zpool list -o name
3802023-11-19T06:14:47.200Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3812023-11-19T06:14:47.238Z+ [[ -z '' ]]
3822023-11-19T06:14:47.276Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3832023-11-19T06:14:47.314Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3842023-11-19T06:14:47.355Z+ set +x
3852023-11-19T06:14:47.393Z+ for ZPOOL in "${ZPOOLS[@]}"
3862023-11-19T06:14:47.435Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-11-19T06:14:47.479Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-11-19T06:14:47.518Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-11-19T06:14:47.556Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-11-19T06:14:47.594Z0+0 records in
3912023-11-19T06:14:47.632Z0+0 records out
3922023-11-19T06:14:47.671Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3932023-11-19T06:14:47.710Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942023-11-19T06:14:47.748Z+ set +x
3952023-11-19T06:14:47.786Z++ zpool list -o name
3962023-11-19T06:14:47.824Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972023-11-19T06:14:47.867Z+ [[ -z '' ]]
3982023-11-19T06:14:47.921Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992023-11-19T06:14:47.960Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002023-11-19T06:14:48.000Z+ set +x
4012023-11-19T06:14:48.039Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-19T06:14:48.077Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4032023-11-19T06:14:48.154Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-11-19T06:14:48.193Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4052023-11-19T06:14:48.231Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4062023-11-19T06:14:48.269Z0+0 records in
4072023-11-19T06:14:48.312Z0+0 records out
4082023-11-19T06:14:48.356Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4092023-11-19T06:14:48.443Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4102023-11-19T06:14:48.482Z+ set +x
4112023-11-19T06:14:48.521Z++ zpool list -o name
4122023-11-19T06:14:48.601Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4132023-11-19T06:14:48.640Z+ [[ -z '' ]]
4142023-11-19T06:14:48.679Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4152023-11-19T06:14:48.718Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4162023-11-19T06:14:48.757Z+ set +x
4172023-11-19T06:14:48.797Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-11-19T06:14:48.835Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4192023-11-19T06:14:48.874Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4202023-11-19T06:14:48.912Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4212023-11-19T06:14:48.951Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4222023-11-19T06:14:48.991Z0+0 records in
4232023-11-19T06:14:49.029Z0+0 records out
4242023-11-19T06:14:49.069Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4252023-11-19T06:14:49.116Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4262023-11-19T06:14:49.209Z+ set +x
4272023-11-19T06:14:49.249Z++ zpool list -o name
4282023-11-19T06:14:49.287Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4292023-11-19T06:14:49.325Z+ [[ -z '' ]]
4302023-11-19T06:14:49.364Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4312023-11-19T06:14:49.402Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4322023-11-19T06:14:49.440Z+ set +x
4332023-11-19T06:14:49.478Z+ for ZPOOL in "${ZPOOLS[@]}"
4342023-11-19T06:14:49.516Zdownloading softnpu ...done
4352023-11-19T06:14:49.558Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4362023-11-19T06:14:49.604Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
4372023-11-19T06:14:49.643Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4382023-11-19T06:14:49.682Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4392023-11-19T06:14:49.720Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4402023-11-19T06:14:49.758Z0+0 records in
4412023-11-19T06:14:49.796Z0+0 records out
4422023-11-19T06:14:49.834Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4432023-11-19T06:14:49.872Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4442023-11-19T06:14:49.910Z+ set +x
4452023-11-19T06:14:49.948Z++ zpool list -o name
4462023-11-19T06:14:49.989Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4472023-11-19T06:14:50.031Z+ [[ -z '' ]]
4482023-11-19T06:14:50.069Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4492023-11-19T06:14:50.108Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4502023-11-19T06:14:50.147Z+ set +x
4512023-11-19T06:14:50.185Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4522023-11-19T06:14:50.244Z+ readarray -t ZPOOLS
4532023-11-19T06:14:50.283Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4542023-11-19T06:14:50.322Z++ sed 's/[ ",]//g'
4552023-11-19T06:14:50.360Z+ for ZPOOL in "${ZPOOLS[@]}"
4562023-11-19T06:14:50.404Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4572023-11-19T06:14:50.451Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4582023-11-19T06:14:50.492Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4592023-11-19T06:14:50.531Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4602023-11-19T06:14:50.571Z0+0 records in
4612023-11-19T06:14:50.610Z0+0 records out
4622023-11-19T06:14:50.649Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4632023-11-19T06:14:50.688Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4642023-11-19T06:14:50.727Z+ set +x
4652023-11-19T06:14:50.766Z++ zpool list -o name
4662023-11-19T06:14:50.804Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4672023-11-19T06:14:50.843Z+ [[ -z '' ]]
4682023-11-19T06:14:50.890Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4692023-11-19T06:14:50.929Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4702023-11-19T06:14:50.971Z+ set +x
4712023-11-19T06:14:51.018Z+ for ZPOOL in "${ZPOOLS[@]}"
4722023-11-19T06:14:51.066Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4732023-11-19T06:14:51.106Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4742023-11-19T06:14:51.147Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4752023-11-19T06:14:51.188Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4762023-11-19T06:14:51.226Z0+0 records in
4772023-11-19T06:14:51.283Z0+0 records out
4782023-11-19T06:14:51.336Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4792023-11-19T06:14:51.376Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4802023-11-19T06:14:51.415Z+ set +x
4812023-11-19T06:14:51.453Z++ zpool list -o name
4822023-11-19T06:14:51.492Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4832023-11-19T06:14:51.531Z+ [[ -z '' ]]
4842023-11-19T06:14:51.570Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4852023-11-19T06:14:51.609Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4862023-11-19T06:14:51.648Z+ set +x
4872023-11-19T06:14:51.687Z+ [[ zone == \z\o\n\e ]]
4882023-11-19T06:14:51.725Z+ ensure_simulated_links igb0
4892023-11-19T06:14:51.764Z+ local PHYSICAL_LINK=igb0
4902023-11-19T06:14:51.804Z+ INDICES=("0" "1")
4912023-11-19T06:14:51.842Z+ for I in "${INDICES[@]}"
4922023-11-19T06:14:51.881Z++ get_simnet_name_if_exists net0
4932023-11-19T06:14:51.920Z++ dladm show-simnet -p -o LINK net0
4942023-11-19T06:14:51.959Z++ echo ''
4952023-11-19T06:14:51.998Z+ [[ -z '' ]]
4962023-11-19T06:14:52.037Z+ dladm create-simnet -t net0
4972023-11-19T06:14:52.076Z+ dladm create-simnet -t sc0_0
4982023-11-19T06:14:52.115Z+ dladm modify-simnet -t -p net0 sc0_0
4992023-11-19T06:14:52.155Z+ dladm set-linkprop -p mtu=9000 sc0_0
5002023-11-19T06:14:52.193Z+ success 'Simnet net0/sc0_0 exists'
5012023-11-19T06:14:52.232Z+ set +x
5022023-11-19T06:14:52.271Z+ for I in "${INDICES[@]}"
5032023-11-19T06:14:52.318Z++ get_simnet_name_if_exists net1
5042023-11-19T06:14:52.356Z++ dladm show-simnet -p -o LINK net1
5052023-11-19T06:14:52.395Z++ echo ''
5062023-11-19T06:14:52.434Z+ [[ -z '' ]]
5072023-11-19T06:14:52.473Z+ dladm create-simnet -t net1
5082023-11-19T06:14:52.512Z+ dladm create-simnet -t sc1_0
5092023-11-19T06:14:52.551Z+ dladm modify-simnet -t -p net1 sc1_0
5102023-11-19T06:14:52.590Z+ dladm set-linkprop -p mtu=9000 sc1_0
5112023-11-19T06:14:52.629Z+ success 'Simnet net1/sc1_0 exists'
5122023-11-19T06:14:52.668Z+ set +x
5132023-11-19T06:14:52.707Z++ get_vnic_name_if_exists sc0_1
5142023-11-19T06:14:52.746Z++ dladm show-vnic -p -o LINK sc0_1
5152023-11-19T06:14:52.785Z++ echo ''
5162023-11-19T06:14:52.824Z+ [[ -z '' ]]
5172023-11-19T06:14:52.863Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5182023-11-19T06:14:52.901Z+ [[ -v PROMISC_FILT_OFF ]]
5192023-11-19T06:14:52.943Z+ success 'Vnic sc0_1 exists'
5202023-11-19T06:14:52.982Z+ set +x
5212023-11-19T06:14:53.021Z+ warn_if_no_proxy_arp
5222023-11-19T06:14:53.059Z+ [[ -v PXA_START ]]
5232023-11-19T06:14:53.097Z+ [[ -v PXA_END ]]
5242023-11-19T06:14:53.189Z+ ensure_softnpu_zone
5252023-11-19T06:14:53.228Z+ zoneadm list
5262023-11-19T06:14:53.267Z+ grep -q sidecar_softnpu
5272023-11-19T06:14:53.306Z+ [[ -f out/npuzone/npuzone ]]
5282023-11-19T06:14:53.356Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5292023-11-19T06:14:53.394Zdownloading asic_program.so ...done
5302023-11-19T06:14:53.433Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-19T06:14:59.483Zdownloading scadm ...done
5322023-11-19T06:14:59.522Zcreating port tfportrear0_0 for sc0_0
5332023-11-19T06:14:59.561Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-19T06:14:59.601Zsoftnpu zone setup
5352023-11-19T06:14:59.639Zconfigure zone
5362023-11-19T06:14:59.678Zinstall zone
5372023-11-19T06:15:00.570Zboot zone
5382023-11-19T06:15:03.678Z[sidecar_softnpu] svcs milestone/network
5392023-11-19T06:15:03.717ZSTATE STIME FMRI
5402023-11-19T06:15:03.756Zonline 6:15:01 svc:/milestone/network:default
5412023-11-19T06:15:03.796Z[sidecar_softnpu] /softnpu/npu start
5422023-11-19T06:15:03.835Z
5432023-11-19T06:15:03.876Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-19T06:15:03.915Z+ GATEWAY_IP=10.151.2.100
5452023-11-19T06:15:03.955Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-11-19T06:15:03.995ZUsing 10.151.2.100 as gateway ip
5472023-11-19T06:15:04.035Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-19T06:15:04.074Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-11-19T06:15:04.113Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-19T06:15:04.153Z+++ echo 18:c0:4d:0d:a0:2a
5512023-11-19T06:15:04.192Z+ COUNT=1
5522023-11-19T06:15:04.231Z+ [[ 1 -ne 1 ]]
5532023-11-19T06:15:04.270Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-11-19T06:15:04.310ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-11-19T06:15:04.350Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-11-19T06:15:04.389Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-11-19T06:15:04.459Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-19T06:15:04.499Z+ [[ -v PXA_START ]]
5592023-11-19T06:15:04.538Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-11-19T06:15:04.577Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-11-19T06:15:04.616Z+ z_scadm dump-state
5622023-11-19T06:15:04.655Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-19T06:15:04.694Zlocal v6:
5642023-11-19T06:15:04.733Zlocal v4:
5652023-11-19T06:15:04.810Zrouter v6:
5662023-11-19T06:15:04.849Zrouter v4_idx:
5672023-11-19T06:15:04.888Zrouter v4_routes:
5682023-11-19T06:15:04.959Zresolver v4:
5692023-11-19T06:15:05.000Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-11-19T06:15:05.039Zresolver v6:
5712023-11-19T06:15:05.078Znat v4:
5722023-11-19T06:15:05.118Znat_v6:
5732023-11-19T06:15:05.157Zport_mac:
5742023-11-19T06:15:05.197Zproxy arp:
5752023-11-19T06:15:05.236Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-11-19T06:15:05.275Z+ success 'softnpu zone exists'
5772023-11-19T06:15:05.315Z+ set +x
5782023-11-19T06:15:05.355Zsoftnpu zone exists
5792023-11-19T06:15:05.395Z
5802023-11-19T06:15:05.435Zreal 26.541131422
5812023-11-19T06:15:05.497Zuser 0.709338203
5822023-11-19T06:15:05.543Zsys 1.605399434
5832023-11-19T06:15:05.582Ztrap 0.000067175
5842023-11-19T06:15:05.621Ztflt 0.000133178
5852023-11-19T06:15:05.660Zdflt 0.001062862
5862023-11-19T06:15:05.699Zkflt 0.000007080
5872023-11-19T06:15:05.738Zlock 20:32.443855160
5882023-11-19T06:15:05.777Zslp 1:19.565403323
5892023-11-19T06:15:05.816Zlat 0.081457875
5902023-11-19T06:15:05.881Zstop 0.336459546
5912023-11-19T06:15:05.921Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-19T06:15:05.960Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-19T06:15:06.000Z+ SILO_NAME=recovery
5942023-11-19T06:15:06.041Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-19T06:15:06.080Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-19T06:15:06.119Z+ sed -i~ '
5972023-11-19T06:15:06.159Z /^external_dns_ips/c\
5982023-11-19T06:15:06.200Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-11-19T06:15:06.239Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-19T06:15:06.278Z /^first/c\
6012023-11-19T06:15:06.318Zfirst = "10.151.2.164"
6022023-11-19T06:15:06.357Z /^last/c\
6032023-11-19T06:15:06.396Zlast = "10.151.2.173"
6042023-11-19T06:15:06.435Z }
6052023-11-19T06:15:06.475Z /^\[rack_network_config/,/^$/ {
6062023-11-19T06:15:06.541Z /^infra_ip_first/c\
6072023-11-19T06:15:06.588Zinfra_ip_first = "10.151.2.174"
6082023-11-19T06:15:06.628Z /^infra_ip_last/c\
6092023-11-19T06:15:06.668Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6102023-11-19T06:15:06.707Zinfra_ip_last = "10.151.2.174"
6112023-11-19T06:15:06.747Z+++ pkg/config-rss.toml Sun Nov 19 06:15:03 2023
6122023-11-19T06:15:06.792Z }
6132023-11-19T06:15:06.832Z@@ -41,7 +41,7 @@
6142023-11-19T06:15:06.874Z /^\[\[rack_network_config.ports/,/^$/ {
6152023-11-19T06:15:06.913Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6162023-11-19T06:15:06.952Z /^routes/c\
6172023-11-19T06:15:06.992Z # must be contained in one of the "internal services" IP Pool ranges listed
6182023-11-19T06:15:07.031Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6192023-11-19T06:15:07.069Z # below.
6202023-11-19T06:15:07.108Z /^addresses/c\
6212023-11-19T06:15:07.147Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6222023-11-19T06:15:07.186Zaddresses = \["10.151.2.174/32"\]
6232023-11-19T06:15:07.225Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6242023-11-19T06:15:07.264Z }
6252023-11-19T06:15:07.304Z
6262023-11-19T06:15:07.343Z' pkg/config-rss.toml
6272023-11-19T06:15:07.382Z # Initial TLS certificates for the external API
6282023-11-19T06:15:07.423Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6292023-11-19T06:15:07.462Z #
6302023-11-19T06:15:07.500Z+ true
6312023-11-19T06:15:07.540Z@@ -77,8 +77,8 @@
6322023-11-19T06:15:07.683Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6332023-11-19T06:15:07.723Z #
6342023-11-19T06:15:07.761Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6352023-11-19T06:15:07.803Z # For more on this and what to put here, see docs/how-to-run.adoc.
6362023-11-19T06:15:07.846Z+ TLS_NAME=recovery.sys.oxide.test
6372023-11-19T06:15:07.885Z [[internal_services_ip_pool_ranges]]
6382023-11-19T06:15:07.925Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6392023-11-19T06:15:07.964Z-first = "192.168.1.20"
6402023-11-19T06:15:08.004Z....+.+...+.....+.........+...............+......+....+...+........+......+....+..+.........+....+..+....+........+...+.........+...+.+............+.....+....+...+.....+.......+.....+......+.........+.+.....+....+.........+..+...+.......+...+.....+.+...+...+..+.............+..+..........+..+...+............+....+.....+....+..+...+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+....+...+..+...+....+...........+...+....+...+..+.......+...+..+....+.....+.+...+........+...+....+..+...+.+......+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.....+...+.............+.....+....+...+.....+.............+......+.....+......+...+......+.........+.+...+........+...+..........+...........+....+......+...+.....+...+............+.............+........+.+..............+.+.....+.+....................+....+...+..+.........+....+...............+...........+......+...+...+....+...+..+...+.....................+....+......+.........+...........+............+...+.+...+...........+.......+...+..+.......+...+.....+..........+..................+......+..........................+...+.+.....+...+.............+..+..................+..........+...+..............+.......+.....+..........+........+...+.............+............+.........+...........+.......+......+..+.............+..+.........+....+..+...+.+......+.....+....+..............+............+...+...............+.......+............+...+...+..............+...............+...............+...............+.+..+...+.+........+.......+...+..+.........................+...........+...+.+...+...........+......+........................+....+...........+.+......+.....+.......+...+.....+....+........+......+.......+.........+........+......+.+........+.....................+..........+..+.......+........+.........+...+.+...............+..+....+......+...+......+..........................+...+.......+...+............+............+........+..........+...+..+..........+...........................+..+..........+.....+.+...+............+............+.....+..........+.....+.......+.....+...............+.+..+...+....+..............+......+...................+..+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6412023-11-19T06:15:08.043Z-last = "192.168.1.29"
6422023-11-19T06:15:08.081Z+first = "10.151.2.164"
6432023-11-19T06:15:08.120Z+last = "10.151.2.173"
6442023-11-19T06:15:08.161Z
6452023-11-19T06:15:08.200Z # TODO - this configuration is subject to change going forward. Ultimately these
6462023-11-19T06:15:08.239Z # parameters should be provided to the control plane via wicket, but we need to
6472023-11-19T06:15:08.278Z@@ -92,8 +92,8 @@
6482023-11-19T06:15:08.317Z # A range of IP addresses used by Boundary Services on the external network. In
6492023-11-19T06:15:08.356Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6502023-11-19T06:15:08.396Z # With softnpu, only one address is used.
6512023-11-19T06:15:08.436Z-infra_ip_first = "192.168.1.30"
6522023-11-19T06:15:08.475Z-infra_ip_last = "192.168.1.30"
6532023-11-19T06:15:08.515Z+infra_ip_first = "10.151.2.174"
6542023-11-19T06:15:08.594Z+infra_ip_last = "10.151.2.174"
6552023-11-19T06:15:08.647Z
6562023-11-19T06:15:08.693Z # Configurations for BGP routers to run on the scrimlets.
6572023-11-19T06:15:08.732Z bgp = []
6582023-11-19T06:15:08.782Z@@ -101,9 +101,9 @@
6592023-11-19T06:15:08.821Z # You can configure multiple uplinks by repeating the following stanza
6602023-11-19T06:15:08.861Z [[rack_network_config.ports]]
6612023-11-19T06:15:08.901Z # Routes associated with this port.
6622023-11-19T06:15:08.940Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6632023-11-19T06:15:08.979Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6642023-11-19T06:15:09.019Z...+........+....+.....+.+......+.........+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.....+.+......+..+...+.......+.....+............+......+...+.+......+........+............+...+....+......+......+...+..+....+..+....+...+........+.......+..+.+..............+.+...+......+...+..+.+..+...+..........+...+..............+...+....+.....+...+.+..+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+.......+..............+.+..............+.......+.....+......................+...+...+............+.........+......+...+..+...................+.....+......+...+........................+......................+........+.............+.........+..............+.......+.....+...+.........+.+......+........+.+..+.............+..+....+......+..............+.........+.+...+..+......+.......+..+.+............+...+...+..+.......+......+..+....+...............+......+..............+.+.........+...+...+..+...+...+.+..................+..+...+............+...+......+...............+.+.....+........................+....+..+.............+...........+............+.......+..+...+......+.+......+..+............+....+...............+....................+.......+...+............+..+...+...................+......+............+...........+.+.....+.+..+...+.......+.....+................+...............+..............+...+.......+......+..............+.............+..+.......+...........+....+...+..+.+.....+...+......+...............+....+.....................+.....+.+..+.+............+.................+......+......+.............+........+..................+.........+......+.........+............+......+.......+........+...+....+...........+......+......+............+...+.......+...+...+..+............+................+..+..........+.........+....................+.............+...........+.......+.....+...+....+..+.+...+..............+...+...+.......+..+...+....+......+.....+.......+..+..........+...+............+...+.....+......................+.....+...............+..........+.....................+..+.+......+.........+............+..+..........+..+.+.........+.....+.........+...............+....+..+...................+.........+...........+....+......+..+.........+................+.........+........+......+.+......+...............+..+.......+...............+........+...+..........+..+.............+...+..+.......+........+...+.+...........+...+......+...+.+.........+......+.....+.......+...+...........+.+..+..................+...+....+........+........................+......+.......+......+..+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6652023-11-19T06:15:09.059Z # Addresses associated with this port.
6662023-11-19T06:15:09.098Z-----
6672023-11-19T06:15:09.137Z-addresses = ["192.168.1.30/32"]
6682023-11-19T06:15:09.178Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6692023-11-19T06:15:09.216Z+addresses = ["10.151.2.174/32"]
6702023-11-19T06:15:09.271Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6712023-11-19T06:15:09.312Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6722023-11-19T06:15:09.351Z+ rmdir pkg
6732023-11-19T06:15:09.390Z port = "qsfp0"
6742023-11-19T06:15:09.429Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6752023-11-19T06:15:09.468Z # The speed of this port.
6762023-11-19T06:15:09.507Z+ pfexec mkdir -p /usr/oxide
6772023-11-19T06:15:09.548Za pkg/config-rss.toml 7K
6782023-11-19T06:15:09.587Z+ pfexec rm -f /usr/oxide/tcpproxy
6792023-11-19T06:15:09.626Za pkg/initial-tls-cert.pem 2K
6802023-11-19T06:15:09.690Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6812023-11-19T06:15:09.736Za pkg/initial-tls-key.pem 4K
6822023-11-19T06:15:09.775Z+ pfexec chmod +x /usr/oxide/tcpproxy
6832023-11-19T06:15:09.814Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6842023-11-19T06:15:09.854Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6852023-11-19T06:15:09.894Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6862023-11-19T06:15:09.933Z+ OMICRON_NO_UNINSTALL=1
6872023-11-19T06:15:09.972Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6882023-11-19T06:15:10.012ZNov 19 06:15:06.089 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6892023-11-19T06:15:10.051ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6902023-11-19T06:15:10.090ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6912023-11-19T06:15:10.129ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6922023-11-19T06:15:10.169ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6932023-11-19T06:15:10.209ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6942023-11-19T06:15:10.249ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6952023-11-19T06:15:10.290ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6962023-11-19T06:15:10.329ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6972023-11-19T06:15:10.369ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6982023-11-19T06:15:10.408ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6992023-11-19T06:15:10.447ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7002023-11-19T06:15:10.487ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7012023-11-19T06:15:10.526ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7022023-11-19T06:15:10.564ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7032023-11-19T06:15:10.603ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7042023-11-19T06:15:10.642ZNov 19 06:15:06.091 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7052023-11-19T06:15:10.680ZNov 19 06:15:06.316 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-11-19T06:15:10.786ZNov 19 06:15:06.350 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-19T06:15:10.825ZNov 19 06:15:06.410 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-19T06:15:10.865Z
7092023-11-19T06:15:10.903Zreal 0.460996268
7102023-11-19T06:15:10.948Zuser 0.097522980
7112023-11-19T06:15:10.987Zsys 1.835037440
7122023-11-19T06:15:11.027Ztrap 0.000002329
7132023-11-19T06:15:11.066Ztflt 0.000124377
7142023-11-19T06:15:11.104Zdflt 0.000804454
7152023-11-19T06:15:11.149Zkflt 0.000000000
7162023-11-19T06:15:11.189Zlock 41.303012523
7172023-11-19T06:15:11.238Zslp 0.666877249
7182023-11-19T06:15:11.278Zlat 0.003067784
7192023-11-19T06:15:11.317Zstop 0.206001799
7202023-11-19T06:15:11.356Z+ retry=0
7212023-11-19T06:15:11.395Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-19T06:15:11.434Z+ [[ 0 -gt 30 ]]
7232023-11-19T06:15:11.473Z+ sleep 1
7242023-11-19T06:15:11.512Z+ retry=1
7252023-11-19T06:15:11.550Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-19T06:15:11.589Z+ [[ 1 -gt 30 ]]
7272023-11-19T06:15:11.627Z+ sleep 1
7282023-11-19T06:15:11.666Z+ retry=2
7292023-11-19T06:15:11.704Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-19T06:15:11.743Z+ [[ 2 -gt 30 ]]
7312023-11-19T06:15:11.851Z+ sleep 1
7322023-11-19T06:15:11.890Z+ retry=3
7332023-11-19T06:15:11.929Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-11-19T06:15:11.968Z+ [[ 3 -gt 30 ]]
7352023-11-19T06:15:12.007Z+ sleep 1
7362023-11-19T06:15:12.046Z+ retry=4
7372023-11-19T06:15:12.085Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-11-19T06:15:31.019Z+ [[ 4 -gt 30 ]]
7392023-11-19T06:15:31.066Z+ sleep 1
7402023-11-19T06:15:32.026Z+ retry=5
7412023-11-19T06:15:32.070Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7422023-11-19T06:15:32.114Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7432023-11-19T06:15:32.164Zlocal v6:
7442023-11-19T06:15:32.215Zfd00:99::1
7452023-11-19T06:15:32.264Zfe80::bc94:8aff:fe53:9304
7462023-11-19T06:15:32.310Zlocal v4:
7472023-11-19T06:15:32.353Z10.151.2.174
7482023-11-19T06:15:32.396Zrouter v6:
7492023-11-19T06:15:32.444Zrouter v4_idx:
7502023-11-19T06:15:32.493Z0.0.0.0/0 -> 0
7512023-11-19T06:15:32.539Zrouter v4_routes:
7522023-11-19T06:15:32.581Z0 -> 10.151.2.100 (2)
7532023-11-19T06:15:32.625Zresolver v4:
7542023-11-19T06:15:32.665Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7552023-11-19T06:15:32.715Z10.151.2.174 -> 2a:84:56:a8:e7:48
7562023-11-19T06:15:32.764Zresolver v6:
7572023-11-19T06:15:32.812Zfe80::a4dc:ebff:fe9e:3164 -> a6:dc:eb:9e:31:64
7582023-11-19T06:15:32.889Zfe80::bc94:8aff:fe53:9304 -> be:94:8a:53:93:04
7592023-11-19T06:15:32.931Znat v4:
7602023-11-19T06:15:32.975Znat_v6:
7612023-11-19T06:15:33.022Zport_mac:
7622023-11-19T06:15:33.074Z1: a8:40:25:03:5e:61
7632023-11-19T06:15:33.122Z1000: a8:40:25:03:5e:60
7642023-11-19T06:15:33.172Zproxy arp:
7652023-11-19T06:15:33.219Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7662023-11-19T06:15:33.344Z+ retry=0
7672023-11-19T06:15:33.390Z+++ zoneadm list -n
7682023-11-19T06:15:33.433Z+++ grep oxz_ntp
7692023-11-19T06:15:33.472Z++ pfexec svcs -z -Hostate oxide/ntp
7702023-11-19T06:15:33.510Zsvcs: zone '-Hostate': entity not found
7712023-11-19T06:15:33.549Z++ true
7722023-11-19T06:15:33.626Z+ [[ '' != online ]]
7732023-11-19T06:15:33.665Z+ [[ 0 -gt 60 ]]
7742023-11-19T06:15:33.703Z+ sleep 1
7752023-11-19T06:15:33.741Z+ retry=1
7762023-11-19T06:15:33.779Z+++ zoneadm list -n
7772023-11-19T06:15:34.278Z+++ grep oxz_ntp
7782023-11-19T06:15:34.322Z++ pfexec svcs -z -Hostate oxide/ntp
7792023-11-19T06:15:34.366Zsvcs: zone '-Hostate': entity not found
7802023-11-19T06:15:34.409Z++ true
7812023-11-19T06:15:34.455Z+ [[ '' != online ]]
7822023-11-19T06:15:34.503Z+ [[ 1 -gt 60 ]]
7832023-11-19T06:15:34.552Z+ sleep 1
7842023-11-19T06:15:34.600Z+ retry=2
7852023-11-19T06:15:34.645Z+++ zoneadm list -n
7862023-11-19T06:15:34.690Z+++ grep oxz_ntp
7872023-11-19T06:15:34.733Z++ pfexec svcs -z -Hostate oxide/ntp
7882023-11-19T06:15:34.779Zsvcs: zone '-Hostate': entity not found
7892023-11-19T06:15:34.830Z++ true
7902023-11-19T06:15:34.873Z+ [[ '' != online ]]
7912023-11-19T06:15:34.911Z+ [[ 2 -gt 60 ]]
7922023-11-19T06:15:34.949Z+ sleep 1
7932023-11-19T06:15:35.126Z+ retry=3
7942023-11-19T06:15:35.176Z+++ zoneadm list -n
7952023-11-19T06:15:35.222Z+++ grep oxz_ntp
7962023-11-19T06:15:35.273Z++ pfexec svcs -z oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5 -Hostate oxide/ntp
7972023-11-19T06:15:35.315Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7982023-11-19T06:15:35.362Z++ true
7992023-11-19T06:15:35.409Z+ [[ '' != online ]]
8002023-11-19T06:15:35.453Z+ [[ 3 -gt 60 ]]
8012023-11-19T06:15:35.494Z+ sleep 1
8022023-11-19T06:15:36.153Z+ retry=4
8032023-11-19T06:15:36.192Z+++ zoneadm list -n
8042023-11-19T06:15:36.231Z+++ grep oxz_ntp
8052023-11-19T06:15:36.269Z++ pfexec svcs -z oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5 -Hostate oxide/ntp
8062023-11-19T06:15:36.307Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8072023-11-19T06:15:36.345Z++ true
8082023-11-19T06:15:36.383Z+ [[ '' != online ]]
8092023-11-19T06:15:36.421Z+ [[ 4 -gt 60 ]]
8102023-11-19T06:15:36.460Z+ sleep 1
8112023-11-19T06:15:37.183Z+ retry=5
8122023-11-19T06:15:37.221Z+++ zoneadm list -n
8132023-11-19T06:15:37.259Z+++ grep oxz_ntp
8142023-11-19T06:15:37.297Z++ pfexec svcs -z oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5 -Hostate oxide/ntp
8152023-11-19T06:15:37.335Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8162023-11-19T06:15:37.373Z++ true
8172023-11-19T06:15:37.419Z+ [[ '' != online ]]
8182023-11-19T06:15:37.468Z+ [[ 5 -gt 60 ]]
8192023-11-19T06:15:37.507Z+ sleep 1
8202023-11-19T06:15:38.211Z+ retry=6
8212023-11-19T06:15:38.251Z+++ zoneadm list -n
8222023-11-19T06:15:38.298Z+++ grep oxz_ntp
8232023-11-19T06:15:38.347Z++ pfexec svcs -z oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5 -Hostate oxide/ntp
8242023-11-19T06:15:38.385Z+ [[ disabled != online ]]
8252023-11-19T06:15:38.423Z+ [[ 6 -gt 60 ]]
8262023-11-19T06:15:38.461Z+ sleep 1
8272023-11-19T06:15:39.234Z+ retry=7
8282023-11-19T06:15:39.272Z+++ zoneadm list -n
8292023-11-19T06:15:39.311Z+++ grep oxz_ntp
8302023-11-19T06:15:39.349Z++ pfexec svcs -z oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5 -Hostate oxide/ntp
8312023-11-19T06:15:39.387Z+ [[ disabled != online ]]
8322023-11-19T06:15:39.425Z+ [[ 7 -gt 60 ]]
8332023-11-19T06:15:39.463Z+ sleep 1
8342023-11-19T06:15:40.256Z+ retry=8
8352023-11-19T06:15:40.294Z+++ zoneadm list -n
8362023-11-19T06:15:40.333Z+++ grep oxz_ntp
8372023-11-19T06:15:40.371Z++ pfexec svcs -z oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5 -Hostate oxide/ntp
8382023-11-19T06:15:40.409Z+ [[ disabled != online ]]
8392023-11-19T06:15:40.448Z+ [[ 8 -gt 60 ]]
8402023-11-19T06:15:40.490Z+ sleep 1
8412023-11-19T06:15:41.279Z+ retry=9
8422023-11-19T06:15:41.317Z+++ zoneadm list -n
8432023-11-19T06:15:41.367Z+++ grep oxz_ntp
8442023-11-19T06:15:41.416Z++ pfexec svcs -z oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5 -Hostate oxide/ntp
8452023-11-19T06:15:41.465Z+ [[ disabled != online ]]
8462023-11-19T06:15:41.514Z+ [[ 9 -gt 60 ]]
8472023-11-19T06:15:41.568Z+ sleep 1
8482023-11-19T06:15:42.301Z+ retry=10
8492023-11-19T06:15:42.340Z+++ zoneadm list -n
8502023-11-19T06:15:42.378Z+++ grep oxz_ntp
8512023-11-19T06:15:42.417Z++ pfexec svcs -z oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5 -Hostate oxide/ntp
8522023-11-19T06:15:42.456Z+ [[ disabled != online ]]
8532023-11-19T06:15:42.494Z+ [[ 10 -gt 60 ]]
8542023-11-19T06:15:42.537Z+ sleep 1
8552023-11-19T06:15:43.322Z+ retry=11
8562023-11-19T06:15:43.360Z+++ zoneadm list -n
8572023-11-19T06:15:43.399Z+++ grep oxz_ntp
8582023-11-19T06:15:43.437Z++ pfexec svcs -z oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5 -Hostate oxide/ntp
8592023-11-19T06:15:43.475Z+ [[ disabled != online ]]
8602023-11-19T06:15:43.515Z+ [[ 11 -gt 60 ]]
8612023-11-19T06:15:43.568Z+ sleep 1
8622023-11-19T06:15:44.345Z+ retry=12
8632023-11-19T06:15:44.383Z+++ zoneadm list -n
8642023-11-19T06:15:44.435Z+++ grep oxz_ntp
8652023-11-19T06:15:44.483Z++ pfexec svcs -z oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5 -Hostate oxide/ntp
8662023-11-19T06:15:44.522Z+ [[ disabled != online ]]
8672023-11-19T06:15:44.560Z+ [[ 12 -gt 60 ]]
8682023-11-19T06:15:44.598Z+ sleep 1
8692023-11-19T06:15:45.370Z+ retry=13
 
8702023-11-19T06:15:45.383Zsock console login: Nov 19 06:15:44 sok ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8712023-11-19T06:15:45.417Z+++ zoneadm list -n
8722023-11-19T06:15:45.464Z+++ grep oxz_ntp
8732023-11-19T06:15:45.514Z++ pfexec svcs -z oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5 -Hostate oxide/ntp
8742023-11-19T06:15:45.564Z+ [[ disabled != online ]]
8752023-11-19T06:15:45.610Z+ [[ 13 -gt 60 ]]
8762023-11-19T06:15:45.656Z+ sleep 1
8772023-11-19T06:15:46.390Z+ retry=14
8782023-11-19T06:15:46.428Z+++ zoneadm list -n
8792023-11-19T06:15:46.467Z+++ grep oxz_ntp
8802023-11-19T06:15:46.507Z++ pfexec svcs -z oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5 -Hostate oxide/ntp
8812023-11-19T06:15:46.549Z+ [[ online != online ]]
8822023-11-19T06:15:46.591Z+ echo 'Waited for chrony: 14s'
8832023-11-19T06:15:46.631Z+ export RUST_BACKTRACE=1
8842023-11-19T06:15:46.674Z+ RUST_BACKTRACE=1
8852023-11-19T06:15:46.717ZWaited for chrony: 14s
8862023-11-19T06:15:46.756Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8872023-11-19T06:15:46.794Z+ ./tests/bootstrap
8882023-11-19T06:15:46.833ZUsing external DNS server at 10.151.2.164:53
8892023-11-19T06:15:46.871Z2023-11-19 06:15:45.816726503 UTC: attempting to log into API
8902023-11-19T06:16:01.473Z2023-11-19 06:16:01.319294825 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8912023-11-19T06:16:02.481Z2023-11-19 06:16:02.320413685 UTC: attempting to log into API
8922023-11-19T06:16:17.508Z2023-11-19 06:16:17.354024701 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8932023-11-19T06:16:18.544Z2023-11-19 06:16:18.355745766 UTC: attempting to log into API
8942023-11-19T06:16:33.602Z2023-11-19 06:16:33.401814314 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8952023-11-19T06:16:34.565Z2023-11-19 06:16:34.402842660 UTC: attempting to log into API
8962023-11-19T06:16:49.608Z2023-11-19 06:16:49.441107140 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8972023-11-19T06:16:50.622Z2023-11-19 06:16:50.442935998 UTC: attempting to log into API
 
8982023-11-19T06:16:58.702ZNov 19 06:16:57 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
8992023-11-19T06:17:04.294ZNov 19 06:17:03 sok ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9002023-11-19T06:17:05.650Z2023-11-19 06:17:05.230623807 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9012023-11-19T06:17:06.651Z2023-11-19 06:17:06.234419166 UTC: attempting to log into API
 
9022023-11-19T06:17:12.118ZNov 19 06:17:10 sok ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9032023-11-19T06:17:18.732ZNov 19 06:17:17 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9042023-11-19T06:17:21.684Z2023-11-19 06:17:21.307812369 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9052023-11-19T06:17:22.684Z2023-11-19 06:17:22.311569992 UTC: attempting to log into API
 
9062023-11-19T06:17:25.313ZNov 19 06:17:24 sok ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9072023-11-19T06:17:37.720Z2023-11-19 06:17:37.385982515 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9082023-11-19T06:17:38.730Z2023-11-19 06:17:38.390589167 UTC: attempting to log into API
9092023-11-19T06:17:53.755Z2023-11-19 06:17:53.465482176 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9102023-11-19T06:17:54.756Z2023-11-19 06:17:54.469303869 UTC: attempting to log into API
9112023-11-19T06:18:00.566Z2023-11-19 06:18:00.295098276 UTC: login succeeded
9122023-11-19T06:18:00.604Zwaiting for nexus to come up...
9132023-11-19T06:18:00.663Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9142023-11-19T06:18:00.707Zensuring datasets are ready...
9152023-11-19T06:18:02.699Zlet's roll.
9162023-11-19T06:18:02.738Z+ rm ./tests/bootstrap
9172023-11-19T06:18:02.777Z+ for test_bin in tests/*
9182023-11-19T06:18:02.816Z+ ./tests/bootstrap-714f127c345e1afe
9192023-11-19T06:18:02.855Z
9202023-11-19T06:18:02.893Zrunning 0 tests
9212023-11-19T06:18:02.932Z
9222023-11-19T06:18:02.971Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9232023-11-19T06:18:03.010Z
9242023-11-19T06:18:03.049Z+ for test_bin in tests/*
9252023-11-19T06:18:03.088Z+ ./tests/end_to_end_tests-2a5f0f6021ab3ba7
9262023-11-19T06:18:03.128Z
9272023-11-19T06:18:03.166Zrunning 2 tests
9282023-11-19T06:18:03.205Ztest no_spoof::no_spoof_here ... ok
9292023-11-19T06:19:02.780Ztest instance_launch::instance_launch has been running for over 60 seconds
9302023-11-19T06:20:58.105Ztest instance_launch::instance_launch ... ok
9312023-11-19T06:20:58.130Z
9322023-11-19T06:20:58.156Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 175.33s
9332023-11-19T06:20:58.181Z
9342023-11-19T06:20:58.206Z+ _exit_trap
9352023-11-19T06:20:58.230Z+ local status=0
9362023-11-19T06:20:58.255Z+ [[ 0 -eq 0 ]]
9372023-11-19T06:20:58.280Z+ exit 0
9382023-11-19T06:20:58.305Zprocess exited: duration 388729 ms, exit code 0
 
9392023-11-19T06:20:58.386Zfound 47 output files
9402023-11-19T06:20:58.454Zuploading: /var/svc/log/oxide-sled-agent:default.log (679913 bytes)
9412023-11-19T06:20:59.676Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9422023-11-19T06:20:59.700Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_0032c326-0d05-47fa-9372-284dc53b8484/root/var/svc/log/oxide-clickhouse:default.log (699780 bytes)
9432023-11-19T06:20:59.964Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_0032c326-0d05-47fa-9372-284dc53b8484/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 699780 -> 702338
9442023-11-19T06:21:01.038Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_0032c326-0d05-47fa-9372-284dc53b8484/root/var/svc/log/oxide-clickhouse:default.log
9452023-11-19T06:21:01.062Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45a461a9-bb97-4ec1-ad8f-928e0aaeca85/root/var/svc/log/oxide-crucible-agent:default.log (23934 bytes)
9462023-11-19T06:21:02.184Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45a461a9-bb97-4ec1-ad8f-928e0aaeca85/root/var/svc/log/oxide-crucible-agent:default.log
9472023-11-19T06:21:02.210Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45a461a9-bb97-4ec1-ad8f-928e0aaeca85/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9482023-11-19T06:21:03.286Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45a461a9-bb97-4ec1-ad8f-928e0aaeca85/root/var/svc/log/oxide-crucible-downstairs:default.log
9492023-11-19T06:21:03.311Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45a461a9-bb97-4ec1-ad8f-928e0aaeca85/root/var/svc/log/oxide-crucible-downstairs:downstairs-a3ae0d41-f723-46ac-9d4d-a42a157f76c9.log (5534 bytes)
9502023-11-19T06:21:04.388Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45a461a9-bb97-4ec1-ad8f-928e0aaeca85/root/var/svc/log/oxide-crucible-downstairs:downstairs-a3ae0d41-f723-46ac-9d4d-a42a157f76c9.log
9512023-11-19T06:21:04.415Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45a461a9-bb97-4ec1-ad8f-928e0aaeca85/root/var/svc/log/oxide-crucible-downstairs:downstairs-cda64192-c412-4b8a-94ef-22a64c8f91ea.log (3047 bytes)
9522023-11-19T06:21:05.491Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45a461a9-bb97-4ec1-ad8f-928e0aaeca85/root/var/svc/log/oxide-crucible-downstairs:downstairs-cda64192-c412-4b8a-94ef-22a64c8f91ea.log
9532023-11-19T06:21:05.515Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5/root/var/svc/log/oxide-ntp:default.log (1186 bytes)
9542023-11-19T06:21:06.590Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5/root/var/svc/log/oxide-ntp:default.log
9552023-11-19T06:21:06.614Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_06ec53e4-0f7c-48de-bedc-e76db7a54bc5/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9562023-11-19T06:21:07.690Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_06ec53e4-0f7c-48de-bedc-e76db7a54bc5/root/var/svc/log/oxide-crucible-agent:default.log
9572023-11-19T06:21:07.717Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_06ec53e4-0f7c-48de-bedc-e76db7a54bc5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9582023-11-19T06:21:08.791Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_06ec53e4-0f7c-48de-bedc-e76db7a54bc5/root/var/svc/log/oxide-crucible-downstairs:default.log
9592023-11-19T06:21:08.815Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7492b742-935c-461d-8ae5-397b8b1b8784/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
9602023-11-19T06:21:09.934Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7492b742-935c-461d-8ae5-397b8b1b8784/root/var/svc/log/oxide-crucible-agent:default.log
9612023-11-19T06:21:09.959Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7492b742-935c-461d-8ae5-397b8b1b8784/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9622023-11-19T06:21:11.032Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7492b742-935c-461d-8ae5-397b8b1b8784/root/var/svc/log/oxide-crucible-downstairs:default.log
9632023-11-19T06:21:11.056Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7492b742-935c-461d-8ae5-397b8b1b8784/root/var/svc/log/oxide-crucible-downstairs:downstairs-e16b71bc-9d32-4e7a-8d42-5adacf5b38a0.log (5536 bytes)
9642023-11-19T06:21:12.130Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7492b742-935c-461d-8ae5-397b8b1b8784/root/var/svc/log/oxide-crucible-downstairs:downstairs-e16b71bc-9d32-4e7a-8d42-5adacf5b38a0.log
9652023-11-19T06:21:12.154Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_4189230b-93cf-4203-a219-b8c12053857e/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
9662023-11-19T06:21:13.229Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_4189230b-93cf-4203-a219-b8c12053857e/root/var/svc/log/oxide-crucible-pantry:default.log
9672023-11-19T06:21:13.253Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_3e086d09-bc98-4194-88d9-221e85d4d9f4/root/var/svc/log/oxide-external_dns:default.log (15610 bytes)
9682023-11-19T06:21:14.351Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_3e086d09-bc98-4194-88d9-221e85d4d9f4/root/var/svc/log/oxide-external_dns:default.log
9692023-11-19T06:21:14.375Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_9385c33b-bcee-4342-bf4d-2649f94e3cd8/root/var/svc/log/oxide-internal_dns:default.log (99559 bytes)
9702023-11-19T06:21:15.564Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_9385c33b-bcee-4342-bf4d-2649f94e3cd8/root/var/svc/log/oxide-internal_dns:default.log
9712023-11-19T06:21:15.588Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c0381bbd-a317-45f4-a559-1446a79c7980/root/var/svc/log/oxide-nexus:default.log (690472 bytes)
9722023-11-19T06:21:15.831Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c0381bbd-a317-45f4-a559-1446a79c7980/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 690472 -> 691080
9732023-11-19T06:21:16.905Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c0381bbd-a317-45f4-a559-1446a79c7980/root/var/svc/log/oxide-nexus:default.log
9742023-11-19T06:21:16.929Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_03926440-d8b5-497e-89bd-c44e1d8d5058/root/var/svc/log/oxide-oximeter:default.log (177678 bytes)
9752023-11-19T06:21:17.094Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_03926440-d8b5-497e-89bd-c44e1d8d5058/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 177678 -> 192893
9762023-11-19T06:21:18.182Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_03926440-d8b5-497e-89bd-c44e1d8d5058/root/var/svc/log/oxide-oximeter:default.log
9772023-11-19T06:21:18.206Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_8dec2685-c04c-4dd5-9408-331c2105d4f9/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
9782023-11-19T06:21:19.282Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_8dec2685-c04c-4dd5-9408-331c2105d4f9/root/var/svc/log/oxide-cockroachdb:default.log
9792023-11-19T06:21:19.306Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_aaa92b54-6b4f-460e-9664-2b17243598e1/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
9802023-11-19T06:21:20.388Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_aaa92b54-6b4f-460e-9664-2b17243598e1/root/var/svc/log/oxide-cockroachdb:default.log
9812023-11-19T06:21:20.412Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_20556fb0-6d68-429c-ad3c-1aa256ad9d77/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
9822023-11-19T06:21:21.509Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_20556fb0-6d68-429c-ad3c-1aa256ad9d77/root/var/svc/log/oxide-crucible-agent:default.log
9832023-11-19T06:21:21.534Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_20556fb0-6d68-429c-ad3c-1aa256ad9d77/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9842023-11-19T06:21:22.609Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_20556fb0-6d68-429c-ad3c-1aa256ad9d77/root/var/svc/log/oxide-crucible-downstairs:default.log
9852023-11-19T06:21:22.633Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_20556fb0-6d68-429c-ad3c-1aa256ad9d77/root/var/svc/log/oxide-crucible-downstairs:downstairs-8639ef8e-59db-4a52-aa58-41b4a1b94654.log (5537 bytes)
9862023-11-19T06:21:24.731Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_20556fb0-6d68-429c-ad3c-1aa256ad9d77/root/var/svc/log/oxide-crucible-downstairs:downstairs-8639ef8e-59db-4a52-aa58-41b4a1b94654.log
9872023-11-19T06:21:24.756Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c423da-51c6-4fe0-8693-445f473a0afb/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
9882023-11-19T06:21:24.828Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c423da-51c6-4fe0-8693-445f473a0afb/root/var/svc/log/oxide-crucible-agent:default.log
9892023-11-19T06:21:24.852Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c423da-51c6-4fe0-8693-445f473a0afb/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9902023-11-19T06:21:25.926Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c423da-51c6-4fe0-8693-445f473a0afb/root/var/svc/log/oxide-crucible-downstairs:default.log
9912023-11-19T06:21:25.950Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c423da-51c6-4fe0-8693-445f473a0afb/root/var/svc/log/oxide-crucible-downstairs:downstairs-19c0fa36-c65e-464b-aa36-dbb21fe36f34.log (3048 bytes)
9922023-11-19T06:21:27.025Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c423da-51c6-4fe0-8693-445f473a0afb/root/var/svc/log/oxide-crucible-downstairs:downstairs-19c0fa36-c65e-464b-aa36-dbb21fe36f34.log
9932023-11-19T06:21:27.049Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_325d9741-632a-46b7-b85c-30d0332a1e02/root/var/svc/log/oxide-crucible-pantry:default.log (2336 bytes)
9942023-11-19T06:21:28.140Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_325d9741-632a-46b7-b85c-30d0332a1e02/root/var/svc/log/oxide-crucible-pantry:default.log
9952023-11-19T06:21:28.164Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_4579a2d0-6e4c-443e-8282-d39f15cd43f6/root/var/svc/log/oxide-internal_dns:default.log (103130 bytes)
9962023-11-19T06:21:28.435Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_4579a2d0-6e4c-443e-8282-d39f15cd43f6/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 103130 -> 106885
9972023-11-19T06:21:29.509Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_4579a2d0-6e4c-443e-8282-d39f15cd43f6/root/var/svc/log/oxide-internal_dns:default.log
9982023-11-19T06:21:29.533Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20dba04d-84be-4113-8c37-593217aafbf2/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9992023-11-19T06:21:30.609Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20dba04d-84be-4113-8c37-593217aafbf2/root/var/svc/log/oxide-crucible-agent:default.log
10002023-11-19T06:21:30.634Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20dba04d-84be-4113-8c37-593217aafbf2/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10012023-11-19T06:21:31.707Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20dba04d-84be-4113-8c37-593217aafbf2/root/var/svc/log/oxide-crucible-downstairs:default.log
10022023-11-19T06:21:31.732Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6fe5bff2-23e6-4edf-80fc-bd8e8a63b212/root/var/svc/log/oxide-crucible-agent:default.log (13685 bytes)
10032023-11-19T06:21:32.829Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6fe5bff2-23e6-4edf-80fc-bd8e8a63b212/root/var/svc/log/oxide-crucible-agent:default.log
10042023-11-19T06:21:32.855Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6fe5bff2-23e6-4edf-80fc-bd8e8a63b212/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10052023-11-19T06:21:33.929Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6fe5bff2-23e6-4edf-80fc-bd8e8a63b212/root/var/svc/log/oxide-crucible-downstairs:default.log
10062023-11-19T06:21:33.953Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6fe5bff2-23e6-4edf-80fc-bd8e8a63b212/root/var/svc/log/oxide-crucible-downstairs:downstairs-22a7dfd8-dea7-4232-89ce-7bc7c89d8265.log (3048 bytes)
10072023-11-19T06:21:35.028Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6fe5bff2-23e6-4edf-80fc-bd8e8a63b212/root/var/svc/log/oxide-crucible-downstairs:downstairs-22a7dfd8-dea7-4232-89ce-7bc7c89d8265.log
10082023-11-19T06:21:35.052Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_ab6876cf-2d38-4a8c-a5bf-19bb08d60c2e/root/var/svc/log/oxide-internal_dns:default.log (99888 bytes)
10092023-11-19T06:21:35.194Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_ab6876cf-2d38-4a8c-a5bf-19bb08d60c2e/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99888 -> 103642
10102023-11-19T06:21:36.268Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_ab6876cf-2d38-4a8c-a5bf-19bb08d60c2e/root/var/svc/log/oxide-internal_dns:default.log
10112023-11-19T06:21:36.291Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_638e639a-14d4-478c-84be-9ec124862d68/root/var/svc/log/oxide-nexus:default.log (4804878 bytes)
10122023-11-19T06:21:38.330Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_638e639a-14d4-478c-84be-9ec124862d68/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4804878 -> 4870833
10132023-11-19T06:21:39.426Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_638e639a-14d4-478c-84be-9ec124862d68/root/var/svc/log/oxide-nexus:default.log
10142023-11-19T06:21:39.450Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_35934ad4-9f6b-4845-a7f6-f2ecebb63d33/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
10152023-11-19T06:21:40.524Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_35934ad4-9f6b-4845-a7f6-f2ecebb63d33/root/var/svc/log/oxide-cockroachdb:default.log
10162023-11-19T06:21:40.548Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_d5a9978d-f1ce-4d2c-ad23-371e40fea5db/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10172023-11-19T06:21:41.623Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_d5a9978d-f1ce-4d2c-ad23-371e40fea5db/root/var/svc/log/oxide-crucible-pantry:default.log
10182023-11-19T06:21:41.647Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_4d5ce470-ee9e-4192-aeee-7e126faaa4b1/root/var/svc/log/oxide-external_dns:default.log (15611 bytes)
10192023-11-19T06:21:41.717Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_4d5ce470-ee9e-4192-aeee-7e126faaa4b1/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15611 -> 19387
10202023-11-19T06:21:42.792Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_4d5ce470-ee9e-4192-aeee-7e126faaa4b1/root/var/svc/log/oxide-external_dns:default.log
10212023-11-19T06:21:42.816Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_9ad3828c-86b7-4026-8615-7a8a43cac3b5/root/var/svc/log/oxide-nexus:default.log (645216 bytes)
10222023-11-19T06:21:43.059Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_9ad3828c-86b7-4026-8615-7a8a43cac3b5/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 645216 -> 709382
10232023-11-19T06:21:44.132Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_9ad3828c-86b7-4026-8615-7a8a43cac3b5/root/var/svc/log/oxide-nexus:default.log
10242023-11-19T06:21:44.156Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_29d9340a-0733-40d6-a185-db0fec9260c9/root/var/svc/log/oxide-cockroachdb:default.log (5341 bytes)
10252023-11-19T06:21:45.231Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_29d9340a-0733-40d6-a185-db0fec9260c9/root/var/svc/log/oxide-cockroachdb:default.log
10262023-11-19T06:21:45.254Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d595ebeb-4ba6-4491-8e56-43c14297897f/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10272023-11-19T06:21:46.329Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d595ebeb-4ba6-4491-8e56-43c14297897f/root/var/svc/log/oxide-crucible-agent:default.log
10282023-11-19T06:21:46.353Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d595ebeb-4ba6-4491-8e56-43c14297897f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10292023-11-19T06:21:47.428Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d595ebeb-4ba6-4491-8e56-43c14297897f/root/var/svc/log/oxide-crucible-downstairs:default.log
10302023-11-19T06:21:47.452Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_61f03afb-d40e-4344-8197-8abc3f06c6d5/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10312023-11-19T06:21:48.551Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_61f03afb-d40e-4344-8197-8abc3f06c6d5/root/var/svc/log/oxide-cockroachdb:default.log
10322023-11-19T06:21:48.574Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_01cffbf3-0f1c-4a10-8016-74fd873b1239/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10332023-11-19T06:21:49.625Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_01cffbf3-0f1c-4a10-8016-74fd873b1239/root/var/svc/log/oxide-crucible-agent:default.log
10342023-11-19T06:21:49.651Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_01cffbf3-0f1c-4a10-8016-74fd873b1239/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10352023-11-19T06:21:50.739Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_01cffbf3-0f1c-4a10-8016-74fd873b1239/root/var/svc/log/oxide-crucible-downstairs:default.log
10362023-11-19T06:21:50.762Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5/root/var/log/chrony/measurements.log (10960 bytes)
10372023-11-19T06:21:50.811Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5/root/var/log/chrony/measurements.log" changed size mid upload: 10960 -> 12056
10382023-11-19T06:21:51.884Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5/root/var/log/chrony/measurements.log
10392023-11-19T06:21:51.908Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5/root/var/log/chrony/statistics.log (6435 bytes)
10402023-11-19T06:21:51.960Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5/root/var/log/chrony/statistics.log" changed size mid upload: 6435 -> 7371
10412023-11-19T06:21:53.036Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5/root/var/log/chrony/statistics.log
10422023-11-19T06:21:53.062Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5/root/var/log/chrony/tracking.log (1717 bytes)
10432023-11-19T06:21:54.137Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2ebf9f15-f14c-4e71-a017-ba91e7d4f4c5/root/var/log/chrony/tracking.log