01HDANRH7RSEEYNVA2RS8JER97: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HDANV9AT0W8Y46G437YBJMD3

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-22T04:04:25.435Zjob dependencies complete; ready to run (waiting for 53 m 2 s)
22023-10-22T04:40:50.230Zjob assigned to worker 01HDATXF2113JR8HX9WFAXNPY8 (queued for 36 m 24 s)
32023-10-22T04:40:50.342Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-22T04:40:50.460Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-10-22T04:40:50.571Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-10-22T04:40:50.666Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-22T04:40:50.820Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-10-22T04:40:50.963ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-22T04:40:51.065Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-22T04:40:51.160ZOxide Helios Version helios-2.0.22241 64-bit
112023-10-22T04:41:05.276ZConfiguring device.
122023-10-22T04:41:05.338Z##################################################
132023-10-22T04:41:05.387Z#### ############# ##
142023-10-22T04:41:05.438Z### ### ############ ##
152023-10-22T04:41:05.486Z## ### # ## ### ## Oxide Computer Company ##
162023-10-22T04:41:05.518Z## ## ## ### # ### ##
172023-10-22T04:41:05.551Z## # ### #### #### This Station Under ##
182023-10-22T04:41:05.585Z### ### #### # ### Computer Control ##
192023-10-22T04:41:05.629Z#### #### ### ## ##
202023-10-22T04:41:05.671Z##################################################
212023-10-22T04:41:05.729ZHostname: buskin
222023-10-22T04:41:05.774ZLoading smf(7) serice descriptions:
232023-10-22T04:41:06.473Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-b4242ab00b4c43e7.gz
242023-10-22T04:41:09.946Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-b4242ab00b4c43e7.gz
252023-10-22T04:41:09.973Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-22T04:41:32.415Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-22T04:41:32.439Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-0e54d5c810ee3c0c.gz
282023-10-22T04:41:55.692Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-0e54d5c810ee3c0c.gz
292023-10-22T04:41:55.719Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-22T04:41:58.346Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-22T04:41:58.371Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-22T04:42:02.381Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-22T04:42:02.408Zdownloading input: /input/package/work/version.txt
342023-10-22T04:42:02.435Zdownloaded input: /input/package/work/version.txt
352023-10-22T04:42:02.462Zdownloading input: /input/package/work/package.tar.gz
362023-10-22T04:42:37.097Zdownloaded input: /input/package/work/package.tar.gz
372023-10-22T04:42:37.126Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-22T04:43:34.510Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-22T04:43:34.536Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-22T04:43:50.741Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-22T04:43:50.772Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-22T04:44:52.361Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-22T04:44:52.426Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-22T04:45:57.151Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-22T04:45:57.180Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-22T04:46:36.788Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-22T04:46:36.814Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-22T04:46:47.355Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-22T04:46:47.384Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-22T04:47:00.586Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-22T04:47:00.623Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-22T04:47:08.433Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-22T04:47:08.460Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-22T04:47:14.116Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-22T04:47:14.147Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-22T04:47:14.172Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-22T04:47:14.197Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-22T04:47:21.268Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-22T04:47:21.293Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-22T04:47:28.198Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-22T04:47:28.229Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-22T04:47:54.104Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-22T04:47:54.139Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-22T04:48:29.263Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-22T04:48:29.700Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-22T04:48:29.727Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-22T04:48:29.755Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-22T04:48:32.520Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-22T04:48:32.586Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-22T04:48:48.717Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-22T04:48:48.741Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-22T04:49:33.950Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-22T04:49:33.981Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-22T04:50:04.195Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-22T04:50:04.222Zstarting task 0: "setup"
762023-10-22T04:50:04.256Z++ uname -s
772023-10-22T04:50:04.282Z+ kern=SunOS
782023-10-22T04:50:04.308Z+ case "$kern" in
792023-10-22T04:50:04.333Z+ groupadd -g 12345 build
802023-10-22T04:50:04.358Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-22T04:50:06.266Z+ zfs create -o mountpoint=/work rpool/work
822023-10-22T04:50:06.291Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-22T04:50:06.316Z+ home_fs=zfs
842023-10-22T04:50:06.341Z+ [[ zfs == autofs ]]
852023-10-22T04:50:06.366Z+ mkdir -p /home/build
862023-10-22T04:50:06.390Z+ chown build:build /home/build /work
872023-10-22T04:50:08.296Z+ chmod 0700 /home/build /work
882023-10-22T04:50:08.321Zprocess exited: duration 4053 ms, exit code 0
 
892023-10-22T04:50:08.371Zstarting task 1: "authentication"
902023-10-22T04:50:08.440Zprocess exited: duration 44 ms, exit code 0
 
912023-10-22T04:50:08.490Zstarting task 2: "build"
922023-10-22T04:50:08.522Z+ trap _exit_trap EXIT
932023-10-22T04:50:08.547Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-22T04:50:08.724Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-22T04:50:08.749Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-22T04:50:08.774Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-22T04:50:08.798Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-22T04:50:08.824Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-22T04:50:08.855Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-22T04:50:08.891Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-22T04:50:08.918Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-22T04:50:08.942Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-22T04:50:08.967Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-22T04:50:08.991Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-22T04:50:09.016Z+ [[ -d /opt/oxide ]]
1062023-10-22T04:50:09.040Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-22T04:50:09.065Z+ pfexec mkdir /opt/oxide
1082023-10-22T04:50:09.089Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-22T04:50:09.114Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-22T04:50:09.138Z+ cd /opt/oxide-underneath
1112023-10-22T04:50:09.163Z+ pfexec tar ceEp@/f - .
1122023-10-22T04:50:09.188Z+ cd /opt/oxide
1132023-10-22T04:50:09.229Z+ pfexec tar xveEp@/f -
1142023-10-22T04:50:09.254Zx ., 0 bytes, 0 tape blocks
1152023-10-22T04:50:09.278Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-22T04:50:09.302Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-22T04:50:09.328Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-10-22T04:50:09.354Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-22T04:50:09.378Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-22T04:50:09.404Z+ pfexec mkdir /opt/oxide/work
1212023-10-22T04:50:09.429Z+ pfexec chown build:build /opt/oxide/work
1222023-10-22T04:50:09.454Z+ cd /opt/oxide/work
1232023-10-22T04:50:09.478Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-22T04:50:10.444ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-22T04:50:10.469Zx out/mg-ddm-gz.tar, 63638528 bytes, 124294 tape blocks
1262023-10-22T04:50:10.494Zx out/omicron-sled-agent.tar, 114954240 bytes, 224520 tape blocks
1272023-10-22T04:50:10.519Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-22T04:50:10.544Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-10-22T04:50:10.569Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-10-22T04:50:10.598Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-22T04:50:10.632Zx target/release/omicron-package, 38196144 bytes, 74602 tape blocks
1322023-10-22T04:50:11.070Zx tools/create_virtual_hardware.sh, 2986 bytes, 6 tape blocks
1332023-10-22T04:50:11.518Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-22T04:50:11.982Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-22T04:50:12.008Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-22T04:50:12.037Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-22T04:50:12.062Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-22T04:50:12.203Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-22T04:50:12.229Z
1402023-10-22T04:50:12.254Zreal 1.556860007
1412023-10-22T04:50:12.279Zuser 1.396620796
1422023-10-22T04:50:12.305Zsys 0.274221327
1432023-10-22T04:50:12.334Ztrap 0.000067250
1442023-10-22T04:50:12.359Ztflt 0.000000000
1452023-10-22T04:50:12.383Zdflt 0.000000000
1462023-10-22T04:50:12.408Zkflt 0.000000000
1472023-10-22T04:50:12.432Zlock 0.000000000
1482023-10-22T04:50:12.458Zslp 1.407270955
1492023-10-22T04:50:12.485Zlat 0.033621872
1502023-10-22T04:50:12.511Zstop 0.000121911
1512023-10-22T04:50:12.536Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-22T04:50:12.560Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-22T04:50:12.585Z+ mkdir tests
1542023-10-22T04:50:12.610Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-22T04:50:12.634Z+ ptime -m gunzip
1562023-10-22T04:50:12.659Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-b4242ab00b4c43e7
1572023-10-22T04:50:12.683Z
1582023-10-22T04:50:12.707Zreal 0.147596379
1592023-10-22T04:50:12.732Zuser 0.132050051
1602023-10-22T04:50:12.757Zsys 0.015296625
1612023-10-22T04:50:12.781Ztrap 0.000012119
1622023-10-22T04:50:12.805Ztflt 0.000000000
1632023-10-22T04:50:12.829Zdflt 0.000000000
1642023-10-22T04:50:12.854Zkflt 0.000000000
1652023-10-22T04:50:12.878Zlock 0.000000000
1662023-10-22T04:50:12.903Zslp 0.000021160
1672023-10-22T04:50:12.927Zlat 0.000023199
1682023-10-22T04:50:12.952Zstop 0.000125821
1692023-10-22T04:50:12.976Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-b4242ab00b4c43e7
1702023-10-22T04:50:13.001Z+ chmod a+x tests/bootstrap-b4242ab00b4c43e7
1712023-10-22T04:50:13.025Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-22T04:50:13.050Z+ ptime -m gunzip
1732023-10-22T04:50:13.074Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-22T04:50:13.926Z
1752023-10-22T04:50:13.950Zreal 2.226254992
1762023-10-22T04:50:13.975Zuser 2.099082267
1772023-10-22T04:50:13.999Zsys 0.126775841
1782023-10-22T04:50:14.024Ztrap 0.000177342
1792023-10-22T04:50:14.048Ztflt 0.000000000
1802023-10-22T04:50:14.073Zdflt 0.000000000
1812023-10-22T04:50:14.097Zkflt 0.000000000
1822023-10-22T04:50:14.122Zlock 0.000000000
1832023-10-22T04:50:14.148Zslp 0.000022250
1842023-10-22T04:50:14.172Zlat 0.000021470
1852023-10-22T04:50:14.197Zstop 0.000123581
1862023-10-22T04:50:14.221Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-22T04:50:14.245Z+ chmod a+x tests/bootstrap
1882023-10-22T04:50:14.270Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-22T04:50:14.294Z+ ptime -m gunzip
1902023-10-22T04:50:14.318Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-0e54d5c810ee3c0c
1912023-10-22T04:50:16.507Z
1922023-10-22T04:50:16.532Zreal 2.568858561
1932023-10-22T04:50:16.556Zuser 2.389915294
1942023-10-22T04:50:16.581Zsys 0.178499450
1952023-10-22T04:50:16.606Ztrap 0.000193212
1962023-10-22T04:50:16.630Ztflt 0.000000000
1972023-10-22T04:50:16.655Zdflt 0.000000000
1982023-10-22T04:50:16.680Zkflt 0.000000000
1992023-10-22T04:50:16.704Zlock 0.000000000
2002023-10-22T04:50:16.728Zslp 0.000026140
2012023-10-22T04:50:16.753Zlat 0.000030650
2022023-10-22T04:50:16.777Zstop 0.000131391
2032023-10-22T04:50:16.801Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-0e54d5c810ee3c0c
2042023-10-22T04:50:16.830Z+ chmod a+x tests/end_to_end_tests-0e54d5c810ee3c0c
2052023-10-22T04:50:16.865Z+ bmat address ls
2062023-10-22T04:50:16.891ZNAME CIDR FIRST LAST COUNT
2072023-10-22T04:50:16.916Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-10-22T04:50:16.940Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-22T04:50:16.965Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-10-22T04:50:16.989Z+ EXTRA_IP_START=10.151.1.164
2112023-10-22T04:50:17.014Z+ EXTRA_IP_END=10.151.1.227
2122023-10-22T04:50:17.038Z+ EXTRA_IP_COUNT=64
2132023-10-22T04:50:17.062Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-10-22T04:50:17.087Z+ EXTRA_IP_BASE=10.151.1
2152023-10-22T04:50:17.112Z+ EXTRA_IP_FOCTET=164
2162023-10-22T04:50:17.136Z+ EXTRA_IP_LOCTET=227
2172023-10-22T04:50:17.161Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-10-22T04:50:17.185Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-10-22T04:50:17.210Z+ DNS_IP1=10.151.1.164
2202023-10-22T04:50:17.235Z+ DNS_IP2=10.151.1.165
2212023-10-22T04:50:17.260Z+ UPLINK_IP=10.151.1.174
2222023-10-22T04:50:17.285Z+ IPPOOL_START=10.151.1.175
2232023-10-22T04:50:17.309Z+ IPPOOL_END=10.151.1.227
2242023-10-22T04:50:17.334Z++ ipadm show-addr -po type,addr
2252023-10-22T04:50:17.358Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-10-22T04:50:17.383Z+ GATEWAY_IP=10.151.1.100
2272023-10-22T04:50:17.407Z+ [[ -n 10.151.1.100 ]]
2282023-10-22T04:50:17.432Z+ ping -s 10.151.1.100 56 1
2292023-10-22T04:50:17.457ZPING 10.151.1.100: 56 data bytes
2302023-10-22T04:50:17.482Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.012 ms
2312023-10-22T04:50:17.509Z
2322023-10-22T04:50:17.534Z----10.151.1.100 PING Statistics----
2332023-10-22T04:50:17.559Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-10-22T04:50:17.583Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2352023-10-22T04:50:17.608Z++ arp -an
2362023-10-22T04:50:17.632Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-10-22T04:50:17.657Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-10-22T04:50:17.682Z+ routeadm -e ipv4-forwarding -u
2392023-10-22T04:50:17.706Z+ PXA_START=10.151.1.164
2402023-10-22T04:50:17.731Z+ PXA_END=10.151.1.227
2412023-10-22T04:50:17.756Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-10-22T04:50:17.780Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-10-22T04:50:17.805Z+ ZPOOL_VDEV_DIR=/scratch
2442023-10-22T04:50:17.830Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-10-22T04:50:17.855Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-10-22T04:50:17.879Z++ cd ./tools
2472023-10-22T04:50:17.905Z++ pwd
2482023-10-22T04:50:17.930Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-10-22T04:50:17.954Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-10-22T04:50:17.979Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-10-22T04:50:18.004Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-10-22T04:50:18.029Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-10-22T04:50:18.056Z++ dladm show-phys -p -o LINK
2542023-10-22T04:50:18.087Z++ head -1
2552023-10-22T04:50:18.117Z+ PHYSICAL_LINK=igb0
2562023-10-22T04:50:18.147Z+ echo 'Using igb0 as physical link'
2572023-10-22T04:50:18.173ZUsing igb0 as physical link
2582023-10-22T04:50:18.198Z+ ensure_run_as_root
2592023-10-22T04:50:18.222Z++ id -u
2602023-10-22T04:50:18.247Z+ [[ 0 -ne 0 ]]
2612023-10-22T04:50:18.271Z+ ensure_zpools
2622023-10-22T04:50:18.296Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-10-22T04:50:18.320Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-10-22T04:50:18.344Z+ readarray -t ZPOOLS
2652023-10-22T04:50:18.369Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-10-22T04:50:18.394Z++ sed 's/[ ",]//g'
2672023-10-22T04:50:18.418Z+ for ZPOOL in "${ZPOOLS[@]}"
2682023-10-22T04:50:18.458Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2692023-10-22T04:50:18.483ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2702023-10-22T04:50:18.508Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2712023-10-22T04:50:18.532Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2722023-10-22T04:50:18.556Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2732023-10-22T04:50:18.581Z0+0 records in
2742023-10-22T04:50:18.605Z0+0 records out
2752023-10-22T04:50:18.632Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2762023-10-22T04:50:18.656ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2772023-10-22T04:50:18.681Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-10-22T04:50:18.705ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2792023-10-22T04:50:18.731Z+ set +x
2802023-10-22T04:50:18.756ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2812023-10-22T04:50:18.783Z++ zpool list -o name
2822023-10-22T04:50:18.813ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2832023-10-22T04:50:18.843Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2842023-10-22T04:50:18.871ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2852023-10-22T04:50:18.903Z+ [[ -z '' ]]
2862023-10-22T04:50:18.928ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2872023-10-22T04:50:18.954Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2882023-10-22T04:50:18.980ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2892023-10-22T04:50:19.005Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2902023-10-22T04:50:19.030ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2912023-10-22T04:50:19.056Z+ set +x
2922023-10-22T04:50:19.087ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2932023-10-22T04:50:19.113Z+ for ZPOOL in "${ZPOOLS[@]}"
2942023-10-22T04:50:19.142ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2952023-10-22T04:50:19.168Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2962023-10-22T04:50:19.194ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2972023-10-22T04:50:19.223Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2982023-10-22T04:50:19.250ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2992023-10-22T04:50:19.279Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3002023-10-22T04:50:19.311ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3012023-10-22T04:50:19.338Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3022023-10-22T04:50:19.364ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3032023-10-22T04:50:19.393Z0+0 records in
3042023-10-22T04:50:19.424ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3052023-10-22T04:50:19.451Z0+0 records out
3062023-10-22T04:50:19.483ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3072023-10-22T04:50:19.518Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3082023-10-22T04:50:19.547ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3092023-10-22T04:50:19.571Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3102023-10-22T04:50:19.597ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3112023-10-22T04:50:19.621Z+ set +x
3122023-10-22T04:50:19.646ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3132023-10-22T04:50:19.670Z++ zpool list -o name
3142023-10-22T04:50:19.695ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3152023-10-22T04:50:19.727Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3162023-10-22T04:50:19.754ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3172023-10-22T04:50:19.779Z+ [[ -z '' ]]
3182023-10-22T04:50:19.812ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3192023-10-22T04:50:19.839Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3202023-10-22T04:50:19.864ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3212023-10-22T04:50:19.888Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3222023-10-22T04:50:19.913ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3232023-10-22T04:50:19.938Z+ set +x
3242023-10-22T04:50:20.369ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3252023-10-22T04:50:20.395Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-10-22T04:50:20.420ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3272023-10-22T04:50:20.445Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3282023-10-22T04:50:20.472ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3292023-10-22T04:50:20.507Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3302023-10-22T04:50:20.532ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3312023-10-22T04:50:20.556Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3322023-10-22T04:50:20.581ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3332023-10-22T04:50:20.606Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3342023-10-22T04:50:20.630ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3352023-10-22T04:50:20.655Z0+0 records in
3362023-10-22T04:50:20.680ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3372023-10-22T04:50:20.714Z0+0 records out
3382023-10-22T04:50:20.739ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3392023-10-22T04:50:20.763Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3402023-10-22T04:50:20.788ZSimnet net0/sc0_0 exists
3412023-10-22T04:50:20.820Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3422023-10-22T04:50:20.878ZSimnet net1/sc1_0 exists
3432023-10-22T04:50:20.909Z+ set +x
3442023-10-22T04:50:20.938ZVnic sc0_1 exists
3452023-10-22T04:50:20.963Z++ zpool list -o name
3462023-10-22T04:50:20.987Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3472023-10-22T04:50:21.011Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3482023-10-22T04:50:21.036Z+ [[ -z '' ]]
3492023-10-22T04:50:21.060Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-10-22T04:50:21.085Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-10-22T04:50:21.110Z+ set +x
3522023-10-22T04:50:21.144Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-10-22T04:50:21.171Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-10-22T04:50:21.196Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-10-22T04:50:21.221Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-10-22T04:50:21.253Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-10-22T04:50:21.278Z0+0 records in
3582023-10-22T04:50:21.302Z0+0 records out
3592023-10-22T04:50:21.328Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3602023-10-22T04:50:21.352Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-10-22T04:50:21.377Z+ set +x
3622023-10-22T04:50:21.428Z++ zpool list -o name
3632023-10-22T04:50:21.452Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-10-22T04:50:21.477Z+ [[ -z '' ]]
3652023-10-22T04:50:21.502Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-10-22T04:50:21.531Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-10-22T04:50:21.557Z+ set +x
3682023-10-22T04:50:21.588Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-10-22T04:50:21.615Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-10-22T04:50:21.648Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-10-22T04:50:21.679Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-10-22T04:50:21.707Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-10-22T04:50:21.736Z0+0 records in
3742023-10-22T04:50:21.763Z0+0 records out
3752023-10-22T04:50:21.797Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3762023-10-22T04:50:21.823Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-10-22T04:50:21.847Z+ set +x
3782023-10-22T04:50:21.873Z++ zpool list -o name
3792023-10-22T04:50:21.897Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-10-22T04:50:21.922Z+ [[ -z '' ]]
3812023-10-22T04:50:21.947Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-10-22T04:50:21.971Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-10-22T04:50:21.996Z+ set +x
3842023-10-22T04:50:22.022Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-10-22T04:50:22.047Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862023-10-22T04:50:22.073Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872023-10-22T04:50:22.097Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882023-10-22T04:50:22.122Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892023-10-22T04:50:22.150Z0+0 records in
3902023-10-22T04:50:22.184Z0+0 records out
3912023-10-22T04:50:22.211Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-10-22T04:50:22.244Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932023-10-22T04:50:22.271Z+ set +x
3942023-10-22T04:50:22.304Z++ zpool list -o name
3952023-10-22T04:50:22.332Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3962023-10-22T04:50:22.356Z+ [[ -z '' ]]
3972023-10-22T04:50:22.381Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3982023-10-22T04:50:22.405Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3992023-10-22T04:50:22.430Z+ set +x
4002023-10-22T04:50:22.454Z+ for ZPOOL in "${ZPOOLS[@]}"
4012023-10-22T04:50:22.483Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4022023-10-22T04:50:22.512Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4032023-10-22T04:50:22.559Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4042023-10-22T04:50:22.584Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4052023-10-22T04:50:22.610Z0+0 records in
4062023-10-22T04:50:22.634Z0+0 records out
4072023-10-22T04:50:22.659Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4082023-10-22T04:50:22.684Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4092023-10-22T04:50:22.709Z+ set +x
4102023-10-22T04:50:23.148Z++ zpool list -o name
4112023-10-22T04:50:23.173Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4122023-10-22T04:50:23.197Z+ [[ -z '' ]]
4132023-10-22T04:50:23.222Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4142023-10-22T04:50:23.246Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4152023-10-22T04:50:23.272Z+ set +x
4162023-10-22T04:50:23.296Z+ for ZPOOL in "${ZPOOLS[@]}"
4172023-10-22T04:50:23.320Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4182023-10-22T04:50:23.344Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192023-10-22T04:50:23.369Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4202023-10-22T04:50:23.393Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4212023-10-22T04:50:23.418Z0+0 records in
4222023-10-22T04:50:23.442Z0+0 records out
4232023-10-22T04:50:23.467Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4242023-10-22T04:50:23.492Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4252023-10-22T04:50:23.517Z+ set +x
4262023-10-22T04:50:23.542Z++ zpool list -o name
4272023-10-22T04:50:23.593Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4282023-10-22T04:50:23.618Z+ [[ -z '' ]]
4292023-10-22T04:50:23.643Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4302023-10-22T04:50:23.667Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4312023-10-22T04:50:23.693Z+ set +x
4322023-10-22T04:50:23.720Z+ for ZPOOL in "${ZPOOLS[@]}"
4332023-10-22T04:50:23.751Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4342023-10-22T04:50:23.778Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4352023-10-22T04:50:23.820Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4362023-10-22T04:50:23.856Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4372023-10-22T04:50:23.887Z0+0 records in
4382023-10-22T04:50:23.916Z0+0 records out
4392023-10-22T04:50:23.947Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4402023-10-22T04:50:23.982Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4412023-10-22T04:50:24.014Z+ set +x
4422023-10-22T04:50:24.056Z++ zpool list -o name
4432023-10-22T04:50:24.096Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4442023-10-22T04:50:24.142Z+ [[ -z '' ]]
4452023-10-22T04:50:24.171Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4462023-10-22T04:50:24.197Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4472023-10-22T04:50:24.224Z+ set +x
4482023-10-22T04:50:24.251Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4492023-10-22T04:50:24.278Z+ readarray -t ZPOOLS
4502023-10-22T04:50:24.304Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4512023-10-22T04:50:24.330Z++ sed 's/[ ",]//g'
4522023-10-22T04:50:24.357Z+ for ZPOOL in "${ZPOOLS[@]}"
4532023-10-22T04:50:24.383Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4542023-10-22T04:50:24.410Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4552023-10-22T04:50:24.434Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4562023-10-22T04:50:24.459Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4572023-10-22T04:50:24.489Z0+0 records in
4582023-10-22T04:50:24.513Z0+0 records out
4592023-10-22T04:50:24.538Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4602023-10-22T04:50:24.563Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4612023-10-22T04:50:24.622Z+ set +x
4622023-10-22T04:50:24.647Z++ zpool list -o name
4632023-10-22T04:50:24.674Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4642023-10-22T04:50:24.699Z+ [[ -z '' ]]
4652023-10-22T04:50:24.724Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-10-22T04:50:24.749Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4672023-10-22T04:50:24.773Z+ set +x
4682023-10-22T04:50:24.798Z+ for ZPOOL in "${ZPOOLS[@]}"
4692023-10-22T04:50:24.822Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4702023-10-22T04:50:24.847Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4712023-10-22T04:50:24.871Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4722023-10-22T04:50:24.897Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4732023-10-22T04:50:24.928Z0+0 records in
4742023-10-22T04:50:24.953Z0+0 records out
4752023-10-22T04:50:24.977Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4762023-10-22T04:50:25.002Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4772023-10-22T04:50:25.033Z+ set +x
4782023-10-22T04:50:25.062Z++ zpool list -o name
4792023-10-22T04:50:25.087Zdownloading softnpu ...done
4802023-10-22T04:50:25.111Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4812023-10-22T04:50:25.136Z+ [[ -z '' ]]
4822023-10-22T04:50:25.161Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4832023-10-22T04:50:25.185Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4842023-10-22T04:50:25.210Z+ set +x
4852023-10-22T04:50:25.235Z+ ensure_simulated_links igb0
4862023-10-22T04:50:25.259Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/libsidecar_lite.so.sha256.txt
4872023-10-22T04:50:25.284Z+ local PHYSICAL_LINK=igb0
4882023-10-22T04:50:25.308Z+ INDICES=("0" "1")
4892023-10-22T04:50:25.335Z+ for I in "${INDICES[@]}"
4902023-10-22T04:50:25.361Z++ get_simnet_name_if_exists net0
4912023-10-22T04:50:25.386Z++ dladm show-simnet -p -o LINK net0
4922023-10-22T04:50:25.411Z++ echo ''
4932023-10-22T04:50:25.435Z+ [[ -z '' ]]
4942023-10-22T04:50:25.464Z+ dladm create-simnet -t net0
4952023-10-22T04:50:25.498Z+ dladm create-simnet -t sc0_0
4962023-10-22T04:50:25.524Z+ dladm modify-simnet -t -p net0 sc0_0
4972023-10-22T04:50:25.550Z+ dladm set-linkprop -p mtu=9000 sc0_0
4982023-10-22T04:50:25.574Z+ success 'Simnet net0/sc0_0 exists'
4992023-10-22T04:50:25.598Z+ set +x
5002023-10-22T04:50:25.647Z+ for I in "${INDICES[@]}"
5012023-10-22T04:50:25.672Z++ get_simnet_name_if_exists net1
5022023-10-22T04:50:25.696Z++ dladm show-simnet -p -o LINK net1
5032023-10-22T04:50:25.721Z++ echo ''
5042023-10-22T04:50:25.745Z+ [[ -z '' ]]
5052023-10-22T04:50:25.770Z+ dladm create-simnet -t net1
5062023-10-22T04:50:25.795Z+ dladm create-simnet -t sc1_0
5072023-10-22T04:50:25.819Z+ dladm modify-simnet -t -p net1 sc1_0
5082023-10-22T04:50:25.844Z+ dladm set-linkprop -p mtu=9000 sc1_0
5092023-10-22T04:50:25.868Z+ success 'Simnet net1/sc1_0 exists'
5102023-10-22T04:50:25.897Z+ set +x
5112023-10-22T04:50:25.924Z++ get_vnic_name_if_exists sc0_1
5122023-10-22T04:50:25.949Z++ dladm show-vnic -p -o LINK sc0_1
5132023-10-22T04:50:25.974Z++ echo ''
5142023-10-22T04:50:25.998Z+ [[ -z '' ]]
5152023-10-22T04:50:26.023Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162023-10-22T04:50:26.048Z+ [[ -v PROMISC_FILT_OFF ]]
5172023-10-22T04:50:26.073Z+ success 'Vnic sc0_1 exists'
5182023-10-22T04:50:26.097Z+ set +x
5192023-10-22T04:50:26.123Z+ warn_if_no_proxy_arp
5202023-10-22T04:50:26.152Z+ [[ -v PXA_START ]]
5212023-10-22T04:50:26.178Z+ [[ -v PXA_END ]]
5222023-10-22T04:50:26.202Z+ ensure_softnpu_zone
5232023-10-22T04:50:26.227Z+ zoneadm list
5242023-10-22T04:50:26.251Z+ grep -q sidecar_softnpu
5252023-10-22T04:50:26.276Z+ [[ -f out/npuzone/npuzone ]]
5262023-10-22T04:50:26.300Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch omicron-tracking
5272023-10-22T04:50:28.256Zdownloading asic_program.so ...done
5282023-10-22T04:50:28.309Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/scadm.sha256.txt
5292023-10-22T04:50:33.316Zdownloading scadm ...done
5302023-10-22T04:50:33.341Zcreating port tfportrear0_0 for sc0_0
5312023-10-22T04:50:33.373Zcreating port tfportqsfp0_0 for sc0_1
5322023-10-22T04:50:33.397Zsoftnpu zone setup
5332023-10-22T04:50:33.422Zconfigure zone
5342023-10-22T04:50:33.452Zinstall zone
5352023-10-22T04:50:34.402Zboot zone
5362023-10-22T04:50:37.511Z[sidecar_softnpu] svcs milestone/network
5372023-10-22T04:50:37.538ZSTATE STIME FMRI
5382023-10-22T04:50:37.564Zonline 4:50:36 svc:/milestone/network:default
5392023-10-22T04:50:37.588Z[sidecar_softnpu] /softnpu/npu start
5402023-10-22T04:50:37.613Z
5412023-10-22T04:50:37.637Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5422023-10-22T04:50:37.662Z+ GATEWAY_IP=10.151.1.100
5432023-10-22T04:50:37.686Z+ echo 'Using 10.151.1.100 as gateway ip'
5442023-10-22T04:50:37.713ZUsing 10.151.1.100 as gateway ip
5452023-10-22T04:50:37.738Z+ [[ ! -v GATEWAY_MAC ]]
5462023-10-22T04:50:37.763Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5472023-10-22T04:50:37.787Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5482023-10-22T04:50:37.811Z+++ echo 18:c0:4d:0d:9f:b2
5492023-10-22T04:50:37.835Z+ COUNT=1
5502023-10-22T04:50:37.862Z+ [[ 1 -ne 1 ]]
5512023-10-22T04:50:37.894Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5522023-10-22T04:50:37.918ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5532023-10-22T04:50:37.943Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5542023-10-22T04:50:37.969Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5552023-10-22T04:50:37.997Z+ PXA_MAC=a8:e1:de:01:70:1d
5562023-10-22T04:50:38.023Z+ [[ -v PXA_START ]]
5572023-10-22T04:50:38.048Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5582023-10-22T04:50:38.072Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5592023-10-22T04:50:38.096Z+ z_scadm dump-state
5602023-10-22T04:50:38.121Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5612023-10-22T04:50:38.145Zlocal v6:
5622023-10-22T04:50:38.169Zlocal v4:
5632023-10-22T04:50:38.193Zrouter v6:
5642023-10-22T04:50:38.217Zrouter v4:
5652023-10-22T04:50:38.242Zresolver v4:
5662023-10-22T04:50:38.266Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5672023-10-22T04:50:38.291Zresolver v6:
5682023-10-22T04:50:38.316Znat v4:
5692023-10-22T04:50:38.341Znat_v6:
5702023-10-22T04:50:38.365Zport_mac:
5712023-10-22T04:50:38.389Zproxy arp:
5722023-10-22T04:50:38.414Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5732023-10-22T04:50:38.438Z+ success 'softnpu zone exists'
5742023-10-22T04:50:38.463Z+ set +x
5752023-10-22T04:50:38.487Zsoftnpu zone exists
5762023-10-22T04:50:38.512Z
5772023-10-22T04:50:38.536Zreal 20.795044140
5782023-10-22T04:50:38.561Zuser 0.701126969
5792023-10-22T04:50:38.585Zsys 1.605966859
5802023-10-22T04:50:38.609Ztrap 0.000079623
5812023-10-22T04:50:38.634Ztflt 0.000110067
5822023-10-22T04:50:38.658Zdflt 0.000945392
5832023-10-22T04:50:38.682Zkflt 0.000000000
5842023-10-22T04:50:38.707Zlock 16:08.820683121
5852023-10-22T04:50:38.732Zslp 1:01.692824977
5862023-10-22T04:50:38.757Zlat 0.069848466
5872023-10-22T04:50:38.783Zstop 0.368401368
5882023-10-22T04:50:38.808Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5892023-10-22T04:50:38.832Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5902023-10-22T04:50:38.857Z+ SILO_NAME=recovery
5912023-10-22T04:50:38.881Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5922023-10-22T04:50:38.905Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5932023-10-22T04:50:38.930Z+ sed -i~ '
5942023-10-22T04:50:38.954Z /^external_dns_ips/c\
5952023-10-22T04:50:38.978Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5962023-10-22T04:50:39.023Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5972023-10-22T04:50:39.048Z /^first/c\
5982023-10-22T04:50:39.072Zfirst = "10.151.1.164"
5992023-10-22T04:50:39.098Z /^last/c\
6002023-10-22T04:50:39.128Zlast = "10.151.1.173"
6012023-10-22T04:50:39.153Z }
6022023-10-22T04:50:39.178Z /^\[rack_network_config/,/^$/ {
6032023-10-22T04:50:39.202Z /^infra_ip_first/c\
6042023-10-22T04:50:39.226Zinfra_ip_first = "10.151.1.174"
6052023-10-22T04:50:39.251Z /^infra_ip_last/c\
6062023-10-22T04:50:39.275Zinfra_ip_last = "10.151.1.174"
6072023-10-22T04:50:39.300Z }
6082023-10-22T04:50:39.324Z /^\[\[rack_network_config.ports/,/^$/ {
6092023-10-22T04:50:39.349Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6102023-10-22T04:50:39.373Z /^routes/c\
6112023-10-22T04:50:39.397Z+++ pkg/config-rss.toml Sun Oct 22 04:50:38 2023
6122023-10-22T04:50:39.422Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6132023-10-22T04:50:39.446Z@@ -41,7 +41,7 @@
6142023-10-22T04:50:39.471Z /^addresses/c\
6152023-10-22T04:50:39.495Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6162023-10-22T04:50:39.519Zaddresses = \["10.151.1.174/32"\]
6172023-10-22T04:50:39.545Z # must be contained in one of the "internal services" IP Pool ranges listed
6182023-10-22T04:50:39.569Z }
6192023-10-22T04:50:39.594Z # below.
6202023-10-22T04:50:39.618Z' pkg/config-rss.toml
6212023-10-22T04:50:39.643Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6222023-10-22T04:50:39.668Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6232023-10-22T04:50:39.692Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6242023-10-22T04:50:39.726Z+ true
6252023-10-22T04:50:39.750Z
6262023-10-22T04:50:39.774Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6272023-10-22T04:50:39.799Z # Initial TLS certificates for the external API
6282023-10-22T04:50:39.823Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6292023-10-22T04:50:39.847Z #
6302023-10-22T04:50:39.872Z+ TLS_NAME=recovery.sys.oxide.test
6312023-10-22T04:50:39.896Z@@ -77,8 +77,8 @@
6322023-10-22T04:50:39.920Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6332023-10-22T04:50:39.945Z #
6342023-10-22T04:50:39.969Z.....+...+..+.+.........+...+......+.........+.....+............+....+..+...+....+.....+.+..+...+......+.+..+.+.....+.......+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..+.+...+.....+................+......+..+...+....+.........+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+......+...............+..+...............+...+...+....+...........+.+........+.............+.....+....+........+.......+......+............+..+....+.........+........+...+.......+...+.............................+..................+.........+......+.......+...+.....+....+...+.......................+...+.+...+.....+.+......+.........+.....+.+......+..+......+......+.+..............+.+.........+........+.+......+...+...+...+..+.............+.....+......+....+.....+.+...+...+.....+.......+.................+...+.+......+..............+.......+........+....+......+.....+......+....+......+........+...+.......+...+.................+.+..+.......+............+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6352023-10-22T04:50:39.995Z # For more on this and what to put here, see docs/how-to-run.adoc.
6362023-10-22T04:50:40.019Z.......+..........+......+.....+...+.+.....+....+..+...+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+..+.........+......+....+......+..+.+..+...+....+......+.....+......+.+..+.......+...............+...+..+.+...+.........+.....+...............+..........+...+..+...+....+...+...+...+...........+...+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+................+...............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6372023-10-22T04:50:40.047Z [[internal_services_ip_pool_ranges]]
6382023-10-22T04:50:40.072Z-----
6392023-10-22T04:50:40.096Z-first = "192.168.1.20"
6402023-10-22T04:50:40.121Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6412023-10-22T04:50:40.145Z-last = "192.168.1.29"
6422023-10-22T04:50:40.170Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6432023-10-22T04:50:40.194Z+first = "10.151.1.164"
6442023-10-22T04:50:40.219Z+ rmdir pkg
6452023-10-22T04:50:40.244Z+last = "10.151.1.173"
6462023-10-22T04:50:40.268Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6472023-10-22T04:50:40.293Z
6482023-10-22T04:50:40.318Z+ pfexec mkdir -p /usr/oxide
6492023-10-22T04:50:40.343Z # TODO - this configuration is subject to change going forward. Ultimately these
6502023-10-22T04:50:40.367Z+ pfexec rm -f /usr/oxide/tcpproxy
6512023-10-22T04:50:40.392Z # parameters should be provided to the control plane via wicket, but we need to
6522023-10-22T04:50:40.417Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6532023-10-22T04:50:40.442Z@@ -92,8 +92,8 @@
6542023-10-22T04:50:40.467Z+ pfexec chmod +x /usr/oxide/tcpproxy
6552023-10-22T04:50:40.491Z # A range of IP addresses used by Boundary Services on the external network. In
6562023-10-22T04:50:40.516Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6572023-10-22T04:50:40.540Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6582023-10-22T04:50:40.565Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6592023-10-22T04:50:40.589Z # With softnpu, only one address is used.
6602023-10-22T04:50:40.614Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6612023-10-22T04:50:40.638Z-infra_ip_first = "192.168.1.30"
6622023-10-22T04:50:40.662Z+ OMICRON_NO_UNINSTALL=1
6632023-10-22T04:50:40.687Z-infra_ip_last = "192.168.1.30"
6642023-10-22T04:50:40.711Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6652023-10-22T04:50:40.740Z+infra_ip_first = "10.151.1.174"
6662023-10-22T04:50:40.766ZOct 22 04:50:39.261 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6672023-10-22T04:50:40.794Z+infra_ip_last = "10.151.1.174"
6682023-10-22T04:50:40.822ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6692023-10-22T04:50:40.847Z
6702023-10-22T04:50:40.871ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6712023-10-22T04:50:40.896Z # Configurations for BGP routers to run on the scrimlets.
6722023-10-22T04:50:40.920ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6732023-10-22T04:50:40.948Z bgp = []
6742023-10-22T04:50:40.974ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6752023-10-22T04:50:40.998Z@@ -101,9 +101,9 @@
6762023-10-22T04:50:41.025ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6772023-10-22T04:50:41.074Z # You can configure multiple uplinks by repeating the following stanza
6782023-10-22T04:50:41.099ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6792023-10-22T04:50:41.123Z [[rack_network_config.ports]]
6802023-10-22T04:50:41.148ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6812023-10-22T04:50:41.173Z # Routes associated with this port.
6822023-10-22T04:50:41.197ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6832023-10-22T04:50:41.222Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6842023-10-22T04:50:41.246ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6852023-10-22T04:50:41.271Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6862023-10-22T04:50:41.296ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6872023-10-22T04:50:41.320Z # Addresses associated with this port.
6882023-10-22T04:50:41.345ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6892023-10-22T04:50:41.371Z-addresses = ["192.168.1.30/32"]
6902023-10-22T04:50:41.399ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6912023-10-22T04:50:41.424Z+addresses = ["10.151.1.174/32"]
6922023-10-22T04:50:41.475ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6932023-10-22T04:50:41.500Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6942023-10-22T04:50:41.533ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6952023-10-22T04:50:41.560Z port = "qsfp0"
6962023-10-22T04:50:41.584ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6972023-10-22T04:50:41.609Z # The speed of this port.
6982023-10-22T04:50:41.638ZOct 22 04:50:39.263 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6992023-10-22T04:50:41.664Za pkg/config-rss.toml 7K
7002023-10-22T04:50:41.689ZOct 22 04:50:39.484 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7012023-10-22T04:50:41.714Za pkg/initial-tls-cert.pem 2K
7022023-10-22T04:50:41.738ZOct 22 04:50:39.518 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7032023-10-22T04:50:41.763Za pkg/initial-tls-key.pem 4K
7042023-10-22T04:50:41.787ZOct 22 04:50:39.579 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7052023-10-22T04:50:41.812Z
7062023-10-22T04:50:41.837Zreal 0.462088190
7072023-10-22T04:50:41.862Zuser 0.098563266
7082023-10-22T04:50:41.886Zsys 1.855739699
7092023-10-22T04:50:41.911Ztrap 0.000001849
7102023-10-22T04:50:41.940Ztflt 0.000060330
7112023-10-22T04:50:41.965Zdflt 0.001370831
7122023-10-22T04:50:41.990Zkflt 0.000000000
7132023-10-22T04:50:42.014Zlock 41.273749946
7142023-10-22T04:50:42.039Zslp 0.682267336
7152023-10-22T04:50:42.063Zlat 0.003085533
7162023-10-22T04:50:42.099Zstop 0.216984528
7172023-10-22T04:50:42.125Z+ retry=0
7182023-10-22T04:50:42.149Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7192023-10-22T04:50:42.174Z+ [[ 0 -gt 30 ]]
7202023-10-22T04:50:42.204Z+ sleep 1
7212023-10-22T04:50:42.230Z+ retry=1
7222023-10-22T04:50:42.254Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232023-10-22T04:50:42.279Z+ [[ 1 -gt 30 ]]
7242023-10-22T04:50:42.303Z+ sleep 1
7252023-10-22T04:50:42.327Z+ retry=2
7262023-10-22T04:50:42.352Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272023-10-22T04:51:06.621Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7282023-10-22T04:51:06.646Zlocal v6:
7292023-10-22T04:51:06.670Zfd00:99::1
7302023-10-22T04:51:06.717Zfe80::e020:bfff:fef9:cb6e
7312023-10-22T04:51:06.742Zlocal v4:
7322023-10-22T04:51:06.766Z10.151.1.174
7332023-10-22T04:51:06.790Zrouter v6:
7342023-10-22T04:51:06.815Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::380f:1eff:fe4b:5efb (1) vid=0
7352023-10-22T04:51:06.839Zfd00:1122:3344:101::/64 -> fe80::380f:1eff:fe4b:5efb (1) vid=0
7362023-10-22T04:51:06.864Zrouter v4:
7372023-10-22T04:51:06.888Z0.0.0.0/0 -> 10.151.1.100 (2) vid=0
7382023-10-22T04:51:06.917Zresolver v4:
7392023-10-22T04:51:06.943Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7402023-10-22T04:51:06.967Z10.151.1.174 -> 42:ca:42:c8:b3:e0
7412023-10-22T04:51:06.991Zresolver v6:
7422023-10-22T04:51:07.016Zfe80::380f:1eff:fe4b:5efb -> 3a:0f:1e:4b:5e:fb
7432023-10-22T04:51:07.042Zfe80::e020:bfff:fef9:cb6e -> e2:20:bf:f9:cb:6e
7442023-10-22T04:51:07.068Znat v4:
7452023-10-22T04:51:07.097Znat_v6:
7462023-10-22T04:51:07.126Zport_mac:
7472023-10-22T04:51:07.152Z1000: a8:40:25:1c:e5:b4
7482023-10-22T04:51:07.177Z1: a8:40:25:1c:e5:b5
7492023-10-22T04:51:07.202Zproxy arp:
7502023-10-22T04:51:07.226Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7512023-10-22T04:51:07.250Z+ retry=0
7522023-10-22T04:51:07.274Z+++ zoneadm list -n
7532023-10-22T04:51:07.298Z+++ grep oxz_ntp
7542023-10-22T04:51:07.333Z++ pfexec svcs -z -Hostate oxide/ntp
7552023-10-22T04:51:07.358Zsvcs: zone '-Hostate': entity not found
7562023-10-22T04:51:07.382Z++ true
7572023-10-22T04:51:07.407Z+ [[ '' != online ]]
7582023-10-22T04:51:07.431Z+ [[ 0 -gt 60 ]]
7592023-10-22T04:51:07.455Z+ sleep 1
7602023-10-22T04:51:07.671Z+ retry=1
7612023-10-22T04:51:07.696Z+++ zoneadm list -n
7622023-10-22T04:51:07.743Z+++ grep oxz_ntp
7632023-10-22T04:51:07.768Z++ pfexec svcs -z -Hostate oxide/ntp
7642023-10-22T04:51:07.795Zsvcs: zone '-Hostate': entity not found
7652023-10-22T04:51:07.819Z++ true
7662023-10-22T04:51:08.254Z+ [[ '' != online ]]
7672023-10-22T04:51:08.283Z+ [[ 1 -gt 60 ]]
7682023-10-22T04:51:08.310Z+ sleep 1
7692023-10-22T04:51:08.694Z+ retry=2
7702023-10-22T04:51:08.726Z+++ zoneadm list -n
7712023-10-22T04:51:08.769Z+++ grep oxz_ntp
7722023-10-22T04:51:08.796Z++ pfexec svcs -z -Hostate oxide/ntp
7732023-10-22T04:51:08.821Zsvcs: zone '-Hostate': entity not found
7742023-10-22T04:51:08.846Z++ true
7752023-10-22T04:51:08.871Z+ [[ '' != online ]]
7762023-10-22T04:51:08.895Z+ [[ 2 -gt 60 ]]
7772023-10-22T04:51:08.920Z+ sleep 1
7782023-10-22T04:51:09.713Z+ retry=3
7792023-10-22T04:51:09.739Z+++ zoneadm list -n
7802023-10-22T04:51:09.764Z+++ grep oxz_ntp
7812023-10-22T04:51:09.794Z++ pfexec svcs -z -Hostate oxide/ntp
7822023-10-22T04:51:09.818Zsvcs: zone '-Hostate': entity not found
7832023-10-22T04:51:09.842Z++ true
7842023-10-22T04:51:09.867Z+ [[ '' != online ]]
7852023-10-22T04:51:09.891Z+ [[ 3 -gt 60 ]]
7862023-10-22T04:51:09.916Z+ sleep 1
7872023-10-22T04:51:10.736Z+ retry=4
7882023-10-22T04:51:10.761Z+++ zoneadm list -n
7892023-10-22T04:51:10.785Z+++ grep oxz_ntp
7902023-10-22T04:51:10.819Z++ pfexec svcs -z -Hostate oxide/ntp
7912023-10-22T04:51:10.843Zsvcs: zone '-Hostate': entity not found
7922023-10-22T04:51:10.868Z++ true
7932023-10-22T04:51:10.893Z+ [[ '' != online ]]
7942023-10-22T04:51:10.917Z+ [[ 4 -gt 60 ]]
7952023-10-22T04:51:10.942Z+ sleep 1
7962023-10-22T04:51:11.758Z+ retry=5
7972023-10-22T04:51:11.793Z+++ zoneadm list -n
7982023-10-22T04:51:11.818Z+++ grep oxz_ntp
7992023-10-22T04:51:11.845Z++ pfexec svcs -z -Hostate oxide/ntp
8002023-10-22T04:51:11.870Zsvcs: zone '-Hostate': entity not found
8012023-10-22T04:51:11.894Z++ true
8022023-10-22T04:51:11.919Z+ [[ '' != online ]]
8032023-10-22T04:51:11.943Z+ [[ 5 -gt 60 ]]
8042023-10-22T04:51:11.967Z+ sleep 1
8052023-10-22T04:51:12.781Z+ retry=6
8062023-10-22T04:51:12.806Z+++ zoneadm list -n
8072023-10-22T04:51:12.831Z+++ grep oxz_ntp
8082023-10-22T04:51:12.870Z++ pfexec svcs -z -Hostate oxide/ntp
8092023-10-22T04:51:13.303Zsvcs: zone '-Hostate': entity not found
8102023-10-22T04:51:13.331Z++ true
8112023-10-22T04:51:13.355Z+ [[ '' != online ]]
8122023-10-22T04:51:13.380Z+ [[ 6 -gt 60 ]]
8132023-10-22T04:51:13.405Z+ sleep 1
8142023-10-22T04:51:13.801Z+ retry=7
8152023-10-22T04:51:13.826Z+++ zoneadm list -n
8162023-10-22T04:51:13.850Z+++ grep oxz_ntp
8172023-10-22T04:51:13.898Z++ pfexec svcs -z -Hostate oxide/ntp
8182023-10-22T04:51:13.922Zsvcs: zone '-Hostate': entity not found
8192023-10-22T04:51:13.947Z++ true
8202023-10-22T04:51:13.971Z+ [[ '' != online ]]
8212023-10-22T04:51:13.996Z+ [[ 7 -gt 60 ]]
8222023-10-22T04:51:14.020Z+ sleep 1
8232023-10-22T04:51:14.823Z+ retry=8
8242023-10-22T04:51:14.848Z+++ zoneadm list -n
8252023-10-22T04:51:14.873Z+++ grep oxz_ntp
8262023-10-22T04:51:15.304Z++ pfexec svcs -z -Hostate oxide/ntp
8272023-10-22T04:51:15.333Zsvcs: zone '-Hostate': entity not found
8282023-10-22T04:51:15.358Z++ true
8292023-10-22T04:51:15.382Z+ [[ '' != online ]]
8302023-10-22T04:51:15.406Z+ [[ 8 -gt 60 ]]
8312023-10-22T04:51:15.430Z+ sleep 1
8322023-10-22T04:51:15.845Z+ retry=9
8332023-10-22T04:51:15.869Z+++ zoneadm list -n
8342023-10-22T04:51:15.894Z+++ grep oxz_ntp
8352023-10-22T04:51:15.918Z++ pfexec svcs -z -Hostate oxide/ntp
8362023-10-22T04:51:15.948Zsvcs: zone '-Hostate': entity not found
8372023-10-22T04:51:15.973Z++ true
8382023-10-22T04:51:15.997Z+ [[ '' != online ]]
8392023-10-22T04:51:16.021Z+ [[ 9 -gt 60 ]]
8402023-10-22T04:51:16.046Z+ sleep 1
8412023-10-22T04:51:16.867Z+ retry=10
8422023-10-22T04:51:16.893Z+++ zoneadm list -n
8432023-10-22T04:51:16.920Z+++ grep oxz_ntp
8442023-10-22T04:51:16.946Z++ pfexec svcs -z oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67 -Hostate oxide/ntp
8452023-10-22T04:51:16.981Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8462023-10-22T04:51:17.006Z++ true
8472023-10-22T04:51:17.031Z+ [[ '' != online ]]
8482023-10-22T04:51:17.055Z+ [[ 10 -gt 60 ]]
8492023-10-22T04:51:17.080Z+ sleep 1
8502023-10-22T04:51:17.905Z+ retry=11
8512023-10-22T04:51:18.365Z+++ zoneadm list -n
8522023-10-22T04:51:18.389Z+++ grep oxz_ntp
8532023-10-22T04:51:18.413Z++ pfexec svcs -z oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67 -Hostate oxide/ntp
8542023-10-22T04:51:18.438Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8552023-10-22T04:51:18.462Z++ true
8562023-10-22T04:51:18.486Z+ [[ '' != online ]]
8572023-10-22T04:51:18.511Z+ [[ 11 -gt 60 ]]
8582023-10-22T04:51:18.535Z+ sleep 1
8592023-10-22T04:51:18.927Z+ retry=12
8602023-10-22T04:51:18.951Z+++ zoneadm list -n
8612023-10-22T04:51:18.976Z+++ grep oxz_ntp
8622023-10-22T04:51:19.002Z++ pfexec svcs -z oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67 -Hostate oxide/ntp
8632023-10-22T04:51:19.032Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8642023-10-22T04:51:19.057Z++ true
8652023-10-22T04:51:19.081Z+ [[ '' != online ]]
8662023-10-22T04:51:19.106Z+ [[ 12 -gt 60 ]]
8672023-10-22T04:51:19.130Z+ sleep 1
8682023-10-22T04:51:19.955Z+ retry=13
8692023-10-22T04:51:19.980Z+++ zoneadm list -n
8702023-10-22T04:51:20.017Z+++ grep oxz_ntp
8712023-10-22T04:51:20.058Z++ pfexec svcs -z oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67 -Hostate oxide/ntp
8722023-10-22T04:51:20.096Z+ [[ disabled != online ]]
8732023-10-22T04:51:20.126Z+ [[ 13 -gt 60 ]]
8742023-10-22T04:51:20.151Z+ sleep 1
8752023-10-22T04:51:20.978Z+ retry=14
8762023-10-22T04:51:21.422Z+++ zoneadm list -n
8772023-10-22T04:51:21.458Z+++ grep oxz_ntp
8782023-10-22T04:51:21.558Z++ pfexec svcs -z oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67 -Hostate oxide/ntp
8792023-10-22T04:51:21.583Z+ [[ disabled != online ]]
8802023-10-22T04:51:21.608Z+ [[ 14 -gt 60 ]]
8812023-10-22T04:51:21.632Z+ sleep 1
8822023-10-22T04:51:21.999Z+ retry=15
8832023-10-22T04:51:22.023Z+++ zoneadm list -n
8842023-10-22T04:51:22.048Z+++ grep oxz_ntp
8852023-10-22T04:51:22.073Z++ pfexec svcs -z oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67 -Hostate oxide/ntp
8862023-10-22T04:51:22.121Z+ [[ disabled != online ]]
8872023-10-22T04:51:22.145Z+ [[ 15 -gt 60 ]]
8882023-10-22T04:51:22.170Z+ sleep 1
8892023-10-22T04:51:23.020Z+ retry=16
8902023-10-22T04:51:23.045Z+++ zoneadm list -n
8912023-10-22T04:51:23.069Z+++ grep oxz_ntp
8922023-10-22T04:51:23.094Z++ pfexec svcs -z oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67 -Hostate oxide/ntp
8932023-10-22T04:51:23.120Z+ [[ disabled != online ]]
8942023-10-22T04:51:23.153Z+ [[ 16 -gt 60 ]]
8952023-10-22T04:51:23.190Z+ sleep 1
8962023-10-22T04:51:24.043Z+ retry=17
8972023-10-22T04:51:24.068Z+++ zoneadm list -n
8982023-10-22T04:51:24.093Z+++ grep oxz_ntp
8992023-10-22T04:51:24.118Z++ pfexec svcs -z oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67 -Hostate oxide/ntp
9002023-10-22T04:51:24.149Z+ [[ disabled != online ]]
9012023-10-22T04:51:24.178Z+ [[ 17 -gt 60 ]]
9022023-10-22T04:51:24.206Z+ sleep 1
9032023-10-22T04:51:25.065Z+ retry=18
9042023-10-22T04:51:25.090Z+++ zoneadm list -n
9052023-10-22T04:51:25.115Z+++ grep oxz_ntp
9062023-10-22T04:51:25.141Z++ pfexec svcs -z oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67 -Hostate oxide/ntp
9072023-10-22T04:51:25.165Z+ [[ disabled != online ]]
9082023-10-22T04:51:25.202Z+ [[ 18 -gt 60 ]]
9092023-10-22T04:51:25.227Z+ sleep 1
9102023-10-22T04:51:26.086Z+ retry=19
9112023-10-22T04:51:26.110Z+++ zoneadm list -n
9122023-10-22T04:51:26.135Z+++ grep oxz_ntp
9132023-10-22T04:51:26.159Z++ pfexec svcs -z oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67 -Hostate oxide/ntp
9142023-10-22T04:51:26.184Z+ [[ disabled != online ]]
9152023-10-22T04:51:26.238Z+ [[ 19 -gt 60 ]]
9162023-10-22T04:51:26.266Z+ sleep 1
9172023-10-22T04:51:27.107Z+ retry=20
9182023-10-22T04:51:27.132Z+++ zoneadm list -n
9192023-10-22T04:51:27.157Z+++ grep oxz_ntp
9202023-10-22T04:51:27.181Z++ pfexec svcs -z oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67 -Hostate oxide/ntp
9212023-10-22T04:51:27.206Z+ [[ disabled != online ]]
9222023-10-22T04:51:27.230Z+ [[ 20 -gt 60 ]]
9232023-10-22T04:51:27.265Z+ sleep 1
 
9242023-10-22T04:51:27.759Zbuskin console login: Oct 22 04:51:27 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9252023-10-22T04:51:28.130Z+ retry=21
9262023-10-22T04:51:28.158Z+++ zoneadm list -n
9272023-10-22T04:51:28.184Z+++ grep oxz_ntp
9282023-10-22T04:51:28.208Z++ pfexec svcs -z oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67 -Hostate oxide/ntp
9292023-10-22T04:51:28.234Z+ [[ online != online ]]
9302023-10-22T04:51:28.293Z+ echo 'Waited for chrony: 21s'
9312023-10-22T04:51:28.325Z+ export RUST_BACKTRACE=1
9322023-10-22T04:51:28.351ZWaited for chrony: 21s
9332023-10-22T04:51:28.379Z+ RUST_BACKTRACE=1
9342023-10-22T04:51:28.403Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9352023-10-22T04:51:28.428Z+ ./tests/bootstrap
9362023-10-22T04:51:28.452ZUsing external DNS server at 10.151.1.164:53
9372023-10-22T04:51:28.477Z2023-10-22 04:51:28.477039584 UTC: attempting to log into API
9382023-10-22T04:51:43.216Z2023-10-22 04:51:43.257429285 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9392023-10-22T04:51:44.218Z2023-10-22 04:51:44.256453099 UTC: attempting to log into API
9402023-10-22T04:51:59.251Z2023-10-22 04:51:59.248412043 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9412023-10-22T04:52:00.253Z2023-10-22 04:52:00.246444275 UTC: attempting to log into API
9422023-10-22T04:52:15.285Z2023-10-22 04:52:15.273938854 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9432023-10-22T04:52:16.285Z2023-10-22 04:52:16.274722427 UTC: attempting to log into API
9442023-10-22T04:52:31.318Z2023-10-22 04:52:31.307333901 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9452023-10-22T04:52:32.609Z2023-10-22 04:52:32.308241814 UTC: attempting to log into API
9462023-10-22T04:52:47.352Z2023-10-22 04:52:47.340972190 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9472023-10-22T04:52:48.353Z2023-10-22 04:52:48.342113117 UTC: attempting to log into API
9482023-10-22T04:53:03.387Z2023-10-22 04:53:03.375253975 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9492023-10-22T04:53:04.387Z2023-10-22 04:53:04.376420512 UTC: attempting to log into API
9502023-10-22T04:53:19.422Z2023-10-22 04:53:19.410284309 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9512023-10-22T04:53:20.429Z2023-10-22 04:53:20.411793220 UTC: attempting to log into API
9522023-10-22T04:53:35.471Z2023-10-22 04:53:35.460175605 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9532023-10-22T04:53:36.476Z2023-10-22 04:53:36.461324952 UTC: attempting to log into API
9542023-10-22T04:54:05.480Z2023-10-22 04:53:51.494452890 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9552023-10-22T04:54:05.549Z2023-10-22 04:53:52.496417817 UTC: attempting to log into API
9562023-10-22T04:54:07.556Z2023-10-22 04:54:07.545190916 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9572023-10-22T04:54:08.041ZOct 22 04:54:07 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9582023-10-22T04:54:08.559Z2023-10-22 04:54:08.546372524 UTC: attempting to log into API
 
9592023-10-22T04:54:15.580ZOct 22 04:54:14 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9602023-10-22T04:54:22.844ZOct 22 04:54:22 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9612023-10-22T04:54:23.590Z2023-10-22 04:54:23.579395 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9622023-10-22T04:54:24.591Z2023-10-22 04:54:24.580569848 UTC: attempting to log into API
 
9632023-10-22T04:54:28.826ZOct 22 04:54:28 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
9642023-10-22T04:54:37.625ZOct 22 04:54:36 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9652023-10-22T04:54:39.627Z2023-10-22 04:54:39.614107877 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9662023-10-22T04:54:40.626Z2023-10-22 04:54:40.615193412 UTC: attempting to log into API
9672023-10-22T04:54:55.666Z2023-10-22 04:54:55.648782119 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9682023-10-22T04:54:56.660Z2023-10-22 04:54:56.649645102 UTC: attempting to log into API
9692023-10-22T04:54:57.459Z2023-10-22 04:54:57.442148039 UTC: login succeeded
9702023-10-22T04:54:57.492Zwaiting for nexus to come up...
9712023-10-22T04:54:57.552Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9722023-10-22T04:54:57.606Zensuring datasets are ready...
9732023-10-22T04:55:00.087Zlet's roll.
9742023-10-22T04:55:00.114Z+ rm ./tests/bootstrap
9752023-10-22T04:55:00.162Z+ for test_bin in tests/*
9762023-10-22T04:55:00.187Z+ ./tests/bootstrap-b4242ab00b4c43e7
9772023-10-22T04:55:00.212Z
9782023-10-22T04:55:00.243Zrunning 0 tests
9792023-10-22T04:55:00.736Z
9802023-10-22T04:55:01.188Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9812023-10-22T04:55:01.220Z
9822023-10-22T04:55:01.249Z+ for test_bin in tests/*
9832023-10-22T04:55:01.281Z+ ./tests/end_to_end_tests-0e54d5c810ee3c0c
9842023-10-22T04:55:01.308Z
9852023-10-22T04:55:01.334Zrunning 2 tests
9862023-10-22T04:55:01.362Ztest no_spoof::no_spoof_here ... ok
9872023-10-22T04:56:00.192Ztest instance_launch::instance_launch has been running for over 60 seconds
9882023-10-22T04:57:56.701Ztest instance_launch::instance_launch ... ok
9892023-10-22T04:57:56.726Z
9902023-10-22T04:57:56.754Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 176.51s
9912023-10-22T04:57:56.790Z
9922023-10-22T04:57:56.826Z+ _exit_trap
9932023-10-22T04:57:56.866Z+ local status=0
9942023-10-22T04:57:56.940Z+ [[ 0 -eq 0 ]]
9952023-10-22T04:57:56.966Z+ exit 0
9962023-10-22T04:57:56.995Zprocess exited: duration 468187 ms, exit code 0
 
9972023-10-22T04:57:57.082Zfound 47 output files
9982023-10-22T04:57:57.136Zuploading: /var/svc/log/oxide-sled-agent:default.log (472082 bytes)
9992023-10-22T04:57:58.823Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10002023-10-22T04:57:58.849Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_d3a188d1-2608-4133-ba19-cbf9824920bb/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10012023-10-22T04:57:59.931Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_d3a188d1-2608-4133-ba19-cbf9824920bb/root/var/svc/log/oxide-cockroachdb:default.log
10022023-10-22T04:57:59.956Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_9699e1ab-2ea6-4e45-8ef0-9fe00c7bf1ce/root/var/svc/log/oxide-external_dns:default.log (15608 bytes)
10032023-10-22T04:58:01.072Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_9699e1ab-2ea6-4e45-8ef0-9fe00c7bf1ce/root/var/svc/log/oxide-external_dns:default.log
10042023-10-22T04:58:01.100Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67/root/var/svc/log/oxide-ntp:default.log (984 bytes)
10052023-10-22T04:58:02.205Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67/root/var/svc/log/oxide-ntp:default.log
10062023-10-22T04:58:02.640Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2bd967e8-91a2-4a0f-a299-823cb2ac26ab/root/var/svc/log/oxide-cockroachdb:default.log (5344 bytes)
10072023-10-22T04:58:03.312Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2bd967e8-91a2-4a0f-a299-823cb2ac26ab/root/var/svc/log/oxide-cockroachdb:default.log
10082023-10-22T04:58:03.339Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3b6cb7b6-cfa6-48e2-912c-42c53ad579a8/root/var/svc/log/oxide-crucible-agent:default.log (24363 bytes)
10092023-10-22T04:58:04.863Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3b6cb7b6-cfa6-48e2-912c-42c53ad579a8/root/var/svc/log/oxide-crucible-agent:default.log
10102023-10-22T04:58:04.888Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3b6cb7b6-cfa6-48e2-912c-42c53ad579a8/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10112023-10-22T04:58:05.972Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3b6cb7b6-cfa6-48e2-912c-42c53ad579a8/root/var/svc/log/oxide-crucible-downstairs:default.log
10122023-10-22T04:58:07.236Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3b6cb7b6-cfa6-48e2-912c-42c53ad579a8/root/var/svc/log/oxide-crucible-downstairs:downstairs-1ff460e0-d542-4825-bdca-71c42f536542.log (3050 bytes)
10132023-10-22T04:58:07.331Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3b6cb7b6-cfa6-48e2-912c-42c53ad579a8/root/var/svc/log/oxide-crucible-downstairs:downstairs-1ff460e0-d542-4825-bdca-71c42f536542.log
10142023-10-22T04:58:07.355Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3b6cb7b6-cfa6-48e2-912c-42c53ad579a8/root/var/svc/log/oxide-crucible-downstairs:downstairs-ec52883d-878b-4214-89d2-da2a4c21b09d.log (5534 bytes)
10152023-10-22T04:58:08.173Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3b6cb7b6-cfa6-48e2-912c-42c53ad579a8/root/var/svc/log/oxide-crucible-downstairs:downstairs-ec52883d-878b-4214-89d2-da2a4c21b09d.log
10162023-10-22T04:58:08.199Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dce6a05f-aa88-45d0-bbc6-842dfcaa9d0b/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10172023-10-22T04:58:09.305Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dce6a05f-aa88-45d0-bbc6-842dfcaa9d0b/root/var/svc/log/oxide-crucible-agent:default.log
10182023-10-22T04:58:10.555Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dce6a05f-aa88-45d0-bbc6-842dfcaa9d0b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10192023-10-22T04:58:10.590Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dce6a05f-aa88-45d0-bbc6-842dfcaa9d0b/root/var/svc/log/oxide-crucible-downstairs:default.log
10202023-10-22T04:58:10.623Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_eb5434fc-3256-4c7a-b3c6-a0baf42f4631/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10212023-10-22T04:58:11.490Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_eb5434fc-3256-4c7a-b3c6-a0baf42f4631/root/var/svc/log/oxide-cockroachdb:default.log
10222023-10-22T04:58:11.515Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_efdf8ec2-cf4c-4fc4-9888-868910ea8454/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
10232023-10-22T04:58:12.626Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_efdf8ec2-cf4c-4fc4-9888-868910ea8454/root/var/svc/log/oxide-crucible-agent:default.log
10242023-10-22T04:58:12.683Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_efdf8ec2-cf4c-4fc4-9888-868910ea8454/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10252023-10-22T04:58:13.768Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_efdf8ec2-cf4c-4fc4-9888-868910ea8454/root/var/svc/log/oxide-crucible-downstairs:default.log
10262023-10-22T04:58:13.792Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_efdf8ec2-cf4c-4fc4-9888-868910ea8454/root/var/svc/log/oxide-crucible-downstairs:downstairs-14a9cfad-1caa-450a-9fb5-12bced5c9b72.log (3049 bytes)
10272023-10-22T04:58:14.867Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_efdf8ec2-cf4c-4fc4-9888-868910ea8454/root/var/svc/log/oxide-crucible-downstairs:downstairs-14a9cfad-1caa-450a-9fb5-12bced5c9b72.log
10282023-10-22T04:58:14.900Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_848c55b1-1b51-48a4-bfe4-d1b1206c457a/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10292023-10-22T04:58:15.985Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_848c55b1-1b51-48a4-bfe4-d1b1206c457a/root/var/svc/log/oxide-crucible-pantry:default.log
10302023-10-22T04:58:16.009Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_eaacdb52-0396-43d3-b34e-dda8d79caa48/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10312023-10-22T04:58:17.080Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_eaacdb52-0396-43d3-b34e-dda8d79caa48/root/var/svc/log/oxide-crucible-pantry:default.log
10322023-10-22T04:58:17.105Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_c9134466-294b-4a5a-8249-1be4b971f755/root/var/svc/log/oxide-external_dns:default.log (15611 bytes)
10332023-10-22T04:58:18.198Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_c9134466-294b-4a5a-8249-1be4b971f755/root/var/svc/log/oxide-external_dns:default.log
10342023-10-22T04:58:18.224Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_b3fc39fd-6cc5-456c-801d-356e173cd69a/root/var/svc/log/oxide-oximeter:default.log (176323 bytes)
10352023-10-22T04:58:18.545Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_b3fc39fd-6cc5-456c-801d-356e173cd69a/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 176323 -> 194521
10362023-10-22T04:58:19.626Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_b3fc39fd-6cc5-456c-801d-356e173cd69a/root/var/svc/log/oxide-oximeter:default.log
10372023-10-22T04:58:19.660Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d294aaaa-b24f-45c3-8142-7c0d7df956f4/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
10382023-10-22T04:58:20.731Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d294aaaa-b24f-45c3-8142-7c0d7df956f4/root/var/svc/log/oxide-crucible-agent:default.log
10392023-10-22T04:58:20.756Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d294aaaa-b24f-45c3-8142-7c0d7df956f4/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10402023-10-22T04:58:21.827Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d294aaaa-b24f-45c3-8142-7c0d7df956f4/root/var/svc/log/oxide-crucible-downstairs:default.log
10412023-10-22T04:58:22.267Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_b8533ab4-2715-4ca8-a53c-91be2e7e190a/root/var/svc/log/oxide-nexus:default.log (579889 bytes)
10422023-10-22T04:58:22.501Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_b8533ab4-2715-4ca8-a53c-91be2e7e190a/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 579889 -> 581713
10432023-10-22T04:58:23.585Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_b8533ab4-2715-4ca8-a53c-91be2e7e190a/root/var/svc/log/oxide-nexus:default.log
10442023-10-22T04:58:23.613Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d550706c-9ea2-48e0-9d64-e1352d6cd68d/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10452023-10-22T04:58:24.699Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d550706c-9ea2-48e0-9d64-e1352d6cd68d/root/var/svc/log/oxide-crucible-agent:default.log
10462023-10-22T04:58:24.746Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d550706c-9ea2-48e0-9d64-e1352d6cd68d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10472023-10-22T04:58:25.820Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d550706c-9ea2-48e0-9d64-e1352d6cd68d/root/var/svc/log/oxide-crucible-downstairs:default.log
10482023-10-22T04:58:25.874Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e2dda4d5-c0f5-4fec-b847-65a52b990635/root/var/svc/log/oxide-crucible-agent:default.log (14113 bytes)
10492023-10-22T04:58:26.975Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e2dda4d5-c0f5-4fec-b847-65a52b990635/root/var/svc/log/oxide-crucible-agent:default.log
10502023-10-22T04:58:27.002Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e2dda4d5-c0f5-4fec-b847-65a52b990635/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10512023-10-22T04:58:28.497Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e2dda4d5-c0f5-4fec-b847-65a52b990635/root/var/svc/log/oxide-crucible-downstairs:default.log
10522023-10-22T04:58:28.524Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e2dda4d5-c0f5-4fec-b847-65a52b990635/root/var/svc/log/oxide-crucible-downstairs:downstairs-6d727cd4-75fa-4c50-9d8f-d6ab3c8f381b.log (3049 bytes)
10532023-10-22T04:58:29.623Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e2dda4d5-c0f5-4fec-b847-65a52b990635/root/var/svc/log/oxide-crucible-downstairs:downstairs-6d727cd4-75fa-4c50-9d8f-d6ab3c8f381b.log
10542023-10-22T04:58:29.687Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_e030e0ea-53f0-49e2-b082-bc2982523e3e/root/var/svc/log/oxide-internal_dns:default.log (103538 bytes)
10552023-10-22T04:58:29.850Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_e030e0ea-53f0-49e2-b082-bc2982523e3e/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 103538 -> 104790
10562023-10-22T04:58:30.923Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_e030e0ea-53f0-49e2-b082-bc2982523e3e/root/var/svc/log/oxide-internal_dns:default.log
10572023-10-22T04:58:30.984Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_1e8938a5-eefb-4ee2-9096-5ace2d632318/root/var/svc/log/oxide-nexus:default.log (4722923 bytes)
10582023-10-22T04:58:37.587Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_1e8938a5-eefb-4ee2-9096-5ace2d632318/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4722923 -> 4783544
10592023-10-22T04:58:38.722Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_1e8938a5-eefb-4ee2-9096-5ace2d632318/root/var/svc/log/oxide-nexus:default.log
10602023-10-22T04:58:38.871Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_991b213d-1202-48fa-8f46-bac3ac4bfbd0/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10612023-10-22T04:58:56.593Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_991b213d-1202-48fa-8f46-bac3ac4bfbd0/root/var/svc/log/oxide-cockroachdb:default.log
10622023-10-22T04:58:56.626Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_e7b06f6f-472e-4555-b6d0-a7cd4cecf5f5/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10632023-10-22T04:58:57.679Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_e7b06f6f-472e-4555-b6d0-a7cd4cecf5f5/root/var/svc/log/oxide-cockroachdb:default.log
10642023-10-22T04:58:57.753Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_423f4206-7330-46ac-8a99-a92058e230da/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10652023-10-22T04:58:58.810Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_423f4206-7330-46ac-8a99-a92058e230da/root/var/svc/log/oxide-crucible-pantry:default.log
10662023-10-22T04:58:58.842Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_40dd9f52-dffc-4612-9fb3-7eb555f6c1e4/root/var/svc/log/oxide-internal_dns:default.log (99461 bytes)
10672023-10-22T04:58:58.974Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_40dd9f52-dffc-4612-9fb3-7eb555f6c1e4/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99461 -> 103215
10682023-10-22T04:59:00.052Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_40dd9f52-dffc-4612-9fb3-7eb555f6c1e4/root/var/svc/log/oxide-internal_dns:default.log
10692023-10-22T04:59:00.076Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8ecd06af-3dcf-4529-be66-b8f6c7a04656/root/var/svc/log/oxide-crucible-agent:default.log (14144 bytes)
10702023-10-22T04:59:01.170Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8ecd06af-3dcf-4529-be66-b8f6c7a04656/root/var/svc/log/oxide-crucible-agent:default.log
10712023-10-22T04:59:01.196Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8ecd06af-3dcf-4529-be66-b8f6c7a04656/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10722023-10-22T04:59:02.265Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8ecd06af-3dcf-4529-be66-b8f6c7a04656/root/var/svc/log/oxide-crucible-downstairs:default.log
10732023-10-22T04:59:02.290Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8ecd06af-3dcf-4529-be66-b8f6c7a04656/root/var/svc/log/oxide-crucible-downstairs:downstairs-65097696-1ee7-4001-bc3a-c3dab997c923.log (5534 bytes)
10742023-10-22T04:59:03.363Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8ecd06af-3dcf-4529-be66-b8f6c7a04656/root/var/svc/log/oxide-crucible-downstairs:downstairs-65097696-1ee7-4001-bc3a-c3dab997c923.log
10752023-10-22T04:59:03.392Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_da1ac6b7-70a3-4686-bb0e-657f31c0919d/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10762023-10-22T04:59:04.459Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_da1ac6b7-70a3-4686-bb0e-657f31c0919d/root/var/svc/log/oxide-crucible-agent:default.log
10772023-10-22T04:59:04.493Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_da1ac6b7-70a3-4686-bb0e-657f31c0919d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10782023-10-22T04:59:05.577Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_da1ac6b7-70a3-4686-bb0e-657f31c0919d/root/var/svc/log/oxide-crucible-downstairs:default.log
10792023-10-22T04:59:05.603Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_a7d5a270-a2ba-42d7-84bd-911419e001aa/root/var/svc/log/oxide-nexus:default.log (677506 bytes)
10802023-10-22T04:59:06.490Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_a7d5a270-a2ba-42d7-84bd-911419e001aa/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 677506 -> 739932
10812023-10-22T04:59:07.993Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_a7d5a270-a2ba-42d7-84bd-911419e001aa/root/var/svc/log/oxide-nexus:default.log
10822023-10-22T04:59:08.024Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_0eadb6c7-ceba-4f04-8dc9-72755a49bdc5/root/var/svc/log/oxide-clickhouse:default.log (648283 bytes)
10832023-10-22T04:59:08.686Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_0eadb6c7-ceba-4f04-8dc9-72755a49bdc5/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 648283 -> 877595
10842023-10-22T04:59:09.769Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_0eadb6c7-ceba-4f04-8dc9-72755a49bdc5/root/var/svc/log/oxide-clickhouse:default.log
10852023-10-22T04:59:09.795Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_81a3166e-f7dc-433e-8286-b44890b6611e/root/var/svc/log/oxide-crucible-agent:default.log (13690 bytes)
10862023-10-22T04:59:10.921Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_81a3166e-f7dc-433e-8286-b44890b6611e/root/var/svc/log/oxide-crucible-agent:default.log
10872023-10-22T04:59:10.953Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_81a3166e-f7dc-433e-8286-b44890b6611e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10882023-10-22T04:59:12.021Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_81a3166e-f7dc-433e-8286-b44890b6611e/root/var/svc/log/oxide-crucible-downstairs:default.log
10892023-10-22T04:59:12.073Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_81a3166e-f7dc-433e-8286-b44890b6611e/root/var/svc/log/oxide-crucible-downstairs:downstairs-34bbc7a0-098f-441d-a959-fe22a09f59fb.log (5535 bytes)
10902023-10-22T04:59:13.154Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_81a3166e-f7dc-433e-8286-b44890b6611e/root/var/svc/log/oxide-crucible-downstairs:downstairs-34bbc7a0-098f-441d-a959-fe22a09f59fb.log
10912023-10-22T04:59:13.182Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4e78f8d3-55ad-40d1-ac29-c2a29708984a/root/var/svc/log/oxide-internal_dns:default.log (101914 bytes)
10922023-10-22T04:59:13.324Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4e78f8d3-55ad-40d1-ac29-c2a29708984a/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 101914 -> 105671
10932023-10-22T04:59:14.395Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4e78f8d3-55ad-40d1-ac29-c2a29708984a/root/var/svc/log/oxide-internal_dns:default.log
10942023-10-22T04:59:14.420Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67/root/var/log/chrony/measurements.log (13015 bytes)
10952023-10-22T04:59:14.493Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67/root/var/log/chrony/measurements.log" changed size mid upload: 13015 -> 16440
10962023-10-22T04:59:15.573Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67/root/var/log/chrony/measurements.log
10972023-10-22T04:59:15.598Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67/root/var/log/chrony/statistics.log (8190 bytes)
10982023-10-22T04:59:15.647Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67/root/var/log/chrony/statistics.log" changed size mid upload: 8190 -> 10647
10992023-10-22T04:59:16.729Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67/root/var/log/chrony/statistics.log
11002023-10-22T04:59:17.971Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67/root/var/log/chrony/tracking.log (1721 bytes)
11012023-10-22T04:59:17.997Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67/root/var/log/chrony/tracking.log" changed size mid upload: 1721 -> 1985
11022023-10-22T04:59:18.259Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_19367231-93c4-4798-bb87-f16408a04b67/root/var/log/chrony/tracking.log