01HBYP8P2DPDGY860T9P2J7QQ2: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.23",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HBYPXDN64GMSC0NSZEGD9F6V

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-05T02:44:47.084Zjob dependencies complete; ready to run (waiting for 1 h 21 m 21 s)
22023-10-05T07:16:15.356Zjob assigned to worker 01HBZB2DPETJP4QA4M8KPBMZDP (queued for 4 h 31 m 28 s)
32023-10-05T07:16:15.357Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-10-05T07:16:15.474Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-10-05T07:16:15.534Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-10-05T07:16:15.568Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-05T07:16:15.601Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-10-05T07:16:15.635ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-05T07:16:15.668Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-05T07:16:15.702ZOxide Helios Version helios-2.0.22216 64-bit
112023-10-05T07:16:15.736ZConfiguring devices.
122023-10-05T07:16:15.769Z#################################################
132023-10-05T07:16:15.802Z#### ############# ##
142023-10-05T07:16:15.836Z### ### ############ ##
152023-10-05T07:16:15.869Z## ### # ## ### ## Oxide Computer Company ##
162023-10-05T07:16:15.903Z## ## ## ### # ### ##
172023-10-05T07:16:15.936Z## # ### #### #### This Station Under ##
182023-10-05T07:16:15.970Z### ### #### # ### Computer Control ##
192023-10-05T07:16:16.003Z#### #### ### ## ##
202023-10-05T07:16:16.037Z##################################################
212023-10-05T07:16:16.070ZHostname: sock
222023-10-05T07:16:16.103ZLoading smf(7) service descriptions:
232023-10-05T07:16:17.115Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-1edfa7f3d5f511cd.gz
242023-10-05T07:16:18.523Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-1edfa7f3d5f511cd.gz
252023-10-05T07:16:18.547Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-05T07:16:35.398Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-05T07:16:35.423Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-1e38f424764e177a.gz
282023-10-05T07:16:54.054Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-1e38f424764e177a.gz
292023-10-05T07:16:54.079Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-05T07:16:54.970Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-05T07:16:54.995Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-05T07:16:57.532Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-05T07:16:57.555Zdownloading input: /input/package/work/version.txt
342023-10-05T07:16:57.579Zdownloaded input: /input/package/work/version.txt
352023-10-05T07:16:57.603Zdownloading input: /input/package/work/package.tar.gz
362023-10-05T07:17:12.227Zdownloaded input: /input/package/work/package.tar.gz
372023-10-05T07:17:12.251Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-05T07:17:33.436Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-05T07:17:33.460Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-05T07:17:40.284Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-05T07:17:40.309Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-05T07:18:10.562Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-05T07:18:10.586Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-05T07:18:40.836Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-05T07:18:40.860Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-05T07:19:01.341Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-05T07:19:01.366Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-05T07:19:05.413Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-05T07:19:05.437Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-05T07:19:12.947Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-05T07:19:12.972Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-05T07:19:16.904Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-05T07:19:16.928Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-05T07:19:20.906Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-05T07:19:20.930Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-05T07:19:20.955Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-05T07:19:20.981Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-05T07:19:23.824Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-05T07:19:23.848Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-05T07:19:26.703Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-05T07:19:26.728Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-05T07:19:38.162Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-05T07:19:38.186Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-05T07:19:50.109Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-05T07:19:50.132Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-05T07:19:50.194Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-05T07:19:50.219Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-05T07:19:53.385Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-05T07:19:53.409Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-05T07:20:05.088Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-05T07:20:05.112Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-05T07:20:43.621Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-05T07:20:43.646Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-05T07:21:07.559Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-05T07:21:07.583Zstarting task 0: "setup"
762023-10-05T07:21:07.616Z++ uname -s
772023-10-05T07:21:07.640Z+ kern=SunOS
782023-10-05T07:21:07.664Z+ case "$kern" in
792023-10-05T07:21:07.688Z+ groupadd -g 12345 build
802023-10-05T07:21:07.712Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-05T07:21:09.625Z+ zfs create -o mountpoint=/work rpool/work
822023-10-05T07:21:09.671Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-05T07:21:09.695Z+ home_fs=zfs
842023-10-05T07:21:09.719Z+ [[ zfs == autofs ]]
852023-10-05T07:21:09.747Z+ mkdir -p /home/build
862023-10-05T07:21:09.773Z+ chown build:build /home/build /work
872023-10-05T07:21:11.678Z+ chmod 0700 /home/build /work
882023-10-05T07:21:11.702Zprocess exited: duration 4074 ms, exit code 0
 
892023-10-05T07:21:11.750Zstarting task 1: "authentication"
902023-10-05T07:21:11.819Zprocess exited: duration 44 ms, exit code 0
 
912023-10-05T07:21:11.866Zstarting task 2: "build"
922023-10-05T07:21:11.897Z+ trap _exit_trap EXIT
932023-10-05T07:21:11.921Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-05T07:21:12.094Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-05T07:21:12.146Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-05T07:21:12.171Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-05T07:21:12.196Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-05T07:21:12.221Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-05T07:21:12.246Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-05T07:21:12.271Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-05T07:21:12.296Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-05T07:21:12.326Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-05T07:21:12.351Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-05T07:21:12.375Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-05T07:21:12.400Z+ [[ -d /opt/oxide ]]
1062023-10-05T07:21:12.424Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-05T07:21:12.449Z+ pfexec mkdir /opt/oxide
1082023-10-05T07:21:12.473Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-05T07:21:12.498Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-05T07:21:12.523Z+ cd /opt/oxide-underneath
1112023-10-05T07:21:12.547Z+ pfexec tar ceEp@/f - .
1122023-10-05T07:21:12.572Z+ cd /opt/oxide
1132023-10-05T07:21:12.596Z+ pfexec tar xveEp@/f -
1142023-10-05T07:21:12.621Zx ., 0 bytes, 0 tape blocks
1152023-10-05T07:21:12.645Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-05T07:21:12.670Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-05T07:21:12.694Zx ./opte/bin/opteadm, 50138640 bytes, 97928 tape blocks
1182023-10-05T07:21:12.719Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-05T07:21:12.744Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-05T07:21:12.769Z+ pfexec mkdir /opt/oxide/work
1212023-10-05T07:21:12.793Z+ pfexec chown build:build /opt/oxide/work
1222023-10-05T07:21:12.818Z+ cd /opt/oxide/work
1232023-10-05T07:21:12.843Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-05T07:21:13.855ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-05T07:21:13.881Zx out/maghemite.tar, 74957312 bytes, 146401 tape blocks
1262023-10-05T07:21:13.907Zx out/omicron-sled-agent.tar, 113334272 bytes, 221356 tape blocks
1272023-10-05T07:21:13.932Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-05T07:21:13.956Zx out/npuzone/npuzone, 36941864 bytes, 72153 tape blocks
1292023-10-05T07:21:13.980Zx package-manifest.toml, 21303 bytes, 42 tape blocks
1302023-10-05T07:21:14.026Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-05T07:21:14.051Zx target/release/omicron-package, 37879296 bytes, 73983 tape blocks
1322023-10-05T07:21:14.077Zx tools/create_virtual_hardware.sh, 2810 bytes, 6 tape blocks
1332023-10-05T07:21:14.102Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-05T07:21:14.137Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-05T07:21:14.163Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-05T07:21:14.188Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-05T07:21:14.213Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-05T07:21:14.239Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-05T07:21:14.264Z
1402023-10-05T07:21:14.288Zreal 1.598974962
1412023-10-05T07:21:14.312Zuser 1.439941651
1422023-10-05T07:21:14.337Zsys 0.282348845
1432023-10-05T07:21:14.362Ztrap 0.000069882
1442023-10-05T07:21:14.386Ztflt 0.000000000
1452023-10-05T07:21:14.411Zdflt 0.000000000
1462023-10-05T07:21:14.436Zkflt 0.000000000
1472023-10-05T07:21:14.460Zlock 0.000000000
1482023-10-05T07:21:14.485Zslp 1.438799670
1492023-10-05T07:21:14.509Zlat 0.034669749
1502023-10-05T07:21:14.535Zstop 0.000126432
1512023-10-05T07:21:14.560Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-05T07:21:14.874Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-05T07:21:14.899Z+ mkdir tests
1542023-10-05T07:21:14.923Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-05T07:21:14.948Z+ ptime -m gunzip
1562023-10-05T07:21:14.973Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-1edfa7f3d5f511cd
1572023-10-05T07:21:15.050Z
1582023-10-05T07:21:15.075Zreal 0.139978148
1592023-10-05T07:21:15.100Zuser 0.129248081
1602023-10-05T07:21:15.126Zsys 0.010492413
1612023-10-05T07:21:15.151Ztrap 0.000011210
1622023-10-05T07:21:15.176Ztflt 0.000000000
1632023-10-05T07:21:15.200Zdflt 0.000000000
1642023-10-05T07:21:15.226Zkflt 0.000000000
1652023-10-05T07:21:15.251Zlock 0.000000000
1662023-10-05T07:21:15.289Zslp 0.000019760
1672023-10-05T07:21:15.313Zlat 0.000019809
1682023-10-05T07:21:15.337Zstop 0.000118152
1692023-10-05T07:21:15.362Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-1edfa7f3d5f511cd
1702023-10-05T07:21:15.386Z+ chmod a+x tests/bootstrap-1edfa7f3d5f511cd
1712023-10-05T07:21:15.411Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-05T07:21:15.435Z+ ptime -m gunzip
1732023-10-05T07:21:15.463Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-05T07:21:17.233Z
1752023-10-05T07:21:17.259Zreal 2.188226321
1762023-10-05T07:21:17.287Zuser 2.064936806
1772023-10-05T07:21:17.311Zsys 0.122938340
1782023-10-05T07:21:17.336Ztrap 0.000147141
1792023-10-05T07:21:17.360Ztflt 0.000000000
1802023-10-05T07:21:17.385Zdflt 0.000000000
1812023-10-05T07:21:17.410Zkflt 0.000000000
1822023-10-05T07:21:17.435Zlock 0.000000000
1832023-10-05T07:21:17.459Zslp 0.000011380
1842023-10-05T07:21:17.486Zlat 0.000017419
1852023-10-05T07:21:17.511Zstop 0.000117122
1862023-10-05T07:21:17.535Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-05T07:21:17.562Z+ chmod a+x tests/bootstrap
1882023-10-05T07:21:17.587Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-05T07:21:17.611Z+ ptime -m gunzip
1902023-10-05T07:21:17.636Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-1e38f424764e177a
1912023-10-05T07:21:19.797Z
1922023-10-05T07:21:19.822Zreal 2.551356466
1932023-10-05T07:21:19.847Zuser 2.390460457
1942023-10-05T07:21:19.871Zsys 0.160462953
1952023-10-05T07:21:19.896Ztrap 0.000203482
1962023-10-05T07:21:19.920Ztflt 0.000000000
1972023-10-05T07:21:19.945Zdflt 0.000000000
1982023-10-05T07:21:19.969Zkflt 0.000000000
1992023-10-05T07:21:19.993Zlock 0.000000000
2002023-10-05T07:21:20.018Zslp 0.000021710
2012023-10-05T07:21:20.042Zlat 0.000016530
2022023-10-05T07:21:20.066Zstop 0.000119821
2032023-10-05T07:21:20.091Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-1e38f424764e177a
2042023-10-05T07:21:20.116Z+ chmod a+x tests/end_to_end_tests-1e38f424764e177a
2052023-10-05T07:21:20.141Z+ bmat address ls
2062023-10-05T07:21:20.180ZNAME CIDR FIRST LAST COUNT
2072023-10-05T07:21:20.204Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-10-05T07:21:20.229Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-05T07:21:20.253Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-10-05T07:21:20.278Z+ EXTRA_IP_START=10.151.2.164
2112023-10-05T07:21:20.302Z+ EXTRA_IP_END=10.151.2.227
2122023-10-05T07:21:20.327Z+ EXTRA_IP_COUNT=64
2132023-10-05T07:21:20.351Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-10-05T07:21:20.376Z+ EXTRA_IP_BASE=10.151.2
2152023-10-05T07:21:20.400Z+ EXTRA_IP_FOCTET=164
2162023-10-05T07:21:20.425Z+ EXTRA_IP_LOCTET=227
2172023-10-05T07:21:20.450Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-10-05T07:21:20.474Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-10-05T07:21:20.499Z+ DNS_IP1=10.151.2.164
2202023-10-05T07:21:20.523Z+ DNS_IP2=10.151.2.165
2212023-10-05T07:21:20.548Z+ UPLINK_IP=10.151.2.174
2222023-10-05T07:21:20.573Z+ IPPOOL_START=10.151.2.175
2232023-10-05T07:21:20.597Z+ IPPOOL_END=10.151.2.227
2242023-10-05T07:21:20.621Z++ ipadm show-addr -po type,addr
2252023-10-05T07:21:20.646Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-10-05T07:21:20.671Z+ GATEWAY_IP=10.151.2.100
2272023-10-05T07:21:20.696Z+ [[ -n 10.151.2.100 ]]
2282023-10-05T07:21:20.721Z+ ping -s 10.151.2.100 56 1
2292023-10-05T07:21:20.751ZPING 10.151.2.100: 56 data bytes
2302023-10-05T07:21:20.776Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.010 ms
2312023-10-05T07:21:20.801Z
2322023-10-05T07:21:20.826Z----10.151.2.100 PING Statistics----
2332023-10-05T07:21:20.851Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-10-05T07:21:20.876Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2352023-10-05T07:21:20.900Z++ arp -an
2362023-10-05T07:21:20.925Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-10-05T07:21:20.949Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-10-05T07:21:20.973Z+ routeadm -e ipv4-forwarding -u
2392023-10-05T07:21:20.998Z+ PXA_START=10.151.2.164
2402023-10-05T07:21:21.022Z+ PXA_END=10.151.2.227
2412023-10-05T07:21:21.047Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-10-05T07:21:21.071Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-10-05T07:21:21.095Z+ ZPOOL_VDEV_DIR=/scratch
2442023-10-05T07:21:21.120Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-10-05T07:21:21.144Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-10-05T07:21:21.168Z++ cd ./tools
2472023-10-05T07:21:21.203Z++ pwd
2482023-10-05T07:21:21.228Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-10-05T07:21:21.253Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-10-05T07:21:21.278Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-10-05T07:21:21.302Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-10-05T07:21:21.327Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-10-05T07:21:21.351Z++ dladm show-phys -p -o LINK
2542023-10-05T07:21:21.376Z++ head -1
2552023-10-05T07:21:21.401Z+ PHYSICAL_LINK=igb0
2562023-10-05T07:21:21.425Z+ echo 'Using igb0 as physical link'
2572023-10-05T07:21:21.450ZUsing igb0 as physical link
2582023-10-05T07:21:21.474Z+ ensure_run_as_root
2592023-10-05T07:21:21.499Z++ id -u
2602023-10-05T07:21:21.524Z+ [[ 0 -ne 0 ]]
2612023-10-05T07:21:21.548Z+ ensure_zpools
2622023-10-05T07:21:21.573Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-10-05T07:21:21.598Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-10-05T07:21:21.622Z+ readarray -t ZPOOLS
2652023-10-05T07:21:21.647Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-10-05T07:21:21.671Z++ sed 's/[ ",]//g'
2672023-10-05T07:21:21.696Z+ for ZPOOL in "${ZPOOLS[@]}"
2682023-10-05T07:21:21.720Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2692023-10-05T07:21:21.745ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2702023-10-05T07:21:21.769Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2712023-10-05T07:21:21.798Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2722023-10-05T07:21:21.834Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2732023-10-05T07:21:21.877Z0+0 records in
2742023-10-05T07:21:21.912Z0+0 records out
2752023-10-05T07:21:21.952Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2762023-10-05T07:21:22.399ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2772023-10-05T07:21:22.434Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-10-05T07:21:22.477ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2792023-10-05T07:21:22.912Z+ set +x
2802023-10-05T07:21:22.948ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2812023-10-05T07:21:22.978Z++ zpool list -o name
2822023-10-05T07:21:23.034ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2832023-10-05T07:21:23.059Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2842023-10-05T07:21:23.085ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2852023-10-05T07:21:23.110Z+ [[ -z '' ]]
2862023-10-05T07:21:23.135ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2872023-10-05T07:21:23.159Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2882023-10-05T07:21:23.184ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2892023-10-05T07:21:23.208Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2902023-10-05T07:21:23.292ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2912023-10-05T07:21:23.316Z+ set +x
2922023-10-05T07:21:23.341ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2932023-10-05T07:21:23.365Z+ for ZPOOL in "${ZPOOLS[@]}"
2942023-10-05T07:21:23.395ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2952023-10-05T07:21:23.436Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2962023-10-05T07:21:23.471ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2972023-10-05T07:21:23.500Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2982023-10-05T07:21:23.528ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2992023-10-05T07:21:23.568Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3002023-10-05T07:21:23.596ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3012023-10-05T07:21:23.622Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3022023-10-05T07:21:23.646ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3032023-10-05T07:21:23.671Z0+0 records in
3042023-10-05T07:21:23.696ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3052023-10-05T07:21:23.721Z0+0 records out
3062023-10-05T07:21:23.746ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3072023-10-05T07:21:23.771Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3082023-10-05T07:21:23.796ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3092023-10-05T07:21:23.821Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3102023-10-05T07:21:23.846ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3112023-10-05T07:21:23.870Z+ set +x
3122023-10-05T07:21:23.895ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3132023-10-05T07:21:23.920Z++ zpool list -o name
3142023-10-05T07:21:23.944ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3152023-10-05T07:21:23.969Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3162023-10-05T07:21:23.994ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3172023-10-05T07:21:24.019Z+ [[ -z '' ]]
3182023-10-05T07:21:24.045ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3192023-10-05T07:21:24.070Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3202023-10-05T07:21:24.096ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3212023-10-05T07:21:24.123Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3222023-10-05T07:21:24.162ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3232023-10-05T07:21:24.189Z+ set +x
3242023-10-05T07:21:24.214ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3252023-10-05T07:21:24.240Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-10-05T07:21:24.266ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3272023-10-05T07:21:24.334Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3282023-10-05T07:21:24.362ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3292023-10-05T07:21:24.388Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3302023-10-05T07:21:24.415ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3312023-10-05T07:21:24.443Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3322023-10-05T07:21:24.470ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3332023-10-05T07:21:24.498Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3342023-10-05T07:21:24.525ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3352023-10-05T07:21:24.553Z0+0 records in
3362023-10-05T07:21:24.581ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3372023-10-05T07:21:24.609Z0+0 records out
3382023-10-05T07:21:24.640ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3392023-10-05T07:21:24.667Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3402023-10-05T07:21:24.698ZSimnet net0/sc0_0 exists
3412023-10-05T07:21:24.726Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3422023-10-05T07:21:24.754ZSimnet net1/sc1_0 exists
3432023-10-05T07:21:24.783Z+ set +x
3442023-10-05T07:21:24.817ZVnic sc0_1 exists
3452023-10-05T07:21:24.846Z++ zpool list -o name
3462023-10-05T07:21:24.878Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3472023-10-05T07:21:24.905Z+ [[ -z '' ]]
3482023-10-05T07:21:24.929Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3492023-10-05T07:21:24.954Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3502023-10-05T07:21:24.979Z+ set +x
3512023-10-05T07:21:25.005Z+ for ZPOOL in "${ZPOOLS[@]}"
3522023-10-05T07:21:25.030Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3532023-10-05T07:21:25.055Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3542023-10-05T07:21:25.080Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3552023-10-05T07:21:25.105Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3562023-10-05T07:21:25.129Z0+0 records in
3572023-10-05T07:21:25.154Z0+0 records out
3582023-10-05T07:21:25.179Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3592023-10-05T07:21:25.204Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3602023-10-05T07:21:25.228Z+ set +x
3612023-10-05T07:21:25.256Z++ zpool list -o name
3622023-10-05T07:21:25.280Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3632023-10-05T07:21:25.305Z+ [[ -z '' ]]
3642023-10-05T07:21:25.329Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3652023-10-05T07:21:25.370Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3662023-10-05T07:21:25.395Z+ set +x
3672023-10-05T07:21:25.422Z+ for ZPOOL in "${ZPOOLS[@]}"
3682023-10-05T07:21:25.446Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3692023-10-05T07:21:25.471Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3702023-10-05T07:21:25.498Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3712023-10-05T07:21:25.523Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3722023-10-05T07:21:25.548Z0+0 records in
3732023-10-05T07:21:25.573Z0+0 records out
3742023-10-05T07:21:25.597Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3752023-10-05T07:21:25.622Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3762023-10-05T07:21:25.647Z+ set +x
3772023-10-05T07:21:25.672Z++ zpool list -o name
3782023-10-05T07:21:25.697Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3792023-10-05T07:21:25.722Z+ [[ -z '' ]]
3802023-10-05T07:21:25.747Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3812023-10-05T07:21:25.772Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3822023-10-05T07:21:25.797Z+ set +x
3832023-10-05T07:21:25.821Z+ for ZPOOL in "${ZPOOLS[@]}"
3842023-10-05T07:21:25.846Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3852023-10-05T07:21:25.873Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3862023-10-05T07:21:25.898Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3872023-10-05T07:21:25.922Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3882023-10-05T07:21:25.948Z0+0 records in
3892023-10-05T07:21:25.972Z0+0 records out
3902023-10-05T07:21:25.997Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3912023-10-05T07:21:26.021Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3922023-10-05T07:21:26.046Z+ set +x
3932023-10-05T07:21:26.070Z++ zpool list -o name
3942023-10-05T07:21:26.095Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3952023-10-05T07:21:26.120Z+ [[ -z '' ]]
3962023-10-05T07:21:26.146Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3972023-10-05T07:21:26.170Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3982023-10-05T07:21:26.195Z+ set +x
3992023-10-05T07:21:26.219Z+ for ZPOOL in "${ZPOOLS[@]}"
4002023-10-05T07:21:26.245Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4012023-10-05T07:21:26.270Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4022023-10-05T07:21:26.294Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4032023-10-05T07:21:26.319Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4042023-10-05T07:21:26.344Z0+0 records in
4052023-10-05T07:21:26.368Z0+0 records out
4062023-10-05T07:21:26.394Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4072023-10-05T07:21:26.419Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4082023-10-05T07:21:26.443Z+ set +x
4092023-10-05T07:21:26.467Z++ zpool list -o name
4102023-10-05T07:21:26.493Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4112023-10-05T07:21:26.518Z+ [[ -z '' ]]
4122023-10-05T07:21:26.543Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4132023-10-05T07:21:26.567Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4142023-10-05T07:21:26.592Z+ set +x
4152023-10-05T07:21:26.618Z+ for ZPOOL in "${ZPOOLS[@]}"
4162023-10-05T07:21:26.643Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4172023-10-05T07:21:26.669Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4182023-10-05T07:21:26.693Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4192023-10-05T07:21:26.718Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4202023-10-05T07:21:26.742Z0+0 records in
4212023-10-05T07:21:26.767Z0+0 records out
4222023-10-05T07:21:26.791Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4232023-10-05T07:21:26.816Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4242023-10-05T07:21:26.840Z+ set +x
4252023-10-05T07:21:26.865Z++ zpool list -o name
4262023-10-05T07:21:26.889Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4272023-10-05T07:21:26.914Z+ [[ -z '' ]]
4282023-10-05T07:21:26.938Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4292023-10-05T07:21:26.963Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4302023-10-05T07:21:26.987Z+ set +x
4312023-10-05T07:21:27.012Z+ for ZPOOL in "${ZPOOLS[@]}"
4322023-10-05T07:21:27.047Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4332023-10-05T07:21:27.075Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4342023-10-05T07:21:27.101Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4352023-10-05T07:21:27.128Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4362023-10-05T07:21:27.190Z0+0 records in
4372023-10-05T07:21:27.221Z0+0 records out
4382023-10-05T07:21:27.248Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4392023-10-05T07:21:27.276Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4402023-10-05T07:21:27.301Z+ set +x
4412023-10-05T07:21:27.326Z++ zpool list -o name
4422023-10-05T07:21:27.353Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4432023-10-05T07:21:27.380Z+ [[ -z '' ]]
4442023-10-05T07:21:27.433Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4452023-10-05T07:21:27.467Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4462023-10-05T07:21:27.912Z+ set +x
4472023-10-05T07:21:28.349Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4482023-10-05T07:21:28.374Z+ readarray -t ZPOOLS
4492023-10-05T07:21:28.398Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4502023-10-05T07:21:28.423Z++ sed 's/[ ",]//g'
4512023-10-05T07:21:28.456Z+ for ZPOOL in "${ZPOOLS[@]}"
4522023-10-05T07:21:28.481Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4532023-10-05T07:21:28.505Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4542023-10-05T07:21:28.530Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4552023-10-05T07:21:28.555Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4562023-10-05T07:21:28.580Z0+0 records in
4572023-10-05T07:21:28.605Z0+0 records out
4582023-10-05T07:21:28.629Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4592023-10-05T07:21:28.653Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4602023-10-05T07:21:28.677Z+ set +x
4612023-10-05T07:21:28.701Z++ zpool list -o name
4622023-10-05T07:21:28.726Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4632023-10-05T07:21:28.751Z+ [[ -z '' ]]
4642023-10-05T07:21:29.199Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4652023-10-05T07:21:29.224Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4662023-10-05T07:21:29.251Z+ set +x
4672023-10-05T07:21:29.275Z+ for ZPOOL in "${ZPOOLS[@]}"
4682023-10-05T07:21:29.300Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4692023-10-05T07:21:29.324Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4702023-10-05T07:21:29.348Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4712023-10-05T07:21:29.373Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4722023-10-05T07:21:29.397Z0+0 records in
4732023-10-05T07:21:29.422Z0+0 records out
4742023-10-05T07:21:29.446Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4752023-10-05T07:21:29.479Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4762023-10-05T07:21:29.503Z+ set +x
4772023-10-05T07:21:29.528Z++ zpool list -o name
4782023-10-05T07:21:29.552Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4792023-10-05T07:21:29.577Z+ [[ -z '' ]]
4802023-10-05T07:21:29.601Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4812023-10-05T07:21:29.626Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4822023-10-05T07:21:29.677Z+ set +x
4832023-10-05T07:21:29.713Z+ ensure_simulated_links igb0
4842023-10-05T07:21:29.737Z+ local PHYSICAL_LINK=igb0
4852023-10-05T07:21:29.762Z+ INDICES=("0" "1")
4862023-10-05T07:21:29.786Zdownloading softnpu ...done
4872023-10-05T07:21:29.811Z+ for I in "${INDICES[@]}"
4882023-10-05T07:21:29.836Z++ get_simnet_name_if_exists net0
4892023-10-05T07:21:29.860Z++ dladm show-simnet -p -o LINK net0
4902023-10-05T07:21:29.884Z++ echo ''
4912023-10-05T07:21:29.909Z+ [[ -z '' ]]
4922023-10-05T07:21:29.933Z+ dladm create-simnet -t net0
4932023-10-05T07:21:29.957Z+ dladm create-simnet -t sc0_0
4942023-10-05T07:21:29.982Z+ dladm modify-simnet -t -p net0 sc0_0
4952023-10-05T07:21:30.006Z+ dladm set-linkprop -p mtu=1600 sc0_0
4962023-10-05T07:21:30.031Z+ success 'Simnet net0/sc0_0 exists'
4972023-10-05T07:21:30.056Z+ set +x
4982023-10-05T07:21:30.081Z+ for I in "${INDICES[@]}"
4992023-10-05T07:21:30.117Z++ get_simnet_name_if_exists net1
5002023-10-05T07:21:30.592Z++ dladm show-simnet -p -o LINK net1
5012023-10-05T07:21:30.616Z++ echo ''
5022023-10-05T07:21:30.641Z+ [[ -z '' ]]
5032023-10-05T07:21:30.665Z+ dladm create-simnet -t net1
5042023-10-05T07:21:30.690Z+ dladm create-simnet -t sc1_0
5052023-10-05T07:21:30.714Z+ dladm modify-simnet -t -p net1 sc1_0
5062023-10-05T07:21:30.739Z+ dladm set-linkprop -p mtu=1600 sc1_0
5072023-10-05T07:21:30.764Z+ success 'Simnet net1/sc1_0 exists'
5082023-10-05T07:21:30.788Z+ set +x
5092023-10-05T07:21:30.813Z++ get_vnic_name_if_exists sc0_1
5102023-10-05T07:21:30.838Z++ dladm show-vnic -p -o LINK sc0_1
5112023-10-05T07:21:30.863Z++ echo ''
5122023-10-05T07:21:30.888Z+ [[ -z '' ]]
5132023-10-05T07:21:30.913Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-10-05T07:21:31.358Z+ success 'Vnic sc0_1 exists'
5152023-10-05T07:21:31.799Z+ set +x
5162023-10-05T07:21:32.249Z+ warn_if_no_proxy_arp
5172023-10-05T07:21:32.688Z+ [[ -v PXA_START ]]
5182023-10-05T07:21:32.713Z+ [[ -v PXA_END ]]
5192023-10-05T07:21:32.738Z+ ensure_softnpu_zone
5202023-10-05T07:21:32.762Z+ zoneadm list
5212023-10-05T07:21:32.786Z+ grep -q sidecar_softnpu
5222023-10-05T07:21:32.811Z+ [[ -f out/npuzone/npuzone ]]
5232023-10-05T07:21:32.835Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0
5242023-10-05T07:21:33.874Zdownloading asic_program.so ...done
5252023-10-05T07:21:39.162Zdownloading scadm ...done
5262023-10-05T07:21:39.187Zcreating port tfportrear0_0 for sc0_0
5272023-10-05T07:21:39.211Zcreating port tfportqsfp0_0 for sc0_1
5282023-10-05T07:21:39.235Zsoftnpu zone setup
5292023-10-05T07:21:39.261Zconfigure zone
5302023-10-05T07:21:39.286Zinstall zone
5312023-10-05T07:21:40.264Zboot zone
5322023-10-05T07:21:43.369Z[sidecar_softnpu] svcs milestone/network
5332023-10-05T07:21:43.395ZSTATE STIME FMRI
5342023-10-05T07:21:43.420Zonline 7:21:41 svc:/milestone/network:default
5352023-10-05T07:21:43.444Z[sidecar_softnpu] /softnpu/npu start
5362023-10-05T07:21:43.468Z
5372023-10-05T07:21:43.493Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5382023-10-05T07:21:43.518Z+ GATEWAY_IP=10.151.2.100
5392023-10-05T07:21:43.543Z+ echo 'Using 10.151.2.100 as gateway ip'
5402023-10-05T07:21:43.567ZUsing 10.151.2.100 as gateway ip
5412023-10-05T07:21:43.591Z+ [[ ! -v GATEWAY_MAC ]]
5422023-10-05T07:21:43.616Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5432023-10-05T07:21:43.640Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5442023-10-05T07:21:43.668Z+++ echo 18:c0:4d:0d:a0:2a
5452023-10-05T07:21:43.692Z+ COUNT=1
5462023-10-05T07:21:43.716Z+ [[ 1 -ne 1 ]]
5472023-10-05T07:21:43.741Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5482023-10-05T07:21:43.765ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5492023-10-05T07:21:43.790Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5502023-10-05T07:21:43.815Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5512023-10-05T07:21:43.846Z+ PXA_MAC=a8:e1:de:01:70:1d
5522023-10-05T07:21:43.872Z+ [[ -v PXA_START ]]
5532023-10-05T07:21:43.896Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5542023-10-05T07:21:43.921Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5552023-10-05T07:21:43.945Z+ z_scadm dump-state
5562023-10-05T07:21:43.969Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5572023-10-05T07:21:43.993Zlocal v6:
5582023-10-05T07:21:44.018Zlocal v4:
5592023-10-05T07:21:44.042Zrouter v6:
5602023-10-05T07:21:44.066Zrouter v4:
5612023-10-05T07:21:44.090Zresolver v4:
5622023-10-05T07:21:44.115Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5632023-10-05T07:21:44.139Zresolver v6:
5642023-10-05T07:21:44.163Znat v4:
5652023-10-05T07:21:44.187Znat_v6:
5662023-10-05T07:21:44.212Zport_mac:
5672023-10-05T07:21:44.237Zproxy arp:
5682023-10-05T07:21:44.261Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5692023-10-05T07:21:44.286Z+ success 'softnpu zone exists'
5702023-10-05T07:21:44.310Z+ set +x
5712023-10-05T07:21:44.335Zsoftnpu zone exists
5722023-10-05T07:21:44.359Z
5732023-10-05T07:21:44.384Zreal 23.365427339
5742023-10-05T07:21:44.408Zuser 0.716561421
5752023-10-05T07:21:44.433Zsys 1.641886885
5762023-10-05T07:21:44.458Ztrap 0.000064058
5772023-10-05T07:21:44.482Ztflt 0.000154777
5782023-10-05T07:21:44.507Zdflt 0.001170938
5792023-10-05T07:21:44.531Zkflt 0.000000000
5802023-10-05T07:21:44.556Zlock 18:12.634217422
5812023-10-05T07:21:44.580Zslp 1:09.598212636
5822023-10-05T07:21:44.605Zlat 0.078990174
5832023-10-05T07:21:44.630Zstop 0.369314381
5842023-10-05T07:21:44.656Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5852023-10-05T07:21:44.681Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5862023-10-05T07:21:44.706Z+ SILO_NAME=recovery
5872023-10-05T07:21:44.730Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5882023-10-05T07:21:44.755Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5892023-10-05T07:21:44.779Z+ sed -i~ '
5902023-10-05T07:21:44.804Z /^external_dns_ips/c\
5912023-10-05T07:21:44.829Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5922023-10-05T07:21:44.869Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5932023-10-05T07:21:44.894Z /^first/c\
5942023-10-05T07:21:44.919Zfirst = "10.151.2.164"
5952023-10-05T07:21:44.944Z /^last/c\
5962023-10-05T07:21:44.968Zlast = "10.151.2.173"
5972023-10-05T07:21:44.993Z }
5982023-10-05T07:21:45.023Z /^\[rack_network_config/,/^$/ {
5992023-10-05T07:21:45.049Z /^infra_ip_first/c\
6002023-10-05T07:21:45.074Zinfra_ip_first = "10.151.2.174"
6012023-10-05T07:21:45.098Z /^infra_ip_last/c\
6022023-10-05T07:21:45.123Zinfra_ip_last = "10.151.2.174"
6032023-10-05T07:21:45.147Z }
6042023-10-05T07:21:45.171Z /^\[\[rack_network_config.uplinks/,/^$/ {
6052023-10-05T07:21:45.196Z /^gateway_ip/c\
6062023-10-05T07:21:45.220Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6072023-10-05T07:21:45.245Zgateway_ip = "10.151.2.100"
6082023-10-05T07:21:45.269Z+++ pkg/config-rss.toml Thu Oct 5 07:21:43 2023
6092023-10-05T07:21:45.293Z /^uplink_cidr/c\
6102023-10-05T07:21:45.317Z@@ -41,7 +41,7 @@
6112023-10-05T07:21:45.342Zuplink_cidr = "10.151.2.174/32"
6122023-10-05T07:21:45.366Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6132023-10-05T07:21:45.391Z }
6142023-10-05T07:21:45.416Z # must be contained in one of the "internal services" IP Pool ranges listed
6152023-10-05T07:21:45.441Z' pkg/config-rss.toml
6162023-10-05T07:21:45.466Z # below.
6172023-10-05T07:21:45.490Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6182023-10-05T07:21:45.515Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6192023-10-05T07:21:45.540Z+ true
6202023-10-05T07:21:45.564Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6212023-10-05T07:21:45.589Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6222023-10-05T07:21:45.613Z
6232023-10-05T07:21:45.638Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6242023-10-05T07:21:45.663Z # Initial TLS certificates for the external API
6252023-10-05T07:21:45.688Z+ TLS_NAME=recovery.sys.oxide.test
6262023-10-05T07:21:45.712Z #
6272023-10-05T07:21:45.736Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6282023-10-05T07:21:45.763Z@@ -77,8 +77,8 @@
6292023-10-05T07:21:45.789Z.+........+.......+...+.....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+.......+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.....+.........+................+...+..+....+....................+....+.....+.+..............+...+..........+.....+...+....+..+......+..........+..............+...+..........+......+..+.........+...............+.+...+..+....+........+...+...+.......+...+.........+...+..+.+..............+..........+...........+.........+.......+......+..+......+............+.......+......+.........+...+..+..........+..+......+....+........+.+..+.........+.........+.+........+.+.....+....................................+...+...+....+.....+.....................+..................+.+.........+...........+.+.....+...+.+...+......+...........+.+.........+.........+.................+....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6302023-10-05T07:21:45.814Z #
6312023-10-05T07:21:45.839Z # For more on this and what to put here, see docs/how-to-run.adoc.
6322023-10-05T07:21:45.864Z [[internal_services_ip_pool_ranges]]
6332023-10-05T07:21:45.893Z-first = "192.168.1.20"
6342023-10-05T07:21:45.917Z-last = "192.168.1.29"
6352023-10-05T07:21:45.942Z+first = "10.151.2.164"
6362023-10-05T07:21:45.967Z+last = "10.151.2.173"
6372023-10-05T07:21:45.991Z
6382023-10-05T07:21:46.016Z # TODO - this configuration is subject to change going forward. Ultimately these
6392023-10-05T07:21:46.040Z.....+.+...........+.......+...+..+.+......+........+.+..+....+.....+..........+...+...+...+.........+..+......+.......+..+...+.+..+....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+..+..........+...........+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+...............+..+..........+.........+..............+.............+..+............+.+.....+.........+.+...+..+.........+.+.....+............+.+.....+....+...................................+.+......+...+..+...+..........+...............+..+......+............+.+.....+......................+...+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6402023-10-05T07:21:46.064Z # parameters should be provided to the control plane via wicket, but we need to
6412023-10-05T07:21:46.089Z-----
6422023-10-05T07:21:46.113Z@@ -91,13 +91,13 @@
6432023-10-05T07:21:46.137Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6442023-10-05T07:21:46.162Z # A range of IP addresses used by Boundary Services on the external network. In
6452023-10-05T07:21:46.186Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6462023-10-05T07:21:46.210Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6472023-10-05T07:21:46.235Z+ rmdir pkg
6482023-10-05T07:21:46.260Z # With softnpu, only one address is used.
6492023-10-05T07:21:46.284Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6502023-10-05T07:21:46.309Z-infra_ip_first = "192.168.1.30"
6512023-10-05T07:21:46.334Z+ pfexec mkdir -p /usr/oxide
6522023-10-05T07:21:46.359Z-infra_ip_last = "192.168.1.30"
6532023-10-05T07:21:46.384Z+ pfexec rm -f /usr/oxide/tcpproxy
6542023-10-05T07:21:46.408Z+infra_ip_first = "10.151.2.174"
6552023-10-05T07:21:46.433Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6562023-10-05T07:21:46.459Z+infra_ip_last = "10.151.2.174"
6572023-10-05T07:21:46.483Z+ pfexec chmod +x /usr/oxide/tcpproxy
6582023-10-05T07:21:46.508Z
6592023-10-05T07:21:46.533Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6602023-10-05T07:21:46.558Z # You can configure multiple uplinks by repeating the following stanza
6612023-10-05T07:21:46.582Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6622023-10-05T07:21:46.607Z [[rack_network_config.uplinks]]
6632023-10-05T07:21:46.632Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6642023-10-05T07:21:46.657Z # The gateway IP for the rack's external network
6652023-10-05T07:21:46.681Z+ OMICRON_NO_UNINSTALL=1
6662023-10-05T07:21:46.706Z-gateway_ip = "192.168.1.199"
6672023-10-05T07:21:46.731Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6682023-10-05T07:21:46.756Z+gateway_ip = "10.151.2.100"
6692023-10-05T07:21:46.786ZOct 05 07:21:44.995 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6702023-10-05T07:21:46.810Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6712023-10-05T07:21:46.835ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6722023-10-05T07:21:46.860Z uplink_port = "qsfp0"
6732023-10-05T07:21:46.890ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6742023-10-05T07:21:46.924Z uplink_port_speed = "40G"
6752023-10-05T07:21:46.949ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6762023-10-05T07:21:46.974Z@@ -104,7 +104,7 @@
6772023-10-05T07:21:46.998ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6782023-10-05T07:21:47.023Z uplink_port_fec="none"
6792023-10-05T07:21:47.047ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6802023-10-05T07:21:47.072Z # For softnpu, an address within the "infra" block above that will be used for
6812023-10-05T07:21:47.097ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6822023-10-05T07:21:47.148Z # the softnpu uplink port. You can just pick the first address in that pool.
6832023-10-05T07:21:47.172ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6842023-10-05T07:21:47.197Z-uplink_cidr = "192.168.1.30/32"
6852023-10-05T07:21:47.222ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6862023-10-05T07:21:47.247Z+uplink_cidr = "10.151.2.174/32"
6872023-10-05T07:21:47.271ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6882023-10-05T07:21:47.295Z # Switch to use for the uplink. For single-rack deployments this can be
6892023-10-05T07:21:47.319ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6902023-10-05T07:21:47.344Z # "switch0" (upper slot) or "switch1" (lower slot). For single-node softnpu
6912023-10-05T07:21:47.368ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/maghemite.tar
6922023-10-05T07:21:47.393Z # and dendrite stub environments, use "switch0"
6932023-10-05T07:21:47.418ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6942023-10-05T07:21:47.442Za pkg/config-rss.toml 7K
6952023-10-05T07:21:47.467ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6962023-10-05T07:21:47.492Za pkg/initial-tls-cert.pem 2K
6972023-10-05T07:21:47.517ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6982023-10-05T07:21:47.541Za pkg/initial-tls-key.pem 4K
6992023-10-05T07:21:47.566ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7002023-10-05T07:21:47.590ZOct 05 07:21:44.997 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7012023-10-05T07:21:47.614ZOct 05 07:21:45.209 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7022023-10-05T07:21:47.639ZOct 05 07:21:45.250 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7032023-10-05T07:21:47.663ZOct 05 07:21:45.309 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7042023-10-05T07:21:47.688Z
7052023-10-05T07:21:47.712Zreal 0.449161519
7062023-10-05T07:21:47.736Zuser 0.094919289
7072023-10-05T07:21:47.761Zsys 1.773710651
7082023-10-05T07:21:47.785Ztrap 0.000001899
7092023-10-05T07:21:47.810Ztflt 0.000029388
7102023-10-05T07:21:47.834Zdflt 0.000386815
7112023-10-05T07:21:47.859Zkflt 0.000003430
7122023-10-05T07:21:47.885Zlock 40.265918500
7132023-10-05T07:21:47.909Zslp 0.647120353
7142023-10-05T07:21:47.948Zlat 0.002837312
7152023-10-05T07:21:47.973Zstop 0.200771856
7162023-10-05T07:21:47.997Z+ retry=0
7172023-10-05T07:21:48.026Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7182023-10-05T07:21:48.051Z+ [[ 0 -gt 30 ]]
7192023-10-05T07:21:48.076Z+ sleep 1
7202023-10-05T07:21:48.101Z+ retry=1
7212023-10-05T07:21:48.125Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-10-05T07:21:48.150Z+ [[ 1 -gt 30 ]]
7232023-10-05T07:21:48.175Z+ sleep 1
7242023-10-05T07:21:48.199Z+ retry=2
7252023-10-05T07:21:48.224Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-10-05T07:22:06.953Z+ [[ 2 -gt 30 ]]
7272023-10-05T07:22:06.978Z+ sleep 1
7282023-10-05T07:22:07.956Z+ retry=3
7292023-10-05T07:22:07.981Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-10-05T07:22:08.005Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7312023-10-05T07:22:08.030Zlocal v6:
7322023-10-05T07:22:08.054Zfe80::f0:b7ff:fee9:17c9
7332023-10-05T07:22:08.079Zlocal v4:
7342023-10-05T07:22:08.103Zrouter v6:
7352023-10-05T07:22:08.127Zrouter v4:
7362023-10-05T07:22:08.151Zresolver v4:
7372023-10-05T07:22:08.176Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7382023-10-05T07:22:08.200Zresolver v6:
7392023-10-05T07:22:08.225Zfe80::2c3d:84ff:fe08:59be -> 2e:3d:84:08:59:be
7402023-10-05T07:22:08.249Zfe80::f0:b7ff:fee9:17c9 -> 02:f0:b7:e9:17:c9
7412023-10-05T07:22:08.274Znat v4:
7422023-10-05T07:22:08.298Znat_v6:
7432023-10-05T07:22:08.324Zport_mac:
7442023-10-05T07:22:08.348Z1: a8:40:25:59:2b:55
7452023-10-05T07:22:08.401Z1000: a8:40:25:59:2b:54
7462023-10-05T07:22:08.427Zproxy arp:
7472023-10-05T07:22:08.463Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7482023-10-05T07:22:08.488Z+ retry=0
7492023-10-05T07:22:08.513Z+++ zoneadm list -n
7502023-10-05T07:22:08.537Z+++ grep oxz_ntp
7512023-10-05T07:22:08.561Z++ pfexec svcs -z -Hostate oxide/ntp
7522023-10-05T07:22:08.586Zsvcs: zone '-Hostate': entity not found
7532023-10-05T07:22:08.643Z++ true
7542023-10-05T07:22:08.667Z+ [[ '' != online ]]
7552023-10-05T07:22:08.692Z+ [[ 0 -gt 60 ]]
7562023-10-05T07:22:08.717Z+ sleep 1
7572023-10-05T07:22:09.056Z+ retry=1
7582023-10-05T07:22:09.081Z+++ zoneadm list -n
7592023-10-05T07:22:09.127Z+++ grep oxz_ntp
7602023-10-05T07:22:09.153Z++ pfexec svcs -z -Hostate oxide/ntp
7612023-10-05T07:22:09.178Zsvcs: zone '-Hostate': entity not found
7622023-10-05T07:22:09.203Z++ true
7632023-10-05T07:22:09.229Z+ [[ '' != online ]]
7642023-10-05T07:22:09.253Z+ [[ 1 -gt 60 ]]
7652023-10-05T07:22:09.277Z+ sleep 1
7662023-10-05T07:22:10.040Z+ retry=2
7672023-10-05T07:22:10.065Z+++ zoneadm list -n
7682023-10-05T07:22:10.090Z+++ grep oxz_ntp
7692023-10-05T07:22:10.115Z++ pfexec svcs -z -Hostate oxide/ntp
7702023-10-05T07:22:10.140Zsvcs: zone '-Hostate': entity not found
7712023-10-05T07:22:10.203Z++ true
7722023-10-05T07:22:10.228Z+ [[ '' != online ]]
7732023-10-05T07:22:10.253Z+ [[ 2 -gt 60 ]]
7742023-10-05T07:22:10.316Z+ sleep 1
7752023-10-05T07:22:11.061Z+ retry=3
7762023-10-05T07:22:11.114Z+++ zoneadm list -n
7772023-10-05T07:22:11.140Z+++ grep oxz_ntp
7782023-10-05T07:22:11.165Z++ pfexec svcs -z -Hostate oxide/ntp
7792023-10-05T07:22:11.210Zsvcs: zone '-Hostate': entity not found
7802023-10-05T07:22:11.235Z++ true
7812023-10-05T07:22:11.281Z+ [[ '' != online ]]
7822023-10-05T07:22:11.306Z+ [[ 3 -gt 60 ]]
7832023-10-05T07:22:11.331Z+ sleep 1
7842023-10-05T07:22:12.103Z+ retry=4
7852023-10-05T07:22:12.167Z+++ zoneadm list -n
7862023-10-05T07:22:12.237Z+++ grep oxz_ntp
7872023-10-05T07:22:12.262Z++ pfexec svcs -z -Hostate oxide/ntp
7882023-10-05T07:22:12.287Zsvcs: zone '-Hostate': entity not found
7892023-10-05T07:22:12.344Z++ true
7902023-10-05T07:22:12.369Z+ [[ '' != online ]]
7912023-10-05T07:22:12.431Z+ [[ 4 -gt 60 ]]
7922023-10-05T07:22:12.456Z+ sleep 1
7932023-10-05T07:22:13.110Z+ retry=5
7942023-10-05T07:22:13.135Z+++ zoneadm list -n
7952023-10-05T07:22:13.202Z+++ grep oxz_ntp
7962023-10-05T07:22:13.227Z++ pfexec svcs -z -Hostate oxide/ntp
7972023-10-05T07:22:13.300Zsvcs: zone '-Hostate': entity not found
7982023-10-05T07:22:13.326Z++ true
7992023-10-05T07:22:13.372Z+ [[ '' != online ]]
8002023-10-05T07:22:13.397Z+ [[ 5 -gt 60 ]]
8012023-10-05T07:22:13.447Z+ sleep 1
8022023-10-05T07:22:14.129Z+ retry=6
8032023-10-05T07:22:14.154Z+++ zoneadm list -n
8042023-10-05T07:22:14.180Z+++ grep oxz_ntp
8052023-10-05T07:22:14.205Z++ pfexec svcs -z -Hostate oxide/ntp
8062023-10-05T07:22:14.229Zsvcs: zone '-Hostate': entity not found
8072023-10-05T07:22:14.254Z++ true
8082023-10-05T07:22:14.278Z+ [[ '' != online ]]
8092023-10-05T07:22:14.303Z+ [[ 6 -gt 60 ]]
8102023-10-05T07:22:14.327Z+ sleep 1
8112023-10-05T07:22:15.151Z+ retry=7
8122023-10-05T07:22:15.175Z+++ zoneadm list -n
8132023-10-05T07:22:15.200Z+++ grep oxz_ntp
8142023-10-05T07:22:15.224Z++ pfexec svcs -z -Hostate oxide/ntp
8152023-10-05T07:22:15.249Zsvcs: zone '-Hostate': entity not found
8162023-10-05T07:22:15.274Z++ true
8172023-10-05T07:22:15.299Z+ [[ '' != online ]]
8182023-10-05T07:22:15.324Z+ [[ 7 -gt 60 ]]
8192023-10-05T07:22:15.348Z+ sleep 1
8202023-10-05T07:22:16.172Z+ retry=8
8212023-10-05T07:22:16.196Z+++ zoneadm list -n
8222023-10-05T07:22:16.221Z+++ grep oxz_ntp
8232023-10-05T07:22:16.245Z++ pfexec svcs -z -Hostate oxide/ntp
8242023-10-05T07:22:16.270Zsvcs: zone '-Hostate': entity not found
8252023-10-05T07:22:16.295Z++ true
8262023-10-05T07:22:16.320Z+ [[ '' != online ]]
8272023-10-05T07:22:16.344Z+ [[ 8 -gt 60 ]]
8282023-10-05T07:22:16.369Z+ sleep 1
8292023-10-05T07:22:17.193Z+ retry=9
8302023-10-05T07:22:17.217Z+++ zoneadm list -n
8312023-10-05T07:22:17.242Z+++ grep oxz_ntp
8322023-10-05T07:22:17.266Z++ pfexec svcs -z -Hostate oxide/ntp
8332023-10-05T07:22:17.290Zsvcs: zone '-Hostate': entity not found
8342023-10-05T07:22:17.315Z++ true
8352023-10-05T07:22:17.339Z+ [[ '' != online ]]
8362023-10-05T07:22:17.364Z+ [[ 9 -gt 60 ]]
8372023-10-05T07:22:17.388Z+ sleep 1
8382023-10-05T07:22:18.214Z+ retry=10
8392023-10-05T07:22:18.238Z+++ zoneadm list -n
8402023-10-05T07:22:18.263Z+++ grep oxz_ntp
8412023-10-05T07:22:18.287Z++ pfexec svcs -z -Hostate oxide/ntp
8422023-10-05T07:22:18.311Zsvcs: zone '-Hostate': entity not found
8432023-10-05T07:22:18.335Z++ true
8442023-10-05T07:22:18.360Z+ [[ '' != online ]]
8452023-10-05T07:22:18.384Z+ [[ 10 -gt 60 ]]
8462023-10-05T07:22:18.408Z+ sleep 1
8472023-10-05T07:22:19.237Z+ retry=11
8482023-10-05T07:22:19.261Z+++ zoneadm list -n
8492023-10-05T07:22:19.285Z+++ grep oxz_ntp
8502023-10-05T07:22:19.310Z++ pfexec svcs -z -Hostate oxide/ntp
8512023-10-05T07:22:19.334Zsvcs: zone '-Hostate': entity not found
8522023-10-05T07:22:19.359Z++ true
8532023-10-05T07:22:19.384Z+ [[ '' != online ]]
8542023-10-05T07:22:19.409Z+ [[ 11 -gt 60 ]]
8552023-10-05T07:22:19.433Z+ sleep 1
8562023-10-05T07:22:20.256Z+ retry=12
8572023-10-05T07:22:20.282Z+++ zoneadm list -n
8582023-10-05T07:22:20.307Z+++ grep oxz_ntp
8592023-10-05T07:22:20.331Z++ pfexec svcs -z -Hostate oxide/ntp
8602023-10-05T07:22:20.356Zsvcs: zone '-Hostate': entity not found
8612023-10-05T07:22:20.381Z++ true
8622023-10-05T07:22:20.405Z+ [[ '' != online ]]
8632023-10-05T07:22:20.430Z+ [[ 12 -gt 60 ]]
8642023-10-05T07:22:20.456Z+ sleep 1
8652023-10-05T07:22:21.278Z+ retry=13
8662023-10-05T07:22:21.303Z+++ zoneadm list -n
8672023-10-05T07:22:21.328Z+++ grep oxz_ntp
8682023-10-05T07:22:21.352Z++ pfexec svcs -z -Hostate oxide/ntp
8692023-10-05T07:22:21.377Zsvcs: zone '-Hostate': entity not found
8702023-10-05T07:22:21.402Z++ true
8712023-10-05T07:22:21.427Z+ [[ '' != online ]]
8722023-10-05T07:22:21.451Z+ [[ 13 -gt 60 ]]
8732023-10-05T07:22:21.476Z+ sleep 1
8742023-10-05T07:22:22.300Z+ retry=14
8752023-10-05T07:22:22.325Z+++ zoneadm list -n
8762023-10-05T07:22:22.350Z+++ grep oxz_ntp
8772023-10-05T07:22:22.375Z++ pfexec svcs -z oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec -Hostate oxide/ntp
8782023-10-05T07:22:22.399Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8792023-10-05T07:22:22.424Z++ true
8802023-10-05T07:22:22.448Z+ [[ '' != online ]]
8812023-10-05T07:22:22.473Z+ [[ 14 -gt 60 ]]
8822023-10-05T07:22:22.497Z+ sleep 1
8832023-10-05T07:22:23.345Z+ retry=15
8842023-10-05T07:22:23.369Z+++ zoneadm list -n
8852023-10-05T07:22:23.394Z+++ grep oxz_ntp
8862023-10-05T07:22:23.418Z++ pfexec svcs -z oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec -Hostate oxide/ntp
8872023-10-05T07:22:23.442Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8882023-10-05T07:22:23.467Z++ true
8892023-10-05T07:22:23.491Z+ [[ '' != online ]]
8902023-10-05T07:22:23.515Z+ [[ 15 -gt 60 ]]
8912023-10-05T07:22:23.540Z+ sleep 1
8922023-10-05T07:22:24.377Z+ retry=16
8932023-10-05T07:22:24.402Z+++ zoneadm list -n
8942023-10-05T07:22:24.427Z+++ grep oxz_ntp
8952023-10-05T07:22:24.452Z++ pfexec svcs -z oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec -Hostate oxide/ntp
8962023-10-05T07:22:24.476Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8972023-10-05T07:22:24.501Z++ true
8982023-10-05T07:22:24.526Z+ [[ '' != online ]]
8992023-10-05T07:22:24.551Z+ [[ 16 -gt 60 ]]
9002023-10-05T07:22:24.575Z+ sleep 1
9012023-10-05T07:22:25.407Z+ retry=17
9022023-10-05T07:22:25.431Z+++ zoneadm list -n
9032023-10-05T07:22:25.456Z+++ grep oxz_ntp
9042023-10-05T07:22:25.480Z++ pfexec svcs -z oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec -Hostate oxide/ntp
9052023-10-05T07:22:25.505Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9062023-10-05T07:22:25.529Z++ true
9072023-10-05T07:22:25.554Z+ [[ '' != online ]]
9082023-10-05T07:22:25.578Z+ [[ 17 -gt 60 ]]
9092023-10-05T07:22:25.603Z+ sleep 1
9102023-10-05T07:22:26.436Z+ retry=18
9112023-10-05T07:22:26.462Z+++ zoneadm list -n
9122023-10-05T07:22:26.487Z+++ grep oxz_ntp
9132023-10-05T07:22:26.514Z++ pfexec svcs -z oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec -Hostate oxide/ntp
9142023-10-05T07:22:26.539Z+ [[ disabled != online ]]
9152023-10-05T07:22:26.564Z+ [[ 18 -gt 60 ]]
9162023-10-05T07:22:26.589Z+ sleep 1
9172023-10-05T07:22:27.461Z+ retry=19
9182023-10-05T07:22:27.485Z+++ zoneadm list -n
9192023-10-05T07:22:27.510Z+++ grep oxz_ntp
9202023-10-05T07:22:27.534Z++ pfexec svcs -z oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec -Hostate oxide/ntp
9212023-10-05T07:22:27.559Z+ [[ disabled != online ]]
9222023-10-05T07:22:27.583Z+ [[ 19 -gt 60 ]]
9232023-10-05T07:22:27.611Z+ sleep 1
9242023-10-05T07:22:28.482Z+ retry=20
9252023-10-05T07:22:28.507Z+++ zoneadm list -n
9262023-10-05T07:22:28.532Z+++ grep oxz_ntp
9272023-10-05T07:22:28.556Z++ pfexec svcs -z oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec -Hostate oxide/ntp
9282023-10-05T07:22:28.581Z+ [[ disabled != online ]]
9292023-10-05T07:22:28.606Z+ [[ 20 -gt 60 ]]
9302023-10-05T07:22:28.631Z+ sleep 1
9312023-10-05T07:22:29.504Z+ retry=21
9322023-10-05T07:22:29.529Z+++ zoneadm list -n
9332023-10-05T07:22:29.553Z+++ grep oxz_ntp
9342023-10-05T07:22:29.578Z++ pfexec svcs -z oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec -Hostate oxide/ntp
9352023-10-05T07:22:29.602Z+ [[ disabled != online ]]
9362023-10-05T07:22:29.626Z+ [[ 21 -gt 60 ]]
9372023-10-05T07:22:29.652Z+ sleep 1
9382023-10-05T07:22:30.526Z+ retry=22
9392023-10-05T07:22:30.550Z+++ zoneadm list -n
9402023-10-05T07:22:30.575Z+++ grep oxz_ntp
9412023-10-05T07:22:30.599Z++ pfexec svcs -z oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec -Hostate oxide/ntp
9422023-10-05T07:22:30.624Z+ [[ disabled != online ]]
9432023-10-05T07:22:30.648Z+ [[ 22 -gt 60 ]]
9442023-10-05T07:22:30.674Z+ sleep 1
9452023-10-05T07:22:31.548Z+ retry=23
9462023-10-05T07:22:31.573Z+++ zoneadm list -n
9472023-10-05T07:22:31.597Z+++ grep oxz_ntp
9482023-10-05T07:22:31.622Z++ pfexec svcs -z oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec -Hostate oxide/ntp
9492023-10-05T07:22:31.646Z+ [[ disabled != online ]]
9502023-10-05T07:22:31.670Z+ [[ 23 -gt 60 ]]
9512023-10-05T07:22:31.695Z+ sleep 1
9522023-10-05T07:22:32.571Z+ retry=24
9532023-10-05T07:22:32.596Z+++ zoneadm list -n
9542023-10-05T07:22:32.620Z+++ grep oxz_ntp
9552023-10-05T07:22:32.645Z++ pfexec svcs -z oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec -Hostate oxide/ntp
9562023-10-05T07:22:32.669Z+ [[ disabled != online ]]
9572023-10-05T07:22:32.700Z+ [[ 24 -gt 60 ]]
9582023-10-05T07:22:32.725Z+ sleep 1
 
9592023-10-05T07:22:32.888Zsock console login: Oct 5 07:22:32 sok ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9602023-10-05T07:22:33.595Z+ retry=25
9612023-10-05T07:22:33.620Z+++ zoneadm list -n
9622023-10-05T07:22:33.645Z+++ grep oxz_ntp
9632023-10-05T07:22:33.669Z++ pfexec svcs -z oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec -Hostate oxide/ntp
9642023-10-05T07:22:33.694Z+ [[ online != online ]]
9652023-10-05T07:22:33.718Z+ echo 'Waited for chrony: 25s'
9662023-10-05T07:22:33.743ZWaited for chrony: 25s
9672023-10-05T07:22:33.767Z+ export RUST_BACKTRACE=1
9682023-10-05T07:22:33.791Z+ RUST_BACKTRACE=1
9692023-10-05T07:22:33.816Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9702023-10-05T07:22:33.841Z+ ./tests/bootstrap
9712023-10-05T07:22:33.865ZUsing external DNS server at 10.151.2.164:53
9722023-10-05T07:22:33.889Z2023-10-05 07:22:33.791021878 UTC: attempting to log into API
9732023-10-05T07:22:48.681Z2023-10-05 07:22:48.668499053 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9742023-10-05T07:22:49.682Z2023-10-05 07:22:49.670210165 UTC: attempting to log into API
9752023-10-05T07:23:04.714Z2023-10-05 07:23:04.702900177 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9762023-10-05T07:23:05.715Z2023-10-05 07:23:05.703913622 UTC: attempting to log into API
9772023-10-05T07:23:20.749Z2023-10-05 07:23:20.737354895 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9782023-10-05T07:23:21.750Z2023-10-05 07:23:21.739201353 UTC: attempting to log into API
9792023-10-05T07:23:36.785Z2023-10-05 07:23:36.773789385 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9802023-10-05T07:23:37.786Z2023-10-05 07:23:37.774926851 UTC: attempting to log into API
 
9812023-10-05T07:23:39.581ZOct 5 07:23:39 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9822023-10-05T07:23:47.362ZOct 5 07:23:46 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9832023-10-05T07:23:52.818Z2023-10-05 07:23:52.807456283 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9842023-10-05T07:23:53.820Z2023-10-05 07:23:53.808553010 UTC: attempting to log into API
 
9852023-10-05T07:23:54.392ZOct 5 07:23:53 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9862023-10-05T07:24:00.638ZOct 5 07:24:00 sok ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
9872023-10-05T07:24:06.616ZOct 5 07:24:05 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9882023-10-05T07:24:08.852Z2023-10-05 07:24:08.840751517 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9892023-10-05T07:24:09.853Z2023-10-05 07:24:09.841832544 UTC: attempting to log into API
9902023-10-05T07:24:20.648Z2023-10-05 07:24:20.636617912 UTC: login succeeded
9912023-10-05T07:24:20.682Zwaiting for nexus to come up...
9922023-10-05T07:24:20.742Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9932023-10-05T07:24:20.823Zensuring datasets are ready...
9942023-10-05T07:24:24.330Zlet's roll.
9952023-10-05T07:24:24.355Z+ rm ./tests/bootstrap
9962023-10-05T07:24:24.414Z+ for test_bin in tests/*
9972023-10-05T07:24:24.438Z+ ./tests/bootstrap-1edfa7f3d5f511cd
9982023-10-05T07:24:24.463Z
9992023-10-05T07:24:24.487Zrunning 0 tests
10002023-10-05T07:24:24.512Z
10012023-10-05T07:24:24.536Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10022023-10-05T07:24:24.561Z
10032023-10-05T07:24:24.585Z+ for test_bin in tests/*
10042023-10-05T07:24:24.610Z+ ./tests/end_to_end_tests-1e38f424764e177a
10052023-10-05T07:24:24.634Z
10062023-10-05T07:24:24.659Zrunning 2 tests
10072023-10-05T07:24:24.683Ztest no_spoof::no_spoof_here ... ok
10082023-10-05T07:25:24.439Ztest instance_launch::instance_launch has been running for over 60 seconds
10092023-10-05T07:27:18.520Ztest instance_launch::instance_launch ... ok
10102023-10-05T07:27:18.550Z
10112023-10-05T07:27:18.578Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 174.08s
10122023-10-05T07:27:18.602Z
10132023-10-05T07:27:18.629Z+ _exit_trap
10142023-10-05T07:27:18.653Z+ local status=0
10152023-10-05T07:27:18.678Z+ [[ 0 -eq 0 ]]
10162023-10-05T07:27:18.703Z+ exit 0
10172023-10-05T07:27:18.727Zprocess exited: duration 366631 ms, exit code 0
 
10182023-10-05T07:27:18.844Zfound 47 output files
10192023-10-05T07:27:19.033Zuploading: /var/svc/log/oxide-sled-agent:default.log (381067 bytes)
10202023-10-05T07:27:19.082Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 381067 -> 381455
10212023-10-05T07:27:20.191Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10222023-10-05T07:27:20.217Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_015371bc-4f6b-4b5d-a427-c011d81e93c1/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
10232023-10-05T07:27:21.291Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_015371bc-4f6b-4b5d-a427-c011d81e93c1/root/var/svc/log/oxide-cockroachdb:default.log
10242023-10-05T07:27:21.316Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_3fb2e032-5df8-487d-a664-174f7df3285e/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10252023-10-05T07:27:22.386Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_3fb2e032-5df8-487d-a664-174f7df3285e/root/var/svc/log/oxide-cockroachdb:default.log
10262023-10-05T07:27:22.411Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_5c949ee4-2677-4e04-a6dd-608985ad5006/root/var/svc/log/oxide-crucible-pantry:default.log (2332 bytes)
10272023-10-05T07:27:23.480Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_5c949ee4-2677-4e04-a6dd-608985ad5006/root/var/svc/log/oxide-crucible-pantry:default.log
10282023-10-05T07:27:23.505Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bcc7e134-8b34-4e0b-9665-57357cea5b07/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10292023-10-05T07:27:24.573Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bcc7e134-8b34-4e0b-9665-57357cea5b07/root/var/svc/log/oxide-crucible-agent:default.log
10302023-10-05T07:27:24.598Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bcc7e134-8b34-4e0b-9665-57357cea5b07/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10312023-10-05T07:27:25.666Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bcc7e134-8b34-4e0b-9665-57357cea5b07/root/var/svc/log/oxide-crucible-downstairs:default.log
10322023-10-05T07:27:25.691Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_01dc5d10-1af6-4024-9cb5-000535ca055c/root/var/svc/log/oxide-internal_dns:default.log (105167 bytes)
10332023-10-05T07:27:26.871Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_01dc5d10-1af6-4024-9cb5-000535ca055c/root/var/svc/log/oxide-internal_dns:default.log
10342023-10-05T07:27:26.895Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_59bd9b2a-f778-41d5-8113-ee6585b7038c/root/var/svc/log/oxide-oximeter:default.log (176843 bytes)
10352023-10-05T07:27:27.057Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_59bd9b2a-f778-41d5-8113-ee6585b7038c/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 176843 -> 183639
10362023-10-05T07:27:28.126Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_59bd9b2a-f778-41d5-8113-ee6585b7038c/root/var/svc/log/oxide-oximeter:default.log
10372023-10-05T07:27:28.151Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_204752ee-7e4d-421d-a261-e4f8704b83de/root/var/svc/log/oxide-crucible-agent:default.log (14581 bytes)
10382023-10-05T07:27:29.256Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_204752ee-7e4d-421d-a261-e4f8704b83de/root/var/svc/log/oxide-crucible-agent:default.log
10392023-10-05T07:27:29.281Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_204752ee-7e4d-421d-a261-e4f8704b83de/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10402023-10-05T07:27:30.349Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_204752ee-7e4d-421d-a261-e4f8704b83de/root/var/svc/log/oxide-crucible-downstairs:default.log
10412023-10-05T07:27:30.373Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_204752ee-7e4d-421d-a261-e4f8704b83de/root/var/svc/log/oxide-crucible-downstairs:downstairs-8b524330-e572-4be9-bae3-0e9bcef6281b.log (3494 bytes)
10422023-10-05T07:27:31.442Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_204752ee-7e4d-421d-a261-e4f8704b83de/root/var/svc/log/oxide-crucible-downstairs:downstairs-8b524330-e572-4be9-bae3-0e9bcef6281b.log
10432023-10-05T07:27:31.468Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_3ab1b07c-3dc4-4993-9f75-d5d120ea5116/root/var/svc/log/oxide-crucible-agent:default.log (13686 bytes)
10442023-10-05T07:27:32.557Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_3ab1b07c-3dc4-4993-9f75-d5d120ea5116/root/var/svc/log/oxide-crucible-agent:default.log
10452023-10-05T07:27:32.582Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_3ab1b07c-3dc4-4993-9f75-d5d120ea5116/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10462023-10-05T07:27:33.650Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_3ab1b07c-3dc4-4993-9f75-d5d120ea5116/root/var/svc/log/oxide-crucible-downstairs:default.log
10472023-10-05T07:27:33.676Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_3ab1b07c-3dc4-4993-9f75-d5d120ea5116/root/var/svc/log/oxide-crucible-downstairs:downstairs-ba0872da-9859-41be-938e-6a783621af29.log (5064 bytes)
10482023-10-05T07:27:34.744Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_3ab1b07c-3dc4-4993-9f75-d5d120ea5116/root/var/svc/log/oxide-crucible-downstairs:downstairs-ba0872da-9859-41be-938e-6a783621af29.log
10492023-10-05T07:27:34.769Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_4357efa6-2a69-4146-869c-6e3347d8072e/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10502023-10-05T07:27:35.836Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_4357efa6-2a69-4146-869c-6e3347d8072e/root/var/svc/log/oxide-crucible-pantry:default.log
10512023-10-05T07:27:35.861Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_65774b97-64dd-48ff-9052-bbcd7a345721/root/var/svc/log/oxide-internal_dns:default.log (99474 bytes)
10522023-10-05T07:27:37.047Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_65774b97-64dd-48ff-9052-bbcd7a345721/root/var/svc/log/oxide-internal_dns:default.log
10532023-10-05T07:27:37.077Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_6dda579d-7f2b-4f48-a92e-193e12ec5c16/root/var/svc/log/oxide-clickhouse:default.log (622329 bytes)
10542023-10-05T07:27:37.837Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_6dda579d-7f2b-4f48-a92e-193e12ec5c16/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 622329 -> 682939
10552023-10-05T07:27:38.906Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_6dda579d-7f2b-4f48-a92e-193e12ec5c16/root/var/svc/log/oxide-clickhouse:default.log
10562023-10-05T07:27:38.931Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_9d0b6435-2511-48a5-84cb-590500cd5a81/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10572023-10-05T07:27:40.000Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_9d0b6435-2511-48a5-84cb-590500cd5a81/root/var/svc/log/oxide-cockroachdb:default.log
10582023-10-05T07:27:40.024Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_46f3f0ee-173a-4a9d-9350-21cdc8d9c469/root/var/svc/log/oxide-internal_dns:default.log (101909 bytes)
10592023-10-05T07:27:41.203Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_46f3f0ee-173a-4a9d-9350-21cdc8d9c469/root/var/svc/log/oxide-internal_dns:default.log
10602023-10-05T07:27:41.227Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1dd19b2f-3cd0-4ec7-8941-9fa25ba93a10/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10612023-10-05T07:27:42.296Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1dd19b2f-3cd0-4ec7-8941-9fa25ba93a10/root/var/svc/log/oxide-crucible-agent:default.log
10622023-10-05T07:27:42.320Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1dd19b2f-3cd0-4ec7-8941-9fa25ba93a10/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10632023-10-05T07:27:43.389Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1dd19b2f-3cd0-4ec7-8941-9fa25ba93a10/root/var/svc/log/oxide-crucible-downstairs:default.log
10642023-10-05T07:27:43.414Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_21dcba64-0fbf-46c9-917b-d34fe72ab717/root/var/svc/log/oxide-crucible-agent:default.log (14578 bytes)
10652023-10-05T07:27:44.508Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_21dcba64-0fbf-46c9-917b-d34fe72ab717/root/var/svc/log/oxide-crucible-agent:default.log
10662023-10-05T07:27:44.538Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_21dcba64-0fbf-46c9-917b-d34fe72ab717/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10672023-10-05T07:27:46.011Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_21dcba64-0fbf-46c9-917b-d34fe72ab717/root/var/svc/log/oxide-crucible-downstairs:default.log
10682023-10-05T07:27:46.035Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_21dcba64-0fbf-46c9-917b-d34fe72ab717/root/var/svc/log/oxide-crucible-downstairs:downstairs-884d13e1-6971-484f-aff0-c16df9f2f6c9.log (5067 bytes)
10692023-10-05T07:27:47.104Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_21dcba64-0fbf-46c9-917b-d34fe72ab717/root/var/svc/log/oxide-crucible-downstairs:downstairs-884d13e1-6971-484f-aff0-c16df9f2f6c9.log
10702023-10-05T07:27:47.129Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f554898a-cfa5-426c-88b4-36532d49edaf/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10712023-10-05T07:27:48.196Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f554898a-cfa5-426c-88b4-36532d49edaf/root/var/svc/log/oxide-crucible-agent:default.log
10722023-10-05T07:27:48.220Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f554898a-cfa5-426c-88b4-36532d49edaf/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10732023-10-05T07:27:49.296Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f554898a-cfa5-426c-88b4-36532d49edaf/root/var/svc/log/oxide-crucible-downstairs:default.log
10742023-10-05T07:27:49.324Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_c2818626-4118-4135-86e0-7d01aa7ad29b/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10752023-10-05T07:27:50.394Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_c2818626-4118-4135-86e0-7d01aa7ad29b/root/var/svc/log/oxide-cockroachdb:default.log
10762023-10-05T07:27:50.418Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ea9d452a-2a8d-4f4e-8ed2-0bda13b94a72/root/var/svc/log/oxide-crucible-agent:default.log (14580 bytes)
10772023-10-05T07:27:51.514Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ea9d452a-2a8d-4f4e-8ed2-0bda13b94a72/root/var/svc/log/oxide-crucible-agent:default.log
10782023-10-05T07:27:51.538Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ea9d452a-2a8d-4f4e-8ed2-0bda13b94a72/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10792023-10-05T07:27:52.610Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ea9d452a-2a8d-4f4e-8ed2-0bda13b94a72/root/var/svc/log/oxide-crucible-downstairs:default.log
10802023-10-05T07:27:52.634Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ea9d452a-2a8d-4f4e-8ed2-0bda13b94a72/root/var/svc/log/oxide-crucible-downstairs:downstairs-f4525fa7-a825-42ce-95cd-5a76e11e0e3c.log (3494 bytes)
10812023-10-05T07:27:53.713Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ea9d452a-2a8d-4f4e-8ed2-0bda13b94a72/root/var/svc/log/oxide-crucible-downstairs:downstairs-f4525fa7-a825-42ce-95cd-5a76e11e0e3c.log
10822023-10-05T07:27:53.737Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_565236fc-4cd8-455d-a7da-4a979b406b0e/root/var/svc/log/oxide-nexus:default.log (400477 bytes)
10832023-10-05T07:27:53.952Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_565236fc-4cd8-455d-a7da-4a979b406b0e/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 400477 -> 402301
10842023-10-05T07:27:55.025Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_565236fc-4cd8-455d-a7da-4a979b406b0e/root/var/svc/log/oxide-nexus:default.log
10852023-10-05T07:27:55.049Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_574b9334-1f34-4b15-82b9-36ff04a8b13d/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10862023-10-05T07:27:56.124Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_574b9334-1f34-4b15-82b9-36ff04a8b13d/root/var/svc/log/oxide-cockroachdb:default.log
10872023-10-05T07:27:56.148Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_448754f8-55bd-46f9-801c-5c778d3002a0/root/var/svc/log/oxide-external_dns:default.log (15606 bytes)
10882023-10-05T07:27:57.245Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_448754f8-55bd-46f9-801c-5c778d3002a0/root/var/svc/log/oxide-external_dns:default.log
10892023-10-05T07:27:57.269Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ebed193d-06d3-4e03-84cc-90f483fbe1d3/root/var/svc/log/oxide-external_dns:default.log (15614 bytes)
10902023-10-05T07:27:58.365Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ebed193d-06d3-4e03-84cc-90f483fbe1d3/root/var/svc/log/oxide-external_dns:default.log
10912023-10-05T07:27:58.389Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_f7c48384-c6a4-49cf-9915-1f5597adc777/root/var/svc/log/oxide-nexus:default.log (480142 bytes)
10922023-10-05T07:27:58.639Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_f7c48384-c6a4-49cf-9915-1f5597adc777/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 480142 -> 482575
10932023-10-05T07:27:59.715Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_f7c48384-c6a4-49cf-9915-1f5597adc777/root/var/svc/log/oxide-nexus:default.log
10942023-10-05T07:27:59.739Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d5dbf010-5241-47f6-819e-0c7cab802175/root/var/svc/log/oxide-crucible-agent:default.log (14579 bytes)
10952023-10-05T07:28:00.861Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d5dbf010-5241-47f6-819e-0c7cab802175/root/var/svc/log/oxide-crucible-agent:default.log
10962023-10-05T07:28:00.891Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d5dbf010-5241-47f6-819e-0c7cab802175/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10972023-10-05T07:28:02.383Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d5dbf010-5241-47f6-819e-0c7cab802175/root/var/svc/log/oxide-crucible-downstairs:default.log
10982023-10-05T07:28:02.408Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d5dbf010-5241-47f6-819e-0c7cab802175/root/var/svc/log/oxide-crucible-downstairs:downstairs-ff3e067c-be04-43bc-84a9-cf18723e2d22.log (3495 bytes)
10992023-10-05T07:28:03.499Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d5dbf010-5241-47f6-819e-0c7cab802175/root/var/svc/log/oxide-crucible-downstairs:downstairs-ff3e067c-be04-43bc-84a9-cf18723e2d22.log
11002023-10-05T07:28:03.524Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f75aafb8-c738-4591-b0d1-7b56d1985e55/root/var/svc/log/oxide-crucible-agent:default.log (14150 bytes)
11012023-10-05T07:28:04.625Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f75aafb8-c738-4591-b0d1-7b56d1985e55/root/var/svc/log/oxide-crucible-agent:default.log
11022023-10-05T07:28:04.651Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f75aafb8-c738-4591-b0d1-7b56d1985e55/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11032023-10-05T07:28:05.724Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f75aafb8-c738-4591-b0d1-7b56d1985e55/root/var/svc/log/oxide-crucible-downstairs:default.log
11042023-10-05T07:28:05.750Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f75aafb8-c738-4591-b0d1-7b56d1985e55/root/var/svc/log/oxide-crucible-downstairs:downstairs-fbd5083d-8ff1-4d23-8acc-473e6f8ff82e.log (5064 bytes)
11052023-10-05T07:28:06.821Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f75aafb8-c738-4591-b0d1-7b56d1985e55/root/var/svc/log/oxide-crucible-downstairs:downstairs-fbd5083d-8ff1-4d23-8acc-473e6f8ff82e.log
11062023-10-05T07:28:06.846Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_7c3adcec-5eb6-4c34-91a0-2d0059550ab5/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
11072023-10-05T07:28:07.913Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_7c3adcec-5eb6-4c34-91a0-2d0059550ab5/root/var/svc/log/oxide-crucible-pantry:default.log
11082023-10-05T07:28:07.950Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_4193a1a9-8e01-450f-ac36-d627a646915c/root/var/svc/log/oxide-nexus:default.log (4019490 bytes)
11092023-10-05T07:28:09.201Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_4193a1a9-8e01-450f-ac36-d627a646915c/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4019490 -> 4081326
11102023-10-05T07:28:10.271Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_4193a1a9-8e01-450f-ac36-d627a646915c/root/var/svc/log/oxide-nexus:default.log
11112023-10-05T07:28:10.297Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec/root/var/svc/log/oxide-ntp:default.log (850 bytes)
11122023-10-05T07:28:11.375Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec/root/var/svc/log/oxide-ntp:default.log
11132023-10-05T07:28:11.400Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec/root/var/log/chrony/measurements.log (10960 bytes)
11142023-10-05T07:28:11.447Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec/root/var/log/chrony/measurements.log" changed size mid upload: 10960 -> 12056
11152023-10-05T07:28:12.517Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec/root/var/log/chrony/measurements.log
11162023-10-05T07:28:12.542Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec/root/var/log/chrony/statistics.log (6435 bytes)
11172023-10-05T07:28:12.566Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec/root/var/log/chrony/statistics.log" changed size mid upload: 6435 -> 7488
11182023-10-05T07:28:13.632Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec/root/var/log/chrony/statistics.log
11192023-10-05T07:28:13.659Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec/root/var/log/chrony/tracking.log (1321 bytes)
11202023-10-05T07:28:13.684Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec/root/var/log/chrony/tracking.log" changed size mid upload: 1321 -> 1453
11212023-10-05T07:28:14.748Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e29e8f38-f4f8-4dd3-956a-cae8c57a28ec/root/var/log/chrony/tracking.log