01HD9GGBP8AGVZWSZYHYEM4MF1: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HD9GH6CQH5XZTGQFFKKFM1SG

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-21T17:10:16.068Zjob dependencies complete; ready to run (waiting for 51 m 1 s)
22023-10-21T17:11:00.413Zjob assigned to worker 01HD9KETRNAH2BGPNY97JRNEFR (queued for 44 s)
32023-10-21T17:11:00.600Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-21T17:11:00.655Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-10-21T17:11:00.688Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-10-21T17:11:00.722Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-21T17:11:00.755Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-10-21T17:11:00.789ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-21T17:11:00.822Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-21T17:11:00.857ZOxide Helios Version helios-2.0.22241 64-bit
112023-10-21T17:11:00.888ZConfiguring device.
122023-10-21T17:11:00.922Z#################################################
132023-10-21T17:11:00.956Z#### ############# ##
142023-10-21T17:11:00.989Z### ### ############ ##
152023-10-21T17:11:01.025Z## ### # ## ### ## Oxide Computer Company ##
162023-10-21T17:11:01.068Z## ## ## ### # ### ##
172023-10-21T17:11:01.100Z## # ### #### #### This Station Under ##
182023-10-21T17:11:01.136Z### ### #### # ### Computer Control ##
192023-10-21T17:11:01.167Z#### #### ### ## ##
202023-10-21T17:11:01.201Z##################################################
212023-10-21T17:11:01.234ZHostname: buskin
222023-10-21T17:11:01.268ZLoading smf(7) serice descriptions:
232023-10-21T17:11:01.488Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622.gz
242023-10-21T17:11:03.041Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622.gz
252023-10-21T17:11:03.068Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-21T17:11:24.396Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-21T17:11:24.420Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280.gz
282023-10-21T17:11:56.580Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280.gz
292023-10-21T17:11:56.605Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-21T17:11:57.585Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-21T17:11:57.609Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-21T17:12:00.740Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-21T17:12:00.769Zdownloading input: /input/package/work/version.txt
342023-10-21T17:12:00.801Zdownloaded input: /input/package/work/version.txt
352023-10-21T17:12:00.833Zdownloading input: /input/package/work/package.tar.gz
362023-10-21T17:12:22.345Zdownloaded input: /input/package/work/package.tar.gz
372023-10-21T17:12:22.370Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-21T17:12:54.571Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-21T17:12:54.597Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-21T17:13:05.635Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-21T17:13:05.666Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-21T17:14:04.693Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-21T17:14:04.717Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-21T17:14:41.042Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-21T17:14:41.068Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-21T17:15:04.560Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-21T17:15:04.586Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-21T17:15:08.923Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-21T17:15:08.950Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-21T17:15:17.366Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-21T17:15:17.397Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-21T17:15:21.395Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-21T17:15:21.420Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-21T17:15:26.761Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-21T17:15:26.785Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-21T17:15:26.809Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-21T17:15:26.833Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-21T17:15:31.571Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-21T17:15:31.599Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-21T17:15:34.887Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-21T17:15:34.912Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-21T17:15:48.991Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-21T17:15:49.015Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-21T17:16:01.854Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-21T17:16:01.878Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-21T17:16:01.904Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-21T17:16:01.928Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-21T17:16:05.294Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-21T17:16:05.321Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-21T17:16:21.576Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-21T17:16:21.621Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-21T17:17:12.337Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-21T17:17:12.362Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-21T17:17:44.535Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-21T17:17:44.567Zstarting task 0: "setup"
762023-10-21T17:17:44.601Z++ uname -s
772023-10-21T17:17:44.626Z+ kern=SunOS
782023-10-21T17:17:44.651Z+ case "$kern" in
792023-10-21T17:17:44.679Z+ groupadd -g 12345 build
802023-10-21T17:17:44.704Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-21T17:17:46.611Z+ zfs create -o mountpoint=/work rpool/work
822023-10-21T17:17:46.635Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-21T17:17:46.668Z+ home_fs=zfs
842023-10-21T17:17:46.696Z+ [[ zfs == autofs ]]
852023-10-21T17:17:46.741Z+ mkdir -p /home/build
862023-10-21T17:17:46.776Z+ chown build:build /home/build /work
872023-10-21T17:17:48.635Z+ chmod 0700 /home/build /work
882023-10-21T17:17:48.660Zprocess exited: duration 4045 ms, exit code 0
 
892023-10-21T17:17:48.709Zstarting task 1: "authentication"
902023-10-21T17:17:48.777Zprocess exited: duration 44 ms, exit code 0
 
912023-10-21T17:17:48.826Zstarting task 2: "build"
922023-10-21T17:17:48.857Z+ trap _exit_trap EXIT
932023-10-21T17:17:48.881Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-21T17:17:49.054Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-21T17:17:49.081Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-21T17:17:49.124Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-21T17:17:49.149Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-21T17:17:49.174Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-21T17:17:49.198Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-21T17:17:49.222Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-21T17:17:49.246Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-21T17:17:49.270Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-21T17:17:49.294Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-21T17:17:49.318Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-21T17:17:49.342Z+ [[ -d /opt/oxide ]]
1062023-10-21T17:17:49.366Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-21T17:17:49.390Z+ pfexec mkdir /opt/oxide
1082023-10-21T17:17:49.416Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-21T17:17:49.441Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-21T17:17:49.465Z+ cd /opt/oxide-underneath
1112023-10-21T17:17:49.489Z+ pfexec tar ceEp@/f - .
1122023-10-21T17:17:49.513Z+ cd /opt/oxide
1132023-10-21T17:17:49.537Z+ pfexec tar xveEp@/f -
1142023-10-21T17:17:49.561Zx ., 0 bytes, 0 tape blocks
1152023-10-21T17:17:49.585Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-21T17:17:49.609Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-21T17:17:49.633Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-10-21T17:17:49.657Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-21T17:17:49.681Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-21T17:17:49.705Z+ pfexec mkdir /opt/oxide/work
1212023-10-21T17:17:49.729Z+ pfexec chown build:build /opt/oxide/work
1222023-10-21T17:17:49.753Z+ cd /opt/oxide/work
1232023-10-21T17:17:49.777Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-21T17:17:50.777ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-21T17:17:50.818Zx out/mg-ddm-gz.tar, 63638528 bytes, 124294 tape blocks
1262023-10-21T17:17:50.845Zx out/omicron-sled-agent.tar, 115460608 bytes, 225509 tape blocks
1272023-10-21T17:17:50.873Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-21T17:17:50.918Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-10-21T17:17:50.965Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-10-21T17:17:51.015Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-21T17:17:51.075Zx target/release/omicron-package, 38196032 bytes, 74602 tape blocks
1322023-10-21T17:17:51.122Zx tools/create_virtual_hardware.sh, 2986 bytes, 6 tape blocks
1332023-10-21T17:17:51.210Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-21T17:17:51.278Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-21T17:17:51.402Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-21T17:17:51.665Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-21T17:18:04.405Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-21T17:18:04.615Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-21T17:18:04.925Z
1402023-10-21T17:18:05.317Zreal 1.557534478
1412023-10-21T17:18:05.518Zuser 1.401837038
1422023-10-21T17:18:05.808Zsys 0.276661540
1432023-10-21T17:18:05.868Ztrap 0.000058271
1442023-10-21T17:18:05.959Ztflt 0.000000000
1452023-10-21T17:18:06.006Zdflt 0.000000000
1462023-10-21T17:18:06.064Zkflt 0.000000000
1472023-10-21T17:18:06.114Zlock 0.000000000
1482023-10-21T17:18:06.155Zslp 1.398581672
1492023-10-21T17:18:06.222Zlat 0.035986401
1502023-10-21T17:18:06.256Zstop 0.000117321
1512023-10-21T17:18:06.285Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-21T17:18:06.326Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-21T17:18:06.392Z+ mkdir tests
1542023-10-21T17:18:06.421Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-21T17:18:06.465Z+ ptime -m gunzip
1562023-10-21T17:18:06.498Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622
1572023-10-21T17:18:06.528Z
1582023-10-21T17:18:06.556Zreal 0.143542176
1592023-10-21T17:18:06.582Zuser 0.129844850
1602023-10-21T17:18:06.609Zsys 0.013473783
1612023-10-21T17:18:06.636Ztrap 0.000009680
1622023-10-21T17:18:06.665Ztflt 0.000000000
1632023-10-21T17:18:06.698Zdflt 0.000000000
1642023-10-21T17:18:06.753Zkflt 0.000000000
1652023-10-21T17:18:06.779Zlock 0.000000000
1662023-10-21T17:18:06.809Zslp 0.000019680
1672023-10-21T17:18:06.840Zlat 0.000019640
1682023-10-21T17:18:06.866Zstop 0.000115371
1692023-10-21T17:18:06.899Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622
1702023-10-21T17:18:06.931Z+ chmod a+x tests/bootstrap-49e31e75b930b622
1712023-10-21T17:18:06.958Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-21T17:18:06.984Z+ ptime -m gunzip
1732023-10-21T17:18:07.020Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-21T17:18:07.054Z
1752023-10-21T17:18:07.088Zreal 2.252661307
1762023-10-21T17:18:07.129Zuser 2.101885747
1772023-10-21T17:18:07.165Zsys 0.150375743
1782023-10-21T17:18:07.191Ztrap 0.000178623
1792023-10-21T17:18:07.279Ztflt 0.000000000
1802023-10-21T17:18:07.311Zdflt 0.000000000
1812023-10-21T17:18:07.340Zkflt 0.000000000
1822023-10-21T17:18:07.370Zlock 0.000000000
1832023-10-21T17:18:07.396Zslp 0.000020770
1842023-10-21T17:18:07.425Zlat 0.000018119
1852023-10-21T17:18:07.458Zstop 0.000119372
1862023-10-21T17:18:07.900Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-21T17:18:07.933Z+ chmod a+x tests/bootstrap
1882023-10-21T17:18:07.986Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-21T17:18:08.018Z+ ptime -m gunzip
1902023-10-21T17:18:08.049Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280
1912023-10-21T17:18:08.077Z
1922023-10-21T17:18:08.103Zreal 2.552525010
1932023-10-21T17:18:08.136Zuser 2.391904430
1942023-10-21T17:18:08.172Zsys 0.160164126
1952023-10-21T17:18:08.216Ztrap 0.000199952
1962023-10-21T17:18:08.257Ztflt 0.000000000
1972023-10-21T17:18:08.320Zdflt 0.000000000
1982023-10-21T17:18:08.351Zkflt 0.000000000
1992023-10-21T17:18:08.410Zlock 0.000000000
2002023-10-21T17:18:08.456Zslp 0.000023850
2012023-10-21T17:18:08.488Zlat 0.000040660
2022023-10-21T17:18:08.520Zstop 0.000125741
2032023-10-21T17:18:08.547Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280
2042023-10-21T17:18:08.582Z+ chmod a+x tests/end_to_end_tests-a91cf9af3457d280
2052023-10-21T17:18:08.608ZNAME CIDR FIRST LAST COUNT
2062023-10-21T17:18:08.643Z+ bmat address ls
2072023-10-21T17:18:08.672Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-10-21T17:18:08.710Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-21T17:18:08.755ZPING 10.151.1.100: 56 data bytes
2102023-10-21T17:18:08.799Z+ set -- 10.151.1.164 10.151.1.227 64
2112023-10-21T17:18:08.833Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.010 ms
2122023-10-21T17:18:08.871Z+ EXTRA_IP_START=10.151.1.164
2132023-10-21T17:18:08.914Z
2142023-10-21T17:18:08.950Z+ EXTRA_IP_END=10.151.1.227
2152023-10-21T17:18:08.993Z----10.151.1.100 PING Statistics----
2162023-10-21T17:18:09.029Z+ EXTRA_IP_COUNT=64
2172023-10-21T17:18:09.057Z1 packets transmitted, 1 packets received, 0% packet loss
2182023-10-21T17:18:09.094Z+ (( EXTRA_IP_COUNT >= 32 ))
2192023-10-21T17:18:09.120Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2202023-10-21T17:18:09.165Z+ EXTRA_IP_BASE=10.151.1
2212023-10-21T17:18:09.200ZUsing igb0 as physical link
2222023-10-21T17:18:09.235Z+ EXTRA_IP_FOCTET=164
2232023-10-21T17:18:09.264ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2242023-10-21T17:18:09.307Z+ EXTRA_IP_LOCTET=227
2252023-10-21T17:18:09.389ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2262023-10-21T17:18:09.414Z+ SERVICE_IP_POOL_START=10.151.1.164
2272023-10-21T17:18:09.455ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2282023-10-21T17:18:09.505Z+ SERVICE_IP_POOL_END=10.151.1.173
2292023-10-21T17:18:09.549ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2302023-10-21T17:18:09.578Z+ DNS_IP1=10.151.1.164
2312023-10-21T17:18:09.621ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2322023-10-21T17:18:09.647Z+ DNS_IP2=10.151.1.165
2332023-10-21T17:18:09.679ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2342023-10-21T17:18:09.718Z+ UPLINK_IP=10.151.1.174
2352023-10-21T17:18:09.765ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2362023-10-21T17:18:09.804Z+ IPPOOL_START=10.151.1.175
2372023-10-21T17:18:09.831ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2382023-10-21T17:18:09.862Z+ IPPOOL_END=10.151.1.227
2392023-10-21T17:18:09.894ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2402023-10-21T17:18:09.926Z++ ipadm show-addr -po type,addr
2412023-10-21T17:18:09.953ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2422023-10-21T17:18:09.978Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2432023-10-21T17:18:10.003ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2442023-10-21T17:18:10.033Z+ GATEWAY_IP=10.151.1.100
2452023-10-21T17:18:10.062ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2462023-10-21T17:18:10.092Z+ [[ -n 10.151.1.100 ]]
2472023-10-21T17:18:10.122ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2482023-10-21T17:18:10.150Z+ ping -s 10.151.1.100 56 1
2492023-10-21T17:18:10.176ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2502023-10-21T17:18:10.206Z++ arp -an
2512023-10-21T17:18:10.236ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2522023-10-21T17:18:10.261Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2532023-10-21T17:18:10.286ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
2542023-10-21T17:18:10.310Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2552023-10-21T17:18:10.336ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
2562023-10-21T17:18:10.390Z+ routeadm -e ipv4-forwarding -u
2572023-10-21T17:18:10.451ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
2582023-10-21T17:18:10.485Z+ PXA_START=10.151.1.164
2592023-10-21T17:18:10.516ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
2602023-10-21T17:18:10.546Z+ PXA_END=10.151.1.227
2612023-10-21T17:18:10.571ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
2622023-10-21T17:18:10.599Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2632023-10-21T17:18:10.629ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
2642023-10-21T17:18:10.659Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2652023-10-21T17:18:11.132ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
2662023-10-21T17:18:11.157Z+ ZPOOL_VDEV_DIR=/scratch
2672023-10-21T17:18:11.183ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
2682023-10-21T17:18:11.209Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2692023-10-21T17:18:11.239ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
2702023-10-21T17:18:11.264Z+++ dirname ./tools/create_virtual_hardware.sh
2712023-10-21T17:18:11.288ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
2722023-10-21T17:18:11.317Z++ cd ./tools
2732023-10-21T17:18:11.350ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
2742023-10-21T17:18:11.379Z++ pwd
2752023-10-21T17:18:11.416ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
2762023-10-21T17:18:11.442Z+ SOURCE_DIR=/opt/oxide/work/tools
2772023-10-21T17:18:11.515ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
2782023-10-21T17:18:11.576Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2792023-10-21T17:18:11.619ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-10-21T17:18:11.725Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2812023-10-21T17:18:11.749ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2822023-10-21T17:18:11.775Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2832023-10-21T17:18:11.799ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
2842023-10-21T17:18:11.831Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2852023-10-21T17:18:11.858ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2862023-10-21T17:18:11.883Z++ dladm show-phys -p -o LINK
2872023-10-21T17:18:11.908ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2882023-10-21T17:18:11.932Z++ head -1
2892023-10-21T17:18:11.956ZSimnet net0/sc0_0 exists
2902023-10-21T17:18:11.980Z+ PHYSICAL_LINK=igb0
2912023-10-21T17:18:12.004ZSimnet net1/sc1_0 exists
2922023-10-21T17:18:12.028Z+ echo 'Using igb0 as physical link'
2932023-10-21T17:18:12.052ZVnic sc0_1 exists
2942023-10-21T17:18:12.076Z+ ensure_run_as_root
2952023-10-21T17:18:12.100Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
2962023-10-21T17:18:12.124Z++ id -u
2972023-10-21T17:18:12.148Z+ [[ 0 -ne 0 ]]
2982023-10-21T17:18:12.172Z+ ensure_zpools
2992023-10-21T17:18:12.196Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
3002023-10-21T17:18:12.227Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
3012023-10-21T17:18:12.253Z+ readarray -t ZPOOLS
3022023-10-21T17:18:12.277Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
3032023-10-21T17:18:12.301Z++ sed 's/[ ",]//g'
3042023-10-21T17:18:12.325Z+ for ZPOOL in "${ZPOOLS[@]}"
3052023-10-21T17:18:12.350Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
3062023-10-21T17:18:12.376Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
3072023-10-21T17:18:12.400Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
3082023-10-21T17:18:12.424Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
3092023-10-21T17:18:12.448Z0+0 records in
3102023-10-21T17:18:12.473Z0+0 records out
3112023-10-21T17:18:12.497Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
3122023-10-21T17:18:12.550Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
3132023-10-21T17:18:12.575Z+ set +x
3142023-10-21T17:18:12.599Z++ zpool list -o name
3152023-10-21T17:18:12.623Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
3162023-10-21T17:18:12.649Z+ [[ -z '' ]]
3172023-10-21T17:18:12.675Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
3182023-10-21T17:18:12.701Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
3192023-10-21T17:18:12.727Z+ set +x
3202023-10-21T17:18:12.760Z+ for ZPOOL in "${ZPOOLS[@]}"
3212023-10-21T17:18:12.787Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3222023-10-21T17:18:12.811Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3232023-10-21T17:18:12.835Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3242023-10-21T17:18:12.859Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3252023-10-21T17:18:12.885Z0+0 records in
3262023-10-21T17:18:12.909Z0+0 records out
3272023-10-21T17:18:12.935Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3282023-10-21T17:18:12.960Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3292023-10-21T17:18:12.984Z+ set +x
3302023-10-21T17:18:13.008Z++ zpool list -o name
3312023-10-21T17:18:13.033Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3322023-10-21T17:18:13.056Z+ [[ -z '' ]]
3332023-10-21T17:18:13.081Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3342023-10-21T17:18:13.105Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3352023-10-21T17:18:13.129Z+ set +x
3362023-10-21T17:18:13.155Z+ for ZPOOL in "${ZPOOLS[@]}"
3372023-10-21T17:18:13.182Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3382023-10-21T17:18:13.208Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3392023-10-21T17:18:13.232Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3402023-10-21T17:18:13.257Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3412023-10-21T17:18:13.281Z0+0 records in
3422023-10-21T17:18:13.305Z0+0 records out
3432023-10-21T17:18:13.329Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3442023-10-21T17:18:13.353Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3452023-10-21T17:18:13.377Z+ set +x
3462023-10-21T17:18:13.409Z++ zpool list -o name
3472023-10-21T17:18:13.439Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3482023-10-21T17:18:13.466Z+ [[ -z '' ]]
3492023-10-21T17:18:13.496Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-10-21T17:18:13.527Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-10-21T17:18:13.604Z+ set +x
3522023-10-21T17:18:13.648Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-10-21T17:18:13.677Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-10-21T17:18:13.703Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-10-21T17:18:13.733Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-10-21T17:18:13.763Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-10-21T17:18:13.790Z0+0 records in
3582023-10-21T17:18:13.817Z0+0 records out
3592023-10-21T17:18:13.844Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3602023-10-21T17:18:13.874Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-10-21T17:18:13.900Z+ set +x
3622023-10-21T17:18:13.926Z++ zpool list -o name
3632023-10-21T17:18:13.954Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-10-21T17:18:13.982Z+ [[ -z '' ]]
3652023-10-21T17:18:14.013Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-10-21T17:18:14.037Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-10-21T17:18:14.062Z+ set +x
3682023-10-21T17:18:14.095Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-10-21T17:18:14.122Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-10-21T17:18:14.146Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-10-21T17:18:14.170Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-10-21T17:18:14.195Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-10-21T17:18:14.220Z0+0 records in
3742023-10-21T17:18:14.244Z0+0 records out
3752023-10-21T17:18:14.280Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3762023-10-21T17:18:14.305Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-10-21T17:18:14.330Z+ set +x
3782023-10-21T17:18:14.354Z++ zpool list -o name
3792023-10-21T17:18:14.378Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-10-21T17:18:14.406Z+ [[ -z '' ]]
3812023-10-21T17:18:14.430Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-10-21T17:18:14.454Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-10-21T17:18:14.478Z+ set +x
3842023-10-21T17:18:14.503Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-10-21T17:18:14.534Zdownloading softnpu ...done
3862023-10-21T17:18:14.560Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-10-21T17:18:14.584Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-10-21T17:18:14.608Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-10-21T17:18:14.661Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-10-21T17:18:14.685Z0+0 records in
3912023-10-21T17:18:14.710Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/libsidecar_lite.so.sha256.txt
3922023-10-21T17:18:14.734Z0+0 records out
3932023-10-21T17:18:14.758Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3942023-10-21T17:18:14.782Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3952023-10-21T17:18:14.806Z+ set +x
3962023-10-21T17:18:14.830Z++ zpool list -o name
3972023-10-21T17:18:14.854Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3982023-10-21T17:18:14.878Z+ [[ -z '' ]]
3992023-10-21T17:18:14.902Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
4002023-10-21T17:18:14.930Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4012023-10-21T17:18:14.980Z+ set +x
4022023-10-21T17:18:15.008Z+ for ZPOOL in "${ZPOOLS[@]}"
4032023-10-21T17:18:15.035Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4042023-10-21T17:18:15.059Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4052023-10-21T17:18:15.084Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4062023-10-21T17:18:15.108Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4072023-10-21T17:18:15.132Z0+0 records in
4082023-10-21T17:18:15.156Z0+0 records out
4092023-10-21T17:18:15.181Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4102023-10-21T17:18:15.207Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4112023-10-21T17:18:15.231Z+ set +x
4122023-10-21T17:18:15.257Z++ zpool list -o name
4132023-10-21T17:18:15.281Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4142023-10-21T17:18:15.305Z+ [[ -z '' ]]
4152023-10-21T17:18:15.332Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4162023-10-21T17:18:15.356Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4172023-10-21T17:18:15.385Z+ set +x
4182023-10-21T17:18:15.824Z+ for ZPOOL in "${ZPOOLS[@]}"
4192023-10-21T17:18:15.851Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4202023-10-21T17:18:15.875Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4212023-10-21T17:18:16.309Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4222023-10-21T17:18:16.344Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4232023-10-21T17:18:16.370Z0+0 records in
4242023-10-21T17:18:16.397Z0+0 records out
4252023-10-21T17:18:16.421Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4262023-10-21T17:18:16.446Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4272023-10-21T17:18:16.470Z+ set +x
4282023-10-21T17:18:16.495Z++ zpool list -o name
4292023-10-21T17:18:16.544Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4302023-10-21T17:18:16.568Z+ [[ -z '' ]]
4312023-10-21T17:18:16.593Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4322023-10-21T17:18:16.617Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4332023-10-21T17:18:16.641Z+ set +x
4342023-10-21T17:18:16.666Z+ for ZPOOL in "${ZPOOLS[@]}"
4352023-10-21T17:18:16.696Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4362023-10-21T17:18:16.727Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4372023-10-21T17:18:16.759Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4382023-10-21T17:18:16.784Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4392023-10-21T17:18:16.809Z0+0 records in
4402023-10-21T17:18:16.833Z0+0 records out
4412023-10-21T17:18:16.857Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4422023-10-21T17:18:16.881Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4432023-10-21T17:18:16.906Z+ set +x
4442023-10-21T17:18:16.930Z++ zpool list -o name
4452023-10-21T17:18:16.954Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4462023-10-21T17:18:16.979Z+ [[ -z '' ]]
4472023-10-21T17:18:17.003Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4482023-10-21T17:18:17.028Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4492023-10-21T17:18:17.053Z+ set +x
4502023-10-21T17:18:17.078Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4512023-10-21T17:18:17.104Z+ readarray -t ZPOOLS
4522023-10-21T17:18:17.131Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4532023-10-21T17:18:17.163Z++ sed 's/[ ",]//g'
4542023-10-21T17:18:17.189Z+ for ZPOOL in "${ZPOOLS[@]}"
4552023-10-21T17:18:17.217Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4562023-10-21T17:18:17.242Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4572023-10-21T17:18:17.267Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4582023-10-21T17:18:17.294Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4592023-10-21T17:18:17.318Z0+0 records in
4602023-10-21T17:18:17.344Z0+0 records out
4612023-10-21T17:18:17.370Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4622023-10-21T17:18:17.395Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4632023-10-21T17:18:17.420Z+ set +x
4642023-10-21T17:18:17.444Z++ zpool list -o name
4652023-10-21T17:18:17.468Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4662023-10-21T17:18:17.493Z+ [[ -z '' ]]
4672023-10-21T17:18:17.517Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4682023-10-21T17:18:17.541Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4692023-10-21T17:18:17.566Z+ set +x
4702023-10-21T17:18:17.595Z+ for ZPOOL in "${ZPOOLS[@]}"
4712023-10-21T17:18:17.622Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4722023-10-21T17:18:17.650Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4732023-10-21T17:18:17.675Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4742023-10-21T17:18:17.699Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4752023-10-21T17:18:17.723Z0+0 records in
4762023-10-21T17:18:17.776Z0+0 records out
4772023-10-21T17:18:17.800Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
4782023-10-21T17:18:17.826Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4792023-10-21T17:18:17.850Z+ set +x
4802023-10-21T17:18:17.874Z++ zpool list -o name
4812023-10-21T17:18:17.898Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4822023-10-21T17:18:17.923Z+ [[ -z '' ]]
4832023-10-21T17:18:17.947Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4842023-10-21T17:18:17.972Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4852023-10-21T17:18:17.996Z+ set +x
4862023-10-21T17:18:18.023Z+ ensure_simulated_links igb0
4872023-10-21T17:18:18.057Z+ local PHYSICAL_LINK=igb0
4882023-10-21T17:18:18.085Z+ INDICES=("0" "1")
4892023-10-21T17:18:18.110Z+ for I in "${INDICES[@]}"
4902023-10-21T17:18:18.134Z++ get_simnet_name_if_exists net0
4912023-10-21T17:18:18.158Z++ dladm show-simnet -p -o LINK net0
4922023-10-21T17:18:18.183Z++ echo ''
4932023-10-21T17:18:18.207Z+ [[ -z '' ]]
4942023-10-21T17:18:18.231Z+ dladm create-simnet -t net0
4952023-10-21T17:18:18.255Z+ dladm create-simnet -t sc0_0
4962023-10-21T17:18:18.279Z+ dladm modify-simnet -t -p net0 sc0_0
4972023-10-21T17:18:18.304Z+ dladm set-linkprop -p mtu=9000 sc0_0
4982023-10-21T17:18:18.328Z+ success 'Simnet net0/sc0_0 exists'
4992023-10-21T17:18:18.355Z+ set +x
5002023-10-21T17:18:18.380Z+ for I in "${INDICES[@]}"
5012023-10-21T17:18:18.407Z++ get_simnet_name_if_exists net1
5022023-10-21T17:18:18.431Z++ dladm show-simnet -p -o LINK net1
5032023-10-21T17:18:18.457Z++ echo ''
5042023-10-21T17:18:18.481Z+ [[ -z '' ]]
5052023-10-21T17:18:18.506Z+ dladm create-simnet -t net1
5062023-10-21T17:18:18.530Z+ dladm create-simnet -t sc1_0
5072023-10-21T17:18:18.554Z+ dladm modify-simnet -t -p net1 sc1_0
5082023-10-21T17:18:18.578Z+ dladm set-linkprop -p mtu=9000 sc1_0
5092023-10-21T17:18:18.603Z+ success 'Simnet net1/sc1_0 exists'
5102023-10-21T17:18:18.627Z+ set +x
5112023-10-21T17:18:18.651Z++ get_vnic_name_if_exists sc0_1
5122023-10-21T17:18:18.675Z++ dladm show-vnic -p -o LINK sc0_1
5132023-10-21T17:18:18.699Z++ echo ''
5142023-10-21T17:18:18.723Z+ [[ -z '' ]]
5152023-10-21T17:18:18.747Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162023-10-21T17:18:18.771Z+ [[ -v PROMISC_FILT_OFF ]]
5172023-10-21T17:18:18.812Z+ success 'Vnic sc0_1 exists'
5182023-10-21T17:18:18.846Z+ set +x
5192023-10-21T17:18:19.279Z+ warn_if_no_proxy_arp
5202023-10-21T17:18:19.719Z+ [[ -v PXA_START ]]
5212023-10-21T17:18:19.743Z+ [[ -v PXA_END ]]
5222023-10-21T17:18:19.768Z+ ensure_softnpu_zone
5232023-10-21T17:18:21.082Z+ zoneadm list
5242023-10-21T17:18:21.112Z+ grep -q sidecar_softnpu
5252023-10-21T17:18:21.136Z+ [[ -f out/npuzone/npuzone ]]
5262023-10-21T17:18:21.161Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch omicron-tracking
5272023-10-21T17:18:21.294Zdownloading asic_program.so ...done
5282023-10-21T17:18:21.485Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/scadm.sha256.txt
5292023-10-21T17:18:28.007Zdownloading scadm ...done
5302023-10-21T17:18:28.032Zcreating port tfportrear0_0 for sc0_0
5312023-10-21T17:18:28.058Zcreating port tfportqsfp0_0 for sc0_1
5322023-10-21T17:18:28.090Zsoftnpu zone setup
5332023-10-21T17:18:28.115Zconfigure zone
5342023-10-21T17:18:28.140Zinstall zone
5352023-10-21T17:18:29.089Zboot zone
5362023-10-21T17:18:32.195Z[sidecar_softnpu] svcs milestone/network
5372023-10-21T17:18:32.222ZSTATE STIME FMRI
5382023-10-21T17:18:32.247Zonline 17:18:30 svc:/milestone/network:default
5392023-10-21T17:18:32.272Z[sidecar_softnpu] /softnpu/npu start
5402023-10-21T17:18:32.296Z
5412023-10-21T17:18:32.321Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5422023-10-21T17:18:32.351Z+ GATEWAY_IP=10.151.1.100
5432023-10-21T17:18:32.375Z+ echo 'Using 10.151.1.100 as gateway ip'
5442023-10-21T17:18:32.400ZUsing 10.151.1.100 as gateway ip
5452023-10-21T17:18:32.424Z+ [[ ! -v GATEWAY_MAC ]]
5462023-10-21T17:18:32.452Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5472023-10-21T17:18:32.478Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5482023-10-21T17:18:32.512Z+++ echo 18:c0:4d:0d:9f:b2
5492023-10-21T17:18:32.554Z+ COUNT=1
5502023-10-21T17:18:32.586Z+ [[ 1 -ne 1 ]]
5512023-10-21T17:18:32.612Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5522023-10-21T17:18:32.637ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5532023-10-21T17:18:32.661Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5542023-10-21T17:18:32.688Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5552023-10-21T17:18:32.712Z+ PXA_MAC=a8:e1:de:01:70:1d
5562023-10-21T17:18:32.741Z+ [[ -v PXA_START ]]
5572023-10-21T17:18:32.765Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5582023-10-21T17:18:32.790Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5592023-10-21T17:18:32.814Z+ z_scadm dump-state
5602023-10-21T17:18:32.838Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5612023-10-21T17:18:32.862Zlocal v6:
5622023-10-21T17:18:32.902Zlocal v4:
5632023-10-21T17:18:32.939Zrouter v6:
5642023-10-21T17:18:32.964Zrouter v4:
5652023-10-21T17:18:32.988Zresolver v4:
5662023-10-21T17:18:33.013Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5672023-10-21T17:18:33.037Zresolver v6:
5682023-10-21T17:18:33.061Znat v4:
5692023-10-21T17:18:33.089Znat_v6:
5702023-10-21T17:18:33.114Zport_mac:
5712023-10-21T17:18:33.139Zproxy arp:
5722023-10-21T17:18:33.164Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5732023-10-21T17:18:33.191Z+ success 'softnpu zone exists'
5742023-10-21T17:18:33.215Z+ set +x
5752023-10-21T17:18:33.240Zsoftnpu zone exists
5762023-10-21T17:18:33.264Z
5772023-10-21T17:18:33.289Zreal 27.075628928
5782023-10-21T17:18:33.314Zuser 0.694494299
5792023-10-21T17:18:33.338Zsys 1.568018320
5802023-10-21T17:18:33.376Ztrap 0.000083948
5812023-10-21T17:18:33.443Ztflt 0.000120005
5822023-10-21T17:18:33.467Zdflt 0.000665902
5832023-10-21T17:18:33.492Zkflt 0.000000000
5842023-10-21T17:18:33.517Zlock 21:10.926837712
5852023-10-21T17:18:33.541Zslp 1:20.459015971
5862023-10-21T17:18:33.566Zlat 0.071139877
5872023-10-21T17:18:33.618Zstop 0.367165929
5882023-10-21T17:18:33.642Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5892023-10-21T17:18:33.668Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5902023-10-21T17:18:33.692Z+ SILO_NAME=recovery
5912023-10-21T17:18:33.717Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5922023-10-21T17:18:33.743Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5932023-10-21T17:18:33.768Z+ sed -i~ '
5942023-10-21T17:18:33.801Z /^external_dns_ips/c\
5952023-10-21T17:18:33.827Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5962023-10-21T17:18:33.852Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5972023-10-21T17:18:33.876Z /^first/c\
5982023-10-21T17:18:33.900Zfirst = "10.151.1.164"
5992023-10-21T17:18:33.925Z /^last/c\
6002023-10-21T17:18:33.949Zlast = "10.151.1.173"
6012023-10-21T17:18:33.973Z }
6022023-10-21T17:18:34.407Z /^\[rack_network_config/,/^$/ {
6032023-10-21T17:18:34.847Z /^infra_ip_first/c\
6042023-10-21T17:18:34.873Zinfra_ip_first = "10.151.1.174"
6052023-10-21T17:18:34.901Z /^infra_ip_last/c\
6062023-10-21T17:18:34.937Zinfra_ip_last = "10.151.1.174"
6072023-10-21T17:18:34.962Z }
6082023-10-21T17:18:34.986Z /^\[\[rack_network_config.ports/,/^$/ {
6092023-10-21T17:18:35.011Z /^routes/c\
6102023-10-21T17:18:35.035Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6112023-10-21T17:18:35.060Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6122023-10-21T17:18:35.087Z+++ pkg/config-rss.toml Sat Oct 21 17:18:32 2023
6132023-10-21T17:18:35.111Z /^addresses/c\
6142023-10-21T17:18:35.135Z@@ -41,7 +41,7 @@
6152023-10-21T17:18:35.159Zaddresses = \["10.151.1.174/32"\]
6162023-10-21T17:18:35.185Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6172023-10-21T17:18:35.210Z }
6182023-10-21T17:18:35.234Z # must be contained in one of the "internal services" IP Pool ranges listed
6192023-10-21T17:18:35.259Z' pkg/config-rss.toml
6202023-10-21T17:18:35.283Z # below.
6212023-10-21T17:18:35.308Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6222023-10-21T17:18:35.332Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6232023-10-21T17:18:35.357Z+ true
6242023-10-21T17:18:35.383Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6252023-10-21T17:18:35.408Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6262023-10-21T17:18:35.442Z
6272023-10-21T17:18:35.467Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6282023-10-21T17:18:35.907Z # Initial TLS certificates for the external API
6292023-10-21T17:18:35.931Z+ TLS_NAME=recovery.sys.oxide.test
6302023-10-21T17:18:35.956Z #
6312023-10-21T17:18:35.980Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6322023-10-21T17:18:36.004Z@@ -77,8 +77,8 @@
6332023-10-21T17:18:36.029Z #
6342023-10-21T17:18:36.053Z # For more on this and what to put here, see docs/how-to-run.adoc.
6352023-10-21T17:18:36.077Z [[internal_services_ip_pool_ranges]]
6362023-10-21T17:18:36.102Z-first = "192.168.1.20"
6372023-10-21T17:18:36.126Z.+.....+....+.........+..+.+..+..........+......+......+.............................+..........+..+...+.+...............+.....+......+..........+............+..+...+...+....+.........+..+.........+.+..............+.........+......+.+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+.......+.....+...+.......+..+.+.........+.....+.......+...+...+........+....+...+..............+...+......+.......+.....+.........+....+..+.............+......+...+.....+.+..+.............+....................+.....................+...+......+....+...+.....+.......+...+............+.....+.......+........+...+...+.........+.+......+......+........+.+..................+..+....+............+..+..........+......+..+...............+.............+......+........+......+....+...........+......+......+.........+.+............+.....+...+.............+.........+...+...+...........+...+.......+..................+......+..+.......+..+......+.......+..+...................+.....+.+.....+.........+............+.........+...+...+.........+..............................+...+......+....+...+.....+...+.+..............+.+.....+.............+.....................+.....+....+..+.+...+...............+............+.................+.............+..+..........+...........+.......+........+..........+...............+.................+.........+.+..+.......+.........+...........+.+.....+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6382023-10-21T17:18:36.151Z-last = "192.168.1.29"
6392023-10-21T17:18:36.177Z+first = "10.151.1.164"
6402023-10-21T17:18:36.202Z+last = "10.151.1.173"
6412023-10-21T17:18:36.227Z
6422023-10-21T17:18:36.253Z # TODO - this configuration is subject to change going forward. Ultimately these
6432023-10-21T17:18:36.277Z # parameters should be provided to the control plane via wicket, but we need to
6442023-10-21T17:18:36.310Z@@ -92,8 +92,8 @@
6452023-10-21T17:18:36.351Z # A range of IP addresses used by Boundary Services on the external network. In
6462023-10-21T17:18:36.377Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6472023-10-21T17:18:36.402Z # With softnpu, only one address is used.
6482023-10-21T17:18:36.428Z-infra_ip_first = "192.168.1.30"
6492023-10-21T17:18:36.452Z-infra_ip_last = "192.168.1.30"
6502023-10-21T17:18:36.477Z+infra_ip_first = "10.151.1.174"
6512023-10-21T17:18:36.502Z+infra_ip_last = "10.151.1.174"
6522023-10-21T17:18:36.526Z
6532023-10-21T17:18:36.551Z # Configurations for BGP routers to run on the scrimlets.
6542023-10-21T17:18:36.575Z bgp = []
6552023-10-21T17:18:36.599Z@@ -101,9 +101,9 @@
6562023-10-21T17:18:36.624Z # You can configure multiple uplinks by repeating the following stanza
6572023-10-21T17:18:36.648Z [[rack_network_config.ports]]
6582023-10-21T17:18:36.698Z # Routes associated with this port.
6592023-10-21T17:18:36.723Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6602023-10-21T17:18:36.748Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6612023-10-21T17:18:36.776Z # Addresses associated with this port.
6622023-10-21T17:18:36.819Z-addresses = ["192.168.1.30/32"]
6632023-10-21T17:18:36.853Z+addresses = ["10.151.1.174/32"]
6642023-10-21T17:18:36.878Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6652023-10-21T17:18:36.902Z port = "qsfp0"
6662023-10-21T17:18:36.926Z # The speed of this port.
6672023-10-21T17:18:36.951Za pkg/config-rss.toml 7K
6682023-10-21T17:18:36.976Z...........+..........+.....+.+...........+.......+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.+.....+.+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......................+.....+....+.....+..........+..+..........+...........+.........+......+..........+............+.....................+.....+.+..............+.+........+.............+..+.+..+.......+..+.+........+.......+........+......+........................+.+...........+.......+...........+.+........+...+....+.....+.+............+..+..................+.+...........+....+..+.............+.....+.+......+..............+................+..+...+...+.+......+............+..+.+.........+.....+...+..........+.....+...+.........+...+...............+......................+...+..+...................+.........+....................+............+...+............+......+............+.......+......+............+.....+.......+...+.....+...+....+......+.........+.........+..............+...+...+......+.........+....+...+...+..............+.+..+.......+..+....+..+....+...+........+.......+...+...+......+......+............+...+...............+...+.....+..........+........................+.....+.............+...+......+.....+......+.........+.............+..+....+..+...+......+.+.....+............+.............+........+.+...+.....+.+...........+.........+....+..........................+..........+..+.........+..........+...+........+...+..............................+............+...+...................+..+.+..+..........+......+.....+...............+......+...+............+.+...+............+..+.............+...+..............+...+.......+......+............+..+......+.+......+...+...........+...+...+.......+..+......+...+.+.....+......+.................................+.+......+..+.......+......+..+......+.+..............+.+...........+.........+...................+..............................+......+...+..+...............+....+......+.....+.+.....+.........+...............+...+.+......+...+........+.......+...+..+.+.....................+......+...+.........+.....+...+.......+........+...+......+...+.+........................+...........+.........+............+....+...........+...+......+......+...............+......+....+...+......+............+........+.+......+..............+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6692023-10-21T17:18:37.001Za pkg/initial-tls-cert.pem 2K
6702023-10-21T17:18:37.025Z-----
6712023-10-21T17:18:37.050Za pkg/initial-tls-key.pem 4K
6722023-10-21T17:18:37.074Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6732023-10-21T17:18:37.102Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-10-21T17:18:37.126Z+ rmdir pkg
6752023-10-21T17:18:37.151Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6762023-10-21T17:18:37.175Z+ pfexec mkdir -p /usr/oxide
6772023-10-21T17:18:37.617Z+ pfexec rm -f /usr/oxide/tcpproxy
6782023-10-21T17:18:37.643Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6792023-10-21T17:18:37.667Z+ pfexec chmod +x /usr/oxide/tcpproxy
6802023-10-21T17:18:37.699Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6812023-10-21T17:18:37.726Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6822023-10-21T17:18:37.751Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6832023-10-21T17:18:37.776Z+ OMICRON_NO_UNINSTALL=1
6842023-10-21T17:18:37.801Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6852023-10-21T17:18:37.825ZOct 21 17:18:34.905 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6862023-10-21T17:18:37.867ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6872023-10-21T17:18:37.899ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6882023-10-21T17:18:37.923ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6892023-10-21T17:18:37.949ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6902023-10-21T17:18:37.973ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6912023-10-21T17:18:37.997ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6922023-10-21T17:18:38.022ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6932023-10-21T17:18:38.047ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6942023-10-21T17:18:38.072ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6952023-10-21T17:18:38.112ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6962023-10-21T17:18:38.137ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6972023-10-21T17:18:38.165ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6982023-10-21T17:18:38.191ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6992023-10-21T17:18:38.264ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7002023-10-21T17:18:38.289ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7012023-10-21T17:18:38.313ZOct 21 17:18:34.907 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7022023-10-21T17:18:38.338ZOct 21 17:18:35.142 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7032023-10-21T17:18:38.362ZOct 21 17:18:35.180 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7042023-10-21T17:18:38.387ZOct 21 17:18:35.243 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7052023-10-21T17:18:38.411Z
7062023-10-21T17:18:38.435Zreal 0.477031506
7072023-10-21T17:18:38.461Zuser 0.097488954
7082023-10-21T17:18:38.486Zsys 1.891369688
7092023-10-21T17:18:38.511Ztrap 0.000000420
7102023-10-21T17:18:38.535Ztflt 0.000013759
7112023-10-21T17:18:38.559Zdflt 0.000347917
7122023-10-21T17:18:38.583Zkflt 0.000003850
7132023-10-21T17:18:38.609Zlock 42.876161137
7142023-10-21T17:18:38.636Zslp 0.685939285
7152023-10-21T17:18:38.667Zlat 0.003082684
7162023-10-21T17:18:38.696Zstop 0.211732022
7172023-10-21T17:18:38.723Z+ retry=0
7182023-10-21T17:18:38.747Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7192023-10-21T17:18:38.771Z+ [[ 0 -gt 30 ]]
7202023-10-21T17:18:38.796Z+ sleep 1
7212023-10-21T17:18:38.820Z+ retry=1
7222023-10-21T17:18:38.844Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232023-10-21T17:18:38.868Z+ [[ 1 -gt 30 ]]
7242023-10-21T17:18:38.928Z+ sleep 1
7252023-10-21T17:18:38.953Z+ retry=2
7262023-10-21T17:18:38.977Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272023-10-21T17:19:02.577Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7282023-10-21T17:19:02.629Zlocal v6:
7292023-10-21T17:19:02.654Zfe80::3c08:4eff:fed1:f73b
7302023-10-21T17:19:02.678Zfd00:99::1
7312023-10-21T17:19:02.705Zlocal v4:
7322023-10-21T17:19:02.730Z10.151.1.174
7332023-10-21T17:19:02.755Zrouter v6:
7342023-10-21T17:19:02.780Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::b03d:c9ff:feb7:37a1 (1) vid=0
7352023-10-21T17:19:02.805Zfd00:1122:3344:101::/64 -> fe80::b03d:c9ff:feb7:37a1 (1) vid=0
7362023-10-21T17:19:02.830Zrouter v4:
7372023-10-21T17:19:02.857Z0.0.0.0/0 -> 10.151.1.100 (2) vid=0
7382023-10-21T17:19:02.882Zresolver v4:
7392023-10-21T17:19:02.907Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7402023-10-21T17:19:02.932Z10.151.1.174 -> b2:78:b2:a1:ee:15
7412023-10-21T17:19:02.958Zresolver v6:
7422023-10-21T17:19:02.982Zfe80::3c08:4eff:fed1:f73b -> 3e:08:4e:d1:f7:3b
7432023-10-21T17:19:03.007Zfe80::b03d:c9ff:feb7:37a1 -> b2:3d:c9:b7:37:a1
7442023-10-21T17:19:03.032Znat v4:
7452023-10-21T17:19:03.056Znat_v6:
7462023-10-21T17:19:03.081Zport_mac:
7472023-10-21T17:19:03.106Z1000: a8:40:25:7a:9d:38
7482023-10-21T17:19:03.130Z1: a8:40:25:7a:9d:39
7492023-10-21T17:19:03.156Zproxy arp:
7502023-10-21T17:19:03.180Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7512023-10-21T17:19:03.204Z+ retry=0
7522023-10-21T17:19:03.229Z+++ zoneadm list -n
7532023-10-21T17:19:03.254Z+++ grep oxz_ntp
7542023-10-21T17:19:03.289Z++ pfexec svcs -z -Hostate oxide/ntp
7552023-10-21T17:19:03.316Zsvcs: zone '-Hostate': entity not found
7562023-10-21T17:19:03.341Z++ true
7572023-10-21T17:19:03.365Z+ [[ '' != online ]]
7582023-10-21T17:19:03.391Z+ [[ 0 -gt 60 ]]
7592023-10-21T17:19:03.416Z+ sleep 1
7602023-10-21T17:19:03.653Z+ retry=1
7612023-10-21T17:19:03.678Z+++ zoneadm list -n
7622023-10-21T17:19:03.702Z+++ grep oxz_ntp
7632023-10-21T17:19:03.727Z++ pfexec svcs -z -Hostate oxide/ntp
7642023-10-21T17:19:03.751Zsvcs: zone '-Hostate': entity not found
7652023-10-21T17:19:03.776Z++ true
7662023-10-21T17:19:03.802Z+ [[ '' != online ]]
7672023-10-21T17:19:03.827Z+ [[ 1 -gt 60 ]]
7682023-10-21T17:19:03.852Z+ sleep 1
7692023-10-21T17:19:04.679Z+ retry=2
7702023-10-21T17:19:04.704Z+++ zoneadm list -n
7712023-10-21T17:19:04.728Z+++ grep oxz_ntp
7722023-10-21T17:19:04.753Z++ pfexec svcs -z -Hostate oxide/ntp
7732023-10-21T17:19:04.779Zsvcs: zone '-Hostate': entity not found
7742023-10-21T17:19:04.804Z++ true
7752023-10-21T17:19:04.828Z+ [[ '' != online ]]
7762023-10-21T17:19:04.853Z+ [[ 2 -gt 60 ]]
7772023-10-21T17:19:04.877Z+ sleep 1
7782023-10-21T17:19:05.680Z+ retry=3
7792023-10-21T17:19:05.706Z+++ zoneadm list -n
7802023-10-21T17:19:05.731Z+++ grep oxz_ntp
7812023-10-21T17:19:05.755Z++ pfexec svcs -z -Hostate oxide/ntp
7822023-10-21T17:19:05.780Zsvcs: zone '-Hostate': entity not found
7832023-10-21T17:19:05.809Z++ true
7842023-10-21T17:19:05.833Z+ [[ '' != online ]]
7852023-10-21T17:19:05.857Z+ [[ 3 -gt 60 ]]
7862023-10-21T17:19:05.882Z+ sleep 1
7872023-10-21T17:19:06.704Z+ retry=4
7882023-10-21T17:19:06.732Z+++ zoneadm list -n
7892023-10-21T17:19:06.756Z+++ grep oxz_ntp
7902023-10-21T17:19:06.780Z++ pfexec svcs -z -Hostate oxide/ntp
7912023-10-21T17:19:06.804Zsvcs: zone '-Hostate': entity not found
7922023-10-21T17:19:06.830Z++ true
7932023-10-21T17:19:06.857Z+ [[ '' != online ]]
7942023-10-21T17:19:06.880Z+ [[ 4 -gt 60 ]]
7952023-10-21T17:19:06.904Z+ sleep 1
7962023-10-21T17:19:07.726Z+ retry=5
7972023-10-21T17:19:07.756Z+++ zoneadm list -n
7982023-10-21T17:19:07.780Z+++ grep oxz_ntp
7992023-10-21T17:19:07.805Z++ pfexec svcs -z -Hostate oxide/ntp
8002023-10-21T17:19:07.828Zsvcs: zone '-Hostate': entity not found
8012023-10-21T17:19:07.852Z++ true
8022023-10-21T17:19:07.876Z+ [[ '' != online ]]
8032023-10-21T17:19:07.901Z+ [[ 5 -gt 60 ]]
8042023-10-21T17:19:07.925Z+ sleep 1
8052023-10-21T17:19:08.747Z+ retry=6
8062023-10-21T17:19:08.781Z+++ zoneadm list -n
8072023-10-21T17:19:08.810Z+++ grep oxz_ntp
8082023-10-21T17:19:08.834Z++ pfexec svcs -z -Hostate oxide/ntp
8092023-10-21T17:19:08.858Zsvcs: zone '-Hostate': entity not found
8102023-10-21T17:19:08.882Z++ true
8112023-10-21T17:19:08.906Z+ [[ '' != online ]]
8122023-10-21T17:19:08.930Z+ [[ 6 -gt 60 ]]
8132023-10-21T17:19:08.954Z+ sleep 1
8142023-10-21T17:19:09.768Z+ retry=7
8152023-10-21T17:19:09.804Z+++ zoneadm list -n
8162023-10-21T17:19:09.829Z+++ grep oxz_ntp
8172023-10-21T17:19:09.853Z++ pfexec svcs -z -Hostate oxide/ntp
8182023-10-21T17:19:09.877Zsvcs: zone '-Hostate': entity not found
8192023-10-21T17:19:09.901Z++ true
8202023-10-21T17:19:09.925Z+ [[ '' != online ]]
8212023-10-21T17:19:09.949Z+ [[ 7 -gt 60 ]]
8222023-10-21T17:19:09.974Z+ sleep 1
8232023-10-21T17:19:10.791Z+ retry=8
8242023-10-21T17:19:10.830Z+++ zoneadm list -n
8252023-10-21T17:19:10.854Z+++ grep oxz_ntp
8262023-10-21T17:19:10.878Z++ pfexec svcs -z -Hostate oxide/ntp
8272023-10-21T17:19:10.902Zsvcs: zone '-Hostate': entity not found
8282023-10-21T17:19:10.926Z++ true
8292023-10-21T17:19:10.951Z+ [[ '' != online ]]
8302023-10-21T17:19:10.974Z+ [[ 8 -gt 60 ]]
8312023-10-21T17:19:10.998Z+ sleep 1
8322023-10-21T17:19:11.813Z+ retry=9
8332023-10-21T17:19:11.855Z+++ zoneadm list -n
8342023-10-21T17:19:11.879Z+++ grep oxz_ntp
8352023-10-21T17:19:11.903Z++ pfexec svcs -z -Hostate oxide/ntp
8362023-10-21T17:19:11.927Zsvcs: zone '-Hostate': entity not found
8372023-10-21T17:19:11.951Z++ true
8382023-10-21T17:19:11.975Z+ [[ '' != online ]]
8392023-10-21T17:19:11.999Z+ [[ 9 -gt 60 ]]
8402023-10-21T17:19:12.027Z+ sleep 1
8412023-10-21T17:19:12.836Z+ retry=10
8422023-10-21T17:19:12.880Z+++ zoneadm list -n
8432023-10-21T17:19:12.904Z+++ grep oxz_ntp
8442023-10-21T17:19:12.928Z++ pfexec svcs -z oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf -Hostate oxide/ntp
8452023-10-21T17:19:12.952Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8462023-10-21T17:19:12.976Z++ true
8472023-10-21T17:19:13.002Z+ [[ '' != online ]]
8482023-10-21T17:19:13.026Z+ [[ 10 -gt 60 ]]
8492023-10-21T17:19:13.050Z+ sleep 1
8502023-10-21T17:19:13.905Z+ retry=11
8512023-10-21T17:19:13.930Z+++ zoneadm list -n
8522023-10-21T17:19:13.954Z+++ grep oxz_ntp
8532023-10-21T17:19:13.978Z++ pfexec svcs -z oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf -Hostate oxide/ntp
8542023-10-21T17:19:14.005Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8552023-10-21T17:19:14.467Z++ true
8562023-10-21T17:19:14.491Z+ [[ '' != online ]]
8572023-10-21T17:19:14.515Z+ [[ 11 -gt 60 ]]
8582023-10-21T17:19:14.538Z+ sleep 1
8592023-10-21T17:19:14.935Z+ retry=12
8602023-10-21T17:19:14.961Z+++ zoneadm list -n
8612023-10-21T17:19:14.989Z+++ grep oxz_ntp
8622023-10-21T17:19:15.017Z++ pfexec svcs -z oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf -Hostate oxide/ntp
8632023-10-21T17:19:15.046Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8642023-10-21T17:19:15.070Z++ true
8652023-10-21T17:19:15.095Z+ [[ '' != online ]]
8662023-10-21T17:19:15.119Z+ [[ 12 -gt 60 ]]
8672023-10-21T17:19:15.143Z+ sleep 1
8682023-10-21T17:19:15.965Z+ retry=13
8692023-10-21T17:19:15.994Z+++ zoneadm list -n
8702023-10-21T17:19:16.021Z+++ grep oxz_ntp
8712023-10-21T17:19:16.045Z++ pfexec svcs -z oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf -Hostate oxide/ntp
8722023-10-21T17:19:16.069Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8732023-10-21T17:19:16.093Z++ true
8742023-10-21T17:19:16.117Z+ [[ '' != online ]]
8752023-10-21T17:19:16.162Z+ [[ 13 -gt 60 ]]
8762023-10-21T17:19:16.186Z+ sleep 1
8772023-10-21T17:19:16.992Z+ retry=14
8782023-10-21T17:19:17.016Z+++ zoneadm list -n
8792023-10-21T17:19:17.041Z+++ grep oxz_ntp
8802023-10-21T17:19:17.065Z++ pfexec svcs -z oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf -Hostate oxide/ntp
8812023-10-21T17:19:17.091Z+ [[ disabled != online ]]
8822023-10-21T17:19:17.115Z+ [[ 14 -gt 60 ]]
8832023-10-21T17:19:17.139Z+ sleep 1
8842023-10-21T17:19:18.017Z+ retry=15
8852023-10-21T17:19:18.041Z+++ zoneadm list -n
8862023-10-21T17:19:18.065Z+++ grep oxz_ntp
8872023-10-21T17:19:18.092Z++ pfexec svcs -z oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf -Hostate oxide/ntp
8882023-10-21T17:19:18.119Z+ [[ disabled != online ]]
8892023-10-21T17:19:18.144Z+ [[ 15 -gt 60 ]]
8902023-10-21T17:19:18.168Z+ sleep 1
8912023-10-21T17:19:19.045Z+ retry=16
8922023-10-21T17:19:19.075Z+++ zoneadm list -n
8932023-10-21T17:19:19.099Z+++ grep oxz_ntp
8942023-10-21T17:19:19.123Z++ pfexec svcs -z oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf -Hostate oxide/ntp
8952023-10-21T17:19:19.147Z+ [[ disabled != online ]]
8962023-10-21T17:19:19.173Z+ [[ 16 -gt 60 ]]
8972023-10-21T17:19:19.624Z+ sleep 1
8982023-10-21T17:19:20.089Z+ retry=17
8992023-10-21T17:19:20.524Z+++ zoneadm list -n
9002023-10-21T17:19:20.966Z+++ grep oxz_ntp
9012023-10-21T17:19:20.998Z++ pfexec svcs -z oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf -Hostate oxide/ntp
9022023-10-21T17:19:21.044Z+ [[ disabled != online ]]
9032023-10-21T17:19:21.482Z+ [[ 17 -gt 60 ]]
9042023-10-21T17:19:21.536Z+ sleep 1
9052023-10-21T17:19:21.574Z+ retry=18
9062023-10-21T17:19:21.609Z+++ zoneadm list -n
9072023-10-21T17:19:21.638Z+++ grep oxz_ntp
9082023-10-21T17:19:21.662Z++ pfexec svcs -z oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf -Hostate oxide/ntp
9092023-10-21T17:19:21.686Z+ [[ disabled != online ]]
9102023-10-21T17:19:21.711Z+ [[ 18 -gt 60 ]]
9112023-10-21T17:19:21.735Z+ sleep 1
9122023-10-21T17:19:22.104Z+ retry=19
9132023-10-21T17:19:22.151Z+++ zoneadm list -n
9142023-10-21T17:19:22.176Z+++ grep oxz_ntp
9152023-10-21T17:19:22.200Z++ pfexec svcs -z oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf -Hostate oxide/ntp
9162023-10-21T17:19:22.224Z+ [[ disabled != online ]]
9172023-10-21T17:19:22.249Z+ [[ 19 -gt 60 ]]
9182023-10-21T17:19:22.273Z+ sleep 1
9192023-10-21T17:19:23.129Z+ retry=20
9202023-10-21T17:19:23.178Z+++ zoneadm list -n
9212023-10-21T17:19:23.205Z+++ grep oxz_ntp
9222023-10-21T17:19:23.231Z++ pfexec svcs -z oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf -Hostate oxide/ntp
9232023-10-21T17:19:23.260Z+ [[ disabled != online ]]
9242023-10-21T17:19:23.287Z+ [[ 20 -gt 60 ]]
9252023-10-21T17:19:23.311Z+ sleep 1
9262023-10-21T17:19:24.149Z+ retry=21
9272023-10-21T17:19:24.174Z+++ zoneadm list -n
9282023-10-21T17:19:24.203Z+++ grep oxz_ntp
9292023-10-21T17:19:24.229Z++ pfexec svcs -z oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf -Hostate oxide/ntp
9302023-10-21T17:19:24.255Z+ [[ disabled != online ]]
9312023-10-21T17:19:24.279Z+ [[ 21 -gt 60 ]]
9322023-10-21T17:19:24.303Z+ sleep 1
9332023-10-21T17:19:25.178Z+ retry=22
9342023-10-21T17:19:25.240Z+++ zoneadm list -n
9352023-10-21T17:19:25.272Z+++ grep oxz_ntp
9362023-10-21T17:19:25.303Z++ pfexec svcs -z oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf -Hostate oxide/ntp
9372023-10-21T17:19:25.331Z+ [[ disabled != online ]]
 
9382023-10-21T17:19:25.346Zbuskin console login: Oct 21 17:19:24 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9392023-10-21T17:19:25.360Z+ [[ 22 -gt 60 ]]
9402023-10-21T17:19:25.394Z+ sleep 1
9412023-10-21T17:19:26.195Z+ retry=23
9422023-10-21T17:19:26.223Z+++ zoneadm list -n
9432023-10-21T17:19:26.281Z+++ grep oxz_ntp
9442023-10-21T17:19:26.313Z++ pfexec svcs -z oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf -Hostate oxide/ntp
9452023-10-21T17:19:26.343Z+ [[ online != online ]]
9462023-10-21T17:19:26.370Z+ echo 'Waited for chrony: 23s'
9472023-10-21T17:19:26.395Z+ export RUST_BACKTRACE=1
9482023-10-21T17:19:26.419Z+ RUST_BACKTRACE=1
9492023-10-21T17:19:26.443ZWaited for chrony: 23s
9502023-10-21T17:19:26.467Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9512023-10-21T17:19:26.519Z+ ./tests/bootstrap
9522023-10-21T17:19:26.547ZUsing external DNS server at 10.151.1.164:53
9532023-10-21T17:19:26.577Z2023-10-21 17:19:26.246966533 UTC: attempting to log into API
9542023-10-21T17:19:41.289Z2023-10-21 17:19:41.278063139 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9552023-10-21T17:19:42.300Z2023-10-21 17:19:42.278973834 UTC: attempting to log into API
9562023-10-21T17:19:57.549Z2023-10-21 17:19:57.312489778 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9572023-10-21T17:19:58.347Z2023-10-21 17:19:58.312915926 UTC: attempting to log into API
9582023-10-21T17:20:13.358Z2023-10-21 17:20:13.345789630 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9592023-10-21T17:20:14.358Z2023-10-21 17:20:14.346921319 UTC: attempting to log into API
9602023-10-21T17:20:29.397Z2023-10-21 17:20:29.380859409 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9612023-10-21T17:20:30.395Z2023-10-21 17:20:30.382598717 UTC: attempting to log into API
 
9622023-10-21T17:20:36.959ZOct 21 17:20:36 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9632023-10-21T17:20:44.255ZOct 21 17:20:43 bukin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9642023-10-21T17:20:45.447Z2023-10-21 17:20:45.425694734 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9652023-10-21T17:20:46.437Z2023-10-21 17:20:46.426849854 UTC: attempting to log into API
 
9662023-10-21T17:20:51.561ZOct 21 17:20:50 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9672023-10-21T17:20:58.115ZOct 21 17:20:57 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9682023-10-21T17:21:01.471Z2023-10-21 17:21:01.460683702 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9692023-10-21T17:21:02.471Z2023-10-21 17:21:02.461458695 UTC: attempting to log into API
 
9702023-10-21T17:21:04.891ZOct 21 17:21:04 bukin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9712023-10-21T17:21:17.503Z2023-10-21 17:21:17.493750775 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9722023-10-21T17:21:18.504Z2023-10-21 17:21:18.494910454 UTC: attempting to log into API
9732023-10-21T17:21:24.295Z2023-10-21 17:21:24.283704375 UTC: login succeeded
9742023-10-21T17:21:24.327Zwaiting for nexus to come up...
9752023-10-21T17:21:24.389Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9762023-10-21T17:21:24.471Zensuring datasets are ready...
9772023-10-21T17:21:27.803Zlet's roll.
9782023-10-21T17:21:27.858Z+ rm ./tests/bootstrap
9792023-10-21T17:21:27.890Z+ for test_bin in tests/*
9802023-10-21T17:21:27.918Z+ ./tests/bootstrap-49e31e75b930b622
9812023-10-21T17:21:27.948Z
9822023-10-21T17:21:27.986Zrunning 0 tests
9832023-10-21T17:21:28.024Z
9842023-10-21T17:21:28.050Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9852023-10-21T17:21:28.085Z
9862023-10-21T17:21:28.111Z+ for test_bin in tests/*
9872023-10-21T17:21:28.134Z+ ./tests/end_to_end_tests-a91cf9af3457d280
9882023-10-21T17:21:28.164Z
9892023-10-21T17:21:28.188Zrunning 2 tests
9902023-10-21T17:21:28.214Ztest no_spoof::no_spoof_here ... ok
9912023-10-21T17:22:27.858Ztest instance_launch::instance_launch has been running for over 60 seconds
9922023-10-21T17:24:26.957Ztest instance_launch::instance_launch ... ok
9932023-10-21T17:24:26.983Z
9942023-10-21T17:24:27.007Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 179.10s
9952023-10-21T17:24:27.035Z
9962023-10-21T17:24:27.058Z+ _exit_trap
9972023-10-21T17:24:27.085Z+ local status=0
9982023-10-21T17:24:27.114Z+ [[ 0 -eq 0 ]]
9992023-10-21T17:24:27.138Z+ exit 0
10002023-10-21T17:24:27.162Zprocess exited: duration 398111 ms, exit code 0
 
10012023-10-21T17:24:27.245Zfound 47 output files
10022023-10-21T17:24:27.295Zuploading: /var/svc/log/oxide-sled-agent:default.log (442745 bytes)
10032023-10-21T17:24:27.508Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 442745 -> 443145
10042023-10-21T17:24:28.602Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10052023-10-21T17:24:28.630Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_6781ac1a-5cf8-4254-8488-5bf9670ead92/root/var/svc/log/oxide-internal_dns:default.log (99076 bytes)
10062023-10-21T17:24:29.872Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_6781ac1a-5cf8-4254-8488-5bf9670ead92/root/var/svc/log/oxide-internal_dns:default.log
10072023-10-21T17:24:29.896Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf/root/var/svc/log/oxide-ntp:default.log (783 bytes)
10082023-10-21T17:24:30.964Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf/root/var/svc/log/oxide-ntp:default.log
10092023-10-21T17:24:30.993Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_56f34b31-da43-4675-8279-581de3be7384/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10102023-10-21T17:24:32.078Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_56f34b31-da43-4675-8279-581de3be7384/root/var/svc/log/oxide-crucible-agent:default.log
10112023-10-21T17:24:32.104Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_56f34b31-da43-4675-8279-581de3be7384/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10122023-10-21T17:24:33.171Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_56f34b31-da43-4675-8279-581de3be7384/root/var/svc/log/oxide-crucible-downstairs:default.log
10132023-10-21T17:24:33.195Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bfebc1e-98c3-4d58-ab01-72e5cbef7726/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10142023-10-21T17:24:34.266Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bfebc1e-98c3-4d58-ab01-72e5cbef7726/root/var/svc/log/oxide-crucible-agent:default.log
10152023-10-21T17:24:34.290Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bfebc1e-98c3-4d58-ab01-72e5cbef7726/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10162023-10-21T17:24:35.359Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bfebc1e-98c3-4d58-ab01-72e5cbef7726/root/var/svc/log/oxide-crucible-downstairs:default.log
10172023-10-21T17:24:35.384Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4ae87c9e-f0f8-4b36-93a9-e9fd4dacc4d6/root/var/svc/log/oxide-external_dns:default.log (15612 bytes)
10182023-10-21T17:24:36.519Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4ae87c9e-f0f8-4b36-93a9-e9fd4dacc4d6/root/var/svc/log/oxide-external_dns:default.log
10192023-10-21T17:24:37.006Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_a1e208a4-26c3-4301-8068-c399578316ef/root/var/svc/log/oxide-oximeter:default.log (180886 bytes)
10202023-10-21T17:24:37.032Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_a1e208a4-26c3-4301-8068-c399578316ef/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 180886 -> 189101
10212023-10-21T17:24:37.770Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_a1e208a4-26c3-4301-8068-c399578316ef/root/var/svc/log/oxide-oximeter:default.log
10222023-10-21T17:24:37.794Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_315062eb-4535-4068-90b1-47746a296c66/root/var/svc/log/oxide-clickhouse:default.log (650823 bytes)
10232023-10-21T17:24:38.468Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_315062eb-4535-4068-90b1-47746a296c66/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 650823 -> 679586
10242023-10-21T17:24:39.162Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_315062eb-4535-4068-90b1-47746a296c66/root/var/svc/log/oxide-clickhouse:default.log
10252023-10-21T17:24:39.195Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_540d66d1-e2b9-49af-aa03-11b43d103dd5/root/var/svc/log/oxide-crucible-agent:default.log (24397 bytes)
10262023-10-21T17:24:40.736Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_540d66d1-e2b9-49af-aa03-11b43d103dd5/root/var/svc/log/oxide-crucible-agent:default.log
10272023-10-21T17:24:40.770Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_540d66d1-e2b9-49af-aa03-11b43d103dd5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10282023-10-21T17:24:41.849Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_540d66d1-e2b9-49af-aa03-11b43d103dd5/root/var/svc/log/oxide-crucible-downstairs:default.log
10292023-10-21T17:24:41.879Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_540d66d1-e2b9-49af-aa03-11b43d103dd5/root/var/svc/log/oxide-crucible-downstairs:downstairs-c85ae805-be5a-457a-a4aa-870c4300d481.log (5534 bytes)
10302023-10-21T17:24:43.444Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_540d66d1-e2b9-49af-aa03-11b43d103dd5/root/var/svc/log/oxide-crucible-downstairs:downstairs-c85ae805-be5a-457a-a4aa-870c4300d481.log
10312023-10-21T17:24:43.470Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_540d66d1-e2b9-49af-aa03-11b43d103dd5/root/var/svc/log/oxide-crucible-downstairs:downstairs-c8d38ee6-5a2b-4138-a61e-af7406f65990.log (3049 bytes)
10322023-10-21T17:24:44.567Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_540d66d1-e2b9-49af-aa03-11b43d103dd5/root/var/svc/log/oxide-crucible-downstairs:downstairs-c8d38ee6-5a2b-4138-a61e-af7406f65990.log
10332023-10-21T17:24:44.601Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cefd3bbd-6e1c-4cc5-939e-624201378420/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10342023-10-21T17:24:45.702Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cefd3bbd-6e1c-4cc5-939e-624201378420/root/var/svc/log/oxide-crucible-agent:default.log
10352023-10-21T17:24:45.737Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cefd3bbd-6e1c-4cc5-939e-624201378420/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10362023-10-21T17:24:46.811Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cefd3bbd-6e1c-4cc5-939e-624201378420/root/var/svc/log/oxide-crucible-downstairs:default.log
10372023-10-21T17:24:46.837Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_3cc9794e-cb06-48bb-8380-3b36a7caf6d9/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10382023-10-21T17:24:47.920Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_3cc9794e-cb06-48bb-8380-3b36a7caf6d9/root/var/svc/log/oxide-crucible-pantry:default.log
10392023-10-21T17:24:47.951Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_84f51e7e-4a6a-40b2-89f5-4ac7c16a2470/root/var/svc/log/oxide-internal_dns:default.log (103944 bytes)
10402023-10-21T17:24:49.312Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_84f51e7e-4a6a-40b2-89f5-4ac7c16a2470/root/var/svc/log/oxide-internal_dns:default.log
10412023-10-21T17:24:49.337Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d98bfd1b-cefc-4fdf-8de5-0c6ab1cac9d6/root/var/svc/log/oxide-crucible-agent:default.log (25255 bytes)
10422023-10-21T17:24:50.494Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d98bfd1b-cefc-4fdf-8de5-0c6ab1cac9d6/root/var/svc/log/oxide-crucible-agent:default.log
10432023-10-21T17:24:50.518Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d98bfd1b-cefc-4fdf-8de5-0c6ab1cac9d6/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10442023-10-21T17:24:51.586Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d98bfd1b-cefc-4fdf-8de5-0c6ab1cac9d6/root/var/svc/log/oxide-crucible-downstairs:default.log
10452023-10-21T17:24:51.612Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d98bfd1b-cefc-4fdf-8de5-0c6ab1cac9d6/root/var/svc/log/oxide-crucible-downstairs:downstairs-095e0d3d-60e7-49ae-b9f1-aa29ebf1c4f1.log (3051 bytes)
10462023-10-21T17:24:52.715Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d98bfd1b-cefc-4fdf-8de5-0c6ab1cac9d6/root/var/svc/log/oxide-crucible-downstairs:downstairs-095e0d3d-60e7-49ae-b9f1-aa29ebf1c4f1.log
10472023-10-21T17:24:52.739Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d98bfd1b-cefc-4fdf-8de5-0c6ab1cac9d6/root/var/svc/log/oxide-crucible-downstairs:downstairs-ef262c8a-c7fb-4127-a8aa-25b1a3e8eece.log (5536 bytes)
10482023-10-21T17:24:53.830Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d98bfd1b-cefc-4fdf-8de5-0c6ab1cac9d6/root/var/svc/log/oxide-crucible-downstairs:downstairs-ef262c8a-c7fb-4127-a8aa-25b1a3e8eece.log
10492023-10-21T17:24:53.854Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_94339dd6-2a9e-4a76-8372-a1279cbc33dd/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10502023-10-21T17:24:54.924Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_94339dd6-2a9e-4a76-8372-a1279cbc33dd/root/var/svc/log/oxide-crucible-pantry:default.log
10512023-10-21T17:24:54.948Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_5466fb83-3627-4cdc-8964-d67e77ee7fb0/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
10522023-10-21T17:24:56.027Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_5466fb83-3627-4cdc-8964-d67e77ee7fb0/root/var/svc/log/oxide-cockroachdb:default.log
10532023-10-21T17:24:56.051Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_7b1edb6b-ca9c-4081-b7db-10a84e008604/root/var/svc/log/oxide-cockroachdb:default.log (5366 bytes)
10542023-10-21T17:24:57.121Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_7b1edb6b-ca9c-4081-b7db-10a84e008604/root/var/svc/log/oxide-cockroachdb:default.log
10552023-10-21T17:24:57.145Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_c6fd47de-05a1-47aa-b2a5-6e755236fdae/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10562023-10-21T17:24:58.230Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_c6fd47de-05a1-47aa-b2a5-6e755236fdae/root/var/svc/log/oxide-crucible-pantry:default.log
10572023-10-21T17:24:58.261Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_e2986a5e-5a4a-4ea2-b8eb-a180f669ff4f/root/var/svc/log/oxide-external_dns:default.log (15607 bytes)
10582023-10-21T17:24:59.390Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_e2986a5e-5a4a-4ea2-b8eb-a180f669ff4f/root/var/svc/log/oxide-external_dns:default.log
10592023-10-21T17:24:59.414Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_81b58c34-c9f6-4010-a244-f19a1f147407/root/var/svc/log/oxide-nexus:default.log (562827 bytes)
10602023-10-21T17:25:00.189Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_81b58c34-c9f6-4010-a244-f19a1f147407/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 562827 -> 564651
10612023-10-21T17:25:01.269Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_81b58c34-c9f6-4010-a244-f19a1f147407/root/var/svc/log/oxide-nexus:default.log
10622023-10-21T17:25:01.303Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_c794c01e-a8b8-4e7e-a838-b95a384b7d27/root/var/svc/log/oxide-nexus:default.log (484848 bytes)
10632023-10-21T17:25:02.375Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_c794c01e-a8b8-4e7e-a838-b95a384b7d27/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 484848 -> 486672
10642023-10-21T17:25:03.457Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_c794c01e-a8b8-4e7e-a838-b95a384b7d27/root/var/svc/log/oxide-nexus:default.log
10652023-10-21T17:25:03.485Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_c7d0803c-7204-455e-b287-d78706a7192c/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
10662023-10-21T17:25:04.562Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_c7d0803c-7204-455e-b287-d78706a7192c/root/var/svc/log/oxide-cockroachdb:default.log
10672023-10-21T17:25:04.587Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_24919ab7-a3ff-49f2-8a8c-60dbae5e9f40/root/var/svc/log/oxide-crucible-agent:default.log (14584 bytes)
10682023-10-21T17:25:05.715Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_24919ab7-a3ff-49f2-8a8c-60dbae5e9f40/root/var/svc/log/oxide-crucible-agent:default.log
10692023-10-21T17:25:06.159Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_24919ab7-a3ff-49f2-8a8c-60dbae5e9f40/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10702023-10-21T17:25:07.379Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_24919ab7-a3ff-49f2-8a8c-60dbae5e9f40/root/var/svc/log/oxide-crucible-downstairs:default.log
10712023-10-21T17:25:07.411Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_24919ab7-a3ff-49f2-8a8c-60dbae5e9f40/root/var/svc/log/oxide-crucible-downstairs:downstairs-cfca498b-73cc-4871-8bb6-ce7a1f026243.log (3051 bytes)
10722023-10-21T17:25:07.952Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_24919ab7-a3ff-49f2-8a8c-60dbae5e9f40/root/var/svc/log/oxide-crucible-downstairs:downstairs-cfca498b-73cc-4871-8bb6-ce7a1f026243.log
10732023-10-21T17:25:07.990Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_c9f43b2a-a2e7-45c3-a9c7-d6412f04a757/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10742023-10-21T17:25:09.493Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_c9f43b2a-a2e7-45c3-a9c7-d6412f04a757/root/var/svc/log/oxide-crucible-agent:default.log
10752023-10-21T17:25:09.518Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_c9f43b2a-a2e7-45c3-a9c7-d6412f04a757/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10762023-10-21T17:25:10.591Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_c9f43b2a-a2e7-45c3-a9c7-d6412f04a757/root/var/svc/log/oxide-crucible-downstairs:default.log
10772023-10-21T17:25:10.618Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ed5b84c3-afde-4dee-a1b9-1beac74019df/root/var/svc/log/oxide-crucible-agent:default.log (13686 bytes)
10782023-10-21T17:25:11.720Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ed5b84c3-afde-4dee-a1b9-1beac74019df/root/var/svc/log/oxide-crucible-agent:default.log
10792023-10-21T17:25:11.756Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ed5b84c3-afde-4dee-a1b9-1beac74019df/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10802023-10-21T17:25:12.840Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ed5b84c3-afde-4dee-a1b9-1beac74019df/root/var/svc/log/oxide-crucible-downstairs:default.log
10812023-10-21T17:25:12.874Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ed5b84c3-afde-4dee-a1b9-1beac74019df/root/var/svc/log/oxide-crucible-downstairs:downstairs-24ff5adc-1853-4203-87a5-22863762d673.log (5536 bytes)
10822023-10-21T17:25:13.984Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ed5b84c3-afde-4dee-a1b9-1beac74019df/root/var/svc/log/oxide-crucible-downstairs:downstairs-24ff5adc-1853-4203-87a5-22863762d673.log
10832023-10-21T17:25:14.026Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_56e3551c-05ec-4be0-b5eb-93986572f140/root/var/svc/log/oxide-nexus:default.log (4570047 bytes)
10842023-10-21T17:25:16.546Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_56e3551c-05ec-4be0-b5eb-93986572f140/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4570047 -> 4631276
10852023-10-21T17:25:17.629Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_56e3551c-05ec-4be0-b5eb-93986572f140/root/var/svc/log/oxide-nexus:default.log
10862023-10-21T17:25:17.655Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_5c8cf79f-14b4-477c-b047-2aab16e5cbd3/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10872023-10-21T17:25:18.726Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_5c8cf79f-14b4-477c-b047-2aab16e5cbd3/root/var/svc/log/oxide-crucible-agent:default.log
10882023-10-21T17:25:18.750Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_5c8cf79f-14b4-477c-b047-2aab16e5cbd3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10892023-10-21T17:25:19.820Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_5c8cf79f-14b4-477c-b047-2aab16e5cbd3/root/var/svc/log/oxide-crucible-downstairs:default.log
10902023-10-21T17:25:19.846Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_ec283ed9-6ec8-49b5-950f-4ad017fb4124/root/var/svc/log/oxide-internal_dns:default.log (101989 bytes)
10912023-10-21T17:25:20.052Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_ec283ed9-6ec8-49b5-950f-4ad017fb4124/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 101989 -> 105746
10922023-10-21T17:25:21.121Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_ec283ed9-6ec8-49b5-950f-4ad017fb4124/root/var/svc/log/oxide-internal_dns:default.log
10932023-10-21T17:25:21.150Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2e24bea2-1bd3-442c-b429-7fad5e1476e1/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
10942023-10-21T17:25:22.219Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2e24bea2-1bd3-442c-b429-7fad5e1476e1/root/var/svc/log/oxide-cockroachdb:default.log
10952023-10-21T17:25:22.244Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_94a6202e-694a-4242-b8e1-96fcd32a23d3/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10962023-10-21T17:25:23.314Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_94a6202e-694a-4242-b8e1-96fcd32a23d3/root/var/svc/log/oxide-cockroachdb:default.log
10972023-10-21T17:25:23.342Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf/root/var/log/chrony/measurements.log (9042 bytes)
10982023-10-21T17:25:23.392Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf/root/var/log/chrony/measurements.log" changed size mid upload: 9042 -> 10549
10992023-10-21T17:25:24.464Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf/root/var/log/chrony/measurements.log
11002023-10-21T17:25:24.488Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf/root/var/log/chrony/statistics.log (5850 bytes)
11012023-10-21T17:25:24.512Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf/root/var/log/chrony/statistics.log" changed size mid upload: 5850 -> 6786
11022023-10-21T17:25:25.581Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf/root/var/log/chrony/statistics.log
11032023-10-21T17:25:25.611Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf/root/var/log/chrony/tracking.log (1323 bytes)
11042023-10-21T17:25:25.636Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf/root/var/log/chrony/tracking.log" changed size mid upload: 1323 -> 1455
11052023-10-21T17:25:26.707Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3c13f373-0476-4552-974e-9e12dc2c1faf/root/var/log/chrony/tracking.log