01HAR4T6TWPXVSGQEY1MKCE94R: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.23",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HAR4VDJ9WJMBABKMEW9BMZA6

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-09-20T02:50:46.116Zjob dependencies complete; ready to run (waiting for 54 m 8 s)
22023-09-20T03:42:05.018Zjob assigned to worker 01HARAV9GKDRMQAGQX6DRRJP1Q (queued for 51 m 18 s)
32023-09-20T03:42:05.107Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-09-20T03:42:05.157Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-09-20T03:42:05.193Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-09-20T03:42:05.227Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-09-20T03:42:05.271Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-09-20T03:42:05.305ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-09-20T03:42:05.338ZOxide Helios Version helios-2.0.22085 64-bit
102023-09-20T03:42:05.371ZConfiguring device.
112023-09-20T03:42:05.405Z##################################################
122023-09-20T03:42:05.438Z#### ############# ##
132023-09-20T03:42:05.874Z### ### ############ ##
142023-09-20T03:42:05.905Z## ### # ## ### ## Oxide Computer Company ##
152023-09-20T03:42:05.938Z## ## ## ### # ### ##
162023-09-20T03:42:05.972Z## # ### #### #### This Station Under ##
172023-09-20T03:42:06.005Z### ### #### # ### Computer Control ##
182023-09-20T03:42:06.040Z#### #### ### ## ##
192023-09-20T03:42:06.072Z##################################################
202023-09-20T03:42:06.105ZHostname: buskin
212023-09-20T03:42:06.139ZLoading smf(7) service descriptions:
222023-09-20T03:42:06.503Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-49c0192018c458f0.gz
232023-09-20T03:42:08.364Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-49c0192018c458f0.gz
242023-09-20T03:42:08.388Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
252023-09-20T03:42:27.090Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-09-20T03:42:27.115Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a5dac3003865f64e.gz
272023-09-20T03:42:48.625Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a5dac3003865f64e.gz
282023-09-20T03:42:48.649Zdownloading input: /input/ci-tools/work/caboose-util.gz
292023-09-20T03:42:49.732Zdownloaded input: /input/ci-tools/work/caboose-util.gz
302023-09-20T03:42:49.756Zdownloading input: /input/ci-tools/work/tufaceous.gz
312023-09-20T03:42:52.652Zdownloaded input: /input/ci-tools/work/tufaceous.gz
322023-09-20T03:42:52.678Zdownloading input: /input/package/work/version.txt
332023-09-20T03:42:52.704Zdownloaded input: /input/package/work/version.txt
342023-09-20T03:42:52.729Zdownloading input: /input/package/work/package.tar.gz
352023-09-20T03:43:08.852Zdownloaded input: /input/package/work/package.tar.gz
362023-09-20T03:43:08.876Zdownloading input: /input/package/work/global-zone-packages.tar.gz
372023-09-20T03:43:38.617Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
382023-09-20T03:43:38.643Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
392023-09-20T03:43:45.725Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-09-20T03:43:45.753Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
412023-09-20T03:44:19.875Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
422023-09-20T03:44:19.899Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
432023-09-20T03:44:53.477Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-09-20T03:44:53.501Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
452023-09-20T03:45:17.561Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
462023-09-20T03:45:17.586Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
472023-09-20T03:45:22.468Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
482023-09-20T03:45:22.494Zdownloading input: /input/package/work/zones/crucible.tar.gz
492023-09-20T03:45:30.266Zdownloaded input: /input/package/work/zones/crucible.tar.gz
502023-09-20T03:45:30.489Zdownloading input: /input/package/work/zones/external-dns.tar.gz
512023-09-20T03:45:43.264Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
522023-09-20T03:45:51.480Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
532023-09-20T03:45:56.585Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
542023-09-20T03:45:56.610Zdownloading input: /input/package/work/zones/ntp.tar.gz
552023-09-20T03:45:56.643Zdownloaded input: /input/package/work/zones/ntp.tar.gz
562023-09-20T03:45:56.674Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
572023-09-20T03:46:00.327Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-09-20T03:46:00.378Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
592023-09-20T03:46:03.483Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-09-20T03:46:03.507Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
612023-09-20T03:46:16.075Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
622023-09-20T03:46:16.099Zdownloading input: /input/package/work/zones/overlay.tar.gz
632023-09-20T03:46:16.124Zdownloaded input: /input/package/work/zones/overlay.tar.gz
642023-09-20T03:46:16.147Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
652023-09-20T03:46:20.738Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
662023-09-20T03:46:20.763Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
672023-09-20T03:46:35.753Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
682023-09-20T03:46:35.777Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
692023-09-20T03:47:17.507Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
702023-09-20T03:47:17.531Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
712023-09-20T03:47:45.213Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
722023-09-20T03:47:45.238Zstarting task 0: "setup"
732023-09-20T03:47:45.272Z++ uname -s
742023-09-20T03:47:45.297Z+ kern=SunOS
752023-09-20T03:47:45.322Z+ case "$kern" in
762023-09-20T03:47:45.346Z+ groupadd -g 12345 build
772023-09-20T03:47:45.370Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
782023-09-20T03:47:47.288Z+ zfs create -o mountpoint=/work rpool/work
792023-09-20T03:47:47.317Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
802023-09-20T03:47:47.342Z+ home_fs=zfs
812023-09-20T03:47:47.377Z+ [[ zfs == autofs ]]
822023-09-20T03:47:47.403Z+ mkdir -p /home/build
832023-09-20T03:47:47.427Z+ chown build:build /home/build /work
842023-09-20T03:47:49.315Z+ chmod 0700 /home/build /work
852023-09-20T03:47:49.351Zprocess exited: duration 4056 ms, exit code 0
 
862023-09-20T03:47:49.410Zstarting task 1: "authentication"
872023-09-20T03:47:49.484Zprocess exited: duration 46 ms, exit code 0
 
882023-09-20T03:47:49.537Zstarting task 2: "build"
892023-09-20T03:47:49.568Z+ trap _exit_trap EXIT
902023-09-20T03:47:49.592Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
912023-09-20T03:47:49.763Zsecpolicy_fs_linkdir: 0x55 = 0x55
922023-09-20T03:47:49.788Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
932023-09-20T03:47:49.812Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
942023-09-20T03:47:49.836Zsecpolicy_fs_linkdir+3: 0 = 0xe5
952023-09-20T03:47:49.866Zsecpolicy_fs_linkdir+4: 0 = 0xb8
962023-09-20T03:47:50.333Zsecpolicy_fs_linkdir+5: 0 = 0x1
972023-09-20T03:47:50.366Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
982023-09-20T03:47:50.393Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
992023-09-20T03:47:50.423Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1002023-09-20T03:47:50.893Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1012023-09-20T03:47:51.790Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1022023-09-20T03:47:51.814Z+ [[ -d /opt/oxide ]]
1032023-09-20T03:47:51.839Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1042023-09-20T03:47:51.864Z+ pfexec mkdir /opt/oxide
1052023-09-20T03:47:52.088Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1062023-09-20T03:47:52.118Z+ [[ -d /opt/oxide-underneath ]]
1072023-09-20T03:47:52.142Z+ cd /opt/oxide-underneath
1082023-09-20T03:47:52.166Z+ pfexec tar ceEp@/f - .
1092023-09-20T03:47:52.190Z+ cd /opt/oxide
1102023-09-20T03:47:52.214Z+ pfexec tar xveEp@/f -
1112023-09-20T03:47:52.238Zx ., 0 bytes, 0 tape blocks
1122023-09-20T03:47:52.261Zx ./opte, 0 bytes, 0 tape blocks
1132023-09-20T03:47:52.285Zx ./opte/bin, 0 bytes, 0 tape blocks
1142023-09-20T03:47:52.310Zx ./opte/bin/opteadm, 46195888 bytes, 90227 tape blocks
1152023-09-20T03:47:52.336Z+ pfexec rm -rf /opt/oxide-underneath
1162023-09-20T03:47:52.360Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1172023-09-20T03:47:52.384Z+ pfexec mkdir /opt/oxide/work
1182023-09-20T03:47:52.408Z+ pfexec chown build:build /opt/oxide/work
1192023-09-20T03:47:52.432Z+ cd /opt/oxide/work
1202023-09-20T03:47:52.456Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1212023-09-20T03:47:52.480ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1222023-09-20T03:47:52.504Zx out/maghemite.tar, 74957312 bytes, 146401 tape blocks
1232023-09-20T03:47:52.531Zx out/omicron-sled-agent.tar, 112352256 bytes, 219438 tape blocks
1242023-09-20T03:47:52.556Zx out/target/test, 49 bytes, 1 tape blocks
1252023-09-20T03:47:52.582Zx out/npuzone/npuzone, 36941864 bytes, 72153 tape blocks
1262023-09-20T03:47:52.608Zx package-manifest.toml, 20968 bytes, 41 tape blocks
1272023-09-20T03:47:52.632Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1282023-09-20T03:47:52.656Zx target/release/omicron-package, 37700856 bytes, 73635 tape blocks
1292023-09-20T03:47:52.680Zx tools/create_virtual_hardware.sh, 2806 bytes, 6 tape blocks
1302023-09-20T03:47:52.704Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1312023-09-20T03:47:52.728Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1322023-09-20T03:47:52.752Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1332023-09-20T03:47:52.776Zx tools/scrimlet/softnpu-init.sh, 2128 bytes, 5 tape blocks
1342023-09-20T03:47:52.800Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1352023-09-20T03:47:52.824Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1362023-09-20T03:47:52.848Z
1372023-09-20T03:47:52.872Zreal 1.595564407
1382023-09-20T03:47:52.895Zuser 1.434565632
1392023-09-20T03:47:52.924Zsys 0.286379010
1402023-09-20T03:47:52.948Ztrap 0.000079232
1412023-09-20T03:47:52.972Ztflt 0.000000000
1422023-09-20T03:47:52.996Zdflt 0.000000000
1432023-09-20T03:47:53.428Zkflt 0.000000000
1442023-09-20T03:47:53.453Zlock 0.000000000
1452023-09-20T03:47:53.477Zslp 1.433140317
1462023-09-20T03:47:53.501Zlat 0.035043224
1472023-09-20T03:47:53.524Zstop 0.000119581
1482023-09-20T03:47:53.548Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1492023-09-20T03:47:53.573Z+ mkdir tests
1502023-09-20T03:47:53.598Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1512023-09-20T03:47:53.622Z+ ptime -m gunzip
1522023-09-20T03:47:53.646Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-49c0192018c458f0
1532023-09-20T03:47:53.670Z
1542023-09-20T03:47:53.693Zreal 0.137403804
1552023-09-20T03:47:53.717Zuser 0.127052213
1562023-09-20T03:47:53.741Zsys 0.010120017
1572023-09-20T03:47:53.766Ztrap 0.000013349
1582023-09-20T03:47:53.789Ztflt 0.000000000
1592023-09-20T03:47:53.813Zdflt 0.000000000
1602023-09-20T03:47:53.844Zkflt 0.000000000
1612023-09-20T03:47:53.871Zlock 0.000000000
1622023-09-20T03:47:53.895Zslp 0.000018340
1632023-09-20T03:47:53.919Zlat 0.000021940
1642023-09-20T03:47:53.942Zstop 0.000118572
1652023-09-20T03:47:53.966Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-49c0192018c458f0
1662023-09-20T03:47:53.990Z+ chmod a+x tests/bootstrap-49c0192018c458f0
1672023-09-20T03:47:54.014Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1682023-09-20T03:47:54.038Z+ ptime -m gunzip
1692023-09-20T03:47:54.062Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1702023-09-20T03:47:54.734Z
1712023-09-20T03:47:54.758Zreal 2.137172922
1722023-09-20T03:47:54.782Zuser 2.018167474
1732023-09-20T03:47:54.806Zsys 0.118610689
1742023-09-20T03:47:54.834Ztrap 0.000153914
1752023-09-20T03:47:54.861Ztflt 0.000000000
1762023-09-20T03:47:54.885Zdflt 0.000000000
1772023-09-20T03:47:54.909Zkflt 0.000000000
1782023-09-20T03:47:54.935Zlock 0.000000000
1792023-09-20T03:47:54.961Zslp 0.000030450
1802023-09-20T03:47:54.984Zlat 0.000030251
1812023-09-20T03:47:55.008Zstop 0.000118802
1822023-09-20T03:47:55.032Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1832023-09-20T03:47:55.056Z+ chmod a+x tests/bootstrap
1842023-09-20T03:47:55.080Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1852023-09-20T03:47:55.106Z+ ptime -m gunzip
1862023-09-20T03:47:55.130Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a5dac3003865f64e
1872023-09-20T03:47:57.193Z
1882023-09-20T03:47:57.219Zreal 2.446556687
1892023-09-20T03:47:57.243Zuser 2.298453431
1902023-09-20T03:47:57.267Zsys 0.147695210
1912023-09-20T03:47:57.293Ztrap 0.000176723
1922023-09-20T03:47:57.317Ztflt 0.000000000
1932023-09-20T03:47:57.343Zdflt 0.000000000
1942023-09-20T03:47:57.371Zkflt 0.000000000
1952023-09-20T03:47:57.403Zlock 0.000000000
1962023-09-20T03:47:57.446Zslp 0.000022850
1972023-09-20T03:47:57.470Zlat 0.000022270
1982023-09-20T03:47:57.494Zstop 0.000123841
1992023-09-20T03:47:57.519Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a5dac3003865f64e
2002023-09-20T03:47:57.563Z+ chmod a+x tests/end_to_end_tests-a5dac3003865f64e
2012023-09-20T03:47:57.589Z+ bmat address ls
2022023-09-20T03:47:57.614ZNAME CIDR FIRST LAST COUNT
2032023-09-20T03:47:57.638Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2042023-09-20T03:47:57.663Z++ bmat address ls -f extra -Ho first,last,count
2052023-09-20T03:47:57.689Z+ set -- 10.151.1.164 10.151.1.227 64
2062023-09-20T03:47:57.714Z+ EXTRA_IP_START=10.151.1.164
2072023-09-20T03:47:57.745Z+ EXTRA_IP_END=10.151.1.227
2082023-09-20T03:47:57.771Z+ EXTRA_IP_COUNT=64
2092023-09-20T03:47:57.795Z+ (( EXTRA_IP_COUNT >= 32 ))
2102023-09-20T03:47:57.818Z+ EXTRA_IP_BASE=10.151.1
2112023-09-20T03:47:57.842Z+ EXTRA_IP_FOCTET=164
2122023-09-20T03:47:57.866Z+ EXTRA_IP_LOCTET=227
2132023-09-20T03:47:57.890Z+ SERVICE_IP_POOL_START=10.151.1.164
2142023-09-20T03:47:57.914Z+ SERVICE_IP_POOL_END=10.151.1.173
2152023-09-20T03:47:57.938Z+ DNS_IP1=10.151.1.164
2162023-09-20T03:47:57.962Z+ DNS_IP2=10.151.1.165
2172023-09-20T03:47:57.986Z+ UPLINK_IP=10.151.1.174
2182023-09-20T03:47:58.010Z+ IPPOOL_START=10.151.1.175
2192023-09-20T03:47:58.038Z+ IPPOOL_END=10.151.1.227
2202023-09-20T03:47:58.065Z++ ipadm show-addr -po type,addr
2212023-09-20T03:47:58.088Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2222023-09-20T03:47:58.112Z+ GATEWAY_IP=10.151.1.100
2232023-09-20T03:47:58.136Z+ [[ -n 10.151.1.100 ]]
2242023-09-20T03:47:58.161Z+ ping -s 10.151.1.100 56 1
2252023-09-20T03:47:58.185ZPING 10.151.1.100: 56 data bytes
2262023-09-20T03:47:58.209Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.025 ms
2272023-09-20T03:47:58.234Z
2282023-09-20T03:47:58.258Z----10.151.1.100 PING Statistics----
2292023-09-20T03:47:58.282Z1 packets transmitted, 1 packets received, 0% packet loss
2302023-09-20T03:47:58.306Zround-trip (ms) min/avg/max/stddev = 0.025/0.025/0.025/-nan
2312023-09-20T03:47:58.330Z++ arp -an
2322023-09-20T03:47:58.353Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2332023-09-20T03:47:58.377Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2342023-09-20T03:47:58.401Z+ routeadm -e ipv4-forwarding -u
2352023-09-20T03:47:58.425Z+ PXA_START=10.151.1.164
2362023-09-20T03:47:58.467Z+ PXA_END=10.151.1.227
2372023-09-20T03:47:58.491Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2382023-09-20T03:47:58.536Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2392023-09-20T03:47:58.560Z+ ZPOOL_VDEV_DIR=/scratch
2402023-09-20T03:47:58.583Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2412023-09-20T03:47:58.607Z+++ dirname ./tools/create_virtual_hardware.sh
2422023-09-20T03:47:58.635Z++ cd ./tools
2432023-09-20T03:47:58.663Z++ pwd
2442023-09-20T03:47:58.687Z+ SOURCE_DIR=/opt/oxide/work/tools
2452023-09-20T03:47:58.710Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2462023-09-20T03:47:58.734Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2472023-09-20T03:47:58.760Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2482023-09-20T03:47:58.784Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2492023-09-20T03:47:58.807Z++ dladm show-phys -p -o LINK
2502023-09-20T03:47:58.831Z++ head -1
2512023-09-20T03:47:58.855Z+ PHYSICAL_LINK=igb0
2522023-09-20T03:47:58.879Z+ echo 'Using igb0 as physical link'
2532023-09-20T03:47:58.902ZUsing igb0 as physical link
2542023-09-20T03:47:58.926Z+ ensure_run_as_root
2552023-09-20T03:47:58.951Z++ id -u
2562023-09-20T03:47:58.974Z+ [[ 0 -ne 0 ]]
2572023-09-20T03:47:58.998Z+ ensure_zpools
2582023-09-20T03:47:59.022Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2592023-09-20T03:47:59.046Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2602023-09-20T03:47:59.070Z+ readarray -t ZPOOLS
2612023-09-20T03:47:59.095Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2622023-09-20T03:47:59.119Z++ sed 's/[ ",]//g'
2632023-09-20T03:47:59.144Z+ for ZPOOL in "${ZPOOLS[@]}"
2642023-09-20T03:47:59.171Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2652023-09-20T03:47:59.199ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2662023-09-20T03:47:59.227Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2672023-09-20T03:47:59.262Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2682023-09-20T03:47:59.293Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2692023-09-20T03:47:59.321ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2702023-09-20T03:47:59.350Z0+0 records in
2712023-09-20T03:47:59.376ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2722023-09-20T03:47:59.404Z0+0 records out
2732023-09-20T03:47:59.438ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2742023-09-20T03:47:59.520Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2752023-09-20T03:47:59.553ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2762023-09-20T03:48:00.026Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2772023-09-20T03:48:00.054ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2782023-09-20T03:48:00.494Z+ set +x
2792023-09-20T03:48:00.525ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2802023-09-20T03:48:00.555Z++ zpool list -o name
2812023-09-20T03:48:00.582ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2822023-09-20T03:48:00.606Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2832023-09-20T03:48:00.630ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2842023-09-20T03:48:00.655Z+ [[ -z '' ]]
2852023-09-20T03:48:00.678ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2862023-09-20T03:48:00.702Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2872023-09-20T03:48:00.727ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2882023-09-20T03:48:00.751Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2892023-09-20T03:48:00.775ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2902023-09-20T03:48:00.799Z+ set +x
2912023-09-20T03:48:00.824ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2922023-09-20T03:48:00.856Z+ for ZPOOL in "${ZPOOLS[@]}"
2932023-09-20T03:48:01.287ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2942023-09-20T03:48:01.312Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2952023-09-20T03:48:01.336ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2962023-09-20T03:48:01.361Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2972023-09-20T03:48:01.385ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
2982023-09-20T03:48:01.412Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2992023-09-20T03:48:01.438ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3002023-09-20T03:48:01.468Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3012023-09-20T03:48:01.493ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3022023-09-20T03:48:01.517Z0+0 records in
3032023-09-20T03:48:01.622ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3042023-09-20T03:48:01.648Z0+0 records out
3052023-09-20T03:48:01.674ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3062023-09-20T03:48:01.698Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3072023-09-20T03:48:01.725ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3082023-09-20T03:48:01.749Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3092023-09-20T03:48:01.776ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3102023-09-20T03:48:01.802Z+ set +x
3112023-09-20T03:48:01.831ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3122023-09-20T03:48:01.855Z++ zpool list -o name
3132023-09-20T03:48:01.882ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3142023-09-20T03:48:01.921Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3152023-09-20T03:48:01.948ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3162023-09-20T03:48:01.973Z+ [[ -z '' ]]
3172023-09-20T03:48:01.997ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3182023-09-20T03:48:02.023Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3192023-09-20T03:48:02.047ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3202023-09-20T03:48:02.477Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3212023-09-20T03:48:02.503ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3222023-09-20T03:48:02.534Z+ set +x
3232023-09-20T03:48:02.568ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3242023-09-20T03:48:02.600Z+ for ZPOOL in "${ZPOOLS[@]}"
3252023-09-20T03:48:02.627ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3262023-09-20T03:48:02.652Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-09-20T03:48:02.676ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3282023-09-20T03:48:02.700Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-09-20T03:48:02.725ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3302023-09-20T03:48:02.749Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3312023-09-20T03:48:02.776ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3322023-09-20T03:48:02.805Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3332023-09-20T03:48:02.830ZSimnet net0/sc0_0 exists
3342023-09-20T03:48:02.854Z0+0 records in
3352023-09-20T03:48:02.878ZSimnet net1/sc1_0 exists
3362023-09-20T03:48:02.903Z0+0 records out
3372023-09-20T03:48:02.927ZVnic sc0_1 exists
3382023-09-20T03:48:02.952Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3392023-09-20T03:48:02.976Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3402023-09-20T03:48:03.008Z+ set +x
3412023-09-20T03:48:03.037Z++ zpool list -o name
3422023-09-20T03:48:03.069Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3432023-09-20T03:48:03.097Z+ [[ -z '' ]]
3442023-09-20T03:48:03.129Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3452023-09-20T03:48:03.155Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3462023-09-20T03:48:03.179Z+ set +x
3472023-09-20T03:48:03.212Z+ for ZPOOL in "${ZPOOLS[@]}"
3482023-09-20T03:48:03.236Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3492023-09-20T03:48:03.260Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-09-20T03:48:03.284Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3512023-09-20T03:48:03.308Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3522023-09-20T03:48:03.332Z0+0 records in
3532023-09-20T03:48:03.357Z0+0 records out
3542023-09-20T03:48:03.381Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3552023-09-20T03:48:03.405Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3562023-09-20T03:48:03.429Z+ set +x
3572023-09-20T03:48:03.453Z++ zpool list -o name
3582023-09-20T03:48:03.477Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3592023-09-20T03:48:03.502Z+ [[ -z '' ]]
3602023-09-20T03:48:03.526Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3612023-09-20T03:48:03.551Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3622023-09-20T03:48:03.589Z+ set +x
3632023-09-20T03:48:03.613Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-09-20T03:48:03.638Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3652023-09-20T03:48:03.662Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-09-20T03:48:03.687Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3672023-09-20T03:48:03.711Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3682023-09-20T03:48:03.735Z0+0 records in
3692023-09-20T03:48:03.759Z0+0 records out
3702023-09-20T03:48:03.784Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3712023-09-20T03:48:03.808Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3722023-09-20T03:48:03.836Z+ set +x
3732023-09-20T03:48:03.861Z++ zpool list -o name
3742023-09-20T03:48:03.885Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3752023-09-20T03:48:03.911Z+ [[ -z '' ]]
3762023-09-20T03:48:03.935Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3772023-09-20T03:48:03.959Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3782023-09-20T03:48:03.984Z+ set +x
3792023-09-20T03:48:04.008Z+ for ZPOOL in "${ZPOOLS[@]}"
3802023-09-20T03:48:04.032Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3812023-09-20T03:48:04.056Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3822023-09-20T03:48:04.080Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3832023-09-20T03:48:04.104Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3842023-09-20T03:48:04.128Z0+0 records in
3852023-09-20T03:48:04.152Z0+0 records out
3862023-09-20T03:48:04.176Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3872023-09-20T03:48:04.200Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3882023-09-20T03:48:04.224Z+ set +x
3892023-09-20T03:48:04.248Z++ zpool list -o name
3902023-09-20T03:48:04.272Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3912023-09-20T03:48:04.296Z+ [[ -z '' ]]
3922023-09-20T03:48:04.321Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3932023-09-20T03:48:04.348Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3942023-09-20T03:48:04.377Z+ set +x
3952023-09-20T03:48:04.409Z+ for ZPOOL in "${ZPOOLS[@]}"
3962023-09-20T03:48:04.435Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3972023-09-20T03:48:04.463Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-09-20T03:48:04.487Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3992023-09-20T03:48:04.512Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4002023-09-20T03:48:04.537Z0+0 records in
4012023-09-20T03:48:04.561Z0+0 records out
4022023-09-20T03:48:04.586Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4032023-09-20T03:48:04.613Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4042023-09-20T03:48:04.637Z+ set +x
4052023-09-20T03:48:04.661Z++ zpool list -o name
4062023-09-20T03:48:04.686Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4072023-09-20T03:48:04.711Z+ [[ -z '' ]]
4082023-09-20T03:48:04.736Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4092023-09-20T03:48:04.761Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4102023-09-20T03:48:04.786Z+ set +x
4112023-09-20T03:48:04.810Z+ for ZPOOL in "${ZPOOLS[@]}"
4122023-09-20T03:48:04.836Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4132023-09-20T03:48:04.867Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4142023-09-20T03:48:04.899Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4152023-09-20T03:48:04.928Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4162023-09-20T03:48:04.956Z0+0 records in
4172023-09-20T03:48:04.982Z0+0 records out
4182023-09-20T03:48:05.015Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4192023-09-20T03:48:05.045Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4202023-09-20T03:48:05.074Z+ set +x
4212023-09-20T03:48:05.108Z++ zpool list -o name
4222023-09-20T03:48:05.137Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4232023-09-20T03:48:05.165Z+ [[ -z '' ]]
4242023-09-20T03:48:05.194Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4252023-09-20T03:48:05.222Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4262023-09-20T03:48:05.251Z+ set +x
4272023-09-20T03:48:05.281Z+ for ZPOOL in "${ZPOOLS[@]}"
4282023-09-20T03:48:05.308Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4292023-09-20T03:48:05.339Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4302023-09-20T03:48:05.375Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4312023-09-20T03:48:05.413Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4322023-09-20T03:48:05.442Z0+0 records in
4332023-09-20T03:48:05.472Z0+0 records out
4342023-09-20T03:48:05.498Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4352023-09-20T03:48:05.523Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4362023-09-20T03:48:05.552Z+ set +x
4372023-09-20T03:48:05.578Z++ zpool list -o name
4382023-09-20T03:48:05.612Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4392023-09-20T03:48:05.639Z+ [[ -z '' ]]
4402023-09-20T03:48:05.667Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4412023-09-20T03:48:05.697Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4422023-09-20T03:48:05.723Z+ set +x
4432023-09-20T03:48:05.747Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4442023-09-20T03:48:05.771Z+ readarray -t ZPOOLS
4452023-09-20T03:48:05.795Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4462023-09-20T03:48:05.819Z++ sed 's/[ ",]//g'
4472023-09-20T03:48:05.845Z+ for ZPOOL in "${ZPOOLS[@]}"
4482023-09-20T03:48:05.869Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4492023-09-20T03:48:05.898Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4502023-09-20T03:48:05.922Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4512023-09-20T03:48:05.947Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4522023-09-20T03:48:05.976Z0+0 records in
4532023-09-20T03:48:06.000Z0+0 records out
4542023-09-20T03:48:06.024Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4552023-09-20T03:48:06.048Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4562023-09-20T03:48:06.072Z+ set +x
4572023-09-20T03:48:06.096Z++ zpool list -o name
4582023-09-20T03:48:06.120Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4592023-09-20T03:48:07.167Z+ [[ -z '' ]]
4602023-09-20T03:48:07.226Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4612023-09-20T03:48:07.250Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4622023-09-20T03:48:07.274Z+ set +x
4632023-09-20T03:48:07.298Z+ for ZPOOL in "${ZPOOLS[@]}"
4642023-09-20T03:48:07.321Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4652023-09-20T03:48:07.350Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-09-20T03:48:07.377Zdownloading softnpu ...done
4672023-09-20T03:48:07.401Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4682023-09-20T03:48:07.425Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4692023-09-20T03:48:07.449Z0+0 records in
4702023-09-20T03:48:07.472Z0+0 records out
4712023-09-20T03:48:07.496Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
4722023-09-20T03:48:07.521Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4732023-09-20T03:48:07.553Z+ set +x
4742023-09-20T03:48:07.577Z++ zpool list -o name
4752023-09-20T03:48:07.601Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-09-20T03:48:07.624Z+ [[ -z '' ]]
4772023-09-20T03:48:07.648Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-09-20T03:48:07.672Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-09-20T03:48:07.713Z+ set +x
4802023-09-20T03:48:07.738Z+ ensure_simulated_links igb0
4812023-09-20T03:48:07.766Z+ local PHYSICAL_LINK=igb0
4822023-09-20T03:48:07.801Z+ INDICES=("0" "1")
4832023-09-20T03:48:07.831Z+ for I in "${INDICES[@]}"
4842023-09-20T03:48:07.855Z++ get_simnet_name_if_exists net0
4852023-09-20T03:48:07.883Z++ dladm show-simnet -p -o LINK net0
4862023-09-20T03:48:07.910Z++ echo ''
4872023-09-20T03:48:07.934Z+ [[ -z '' ]]
4882023-09-20T03:48:07.958Z+ dladm create-simnet -t net0
4892023-09-20T03:48:07.982Z+ dladm create-simnet -t sc0_0
4902023-09-20T03:48:08.006Z+ dladm modify-simnet -t -p net0 sc0_0
4912023-09-20T03:48:08.030Z+ dladm set-linkprop -p mtu=1600 sc0_0
4922023-09-20T03:48:08.054Z+ success 'Simnet net0/sc0_0 exists'
4932023-09-20T03:48:08.080Z+ set +x
4942023-09-20T03:48:08.105Z+ for I in "${INDICES[@]}"
4952023-09-20T03:48:08.129Z++ get_simnet_name_if_exists net1
4962023-09-20T03:48:08.152Z++ dladm show-simnet -p -o LINK net1
4972023-09-20T03:48:08.176Z++ echo ''
4982023-09-20T03:48:08.201Z+ [[ -z '' ]]
4992023-09-20T03:48:08.225Z+ dladm create-simnet -t net1
5002023-09-20T03:48:08.252Z+ dladm create-simnet -t sc1_0
5012023-09-20T03:48:08.275Z+ dladm modify-simnet -t -p net1 sc1_0
5022023-09-20T03:48:08.300Z+ dladm set-linkprop -p mtu=1600 sc1_0
5032023-09-20T03:48:08.323Z+ success 'Simnet net1/sc1_0 exists'
5042023-09-20T03:48:08.347Z+ set +x
5052023-09-20T03:48:08.378Z++ get_vnic_name_if_exists sc0_1
5062023-09-20T03:48:08.403Z++ dladm show-vnic -p -o LINK sc0_1
5072023-09-20T03:48:08.427Z++ echo ''
5082023-09-20T03:48:08.451Z+ [[ -z '' ]]
5092023-09-20T03:48:08.475Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5102023-09-20T03:48:08.499Z+ success 'Vnic sc0_1 exists'
5112023-09-20T03:48:08.523Z+ set +x
5122023-09-20T03:48:08.546Z+ warn_if_no_proxy_arp
5132023-09-20T03:48:08.572Z+ [[ -v PXA_START ]]
5142023-09-20T03:48:08.595Z+ [[ -v PXA_END ]]
5152023-09-20T03:48:08.619Z+ ensure_softnpu_zone
5162023-09-20T03:48:08.644Z+ zoneadm list
5172023-09-20T03:48:08.668Z+ grep -q sidecar_softnpu
5182023-09-20T03:48:08.692Z+ [[ -f out/npuzone/npuzone ]]
5192023-09-20T03:48:08.745Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0
5202023-09-20T03:48:10.452Zdownloading asic_program.so ...done
5212023-09-20T03:48:22.715Zdownloading scadm ...done
5222023-09-20T03:48:22.744Zcreating port tfportrear0_0 for sc0_0
5232023-09-20T03:48:22.774Zcreating port tfportqsfp0_0 for sc0_1
5242023-09-20T03:48:22.805Zsoftnpu zone setup
5252023-09-20T03:48:22.835Zconfigure zone
5262023-09-20T03:48:22.865Zinstall zone
5272023-09-20T03:48:23.800Zboot zone
5282023-09-20T03:48:26.913Z[sidecar_softnpu] svcs milestone/network
5292023-09-20T03:48:26.956ZSTATE STIME FMRI
5302023-09-20T03:48:27.073Zonline 3:48:25 svc:/milestone/network:default
5312023-09-20T03:48:27.146Z[sidecar_softnpu] /softnpu/npu start
5322023-09-20T03:48:27.222Z
5332023-09-20T03:48:27.313Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5342023-09-20T03:48:27.364Z+ GATEWAY_IP=10.151.1.100
5352023-09-20T03:48:27.419Z+ echo 'Using 10.151.1.100 as gateway ip'
5362023-09-20T03:48:27.542ZUsing 10.151.1.100 as gateway ip
5372023-09-20T03:48:27.673Z+ [[ ! -v GATEWAY_MAC ]]
5382023-09-20T03:48:27.803Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5392023-09-20T03:48:27.924Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5402023-09-20T03:48:28.060Z+++ echo 18:c0:4d:0d:9f:b2
5412023-09-20T03:48:28.186Z+ COUNT=1
5422023-09-20T03:48:28.307Z+ [[ 1 -ne 1 ]]
5432023-09-20T03:48:28.504Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5442023-09-20T03:48:28.576ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5452023-09-20T03:48:28.869Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5462023-09-20T03:48:29.008Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5472023-09-20T03:48:29.084Z+ PXA_MAC=a8:e1:de:01:70:1d
5482023-09-20T03:48:29.333Z+ [[ -v PXA_START ]]
5492023-09-20T03:48:29.555Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5502023-09-20T03:48:29.768Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5512023-09-20T03:48:30.008Z+ z_scadm dump-state
5522023-09-20T03:48:30.178Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5532023-09-20T03:48:30.383Zlocal v6:
5542023-09-20T03:48:30.534Zlocal v4:
5552023-09-20T03:48:31.943Zrouter v6:
5562023-09-20T03:48:32.559Zrouter v4:
5572023-09-20T03:48:34.795Zresolver v4:
5582023-09-20T03:48:35.349Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5592023-09-20T03:48:35.538Zresolver v6:
5602023-09-20T03:48:55.515Znat v4:
5612023-09-20T03:48:55.936Znat_v6:
5622023-09-20T03:48:56.076Zport_mac:
5632023-09-20T03:48:56.345Zproxy arp:
5642023-09-20T03:48:56.426Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5652023-09-20T03:48:56.718Z+ success 'softnpu zone exists'
5662023-09-20T03:48:56.746Z+ set +x
5672023-09-20T03:48:56.855Zsoftnpu zone exists
5682023-09-20T03:48:56.978Z
5692023-09-20T03:48:57.032Zreal 29.540901727
5702023-09-20T03:48:57.073Zuser 0.699202243
5712023-09-20T03:48:57.125Zsys 1.615103038
5722023-09-20T03:48:57.167Ztrap 0.000082649
5732023-09-20T03:48:57.200Ztflt 0.000121047
5742023-09-20T03:48:57.252Zdflt 0.003382735
5752023-09-20T03:48:57.303Zkflt 0.000000000
5762023-09-20T03:48:57.362Zlock 23:05.138972814
5772023-09-20T03:48:57.443Zslp 1:28.225305424
5782023-09-20T03:48:57.490Zlat 0.065838504
5792023-09-20T03:48:57.552Zstop 0.398091127
5802023-09-20T03:48:57.596Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5812023-09-20T03:48:57.627Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5822023-09-20T03:48:57.706Z+ SILO_NAME=recovery
5832023-09-20T03:48:57.805Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5842023-09-20T03:48:57.900Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5852023-09-20T03:48:57.950Z+ sed -i~ '
5862023-09-20T03:48:57.987Z /^external_dns_ips/c\
5872023-09-20T03:48:58.049Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5882023-09-20T03:48:58.100Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5892023-09-20T03:48:58.135Z /^first/c\
5902023-09-20T03:48:58.185Zfirst = "10.151.1.164"
5912023-09-20T03:48:58.215Z /^last/c\
5922023-09-20T03:48:58.243Zlast = "10.151.1.173"
5932023-09-20T03:48:58.281Z }
5942023-09-20T03:48:58.311Z /^\[rack_network_config/,/^$/ {
5952023-09-20T03:48:58.368Z /^infra_ip_first/c\
5962023-09-20T03:48:58.400Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
5972023-09-20T03:48:58.426Zinfra_ip_first = "10.151.1.174"
5982023-09-20T03:48:58.862Z+++ pkg/config-rss.toml Wed Sep 20 03:48:27 2023
5992023-09-20T03:48:58.907Z /^infra_ip_last/c\
6002023-09-20T03:48:58.939Z@@ -41,7 +41,7 @@
6012023-09-20T03:48:58.964Zinfra_ip_last = "10.151.1.174"
6022023-09-20T03:48:58.998Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6032023-09-20T03:48:59.038Z }
6042023-09-20T03:48:59.077Z # must be contained in one of the "internal services" IP Pool ranges listed
6052023-09-20T03:48:59.112Z /^\[\[rack_network_config.uplinks/,/^$/ {
6062023-09-20T03:48:59.143Z # below.
6072023-09-20T03:48:59.174Z /^gateway_ip/c\
6082023-09-20T03:48:59.199Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6092023-09-20T03:48:59.227Zgateway_ip = "10.151.1.100"
6102023-09-20T03:48:59.255Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6112023-09-20T03:48:59.297Z /^uplink_cidr/c\
6122023-09-20T03:48:59.337Z
6132023-09-20T03:48:59.366Zuplink_cidr = "10.151.1.174/32"
6142023-09-20T03:48:59.405Z # Initial TLS certificates for the external API
6152023-09-20T03:48:59.461Z }
6162023-09-20T03:48:59.505Z #
6172023-09-20T03:48:59.530Z' pkg/config-rss.toml
6182023-09-20T03:48:59.569Z@@ -77,8 +77,8 @@
6192023-09-20T03:48:59.596Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6202023-09-20T03:48:59.633Z #
6212023-09-20T03:48:59.659Z+ true
6222023-09-20T03:48:59.694Z # For more on this and what to put here, see docs/how-to-run.adoc.
6232023-09-20T03:48:59.721Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6242023-09-20T03:48:59.749Z [[internal_services_ip_pool_ranges]]
6252023-09-20T03:48:59.790Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6262023-09-20T03:48:59.831Z-first = "192.168.1.20"
6272023-09-20T03:48:59.861Z+ TLS_NAME=recovery.sys.oxide.test
6282023-09-20T03:48:59.889Z-last = "192.168.1.29"
6292023-09-20T03:48:59.943Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6302023-09-20T03:48:59.980Z+first = "10.151.1.164"
6312023-09-20T03:49:00.016Z.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+......+...+...+...+.....+......+......+....+.....+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+...+...........+.........................+............+...+..+...+......+....+............+........+....+..+..........+.....+.............+........+.......+.....+.......+......+..+.......+...+.................+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6322023-09-20T03:49:00.043Z+last = "10.151.1.173"
6332023-09-20T03:49:00.081Z..............+.+..............+...+.+.....+....+.....+.+..+.........+...+......+.+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..............+..+......+............+...+....+..+....+.........+...+..+...+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6342023-09-20T03:49:00.112Z
6352023-09-20T03:49:00.147Z-----
6362023-09-20T03:49:00.189Z # TODO - this configuration is subject to change going forward. Ultimately these
6372023-09-20T03:49:00.217Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6382023-09-20T03:49:00.255Z # parameters should be provided to the control plane via wicket, but we need to
6392023-09-20T03:49:00.307Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6402023-09-20T03:49:00.333Z@@ -91,13 +91,13 @@
6412023-09-20T03:49:00.370Z+ rmdir pkg
6422023-09-20T03:49:00.401Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-09-20T03:49:00.440Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6442023-09-20T03:49:00.482Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6452023-09-20T03:49:00.534Z+ pfexec mkdir -p /usr/oxide
6462023-09-20T03:49:00.559Z # With softnpu, only one address is used.
6472023-09-20T03:49:00.611Z+ pfexec rm -f /usr/oxide/tcpproxy
6482023-09-20T03:49:00.662Z-infra_ip_first = "192.168.1.30"
6492023-09-20T03:49:00.692Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6502023-09-20T03:49:00.731Z-infra_ip_last = "192.168.1.30"
6512023-09-20T03:49:00.758Z+ pfexec chmod +x /usr/oxide/tcpproxy
6522023-09-20T03:49:00.801Z+infra_ip_first = "10.151.1.174"
6532023-09-20T03:49:00.847Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6542023-09-20T03:49:00.880Z+infra_ip_last = "10.151.1.174"
6552023-09-20T03:49:00.906Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6562023-09-20T03:49:00.934Z
6572023-09-20T03:49:00.989Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6582023-09-20T03:49:01.018Z # You can configure multiple uplinks by repeating the following stanza
6592023-09-20T03:49:01.049Z+ OMICRON_NO_UNINSTALL=1
6602023-09-20T03:49:01.087Z [[rack_network_config.uplinks]]
6612023-09-20T03:49:01.118Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6622023-09-20T03:49:01.143Z # The gateway IP for the rack's external network
6632023-09-20T03:49:01.169ZSep 20 03:48:28.370 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "switch": "softnpu"})
6642023-09-20T03:49:01.196Z-gateway_ip = "192.168.1.199"
6652023-09-20T03:49:01.220ZSep 20 03:48:28.372 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6662023-09-20T03:49:01.252Z+gateway_ip = "10.151.1.100"
6672023-09-20T03:49:01.288ZSep 20 03:48:28.373 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6682023-09-20T03:49:01.331Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6692023-09-20T03:49:01.358ZSep 20 03:48:28.373 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6702023-09-20T03:49:01.397Z uplink_port = "qsfp0"
6712023-09-20T03:49:01.428ZSep 20 03:48:28.373 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6722023-09-20T03:49:01.458Z uplink_port_speed = "40G"
6732023-09-20T03:49:01.488ZSep 20 03:48:28.373 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6742023-09-20T03:49:01.517Z@@ -104,7 +104,7 @@
6752023-09-20T03:49:01.587ZSep 20 03:48:28.373 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6762023-09-20T03:49:01.615Z uplink_port_fec="none"
6772023-09-20T03:49:01.649ZSep 20 03:48:28.373 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6782023-09-20T03:49:01.677Z # For softnpu, an address within the "infra" block above that will be used for
6792023-09-20T03:49:01.710ZSep 20 03:48:28.373 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6802023-09-20T03:49:01.736Z # the softnpu uplink port. You can just pick the first address in that pool.
6812023-09-20T03:49:01.770ZSep 20 03:48:28.373 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6822023-09-20T03:49:01.816Z-uplink_cidr = "192.168.1.30/32"
6832023-09-20T03:49:02.048ZSep 20 03:48:28.373 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/maghemite.tar
6842023-09-20T03:49:02.076Z+uplink_cidr = "10.151.1.174/32"
6852023-09-20T03:49:02.224ZSep 20 03:48:28.373 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6862023-09-20T03:49:02.398Z # Switch to use for the uplink. For single-rack deployments this can be
6872023-09-20T03:49:02.507ZSep 20 03:48:28.373 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6882023-09-20T03:49:02.545Z # "switch0" (upper slot) or "switch1" (lower slot). For single-node softnpu
6892023-09-20T03:49:02.580ZSep 20 03:48:28.373 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6902023-09-20T03:49:02.620Z # and dendrite stub environments, use "switch0"
6912023-09-20T03:49:02.704ZSep 20 03:48:28.373 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6922023-09-20T03:49:02.794Za pkg/config-rss.toml 7K
6932023-09-20T03:49:02.830ZSep 20 03:48:28.373 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6942023-09-20T03:49:02.871Za pkg/initial-tls-cert.pem 2K
6952023-09-20T03:49:02.907ZSep 20 03:48:28.373 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6962023-09-20T03:49:02.937Za pkg/initial-tls-key.pem 4K
6972023-09-20T03:49:02.967ZSep 20 03:48:28.592 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
6982023-09-20T03:49:03.004Zlocal v6:
6992023-09-20T03:49:03.032ZSep 20 03:48:28.634 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7002023-09-20T03:49:03.087Zfd00:99::1
7012023-09-20T03:49:03.115ZSep 20 03:48:28.697 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7022023-09-20T03:49:03.150Zfe80::7889:8ff:fe14:e849
7032023-09-20T03:49:03.189Z
7042023-09-20T03:49:03.221Zlocal v4:
7052023-09-20T03:49:03.247Zreal 0.461090437
7062023-09-20T03:49:03.286Z10.151.1.174
7072023-09-20T03:49:03.316Zuser 0.095491966
7082023-09-20T03:49:03.342Zrouter v6:
7092023-09-20T03:49:03.367Zsys 1.720739564
7102023-09-20T03:49:03.391Zrouter v4:
7112023-09-20T03:49:03.416Ztrap 0.000000659
7122023-09-20T03:49:03.450Z0.0.0.0/0 -> 10.151.1.100 (2) vid=0
7132023-09-20T03:49:03.524Ztflt 0.000125331
7142023-09-20T03:49:03.551Zresolver v4:
7152023-09-20T03:49:03.586Zdflt 0.001039793
7162023-09-20T03:49:03.611Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7172023-09-20T03:49:03.655Zkflt 0.000000000
7182023-09-20T03:49:03.680Zresolver v6:
7192023-09-20T03:49:03.706Zlock 41.595224623
7202023-09-20T03:49:03.735Zfe80::7889:8ff:fe14:e849 -> 7a:89:08:14:e8:49
7212023-09-20T03:49:03.762Zslp 0.667500723
7222023-09-20T03:49:03.794Zfe80::e8f1:c4ff:fe1c:969 -> ea:f1:c4:1c:09:69
7232023-09-20T03:49:03.819Zlat 0.003086087
7242023-09-20T03:49:03.848Znat v4:
7252023-09-20T03:49:03.873Zstop 0.203363350
7262023-09-20T03:49:03.897Znat_v6:
7272023-09-20T03:49:03.923Z+ retry=0
7282023-09-20T03:49:03.948Zport_mac:
7292023-09-20T03:49:03.973Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-09-20T03:49:03.997Z1000: a8:40:25:23:48:a2
7312023-09-20T03:49:04.023Z+ [[ 0 -gt 30 ]]
7322023-09-20T03:49:04.048Z1: a8:40:25:23:48:a3
7332023-09-20T03:49:04.072Z+ sleep 1
7342023-09-20T03:49:04.115Zproxy arp:
7352023-09-20T03:49:04.141Z+ retry=1
7362023-09-20T03:49:04.166Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7372023-09-20T03:49:04.190Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-09-20T03:49:04.215Z+ [[ 1 -gt 30 ]]
7392023-09-20T03:49:04.239Z+ sleep 1
7402023-09-20T03:49:04.264Z+ retry=2
7412023-09-20T03:49:04.288Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7422023-09-20T03:49:04.313Z+ [[ 2 -gt 30 ]]
7432023-09-20T03:49:04.337Z+ sleep 1
7442023-09-20T03:49:04.362Z+ retry=3
7452023-09-20T03:49:04.387Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7462023-09-20T03:49:04.412Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7472023-09-20T03:49:04.436Z+ retry=0
7482023-09-20T03:49:04.462Z+++ zoneadm list -n
7492023-09-20T03:49:04.488Z+++ grep oxz_ntp
7502023-09-20T03:49:04.513Z++ pfexec svcs -z -Hostate oxide/ntp
7512023-09-20T03:49:04.538Zsvcs: zone '-Hostate': entity not found
7522023-09-20T03:49:04.563Z++ true
7532023-09-20T03:49:04.590Z+ [[ '' != online ]]
7542023-09-20T03:49:04.616Z+ [[ 0 -gt 60 ]]
7552023-09-20T03:49:05.051Z+ sleep 1
7562023-09-20T03:49:05.085Z+ retry=1
7572023-09-20T03:49:05.111Z+++ zoneadm list -n
7582023-09-20T03:49:05.150Z+++ grep oxz_ntp
7592023-09-20T03:49:05.176Z++ pfexec svcs -z -Hostate oxide/ntp
7602023-09-20T03:49:05.200Zsvcs: zone '-Hostate': entity not found
7612023-09-20T03:49:05.225Z++ true
7622023-09-20T03:49:05.250Z+ [[ '' != online ]]
7632023-09-20T03:49:05.274Z+ [[ 1 -gt 60 ]]
7642023-09-20T03:49:05.298Z+ sleep 1
7652023-09-20T03:49:05.323Z+ retry=2
7662023-09-20T03:49:05.347Z+++ zoneadm list -n
7672023-09-20T03:49:05.372Z+++ grep oxz_ntp
7682023-09-20T03:49:05.396Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-09-20T03:49:05.421Zsvcs: zone '-Hostate': entity not found
7702023-09-20T03:49:05.451Z++ true
7712023-09-20T03:49:05.480Z+ [[ '' != online ]]
7722023-09-20T03:49:05.505Z+ [[ 2 -gt 60 ]]
7732023-09-20T03:49:05.532Z+ sleep 1
7742023-09-20T03:49:05.564Z+ retry=3
7752023-09-20T03:49:05.591Z+++ zoneadm list -n
7762023-09-20T03:49:05.618Z+++ grep oxz_ntp
7772023-09-20T03:49:05.643Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-09-20T03:49:05.668Zsvcs: zone '-Hostate': entity not found
7792023-09-20T03:49:05.695Z++ true
7802023-09-20T03:49:05.721Z+ [[ '' != online ]]
7812023-09-20T03:49:05.747Z+ [[ 3 -gt 60 ]]
7822023-09-20T03:49:05.772Z+ sleep 1
7832023-09-20T03:49:05.798Z+ retry=4
7842023-09-20T03:49:05.822Z+++ zoneadm list -n
7852023-09-20T03:49:05.847Z+++ grep oxz_ntp
7862023-09-20T03:49:05.871Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-09-20T03:49:05.896Zsvcs: zone '-Hostate': entity not found
7882023-09-20T03:49:05.920Z++ true
7892023-09-20T03:49:05.944Z+ [[ '' != online ]]
7902023-09-20T03:49:05.969Z+ [[ 4 -gt 60 ]]
7912023-09-20T03:49:05.994Z+ sleep 1
7922023-09-20T03:49:06.021Z+ retry=5
7932023-09-20T03:49:06.049Z+++ zoneadm list -n
7942023-09-20T03:49:06.073Z+++ grep oxz_ntp
7952023-09-20T03:49:06.098Z++ pfexec svcs -z -Hostate oxide/ntp
7962023-09-20T03:49:06.123Zsvcs: zone '-Hostate': entity not found
7972023-09-20T03:49:06.147Z++ true
7982023-09-20T03:49:06.183Z+ [[ '' != online ]]
7992023-09-20T03:49:06.209Z+ [[ 5 -gt 60 ]]
8002023-09-20T03:49:06.234Z+ sleep 1
8012023-09-20T03:49:06.259Z+ retry=6
8022023-09-20T03:49:06.284Z+++ zoneadm list -n
8032023-09-20T03:49:06.308Z+++ grep oxz_ntp
8042023-09-20T03:49:06.332Z++ pfexec svcs -z -Hostate oxide/ntp
8052023-09-20T03:49:06.357Zsvcs: zone '-Hostate': entity not found
8062023-09-20T03:49:06.381Z++ true
8072023-09-20T03:49:06.405Z+ [[ '' != online ]]
8082023-09-20T03:49:06.430Z+ [[ 6 -gt 60 ]]
8092023-09-20T03:49:06.458Z+ sleep 1
8102023-09-20T03:49:06.482Z+ retry=7
8112023-09-20T03:49:06.533Z+++ zoneadm list -n
8122023-09-20T03:49:06.558Z+++ grep oxz_ntp
8132023-09-20T03:49:06.583Z++ pfexec svcs -z -Hostate oxide/ntp
8142023-09-20T03:49:06.608Zsvcs: zone '-Hostate': entity not found
8152023-09-20T03:49:06.634Z++ true
8162023-09-20T03:49:06.658Z+ [[ '' != online ]]
8172023-09-20T03:49:06.683Z+ [[ 7 -gt 60 ]]
8182023-09-20T03:49:06.708Z+ sleep 1
8192023-09-20T03:49:06.733Z+ retry=8
8202023-09-20T03:49:06.757Z+++ zoneadm list -n
8212023-09-20T03:49:06.782Z+++ grep oxz_ntp
8222023-09-20T03:49:06.806Z++ pfexec svcs -z -Hostate oxide/ntp
8232023-09-20T03:49:06.831Zsvcs: zone '-Hostate': entity not found
8242023-09-20T03:49:06.860Z++ true
8252023-09-20T03:49:06.885Z+ [[ '' != online ]]
8262023-09-20T03:49:06.910Z+ [[ 8 -gt 60 ]]
8272023-09-20T03:49:06.934Z+ sleep 1
8282023-09-20T03:49:06.959Z+ retry=9
8292023-09-20T03:49:06.983Z+++ zoneadm list -n
8302023-09-20T03:49:07.008Z+++ grep oxz_ntp
8312023-09-20T03:49:07.033Z++ pfexec svcs -z -Hostate oxide/ntp
8322023-09-20T03:49:07.058Zsvcs: zone '-Hostate': entity not found
8332023-09-20T03:49:07.083Z++ true
8342023-09-20T03:49:07.107Z+ [[ '' != online ]]
8352023-09-20T03:49:07.132Z+ [[ 9 -gt 60 ]]
8362023-09-20T03:49:07.157Z+ sleep 1
8372023-09-20T03:49:07.181Z+ retry=10
8382023-09-20T03:49:07.218Z+++ zoneadm list -n
8392023-09-20T03:49:07.244Z+++ grep oxz_ntp
8402023-09-20T03:49:07.269Z++ pfexec svcs -z -Hostate oxide/ntp
8412023-09-20T03:49:07.294Zsvcs: zone '-Hostate': entity not found
8422023-09-20T03:49:07.319Z++ true
8432023-09-20T03:49:07.345Z+ [[ '' != online ]]
8442023-09-20T03:49:07.369Z+ [[ 10 -gt 60 ]]
8452023-09-20T03:49:07.393Z+ sleep 1
8462023-09-20T03:49:07.451Z+ retry=11
8472023-09-20T03:49:07.478Z+++ zoneadm list -n
8482023-09-20T03:49:07.502Z+++ grep oxz_ntp
8492023-09-20T03:49:07.526Z++ pfexec svcs -z -Hostate oxide/ntp
8502023-09-20T03:49:07.551Zsvcs: zone '-Hostate': entity not found
8512023-09-20T03:49:07.576Z++ true
8522023-09-20T03:49:07.600Z+ [[ '' != online ]]
8532023-09-20T03:49:07.625Z+ [[ 11 -gt 60 ]]
8542023-09-20T03:49:07.650Z+ sleep 1
8552023-09-20T03:49:07.680Z+ retry=12
8562023-09-20T03:49:07.705Z+++ zoneadm list -n
8572023-09-20T03:49:07.730Z+++ grep oxz_ntp
8582023-09-20T03:49:07.756Z++ pfexec svcs -z -Hostate oxide/ntp
8592023-09-20T03:49:07.781Zsvcs: zone '-Hostate': entity not found
8602023-09-20T03:49:07.805Z++ true
8612023-09-20T03:49:07.830Z+ [[ '' != online ]]
8622023-09-20T03:49:07.854Z+ [[ 12 -gt 60 ]]
8632023-09-20T03:49:07.880Z+ sleep 1
8642023-09-20T03:49:07.911Z+ retry=13
8652023-09-20T03:49:07.935Z+++ zoneadm list -n
8662023-09-20T03:49:07.960Z+++ grep oxz_ntp
8672023-09-20T03:49:07.984Z++ pfexec svcs -z oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915 -Hostate oxide/ntp
8682023-09-20T03:49:08.010Zsvcs: Could not bind to repository server for zone oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915: repository server unavailable
8692023-09-20T03:49:08.035Z++ true
8702023-09-20T03:49:08.060Z+ [[ '' != online ]]
8712023-09-20T03:49:08.085Z+ [[ 13 -gt 60 ]]
8722023-09-20T03:49:08.116Z+ sleep 1
8732023-09-20T03:49:08.141Z+ retry=14
8742023-09-20T03:49:08.166Z+++ zoneadm list -n
8752023-09-20T03:49:08.191Z+++ grep oxz_ntp
8762023-09-20T03:49:08.219Z++ pfexec svcs -z oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915 -Hostate oxide/ntp
8772023-09-20T03:49:08.255Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8782023-09-20T03:49:08.281Z++ true
8792023-09-20T03:49:08.306Z+ [[ '' != online ]]
8802023-09-20T03:49:08.331Z+ [[ 14 -gt 60 ]]
8812023-09-20T03:49:08.357Z+ sleep 1
8822023-09-20T03:49:08.381Z+ retry=15
8832023-09-20T03:49:08.406Z+++ zoneadm list -n
8842023-09-20T03:49:08.465Z+++ grep oxz_ntp
8852023-09-20T03:49:08.491Z++ pfexec svcs -z oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915 -Hostate oxide/ntp
8862023-09-20T03:49:08.515Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8872023-09-20T03:49:08.541Z++ true
8882023-09-20T03:49:08.565Z+ [[ '' != online ]]
8892023-09-20T03:49:08.590Z+ [[ 15 -gt 60 ]]
8902023-09-20T03:49:08.615Z+ sleep 1
8912023-09-20T03:49:08.677Z+ retry=16
8922023-09-20T03:49:08.703Z+++ zoneadm list -n
8932023-09-20T03:49:08.760Z+++ grep oxz_ntp
8942023-09-20T03:49:08.795Z++ pfexec svcs -z oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915 -Hostate oxide/ntp
8952023-09-20T03:49:08.820Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8962023-09-20T03:49:08.844Z++ true
8972023-09-20T03:49:08.871Z+ [[ '' != online ]]
8982023-09-20T03:49:08.896Z+ [[ 16 -gt 60 ]]
8992023-09-20T03:49:08.921Z+ sleep 1
9002023-09-20T03:49:08.946Z+ retry=17
9012023-09-20T03:49:08.970Z+++ zoneadm list -n
9022023-09-20T03:49:08.995Z+++ grep oxz_ntp
9032023-09-20T03:49:09.019Z++ pfexec svcs -z oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915 -Hostate oxide/ntp
9042023-09-20T03:49:09.044Z+ [[ disabled != online ]]
9052023-09-20T03:49:09.070Z+ [[ 17 -gt 60 ]]
9062023-09-20T03:49:09.094Z+ sleep 1
9072023-09-20T03:49:09.584Z+ retry=18
9082023-09-20T03:49:09.610Z+++ zoneadm list -n
9092023-09-20T03:49:09.634Z+++ grep oxz_ntp
9102023-09-20T03:49:09.659Z++ pfexec svcs -z oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915 -Hostate oxide/ntp
9112023-09-20T03:49:09.686Z+ [[ disabled != online ]]
9122023-09-20T03:49:09.713Z+ [[ 18 -gt 60 ]]
9132023-09-20T03:49:09.738Z+ sleep 1
9142023-09-20T03:49:10.607Z+ retry=19
9152023-09-20T03:49:10.631Z+++ zoneadm list -n
9162023-09-20T03:49:10.656Z+++ grep oxz_ntp
9172023-09-20T03:49:10.681Z++ pfexec svcs -z oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915 -Hostate oxide/ntp
9182023-09-20T03:49:10.708Z+ [[ disabled != online ]]
9192023-09-20T03:49:10.732Z+ [[ 19 -gt 60 ]]
9202023-09-20T03:49:10.757Z+ sleep 1
9212023-09-20T03:49:11.629Z+ retry=20
9222023-09-20T03:49:11.654Z+++ zoneadm list -n
9232023-09-20T03:49:11.678Z+++ grep oxz_ntp
9242023-09-20T03:49:11.703Z++ pfexec svcs -z oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915 -Hostate oxide/ntp
9252023-09-20T03:49:11.728Z+ [[ disabled != online ]]
9262023-09-20T03:49:11.753Z+ [[ 20 -gt 60 ]]
9272023-09-20T03:49:11.778Z+ sleep 1
9282023-09-20T03:49:12.651Z+ retry=21
9292023-09-20T03:49:12.676Z+++ zoneadm list -n
9302023-09-20T03:49:12.702Z+++ grep oxz_ntp
9312023-09-20T03:49:12.727Z++ pfexec svcs -z oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915 -Hostate oxide/ntp
9322023-09-20T03:49:12.751Z+ [[ disabled != online ]]
9332023-09-20T03:49:12.776Z+ [[ 21 -gt 60 ]]
9342023-09-20T03:49:12.805Z+ sleep 1
9352023-09-20T03:49:13.672Z+ retry=22
9362023-09-20T03:49:13.704Z+++ zoneadm list -n
9372023-09-20T03:49:13.728Z+++ grep oxz_ntp
9382023-09-20T03:49:13.753Z++ pfexec svcs -z oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915 -Hostate oxide/ntp
9392023-09-20T03:49:13.779Z+ [[ disabled != online ]]
9402023-09-20T03:49:13.803Z+ [[ 22 -gt 60 ]]
9412023-09-20T03:49:13.828Z+ sleep 1
9422023-09-20T03:49:14.696Z+ retry=23
9432023-09-20T03:49:14.720Z+++ zoneadm list -n
9442023-09-20T03:49:14.745Z+++ grep oxz_ntp
9452023-09-20T03:49:14.769Z++ pfexec svcs -z oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915 -Hostate oxide/ntp
9462023-09-20T03:49:14.794Z+ [[ disabled != online ]]
9472023-09-20T03:49:14.819Z+ [[ 23 -gt 60 ]]
9482023-09-20T03:49:14.843Z+ sleep 1
9492023-09-20T03:49:15.724Z+ retry=24
9502023-09-20T03:49:15.751Z+++ zoneadm list -n
9512023-09-20T03:49:15.775Z+++ grep oxz_ntp
9522023-09-20T03:49:15.800Z++ pfexec svcs -z oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915 -Hostate oxide/ntp
9532023-09-20T03:49:15.825Z+ [[ online != online ]]
9542023-09-20T03:49:15.849Z+ echo 'Waited for chrony: 24s'
 
9552023-09-20T03:49:15.861Zbuskin console login: Sep 20 03:49:15 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9562023-09-20T03:49:15.873ZWaited for chrony: 24s
9572023-09-20T03:49:15.898Z+ export RUST_BACKTRACE=1
9582023-09-20T03:49:15.925Z+ RUST_BACKTRACE=1
9592023-09-20T03:49:15.950Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9602023-09-20T03:49:16.380Z+ ./tests/bootstrap
9612023-09-20T03:49:16.411ZUsing external DNS server at 10.151.1.164:53
9622023-09-20T03:49:16.444Z2023-09-20 03:49:16.067566490 UTC: attempting to log into API
9632023-09-20T03:49:30.805Z2023-09-20 03:49:31.223883339 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9642023-09-20T03:49:31.806Z2023-09-20 03:49:32.222070268 UTC: attempting to log into API
9652023-09-20T03:49:46.841Z2023-09-20 03:49:47.215210721 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9662023-09-20T03:49:47.842Z2023-09-20 03:49:48.213657474 UTC: attempting to log into API
9672023-09-20T03:50:02.894Z2023-09-20 03:50:03.224550679 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9682023-09-20T03:50:04.199Z2023-09-20 03:50:04.222933980 UTC: attempting to log into API
9692023-09-20T03:50:18.929Z2023-09-20 03:50:19.216027175 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9702023-09-20T03:50:19.930Z2023-09-20 03:50:20.214368946 UTC: attempting to log into API
 
9712023-09-20T03:50:32.853ZSep 20 03:50:32 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9722023-09-20T03:50:34.967Z2023-09-20 03:50:35.209932195 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9732023-09-20T03:50:35.967Z2023-09-20 03:50:36.208261176 UTC: attempting to log into API
 
9742023-09-20T03:50:39.812ZSep 20 03:50:39 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9752023-09-20T03:50:48.037ZSep 20 03:50:47 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9762023-09-20T03:50:51.001Z2023-09-20 03:50:51.200665967 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9772023-09-20T03:50:52.002Z2023-09-20 03:50:52.199265524 UTC: attempting to log into API
 
9782023-09-20T03:50:54.730ZSep 20 03:50:54 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9792023-09-20T03:51:00.397ZSep 20 03:51:00 bukin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9802023-09-20T03:51:07.037Z2023-09-20 03:51:07.193102780 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9812023-09-20T03:51:08.039Z2023-09-20 03:51:08.191497002 UTC: attempting to log into API
9822023-09-20T03:51:23.071Z2023-09-20 03:51:23.183592573 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9832023-09-20T03:51:24.072Z2023-09-20 03:51:24.181956584 UTC: attempting to log into API
9842023-09-20T03:51:39.105Z2023-09-20 03:51:39.174438822 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9852023-09-20T03:51:40.106Z2023-09-20 03:51:40.172537018 UTC: attempting to log into API
9862023-09-20T03:51:40.896Z2023-09-20 03:51:40.959927127 UTC: login succeeded
9872023-09-20T03:51:40.929Zwaiting for nexus to come up...
9882023-09-20T03:51:40.989Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9892023-09-20T03:51:41.038Zensuring datasets are ready...
9902023-09-20T03:51:43.051Zlet's roll.
9912023-09-20T03:51:43.076Z+ rm ./tests/bootstrap
9922023-09-20T03:51:43.133Z+ for test_bin in tests/*
9932023-09-20T03:51:43.157Z+ ./tests/bootstrap-49c0192018c458f0
9942023-09-20T03:51:43.182Z
9952023-09-20T03:51:43.206Zrunning 0 tests
9962023-09-20T03:51:43.233Z
9972023-09-20T03:51:43.258Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9982023-09-20T03:51:43.282Z
9992023-09-20T03:51:43.306Z+ for test_bin in tests/*
10002023-09-20T03:51:43.330Z+ ./tests/end_to_end_tests-a5dac3003865f64e
10012023-09-20T03:51:43.354Z
10022023-09-20T03:51:43.379Zrunning 2 tests
10032023-09-20T03:51:43.403Ztest no_spoof::no_spoof_here ... ok
10042023-09-20T03:52:43.224Ztest instance_launch::instance_launch has been running for over 60 seconds
10052023-09-20T03:54:37.699Ztest instance_launch::instance_launch ... ok
10062023-09-20T03:54:37.724Z
10072023-09-20T03:54:37.748Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 174.54s
10082023-09-20T03:54:37.772Z
10092023-09-20T03:54:37.796Z+ _exit_trap
10102023-09-20T03:54:37.820Z+ local status=0
10112023-09-20T03:54:37.844Z+ [[ 0 -eq 0 ]]
10122023-09-20T03:54:37.868Z+ exit 0
10132023-09-20T03:54:37.892Zprocess exited: duration 408142 ms, exit code 0
 
10142023-09-20T03:54:37.997Zfound 47 output files
10152023-09-20T03:54:38.161Zuploading: /var/svc/log/oxide-sled-agent:default.log (382390 bytes)
10162023-09-20T03:54:38.216Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 382390 -> 382952
10172023-09-20T03:54:39.291Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10182023-09-20T03:54:39.319Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_30153f90-1fce-4f1d-a7cc-a160a8286745/root/var/svc/log/oxide-clickhouse:default.log (480392 bytes)
10192023-09-20T03:54:40.596Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_30153f90-1fce-4f1d-a7cc-a160a8286745/root/var/svc/log/oxide-clickhouse:default.log
10202023-09-20T03:54:40.620Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_f840ab86-b51b-4dfc-87f5-1e94da606b2f/root/var/svc/log/oxide-external_dns:default.log (15610 bytes)
10212023-09-20T03:54:41.714Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_f840ab86-b51b-4dfc-87f5-1e94da606b2f/root/var/svc/log/oxide-external_dns:default.log
10222023-09-20T03:54:41.754Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8e413664-c870-4e1d-9bd8-441f1c8249bc/root/var/svc/log/oxide-nexus:default.log (4334780 bytes)
10232023-09-20T03:54:44.829Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8e413664-c870-4e1d-9bd8-441f1c8249bc/root/var/svc/log/oxide-nexus:default.log
10242023-09-20T03:54:44.853Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_68e17930-c4e8-46a4-b943-ce2dfb9c32f7/root/var/svc/log/oxide-crucible-agent:default.log (14296 bytes)
10252023-09-20T03:54:45.947Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_68e17930-c4e8-46a4-b943-ce2dfb9c32f7/root/var/svc/log/oxide-crucible-agent:default.log
10262023-09-20T03:54:45.971Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_68e17930-c4e8-46a4-b943-ce2dfb9c32f7/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10272023-09-20T03:54:47.042Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_68e17930-c4e8-46a4-b943-ce2dfb9c32f7/root/var/svc/log/oxide-crucible-downstairs:default.log
10282023-09-20T03:54:47.066Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_68e17930-c4e8-46a4-b943-ce2dfb9c32f7/root/var/svc/log/oxide-crucible-downstairs:downstairs-1ba09f7d-c664-49b7-9561-eed1b53f2403.log (5066 bytes)
10292023-09-20T03:54:48.139Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_68e17930-c4e8-46a4-b943-ce2dfb9c32f7/root/var/svc/log/oxide-crucible-downstairs:downstairs-1ba09f7d-c664-49b7-9561-eed1b53f2403.log
10302023-09-20T03:54:48.164Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8f2b6ec7-09eb-4cc0-b44a-b1b0c56c1b60/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10312023-09-20T03:54:49.235Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8f2b6ec7-09eb-4cc0-b44a-b1b0c56c1b60/root/var/svc/log/oxide-crucible-agent:default.log
10322023-09-20T03:54:49.259Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8f2b6ec7-09eb-4cc0-b44a-b1b0c56c1b60/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10332023-09-20T03:54:50.330Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8f2b6ec7-09eb-4cc0-b44a-b1b0c56c1b60/root/var/svc/log/oxide-crucible-downstairs:default.log
10342023-09-20T03:54:50.355Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3e3f164a-de35-4de8-a1bc-1a8d70df38c4/root/var/svc/log/oxide-oximeter:default.log (153144 bytes)
10352023-09-20T03:54:50.514Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3e3f164a-de35-4de8-a1bc-1a8d70df38c4/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 153144 -> 159393
10362023-09-20T03:54:51.585Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3e3f164a-de35-4de8-a1bc-1a8d70df38c4/root/var/svc/log/oxide-oximeter:default.log
10372023-09-20T03:54:51.609Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_14a002ac-4996-43b3-92d5-ecae98936197/root/var/svc/log/oxide-cockroachdb:default.log (5342 bytes)
10382023-09-20T03:54:52.683Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_14a002ac-4996-43b3-92d5-ecae98936197/root/var/svc/log/oxide-cockroachdb:default.log
10392023-09-20T03:54:52.707Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6dd2ceaa-bb97-452d-b091-627d2fb3ff54/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10402023-09-20T03:54:53.788Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6dd2ceaa-bb97-452d-b091-627d2fb3ff54/root/var/svc/log/oxide-cockroachdb:default.log
10412023-09-20T03:54:53.812Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ae15ceaa-bd15-472e-9980-f9899a74fd71/root/var/svc/log/oxide-crucible-agent:default.log (13870 bytes)
10422023-09-20T03:54:54.907Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ae15ceaa-bd15-472e-9980-f9899a74fd71/root/var/svc/log/oxide-crucible-agent:default.log
10432023-09-20T03:54:54.931Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ae15ceaa-bd15-472e-9980-f9899a74fd71/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10442023-09-20T03:54:56.003Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ae15ceaa-bd15-472e-9980-f9899a74fd71/root/var/svc/log/oxide-crucible-downstairs:default.log
10452023-09-20T03:54:56.027Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ae15ceaa-bd15-472e-9980-f9899a74fd71/root/var/svc/log/oxide-crucible-downstairs:downstairs-14041659-d215-48cb-baa4-9d8907ec81cc.log (3490 bytes)
10462023-09-20T03:54:57.098Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ae15ceaa-bd15-472e-9980-f9899a74fd71/root/var/svc/log/oxide-crucible-downstairs:downstairs-14041659-d215-48cb-baa4-9d8907ec81cc.log
10472023-09-20T03:54:57.124Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_50266584-65cc-4749-90f5-03bc0b287e1a/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10482023-09-20T03:54:58.193Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_50266584-65cc-4749-90f5-03bc0b287e1a/root/var/svc/log/oxide-crucible-pantry:default.log
10492023-09-20T03:54:58.217Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_b023678b-9619-4790-b197-c706bb2ff170/root/var/svc/log/oxide-nexus:default.log (453536 bytes)
10502023-09-20T03:54:58.425Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_b023678b-9619-4790-b197-c706bb2ff170/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 453536 -> 454752
10512023-09-20T03:54:59.496Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_b023678b-9619-4790-b197-c706bb2ff170/root/var/svc/log/oxide-nexus:default.log
10522023-09-20T03:54:59.521Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_e2efcbab-f26d-427a-9878-d9bbcbd54b26/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10532023-09-20T03:55:00.592Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_e2efcbab-f26d-427a-9878-d9bbcbd54b26/root/var/svc/log/oxide-cockroachdb:default.log
10542023-09-20T03:55:00.616Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6eead47c-e516-4cfb-b7a6-4469a75c6c3f/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
10552023-09-20T03:55:01.688Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6eead47c-e516-4cfb-b7a6-4469a75c6c3f/root/var/svc/log/oxide-crucible-agent:default.log
10562023-09-20T03:55:01.712Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6eead47c-e516-4cfb-b7a6-4469a75c6c3f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10572023-09-20T03:55:02.784Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6eead47c-e516-4cfb-b7a6-4469a75c6c3f/root/var/svc/log/oxide-crucible-downstairs:default.log
10582023-09-20T03:55:02.809Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_37b2e6d9-6cb2-4beb-a5d7-3d31f3719977/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10592023-09-20T03:55:03.882Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_37b2e6d9-6cb2-4beb-a5d7-3d31f3719977/root/var/svc/log/oxide-crucible-pantry:default.log
10602023-09-20T03:55:03.910Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a96a5506-7b52-4625-ab6d-ab75e188beea/root/var/svc/log/oxide-crucible-agent:default.log (25274 bytes)
10612023-09-20T03:55:05.028Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a96a5506-7b52-4625-ab6d-ab75e188beea/root/var/svc/log/oxide-crucible-agent:default.log
10622023-09-20T03:55:05.054Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a96a5506-7b52-4625-ab6d-ab75e188beea/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10632023-09-20T03:55:06.125Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a96a5506-7b52-4625-ab6d-ab75e188beea/root/var/svc/log/oxide-crucible-downstairs:default.log
10642023-09-20T03:55:06.149Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a96a5506-7b52-4625-ab6d-ab75e188beea/root/var/svc/log/oxide-crucible-downstairs:downstairs-6dd50b14-a5bb-419d-ace1-4817c219a234.log (5064 bytes)
10652023-09-20T03:55:07.223Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a96a5506-7b52-4625-ab6d-ab75e188beea/root/var/svc/log/oxide-crucible-downstairs:downstairs-6dd50b14-a5bb-419d-ace1-4817c219a234.log
10662023-09-20T03:55:07.247Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a96a5506-7b52-4625-ab6d-ab75e188beea/root/var/svc/log/oxide-crucible-downstairs:downstairs-e254180d-41a0-4fa6-acba-8464956708e9.log (3494 bytes)
10672023-09-20T03:55:08.320Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a96a5506-7b52-4625-ab6d-ab75e188beea/root/var/svc/log/oxide-crucible-downstairs:downstairs-e254180d-41a0-4fa6-acba-8464956708e9.log
10682023-09-20T03:55:08.344Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_a296927f-5e12-488a-867b-511d5864f1fe/root/var/svc/log/oxide-internal_dns:default.log (99564 bytes)
10692023-09-20T03:55:08.480Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_a296927f-5e12-488a-867b-511d5864f1fe/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99564 -> 103314
10702023-09-20T03:55:09.552Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_a296927f-5e12-488a-867b-511d5864f1fe/root/var/svc/log/oxide-internal_dns:default.log
10712023-09-20T03:55:09.577Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_237e1ccf-9c97-48c5-bdd2-06af0c9412de/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10722023-09-20T03:55:10.649Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_237e1ccf-9c97-48c5-bdd2-06af0c9412de/root/var/svc/log/oxide-cockroachdb:default.log
10732023-09-20T03:55:10.678Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_f207d0a2-721a-48a5-8439-beab99cf8636/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10742023-09-20T03:55:11.745Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_f207d0a2-721a-48a5-8439-beab99cf8636/root/var/svc/log/oxide-cockroachdb:default.log
10752023-09-20T03:55:11.770Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44a4726c-52f3-4e02-8693-d4a73a0edc01/root/var/svc/log/oxide-crucible-agent:default.log (25277 bytes)
10762023-09-20T03:55:12.886Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44a4726c-52f3-4e02-8693-d4a73a0edc01/root/var/svc/log/oxide-crucible-agent:default.log
10772023-09-20T03:55:12.910Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44a4726c-52f3-4e02-8693-d4a73a0edc01/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10782023-09-20T03:55:13.981Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44a4726c-52f3-4e02-8693-d4a73a0edc01/root/var/svc/log/oxide-crucible-downstairs:default.log
10792023-09-20T03:55:14.005Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44a4726c-52f3-4e02-8693-d4a73a0edc01/root/var/svc/log/oxide-crucible-downstairs:downstairs-eb39c009-a752-49ba-b072-e246ed4f9f70.log (3493 bytes)
10802023-09-20T03:55:15.076Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44a4726c-52f3-4e02-8693-d4a73a0edc01/root/var/svc/log/oxide-crucible-downstairs:downstairs-eb39c009-a752-49ba-b072-e246ed4f9f70.log
10812023-09-20T03:55:15.100Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44a4726c-52f3-4e02-8693-d4a73a0edc01/root/var/svc/log/oxide-crucible-downstairs:downstairs-faebcf53-e2da-4bab-89d0-6a4e2e117803.log (5066 bytes)
10822023-09-20T03:55:16.172Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44a4726c-52f3-4e02-8693-d4a73a0edc01/root/var/svc/log/oxide-crucible-downstairs:downstairs-faebcf53-e2da-4bab-89d0-6a4e2e117803.log
10832023-09-20T03:55:16.196Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_a29497ec-9b94-42a0-9136-a20c59f4c77e/root/var/svc/log/oxide-internal_dns:default.log (102738 bytes)
10842023-09-20T03:55:16.332Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_a29497ec-9b94-42a0-9136-a20c59f4c77e/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 102738 -> 106490
10852023-09-20T03:55:17.403Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_a29497ec-9b94-42a0-9136-a20c59f4c77e/root/var/svc/log/oxide-internal_dns:default.log
10862023-09-20T03:55:17.428Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_a5db5e2c-8e54-4f39-9689-3ea4f264af81/root/var/svc/log/oxide-internal_dns:default.log (99884 bytes)
10872023-09-20T03:55:17.564Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_a5db5e2c-8e54-4f39-9689-3ea4f264af81/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99884 -> 103639
10882023-09-20T03:55:18.636Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_a5db5e2c-8e54-4f39-9689-3ea4f264af81/root/var/svc/log/oxide-internal_dns:default.log
10892023-09-20T03:55:18.661Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_b2e743e1-ac6b-4306-a859-2ab1114b836a/root/var/svc/log/oxide-nexus:default.log (457891 bytes)
10902023-09-20T03:55:18.893Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_b2e743e1-ac6b-4306-a859-2ab1114b836a/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 457891 -> 518505
10912023-09-20T03:55:19.964Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_b2e743e1-ac6b-4306-a859-2ab1114b836a/root/var/svc/log/oxide-nexus:default.log
10922023-09-20T03:55:19.988Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915/root/var/svc/log/oxide-ntp:default.log (1309 bytes)
10932023-09-20T03:55:21.059Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915/root/var/svc/log/oxide-ntp:default.log
10942023-09-20T03:55:21.085Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_66dae034-34bf-419a-9679-63da99019e56/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10952023-09-20T03:55:22.154Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_66dae034-34bf-419a-9679-63da99019e56/root/var/svc/log/oxide-crucible-agent:default.log
10962023-09-20T03:55:22.178Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_66dae034-34bf-419a-9679-63da99019e56/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10972023-09-20T03:55:23.248Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_66dae034-34bf-419a-9679-63da99019e56/root/var/svc/log/oxide-crucible-downstairs:default.log
10982023-09-20T03:55:23.272Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d7ca13eb-fdb6-4c57-9d57-0d46d1616447/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10992023-09-20T03:55:24.345Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d7ca13eb-fdb6-4c57-9d57-0d46d1616447/root/var/svc/log/oxide-crucible-agent:default.log
11002023-09-20T03:55:24.371Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d7ca13eb-fdb6-4c57-9d57-0d46d1616447/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11012023-09-20T03:55:25.454Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d7ca13eb-fdb6-4c57-9d57-0d46d1616447/root/var/svc/log/oxide-crucible-downstairs:default.log
11022023-09-20T03:55:25.478Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bdeabda1-0efa-4bfe-bfa6-fe98d421d013/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
11032023-09-20T03:55:26.549Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bdeabda1-0efa-4bfe-bfa6-fe98d421d013/root/var/svc/log/oxide-crucible-agent:default.log
11042023-09-20T03:55:26.574Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bdeabda1-0efa-4bfe-bfa6-fe98d421d013/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11052023-09-20T03:55:27.643Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bdeabda1-0efa-4bfe-bfa6-fe98d421d013/root/var/svc/log/oxide-crucible-downstairs:default.log
11062023-09-20T03:55:27.668Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_c69b0abd-69bf-4441-b75b-7a922e5e21aa/root/var/svc/log/oxide-crucible-pantry:default.log (2330 bytes)
11072023-09-20T03:55:28.741Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_c69b0abd-69bf-4441-b75b-7a922e5e21aa/root/var/svc/log/oxide-crucible-pantry:default.log
11082023-09-20T03:55:28.765Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_3982ba39-4a3c-46be-b1d4-f0505bc72832/root/var/svc/log/oxide-external_dns:default.log (15611 bytes)
11092023-09-20T03:55:28.833Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_3982ba39-4a3c-46be-b1d4-f0505bc72832/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15611 -> 19389
11102023-09-20T03:55:29.904Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_3982ba39-4a3c-46be-b1d4-f0505bc72832/root/var/svc/log/oxide-external_dns:default.log
11112023-09-20T03:55:29.929Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915/root/var/log/chrony/measurements.log (13289 bytes)
11122023-09-20T03:55:29.974Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915/root/var/log/chrony/measurements.log" changed size mid upload: 13289 -> 15344
11132023-09-20T03:55:31.044Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915/root/var/log/chrony/measurements.log
11142023-09-20T03:55:31.069Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915/root/var/log/chrony/statistics.log (7020 bytes)
11152023-09-20T03:55:31.113Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915/root/var/log/chrony/statistics.log" changed size mid upload: 7020 -> 8658
11162023-09-20T03:55:32.184Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915/root/var/log/chrony/statistics.log
11172023-09-20T03:55:32.209Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915/root/var/log/chrony/tracking.log (1854 bytes)
11182023-09-20T03:55:32.233Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915/root/var/log/chrony/tracking.log" changed size mid upload: 1854 -> 1986
11192023-09-20T03:55:33.301Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_53a0e933-3473-4a9f-b17a-039ac7ee8915/root/var/log/chrony/tracking.log