01HDFWQ8Z8ND9DPHE5SKC91QV9: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HDFWQTQAE3RVZ9TBSCMEHKDM

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-24T04:39:06.263Zjob dependencies complete; ready to run (waiting for 51 m 4 s)
22023-10-24T04:39:57.875Zjob assigned to worker 01HDFZNHRC80VYXCDSBPKQY3Y7 (queued for 51 s)
32023-10-24T04:39:57.903Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-24T04:39:57.956Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-10-24T04:39:57.990Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-10-24T04:39:58.023Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-24T04:39:58.085Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-10-24T04:39:58.123ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-24T04:39:58.157Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-24T04:39:58.190ZOxide Helios Version helios-2.0.22241 64-bit
112023-10-24T04:39:58.228ZConfiguring device.
122023-10-24T04:39:58.276Z##################################################
132023-10-24T04:39:58.313Z#### ############# ##
142023-10-24T04:39:58.348Z### ### ############ ##
152023-10-24T04:39:58.786Z## ### # ## ### ## Oxide Computer Company ##
162023-10-24T04:39:58.824Z## ## ## ### # ### ##
172023-10-24T04:39:58.857Z## # ### #### #### This Station Under ##
182023-10-24T04:39:58.891Z### ### #### # ### Computer Control ##
192023-10-24T04:39:58.924Z#### #### ### ## ##
202023-10-24T04:39:58.958Z##################################################
212023-10-24T04:39:58.991ZHostname: buskin
222023-10-24T04:39:59.024ZLoading smf(7) serice descriptions:
232023-10-24T04:39:59.301Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622.gz
242023-10-24T04:40:00.743Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622.gz
252023-10-24T04:40:00.768Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-24T04:40:26.000Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-24T04:40:26.025Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280.gz
282023-10-24T04:40:50.648Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280.gz
292023-10-24T04:40:50.676Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-24T04:40:51.655Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-24T04:40:51.678Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-24T04:40:56.213Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-24T04:40:56.251Zdownloading input: /input/package/work/version.txt
342023-10-24T04:40:56.281Zdownloaded input: /input/package/work/version.txt
352023-10-24T04:40:56.315Zdownloading input: /input/package/work/package.tar.gz
362023-10-24T04:41:14.048Zdownloaded input: /input/package/work/package.tar.gz
372023-10-24T04:41:14.074Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-24T04:41:40.989Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-24T04:41:41.013Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-24T04:41:48.865Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-24T04:41:48.893Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-24T04:42:30.068Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-24T04:42:30.093Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-24T04:43:08.655Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-24T04:43:08.680Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-24T04:43:46.329Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-24T04:43:46.742Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-24T04:43:50.914Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-24T04:43:50.986Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-24T04:43:58.932Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-24T04:43:58.956Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-24T04:44:02.991Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-24T04:44:03.016Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-24T04:44:08.364Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-24T04:44:08.388Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-24T04:44:08.412Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-24T04:44:08.437Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-24T04:44:11.457Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-24T04:44:11.481Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-24T04:44:14.555Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-24T04:44:14.580Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-24T04:44:26.917Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-24T04:44:26.942Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-24T04:44:40.596Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-24T04:44:40.620Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-24T04:44:40.657Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-24T04:44:40.681Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-24T04:44:44.043Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-24T04:44:44.067Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-24T04:44:55.874Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-24T04:44:55.898Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-24T04:45:39.735Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-24T04:45:39.760Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-24T04:46:08.144Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-24T04:46:08.180Zstarting task 0: "setup"
762023-10-24T04:46:08.213Z++ uname -s
772023-10-24T04:46:08.237Z+ kern=SunOS
782023-10-24T04:46:08.261Z+ case "$kern" in
792023-10-24T04:46:08.286Z+ groupadd -g 12345 build
802023-10-24T04:46:08.310Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-24T04:46:10.237Z+ zfs create -o mountpoint=/work rpool/work
822023-10-24T04:46:10.262Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-24T04:46:10.291Z+ home_fs=zfs
842023-10-24T04:46:10.318Z+ [[ zfs == autofs ]]
852023-10-24T04:46:10.342Z+ mkdir -p /home/build
862023-10-24T04:46:10.366Z+ chown build:build /home/build /work
872023-10-24T04:46:12.261Z+ chmod 0700 /home/build /work
882023-10-24T04:46:12.307Zprocess exited: duration 4058 ms, exit code 0
 
892023-10-24T04:46:12.367Zstarting task 1: "authentication"
902023-10-24T04:46:12.437Zprocess exited: duration 45 ms, exit code 0
 
912023-10-24T04:46:12.485Zstarting task 2: "build"
922023-10-24T04:46:12.516Z+ trap _exit_trap EXIT
932023-10-24T04:46:12.540Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-24T04:46:12.712Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-24T04:46:12.736Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-24T04:46:12.760Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-24T04:46:12.786Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-24T04:46:12.811Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-24T04:46:12.834Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-24T04:46:12.858Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-24T04:46:12.885Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-24T04:46:12.911Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-24T04:46:12.936Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-24T04:46:12.973Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-24T04:46:12.999Z+ [[ -d /opt/oxide ]]
1062023-10-24T04:46:13.023Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-24T04:46:13.050Z+ pfexec mkdir /opt/oxide
1082023-10-24T04:46:13.074Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-24T04:46:13.098Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-24T04:46:13.122Z+ cd /opt/oxide-underneath
1112023-10-24T04:46:13.145Z+ pfexec tar ceEp@/f - .
1122023-10-24T04:46:13.169Z+ cd /opt/oxide
1132023-10-24T04:46:13.194Z+ pfexec tar xveEp@/f -
1142023-10-24T04:46:13.218Zx ., 0 bytes, 0 tape blocks
1152023-10-24T04:46:13.242Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-24T04:46:13.266Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-24T04:46:13.290Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-10-24T04:46:13.332Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-24T04:46:13.357Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-24T04:46:13.381Z+ pfexec mkdir /opt/oxide/work
1212023-10-24T04:46:13.411Z+ pfexec chown build:build /opt/oxide/work
1222023-10-24T04:46:13.868Z+ cd /opt/oxide/work
1232023-10-24T04:46:13.894Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-24T04:46:14.438ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-24T04:46:14.464Zx out/mg-ddm-gz.tar, 63638528 bytes, 124294 tape blocks
1262023-10-24T04:46:14.896Zx out/omicron-sled-agent.tar, 115460608 bytes, 225509 tape blocks
1272023-10-24T04:46:14.920Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-24T04:46:14.982Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-10-24T04:46:15.007Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-10-24T04:46:15.031Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-24T04:46:15.056Zx target/release/omicron-package, 38196032 bytes, 74602 tape blocks
1322023-10-24T04:46:15.081Zx tools/create_virtual_hardware.sh, 2986 bytes, 6 tape blocks
1332023-10-24T04:46:15.106Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-24T04:46:15.130Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-24T04:46:15.154Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-24T04:46:15.178Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-24T04:46:15.203Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-24T04:46:15.227Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-24T04:46:15.252Z
1402023-10-24T04:46:15.276Zreal 1.562385424
1412023-10-24T04:46:15.301Zuser 1.402471475
1422023-10-24T04:46:15.325Zsys 0.277611090
1432023-10-24T04:46:15.349Ztrap 0.000069750
1442023-10-24T04:46:15.387Ztflt 0.000000000
1452023-10-24T04:46:15.411Zdflt 0.000000000
1462023-10-24T04:46:15.435Zkflt 0.000000000
1472023-10-24T04:46:15.459Zlock 0.000000000
1482023-10-24T04:46:15.483Zslp 1.408914095
1492023-10-24T04:46:15.507Zlat 0.033757343
1502023-10-24T04:46:15.531Zstop 0.000120112
1512023-10-24T04:46:15.555Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-24T04:46:15.580Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-24T04:46:15.604Z+ mkdir tests
1542023-10-24T04:46:15.628Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-24T04:46:15.652Z+ ptime -m gunzip
1562023-10-24T04:46:15.676Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622
1572023-10-24T04:46:15.700Z
1582023-10-24T04:46:15.728Zreal 0.142596893
1592023-10-24T04:46:15.756Zuser 0.129701899
1602023-10-24T04:46:15.780Zsys 0.012646428
1612023-10-24T04:46:15.806Ztrap 0.000011281
1622023-10-24T04:46:15.830Ztflt 0.000000000
1632023-10-24T04:46:15.855Zdflt 0.000000000
1642023-10-24T04:46:15.888Zkflt 0.000000000
1652023-10-24T04:46:15.912Zlock 0.000000000
1662023-10-24T04:46:15.936Zslp 0.000022810
1672023-10-24T04:46:15.960Zlat 0.000022110
1682023-10-24T04:46:15.984Zstop 0.000128103
1692023-10-24T04:46:16.008Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622
1702023-10-24T04:46:16.034Z+ chmod a+x tests/bootstrap-49e31e75b930b622
1712023-10-24T04:46:16.058Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-24T04:46:16.082Z+ ptime -m gunzip
1732023-10-24T04:46:16.106Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-24T04:46:17.911Z
1752023-10-24T04:46:17.937Zreal 2.252357795
1762023-10-24T04:46:17.961Zuser 2.101425248
1772023-10-24T04:46:17.985Zsys 0.150510447
1782023-10-24T04:46:18.009Ztrap 0.000180483
1792023-10-24T04:46:18.041Ztflt 0.000000000
1802023-10-24T04:46:18.069Zdflt 0.000000000
1812023-10-24T04:46:18.093Zkflt 0.000000000
1822023-10-24T04:46:18.117Zlock 0.000000000
1832023-10-24T04:46:18.141Zslp 0.000021060
1842023-10-24T04:46:18.165Zlat 0.000023221
1852023-10-24T04:46:18.189Zstop 0.000135453
1862023-10-24T04:46:18.213Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-24T04:46:18.237Z+ chmod a+x tests/bootstrap
1882023-10-24T04:46:18.261Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-24T04:46:18.285Z+ ptime -m gunzip
1902023-10-24T04:46:18.309Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280
1912023-10-24T04:46:20.503Z
1922023-10-24T04:46:20.532Zreal 2.578740000
1932023-10-24T04:46:20.564Zuser 2.416513150
1942023-10-24T04:46:20.591Zsys 0.161776881
1952023-10-24T04:46:20.615Ztrap 0.000210952
1962023-10-24T04:46:20.639Ztflt 0.000000000
1972023-10-24T04:46:20.663Zdflt 0.000000000
1982023-10-24T04:46:20.686Zkflt 0.000000000
1992023-10-24T04:46:20.710Zlock 0.000000000
2002023-10-24T04:46:20.734Zslp 0.000023500
2012023-10-24T04:46:20.758Zlat 0.000022200
2022023-10-24T04:46:20.782Zstop 0.000127513
2032023-10-24T04:46:20.806Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280
2042023-10-24T04:46:20.829Z+ chmod a+x tests/end_to_end_tests-a91cf9af3457d280
2052023-10-24T04:46:20.853Z+ bmat address ls
2062023-10-24T04:46:20.878ZNAME CIDR FIRST LAST COUNT
2072023-10-24T04:46:20.902Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-10-24T04:46:20.926Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-24T04:46:20.951Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-10-24T04:46:20.977Z+ EXTRA_IP_START=10.151.1.164
2112023-10-24T04:46:21.005Z+ EXTRA_IP_END=10.151.1.227
2122023-10-24T04:46:21.031Z+ EXTRA_IP_COUNT=64
2132023-10-24T04:46:21.060Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-10-24T04:46:21.084Z+ EXTRA_IP_BASE=10.151.1
2152023-10-24T04:46:21.117Z+ EXTRA_IP_FOCTET=164
2162023-10-24T04:46:21.141Z+ EXTRA_IP_LOCTET=227
2172023-10-24T04:46:21.165Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-10-24T04:46:21.189Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-10-24T04:46:21.213Z+ DNS_IP1=10.151.1.164
2202023-10-24T04:46:21.237Z+ DNS_IP2=10.151.1.165
2212023-10-24T04:46:21.261Z+ UPLINK_IP=10.151.1.174
2222023-10-24T04:46:21.285Z+ IPPOOL_START=10.151.1.175
2232023-10-24T04:46:21.309Z+ IPPOOL_END=10.151.1.227
2242023-10-24T04:46:21.333Z++ ipadm show-addr -po type,addr
2252023-10-24T04:46:21.357Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-10-24T04:46:21.382Z+ GATEWAY_IP=10.151.1.100
2272023-10-24T04:46:21.406Z+ [[ -n 10.151.1.100 ]]
2282023-10-24T04:46:21.430Z+ ping -s 10.151.1.100 56 1
2292023-10-24T04:46:21.454ZPING 10.151.1.100: 56 data bytes
2302023-10-24T04:46:21.479Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
2312023-10-24T04:46:21.503Z
2322023-10-24T04:46:21.562Z----10.151.1.100 PING Statistics----
2332023-10-24T04:46:21.586Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-10-24T04:46:21.610Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-10-24T04:46:21.635Z++ arp -an
2362023-10-24T04:46:21.659Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-10-24T04:46:21.683Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-10-24T04:46:21.707Z+ routeadm -e ipv4-forwarding -u
2392023-10-24T04:46:21.731Z+ PXA_START=10.151.1.164
2402023-10-24T04:46:21.755Z+ PXA_END=10.151.1.227
2412023-10-24T04:46:21.779Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-10-24T04:46:21.803Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-10-24T04:46:21.827Z+ ZPOOL_VDEV_DIR=/scratch
2442023-10-24T04:46:21.852Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-10-24T04:46:21.876Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-10-24T04:46:21.901Z++ cd ./tools
2472023-10-24T04:46:21.925Z++ pwd
2482023-10-24T04:46:21.950Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-10-24T04:46:21.976Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-10-24T04:46:22.002Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-10-24T04:46:22.028Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-10-24T04:46:22.055Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-10-24T04:46:22.083Z++ dladm show-phys -p -o LINK
2542023-10-24T04:46:22.107Z++ head -1
2552023-10-24T04:46:22.131Z+ PHYSICAL_LINK=igb0
2562023-10-24T04:46:22.155Z+ echo 'Using igb0 as physical link'
2572023-10-24T04:46:22.179ZUsing igb0 as physical link
2582023-10-24T04:46:22.203Z+ ensure_run_as_root
2592023-10-24T04:46:22.227Z++ id -u
2602023-10-24T04:46:22.251Z+ [[ 0 -ne 0 ]]
2612023-10-24T04:46:22.281Z+ ensure_zpools
2622023-10-24T04:46:22.305Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-10-24T04:46:22.329Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-10-24T04:46:22.357Z+ readarray -t ZPOOLS
2652023-10-24T04:46:22.381Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-10-24T04:46:22.406Z++ sed 's/[ ",]//g'
2672023-10-24T04:46:22.430Z+ for ZPOOL in "${ZPOOLS[@]}"
2682023-10-24T04:46:22.454Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2692023-10-24T04:46:22.481ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2702023-10-24T04:46:22.510Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2712023-10-24T04:46:22.534Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2722023-10-24T04:46:22.558Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2732023-10-24T04:46:22.586Z0+0 records in
2742023-10-24T04:46:22.610Z0+0 records out
2752023-10-24T04:46:22.634ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2762023-10-24T04:46:22.659Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2772023-10-24T04:46:22.686ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2782023-10-24T04:46:22.714Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2792023-10-24T04:46:22.749ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2802023-10-24T04:46:22.784Z+ set +x
2812023-10-24T04:46:22.816ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2822023-10-24T04:46:22.844Z++ zpool list -o name
2832023-10-24T04:46:22.869ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2842023-10-24T04:46:22.893Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2852023-10-24T04:46:22.921ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2862023-10-24T04:46:22.945Z+ [[ -z '' ]]
2872023-10-24T04:46:22.970ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2882023-10-24T04:46:22.994Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2892023-10-24T04:46:23.018ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2902023-10-24T04:46:23.041Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2912023-10-24T04:46:23.065ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2922023-10-24T04:46:23.089Z+ set +x
2932023-10-24T04:46:23.114ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2942023-10-24T04:46:23.138Z+ for ZPOOL in "${ZPOOLS[@]}"
2952023-10-24T04:46:23.163ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2962023-10-24T04:46:23.188Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2972023-10-24T04:46:23.217ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2982023-10-24T04:46:23.249Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2992023-10-24T04:46:23.276ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3002023-10-24T04:46:23.300Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3012023-10-24T04:46:23.329ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3022023-10-24T04:46:23.355Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3032023-10-24T04:46:23.381ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3042023-10-24T04:46:23.407Z0+0 records in
3052023-10-24T04:46:23.434ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3062023-10-24T04:46:23.461Z0+0 records out
3072023-10-24T04:46:23.485ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3082023-10-24T04:46:23.509Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3092023-10-24T04:46:23.533ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3102023-10-24T04:46:23.557Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3112023-10-24T04:46:23.581ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3122023-10-24T04:46:23.610Z+ set +x
3132023-10-24T04:46:23.635ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3142023-10-24T04:46:23.662Z++ zpool list -o name
3152023-10-24T04:46:23.690ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3162023-10-24T04:46:23.721Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3172023-10-24T04:46:23.747ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3182023-10-24T04:46:23.771Z+ [[ -z '' ]]
3192023-10-24T04:46:23.796ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3202023-10-24T04:46:23.820Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3212023-10-24T04:46:23.844ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3222023-10-24T04:46:23.873Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-10-24T04:46:23.901ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3242023-10-24T04:46:23.924Z+ set +x
3252023-10-24T04:46:23.950ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3262023-10-24T04:46:23.974Z+ for ZPOOL in "${ZPOOLS[@]}"
3272023-10-24T04:46:23.998ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3282023-10-24T04:46:24.023Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3292023-10-24T04:46:24.047ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3302023-10-24T04:46:24.077Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3312023-10-24T04:46:24.106ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3322023-10-24T04:46:24.570Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3332023-10-24T04:46:24.595ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3342023-10-24T04:46:24.638Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3352023-10-24T04:46:24.662ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3362023-10-24T04:46:24.687Z0+0 records in
3372023-10-24T04:46:24.711ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3382023-10-24T04:46:24.735Z0+0 records out
3392023-10-24T04:46:24.759ZSimnet net0/sc0_0 exists
3402023-10-24T04:46:24.783Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3412023-10-24T04:46:24.808ZSimnet net1/sc1_0 exists
3422023-10-24T04:46:24.837Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3432023-10-24T04:46:24.864ZVnic sc0_1 exists
3442023-10-24T04:46:24.889Z+ set +x
3452023-10-24T04:46:25.323Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3462023-10-24T04:46:25.353Z++ zpool list -o name
3472023-10-24T04:46:25.378Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3482023-10-24T04:46:25.403Z+ [[ -z '' ]]
3492023-10-24T04:46:25.432Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-10-24T04:46:25.459Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-10-24T04:46:25.486Z+ set +x
3522023-10-24T04:46:25.513Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-10-24T04:46:25.539Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-10-24T04:46:25.563Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-10-24T04:46:25.586Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-10-24T04:46:25.610Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-10-24T04:46:25.635Z0+0 records in
3582023-10-24T04:46:25.673Z0+0 records out
3592023-10-24T04:46:25.698Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3602023-10-24T04:46:25.730Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-10-24T04:46:25.754Z+ set +x
3622023-10-24T04:46:25.778Z++ zpool list -o name
3632023-10-24T04:46:25.802Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-10-24T04:46:25.826Z+ [[ -z '' ]]
3652023-10-24T04:46:25.850Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-10-24T04:46:25.874Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-10-24T04:46:25.901Z+ set +x
3682023-10-24T04:46:25.926Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-10-24T04:46:25.951Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-10-24T04:46:25.975Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-10-24T04:46:26.001Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-10-24T04:46:26.025Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-10-24T04:46:26.049Z0+0 records in
3742023-10-24T04:46:26.074Z0+0 records out
3752023-10-24T04:46:26.108Z0 bytes (0 iB) transferred in 0.000007 secs (0 iB/sec)
3762023-10-24T04:46:26.135Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-10-24T04:46:26.168Z+ set +x
3782023-10-24T04:46:26.197Z++ zpool list -o name
3792023-10-24T04:46:26.221Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-10-24T04:46:26.245Z+ [[ -z '' ]]
3812023-10-24T04:46:26.269Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-10-24T04:46:26.293Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-10-24T04:46:26.317Z+ set +x
3842023-10-24T04:46:26.348Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-10-24T04:46:26.380Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862023-10-24T04:46:26.404Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872023-10-24T04:46:26.428Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882023-10-24T04:46:26.452Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892023-10-24T04:46:26.476Z0+0 records in
3902023-10-24T04:46:26.500Z0+0 records out
3912023-10-24T04:46:26.525Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-10-24T04:46:26.549Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932023-10-24T04:46:26.574Z+ set +x
3942023-10-24T04:46:26.604Z++ zpool list -o name
3952023-10-24T04:46:26.632Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3962023-10-24T04:46:26.656Z+ [[ -z '' ]]
3972023-10-24T04:46:26.699Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3982023-10-24T04:46:26.723Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3992023-10-24T04:46:26.747Z+ set +x
4002023-10-24T04:46:26.774Z+ for ZPOOL in "${ZPOOLS[@]}"
4012023-10-24T04:46:26.804Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4022023-10-24T04:46:26.830Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4032023-10-24T04:46:26.854Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4042023-10-24T04:46:26.878Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4052023-10-24T04:46:26.903Z0+0 records in
4062023-10-24T04:46:26.927Z0+0 records out
4072023-10-24T04:46:26.951Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4082023-10-24T04:46:26.975Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4092023-10-24T04:46:26.999Z+ set +x
4102023-10-24T04:46:27.023Z++ zpool list -o name
4112023-10-24T04:46:27.047Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4122023-10-24T04:46:27.071Z+ [[ -z '' ]]
4132023-10-24T04:46:27.095Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4142023-10-24T04:46:27.119Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4152023-10-24T04:46:27.143Z+ set +x
4162023-10-24T04:46:27.167Z+ for ZPOOL in "${ZPOOLS[@]}"
4172023-10-24T04:46:27.192Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4182023-10-24T04:46:27.219Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192023-10-24T04:46:27.244Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4202023-10-24T04:46:27.267Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4212023-10-24T04:46:27.292Z0+0 records in
4222023-10-24T04:46:27.316Z0+0 records out
4232023-10-24T04:46:27.340Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4242023-10-24T04:46:27.364Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4252023-10-24T04:46:27.388Z+ set +x
4262023-10-24T04:46:27.413Z++ zpool list -o name
4272023-10-24T04:46:27.447Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4282023-10-24T04:46:27.473Z+ [[ -z '' ]]
4292023-10-24T04:46:27.501Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4302023-10-24T04:46:27.934Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4312023-10-24T04:46:27.960Z+ set +x
4322023-10-24T04:46:27.984Z+ for ZPOOL in "${ZPOOLS[@]}"
4332023-10-24T04:46:28.008Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4342023-10-24T04:46:28.032Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4352023-10-24T04:46:28.057Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4362023-10-24T04:46:28.081Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4372023-10-24T04:46:28.106Z0+0 records in
4382023-10-24T04:46:28.130Z0+0 records out
4392023-10-24T04:46:28.154Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4402023-10-24T04:46:28.178Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4412023-10-24T04:46:28.202Z+ set +x
4422023-10-24T04:46:28.226Z++ zpool list -o name
4432023-10-24T04:46:28.250Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4442023-10-24T04:46:28.275Z+ [[ -z '' ]]
4452023-10-24T04:46:28.299Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4462023-10-24T04:46:28.323Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4472023-10-24T04:46:28.348Z+ set +x
4482023-10-24T04:46:28.372Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4492023-10-24T04:46:28.396Z+ readarray -t ZPOOLS
4502023-10-24T04:46:28.420Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4512023-10-24T04:46:28.445Z++ sed 's/[ ",]//g'
4522023-10-24T04:46:28.469Z+ for ZPOOL in "${ZPOOLS[@]}"
4532023-10-24T04:46:28.493Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4542023-10-24T04:46:28.579Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4552023-10-24T04:46:28.605Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4562023-10-24T04:46:28.629Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4572023-10-24T04:46:28.654Z0+0 records in
4582023-10-24T04:46:28.677Z0+0 records out
4592023-10-24T04:46:28.701Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4602023-10-24T04:46:28.748Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4612023-10-24T04:46:28.783Z+ set +x
4622023-10-24T04:46:28.809Z++ zpool list -o name
4632023-10-24T04:46:28.836Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4642023-10-24T04:46:28.860Z+ [[ -z '' ]]
4652023-10-24T04:46:28.889Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-10-24T04:46:28.913Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4672023-10-24T04:46:28.939Z+ set +x
4682023-10-24T04:46:28.963Z+ for ZPOOL in "${ZPOOLS[@]}"
4692023-10-24T04:46:28.987Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4702023-10-24T04:46:29.011Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4712023-10-24T04:46:29.036Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4722023-10-24T04:46:29.060Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4732023-10-24T04:46:29.084Z0+0 records in
4742023-10-24T04:46:29.108Z0+0 records out
4752023-10-24T04:46:29.132Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4762023-10-24T04:46:29.156Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4772023-10-24T04:46:29.180Z+ set +x
4782023-10-24T04:46:29.203Z++ zpool list -o name
4792023-10-24T04:46:29.227Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4802023-10-24T04:46:29.252Z+ [[ -z '' ]]
4812023-10-24T04:46:29.276Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4822023-10-24T04:46:29.340Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4832023-10-24T04:46:29.371Z+ set +x
4842023-10-24T04:46:29.397Z+ ensure_simulated_links igb0
4852023-10-24T04:46:29.423Z+ local PHYSICAL_LINK=igb0
4862023-10-24T04:46:29.454Z+ INDICES=("0" "1")
4872023-10-24T04:46:29.485Z+ for I in "${INDICES[@]}"
4882023-10-24T04:46:29.511Z++ get_simnet_name_if_exists net0
4892023-10-24T04:46:29.538Z++ dladm show-simnet -p -o LINK net0
4902023-10-24T04:46:29.565Z++ echo ''
4912023-10-24T04:46:29.593Z+ [[ -z '' ]]
4922023-10-24T04:46:29.619Z+ dladm create-simnet -t net0
4932023-10-24T04:46:29.647Z+ dladm create-simnet -t sc0_0
4942023-10-24T04:46:29.675Z+ dladm modify-simnet -t -p net0 sc0_0
4952023-10-24T04:46:29.704Z+ dladm set-linkprop -p mtu=9000 sc0_0
4962023-10-24T04:46:29.731Z+ success 'Simnet net0/sc0_0 exists'
4972023-10-24T04:46:29.813Z+ set +x
4982023-10-24T04:46:29.839Z+ for I in "${INDICES[@]}"
4992023-10-24T04:46:29.863Z++ get_simnet_name_if_exists net1
5002023-10-24T04:46:29.888Z++ dladm show-simnet -p -o LINK net1
5012023-10-24T04:46:29.912Z++ echo ''
5022023-10-24T04:46:29.937Z+ [[ -z '' ]]
5032023-10-24T04:46:29.966Z+ dladm create-simnet -t net1
5042023-10-24T04:46:30.032Z+ dladm create-simnet -t sc1_0
5052023-10-24T04:46:30.056Z+ dladm modify-simnet -t -p net1 sc1_0
5062023-10-24T04:46:30.081Z+ dladm set-linkprop -p mtu=9000 sc1_0
5072023-10-24T04:46:30.108Z+ success 'Simnet net1/sc1_0 exists'
5082023-10-24T04:46:30.154Z+ set +x
5092023-10-24T04:46:30.180Z++ get_vnic_name_if_exists sc0_1
5102023-10-24T04:46:30.208Z++ dladm show-vnic -p -o LINK sc0_1
5112023-10-24T04:46:30.233Z++ echo ''
5122023-10-24T04:46:30.260Z+ [[ -z '' ]]
5132023-10-24T04:46:30.288Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-10-24T04:46:30.314Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-10-24T04:46:30.344Z+ success 'Vnic sc0_1 exists'
5162023-10-24T04:46:30.376Z+ set +x
5172023-10-24T04:46:30.402Z+ warn_if_no_proxy_arp
5182023-10-24T04:46:30.431Z+ [[ -v PXA_START ]]
5192023-10-24T04:46:30.459Z+ [[ -v PXA_END ]]
5202023-10-24T04:46:30.489Z+ ensure_softnpu_zone
5212023-10-24T04:46:30.515Z+ zoneadm list
5222023-10-24T04:46:30.544Z+ grep -q sidecar_softnpu
5232023-10-24T04:46:30.577Z+ [[ -f out/npuzone/npuzone ]]
5242023-10-24T04:46:30.606Zdownloading softnpu ...done
5252023-10-24T04:46:30.636Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch omicron-tracking
5262023-10-24T04:46:30.663Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/libsidecar_lite.so.sha256.txt
5272023-10-24T04:46:34.079Zdownloading asic_program.so ...done
5282023-10-24T04:46:34.257Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/scadm.sha256.txt
5292023-10-24T04:46:39.635Zdownloading scadm ...done
5302023-10-24T04:46:39.660Zcreating port tfportrear0_0 for sc0_0
5312023-10-24T04:46:39.684Zcreating port tfportqsfp0_0 for sc0_1
5322023-10-24T04:46:39.708Zsoftnpu zone setup
5332023-10-24T04:46:39.732Zconfigure zone
5342023-10-24T04:46:39.756Zinstall zone
5352023-10-24T04:46:40.737Zboot zone
5362023-10-24T04:46:43.840Z[sidecar_softnpu] svcs milestone/network
5372023-10-24T04:46:43.866ZSTATE STIME FMRI
5382023-10-24T04:46:43.891Zonline 4:46:42 svc:/milestone/network:default
5392023-10-24T04:46:43.915Z[sidecar_softnpu] /softnpu/npu start
5402023-10-24T04:46:43.939Z
5412023-10-24T04:46:43.963Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5422023-10-24T04:46:43.987Z+ GATEWAY_IP=10.151.1.100
5432023-10-24T04:46:44.014Z+ echo 'Using 10.151.1.100 as gateway ip'
5442023-10-24T04:46:44.042ZUsing 10.151.1.100 as gateway ip
5452023-10-24T04:46:44.068Z+ [[ ! -v GATEWAY_MAC ]]
5462023-10-24T04:46:44.093Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5472023-10-24T04:46:44.118Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5482023-10-24T04:46:44.142Z+++ echo 18:c0:4d:0d:9f:b2
5492023-10-24T04:46:44.168Z+ COUNT=1
5502023-10-24T04:46:44.192Z+ [[ 1 -ne 1 ]]
5512023-10-24T04:46:44.228Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5522023-10-24T04:46:44.252ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5532023-10-24T04:46:44.276Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5542023-10-24T04:46:44.300Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5552023-10-24T04:46:44.325Z+ PXA_MAC=a8:e1:de:01:70:1d
5562023-10-24T04:46:44.374Z+ [[ -v PXA_START ]]
5572023-10-24T04:46:44.398Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5582023-10-24T04:46:44.422Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5592023-10-24T04:46:44.446Z+ z_scadm dump-state
5602023-10-24T04:46:44.470Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5612023-10-24T04:46:44.494Zlocal v6:
5622023-10-24T04:46:44.531Zlocal v4:
5632023-10-24T04:46:44.558Zrouter v6:
5642023-10-24T04:46:44.582Zrouter v4:
5652023-10-24T04:46:44.606Zresolver v4:
5662023-10-24T04:46:44.630Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5672023-10-24T04:46:44.654Zresolver v6:
5682023-10-24T04:46:44.679Znat v4:
5692023-10-24T04:46:44.703Znat_v6:
5702023-10-24T04:46:44.727Zport_mac:
5712023-10-24T04:46:44.751Zproxy arp:
5722023-10-24T04:46:44.774Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5732023-10-24T04:46:44.798Z+ success 'softnpu zone exists'
5742023-10-24T04:46:44.822Z+ set +x
5752023-10-24T04:46:44.846Zsoftnpu zone exists
5762023-10-24T04:46:44.870Z
5772023-10-24T04:46:44.893Zreal 23.142335040
5782023-10-24T04:46:44.917Zuser 0.702886725
5792023-10-24T04:46:44.941Zsys 1.625394573
5802023-10-24T04:46:44.965Ztrap 0.000090179
5812023-10-24T04:46:44.989Ztflt 0.000223675
5822023-10-24T04:46:45.013Zdflt 0.001393918
5832023-10-24T04:46:45.078Zkflt 0.000002440
5842023-10-24T04:46:45.104Zlock 17:58.896031560
5852023-10-24T04:46:45.128Zslp 1:08.641248039
5862023-10-24T04:46:45.152Zlat 0.074002885
5872023-10-24T04:46:45.176Zstop 0.371395185
5882023-10-24T04:46:45.200Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5892023-10-24T04:46:45.224Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5902023-10-24T04:46:45.253Z+ SILO_NAME=recovery
5912023-10-24T04:46:45.277Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5922023-10-24T04:46:45.301Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5932023-10-24T04:46:45.326Z+ sed -i~ '
5942023-10-24T04:46:45.351Z /^external_dns_ips/c\
5952023-10-24T04:46:45.375Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5962023-10-24T04:46:45.399Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5972023-10-24T04:46:45.423Z /^first/c\
5982023-10-24T04:46:45.447Zfirst = "10.151.1.164"
5992023-10-24T04:46:45.472Z /^last/c\
6002023-10-24T04:46:45.496Zlast = "10.151.1.173"
6012023-10-24T04:46:45.520Z }
6022023-10-24T04:46:45.544Z /^\[rack_network_config/,/^$/ {
6032023-10-24T04:46:45.568Z /^infra_ip_first/c\
6042023-10-24T04:46:45.592Zinfra_ip_first = "10.151.1.174"
6052023-10-24T04:46:45.616Z /^infra_ip_last/c\
6062023-10-24T04:46:45.640Zinfra_ip_last = "10.151.1.174"
6072023-10-24T04:46:45.664Z }
6082023-10-24T04:46:45.690Z /^\[\[rack_network_config.ports/,/^$/ {
6092023-10-24T04:46:45.714Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6102023-10-24T04:46:45.738Z /^routes/c\
6112023-10-24T04:46:45.762Z+++ pkg/config-rss.toml Tue Oct 24 04:46:44 2023
6122023-10-24T04:46:45.789Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6132023-10-24T04:46:45.815Z@@ -41,7 +41,7 @@
6142023-10-24T04:46:45.840Z /^addresses/c\
6152023-10-24T04:46:45.864Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6162023-10-24T04:46:45.889Zaddresses = \["10.151.1.174/32"\]
6172023-10-24T04:46:45.921Z # must be contained in one of the "internal services" IP Pool ranges listed
6182023-10-24T04:46:45.947Z }
6192023-10-24T04:46:45.973Z # below.
6202023-10-24T04:46:45.997Z' pkg/config-rss.toml
6212023-10-24T04:46:46.021Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6222023-10-24T04:46:46.047Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6232023-10-24T04:46:46.071Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6242023-10-24T04:46:46.096Z+ true
6252023-10-24T04:46:46.119Z
6262023-10-24T04:46:46.144Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6272023-10-24T04:46:46.168Z # Initial TLS certificates for the external API
6282023-10-24T04:46:46.191Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6292023-10-24T04:46:46.215Z #
6302023-10-24T04:46:46.239Z+ TLS_NAME=recovery.sys.oxide.test
6312023-10-24T04:46:46.280Z@@ -77,8 +77,8 @@
6322023-10-24T04:46:46.304Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6332023-10-24T04:46:46.333Z #
6342023-10-24T04:46:46.362Z # For more on this and what to put here, see docs/how-to-run.adoc.
6352023-10-24T04:46:46.386Z [[internal_services_ip_pool_ranges]]
6362023-10-24T04:46:46.410Z-first = "192.168.1.20"
6372023-10-24T04:46:46.434Z-last = "192.168.1.29"
6382023-10-24T04:46:46.458Z+first = "10.151.1.164"
6392023-10-24T04:46:46.482Z+last = "10.151.1.173"
6402023-10-24T04:46:46.506Z
6412023-10-24T04:46:46.530Z # TODO - this configuration is subject to change going forward. Ultimately these
6422023-10-24T04:46:46.554Z # parameters should be provided to the control plane via wicket, but we need to
6432023-10-24T04:46:46.578Z@@ -92,8 +92,8 @@
6442023-10-24T04:46:46.602Z # A range of IP addresses used by Boundary Services on the external network. In
6452023-10-24T04:46:46.626Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6462023-10-24T04:46:46.652Z # With softnpu, only one address is used.
6472023-10-24T04:46:46.676Z.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+............+...+...+...+...............+.+.........+.....+.+...............+...+...+..+...+.......+..+..................+...+...........................+...+...+..........+.....+......+......+.......+...+............+......+.....+......+......+.........+....+..+...+....+...+..+.+...........+....+.....+.+...........+....+.....+...+.......+.....+.+............+..+..........+.....+..................+..........+...+..+...............+...+...................+...........+.......+...+..+..........+.....+......+.......+...+.........+...+..............+....+.....+....+...........+....+..+......+.............+..+....+...+........+....+...+........+.......+...........+......+.+...+.........+..+...............+....+..+.............+......+...............+......+..+.+..............+..................+....+..............+.+..............+..........+....................+.......+..+...+...+.+.....+.+...+......+.....+......+....+...........+.......+......+.....+.........+...+............+...+.............+..+....+.....+...............+....+..+...+.....................+.........+......+.+............+..+..........+........+.......+...+..+.+...............+..+.......+...+........+...+.......+...+.....................+.....+....+..+............+..........+..+.+...+......+.........+..............+.......+...+........+..........+.....+....+.....................+..+..........+.....................+.................+.........+.+..............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6482023-10-24T04:46:46.702Z-infra_ip_first = "192.168.1.30"
6492023-10-24T04:46:46.726Z-infra_ip_last = "192.168.1.30"
6502023-10-24T04:46:46.751Z+infra_ip_first = "10.151.1.174"
6512023-10-24T04:46:46.775Z+infra_ip_last = "10.151.1.174"
6522023-10-24T04:46:46.799Z
6532023-10-24T04:46:46.823Z # Configurations for BGP routers to run on the scrimlets.
6542023-10-24T04:46:46.847Z bgp = []
6552023-10-24T04:46:46.871Z@@ -101,9 +101,9 @@
6562023-10-24T04:46:46.895Z # You can configure multiple uplinks by repeating the following stanza
6572023-10-24T04:46:46.919Z [[rack_network_config.ports]]
6582023-10-24T04:46:46.943Z # Routes associated with this port.
6592023-10-24T04:46:46.967Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6602023-10-24T04:46:46.991Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6612023-10-24T04:46:47.016Z # Addresses associated with this port.
6622023-10-24T04:46:47.040Z-addresses = ["192.168.1.30/32"]
6632023-10-24T04:46:47.064Z+addresses = ["10.151.1.174/32"]
6642023-10-24T04:46:47.088Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6652023-10-24T04:46:47.111Z port = "qsfp0"
6662023-10-24T04:46:47.136Z # The speed of this port.
6672023-10-24T04:46:47.572Za pkg/config-rss.toml 7K
6682023-10-24T04:46:47.598Z.....+.........+.......+......+...+..........................+..........+...+...+..+.............+..+......+......+......+...+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.............+.....+....+.....+.......+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+........................+..+...+......+..............................+..........................................+............+......+.......+..+............+......+...+.........+...+....+.....+......+............................+.....+.+......+............+............+......+..+.+..+...+.+...+...+.........+.........+...........+...............+....+.....+.+........+................+.....+.+...........+...+.+...+...+..+.............+...............+...+.........+............+...+............+...+..+...+................+......+.....+..........+......+.....+...............+....+.........+......+......+...+..+.......+..+...+......+........................+......+....+..+.+...+.....+...................+..+..........+......+...+............+..+.+...........+...+.+..+...............+.......+........+............+..........+......+..+....+........+...+...+..........+......+..................+..+...+............+.........+......+.+.................+...........................+.+..............+.+.........+......+..+...+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6692023-10-24T04:46:47.622Z-----
6702023-10-24T04:46:47.645Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6712023-10-24T04:46:47.676Za pkg/initial-tls-cert.pem 2K
6722023-10-24T04:46:47.702Za pkg/initial-tls-key.pem 4K
6732023-10-24T04:46:47.726Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-10-24T04:46:47.750Z+ rmdir pkg
6752023-10-24T04:46:47.774Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6762023-10-24T04:46:47.800Z+ pfexec mkdir -p /usr/oxide
6772023-10-24T04:46:47.824Z+ pfexec rm -f /usr/oxide/tcpproxy
6782023-10-24T04:46:47.847Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6792023-10-24T04:46:47.872Z+ pfexec chmod +x /usr/oxide/tcpproxy
6802023-10-24T04:46:47.896Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6812023-10-24T04:46:47.920Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6822023-10-24T04:46:47.944Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6832023-10-24T04:46:47.967Z+ OMICRON_NO_UNINSTALL=1
6842023-10-24T04:46:47.991Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6852023-10-24T04:46:48.015ZOct 24 04:46:46.588 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6862023-10-24T04:46:48.039ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6872023-10-24T04:46:48.063ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6882023-10-24T04:46:48.088ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6892023-10-24T04:46:48.118ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6902023-10-24T04:46:48.143ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6912023-10-24T04:46:48.166ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6922023-10-24T04:46:48.190ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6932023-10-24T04:46:48.214ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6942023-10-24T04:46:48.238ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6952023-10-24T04:46:48.263ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6962023-10-24T04:46:48.287ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6972023-10-24T04:46:48.330ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6982023-10-24T04:46:48.354ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6992023-10-24T04:46:48.378ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7002023-10-24T04:46:48.402ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7012023-10-24T04:46:48.426ZOct 24 04:46:46.590 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7022023-10-24T04:46:48.450ZOct 24 04:46:46.814 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7032023-10-24T04:46:48.474ZOct 24 04:46:46.851 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7042023-10-24T04:46:48.499ZOct 24 04:46:46.914 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7052023-10-24T04:46:48.524Z
7062023-10-24T04:46:48.549Zreal 0.467508381
7072023-10-24T04:46:48.577Zuser 0.095922020
7082023-10-24T04:46:48.601Zsys 1.810302475
7092023-10-24T04:46:48.634Ztrap 0.000009340
7102023-10-24T04:46:48.659Ztflt 0.000009950
7112023-10-24T04:46:48.683Zdflt 0.000919540
7122023-10-24T04:46:48.707Zkflt 0.000000000
7132023-10-24T04:46:48.731Zlock 42.047888313
7142023-10-24T04:46:48.755Zslp 0.685188300
7152023-10-24T04:46:48.779Zlat 0.003000042
7162023-10-24T04:46:48.803Zstop 0.205573094
7172023-10-24T04:46:48.829Z+ retry=0
7182023-10-24T04:46:48.853Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7192023-10-24T04:46:48.877Z+ [[ 0 -gt 30 ]]
7202023-10-24T04:46:48.901Z+ sleep 1
7212023-10-24T04:46:48.926Z+ retry=1
7222023-10-24T04:46:48.950Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232023-10-24T04:46:48.974Z+ [[ 1 -gt 30 ]]
7242023-10-24T04:46:48.998Z+ sleep 1
7252023-10-24T04:46:49.022Z+ retry=2
7262023-10-24T04:46:49.048Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272023-10-24T04:47:13.813Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7282023-10-24T04:47:13.842Zlocal v6:
7292023-10-24T04:47:13.866Zfd00:99::1
7302023-10-24T04:47:13.891Zfe80::2c11:6eff:fe92:676a
7312023-10-24T04:47:13.918Zlocal v4:
7322023-10-24T04:47:13.945Z10.151.1.174
7332023-10-24T04:47:13.988Zrouter v6:
7342023-10-24T04:47:14.452Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::9cb0:54ff:fedb:8f6b (1) vid=0
7352023-10-24T04:47:14.477Zfd00:1122:3344:101::/64 -> fe80::9cb0:54ff:fedb:8f6b (1) vid=0
7362023-10-24T04:47:14.502Zrouter v4:
7372023-10-24T04:47:14.526Z0.0.0.0/0 -> 10.151.1.100 (2) vid=0
7382023-10-24T04:47:14.663Zresolver v4:
7392023-10-24T04:47:14.687Z10.151.1.174 -> 62:0c:dc:bc:00:50
7402023-10-24T04:47:14.710Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7412023-10-24T04:47:14.734Zresolver v6:
7422023-10-24T04:47:14.758Zfe80::9cb0:54ff:fedb:8f6b -> 9e:b0:54:db:8f:6b
7432023-10-24T04:47:14.783Zfe80::2c11:6eff:fe92:676a -> 2e:11:6e:92:67:6a
7442023-10-24T04:47:14.807Znat v4:
7452023-10-24T04:47:14.833Znat_v6:
7462023-10-24T04:47:14.859Zport_mac:
7472023-10-24T04:47:14.885Z1: a8:40:25:56:67:31
7482023-10-24T04:47:14.929Z1000: a8:40:25:56:67:30
7492023-10-24T04:47:14.955Zproxy arp:
7502023-10-24T04:47:14.979Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7512023-10-24T04:47:15.012Z+ retry=0
7522023-10-24T04:47:15.037Z+++ zoneadm list -n
7532023-10-24T04:47:15.062Z+++ grep oxz_ntp
7542023-10-24T04:47:15.089Z++ pfexec svcs -z -Hostate oxide/ntp
7552023-10-24T04:47:15.113Zsvcs: zone '-Hostate': entity not found
7562023-10-24T04:47:15.178Z++ true
7572023-10-24T04:47:15.203Z+ [[ '' != online ]]
7582023-10-24T04:47:15.229Z+ [[ 0 -gt 60 ]]
7592023-10-24T04:47:15.253Z+ sleep 1
7602023-10-24T04:47:15.278Z+ retry=1
7612023-10-24T04:47:15.302Z+++ zoneadm list -n
7622023-10-24T04:47:15.326Z+++ grep oxz_ntp
7632023-10-24T04:47:15.350Z++ pfexec svcs -z -Hostate oxide/ntp
7642023-10-24T04:47:15.373Zsvcs: zone '-Hostate': entity not found
7652023-10-24T04:47:15.397Z++ true
7662023-10-24T04:47:15.421Z+ [[ '' != online ]]
7672023-10-24T04:47:15.445Z+ [[ 1 -gt 60 ]]
7682023-10-24T04:47:15.469Z+ sleep 1
7692023-10-24T04:47:15.896Z+ retry=2
7702023-10-24T04:47:15.926Z+++ zoneadm list -n
7712023-10-24T04:47:16.391Z+++ grep oxz_ntp
7722023-10-24T04:47:16.416Z++ pfexec svcs -z -Hostate oxide/ntp
7732023-10-24T04:47:16.441Zsvcs: zone '-Hostate': entity not found
7742023-10-24T04:47:16.465Z++ true
7752023-10-24T04:47:16.489Z+ [[ '' != online ]]
7762023-10-24T04:47:16.513Z+ [[ 2 -gt 60 ]]
7772023-10-24T04:47:16.537Z+ sleep 1
7782023-10-24T04:47:16.917Z+ retry=3
7792023-10-24T04:47:16.942Z+++ zoneadm list -n
7802023-10-24T04:47:16.969Z+++ grep oxz_ntp
7812023-10-24T04:47:16.996Z++ pfexec svcs -z -Hostate oxide/ntp
7822023-10-24T04:47:17.021Zsvcs: zone '-Hostate': entity not found
7832023-10-24T04:47:17.066Z++ true
7842023-10-24T04:47:17.090Z+ [[ '' != online ]]
7852023-10-24T04:47:17.114Z+ [[ 3 -gt 60 ]]
7862023-10-24T04:47:17.139Z+ sleep 1
7872023-10-24T04:47:17.940Z+ retry=4
7882023-10-24T04:47:17.970Z+++ zoneadm list -n
7892023-10-24T04:47:17.993Z+++ grep oxz_ntp
7902023-10-24T04:47:18.018Z++ pfexec svcs -z -Hostate oxide/ntp
7912023-10-24T04:47:18.041Zsvcs: zone '-Hostate': entity not found
7922023-10-24T04:47:18.065Z++ true
7932023-10-24T04:47:19.121Z+ [[ '' != online ]]
7942023-10-24T04:47:19.150Z+ [[ 4 -gt 60 ]]
7952023-10-24T04:47:19.174Z+ sleep 1
7962023-10-24T04:47:19.198Z+ retry=5
7972023-10-24T04:47:19.222Z+++ zoneadm list -n
7982023-10-24T04:47:19.316Z+++ grep oxz_ntp
7992023-10-24T04:47:19.340Z++ pfexec svcs -z -Hostate oxide/ntp
8002023-10-24T04:47:19.364Zsvcs: zone '-Hostate': entity not found
8012023-10-24T04:47:19.389Z++ true
8022023-10-24T04:47:19.413Z+ [[ '' != online ]]
8032023-10-24T04:47:19.437Z+ [[ 5 -gt 60 ]]
8042023-10-24T04:47:19.461Z+ sleep 1
8052023-10-24T04:47:19.984Z+ retry=6
8062023-10-24T04:47:20.009Z+++ zoneadm list -n
8072023-10-24T04:47:20.033Z+++ grep oxz_ntp
8082023-10-24T04:47:20.057Z++ pfexec svcs -z -Hostate oxide/ntp
8092023-10-24T04:47:20.085Zsvcs: zone '-Hostate': entity not found
8102023-10-24T04:47:20.115Z++ true
8112023-10-24T04:47:20.144Z+ [[ '' != online ]]
8122023-10-24T04:47:20.168Z+ [[ 6 -gt 60 ]]
8132023-10-24T04:47:20.192Z+ sleep 1
8142023-10-24T04:47:21.008Z+ retry=7
8152023-10-24T04:47:21.039Z+++ zoneadm list -n
8162023-10-24T04:47:21.063Z+++ grep oxz_ntp
8172023-10-24T04:47:21.087Z++ pfexec svcs -z -Hostate oxide/ntp
8182023-10-24T04:47:21.111Zsvcs: zone '-Hostate': entity not found
8192023-10-24T04:47:21.135Z++ true
8202023-10-24T04:47:21.167Z+ [[ '' != online ]]
8212023-10-24T04:47:21.191Z+ [[ 7 -gt 60 ]]
8222023-10-24T04:47:21.215Z+ sleep 1
8232023-10-24T04:47:22.026Z+ retry=8
8242023-10-24T04:47:22.051Z+++ zoneadm list -n
8252023-10-24T04:47:22.079Z+++ grep oxz_ntp
8262023-10-24T04:47:22.103Z++ pfexec svcs -z -Hostate oxide/ntp
8272023-10-24T04:47:22.127Zsvcs: zone '-Hostate': entity not found
8282023-10-24T04:47:22.151Z++ true
8292023-10-24T04:47:22.193Z+ [[ '' != online ]]
8302023-10-24T04:47:22.217Z+ [[ 8 -gt 60 ]]
8312023-10-24T04:47:22.241Z+ sleep 1
8322023-10-24T04:47:23.052Z+ retry=9
8332023-10-24T04:47:23.076Z+++ zoneadm list -n
8342023-10-24T04:47:23.104Z+++ grep oxz_ntp
8352023-10-24T04:47:23.128Z++ pfexec svcs -z oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097 -Hostate oxide/ntp
8362023-10-24T04:47:23.152Zsvcs: Could not bind to repository server for zone oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097: repository server unavailable
8372023-10-24T04:47:23.176Z++ true
8382023-10-24T04:47:23.219Z+ [[ '' != online ]]
8392023-10-24T04:47:23.244Z+ [[ 9 -gt 60 ]]
8402023-10-24T04:47:23.268Z+ sleep 1
8412023-10-24T04:47:24.070Z+ retry=10
8422023-10-24T04:47:24.094Z+++ zoneadm list -n
8432023-10-24T04:47:24.118Z+++ grep oxz_ntp
8442023-10-24T04:47:24.144Z++ pfexec svcs -z oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097 -Hostate oxide/ntp
8452023-10-24T04:47:24.169Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8462023-10-24T04:47:24.193Z++ true
8472023-10-24T04:47:24.217Z+ [[ '' != online ]]
8482023-10-24T04:47:24.292Z+ [[ 10 -gt 60 ]]
8492023-10-24T04:47:24.316Z+ sleep 1
8502023-10-24T04:47:25.103Z+ retry=11
8512023-10-24T04:47:25.127Z+++ zoneadm list -n
8522023-10-24T04:47:25.151Z+++ grep oxz_ntp
8532023-10-24T04:47:25.175Z++ pfexec svcs -z oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097 -Hostate oxide/ntp
8542023-10-24T04:47:25.200Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8552023-10-24T04:47:25.224Z++ true
8562023-10-24T04:47:25.267Z+ [[ '' != online ]]
8572023-10-24T04:47:25.292Z+ [[ 11 -gt 60 ]]
8582023-10-24T04:47:25.316Z+ sleep 1
8592023-10-24T04:47:26.132Z+ retry=12
8602023-10-24T04:47:26.157Z+++ zoneadm list -n
8612023-10-24T04:47:26.181Z+++ grep oxz_ntp
8622023-10-24T04:47:26.205Z++ pfexec svcs -z oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097 -Hostate oxide/ntp
8632023-10-24T04:47:26.230Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8642023-10-24T04:47:26.254Z++ true
8652023-10-24T04:47:26.292Z+ [[ '' != online ]]
8662023-10-24T04:47:26.316Z+ [[ 12 -gt 60 ]]
8672023-10-24T04:47:26.340Z+ sleep 1
8682023-10-24T04:47:27.162Z+ retry=13
8692023-10-24T04:47:27.186Z+++ zoneadm list -n
8702023-10-24T04:47:27.210Z+++ grep oxz_ntp
8712023-10-24T04:47:27.234Z++ pfexec svcs -z oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097 -Hostate oxide/ntp
8722023-10-24T04:47:27.261Z+ [[ disabled != online ]]
8732023-10-24T04:47:27.285Z+ [[ 13 -gt 60 ]]
8742023-10-24T04:47:27.316Z+ sleep 1
8752023-10-24T04:47:28.185Z+ retry=14
8762023-10-24T04:47:28.210Z+++ zoneadm list -n
8772023-10-24T04:47:28.234Z+++ grep oxz_ntp
8782023-10-24T04:47:28.262Z++ pfexec svcs -z oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097 -Hostate oxide/ntp
8792023-10-24T04:47:28.286Z+ [[ disabled != online ]]
8802023-10-24T04:47:28.310Z+ [[ 14 -gt 60 ]]
8812023-10-24T04:47:28.344Z+ sleep 1
8822023-10-24T04:47:29.207Z+ retry=15
8832023-10-24T04:47:29.232Z+++ zoneadm list -n
8842023-10-24T04:47:29.256Z+++ grep oxz_ntp
8852023-10-24T04:47:29.282Z++ pfexec svcs -z oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097 -Hostate oxide/ntp
8862023-10-24T04:47:29.308Z+ [[ disabled != online ]]
8872023-10-24T04:47:29.335Z+ [[ 15 -gt 60 ]]
8882023-10-24T04:47:29.369Z+ sleep 1
8892023-10-24T04:47:30.232Z+ retry=16
8902023-10-24T04:47:30.257Z+++ zoneadm list -n
8912023-10-24T04:47:30.284Z+++ grep oxz_ntp
8922023-10-24T04:47:30.318Z++ pfexec svcs -z oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097 -Hostate oxide/ntp
8932023-10-24T04:47:30.346Z+ [[ disabled != online ]]
8942023-10-24T04:47:30.398Z+ [[ 16 -gt 60 ]]
8952023-10-24T04:47:30.423Z+ sleep 1
8962023-10-24T04:47:31.252Z+ retry=17
8972023-10-24T04:47:31.276Z+++ zoneadm list -n
8982023-10-24T04:47:31.301Z+++ grep oxz_ntp
8992023-10-24T04:47:31.325Z++ pfexec svcs -z oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097 -Hostate oxide/ntp
9002023-10-24T04:47:31.349Z+ [[ disabled != online ]]
9012023-10-24T04:47:31.373Z+ [[ 17 -gt 60 ]]
9022023-10-24T04:47:31.421Z+ sleep 1
9032023-10-24T04:47:32.275Z+ retry=18
9042023-10-24T04:47:32.299Z+++ zoneadm list -n
9052023-10-24T04:47:32.324Z+++ grep oxz_ntp
9062023-10-24T04:47:32.348Z++ pfexec svcs -z oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097 -Hostate oxide/ntp
9072023-10-24T04:47:32.372Z+ [[ disabled != online ]]
9082023-10-24T04:47:32.396Z+ [[ 18 -gt 60 ]]
9092023-10-24T04:47:32.420Z+ sleep 1
9102023-10-24T04:47:33.298Z+ retry=19
9112023-10-24T04:47:33.323Z+++ zoneadm list -n
9122023-10-24T04:47:33.347Z+++ grep oxz_ntp
9132023-10-24T04:47:33.371Z++ pfexec svcs -z oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097 -Hostate oxide/ntp
9142023-10-24T04:47:33.395Z+ [[ disabled != online ]]
9152023-10-24T04:47:33.420Z+ [[ 19 -gt 60 ]]
9162023-10-24T04:47:33.443Z+ sleep 1
9172023-10-24T04:47:34.319Z+ retry=20
9182023-10-24T04:47:34.344Z+++ zoneadm list -n
9192023-10-24T04:47:34.368Z+++ grep oxz_ntp
9202023-10-24T04:47:34.392Z++ pfexec svcs -z oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097 -Hostate oxide/ntp
9212023-10-24T04:47:34.416Z+ [[ disabled != online ]]
9222023-10-24T04:47:34.440Z+ [[ 20 -gt 60 ]]
9232023-10-24T04:47:34.464Z+ sleep 1
9242023-10-24T04:47:35.341Z+ retry=21
9252023-10-24T04:47:35.365Z+++ zoneadm list -n
9262023-10-24T04:47:35.389Z+++ grep oxz_ntp
 
9272023-10-24T04:47:35.401Zbuskin console login: Oct 24 04:47:35 bukin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9282023-10-24T04:47:35.413Z++ pfexec svcs -z oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097 -Hostate oxide/ntp
9292023-10-24T04:47:35.437Z+ [[ disabled != online ]]
9302023-10-24T04:47:35.461Z+ [[ 21 -gt 60 ]]
9312023-10-24T04:47:35.485Z+ sleep 1
9322023-10-24T04:47:36.363Z+ retry=22
9332023-10-24T04:47:36.389Z+++ zoneadm list -n
9342023-10-24T04:47:36.413Z+++ grep oxz_ntp
9352023-10-24T04:47:36.436Z++ pfexec svcs -z oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097 -Hostate oxide/ntp
9362023-10-24T04:47:36.460Z+ [[ online != online ]]
9372023-10-24T04:47:36.484Z+ echo 'Waited for chrony: 22s'
9382023-10-24T04:47:36.508ZWaited for chrony: 22s
9392023-10-24T04:47:36.550Z+ export RUST_BACKTRACE=1
9402023-10-24T04:47:36.574Z+ RUST_BACKTRACE=1
9412023-10-24T04:47:36.598Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9422023-10-24T04:47:36.622Z+ ./tests/bootstrap
9432023-10-24T04:47:36.646ZUsing external DNS server at 10.151.1.164:53
9442023-10-24T04:47:36.672Z2023-10-24 04:47:36.850886176 UTC: attempting to log into API
9452023-10-24T04:47:51.451Z2023-10-24 04:47:51.744414008 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9462023-10-24T04:47:52.452Z2023-10-24 04:47:52.743352396 UTC: attempting to log into API
9472023-10-24T04:48:07.485Z2023-10-24 04:48:07.735425579 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9482023-10-24T04:48:08.486Z2023-10-24 04:48:08.733454051 UTC: attempting to log into API
9492023-10-24T04:48:23.519Z2023-10-24 04:48:23.725934396 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9502023-10-24T04:48:24.520Z2023-10-24 04:48:24.723911847 UTC: attempting to log into API
9512023-10-24T04:48:39.553Z2023-10-24 04:48:39.716077453 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9522023-10-24T04:48:40.554Z2023-10-24 04:48:40.715026400 UTC: attempting to log into API
9532023-10-24T04:48:55.593Z2023-10-24 04:48:55.707324920 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9542023-10-24T04:48:56.588Z2023-10-24 04:48:56.705319692 UTC: attempting to log into API
9552023-10-24T04:49:11.622Z2023-10-24 04:49:11.697687145 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9562023-10-24T04:49:12.622Z2023-10-24 04:49:12.695720058 UTC: attempting to log into API
9572023-10-24T04:49:27.655Z2023-10-24 04:49:27.687657248 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9582023-10-24T04:49:28.656Z2023-10-24 04:49:28.686620475 UTC: attempting to log into API
9592023-10-24T04:49:43.695Z2023-10-24 04:49:43.679354638 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9602023-10-24T04:49:44.691Z2023-10-24 04:49:44.678255184 UTC: attempting to log into API
9612023-10-24T04:49:59.725Z2023-10-24 04:49:59.711984852 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9622023-10-24T04:50:00.725Z2023-10-24 04:50:00.712923580 UTC: attempting to log into API
9632023-10-24T04:50:15.764Z2023-10-24 04:50:15.745594470 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9642023-10-24T04:50:16.759Z2023-10-24 04:50:16.746288142 UTC: attempting to log into API
9652023-10-24T04:50:31.805Z2023-10-24 04:50:31.786206360 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9662023-10-24T04:50:32.800Z2023-10-24 04:50:32.786874701 UTC: attempting to log into API
9672023-10-24T04:50:47.843Z2023-10-24 04:50:47.829462399 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9682023-10-24T04:51:08.337Z2023-10-24 04:50:48.830410028 UTC: attempting to log into API
9692023-10-24T04:51:08.535Z2023-10-24 04:51:03.865266693 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9702023-10-24T04:51:08.735Z2023-10-24 04:51:04.866504488 UTC: attempting to log into API
9712023-10-24T04:51:19.913Z2023-10-24 04:51:19.899839719 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9722023-10-24T04:51:20.913Z2023-10-24 04:51:20.900529359 UTC: attempting to log into API
 
9732023-10-24T04:51:32.269ZOct 24 04:51:31 bukin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9742023-10-24T04:51:35.957Z2023-10-24 04:51:35.943195423 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9752023-10-24T04:51:36.956Z2023-10-24 04:51:36.944032476 UTC: attempting to log into API
 
9762023-10-24T04:51:39.777ZOct 24 04:51:39 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9772023-10-24T04:51:47.026ZOct 24 04:51:46 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9782023-10-24T04:51:51.989Z2023-10-24 04:51:51.978061057 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9792023-10-24T04:51:52.990Z2023-10-24 04:51:52.979135667 UTC: attempting to log into API
 
9802023-10-24T04:51:54.021ZOct 24 04:51:53 bukin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9812023-10-24T04:52:01.304ZOct 24 04:52:00 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9822023-10-24T04:52:08.024Z2023-10-24 04:52:08.012249388 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9832023-10-24T04:52:09.024Z2023-10-24 04:52:09.013228836 UTC: attempting to log into API
9842023-10-24T04:52:19.065Z2023-10-24 04:52:19.053423449 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: dns error: no record found for Query { name: Name("recovery.sys.oxide.test."), query_type: AAAA, query_class: IN }
9852023-10-24T04:52:20.070Z2023-10-24 04:52:20.054701056 UTC: attempting to log into API
9862023-10-24T04:52:20.101Z2023-10-24 04:52:20.089603382 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: tcp connect error: Connection refused (os error 146)
9872023-10-24T04:52:21.102Z2023-10-24 04:52:21.090901750 UTC: attempting to log into API
9882023-10-24T04:52:21.869Z2023-10-24 04:52:21.858368780 UTC: login succeeded
9892023-10-24T04:52:21.902Zwaiting for nexus to come up...
9902023-10-24T04:52:21.953Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9912023-10-24T04:52:22.110Zensuring datasets are ready...
9922023-10-24T04:52:24.010Zlet's roll.
9932023-10-24T04:52:24.035Z+ rm ./tests/bootstrap
9942023-10-24T04:52:24.086Z+ for test_bin in tests/*
9952023-10-24T04:52:24.110Z+ ./tests/bootstrap-49e31e75b930b622
9962023-10-24T04:52:24.135Z
9972023-10-24T04:52:24.159Zrunning 0 tests
9982023-10-24T04:52:24.183Z
9992023-10-24T04:52:24.207Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10002023-10-24T04:52:24.235Z
10012023-10-24T04:52:24.266Z+ for test_bin in tests/*
10022023-10-24T04:52:24.290Z+ ./tests/end_to_end_tests-a91cf9af3457d280
10032023-10-24T04:52:24.314Z
10042023-10-24T04:52:24.338Zrunning 2 tests
10052023-10-24T04:52:24.388Ztest no_spoof::no_spoof_here ... ok
10062023-10-24T04:53:24.109Ztest instance_launch::instance_launch has been running for over 60 seconds
10072023-10-24T04:55:15.796Ztest instance_launch::instance_launch ... ok
10082023-10-24T04:55:15.822Z
10092023-10-24T04:55:15.846Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 171.68s
10102023-10-24T04:55:15.871Z
10112023-10-24T04:55:15.896Z+ _exit_trap
10122023-10-24T04:55:15.920Z+ local status=0
10132023-10-24T04:55:15.944Z+ [[ 0 -eq 0 ]]
10142023-10-24T04:55:15.968Z+ exit 0
10152023-10-24T04:55:15.992Zprocess exited: duration 543287 ms, exit code 0
 
10162023-10-24T04:55:16.072Zfound 47 output files
10172023-10-24T04:55:16.123Zuploading: /var/svc/log/oxide-sled-agent:default.log (486795 bytes)
10182023-10-24T04:55:17.339Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10192023-10-24T04:55:17.364Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6e9c5aaf-7917-477d-ae73-ae16f941ce4f/root/var/svc/log/oxide-crucible-agent:default.log (24398 bytes)
10202023-10-24T04:55:18.479Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6e9c5aaf-7917-477d-ae73-ae16f941ce4f/root/var/svc/log/oxide-crucible-agent:default.log
10212023-10-24T04:55:18.503Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6e9c5aaf-7917-477d-ae73-ae16f941ce4f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10222023-10-24T04:55:19.574Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6e9c5aaf-7917-477d-ae73-ae16f941ce4f/root/var/svc/log/oxide-crucible-downstairs:default.log
10232023-10-24T04:55:19.604Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6e9c5aaf-7917-477d-ae73-ae16f941ce4f/root/var/svc/log/oxide-crucible-downstairs:downstairs-133f7b2d-311a-4b08-a8cf-e820fa89bea2.log (5537 bytes)
10242023-10-24T04:55:20.683Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6e9c5aaf-7917-477d-ae73-ae16f941ce4f/root/var/svc/log/oxide-crucible-downstairs:downstairs-133f7b2d-311a-4b08-a8cf-e820fa89bea2.log
10252023-10-24T04:55:20.708Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6e9c5aaf-7917-477d-ae73-ae16f941ce4f/root/var/svc/log/oxide-crucible-downstairs:downstairs-b469ef74-4ed0-4352-802f-6c14d66e36dd.log (3051 bytes)
10262023-10-24T04:55:21.778Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6e9c5aaf-7917-477d-ae73-ae16f941ce4f/root/var/svc/log/oxide-crucible-downstairs:downstairs-b469ef74-4ed0-4352-802f-6c14d66e36dd.log
10272023-10-24T04:55:21.812Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0a1a067d-f758-4edc-9c2a-99f2f1aae6ba/root/var/svc/log/oxide-cockroachdb:default.log (5344 bytes)
10282023-10-24T04:55:22.886Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0a1a067d-f758-4edc-9c2a-99f2f1aae6ba/root/var/svc/log/oxide-cockroachdb:default.log
10292023-10-24T04:55:22.915Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_7e1997ec-db47-4020-b04d-cf2a0ade03b9/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10302023-10-24T04:55:23.983Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_7e1997ec-db47-4020-b04d-cf2a0ade03b9/root/var/svc/log/oxide-cockroachdb:default.log
10312023-10-24T04:55:24.007Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_01666cff-7baf-4478-8b77-ada4a68267ac/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10322023-10-24T04:55:25.077Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_01666cff-7baf-4478-8b77-ada4a68267ac/root/var/svc/log/oxide-crucible-agent:default.log
10332023-10-24T04:55:25.101Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_01666cff-7baf-4478-8b77-ada4a68267ac/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10342023-10-24T04:55:26.171Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_01666cff-7baf-4478-8b77-ada4a68267ac/root/var/svc/log/oxide-crucible-downstairs:default.log
10352023-10-24T04:55:26.196Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_c2e5af6e-b796-468a-b77f-d939569e8deb/root/var/svc/log/oxide-clickhouse:default.log (656766 bytes)
10362023-10-24T04:55:27.030Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_c2e5af6e-b796-468a-b77f-d939569e8deb/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 656766 -> 715754
10372023-10-24T04:55:28.105Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_c2e5af6e-b796-468a-b77f-d939569e8deb/root/var/svc/log/oxide-clickhouse:default.log
10382023-10-24T04:55:28.131Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_e53a6c2b-53a8-4a8b-ab99-e50be7aabc0a/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10392023-10-24T04:55:29.211Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_e53a6c2b-53a8-4a8b-ab99-e50be7aabc0a/root/var/svc/log/oxide-cockroachdb:default.log
10402023-10-24T04:55:29.235Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_660558de-a472-4e28-9258-d4b6c06145ec/root/var/svc/log/oxide-crucible-pantry:default.log (2337 bytes)
10412023-10-24T04:55:30.305Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_660558de-a472-4e28-9258-d4b6c06145ec/root/var/svc/log/oxide-crucible-pantry:default.log
10422023-10-24T04:55:30.344Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_0e1e0de3-f430-437b-888f-ca350f52222e/root/var/svc/log/oxide-nexus:default.log (4435806 bytes)
10432023-10-24T04:55:31.571Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_0e1e0de3-f430-437b-888f-ca350f52222e/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4435806 -> 4437024
10442023-10-24T04:55:32.642Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_0e1e0de3-f430-437b-888f-ca350f52222e/root/var/svc/log/oxide-nexus:default.log
10452023-10-24T04:55:32.666Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_28b94fa0-8efa-4dfa-b6d5-9bcf93bf7970/root/var/svc/log/oxide-crucible-agent:default.log (14149 bytes)
10462023-10-24T04:55:33.761Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_28b94fa0-8efa-4dfa-b6d5-9bcf93bf7970/root/var/svc/log/oxide-crucible-agent:default.log
10472023-10-24T04:55:33.785Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_28b94fa0-8efa-4dfa-b6d5-9bcf93bf7970/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10482023-10-24T04:55:34.855Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_28b94fa0-8efa-4dfa-b6d5-9bcf93bf7970/root/var/svc/log/oxide-crucible-downstairs:default.log
10492023-10-24T04:55:34.880Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_28b94fa0-8efa-4dfa-b6d5-9bcf93bf7970/root/var/svc/log/oxide-crucible-downstairs:downstairs-83a94069-780c-49d7-9ae2-78199c828368.log (3046 bytes)
10502023-10-24T04:55:35.969Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_28b94fa0-8efa-4dfa-b6d5-9bcf93bf7970/root/var/svc/log/oxide-crucible-downstairs:downstairs-83a94069-780c-49d7-9ae2-78199c828368.log
10512023-10-24T04:55:35.993Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_16570845-726b-4d47-8620-f61d8b667138/root/var/svc/log/oxide-external_dns:default.log (16373 bytes)
10522023-10-24T04:55:37.114Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_16570845-726b-4d47-8620-f61d8b667138/root/var/svc/log/oxide-external_dns:default.log
10532023-10-24T04:55:37.138Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_1a33dc7b-6e14-48ff-a69f-6934bc50a038/root/var/svc/log/oxide-oximeter:default.log (182149 bytes)
10542023-10-24T04:55:37.320Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_1a33dc7b-6e14-48ff-a69f-6934bc50a038/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 182149 -> 202581
10552023-10-24T04:55:38.391Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_1a33dc7b-6e14-48ff-a69f-6934bc50a038/root/var/svc/log/oxide-oximeter:default.log
10562023-10-24T04:55:38.416Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_64f29599-8e25-448d-bab9-7d064f284a07/root/var/svc/log/oxide-crucible-agent:default.log (14150 bytes)
10572023-10-24T04:55:39.913Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_64f29599-8e25-448d-bab9-7d064f284a07/root/var/svc/log/oxide-crucible-agent:default.log
10582023-10-24T04:55:39.937Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_64f29599-8e25-448d-bab9-7d064f284a07/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10592023-10-24T04:55:41.010Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_64f29599-8e25-448d-bab9-7d064f284a07/root/var/svc/log/oxide-crucible-downstairs:default.log
10602023-10-24T04:55:41.035Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_64f29599-8e25-448d-bab9-7d064f284a07/root/var/svc/log/oxide-crucible-downstairs:downstairs-6a1a29bd-b8da-46a4-95d2-1ddbc1b3ab77.log (5536 bytes)
10612023-10-24T04:55:42.107Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_64f29599-8e25-448d-bab9-7d064f284a07/root/var/svc/log/oxide-crucible-downstairs:downstairs-6a1a29bd-b8da-46a4-95d2-1ddbc1b3ab77.log
10622023-10-24T04:55:42.133Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_de65baba-b170-4944-867b-039234889c58/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10632023-10-24T04:55:43.216Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_de65baba-b170-4944-867b-039234889c58/root/var/svc/log/oxide-crucible-pantry:default.log
10642023-10-24T04:55:43.240Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_d88f34f8-b427-431b-b75e-87e8fb6b2005/root/var/svc/log/oxide-external_dns:default.log (15614 bytes)
10652023-10-24T04:55:44.334Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_d88f34f8-b427-431b-b75e-87e8fb6b2005/root/var/svc/log/oxide-external_dns:default.log
10662023-10-24T04:55:44.374Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097/root/var/svc/log/oxide-ntp:default.log (1383 bytes)
10672023-10-24T04:55:45.447Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097/root/var/svc/log/oxide-ntp:default.log
10682023-10-24T04:55:45.471Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_c7fbcda3-b8c7-42d8-b5de-2a2d4a292393/root/var/svc/log/oxide-crucible-pantry:default.log (2332 bytes)
10692023-10-24T04:55:46.541Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_c7fbcda3-b8c7-42d8-b5de-2a2d4a292393/root/var/svc/log/oxide-crucible-pantry:default.log
10702023-10-24T04:55:46.567Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_7a17f684-8334-4827-83e9-a188bfacbe50/root/var/svc/log/oxide-internal_dns:default.log (99556 bytes)
10712023-10-24T04:55:47.749Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_7a17f684-8334-4827-83e9-a188bfacbe50/root/var/svc/log/oxide-internal_dns:default.log
10722023-10-24T04:55:47.773Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_a15208ca-8729-4588-ad9e-eb4bac4be1b6/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10732023-10-24T04:55:48.851Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_a15208ca-8729-4588-ad9e-eb4bac4be1b6/root/var/svc/log/oxide-cockroachdb:default.log
10742023-10-24T04:55:48.878Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0b6edf79-e4d2-49b5-ba76-140280126a2a/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10752023-10-24T04:55:50.359Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0b6edf79-e4d2-49b5-ba76-140280126a2a/root/var/svc/log/oxide-crucible-agent:default.log
10762023-10-24T04:55:50.384Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0b6edf79-e4d2-49b5-ba76-140280126a2a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10772023-10-24T04:55:51.457Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0b6edf79-e4d2-49b5-ba76-140280126a2a/root/var/svc/log/oxide-crucible-downstairs:default.log
10782023-10-24T04:55:51.481Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_d96a9e31-6997-47d5-851e-001ca2b93137/root/var/svc/log/oxide-internal_dns:default.log (104346 bytes)
10792023-10-24T04:55:52.666Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_d96a9e31-6997-47d5-851e-001ca2b93137/root/var/svc/log/oxide-internal_dns:default.log
10802023-10-24T04:55:52.691Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_453e7a95-683e-47a8-a0f2-ca1aaa433056/root/var/svc/log/oxide-nexus:default.log (683892 bytes)
10812023-10-24T04:55:52.929Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_453e7a95-683e-47a8-a0f2-ca1aaa433056/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 683892 -> 686324
10822023-10-24T04:55:54.006Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_453e7a95-683e-47a8-a0f2-ca1aaa433056/root/var/svc/log/oxide-nexus:default.log
10832023-10-24T04:55:54.032Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_83db1b9a-1e52-4821-a607-efab064acfd8/root/var/svc/log/oxide-nexus:default.log (490945 bytes)
10842023-10-24T04:55:54.306Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_83db1b9a-1e52-4821-a607-efab064acfd8/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 490945 -> 493375
10852023-10-24T04:55:55.379Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_83db1b9a-1e52-4821-a607-efab064acfd8/root/var/svc/log/oxide-nexus:default.log
10862023-10-24T04:55:55.404Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e9745ec5-ad9a-4b51-baf4-896bf8cd4e98/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10872023-10-24T04:55:56.475Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e9745ec5-ad9a-4b51-baf4-896bf8cd4e98/root/var/svc/log/oxide-cockroachdb:default.log
10882023-10-24T04:55:56.499Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7579e452-8e92-47aa-a335-e9d69de838ed/root/var/svc/log/oxide-crucible-agent:default.log (13684 bytes)
10892023-10-24T04:55:57.594Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7579e452-8e92-47aa-a335-e9d69de838ed/root/var/svc/log/oxide-crucible-agent:default.log
10902023-10-24T04:55:57.618Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7579e452-8e92-47aa-a335-e9d69de838ed/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10912023-10-24T04:55:58.689Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7579e452-8e92-47aa-a335-e9d69de838ed/root/var/svc/log/oxide-crucible-downstairs:default.log
10922023-10-24T04:55:58.713Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7579e452-8e92-47aa-a335-e9d69de838ed/root/var/svc/log/oxide-crucible-downstairs:downstairs-4bb32bda-bf4f-4e34-9ebf-52965e8c3862.log (3049 bytes)
10932023-10-24T04:55:59.810Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7579e452-8e92-47aa-a335-e9d69de838ed/root/var/svc/log/oxide-crucible-downstairs:downstairs-4bb32bda-bf4f-4e34-9ebf-52965e8c3862.log
10942023-10-24T04:55:59.834Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_85a4566d-95e3-4dec-a290-7357329d4968/root/var/svc/log/oxide-crucible-agent:default.log (14580 bytes)
10952023-10-24T04:56:00.929Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_85a4566d-95e3-4dec-a290-7357329d4968/root/var/svc/log/oxide-crucible-agent:default.log
10962023-10-24T04:56:00.953Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_85a4566d-95e3-4dec-a290-7357329d4968/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10972023-10-24T04:56:02.028Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_85a4566d-95e3-4dec-a290-7357329d4968/root/var/svc/log/oxide-crucible-downstairs:default.log
10982023-10-24T04:56:02.053Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_85a4566d-95e3-4dec-a290-7357329d4968/root/var/svc/log/oxide-crucible-downstairs:downstairs-f0d388f1-02f3-45ab-964b-fc9753ed946f.log (5535 bytes)
10992023-10-24T04:56:03.133Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_85a4566d-95e3-4dec-a290-7357329d4968/root/var/svc/log/oxide-crucible-downstairs:downstairs-f0d388f1-02f3-45ab-964b-fc9753ed946f.log
11002023-10-24T04:56:03.160Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bd526fc1-466f-4510-a209-6e66ff91687f/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
11012023-10-24T04:56:04.265Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bd526fc1-466f-4510-a209-6e66ff91687f/root/var/svc/log/oxide-crucible-agent:default.log
11022023-10-24T04:56:04.294Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bd526fc1-466f-4510-a209-6e66ff91687f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11032023-10-24T04:56:05.346Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bd526fc1-466f-4510-a209-6e66ff91687f/root/var/svc/log/oxide-crucible-downstairs:default.log
11042023-10-24T04:56:05.372Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_42ae1434-8c62-451d-bf60-0a319a903ecd/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
11052023-10-24T04:56:06.443Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_42ae1434-8c62-451d-bf60-0a319a903ecd/root/var/svc/log/oxide-crucible-agent:default.log
11062023-10-24T04:56:06.467Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_42ae1434-8c62-451d-bf60-0a319a903ecd/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11072023-10-24T04:56:07.538Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_42ae1434-8c62-451d-bf60-0a319a903ecd/root/var/svc/log/oxide-crucible-downstairs:default.log
11082023-10-24T04:56:07.563Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_36b2d858-62b5-4497-a73c-d77d1688f404/root/var/svc/log/oxide-internal_dns:default.log (100293 bytes)
11092023-10-24T04:56:07.699Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_36b2d858-62b5-4497-a73c-d77d1688f404/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 100293 -> 104045
11102023-10-24T04:56:08.770Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_36b2d858-62b5-4497-a73c-d77d1688f404/root/var/svc/log/oxide-internal_dns:default.log
11112023-10-24T04:56:08.795Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097/root/var/log/chrony/measurements.log (12330 bytes)
11122023-10-24T04:56:08.841Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097/root/var/log/chrony/measurements.log" changed size mid upload: 12330 -> 14385
11132023-10-24T04:56:09.913Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097/root/var/log/chrony/measurements.log
11142023-10-24T04:56:09.937Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097/root/var/log/chrony/statistics.log (7839 bytes)
11152023-10-24T04:56:09.982Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097/root/var/log/chrony/statistics.log" changed size mid upload: 7839 -> 9243
11162023-10-24T04:56:11.077Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097/root/var/log/chrony/statistics.log
11172023-10-24T04:56:11.102Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097/root/var/log/chrony/tracking.log (2118 bytes)
11182023-10-24T04:56:11.126Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097/root/var/log/chrony/tracking.log" changed size mid upload: 2118 -> 2250
11192023-10-24T04:56:12.195Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_f78c6c50-7d6e-4f6d-8927-a8405d63b097/root/var/log/chrony/tracking.log