01HCZP8884FQJKS43QCV2C7CR0: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.23",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HCZPA1TB4JX5KWAZVNM72CHH

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-17T21:39:44.662Zjob dependencies complete; ready to run (waiting for 51 m 57 s)
22023-10-17T21:40:25.839Zjob assigned to worker 01HCZS99G5QD3VQ60Q4SRCM3YH (queued for 41 s)
32023-10-17T21:40:25.899Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-17T21:40:25.946Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-10-17T21:40:25.979Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-10-17T21:40:26.016Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-17T21:40:26.047Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-10-17T21:40:26.080ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-17T21:40:26.114Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-17T21:40:26.148ZOxide Helios Version helios-2.0.22216 64-bit
112023-10-17T21:40:26.183ZConfiguring devices.
122023-10-17T21:40:26.217Z#################################################
132023-10-17T21:40:26.250Z#### ############# ##
142023-10-17T21:40:26.284Z### ### ############ ##
152023-10-17T21:40:26.318Z## ### # ## ### ## Oxide Computer Company ##
162023-10-17T21:40:26.351Z## ## ## ### # ### ##
172023-10-17T21:40:26.385Z## # ### #### #### This Station Under ##
182023-10-17T21:40:26.425Z### ### #### # ### Computer Control ##
192023-10-17T21:40:26.923Z#### #### ### ## ##
202023-10-17T21:40:26.965Z##################################################
212023-10-17T21:40:27.004ZHostname: buskin
222023-10-17T21:40:27.042ZLoading smf(7) service descriptions:
232023-10-17T21:40:28.098Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-8f412cf8dfc7d0db.gz
242023-10-17T21:40:29.820Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-8f412cf8dfc7d0db.gz
252023-10-17T21:40:30.254Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-17T21:40:46.839Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-17T21:40:46.863Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2c7e0af8d3a77456.gz
282023-10-17T21:41:06.371Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2c7e0af8d3a77456.gz
292023-10-17T21:41:06.395Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-17T21:41:07.336Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-17T21:41:07.361Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-17T21:41:09.931Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-17T21:41:09.955Zdownloading input: /input/package/work/version.txt
342023-10-17T21:41:09.990Zdownloaded input: /input/package/work/version.txt
352023-10-17T21:41:10.015Zdownloading input: /input/package/work/package.tar.gz
362023-10-17T21:41:27.108Zdownloaded input: /input/package/work/package.tar.gz
372023-10-17T21:41:27.132Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-17T21:42:03.261Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-17T21:42:03.287Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-17T21:42:14.893Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-17T21:42:14.921Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-17T21:43:09.634Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-17T21:43:09.660Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-17T21:44:14.939Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-17T21:44:14.963Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-17T21:44:53.820Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-17T21:44:53.844Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-17T21:44:58.880Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-17T21:44:59.344Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-17T21:45:11.821Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-17T21:45:11.845Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-17T21:45:26.878Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-17T21:45:27.679Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-17T21:45:31.819Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-17T21:45:31.843Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-17T21:45:31.903Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-17T21:45:31.936Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-17T21:45:34.919Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-17T21:45:34.943Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-17T21:45:38.072Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-17T21:45:38.097Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-17T21:45:50.664Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-17T21:45:50.689Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-17T21:46:04.084Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-17T21:46:04.113Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-17T21:46:04.551Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-17T21:46:04.630Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-17T21:46:07.402Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-17T21:46:07.427Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-17T21:46:19.723Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-17T21:46:19.754Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-17T21:47:19.167Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-17T21:47:19.191Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-17T21:47:53.459Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-17T21:47:53.483Zstarting task 0: "setup"
762023-10-17T21:47:53.519Z++ uname -s
772023-10-17T21:47:53.544Z+ kern=SunOS
782023-10-17T21:47:53.570Z+ case "$kern" in
792023-10-17T21:47:53.594Z+ groupadd -g 12345 build
802023-10-17T21:47:53.619Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-17T21:47:55.526Z+ zfs create -o mountpoint=/work rpool/work
822023-10-17T21:47:55.551Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-17T21:47:55.575Z+ home_fs=zfs
842023-10-17T21:47:55.599Z+ [[ zfs == autofs ]]
852023-10-17T21:47:55.623Z+ mkdir -p /home/build
862023-10-17T21:47:55.647Z+ chown build:build /home/build /work
872023-10-17T21:47:57.554Z+ chmod 0700 /home/build /work
882023-10-17T21:47:57.578Zprocess exited: duration 4048 ms, exit code 0
 
892023-10-17T21:47:57.625Zstarting task 1: "authentication"
902023-10-17T21:47:57.707Zprocess exited: duration 57 ms, exit code 0
 
912023-10-17T21:47:57.766Zstarting task 2: "build"
922023-10-17T21:47:57.799Z+ trap _exit_trap EXIT
932023-10-17T21:47:57.824Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-17T21:47:57.995Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-17T21:47:58.019Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-17T21:47:58.043Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-17T21:47:58.067Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-17T21:47:58.091Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-17T21:47:58.140Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-17T21:47:58.163Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-17T21:47:58.187Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-17T21:47:58.210Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-17T21:47:58.240Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-17T21:47:58.266Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-17T21:47:58.290Z+ [[ -d /opt/oxide ]]
1062023-10-17T21:47:58.315Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-17T21:47:58.341Z+ pfexec mkdir /opt/oxide
1082023-10-17T21:47:58.416Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-17T21:47:58.448Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-17T21:47:58.478Z+ cd /opt/oxide-underneath
1112023-10-17T21:47:58.503Z+ pfexec tar ceEp@/f - .
1122023-10-17T21:47:58.526Z+ cd /opt/oxide
1132023-10-17T21:47:58.551Z+ pfexec tar xveEp@/f -
1142023-10-17T21:47:58.593Zx ., 0 bytes, 0 tape blocks
1152023-10-17T21:47:58.624Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-17T21:47:58.649Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-17T21:47:58.674Zx ./opte/bin/opteadm, 50138640 bytes, 97928 tape blocks
1182023-10-17T21:47:58.704Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-17T21:47:58.729Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-17T21:47:58.753Z+ pfexec mkdir /opt/oxide/work
1212023-10-17T21:47:58.787Z+ pfexec chown build:build /opt/oxide/work
1222023-10-17T21:47:58.811Z+ cd /opt/oxide/work
1232023-10-17T21:47:58.834Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-17T21:47:59.763ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-17T21:47:59.799Zx out/maghemite.tar, 74957312 bytes, 146401 tape blocks
1262023-10-17T21:47:59.827Zx out/omicron-sled-agent.tar, 114037248 bytes, 222729 tape blocks
1272023-10-17T21:47:59.851Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-17T21:47:59.876Zx out/npuzone/npuzone, 37379944 bytes, 73008 tape blocks
1292023-10-17T21:47:59.900Zx package-manifest.toml, 21303 bytes, 42 tape blocks
1302023-10-17T21:47:59.925Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-17T21:47:59.949Zx target/release/omicron-package, 38061408 bytes, 74339 tape blocks
1322023-10-17T21:47:59.974Zx tools/create_virtual_hardware.sh, 2818 bytes, 6 tape blocks
1332023-10-17T21:47:59.999Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-17T21:48:00.028Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-17T21:48:00.052Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-17T21:48:00.076Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-17T21:48:00.102Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-17T21:48:00.174Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-17T21:48:00.199Z
1402023-10-17T21:48:00.638Zreal 1.604101435
1412023-10-17T21:48:00.676Zuser 1.445443495
1422023-10-17T21:48:00.723Zsys 0.274959096
1432023-10-17T21:48:00.760Ztrap 0.000067660
1442023-10-17T21:48:00.784Ztflt 0.000000000
1452023-10-17T21:48:00.808Zdflt 0.000000000
1462023-10-17T21:48:00.833Zkflt 0.000000000
1472023-10-17T21:48:01.267Zlock 0.000000000
1482023-10-17T21:48:01.299Zslp 1.449568106
1492023-10-17T21:48:01.800Zlat 0.036037543
1502023-10-17T21:48:01.827Zstop 0.000127502
1512023-10-17T21:48:01.856Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-17T21:48:01.881Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-17T21:48:01.905Z+ mkdir tests
1542023-10-17T21:48:01.931Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-17T21:48:01.962Z+ ptime -m gunzip
1562023-10-17T21:48:01.987Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8f412cf8dfc7d0db
1572023-10-17T21:48:02.011Z
1582023-10-17T21:48:02.036Zreal 0.139607457
1592023-10-17T21:48:02.061Zuser 0.127266377
1602023-10-17T21:48:02.091Zsys 0.012104156
1612023-10-17T21:48:02.116Ztrap 0.000009579
1622023-10-17T21:48:02.141Ztflt 0.000000000
1632023-10-17T21:48:02.167Zdflt 0.000000000
1642023-10-17T21:48:02.191Zkflt 0.000000000
1652023-10-17T21:48:02.216Zlock 0.000000000
1662023-10-17T21:48:02.241Zslp 0.000018740
1672023-10-17T21:48:02.265Zlat 0.000020170
1682023-10-17T21:48:02.291Zstop 0.000118332
1692023-10-17T21:48:02.319Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8f412cf8dfc7d0db
1702023-10-17T21:48:02.344Z+ chmod a+x tests/bootstrap-8f412cf8dfc7d0db
1712023-10-17T21:48:02.369Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-17T21:48:02.394Z+ ptime -m gunzip
1732023-10-17T21:48:02.419Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-17T21:48:03.208Z
1752023-10-17T21:48:03.235Zreal 2.261765677
1762023-10-17T21:48:03.260Zuser 2.115763363
1772023-10-17T21:48:03.285Zsys 0.145567878
1782023-10-17T21:48:03.310Ztrap 0.000183513
1792023-10-17T21:48:03.335Ztflt 0.000000000
1802023-10-17T21:48:03.360Zdflt 0.000000000
1812023-10-17T21:48:03.386Zkflt 0.000000000
1822023-10-17T21:48:03.419Zlock 0.000000000
1832023-10-17T21:48:03.448Zslp 0.000022520
1842023-10-17T21:48:03.481Zlat 0.000048809
1852023-10-17T21:48:03.521Zstop 0.000121352
1862023-10-17T21:48:03.558Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-17T21:48:03.586Z+ chmod a+x tests/bootstrap
1882023-10-17T21:48:03.613Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-17T21:48:03.655Z+ ptime -m gunzip
1902023-10-17T21:48:03.685Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2c7e0af8d3a77456
1912023-10-17T21:48:05.737Z
1922023-10-17T21:48:05.770Zreal 2.515281254
1932023-10-17T21:48:05.800Zuser 2.357543962
1942023-10-17T21:48:05.829Zsys 0.157360022
1952023-10-17T21:48:05.854Ztrap 0.000159798
1962023-10-17T21:48:05.921Ztflt 0.000000000
1972023-10-17T21:48:05.946Zdflt 0.000000000
1982023-10-17T21:48:05.970Zkflt 0.000000000
1992023-10-17T21:48:05.993Zlock 0.000000000
2002023-10-17T21:48:06.020Zslp 0.000019440
2012023-10-17T21:48:06.050Zlat 0.000017830
2022023-10-17T21:48:06.098Zstop 0.000113721
2032023-10-17T21:48:06.138Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2c7e0af8d3a77456
2042023-10-17T21:48:06.166Z+ chmod a+x tests/end_to_end_tests-2c7e0af8d3a77456
2052023-10-17T21:48:06.195Z+ bmat address ls
2062023-10-17T21:48:06.221ZNAME CIDR FIRST LAST COUNT
2072023-10-17T21:48:06.250Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-10-17T21:48:06.278Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-17T21:48:06.304Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-10-17T21:48:06.333Z+ EXTRA_IP_START=10.151.1.164
2112023-10-17T21:48:06.360Z+ EXTRA_IP_END=10.151.1.227
2122023-10-17T21:48:06.389Z+ EXTRA_IP_COUNT=64
2132023-10-17T21:48:06.419Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-10-17T21:48:06.447Z+ EXTRA_IP_BASE=10.151.1
2152023-10-17T21:48:06.477Z+ EXTRA_IP_FOCTET=164
2162023-10-17T21:48:06.505Z+ EXTRA_IP_LOCTET=227
2172023-10-17T21:48:06.535Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-10-17T21:48:06.564Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-10-17T21:48:07.001Z+ DNS_IP1=10.151.1.164
2202023-10-17T21:48:07.037Z+ DNS_IP2=10.151.1.165
2212023-10-17T21:48:07.073Z+ UPLINK_IP=10.151.1.174
2222023-10-17T21:48:07.098Z+ IPPOOL_START=10.151.1.175
2232023-10-17T21:48:07.122Z+ IPPOOL_END=10.151.1.227
2242023-10-17T21:48:07.146Z++ ipadm show-addr -po type,addr
2252023-10-17T21:48:07.586Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-10-17T21:48:07.612Z+ GATEWAY_IP=10.151.1.100
2272023-10-17T21:48:07.640Z+ [[ -n 10.151.1.100 ]]
2282023-10-17T21:48:07.669Z+ ping -s 10.151.1.100 56 1
2292023-10-17T21:48:07.696ZPING 10.151.1.100: 56 data bytes
2302023-10-17T21:48:07.720Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.012 ms
2312023-10-17T21:48:07.749Z
2322023-10-17T21:48:07.776Z----10.151.1.100 PING Statistics----
2332023-10-17T21:48:07.802Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-10-17T21:48:07.839Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2352023-10-17T21:48:07.875Z++ arp -an
2362023-10-17T21:48:07.914Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-10-17T21:48:07.952Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-10-17T21:48:08.054Z+ routeadm -e ipv4-forwarding -u
2392023-10-17T21:48:08.095Z+ PXA_START=10.151.1.164
2402023-10-17T21:48:08.131Z+ PXA_END=10.151.1.227
2412023-10-17T21:48:08.591Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-10-17T21:48:08.620Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-10-17T21:48:09.174Z+ ZPOOL_VDEV_DIR=/scratch
2442023-10-17T21:48:09.198Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-10-17T21:48:09.222Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-10-17T21:48:09.246Z++ cd ./tools
2472023-10-17T21:48:09.273Z++ pwd
2482023-10-17T21:48:09.299Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-10-17T21:48:09.330Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-10-17T21:48:09.361Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-10-17T21:48:09.389Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-10-17T21:48:09.421Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-10-17T21:48:09.458Z++ dladm show-phys -p -o LINK
2542023-10-17T21:48:09.491Z++ head -1
2552023-10-17T21:48:09.517Z+ PHYSICAL_LINK=igb0
2562023-10-17T21:48:09.541Z+ echo 'Using igb0 as physical link'
2572023-10-17T21:48:09.566ZUsing igb0 as physical link
2582023-10-17T21:48:09.590Z+ ensure_run_as_root
2592023-10-17T21:48:09.614Z++ id -u
2602023-10-17T21:48:09.638Z+ [[ 0 -ne 0 ]]
2612023-10-17T21:48:09.662Z+ ensure_zpools
2622023-10-17T21:48:09.687Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-10-17T21:48:09.712Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-10-17T21:48:09.735Z+ readarray -t ZPOOLS
2652023-10-17T21:48:09.759Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-10-17T21:48:09.783Z++ sed 's/[ ",]//g'
2672023-10-17T21:48:09.811Z+ for ZPOOL in "${ZPOOLS[@]}"
2682023-10-17T21:48:09.840Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2692023-10-17T21:48:09.866ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2702023-10-17T21:48:09.891Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2712023-10-17T21:48:09.914ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2722023-10-17T21:48:09.938Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-10-17T21:48:09.962ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2742023-10-17T21:48:09.988Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2752023-10-17T21:48:10.017ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2762023-10-17T21:48:10.046Z0+0 records in
2772023-10-17T21:48:10.086ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2782023-10-17T21:48:10.110Z0+0 records out
2792023-10-17T21:48:10.134ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2802023-10-17T21:48:10.158Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2812023-10-17T21:48:10.183ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2822023-10-17T21:48:10.207Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2832023-10-17T21:48:10.231ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2842023-10-17T21:48:10.255Z+ set +x
2852023-10-17T21:48:10.279ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2862023-10-17T21:48:10.307Z++ zpool list -o name
2872023-10-17T21:48:10.331ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2882023-10-17T21:48:10.358Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2892023-10-17T21:48:10.385ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2902023-10-17T21:48:10.420Z+ [[ -z '' ]]
2912023-10-17T21:48:10.450ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2922023-10-17T21:48:10.477Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2932023-10-17T21:48:10.511ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2942023-10-17T21:48:10.539Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2952023-10-17T21:48:10.569ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2962023-10-17T21:48:10.593Z+ set +x
2972023-10-17T21:48:10.617ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2982023-10-17T21:48:10.641Z+ for ZPOOL in "${ZPOOLS[@]}"
2992023-10-17T21:48:10.664ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3002023-10-17T21:48:10.689Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3012023-10-17T21:48:10.714ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3022023-10-17T21:48:10.739Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-10-17T21:48:10.764ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3042023-10-17T21:48:10.790Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3052023-10-17T21:48:10.818ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3062023-10-17T21:48:10.846Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3072023-10-17T21:48:10.887ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3082023-10-17T21:48:10.917Z0+0 records in
3092023-10-17T21:48:10.947ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3102023-10-17T21:48:10.978Z0+0 records out
3112023-10-17T21:48:11.010ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3122023-10-17T21:48:11.040Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3132023-10-17T21:48:11.067ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3142023-10-17T21:48:11.114Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3152023-10-17T21:48:11.149ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3162023-10-17T21:48:11.177Z+ set +x
3172023-10-17T21:48:11.201ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3182023-10-17T21:48:11.225Z++ zpool list -o name
3192023-10-17T21:48:11.248ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3202023-10-17T21:48:11.274Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3212023-10-17T21:48:11.298ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3222023-10-17T21:48:11.321Z+ [[ -z '' ]]
3232023-10-17T21:48:11.345ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3242023-10-17T21:48:11.369Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3252023-10-17T21:48:11.395ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3262023-10-17T21:48:11.426Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3272023-10-17T21:48:11.472ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3282023-10-17T21:48:11.506Z+ set +x
3292023-10-17T21:48:11.545ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3302023-10-17T21:48:11.574Z+ for ZPOOL in "${ZPOOLS[@]}"
3312023-10-17T21:48:11.605ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3322023-10-17T21:48:11.636Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3332023-10-17T21:48:11.663ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3342023-10-17T21:48:11.687Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3352023-10-17T21:48:11.712ZSimnet net0/sc0_0 exists
3362023-10-17T21:48:11.736Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3372023-10-17T21:48:11.760ZSimnet net1/sc1_0 exists
3382023-10-17T21:48:11.784Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3392023-10-17T21:48:11.807ZVnic sc0_1 exists
3402023-10-17T21:48:11.831Z0+0 records in
3412023-10-17T21:48:11.855Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3422023-10-17T21:48:11.879Z0+0 records out
3432023-10-17T21:48:11.902Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3442023-10-17T21:48:11.926Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3452023-10-17T21:48:11.955Z+ set +x
3462023-10-17T21:48:11.983Z++ zpool list -o name
3472023-10-17T21:48:12.006Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3482023-10-17T21:48:12.030Z+ [[ -z '' ]]
3492023-10-17T21:48:12.054Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-10-17T21:48:12.082Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-10-17T21:48:12.540Z+ set +x
3522023-10-17T21:48:12.571Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-10-17T21:48:12.599Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-10-17T21:48:12.628Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-10-17T21:48:12.656Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-10-17T21:48:12.684Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-10-17T21:48:12.710Z0+0 records in
3582023-10-17T21:48:12.737Z0+0 records out
3592023-10-17T21:48:12.762Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3602023-10-17T21:48:12.788Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-10-17T21:48:12.821Z+ set +x
3622023-10-17T21:48:12.855Z++ zpool list -o name
3632023-10-17T21:48:12.895Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-10-17T21:48:12.936Z+ [[ -z '' ]]
3652023-10-17T21:48:12.969Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-10-17T21:48:12.996Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-10-17T21:48:13.025Z+ set +x
3682023-10-17T21:48:13.050Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-10-17T21:48:13.076Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-10-17T21:48:13.101Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-10-17T21:48:13.128Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-10-17T21:48:13.196Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-10-17T21:48:13.223Z0+0 records in
3742023-10-17T21:48:13.336Z0+0 records out
3752023-10-17T21:48:13.376Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3762023-10-17T21:48:13.407Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-10-17T21:48:13.443Z+ set +x
3782023-10-17T21:48:13.480Z++ zpool list -o name
3792023-10-17T21:48:13.516Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-10-17T21:48:13.556Z+ [[ -z '' ]]
3812023-10-17T21:48:13.589Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-10-17T21:48:13.614Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-10-17T21:48:13.637Z+ set +x
3842023-10-17T21:48:13.662Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-10-17T21:48:13.687Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862023-10-17T21:48:13.711Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872023-10-17T21:48:13.734Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882023-10-17T21:48:13.758Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892023-10-17T21:48:13.782Z0+0 records in
3902023-10-17T21:48:13.809Z0+0 records out
3912023-10-17T21:48:13.841Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-10-17T21:48:13.874Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932023-10-17T21:48:13.904Z+ set +x
3942023-10-17T21:48:13.931Z++ zpool list -o name
3952023-10-17T21:48:13.959Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3962023-10-17T21:48:13.992Z+ [[ -z '' ]]
3972023-10-17T21:48:14.017Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3982023-10-17T21:48:14.042Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3992023-10-17T21:48:14.067Z+ set +x
4002023-10-17T21:48:14.094Z+ for ZPOOL in "${ZPOOLS[@]}"
4012023-10-17T21:48:14.120Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4022023-10-17T21:48:14.146Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4032023-10-17T21:48:14.175Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4042023-10-17T21:48:14.235Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4052023-10-17T21:48:14.267Z0+0 records in
4062023-10-17T21:48:14.296Z0+0 records out
4072023-10-17T21:48:14.327Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4082023-10-17T21:48:14.360Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4092023-10-17T21:48:14.392Z+ set +x
4102023-10-17T21:48:14.422Z++ zpool list -o name
4112023-10-17T21:48:14.447Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4122023-10-17T21:48:14.472Z+ [[ -z '' ]]
4132023-10-17T21:48:14.498Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4142023-10-17T21:48:14.523Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4152023-10-17T21:48:14.553Z+ set +x
4162023-10-17T21:48:14.581Zdownloading softnpu ...done
4172023-10-17T21:48:14.608Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-10-17T21:48:14.636Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/4ba9c18928edece80f83153b9d78b76cf093da19/libsidecar_lite.so.sha256.txt
4192023-10-17T21:48:14.663Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4202023-10-17T21:48:14.693Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4212023-10-17T21:48:14.733Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4222023-10-17T21:48:14.757Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4232023-10-17T21:48:15.305Z0+0 records in
4242023-10-17T21:48:15.343Z0+0 records out
4252023-10-17T21:48:15.380Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4262023-10-17T21:48:15.414Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4272023-10-17T21:48:15.452Z+ set +x
4282023-10-17T21:48:15.480Z++ zpool list -o name
4292023-10-17T21:48:15.505Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4302023-10-17T21:48:15.531Z+ [[ -z '' ]]
4312023-10-17T21:48:15.555Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4322023-10-17T21:48:15.579Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4332023-10-17T21:48:15.603Z+ set +x
4342023-10-17T21:48:15.626Z+ for ZPOOL in "${ZPOOLS[@]}"
4352023-10-17T21:48:15.651Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4362023-10-17T21:48:15.675Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4372023-10-17T21:48:15.699Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4382023-10-17T21:48:15.723Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4392023-10-17T21:48:15.750Z0+0 records in
4402023-10-17T21:48:15.774Z0+0 records out
4412023-10-17T21:48:15.798Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4422023-10-17T21:48:15.821Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4432023-10-17T21:48:15.845Z+ set +x
4442023-10-17T21:48:15.875Z++ zpool list -o name
4452023-10-17T21:48:16.349Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4462023-10-17T21:48:16.373Z+ [[ -z '' ]]
4472023-10-17T21:48:16.397Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4482023-10-17T21:48:16.420Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4492023-10-17T21:48:16.444Z+ set +x
4502023-10-17T21:48:16.467Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4512023-10-17T21:48:16.491Z+ readarray -t ZPOOLS
4522023-10-17T21:48:16.515Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4532023-10-17T21:48:16.539Z++ sed 's/[ ",]//g'
4542023-10-17T21:48:16.563Z+ for ZPOOL in "${ZPOOLS[@]}"
4552023-10-17T21:48:16.589Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4562023-10-17T21:48:16.620Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4572023-10-17T21:48:16.648Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4582023-10-17T21:48:16.672Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4592023-10-17T21:48:16.696Z0+0 records in
4602023-10-17T21:48:16.737Z0+0 records out
4612023-10-17T21:48:16.780Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4622023-10-17T21:48:16.808Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4632023-10-17T21:48:17.235Z+ set +x
4642023-10-17T21:48:17.259Z++ zpool list -o name
4652023-10-17T21:48:17.282Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4662023-10-17T21:48:17.306Z+ [[ -z '' ]]
4672023-10-17T21:48:17.330Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4682023-10-17T21:48:17.374Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4692023-10-17T21:48:17.398Z+ set +x
4702023-10-17T21:48:17.835Z+ for ZPOOL in "${ZPOOLS[@]}"
4712023-10-17T21:48:17.859Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4722023-10-17T21:48:17.890Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4732023-10-17T21:48:17.941Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4742023-10-17T21:48:17.969Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4752023-10-17T21:48:18.000Z0+0 records in
4762023-10-17T21:48:18.025Z0+0 records out
4772023-10-17T21:48:18.049Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4782023-10-17T21:48:18.072Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4792023-10-17T21:48:18.096Z+ set +x
4802023-10-17T21:48:18.167Z++ zpool list -o name
4812023-10-17T21:48:18.190Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4822023-10-17T21:48:18.214Z+ [[ -z '' ]]
4832023-10-17T21:48:18.239Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4842023-10-17T21:48:18.262Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4852023-10-17T21:48:18.288Z+ set +x
4862023-10-17T21:48:18.312Z+ ensure_simulated_links igb0
4872023-10-17T21:48:18.336Z+ local PHYSICAL_LINK=igb0
4882023-10-17T21:48:18.359Z+ INDICES=("0" "1")
4892023-10-17T21:48:18.405Z+ for I in "${INDICES[@]}"
4902023-10-17T21:48:18.430Z++ get_simnet_name_if_exists net0
4912023-10-17T21:48:18.454Z++ dladm show-simnet -p -o LINK net0
4922023-10-17T21:48:18.478Z++ echo ''
4932023-10-17T21:48:18.504Z+ [[ -z '' ]]
4942023-10-17T21:48:18.532Z+ dladm create-simnet -t net0
4952023-10-17T21:48:18.560Z+ dladm create-simnet -t sc0_0
4962023-10-17T21:48:18.585Z+ dladm modify-simnet -t -p net0 sc0_0
4972023-10-17T21:48:18.613Z+ dladm set-linkprop -p mtu=9000 sc0_0
4982023-10-17T21:48:18.638Z+ success 'Simnet net0/sc0_0 exists'
4992023-10-17T21:48:18.662Z+ set +x
5002023-10-17T21:48:18.685Z+ for I in "${INDICES[@]}"
5012023-10-17T21:48:18.713Z++ get_simnet_name_if_exists net1
5022023-10-17T21:48:18.739Z++ dladm show-simnet -p -o LINK net1
5032023-10-17T21:48:18.767Z++ echo ''
5042023-10-17T21:48:18.797Z+ [[ -z '' ]]
5052023-10-17T21:48:18.820Z+ dladm create-simnet -t net1
5062023-10-17T21:48:18.844Z+ dladm create-simnet -t sc1_0
5072023-10-17T21:48:18.867Z+ dladm modify-simnet -t -p net1 sc1_0
5082023-10-17T21:48:18.891Z+ dladm set-linkprop -p mtu=9000 sc1_0
5092023-10-17T21:48:18.915Z+ success 'Simnet net1/sc1_0 exists'
5102023-10-17T21:48:18.938Z+ set +x
5112023-10-17T21:48:19.374Z++ get_vnic_name_if_exists sc0_1
5122023-10-17T21:48:19.459Z++ dladm show-vnic -p -o LINK sc0_1
5132023-10-17T21:48:19.484Z++ echo ''
5142023-10-17T21:48:19.508Z+ [[ -z '' ]]
5152023-10-17T21:48:19.531Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162023-10-17T21:48:19.556Z+ success 'Vnic sc0_1 exists'
5172023-10-17T21:48:19.579Z+ set +x
5182023-10-17T21:48:19.603Z+ warn_if_no_proxy_arp
5192023-10-17T21:48:19.627Z+ [[ -v PXA_START ]]
5202023-10-17T21:48:19.651Z+ [[ -v PXA_END ]]
5212023-10-17T21:48:19.675Z+ ensure_softnpu_zone
5222023-10-17T21:48:19.700Z+ zoneadm list
5232023-10-17T21:48:19.724Z+ grep -q sidecar_softnpu
5242023-10-17T21:48:19.749Z+ [[ -f out/npuzone/npuzone ]]
5252023-10-17T21:48:19.775Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0
5262023-10-17T21:48:19.800Zdownloading asic_program.so ...done
5272023-10-17T21:48:19.841Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/4ba9c18928edece80f83153b9d78b76cf093da19/scadm.sha256.txt
5282023-10-17T21:48:24.509Zdownloading scadm ...done
5292023-10-17T21:48:24.533Zcreating port tfportrear0_0 for sc0_0
5302023-10-17T21:48:24.557Zcreating port tfportqsfp0_0 for sc0_1
5312023-10-17T21:48:24.585Zsoftnpu zone setup
5322023-10-17T21:48:24.609Zconfigure zone
5332023-10-17T21:48:24.633Zinstall zone
5342023-10-17T21:48:25.547Zboot zone
5352023-10-17T21:48:28.652Z[sidecar_softnpu] svcs milestone/network
5362023-10-17T21:48:28.714ZSTATE STIME FMRI
5372023-10-17T21:48:28.743Zonline 21:48:26 svc:/milestone/network:default
5382023-10-17T21:48:28.770Z[sidecar_softnpu] /softnpu/npu start
5392023-10-17T21:48:28.797Z
5402023-10-17T21:48:28.825Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5412023-10-17T21:48:28.849Z+ GATEWAY_IP=10.151.1.100
5422023-10-17T21:48:28.872Z+ echo 'Using 10.151.1.100 as gateway ip'
5432023-10-17T21:48:28.896ZUsing 10.151.1.100 as gateway ip
5442023-10-17T21:48:28.920Z+ [[ ! -v GATEWAY_MAC ]]
5452023-10-17T21:48:28.943Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5462023-10-17T21:48:28.967Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5472023-10-17T21:48:28.994Z+++ echo 18:c0:4d:0d:9f:b2
5482023-10-17T21:48:29.027Z+ COUNT=1
5492023-10-17T21:48:29.056Z+ [[ 1 -ne 1 ]]
5502023-10-17T21:48:29.532Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5512023-10-17T21:48:29.560ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5522023-10-17T21:48:29.584Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5532023-10-17T21:48:29.607Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5542023-10-17T21:48:29.631Z+ PXA_MAC=a8:e1:de:01:70:1d
5552023-10-17T21:48:29.656Z+ [[ -v PXA_START ]]
5562023-10-17T21:48:29.680Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5572023-10-17T21:48:29.704Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5582023-10-17T21:48:29.738Z+ z_scadm dump-state
5592023-10-17T21:48:29.763Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5602023-10-17T21:48:29.787Zlocal v6:
5612023-10-17T21:48:29.811Zlocal v4:
5622023-10-17T21:48:29.835Zrouter v6:
5632023-10-17T21:48:29.858Zrouter v4:
5642023-10-17T21:48:29.882Zresolver v4:
5652023-10-17T21:48:29.906Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5662023-10-17T21:48:29.932Zresolver v6:
5672023-10-17T21:48:30.374Znat v4:
5682023-10-17T21:48:30.406Znat_v6:
5692023-10-17T21:48:30.435Zport_mac:
5702023-10-17T21:48:30.464Zproxy arp:
5712023-10-17T21:48:30.493Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5722023-10-17T21:48:30.523Z+ success 'softnpu zone exists'
5732023-10-17T21:48:30.547Z+ set +x
5742023-10-17T21:48:30.575Zsoftnpu zone exists
5752023-10-17T21:48:30.598Z
5762023-10-17T21:48:30.622Zreal 22.727266240
5772023-10-17T21:48:30.646Zuser 0.696467938
5782023-10-17T21:48:30.669Zsys 1.560876504
5792023-10-17T21:48:30.693Ztrap 0.000083127
5802023-10-17T21:48:30.716Ztflt 0.000496520
5812023-10-17T21:48:30.762Zdflt 0.001404192
5822023-10-17T21:48:30.785Zkflt 0.000000000
5832023-10-17T21:48:30.809Zlock 17:43.314861614
5842023-10-17T21:48:30.836Zslp 1:07.732545977
5852023-10-17T21:48:30.861Zlat 0.052638744
5862023-10-17T21:48:30.889Zstop 0.337895431
5872023-10-17T21:48:31.327Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5882023-10-17T21:48:31.353Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5892023-10-17T21:48:31.384Z+ SILO_NAME=recovery
5902023-10-17T21:48:31.413Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5912023-10-17T21:48:31.844Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5922023-10-17T21:48:31.874Z+ sed -i~ '
5932023-10-17T21:48:31.908Z /^external_dns_ips/c\
5942023-10-17T21:48:31.933Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5952023-10-17T21:48:31.956Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5962023-10-17T21:48:31.980Z /^first/c\
5972023-10-17T21:48:32.003Zfirst = "10.151.1.164"
5982023-10-17T21:48:32.027Z /^last/c\
5992023-10-17T21:48:32.051Zlast = "10.151.1.173"
6002023-10-17T21:48:32.075Z }
6012023-10-17T21:48:32.099Z /^\[rack_network_config/,/^$/ {
6022023-10-17T21:48:32.122Z /^infra_ip_first/c\
6032023-10-17T21:48:32.151Zinfra_ip_first = "10.151.1.174"
6042023-10-17T21:48:32.176Z /^infra_ip_last/c\
6052023-10-17T21:48:32.199Zinfra_ip_last = "10.151.1.174"
6062023-10-17T21:48:32.223Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6072023-10-17T21:48:32.664Z }
6082023-10-17T21:48:32.701Z+++ pkg/config-rss.toml Tue Oct 17 21:48:28 2023
6092023-10-17T21:48:32.729Z /^\[\[rack_network_config.uplinks/,/^$/ {
6102023-10-17T21:48:32.754Z@@ -41,7 +41,7 @@
6112023-10-17T21:48:32.777Z /^gateway_ip/c\
6122023-10-17T21:48:32.809Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6132023-10-17T21:48:32.833Zgateway_ip = "10.151.1.100"
6142023-10-17T21:48:32.857Z # must be contained in one of the "internal services" IP Pool ranges listed
6152023-10-17T21:48:32.880Z /^uplink_cidr/c\
6162023-10-17T21:48:32.904Z # below.
6172023-10-17T21:48:32.927Zuplink_cidr = "10.151.1.174/32"
6182023-10-17T21:48:32.951Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6192023-10-17T21:48:32.975Z }
6202023-10-17T21:48:32.998Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6212023-10-17T21:48:33.022Z' pkg/config-rss.toml
6222023-10-17T21:48:33.046Z
6232023-10-17T21:48:33.075Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6242023-10-17T21:48:33.102Z # Initial TLS certificates for the external API
6252023-10-17T21:48:33.188Z+ true
6262023-10-17T21:48:33.212Z #
6272023-10-17T21:48:33.236Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6282023-10-17T21:48:33.259Z@@ -77,8 +77,8 @@
6292023-10-17T21:48:33.283Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6302023-10-17T21:48:33.306Z #
6312023-10-17T21:48:33.330Z+ TLS_NAME=recovery.sys.oxide.test
6322023-10-17T21:48:33.354Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-10-17T21:48:33.378Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6342023-10-17T21:48:33.403Z [[internal_services_ip_pool_ranges]]
6352023-10-17T21:48:33.426Z..+.........+.+......+...+...+............+..+............................+..+...+......+.......+..+...+.......+...+..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+......+....+..+....+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6362023-10-17T21:48:33.450Z-first = "192.168.1.20"
6372023-10-17T21:48:33.474Z.+..+.......+......+..............+.+.................+...+......+.+..+.....................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+........+...+.+......+..+......+.......+.....+...+.+......+..+..........+..+.+.....+...............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6382023-10-17T21:48:33.906Z-last = "192.168.1.29"
6392023-10-17T21:48:33.939Z-----
6402023-10-17T21:48:33.965Z+first = "10.151.1.164"
6412023-10-17T21:48:33.991Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6422023-10-17T21:48:34.029Z+last = "10.151.1.173"
6432023-10-17T21:48:34.053Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6442023-10-17T21:48:34.077Z
6452023-10-17T21:48:34.101Z+ rmdir pkg
6462023-10-17T21:48:34.125Z # TODO - this configuration is subject to change going forward. Ultimately these
6472023-10-17T21:48:34.148Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6482023-10-17T21:48:34.172Z # parameters should be provided to the control plane via wicket, but we need to
6492023-10-17T21:48:34.196Z+ pfexec mkdir -p /usr/oxide
6502023-10-17T21:48:34.220Z@@ -91,13 +91,13 @@
6512023-10-17T21:48:34.243Z+ pfexec rm -f /usr/oxide/tcpproxy
6522023-10-17T21:48:34.267Z # A range of IP addresses used by Boundary Services on the external network. In
6532023-10-17T21:48:34.291Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6542023-10-17T21:48:34.327Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6552023-10-17T21:48:34.355Z+ pfexec chmod +x /usr/oxide/tcpproxy
6562023-10-17T21:48:34.379Z # With softnpu, only one address is used.
6572023-10-17T21:48:34.403Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6582023-10-17T21:48:34.427Z-infra_ip_first = "192.168.1.30"
6592023-10-17T21:48:34.451Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6602023-10-17T21:48:34.479Z-infra_ip_last = "192.168.1.30"
6612023-10-17T21:48:34.503Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6622023-10-17T21:48:34.527Z+infra_ip_first = "10.151.1.174"
6632023-10-17T21:48:34.551Z+ OMICRON_NO_UNINSTALL=1
6642023-10-17T21:48:34.575Z+infra_ip_last = "10.151.1.174"
6652023-10-17T21:48:34.599Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6662023-10-17T21:48:34.623Z
6672023-10-17T21:48:34.646ZOct 17 21:48:28.931 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6682023-10-17T21:48:34.670Z # You can configure multiple uplinks by repeating the following stanza
6692023-10-17T21:48:34.694ZOct 17 21:48:28.933 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6702023-10-17T21:48:34.717Z [[rack_network_config.uplinks]]
6712023-10-17T21:48:34.745ZOct 17 21:48:28.933 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6722023-10-17T21:48:34.780Z # The gateway IP for the rack's external network
6732023-10-17T21:48:34.805ZOct 17 21:48:28.933 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6742023-10-17T21:48:34.829Z-gateway_ip = "192.168.1.199"
6752023-10-17T21:48:34.870ZOct 17 21:48:28.933 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6762023-10-17T21:48:34.894Z+gateway_ip = "10.151.1.100"
6772023-10-17T21:48:34.918ZOct 17 21:48:28.933 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6782023-10-17T21:48:34.942Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6792023-10-17T21:48:34.965ZOct 17 21:48:28.933 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6802023-10-17T21:48:34.989Z uplink_port = "qsfp0"
6812023-10-17T21:48:35.013ZOct 17 21:48:28.933 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/maghemite.tar
6822023-10-17T21:48:35.036Z uplink_port_speed = "40G"
6832023-10-17T21:48:35.060ZOct 17 21:48:28.933 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6842023-10-17T21:48:35.084Z@@ -104,7 +104,7 @@
6852023-10-17T21:48:35.108ZOct 17 21:48:28.933 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6862023-10-17T21:48:35.131Z uplink_port_fec="none"
6872023-10-17T21:48:35.155ZOct 17 21:48:28.934 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6882023-10-17T21:48:35.185Z # For softnpu, an address within the "infra" block above that will be used for
6892023-10-17T21:48:35.220ZOct 17 21:48:28.934 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6902023-10-17T21:48:35.245Z # the softnpu uplink port. You can just pick the first address in that pool.
6912023-10-17T21:48:35.269ZOct 17 21:48:28.934 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6922023-10-17T21:48:35.293Z-uplink_cidr = "192.168.1.30/32"
6932023-10-17T21:48:35.316ZOct 17 21:48:28.934 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6942023-10-17T21:48:35.340Z+uplink_cidr = "10.151.1.174/32"
6952023-10-17T21:48:35.363ZOct 17 21:48:28.934 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6962023-10-17T21:48:35.387Z # Switch to use for the uplink. For single-rack deployments this can be
6972023-10-17T21:48:35.411ZOct 17 21:48:28.934 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6982023-10-17T21:48:35.435Z # "switch0" (upper slot) or "switch1" (lower slot). For single-node softnpu
6992023-10-17T21:48:35.458ZOct 17 21:48:28.934 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7002023-10-17T21:48:35.482Z # and dendrite stub environments, use "switch0"
7012023-10-17T21:48:35.506ZOct 17 21:48:29.203 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7022023-10-17T21:48:35.530Za pkg/config-rss.toml 7K
7032023-10-17T21:48:35.555ZOct 17 21:48:29.246 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7042023-10-17T21:48:35.578Za pkg/initial-tls-cert.pem 2K
7052023-10-17T21:48:35.602ZOct 17 21:48:29.308 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7062023-10-17T21:48:35.633Za pkg/initial-tls-key.pem 4K
7072023-10-17T21:48:35.659Z
7082023-10-17T21:48:35.685Zreal 0.514576837
7092023-10-17T21:48:35.712Zuser 0.095087338
7102023-10-17T21:48:35.740Zsys 2.145346274
7112023-10-17T21:48:35.765Ztrap 0.000000659
7122023-10-17T21:48:35.789Ztflt 0.000024198
7132023-10-17T21:48:35.813Zdflt 0.000419936
7142023-10-17T21:48:35.837Zkflt 0.000000000
7152023-10-17T21:48:35.861Zlock 46.242321795
7162023-10-17T21:48:35.911Zslp 0.723649476
7172023-10-17T21:48:35.935Zlat 0.003216182
7182023-10-17T21:48:35.960Zstop 0.211507397
7192023-10-17T21:48:35.984Z+ retry=0
7202023-10-17T21:48:36.008Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7212023-10-17T21:48:36.031Z+ [[ 0 -gt 30 ]]
7222023-10-17T21:48:36.055Z+ sleep 1
7232023-10-17T21:48:36.079Z+ retry=1
7242023-10-17T21:48:36.103Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7252023-10-17T21:48:36.126Z+ [[ 1 -gt 30 ]]
7262023-10-17T21:48:36.160Z+ sleep 1
7272023-10-17T21:48:36.186Z+ retry=2
7282023-10-17T21:48:36.210Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7292023-10-17T21:48:51.872Z+ [[ 2 -gt 30 ]]
7302023-10-17T21:48:51.898Z+ sleep 1
7312023-10-17T21:48:52.869Z+ retry=3
7322023-10-17T21:48:52.895Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7332023-10-17T21:48:52.931Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7342023-10-17T21:48:52.967Zlocal v6:
7352023-10-17T21:48:52.997Zfe80::ec89:5eff:fe36:fe21
7362023-10-17T21:48:53.031Zfd00:99::1
7372023-10-17T21:48:53.069Zlocal v4:
7382023-10-17T21:48:53.103Z10.151.1.174
7392023-10-17T21:48:53.128Zrouter v6:
7402023-10-17T21:48:53.154Zrouter v4:
7412023-10-17T21:48:53.616Zexpected [port, address], found: [2, 0, 64, 1, 97, a, 0, 0]
7422023-10-17T21:48:53.642Z0.0.0.0/0 -> ?
7432023-10-17T21:48:53.671Zresolver v4:
7442023-10-17T21:48:53.699Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7452023-10-17T21:48:53.725Zresolver v6:
7462023-10-17T21:48:53.751Zfe80::ec89:5eff:fe36:fe21 -> ee:89:5e:36:fe:21
7472023-10-17T21:48:53.776Zfe80::9ca2:35ff:fecd:6d80 -> 9e:a2:35:cd:6d:80
7482023-10-17T21:48:53.804Znat v4:
7492023-10-17T21:48:53.834Znat_v6:
7502023-10-17T21:48:54.292Zport_mac:
7512023-10-17T21:48:54.317Z1000: a8:40:25:5b:1e:0c
7522023-10-17T21:48:54.341Z1: a8:40:25:5b:1e:0d
7532023-10-17T21:48:54.365Zproxy arp:
7542023-10-17T21:48:54.391Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7552023-10-17T21:48:54.426Z+ retry=0
7562023-10-17T21:48:54.455Z+++ zoneadm list -n
7572023-10-17T21:48:54.483Z+++ grep oxz_ntp
7582023-10-17T21:48:54.517Z++ pfexec svcs -z -Hostate oxide/ntp
7592023-10-17T21:48:54.542Zsvcs: zone '-Hostate': entity not found
7602023-10-17T21:48:54.566Z++ true
7612023-10-17T21:48:54.597Z+ [[ '' != online ]]
7622023-10-17T21:48:54.632Z+ [[ 0 -gt 60 ]]
7632023-10-17T21:48:54.664Z+ sleep 1
7642023-10-17T21:48:54.701Z+ retry=1
7652023-10-17T21:48:54.735Z+++ zoneadm list -n
7662023-10-17T21:48:54.776Z+++ grep oxz_ntp
7672023-10-17T21:48:55.247Z++ pfexec svcs -z -Hostate oxide/ntp
7682023-10-17T21:48:55.277Zsvcs: zone '-Hostate': entity not found
7692023-10-17T21:48:55.308Z++ true
7702023-10-17T21:48:55.338Z+ [[ '' != online ]]
7712023-10-17T21:48:55.366Z+ [[ 1 -gt 60 ]]
7722023-10-17T21:48:55.390Z+ sleep 1
7732023-10-17T21:48:55.414Z+ retry=2
7742023-10-17T21:48:55.438Z+++ zoneadm list -n
7752023-10-17T21:48:55.462Z+++ grep oxz_ntp
7762023-10-17T21:48:55.486Z++ pfexec svcs -z -Hostate oxide/ntp
7772023-10-17T21:48:55.510Zsvcs: zone '-Hostate': entity not found
7782023-10-17T21:48:55.540Z++ true
7792023-10-17T21:48:55.573Z+ [[ '' != online ]]
7802023-10-17T21:48:55.598Z+ [[ 2 -gt 60 ]]
7812023-10-17T21:48:55.623Z+ sleep 1
7822023-10-17T21:48:55.979Z+ retry=3
7832023-10-17T21:48:56.011Z+++ zoneadm list -n
7842023-10-17T21:48:56.040Z+++ grep oxz_ntp
7852023-10-17T21:48:56.070Z++ pfexec svcs -z -Hostate oxide/ntp
7862023-10-17T21:48:56.095Zsvcs: zone '-Hostate': entity not found
7872023-10-17T21:48:56.119Z++ true
7882023-10-17T21:48:56.143Z+ [[ '' != online ]]
7892023-10-17T21:48:56.167Z+ [[ 3 -gt 60 ]]
7902023-10-17T21:48:56.192Z+ sleep 1
7912023-10-17T21:48:57.002Z+ retry=4
7922023-10-17T21:48:57.441Z+++ zoneadm list -n
7932023-10-17T21:48:57.471Z+++ grep oxz_ntp
7942023-10-17T21:48:57.496Z++ pfexec svcs -z -Hostate oxide/ntp
7952023-10-17T21:48:57.527Zsvcs: zone '-Hostate': entity not found
7962023-10-17T21:48:57.554Z++ true
7972023-10-17T21:48:57.580Z+ [[ '' != online ]]
7982023-10-17T21:48:57.610Z+ [[ 4 -gt 60 ]]
7992023-10-17T21:48:57.634Z+ sleep 1
8002023-10-17T21:48:58.020Z+ retry=5
8012023-10-17T21:48:58.045Z+++ zoneadm list -n
8022023-10-17T21:48:58.069Z+++ grep oxz_ntp
8032023-10-17T21:48:58.092Z++ pfexec svcs -z -Hostate oxide/ntp
8042023-10-17T21:48:58.141Zsvcs: zone '-Hostate': entity not found
8052023-10-17T21:48:58.165Z++ true
8062023-10-17T21:48:58.188Z+ [[ '' != online ]]
8072023-10-17T21:48:58.212Z+ [[ 5 -gt 60 ]]
8082023-10-17T21:48:58.240Z+ sleep 1
8092023-10-17T21:48:59.040Z+ retry=6
8102023-10-17T21:48:59.065Z+++ zoneadm list -n
8112023-10-17T21:48:59.089Z+++ grep oxz_ntp
8122023-10-17T21:48:59.113Z++ pfexec svcs -z -Hostate oxide/ntp
8132023-10-17T21:48:59.137Zsvcs: zone '-Hostate': entity not found
8142023-10-17T21:48:59.161Z++ true
8152023-10-17T21:48:59.187Z+ [[ '' != online ]]
8162023-10-17T21:48:59.212Z+ [[ 6 -gt 60 ]]
8172023-10-17T21:48:59.242Z+ sleep 1
8182023-10-17T21:49:00.062Z+ retry=7
8192023-10-17T21:49:00.086Z+++ zoneadm list -n
8202023-10-17T21:49:00.111Z+++ grep oxz_ntp
8212023-10-17T21:49:00.146Z++ pfexec svcs -z -Hostate oxide/ntp
8222023-10-17T21:49:00.172Zsvcs: zone '-Hostate': entity not found
8232023-10-17T21:49:00.205Z++ true
8242023-10-17T21:49:00.236Z+ [[ '' != online ]]
8252023-10-17T21:49:00.261Z+ [[ 7 -gt 60 ]]
8262023-10-17T21:49:00.285Z+ sleep 1
8272023-10-17T21:49:01.088Z+ retry=8
8282023-10-17T21:49:01.112Z+++ zoneadm list -n
8292023-10-17T21:49:01.137Z+++ grep oxz_ntp
8302023-10-17T21:49:01.162Z++ pfexec svcs -z -Hostate oxide/ntp
8312023-10-17T21:49:01.189Zsvcs: zone '-Hostate': entity not found
8322023-10-17T21:49:01.217Z++ true
8332023-10-17T21:49:01.246Z+ [[ '' != online ]]
8342023-10-17T21:49:01.270Z+ [[ 8 -gt 60 ]]
8352023-10-17T21:49:01.297Z+ sleep 1
8362023-10-17T21:49:02.111Z+ retry=9
8372023-10-17T21:49:02.577Z+++ zoneadm list -n
8382023-10-17T21:49:02.608Z+++ grep oxz_ntp
8392023-10-17T21:49:02.639Z++ pfexec svcs -z -Hostate oxide/ntp
8402023-10-17T21:49:02.668Zsvcs: zone '-Hostate': entity not found
8412023-10-17T21:49:02.712Z++ true
8422023-10-17T21:49:02.767Z+ [[ '' != online ]]
8432023-10-17T21:49:02.793Z+ [[ 9 -gt 60 ]]
8442023-10-17T21:49:02.818Z+ sleep 1
8452023-10-17T21:49:03.228Z+ retry=10
8462023-10-17T21:49:03.257Z+++ zoneadm list -n
8472023-10-17T21:49:03.284Z+++ grep oxz_ntp
8482023-10-17T21:49:03.308Z++ pfexec svcs -z -Hostate oxide/ntp
8492023-10-17T21:49:03.334Zsvcs: zone '-Hostate': entity not found
8502023-10-17T21:49:03.359Z++ true
8512023-10-17T21:49:03.383Z+ [[ '' != online ]]
8522023-10-17T21:49:03.408Z+ [[ 10 -gt 60 ]]
8532023-10-17T21:49:03.432Z+ sleep 1
8542023-10-17T21:49:04.157Z+ retry=11
8552023-10-17T21:49:04.188Z+++ zoneadm list -n
8562023-10-17T21:49:04.224Z+++ grep oxz_ntp
8572023-10-17T21:49:04.692Z++ pfexec svcs -z -Hostate oxide/ntp
8582023-10-17T21:49:04.717Zsvcs: zone '-Hostate': entity not found
8592023-10-17T21:49:04.744Z++ true
8602023-10-17T21:49:04.771Z+ [[ '' != online ]]
8612023-10-17T21:49:04.812Z+ [[ 11 -gt 60 ]]
8622023-10-17T21:49:04.839Z+ sleep 1
8632023-10-17T21:49:05.174Z+ retry=12
8642023-10-17T21:49:05.199Z+++ zoneadm list -n
8652023-10-17T21:49:05.224Z+++ grep oxz_ntp
8662023-10-17T21:49:05.249Z++ pfexec svcs -z -Hostate oxide/ntp
8672023-10-17T21:49:05.275Zsvcs: zone '-Hostate': entity not found
8682023-10-17T21:49:05.310Z++ true
8692023-10-17T21:49:05.354Z+ [[ '' != online ]]
8702023-10-17T21:49:05.805Z+ [[ 12 -gt 60 ]]
8712023-10-17T21:49:05.842Z+ sleep 1
8722023-10-17T21:49:06.204Z+ retry=13
8732023-10-17T21:49:06.232Z+++ zoneadm list -n
8742023-10-17T21:49:06.262Z+++ grep oxz_ntp
8752023-10-17T21:49:06.291Z++ pfexec svcs -z oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17 -Hostate oxide/ntp
8762023-10-17T21:49:06.320Zsvcs: Could not bind to repository server for zone oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17: repository server unavailable
8772023-10-17T21:49:06.347Z++ true
8782023-10-17T21:49:06.371Z+ [[ '' != online ]]
8792023-10-17T21:49:06.395Z+ [[ 13 -gt 60 ]]
8802023-10-17T21:49:06.425Z+ sleep 1
8812023-10-17T21:49:07.235Z+ retry=14
8822023-10-17T21:49:07.260Z+++ zoneadm list -n
8832023-10-17T21:49:07.284Z+++ grep oxz_ntp
8842023-10-17T21:49:07.308Z++ pfexec svcs -z oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17 -Hostate oxide/ntp
8852023-10-17T21:49:07.333Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8862023-10-17T21:49:07.367Z++ true
8872023-10-17T21:49:07.818Z+ [[ '' != online ]]
8882023-10-17T21:49:07.851Z+ [[ 14 -gt 60 ]]
8892023-10-17T21:49:07.912Z+ sleep 1
8902023-10-17T21:49:08.262Z+ retry=15
8912023-10-17T21:49:08.286Z+++ zoneadm list -n
8922023-10-17T21:49:08.310Z+++ grep oxz_ntp
8932023-10-17T21:49:08.335Z++ pfexec svcs -z oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17 -Hostate oxide/ntp
8942023-10-17T21:49:08.360Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8952023-10-17T21:49:08.387Z++ true
8962023-10-17T21:49:08.412Z+ [[ '' != online ]]
8972023-10-17T21:49:08.439Z+ [[ 15 -gt 60 ]]
8982023-10-17T21:49:08.464Z+ sleep 1
8992023-10-17T21:49:17.331Z+ retry=16
9002023-10-17T21:49:17.432Z+++ zoneadm list -n
9012023-10-17T21:49:17.620Z+++ grep oxz_ntp
9022023-10-17T21:49:17.665Z++ pfexec svcs -z oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17 -Hostate oxide/ntp
9032023-10-17T21:49:17.696Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9042023-10-17T21:49:17.748Z++ true
 
9052023-10-17T21:49:17.751Zbuskin console login: Oct 17 21:49:16 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9062023-10-17T21:49:17.774Z+ [[ '' != online ]]
9072023-10-17T21:49:17.832Z+ [[ 16 -gt 60 ]]
9082023-10-17T21:49:17.865Z+ sleep 1
9092023-10-17T21:49:17.894Z+ retry=17
9102023-10-17T21:49:17.925Z+++ zoneadm list -n
9112023-10-17T21:49:17.958Z+++ grep oxz_ntp
9122023-10-17T21:49:17.994Z++ pfexec svcs -z oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17 -Hostate oxide/ntp
9132023-10-17T21:49:18.030Z+ [[ disabled != online ]]
9142023-10-17T21:49:18.061Z+ [[ 17 -gt 60 ]]
9152023-10-17T21:49:18.097Z+ sleep 1
9162023-10-17T21:49:18.125Z+ retry=18
9172023-10-17T21:49:18.159Z+++ zoneadm list -n
9182023-10-17T21:49:18.228Z+++ grep oxz_ntp
9192023-10-17T21:49:18.255Z++ pfexec svcs -z oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17 -Hostate oxide/ntp
9202023-10-17T21:49:18.279Z+ [[ disabled != online ]]
9212023-10-17T21:49:18.305Z+ [[ 18 -gt 60 ]]
9222023-10-17T21:49:18.331Z+ sleep 1
9232023-10-17T21:49:18.356Z+ retry=19
9242023-10-17T21:49:18.381Z+++ zoneadm list -n
9252023-10-17T21:49:18.406Z+++ grep oxz_ntp
9262023-10-17T21:49:18.431Z++ pfexec svcs -z oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17 -Hostate oxide/ntp
9272023-10-17T21:49:18.456Z+ [[ disabled != online ]]
9282023-10-17T21:49:18.483Z+ [[ 19 -gt 60 ]]
9292023-10-17T21:49:18.513Z+ sleep 1
9302023-10-17T21:49:18.546Z+ retry=20
9312023-10-17T21:49:18.572Z+++ zoneadm list -n
9322023-10-17T21:49:18.596Z+++ grep oxz_ntp
9332023-10-17T21:49:18.620Z++ pfexec svcs -z oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17 -Hostate oxide/ntp
9342023-10-17T21:49:18.647Z+ [[ disabled != online ]]
9352023-10-17T21:49:18.674Z+ [[ 20 -gt 60 ]]
9362023-10-17T21:49:18.700Z+ sleep 1
9372023-10-17T21:49:18.725Z+ retry=21
9382023-10-17T21:49:18.755Z+++ zoneadm list -n
9392023-10-17T21:49:18.787Z+++ grep oxz_ntp
9402023-10-17T21:49:18.817Z++ pfexec svcs -z oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17 -Hostate oxide/ntp
9412023-10-17T21:49:18.882Z+ [[ disabled != online ]]
9422023-10-17T21:49:18.917Z+ [[ 21 -gt 60 ]]
9432023-10-17T21:49:18.955Z+ sleep 1
9442023-10-17T21:49:18.990Z+ retry=22
9452023-10-17T21:49:20.038Z+++ zoneadm list -n
9462023-10-17T21:49:20.087Z+++ grep oxz_ntp
9472023-10-17T21:49:20.115Z++ pfexec svcs -z oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17 -Hostate oxide/ntp
9482023-10-17T21:49:20.144Z+ [[ disabled != online ]]
9492023-10-17T21:49:20.183Z+ [[ 22 -gt 60 ]]
9502023-10-17T21:49:20.217Z+ sleep 1
9512023-10-17T21:49:20.243Z+ retry=23
9522023-10-17T21:49:20.266Z+++ zoneadm list -n
9532023-10-17T21:49:20.708Z+++ grep oxz_ntp
9542023-10-17T21:49:20.746Z++ pfexec svcs -z oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17 -Hostate oxide/ntp
9552023-10-17T21:49:21.188Z+ [[ disabled != online ]]
9562023-10-17T21:49:21.212Z+ [[ 23 -gt 60 ]]
9572023-10-17T21:49:21.236Z+ sleep 1
9582023-10-17T21:49:21.265Z+ retry=24
9592023-10-17T21:49:21.698Z+++ zoneadm list -n
9602023-10-17T21:49:21.723Z+++ grep oxz_ntp
9612023-10-17T21:49:21.748Z++ pfexec svcs -z oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17 -Hostate oxide/ntp
9622023-10-17T21:49:21.772Z+ [[ disabled != online ]]
9632023-10-17T21:49:21.797Z+ [[ 24 -gt 60 ]]
9642023-10-17T21:49:21.821Z+ sleep 1
9652023-10-17T21:49:21.858Z+ retry=25
9662023-10-17T21:49:21.889Z+++ zoneadm list -n
9672023-10-17T21:49:21.914Z+++ grep oxz_ntp
9682023-10-17T21:49:21.940Z++ pfexec svcs -z oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17 -Hostate oxide/ntp
9692023-10-17T21:49:21.985Z+ [[ online != online ]]
9702023-10-17T21:49:22.009Z+ echo 'Waited for chrony: 25s'
9712023-10-17T21:49:22.034ZWaited for chrony: 25s
9722023-10-17T21:49:22.058Z+ export RUST_BACKTRACE=1
9732023-10-17T21:49:22.084Z+ RUST_BACKTRACE=1
9742023-10-17T21:49:22.118Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9752023-10-17T21:49:22.147Z+ ./tests/bootstrap
9762023-10-17T21:49:22.172ZUsing external DNS server at 10.151.1.164:53
9772023-10-17T21:49:22.196Z2023-10-17 21:49:17.780777832 UTC: attempting to log into API
9782023-10-17T21:49:33.584Z2023-10-17 21:49:33.840189978 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9792023-10-17T21:49:34.584Z2023-10-17 21:49:34.841002768 UTC: attempting to log into API
9802023-10-17T21:49:49.616Z2023-10-17 21:49:49.873359617 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9812023-10-17T21:49:50.623Z2023-10-17 21:49:50.874233928 UTC: attempting to log into API
9822023-10-17T21:50:05.664Z2023-10-17 21:50:05.907024353 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9832023-10-17T21:50:06.653Z2023-10-17 21:50:06.908161737 UTC: attempting to log into API
9842023-10-17T21:50:21.685Z2023-10-17 21:50:21.941782633 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9852023-10-17T21:50:22.686Z2023-10-17 21:50:22.942653364 UTC: attempting to log into API
 
9862023-10-17T21:50:28.519ZOct 17 21:50:28 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9872023-10-17T21:50:35.808ZOct 17 21:50:35 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9882023-10-17T21:50:37.742Z2023-10-17 21:50:37.856738859 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9892023-10-17T21:50:38.744Z2023-10-17 21:50:38.855870823 UTC: attempting to log into API
 
9902023-10-17T21:50:41.810ZOct 17 21:50:41 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9912023-10-17T21:50:50.134ZOct 17 21:50:49 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9922023-10-17T21:50:53.777Z2023-10-17 21:50:53.848775969 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9932023-10-17T21:50:54.779Z2023-10-17 21:50:54.847160483 UTC: attempting to log into API
 
9942023-10-17T21:50:55.102ZOct 17 21:50:54 bukin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9952023-10-17T21:51:09.808Z2023-10-17 21:51:09.838480107 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9962023-10-17T21:51:10.825Z2023-10-17 21:51:10.836886392 UTC: attempting to log into API
9972023-10-17T21:51:21.623Z2023-10-17 21:51:21.611283814 UTC: login succeeded
9982023-10-17T21:51:21.648Zwaiting for nexus to come up...
9992023-10-17T21:51:21.709Zcreating IP pool... 10.151.1.175 - 10.151.1.227
10002023-10-17T21:51:21.762Zensuring datasets are ready...
10012023-10-17T21:51:24.532Zlet's roll.
10022023-10-17T21:51:24.556Z+ rm ./tests/bootstrap
10032023-10-17T21:51:24.598Z+ for test_bin in tests/*
10042023-10-17T21:51:24.622Z+ ./tests/bootstrap-8f412cf8dfc7d0db
10052023-10-17T21:51:24.646Z
10062023-10-17T21:51:24.670Zrunning 0 tests
10072023-10-17T21:51:24.694Z
10082023-10-17T21:51:24.718Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10092023-10-17T21:51:24.743Z
10102023-10-17T21:51:24.767Z+ for test_bin in tests/*
10112023-10-17T21:51:24.791Z+ ./tests/end_to_end_tests-2c7e0af8d3a77456
10122023-10-17T21:51:24.814Z
10132023-10-17T21:51:24.838Zrunning 2 tests
10142023-10-17T21:51:24.863Ztest no_spoof::no_spoof_here ... ok
10152023-10-17T21:52:24.629Ztest instance_launch::instance_launch has been running for over 60 seconds
10162023-10-17T21:54:21.597Ztest instance_launch::instance_launch ... ok
10172023-10-17T21:54:21.622Z
10182023-10-17T21:54:21.650Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 176.97s
10192023-10-17T21:54:21.680Z
10202023-10-17T21:54:21.704Z+ _exit_trap
10212023-10-17T21:54:21.751Z+ local status=0
10222023-10-17T21:54:21.775Z+ [[ 0 -eq 0 ]]
10232023-10-17T21:54:21.803Z+ exit 0
10242023-10-17T21:54:21.836Zprocess exited: duration 383810 ms, exit code 0
 
10252023-10-17T21:54:21.947Zfound 47 output files
10262023-10-17T21:54:22.132Zuploading: /var/svc/log/oxide-sled-agent:default.log (435634 bytes)
10272023-10-17T21:54:23.208Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10282023-10-17T21:54:23.235Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e988bc77-03f0-44e5-93bc-b780aea7f994/root/var/svc/log/oxide-crucible-agent:default.log (24825 bytes)
10292023-10-17T21:54:24.355Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e988bc77-03f0-44e5-93bc-b780aea7f994/root/var/svc/log/oxide-crucible-agent:default.log
10302023-10-17T21:54:24.378Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e988bc77-03f0-44e5-93bc-b780aea7f994/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10312023-10-17T21:54:25.449Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e988bc77-03f0-44e5-93bc-b780aea7f994/root/var/svc/log/oxide-crucible-downstairs:default.log
10322023-10-17T21:54:25.473Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e988bc77-03f0-44e5-93bc-b780aea7f994/root/var/svc/log/oxide-crucible-downstairs:downstairs-07080d1d-a46a-4089-96ff-698780701e25.log (5536 bytes)
10332023-10-17T21:54:26.560Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e988bc77-03f0-44e5-93bc-b780aea7f994/root/var/svc/log/oxide-crucible-downstairs:downstairs-07080d1d-a46a-4089-96ff-698780701e25.log
10342023-10-17T21:54:26.586Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e988bc77-03f0-44e5-93bc-b780aea7f994/root/var/svc/log/oxide-crucible-downstairs:downstairs-4b735d01-6645-44f2-a3ac-6a4cd686d120.log (3050 bytes)
10352023-10-17T21:54:27.655Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e988bc77-03f0-44e5-93bc-b780aea7f994/root/var/svc/log/oxide-crucible-downstairs:downstairs-4b735d01-6645-44f2-a3ac-6a4cd686d120.log
10362023-10-17T21:54:27.680Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f38be4bf-a23c-4d29-bf85-6630ed04ab77/root/var/svc/log/oxide-crucible-agent:default.log (14118 bytes)
10372023-10-17T21:54:28.772Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f38be4bf-a23c-4d29-bf85-6630ed04ab77/root/var/svc/log/oxide-crucible-agent:default.log
10382023-10-17T21:54:28.796Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f38be4bf-a23c-4d29-bf85-6630ed04ab77/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10392023-10-17T21:54:29.873Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f38be4bf-a23c-4d29-bf85-6630ed04ab77/root/var/svc/log/oxide-crucible-downstairs:default.log
10402023-10-17T21:54:29.908Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f38be4bf-a23c-4d29-bf85-6630ed04ab77/root/var/svc/log/oxide-crucible-downstairs:downstairs-616f7def-62f6-45e1-aa3c-9ac1746f8f00.log (3051 bytes)
10412023-10-17T21:54:30.987Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f38be4bf-a23c-4d29-bf85-6630ed04ab77/root/var/svc/log/oxide-crucible-downstairs:downstairs-616f7def-62f6-45e1-aa3c-9ac1746f8f00.log
10422023-10-17T21:54:31.051Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_dec25c9c-0de8-4c4c-8c4b-698293bd71eb/root/var/svc/log/oxide-internal_dns:default.log (99570 bytes)
10432023-10-17T21:54:32.219Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_dec25c9c-0de8-4c4c-8c4b-698293bd71eb/root/var/svc/log/oxide-internal_dns:default.log
10442023-10-17T21:54:32.245Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_52c56600-a42d-4052-9275-9f6fe4047086/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10452023-10-17T21:54:33.339Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_52c56600-a42d-4052-9275-9f6fe4047086/root/var/svc/log/oxide-cockroachdb:default.log
10462023-10-17T21:54:33.371Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_59bfd7bc-6e82-4513-b2f4-173f0a7a2ff0/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10472023-10-17T21:54:34.442Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_59bfd7bc-6e82-4513-b2f4-173f0a7a2ff0/root/var/svc/log/oxide-crucible-agent:default.log
10482023-10-17T21:54:34.466Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_59bfd7bc-6e82-4513-b2f4-173f0a7a2ff0/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10492023-10-17T21:54:35.537Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_59bfd7bc-6e82-4513-b2f4-173f0a7a2ff0/root/var/svc/log/oxide-crucible-downstairs:default.log
10502023-10-17T21:54:35.583Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c223de37-0e7b-42e9-96e2-4a32e54166ea/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10512023-10-17T21:54:36.663Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c223de37-0e7b-42e9-96e2-4a32e54166ea/root/var/svc/log/oxide-crucible-agent:default.log
10522023-10-17T21:54:36.687Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c223de37-0e7b-42e9-96e2-4a32e54166ea/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10532023-10-17T21:54:38.168Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c223de37-0e7b-42e9-96e2-4a32e54166ea/root/var/svc/log/oxide-crucible-downstairs:default.log
10542023-10-17T21:54:38.192Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_c7ecb282-d460-4a41-b06d-f07fedce7987/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10552023-10-17T21:54:39.272Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_c7ecb282-d460-4a41-b06d-f07fedce7987/root/var/svc/log/oxide-crucible-pantry:default.log
10562023-10-17T21:54:39.328Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_9fbc7e13-4b77-4a2b-a573-5f2b93ad6ac6/root/var/svc/log/oxide-external_dns:default.log (15601 bytes)
10572023-10-17T21:54:40.428Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_9fbc7e13-4b77-4a2b-a573-5f2b93ad6ac6/root/var/svc/log/oxide-external_dns:default.log
10582023-10-17T21:54:40.473Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_762640f8-d24d-46b3-846f-8b4dc942b589/root/var/svc/log/oxide-nexus:default.log (500319 bytes)
10592023-10-17T21:54:40.703Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_762640f8-d24d-46b3-846f-8b4dc942b589/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 500319 -> 501534
10602023-10-17T21:54:41.774Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_762640f8-d24d-46b3-846f-8b4dc942b589/root/var/svc/log/oxide-nexus:default.log
10612023-10-17T21:54:41.798Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7cd6c81a-6277-4a98-854a-9f7371177895/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10622023-10-17T21:54:42.869Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7cd6c81a-6277-4a98-854a-9f7371177895/root/var/svc/log/oxide-crucible-agent:default.log
10632023-10-17T21:54:42.894Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7cd6c81a-6277-4a98-854a-9f7371177895/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10642023-10-17T21:54:43.973Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7cd6c81a-6277-4a98-854a-9f7371177895/root/var/svc/log/oxide-crucible-downstairs:default.log
10652023-10-17T21:54:44.005Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17/root/var/svc/log/oxide-ntp:default.log (1112 bytes)
10662023-10-17T21:54:45.085Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17/root/var/svc/log/oxide-ntp:default.log
10672023-10-17T21:54:45.117Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_dc1d7f3b-1c31-4615-be1b-ee84844cb805/root/var/svc/log/oxide-cockroachdb:default.log (5366 bytes)
10682023-10-17T21:54:46.199Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_dc1d7f3b-1c31-4615-be1b-ee84844cb805/root/var/svc/log/oxide-cockroachdb:default.log
10692023-10-17T21:54:46.224Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_6ecfe85f-5e33-4839-9fe0-e55b41123267/root/var/svc/log/oxide-external_dns:default.log (15603 bytes)
10702023-10-17T21:54:47.749Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_6ecfe85f-5e33-4839-9fe0-e55b41123267/root/var/svc/log/oxide-external_dns:default.log
10712023-10-17T21:54:47.778Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_7602efb5-bbfc-49be-baf0-f8079f390002/root/var/svc/log/oxide-nexus:default.log (4634504 bytes)
10722023-10-17T21:54:55.282Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_7602efb5-bbfc-49be-baf0-f8079f390002/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4634504 -> 4636331
10732023-10-17T21:54:56.356Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_7602efb5-bbfc-49be-baf0-f8079f390002/root/var/svc/log/oxide-nexus:default.log
10742023-10-17T21:54:56.410Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_a8b43527-a6aa-43f7-8738-4652d467183e/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10752023-10-17T21:54:57.491Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_a8b43527-a6aa-43f7-8738-4652d467183e/root/var/svc/log/oxide-cockroachdb:default.log
10762023-10-17T21:54:57.519Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_356dbb58-41c3-4d3f-83d1-ffade1495d2b/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10772023-10-17T21:54:58.618Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_356dbb58-41c3-4d3f-83d1-ffade1495d2b/root/var/svc/log/oxide-crucible-agent:default.log
10782023-10-17T21:54:58.646Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_356dbb58-41c3-4d3f-83d1-ffade1495d2b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10792023-10-17T21:54:59.749Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_356dbb58-41c3-4d3f-83d1-ffade1495d2b/root/var/svc/log/oxide-crucible-downstairs:default.log
10802023-10-17T21:54:59.799Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ea8bca7f-8624-41a5-b46e-d6525f53b994/root/var/svc/log/oxide-crucible-agent:default.log (14577 bytes)
10812023-10-17T21:55:01.285Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ea8bca7f-8624-41a5-b46e-d6525f53b994/root/var/svc/log/oxide-crucible-agent:default.log
10822023-10-17T21:55:01.310Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ea8bca7f-8624-41a5-b46e-d6525f53b994/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10832023-10-17T21:55:02.390Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ea8bca7f-8624-41a5-b46e-d6525f53b994/root/var/svc/log/oxide-crucible-downstairs:default.log
10842023-10-17T21:55:02.576Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ea8bca7f-8624-41a5-b46e-d6525f53b994/root/var/svc/log/oxide-crucible-downstairs:downstairs-214f403c-555a-4bb1-8931-984a440d918f.log (5536 bytes)
10852023-10-17T21:55:03.766Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ea8bca7f-8624-41a5-b46e-d6525f53b994/root/var/svc/log/oxide-crucible-downstairs:downstairs-214f403c-555a-4bb1-8931-984a440d918f.log
10862023-10-17T21:55:03.912Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_3462c163-87db-4a32-b2e7-ac72cb52a5ff/root/var/svc/log/oxide-internal_dns:default.log (101093 bytes)
10872023-10-17T21:55:04.308Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_3462c163-87db-4a32-b2e7-ac72cb52a5ff/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 101093 -> 104849
10882023-10-17T21:55:05.076Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_3462c163-87db-4a32-b2e7-ac72cb52a5ff/root/var/svc/log/oxide-internal_dns:default.log
10892023-10-17T21:55:05.267Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_37a7f3eb-5b1d-4173-a3b1-69d1d7c00308/root/var/svc/log/oxide-cockroachdb:default.log (5342 bytes)
10902023-10-17T21:55:06.485Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_37a7f3eb-5b1d-4173-a3b1-69d1d7c00308/root/var/svc/log/oxide-cockroachdb:default.log
10912023-10-17T21:55:06.727Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_75a57950-bc73-4891-8fe7-9c85737bf271/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10922023-10-17T21:55:07.997Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_75a57950-bc73-4891-8fe7-9c85737bf271/root/var/svc/log/oxide-cockroachdb:default.log
10932023-10-17T21:55:08.079Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_af4a2ddf-f544-4897-b5c6-11e933fb601a/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10942023-10-17T21:55:09.238Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_af4a2ddf-f544-4897-b5c6-11e933fb601a/root/var/svc/log/oxide-crucible-pantry:default.log
10952023-10-17T21:55:09.457Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_3801ec6d-666d-4371-a384-bf9ae19dc090/root/var/svc/log/oxide-internal_dns:default.log (104341 bytes)
10962023-10-17T21:55:09.586Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_3801ec6d-666d-4371-a384-bf9ae19dc090/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 104341 -> 108094
10972023-10-17T21:55:10.751Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_3801ec6d-666d-4371-a384-bf9ae19dc090/root/var/svc/log/oxide-internal_dns:default.log
10982023-10-17T21:55:10.982Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_96a43284-0332-4381-9e2d-fd9e127feca9/root/var/svc/log/oxide-clickhouse:default.log (668984 bytes)
10992023-10-17T21:55:29.656Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_96a43284-0332-4381-9e2d-fd9e127feca9/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 668984 -> 881000
11002023-10-17T21:55:30.776Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_96a43284-0332-4381-9e2d-fd9e127feca9/root/var/svc/log/oxide-clickhouse:default.log
11012023-10-17T21:55:30.814Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b73cc404-e170-48c3-a4b1-5e2f2a361e10/root/var/svc/log/oxide-crucible-agent:default.log (15047 bytes)
11022023-10-17T21:55:31.914Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b73cc404-e170-48c3-a4b1-5e2f2a361e10/root/var/svc/log/oxide-crucible-agent:default.log
11032023-10-17T21:55:31.940Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b73cc404-e170-48c3-a4b1-5e2f2a361e10/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11042023-10-17T21:55:33.015Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b73cc404-e170-48c3-a4b1-5e2f2a361e10/root/var/svc/log/oxide-crucible-downstairs:default.log
11052023-10-17T21:55:33.039Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b73cc404-e170-48c3-a4b1-5e2f2a361e10/root/var/svc/log/oxide-crucible-downstairs:downstairs-beae77cc-7830-445c-a72b-f3e11fe56740.log (5537 bytes)
11062023-10-17T21:55:34.117Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b73cc404-e170-48c3-a4b1-5e2f2a361e10/root/var/svc/log/oxide-crucible-downstairs:downstairs-beae77cc-7830-445c-a72b-f3e11fe56740.log
11072023-10-17T21:55:34.144Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e3108364-610c-4bff-8cae-6661b4a348fc/root/var/svc/log/oxide-crucible-agent:default.log (14583 bytes)
11082023-10-17T21:55:35.251Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e3108364-610c-4bff-8cae-6661b4a348fc/root/var/svc/log/oxide-crucible-agent:default.log
11092023-10-17T21:55:35.276Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e3108364-610c-4bff-8cae-6661b4a348fc/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11102023-10-17T21:55:36.346Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e3108364-610c-4bff-8cae-6661b4a348fc/root/var/svc/log/oxide-crucible-downstairs:default.log
11112023-10-17T21:55:36.370Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e3108364-610c-4bff-8cae-6661b4a348fc/root/var/svc/log/oxide-crucible-downstairs:downstairs-07c46c80-01b4-4642-b6da-023de917ae28.log (3051 bytes)
11122023-10-17T21:55:37.446Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e3108364-610c-4bff-8cae-6661b4a348fc/root/var/svc/log/oxide-crucible-downstairs:downstairs-07c46c80-01b4-4642-b6da-023de917ae28.log
11132023-10-17T21:55:37.482Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8b050ed6-4d32-4d78-83ac-91b05fd34d26/root/var/svc/log/oxide-nexus:default.log (504945 bytes)
11142023-10-17T21:55:37.725Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8b050ed6-4d32-4d78-83ac-91b05fd34d26/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 504945 -> 567990
11152023-10-17T21:55:38.816Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8b050ed6-4d32-4d78-83ac-91b05fd34d26/root/var/svc/log/oxide-nexus:default.log
11162023-10-17T21:55:38.843Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_28b5d261-a5e8-42e1-b98e-df37ac0d2502/root/var/svc/log/oxide-oximeter:default.log (185347 bytes)
11172023-10-17T21:55:39.918Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_28b5d261-a5e8-42e1-b98e-df37ac0d2502/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 185347 -> 248043
11182023-10-17T21:55:41.372Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_28b5d261-a5e8-42e1-b98e-df37ac0d2502/root/var/svc/log/oxide-oximeter:default.log
11192023-10-17T21:55:41.418Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_271c413c-118a-41cd-96da-43bc96cd7ff8/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
11202023-10-17T21:55:42.486Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_271c413c-118a-41cd-96da-43bc96cd7ff8/root/var/svc/log/oxide-crucible-pantry:default.log
11212023-10-17T21:55:42.518Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17/root/var/log/chrony/measurements.log (12056 bytes)
11222023-10-17T21:55:42.577Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17/root/var/log/chrony/measurements.log" changed size mid upload: 12056 -> 14111
11232023-10-17T21:55:43.654Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17/root/var/log/chrony/measurements.log
11242023-10-17T21:55:43.687Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17/root/var/log/chrony/statistics.log (6084 bytes)
11252023-10-17T21:55:43.731Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17/root/var/log/chrony/statistics.log" changed size mid upload: 6084 -> 7722
11262023-10-17T21:55:44.809Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17/root/var/log/chrony/statistics.log
11272023-10-17T21:55:44.835Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17/root/var/log/chrony/tracking.log (1585 bytes)
11282023-10-17T21:55:44.859Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17/root/var/log/chrony/tracking.log" changed size mid upload: 1585 -> 1717
11292023-10-17T21:55:45.928Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e92483ca-f4df-43f6-9dd2-60458801cb17/root/var/log/chrony/tracking.log