01HHCZV1X4KEXGGMSH7MM8WZJ4: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.27",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HHCZVWDY9D7STTAW47YEXYMK

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-11T18:33:34.390Zjob dependencies complete; ready to run (waiting for 45 m 50 s)
22023-12-11T18:39:36.453Zjob assigned to worker 01HHD2FVZGXSSJSPY1Z6VP7D85 (queued for 6 m 2 s)
32023-12-11T18:39:36.570Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-12-11T18:39:36.610Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-12-11T18:39:36.643Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-12-11T18:39:36.677Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-11T18:39:36.711Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-12-11T18:39:36.744ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-11T18:39:36.778Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-11T18:39:36.814ZOxide Helios Version helios-2.0.22333 64-bit
112023-12-11T18:39:36.851ZConfiguring devices.
122023-12-11T18:39:36.889Z##################################################
132023-12-11T18:39:36.923Z#### ############# ##
142023-12-11T18:39:36.956Z### ### ############ ##
152023-12-11T18:39:36.990Z## ### # ## ### ## Oxide Computer Company ##
162023-12-11T18:39:37.024Z## ## ## ### # ### ##
172023-12-11T18:39:37.066Z## # ### #### #### This Station Under ##
182023-12-11T18:39:37.100Z### ### #### # ### Computer Control ##
192023-12-11T18:39:37.134Z#### #### ### ## ##
202023-12-11T18:39:37.167Z##################################################
212023-12-11T18:39:37.201ZHostname: sock
222023-12-11T18:39:37.293ZLoading smf(7) service descriptions:
232023-12-11T18:39:37.926Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-8898bd4b8ded3394.gz
242023-12-11T18:39:39.201Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-8898bd4b8ded3394.gz
252023-12-11T18:39:39.226Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-11T18:39:58.167Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-11T18:39:58.192Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-76081d63fda74637.gz
282023-12-11T18:40:18.369Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-76081d63fda74637.gz
292023-12-11T18:40:18.394Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-11T18:40:19.329Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-11T18:40:19.329Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-11T18:40:24.245Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-11T18:40:24.245Zdownloading input: /input/package/work/version.txt
342023-12-11T18:40:24.245Zdownloaded input: /input/package/work/version.txt
352023-12-11T18:40:24.245Zdownloading input: /input/package/work/package.tar.gz
362023-12-11T18:40:38.531Zdownloaded input: /input/package/work/package.tar.gz
372023-12-11T18:40:38.531Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-11T18:41:00.548Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-11T18:41:00.548Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-11T18:41:07.881Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-11T18:41:07.881Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-11T18:41:39.515Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-11T18:41:39.515Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-11T18:42:11.422Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-11T18:42:11.422Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-11T18:42:33.375Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-11T18:42:33.404Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-11T18:42:38.336Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-11T18:42:38.361Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-11T18:42:46.969Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-11T18:42:47.002Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-11T18:42:52.596Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-11T18:42:52.596Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-11T18:42:57.885Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-11T18:42:57.886Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-11T18:42:57.931Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-11T18:42:57.957Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-11T18:43:02.116Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-11T18:43:02.117Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-11T18:43:06.326Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-11T18:43:06.326Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-11T18:43:25.931Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-11T18:43:25.931Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-11T18:43:43.982Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-11T18:43:43.983Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-11T18:43:44.048Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-11T18:43:44.048Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-11T18:43:51.374Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-11T18:43:51.374Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-11T18:44:05.483Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-11T18:44:05.484Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-11T18:44:56.812Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-11T18:44:56.812Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-11T18:45:36.295Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-11T18:45:36.321Zstarting task 0: "setup"
762023-12-11T18:45:36.322Z++ uname -s
772023-12-11T18:45:36.322Z+ kern=SunOS
782023-12-11T18:45:36.322Z+ case "$kern" in
792023-12-11T18:45:36.322Z+ groupadd -g 12345 build
802023-12-11T18:45:36.322Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-11T18:45:38.314Z+ zfs create -o mountpoint=/work rpool/work
822023-12-11T18:45:38.338Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-11T18:45:38.363Z+ home_fs=zfs
842023-12-11T18:45:38.363Z+ [[ zfs == autofs ]]
852023-12-11T18:45:38.364Z+ mkdir -p /home/build
862023-12-11T18:45:38.364Z+ chown build:build /home/build /work
872023-12-11T18:45:40.364Z+ chmod 0700 /home/build /work
882023-12-11T18:45:40.390Zprocess exited: duration 4054 ms, exit code 0
 
892023-12-11T18:45:40.451Zstarting task 1: "authentication"
902023-12-11T18:45:40.493Zprocess exited: duration 43 ms, exit code 0
 
912023-12-11T18:45:40.542Zstarting task 2: "build"
922023-12-11T18:45:40.568Z+ trap _exit_trap EXIT
932023-12-11T18:45:40.568Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-11T18:45:40.744Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-11T18:45:40.744Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-11T18:45:40.744Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-11T18:45:40.744Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-11T18:45:40.744Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-11T18:45:40.744Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-11T18:45:40.769Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-11T18:45:40.769Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-11T18:45:40.769Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-11T18:45:40.769Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-11T18:45:40.769Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-11T18:45:40.769Z+ [[ -d /opt/oxide ]]
1062023-12-11T18:45:40.769Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-11T18:45:40.769Z+ pfexec mkdir /opt/oxide
1082023-12-11T18:45:40.769Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-11T18:45:40.769Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-11T18:45:40.769Z+ cd /opt/oxide-underneath
1112023-12-11T18:45:40.769Z+ pfexec tar ceEp@/f - .
1122023-12-11T18:45:40.769Z+ cd /opt/oxide
1132023-12-11T18:45:40.770Z+ pfexec tar xveEp@/f -
1142023-12-11T18:45:40.876Zx ., 0 bytes, 0 tape blocks
1152023-12-11T18:45:40.877Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-11T18:45:40.877Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-11T18:45:40.877Zx ./opte/bin/opteadm, 52567720 bytes, 102672 tape blocks
1182023-12-11T18:45:40.908Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-11T18:45:40.909Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-11T18:45:40.909Z+ pfexec mkdir /opt/oxide/work
1212023-12-11T18:45:40.909Z+ pfexec chown build:build /opt/oxide/work
1222023-12-11T18:45:40.939Z+ cd /opt/oxide/work
1232023-12-11T18:45:40.940Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-11T18:45:42.463ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-12-11T18:45:42.463Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-12-11T18:45:42.464Zx out/omicron-sled-agent.tar, 111842816 bytes, 218443 tape blocks
1272023-12-11T18:45:42.464Zx out/target/test, 75 bytes, 1 tape blocks
1282023-12-11T18:45:42.464Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1292023-12-11T18:45:42.464Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-12-11T18:45:42.464Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1312023-12-11T18:45:42.464Zx target/release/omicron-package, 42512064 bytes, 83032 tape blocks
1322023-12-11T18:45:42.465Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-12-11T18:45:42.465Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-12-11T18:45:42.465Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-12-11T18:45:42.465Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-12-11T18:45:42.467Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-12-11T18:45:42.467Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-12-11T18:45:42.467Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-12-11T18:45:42.497Z
1402023-12-11T18:45:42.497Zreal 1.545462330
1412023-12-11T18:45:42.498Zuser 1.383264634
1422023-12-11T18:45:42.498Zsys 0.279695128
1432023-12-11T18:45:42.498Ztrap 0.000056352
1442023-12-11T18:45:42.498Ztflt 0.000000000
1452023-12-11T18:45:42.498Zdflt 0.000000000
1462023-12-11T18:45:42.498Zkflt 0.000000000
1472023-12-11T18:45:42.498Zlock 0.000000000
1482023-12-11T18:45:42.498Zslp 1.391505205
1492023-12-11T18:45:42.498Zlat 0.034177124
1502023-12-11T18:45:42.498Zstop 0.000127842
1512023-12-11T18:45:42.499Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-12-11T18:45:43.543Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-12-11T18:45:43.571Z+ mkdir tests
1542023-12-11T18:45:43.571Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-12-11T18:45:43.571Z+ ptime -m gunzip
1562023-12-11T18:45:43.571Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8898bd4b8ded3394
1572023-12-11T18:45:43.696Z
1582023-12-11T18:45:43.721Zreal 0.139917305
1592023-12-11T18:45:43.721Zuser 0.126378737
1602023-12-11T18:45:43.721Zsys 0.013302221
1612023-12-11T18:45:43.721Ztrap 0.000010130
1622023-12-11T18:45:43.721Ztflt 0.000000000
1632023-12-11T18:45:43.721Zdflt 0.000000000
1642023-12-11T18:45:43.721Zkflt 0.000000000
1652023-12-11T18:45:43.721Zlock 0.000000000
1662023-12-11T18:45:43.721Zslp 0.000024360
1672023-12-11T18:45:43.721Zlat 0.000024270
1682023-12-11T18:45:43.721Zstop 0.000122993
1692023-12-11T18:45:43.722Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8898bd4b8ded3394
1702023-12-11T18:45:43.722Z+ chmod a+x tests/bootstrap-8898bd4b8ded3394
1712023-12-11T18:45:43.722Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-12-11T18:45:43.722Z+ ptime -m gunzip
1732023-12-11T18:45:43.722Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-12-11T18:45:45.916Z
1752023-12-11T18:45:45.940Zreal 2.207942699
1762023-12-11T18:45:45.940Zuser 2.080229540
1772023-12-11T18:45:45.941Zsys 0.127333423
1782023-12-11T18:45:45.941Ztrap 0.000170542
1792023-12-11T18:45:45.941Ztflt 0.000000000
1802023-12-11T18:45:45.941Zdflt 0.000000000
1812023-12-11T18:45:45.941Zkflt 0.000000000
1822023-12-11T18:45:45.941Zlock 0.000000000
1832023-12-11T18:45:45.941Zslp 0.000021570
1842023-12-11T18:45:45.941Zlat 0.000016740
1852023-12-11T18:45:45.941Zstop 0.000109012
1862023-12-11T18:45:45.941Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-12-11T18:45:45.941Z+ chmod a+x tests/bootstrap
1882023-12-11T18:45:45.941Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-12-11T18:45:45.941Z+ ptime -m gunzip
1902023-12-11T18:45:45.941Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-76081d63fda74637
1912023-12-11T18:45:48.217Z
1922023-12-11T18:45:48.243Zreal 2.289204416
1932023-12-11T18:45:48.243Zuser 2.158336588
1942023-12-11T18:45:48.243Zsys 0.130440550
1952023-12-11T18:45:48.243Ztrap 0.000192553
1962023-12-11T18:45:48.243Ztflt 0.000000000
1972023-12-11T18:45:48.243Zdflt 0.000000000
1982023-12-11T18:45:48.243Zkflt 0.000000000
1992023-12-11T18:45:48.243Zlock 0.000000000
2002023-12-11T18:45:48.243Zslp 0.000022200
2012023-12-11T18:45:48.243Zlat 0.000022801
2022023-12-11T18:45:48.243Zstop 0.000123652
2032023-12-11T18:45:48.243Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-76081d63fda74637
2042023-12-11T18:45:48.243Z+ chmod a+x tests/end_to_end_tests-76081d63fda74637
2052023-12-11T18:45:48.243Z+ bmat address ls
2062023-12-11T18:45:48.243ZNAME CIDR FIRST LAST COUNT
2072023-12-11T18:45:48.243Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-12-11T18:45:48.244Z++ bmat address ls -f extra -Ho first,last,count
2092023-12-11T18:45:48.268Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-12-11T18:45:48.268Z+ EXTRA_IP_START=10.151.2.164
2112023-12-11T18:45:48.268Z+ EXTRA_IP_END=10.151.2.227
2122023-12-11T18:45:48.270Z+ EXTRA_IP_COUNT=64
2132023-12-11T18:45:48.270Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-12-11T18:45:48.270Z+ EXTRA_IP_BASE=10.151.2
2152023-12-11T18:45:48.270Z+ EXTRA_IP_FOCTET=164
2162023-12-11T18:45:48.270Z+ EXTRA_IP_LOCTET=227
2172023-12-11T18:45:48.270Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-12-11T18:45:48.270Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-12-11T18:45:48.270Z+ DNS_IP1=10.151.2.164
2202023-12-11T18:45:48.270Z+ DNS_IP2=10.151.2.165
2212023-12-11T18:45:48.270Z+ UPLINK_IP=10.151.2.174
2222023-12-11T18:45:48.270Z+ IPPOOL_START=10.151.2.175
2232023-12-11T18:45:48.270Z+ IPPOOL_END=10.151.2.227
2242023-12-11T18:45:48.270Z++ ipadm show-addr -po type,addr
2252023-12-11T18:45:48.270Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-12-11T18:45:48.270Z+ GATEWAY_IP=10.151.2.100
2272023-12-11T18:45:48.270Z+ [[ -n 10.151.2.100 ]]
2282023-12-11T18:45:48.270Z+ ping -s 10.151.2.100 56 1
2292023-12-11T18:45:48.271ZPING 10.151.2.100: 56 data bytes
2302023-12-11T18:45:48.271Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2312023-12-11T18:45:48.271Z
2322023-12-11T18:45:48.271Z----10.151.2.100 PING Statistics----
2332023-12-11T18:45:48.273Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-12-11T18:45:48.273Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-12-11T18:45:48.273Z++ arp -an
2362023-12-11T18:45:48.273Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-12-11T18:45:48.298Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-12-11T18:45:48.298Z+ routeadm -e ipv4-forwarding -u
2392023-12-11T18:45:48.379Z+ PXA_START=10.151.2.164
2402023-12-11T18:45:48.410Z+ PXA_END=10.151.2.227
2412023-12-11T18:45:48.410Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-12-11T18:45:48.410Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-12-11T18:45:48.572Z+ ZPOOL_VDEV_DIR=/scratch
2442023-12-11T18:45:48.572Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-12-11T18:45:48.597Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-12-11T18:45:48.597Z++ cd ./tools
2472023-12-11T18:45:48.597Z++ pwd
2482023-12-11T18:45:48.597Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-12-11T18:45:48.598Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-12-11T18:45:48.598Z+ SOFTNPU_MODE=zone
2512023-12-11T18:45:48.598Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-12-11T18:45:48.598Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-12-11T18:45:48.598Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-12-11T18:45:48.598Z++ dladm show-phys -p -o LINK
2552023-12-11T18:45:48.598Z++ head -1
2562023-12-11T18:45:48.670Z+ PHYSICAL_LINK=igb0
2572023-12-11T18:45:48.702Z+ echo 'Using igb0 as physical link'
2582023-12-11T18:45:48.702ZUsing igb0 as physical link
2592023-12-11T18:45:48.703Z+ ensure_run_as_root
2602023-12-11T18:45:48.703Z++ id -u
2612023-12-11T18:45:48.703Z+ [[ 0 -ne 0 ]]
2622023-12-11T18:45:48.703Z+ ensure_zpools
2632023-12-11T18:45:48.703Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-12-11T18:45:48.704Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-12-11T18:45:48.704Z+ readarray -t ZPOOLS
2662023-12-11T18:45:48.704Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-12-11T18:45:48.704Z++ sed 's/[ ",]//g'
2682023-12-11T18:45:48.704Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-12-11T18:45:48.704Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-12-11T18:45:48.704ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-12-11T18:45:48.704Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-12-11T18:45:48.704Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-12-11T18:45:48.704Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-12-11T18:45:48.704Z0+0 records in
2752023-12-11T18:45:48.704Z0+0 records out
2762023-12-11T18:45:48.704Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2772023-12-11T18:45:48.704Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-12-11T18:45:48.704Z+ set +x
2792023-12-11T18:45:48.704ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-12-11T18:45:48.704Z++ zpool list -o name
2812023-12-11T18:45:48.705Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-12-11T18:45:48.705Z+ [[ -z '' ]]
2832023-12-11T18:45:48.705Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-12-11T18:45:48.753Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-12-11T18:45:48.779Z+ set +x
2862023-12-11T18:45:48.779ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-12-11T18:45:48.779Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-12-11T18:45:48.779Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-12-11T18:45:48.779ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-12-11T18:45:48.779Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-12-11T18:45:48.779Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-12-11T18:45:48.779Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-12-11T18:45:48.779Z0+0 records in
2942023-12-11T18:45:48.779Z0+0 records out
2952023-12-11T18:45:48.779Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-12-11T18:45:48.779Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-12-11T18:45:48.779Z+ set +x
2982023-12-11T18:45:48.779ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-12-11T18:45:48.780Z++ zpool list -o name
3002023-12-11T18:45:48.780Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-12-11T18:45:48.780Z+ [[ -z '' ]]
3022023-12-11T18:45:48.780Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-12-11T18:45:48.804Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-12-11T18:45:48.804Z+ set +x
3052023-12-11T18:45:48.805ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-12-11T18:45:48.805Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-12-11T18:45:48.805ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3082023-12-11T18:45:48.805Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3092023-12-11T18:45:48.805Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-12-11T18:45:48.805Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-12-11T18:45:48.805Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-12-11T18:45:48.806Z0+0 records in
3132023-12-11T18:45:48.806Z0+0 records out
3142023-12-11T18:45:48.806Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3152023-12-11T18:45:48.806Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-12-11T18:45:48.806Z+ set +x
3172023-12-11T18:45:48.806ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-12-11T18:45:48.830Z++ zpool list -o name
3192023-12-11T18:45:48.830Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-12-11T18:45:48.830Z+ [[ -z '' ]]
3212023-12-11T18:45:48.830Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-12-11T18:45:48.858Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-12-11T18:45:48.859Z+ set +x
3242023-12-11T18:45:48.859ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-12-11T18:45:48.859Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-12-11T18:45:48.859Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-12-11T18:45:48.859ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-12-11T18:45:48.860Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-12-11T18:45:48.860Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-12-11T18:45:48.860Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-12-11T18:45:48.860Z0+0 records in
3322023-12-11T18:45:48.860Z0+0 records out
3332023-12-11T18:45:48.860Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-12-11T18:45:48.860Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-12-11T18:45:48.861Z+ set +x
3362023-12-11T18:45:48.861ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-12-11T18:45:48.889Z++ zpool list -o name
3382023-12-11T18:45:48.889Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-12-11T18:45:48.889Z+ [[ -z '' ]]
3402023-12-11T18:45:48.889Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-12-11T18:45:48.915Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-12-11T18:45:48.915Z+ set +x
3432023-12-11T18:45:48.915ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-12-11T18:45:48.915Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-12-11T18:45:48.915Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-12-11T18:45:48.915ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-12-11T18:45:48.915Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-12-11T18:45:48.915Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-12-11T18:45:48.916Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-12-11T18:45:48.916Z0+0 records in
3512023-12-11T18:45:48.916Z0+0 records out
3522023-12-11T18:45:48.916Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3532023-12-11T18:45:48.916Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-12-11T18:45:48.916Z+ set +x
3552023-12-11T18:45:48.916ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-12-11T18:45:48.916Z++ zpool list -o name
3572023-12-11T18:45:48.916Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-12-11T18:45:48.942Z+ [[ -z '' ]]
3592023-12-11T18:45:48.942Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-12-11T18:45:48.968Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-12-11T18:45:48.968Z+ set +x
3622023-12-11T18:45:48.968ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-12-11T18:45:48.968Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-12-11T18:45:48.968Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-12-11T18:45:48.968ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-12-11T18:45:48.969Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-12-11T18:45:48.969Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-12-11T18:45:48.969Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-12-11T18:45:48.969Z0+0 records in
3702023-12-11T18:45:48.969Z0+0 records out
3712023-12-11T18:45:48.969Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3722023-12-11T18:45:48.969Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-12-11T18:45:48.969Z+ set +x
3742023-12-11T18:45:48.969ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-12-11T18:45:48.969Z++ zpool list -o name
3762023-12-11T18:45:48.970Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-12-11T18:45:48.994Z+ [[ -z '' ]]
3782023-12-11T18:45:48.994Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-12-11T18:45:49.019Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-12-11T18:45:49.019Z+ set +x
3812023-12-11T18:45:49.019ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-12-11T18:45:49.019Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-12-11T18:45:49.019Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-12-11T18:45:49.019Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3852023-12-11T18:45:49.019Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3862023-12-11T18:45:49.020ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3872023-12-11T18:45:49.020Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-12-11T18:45:49.020Z0+0 records in
3892023-12-11T18:45:49.020Z0+0 records out
3902023-12-11T18:45:49.020Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-12-11T18:45:49.020Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-12-11T18:45:49.020Z+ set +x
3932023-12-11T18:45:49.020ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-12-11T18:45:49.020Z++ zpool list -o name
3952023-12-11T18:45:49.020Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-12-11T18:45:49.045Z+ [[ -z '' ]]
3972023-12-11T18:45:49.046Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-12-11T18:45:49.070Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-12-11T18:45:49.070Z+ set +x
4002023-12-11T18:45:49.071ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-12-11T18:45:49.071Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-12-11T18:45:49.071Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-12-11T18:45:49.071ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-12-11T18:45:49.071Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-12-11T18:45:49.071Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-12-11T18:45:49.071Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-12-11T18:45:49.071Z0+0 records in
4082023-12-11T18:45:49.072Z0+0 records out
4092023-12-11T18:45:49.072Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-12-11T18:45:49.072Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-12-11T18:45:49.072Z+ set +x
4122023-12-11T18:45:49.072ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-12-11T18:45:49.072Z++ zpool list -o name
4142023-12-11T18:45:49.072Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-12-11T18:45:49.096Z+ [[ -z '' ]]
4162023-12-11T18:45:49.097Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-12-11T18:45:49.121Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-12-11T18:45:49.121Z+ set +x
4192023-12-11T18:45:49.121ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-12-11T18:45:49.121Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-12-11T18:45:49.121Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-12-11T18:45:49.121ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-12-11T18:45:49.121Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-12-11T18:45:49.121Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-12-11T18:45:49.121Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-12-11T18:45:49.122Z0+0 records in
4272023-12-11T18:45:49.122Z0+0 records out
4282023-12-11T18:45:49.122Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4292023-12-11T18:45:49.122Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-12-11T18:45:49.122Z+ set +x
4312023-12-11T18:45:49.122ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-12-11T18:45:49.146Z++ zpool list -o name
4332023-12-11T18:45:49.147Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-12-11T18:45:49.147Z+ [[ -z '' ]]
4352023-12-11T18:45:49.147Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-12-11T18:45:49.172Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-12-11T18:45:49.172Z+ set +x
4382023-12-11T18:45:49.172ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-12-11T18:45:49.172Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-12-11T18:45:49.172Z+ readarray -t ZPOOLS
4412023-12-11T18:45:49.172Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-12-11T18:45:49.197Z++ sed 's/[ ",]//g'
4432023-12-11T18:45:49.197Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-12-11T18:45:49.197Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-12-11T18:45:49.197ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-12-11T18:45:49.197Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-12-11T18:45:49.197Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-12-11T18:45:49.197Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-12-11T18:45:49.197Z0+0 records in
4502023-12-11T18:45:49.197Z0+0 records out
4512023-12-11T18:45:49.197Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4522023-12-11T18:45:49.197Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-12-11T18:45:49.197Z+ set +x
4542023-12-11T18:45:49.197ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-12-11T18:45:49.197Z++ zpool list -o name
4562023-12-11T18:45:49.198Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-12-11T18:45:49.198Z+ [[ -z '' ]]
4582023-12-11T18:45:49.198Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-12-11T18:45:49.227Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-12-11T18:45:49.227Z+ set +x
4612023-12-11T18:45:49.251ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-12-11T18:45:49.251Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-12-11T18:45:49.251Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-12-11T18:45:49.251ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-12-11T18:45:49.251Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-12-11T18:45:49.251Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-12-11T18:45:49.251Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-12-11T18:45:49.252Z0+0 records in
4692023-12-11T18:45:49.252Z0+0 records out
4702023-12-11T18:45:49.252Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4712023-12-11T18:45:49.252Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-12-11T18:45:49.252Z+ set +x
4732023-12-11T18:45:49.252ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-12-11T18:45:49.252Z++ zpool list -o name
4752023-12-11T18:45:49.252Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-12-11T18:45:49.252Z+ [[ -z '' ]]
4772023-12-11T18:45:49.252Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-12-11T18:45:49.287Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-12-11T18:45:49.287Z+ set +x
4802023-12-11T18:45:49.321ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-12-11T18:45:49.322Z+ [[ zone == \z\o\n\e ]]
4822023-12-11T18:45:49.322Z+ ensure_simulated_links igb0
4832023-12-11T18:45:49.322Z+ local PHYSICAL_LINK=igb0
4842023-12-11T18:45:49.323Z+ INDICES=("0" "1")
4852023-12-11T18:45:49.323Z+ for I in "${INDICES[@]}"
4862023-12-11T18:45:49.323Z++ get_simnet_name_if_exists net0
4872023-12-11T18:45:49.323Z++ dladm show-simnet -p -o LINK net0
4882023-12-11T18:45:49.323Z++ echo ''
4892023-12-11T18:45:49.323Z+ [[ -z '' ]]
4902023-12-11T18:45:49.323Z+ dladm create-simnet -t net0
4912023-12-11T18:45:49.324Z+ dladm create-simnet -t sc0_0
4922023-12-11T18:45:49.324Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-12-11T18:45:49.324Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-12-11T18:45:49.325Z+ success 'Simnet net0/sc0_0 exists'
4952023-12-11T18:45:49.325Z+ set +x
4962023-12-11T18:45:49.325ZSimnet net0/sc0_0 exists
4972023-12-11T18:45:49.325Z+ for I in "${INDICES[@]}"
4982023-12-11T18:45:49.325Z++ get_simnet_name_if_exists net1
4992023-12-11T18:45:49.325Z++ dladm show-simnet -p -o LINK net1
5002023-12-11T18:45:49.325Z++ echo ''
5012023-12-11T18:45:49.325Z+ [[ -z '' ]]
5022023-12-11T18:45:49.325Z+ dladm create-simnet -t net1
5032023-12-11T18:45:49.325Z+ dladm create-simnet -t sc1_0
5042023-12-11T18:45:49.351Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-12-11T18:45:49.352Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-12-11T18:45:49.352Z+ success 'Simnet net1/sc1_0 exists'
5072023-12-11T18:45:49.352Z+ set +x
5082023-12-11T18:45:49.352ZSimnet net1/sc1_0 exists
5092023-12-11T18:45:49.352Z++ get_vnic_name_if_exists sc0_1
5102023-12-11T18:45:49.352Z++ dladm show-vnic -p -o LINK sc0_1
5112023-12-11T18:45:49.352Z++ echo ''
5122023-12-11T18:45:49.352Z+ [[ -z '' ]]
5132023-12-11T18:45:49.352Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-12-11T18:45:49.353Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-12-11T18:45:49.353Z+ success 'Vnic sc0_1 exists'
5162023-12-11T18:45:49.353Z+ set +x
5172023-12-11T18:45:49.353ZVnic sc0_1 exists
5182023-12-11T18:45:49.353Z+ warn_if_no_proxy_arp
5192023-12-11T18:45:49.353Z+ [[ -v PXA_START ]]
5202023-12-11T18:45:49.353Z+ [[ -v PXA_END ]]
5212023-12-11T18:45:49.353Z+ ensure_softnpu_zone
5222023-12-11T18:45:49.353Z+ zoneadm list
5232023-12-11T18:45:49.353Z+ grep -q sidecar_softnpu
5242023-12-11T18:45:49.354Z+ [[ -f out/npuzone/npuzone ]]
5252023-12-11T18:45:49.354Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit 45ed98fea5824feb4d42f45bbf218e597dc9fc58 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5262023-12-11T18:45:49.378Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5272023-12-11T18:45:54.476Zdownloading softnpu ...done
5282023-12-11T18:45:54.500Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/libsidecar_lite.so.sha256.txt
5292023-12-11T18:45:59.029Zdownloading asic_program.so ...done
5302023-12-11T18:45:59.054Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/scadm.sha256.txt
5312023-12-11T18:46:04.517Zdownloading scadm ...done
5322023-12-11T18:46:04.542Zcreating port tfportrear0_0 for sc0_0
5332023-12-11T18:46:04.542Zcreating port tfportqsfp0_0 for sc0_1
5342023-12-11T18:46:04.542Zsoftnpu zone setup
5352023-12-11T18:46:04.542Zconfigure zone
5362023-12-11T18:46:04.566Zinstall zone
5372023-12-11T18:46:05.624Zboot zone
5382023-12-11T18:46:08.733Z[sidecar_softnpu] svcs milestone/network
5392023-12-11T18:46:08.761ZSTATE STIME FMRI
5402023-12-11T18:46:08.761Zonline 18:46:07 svc:/milestone/network:default
5412023-12-11T18:46:08.762Z[sidecar_softnpu] /softnpu/npu start
5422023-12-11T18:46:08.786Z
5432023-12-11T18:46:08.811Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-12-11T18:46:08.811Z+ GATEWAY_IP=10.151.2.100
5452023-12-11T18:46:08.811ZUsing 10.151.2.100 as gateway ip
5462023-12-11T18:46:08.811Z+ echo 'Using 10.151.2.100 as gateway ip'
5472023-12-11T18:46:08.811Z+ [[ ! -v GATEWAY_MAC ]]
5482023-12-11T18:46:08.811Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-12-11T18:46:08.814Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-12-11T18:46:08.814Z+++ echo 18:c0:4d:0d:a0:2a
5512023-12-11T18:46:08.814Z+ COUNT=1
5522023-12-11T18:46:08.814Z+ [[ 1 -ne 1 ]]
5532023-12-11T18:46:08.814Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-12-11T18:46:08.814ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-12-11T18:46:08.814Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-12-11T18:46:08.814Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-12-11T18:46:08.839Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-12-11T18:46:08.839Z+ [[ -v PXA_START ]]
5592023-12-11T18:46:08.839Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-12-11T18:46:08.839Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-12-11T18:46:08.863Z+ z_scadm dump-state
5622023-12-11T18:46:08.864Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-12-11T18:46:08.891Zlocal v6:
5642023-12-11T18:46:08.891Zlocal v4:
5652023-12-11T18:46:08.891Zrouter v6:
5662023-12-11T18:46:08.891Zrouter v4_idx:
5672023-12-11T18:46:08.891Zrouter v4_routes:
5682023-12-11T18:46:08.892Zresolver v4:
5692023-12-11T18:46:08.892Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-12-11T18:46:08.892Zresolver v6:
5712023-12-11T18:46:08.892Znat v4:
5722023-12-11T18:46:08.892Znat_v6:
5732023-12-11T18:46:08.892Zport_mac:
5742023-12-11T18:46:08.892Zproxy arp:
5752023-12-11T18:46:08.892Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-12-11T18:46:08.895Z+ success 'softnpu zone exists'
5772023-12-11T18:46:08.895Z+ set +x
5782023-12-11T18:46:08.895Zsoftnpu zone exists
5792023-12-11T18:46:08.895Z
5802023-12-11T18:46:08.895Zreal 20.305627313
5812023-12-11T18:46:08.895Zuser 0.705856829
5822023-12-11T18:46:08.895Zsys 1.644845606
5832023-12-11T18:46:08.895Ztrap 0.000088390
5842023-12-11T18:46:08.902Ztflt 0.000473193
5852023-12-11T18:46:08.902Zdflt 0.001545993
5862023-12-11T18:46:08.902Zkflt 0.000000000
5872023-12-11T18:46:08.902Zlock 15:31.166948460
5882023-12-11T18:46:08.902Zslp 1:00.776423175
5892023-12-11T18:46:08.902Zlat 0.079495784
5902023-12-11T18:46:08.902Zstop 0.356125122
5912023-12-11T18:46:08.902Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-12-11T18:46:08.928Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-12-11T18:46:08.928Z+ SILO_NAME=recovery
5942023-12-11T18:46:08.928Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-11T18:46:08.928Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-12-11T18:46:08.928Z+ sed -i~ '
5972023-12-11T18:46:08.928Z /^external_dns_ips/c\
5982023-12-11T18:46:08.928Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-12-11T18:46:08.928Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-12-11T18:46:08.928Z /^first/c\
6012023-12-11T18:46:08.928Zfirst = "10.151.2.164"
6022023-12-11T18:46:08.929Z /^last/c\
6032023-12-11T18:46:08.929Zlast = "10.151.2.173"
6042023-12-11T18:46:08.929Z }
6052023-12-11T18:46:08.930Z /^\[rack_network_config/,/^$/ {
6062023-12-11T18:46:08.930Z /^infra_ip_first/c\
6072023-12-11T18:46:08.930Zinfra_ip_first = "10.151.2.174"
6082023-12-11T18:46:08.930Z /^infra_ip_last/c\
6092023-12-11T18:46:08.930Zinfra_ip_last = "10.151.2.174"
6102023-12-11T18:46:08.930Z }
6112023-12-11T18:46:08.930Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-12-11T18:46:08.930Z /^routes/c\
6132023-12-11T18:46:08.930Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6142023-12-11T18:46:08.930Z /^addresses/c\
6152023-12-11T18:46:08.930Zaddresses = \["10.151.2.174/32"\]
6162023-12-11T18:46:08.930Z }
6172023-12-11T18:46:08.931Z' pkg/config-rss.toml
6182023-12-11T18:46:08.931Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-12-11T18:46:08.931Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-12-11T18:46:08.931Z+++ pkg/config-rss.toml Mon Dec 11 18:46:09 2023
6212023-12-11T18:46:08.931Z@@ -41,7 +41,7 @@
6222023-12-11T18:46:08.931Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-12-11T18:46:08.931Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-12-11T18:46:08.931Z # below.
6252023-12-11T18:46:08.931Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-12-11T18:46:08.931Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6272023-12-11T18:46:08.931Z
6282023-12-11T18:46:08.931Z # Initial TLS certificates for the external API
6292023-12-11T18:46:08.931Z #
6302023-12-11T18:46:08.931Z@@ -77,8 +77,8 @@
6312023-12-11T18:46:08.931Z #
6322023-12-11T18:46:08.931Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-12-11T18:46:08.931Z [[internal_services_ip_pool_ranges]]
6342023-12-11T18:46:08.931Z-first = "192.168.1.20"
6352023-12-11T18:46:08.932Z-last = "192.168.1.29"
6362023-12-11T18:46:08.932Z+first = "10.151.2.164"
6372023-12-11T18:46:08.932Z+last = "10.151.2.173"
6382023-12-11T18:46:08.932Z
6392023-12-11T18:46:08.932Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-12-11T18:46:08.932Z # parameters should be provided to the control plane via wicket, but we need to
6412023-12-11T18:46:08.932Z@@ -92,8 +92,8 @@
6422023-12-11T18:46:08.932Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-12-11T18:46:08.932Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-12-11T18:46:08.932Z # With softnpu, only one address is used.
6452023-12-11T18:46:08.932Z-infra_ip_first = "192.168.1.30"
6462023-12-11T18:46:08.932Z-infra_ip_last = "192.168.1.30"
6472023-12-11T18:46:08.932Z+infra_ip_first = "10.151.2.174"
6482023-12-11T18:46:08.932Z+infra_ip_last = "10.151.2.174"
6492023-12-11T18:46:08.932Z
6502023-12-11T18:46:08.932Z # Configurations for BGP routers to run on the scrimlets.
6512023-12-11T18:46:08.932Z bgp = []
6522023-12-11T18:46:08.932Z@@ -101,9 +101,9 @@
6532023-12-11T18:46:08.932Z # You can configure multiple uplinks by repeating the following stanza
6542023-12-11T18:46:08.933Z [[rack_network_config.ports]]
6552023-12-11T18:46:08.933Z # Routes associated with this port.
6562023-12-11T18:46:08.933Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-12-11T18:46:08.933Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6582023-12-11T18:46:08.933Z # Addresses associated with this port.
6592023-12-11T18:46:08.935Z-addresses = ["192.168.1.30/32"]
6602023-12-11T18:46:08.935Z+addresses = ["10.151.2.174/32"]
6612023-12-11T18:46:08.935Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-12-11T18:46:08.935Z port = "qsfp0"
6632023-12-11T18:46:08.935Z # The speed of this port.
6642023-12-11T18:46:08.935Z+ true
6652023-12-11T18:46:08.936Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-12-11T18:46:08.936Za pkg/config-rss.toml 7K
6672023-12-11T18:46:08.936Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-12-11T18:46:08.960Z+ TLS_NAME=recovery.sys.oxide.test
6692023-12-11T18:46:08.960Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-12-11T18:46:09.164Z.....+.....+.........+......+.+........+.+...+...............+...+..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.+....................+...+......+.+..............+.+...+...+.....+...+....+...+...+...........+...+.+.....+....+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+......+...+............+........+....+..+.+............+..+......+...+....+......+......+..............+.........+.+...........+.+..+..................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-12-11T18:46:09.747Z..........+.........+...+..+.+........+.........+...+...+.+.....+...+......+.+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+......+....+..+.........+...+.+...........+......+...+.+.........+......+...+.....+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.....+.+........+.......+........+.......+..............+.+........+......+.+.........+.....+...+.+..+...............+.........+.+.................+.......+...+........+...+.+..................+..+..........+..............+.........+......+.+...+..+.+..+.......+...............+.....+.........+...................+...+.................+..........+...+........+.........+.......+.....+................+.....+...+....+..+....+...........+.+.........+...........+............+.......+...............+.....+....+..+.........+...+.......+...+......+.........+.....+...+.......+......+.....+.........+.+.........+...+.........+........+.+..+.........+..........+.....+.............+..+.............+.....+.+.........+.....+...+....+...............+........+.+...........+.........+.......+...+..+..........+...+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-11T18:46:09.773Z-----
6732023-12-11T18:46:09.773Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-12-11T18:46:09.797Za pkg/initial-tls-cert.pem 2K
6752023-12-11T18:46:09.797Za pkg/initial-tls-key.pem 4K
6762023-12-11T18:46:09.797Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-12-11T18:46:09.797Z+ rmdir pkg
6782023-12-11T18:46:09.797Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-12-11T18:46:09.797Z+ pfexec mkdir -p /usr/oxide
6802023-12-11T18:46:09.798Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6812023-12-11T18:46:10.527Z+ pfexec chmod +x /usr/oxide/oxide
6822023-12-11T18:46:10.555Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6832023-12-11T18:46:28.800Z+ OMICRON_NO_UNINSTALL=1
6842023-12-11T18:46:28.801Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6852023-12-11T18:46:28.825ZDec 11 18:46:29.301 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6862023-12-11T18:46:28.825ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6872023-12-11T18:46:28.825ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6882023-12-11T18:46:28.825ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6892023-12-11T18:46:28.825ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6902023-12-11T18:46:28.825ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6912023-12-11T18:46:28.826ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6922023-12-11T18:46:28.826ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6932023-12-11T18:46:28.826ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6942023-12-11T18:46:28.826ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6952023-12-11T18:46:28.826ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6962023-12-11T18:46:28.826ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6972023-12-11T18:46:28.826ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6982023-12-11T18:46:28.826ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6992023-12-11T18:46:28.827ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7002023-12-11T18:46:28.827ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7012023-12-11T18:46:28.827ZDec 11 18:46:29.304 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7022023-12-11T18:46:29.049ZDec 11 18:46:29.535 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7032023-12-11T18:46:29.085ZDec 11 18:46:29.571 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7042023-12-11T18:46:29.145ZDec 11 18:46:29.631 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7052023-12-11T18:46:29.274Z
7062023-12-11T18:46:29.299Zreal 0.468478464
7072023-12-11T18:46:29.299Zuser 0.098381659
7082023-12-11T18:46:29.299Zsys 1.833116981
7092023-12-11T18:46:29.299Ztrap 0.000000500
7102023-12-11T18:46:29.299Ztflt 0.000047090
7112023-12-11T18:46:29.299Zdflt 0.000937596
7122023-12-11T18:46:29.299Zkflt 0.000003120
7132023-12-11T18:46:29.299Zlock 42.057567943
7142023-12-11T18:46:29.299Zslp 0.679760433
7152023-12-11T18:46:29.299Zlat 0.003043049
7162023-12-11T18:46:29.299Zstop 0.220242703
7172023-12-11T18:46:29.300Z+ retry=0
7182023-12-11T18:46:29.300Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7192023-12-11T18:46:29.300Z+ [[ 0 -gt 30 ]]
7202023-12-11T18:46:29.300Z+ sleep 1
7212023-12-11T18:46:30.288Z+ retry=1
7222023-12-11T18:46:30.288Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232023-12-11T18:46:30.313Z+ [[ 1 -gt 30 ]]
7242023-12-11T18:46:30.313Z+ sleep 1
7252023-12-11T18:46:31.303Z+ retry=2
7262023-12-11T18:46:31.304Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272023-12-11T18:46:31.332Z+ [[ 2 -gt 30 ]]
7282023-12-11T18:46:31.332Z+ sleep 1
7292023-12-11T18:46:32.317Z+ retry=3
7302023-12-11T18:46:32.317Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312023-12-11T18:46:32.342Z+ [[ 3 -gt 30 ]]
7322023-12-11T18:46:32.342Z+ sleep 1
7332023-12-11T18:46:33.330Z+ retry=4
7342023-12-11T18:46:33.331Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352023-12-11T18:46:58.505Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7362023-12-11T18:46:58.533Zlocal v6:
7372023-12-11T18:46:58.534Zfe80::c036:ff:fe9f:e345
7382023-12-11T18:46:58.534Zfd00:99::1
7392023-12-11T18:46:58.534Zlocal v4:
7402023-12-11T18:46:58.534Z10.151.2.174
7412023-12-11T18:46:58.534Zrouter v6:
7422023-12-11T18:46:58.534Zfdb0:18c0:4d0d:a02a::/64 -> fe80::acb8:d0ff:fea7:c8bf (1)
7432023-12-11T18:46:58.534Zfd00:1122:3344:101::/64 -> fe80::acb8:d0ff:fea7:c8bf (1)
7442023-12-11T18:46:58.534Zfd00:1122:3344:2::/64 -> fe80::acb8:d0ff:fea7:c8bf (1)
7452023-12-11T18:46:58.534Zfd00:1122:3344:3::/64 -> fe80::acb8:d0ff:fea7:c8bf (1)
7462023-12-11T18:46:58.534Zfd00:1122:3344:1::/64 -> fe80::acb8:d0ff:fea7:c8bf (1)
7472023-12-11T18:46:58.534Zrouter v4_idx:
7482023-12-11T18:46:58.559Z0.0.0.0/0 -> 0
7492023-12-11T18:46:58.559Zrouter v4_routes:
7502023-12-11T18:46:58.559Z0 -> 10.151.2.100 (2)
7512023-12-11T18:46:58.559Zresolver v4:
7522023-12-11T18:46:58.559Z10.151.2.174 -> 0a:94:cc:04:73:42
7532023-12-11T18:46:58.559Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7542023-12-11T18:46:58.559Zresolver v6:
7552023-12-11T18:46:58.559Zfe80::acb8:d0ff:fea7:c8bf -> ae:b8:d0:a7:c8:bf
7562023-12-11T18:46:58.559Zfe80::c036:ff:fe9f:e345 -> c2:36:00:9f:e3:45
7572023-12-11T18:46:58.559Znat v4:
7582023-12-11T18:46:58.559Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:c7:65
7592023-12-11T18:46:58.560Znat_v6:
7602023-12-11T18:46:58.560Zport_mac:
7612023-12-11T18:46:58.560Z1: a8:40:25:16:92:99
7622023-12-11T18:46:58.560Z1000: a8:40:25:16:92:98
7632023-12-11T18:46:58.560Zproxy arp:
7642023-12-11T18:46:58.560Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7652023-12-11T18:46:58.560Z+ retry=0
7662023-12-11T18:46:58.560Z+++ zoneadm list -n
7672023-12-11T18:46:58.560Z+++ grep oxz_ntp
7682023-12-11T18:46:58.560Z++ pfexec svcs -z oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c -Hostate oxide/ntp
7692023-12-11T18:46:58.584Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7702023-12-11T18:46:58.591Z++ true
7712023-12-11T18:46:58.591Z+ [[ '' != online ]]
7722023-12-11T18:46:58.591Z+ [[ 0 -gt 60 ]]
7732023-12-11T18:46:58.591Z+ sleep 1
7742023-12-11T18:46:59.574Z+ retry=1
7752023-12-11T18:46:59.598Z+++ zoneadm list -n
7762023-12-11T18:46:59.598Z+++ grep oxz_ntp
7772023-12-11T18:46:59.598Z++ pfexec svcs -z oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c -Hostate oxide/ntp
7782023-12-11T18:46:59.622Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7792023-12-11T18:46:59.623Z++ true
7802023-12-11T18:46:59.623Z+ [[ '' != online ]]
7812023-12-11T18:46:59.623Z+ [[ 1 -gt 60 ]]
7822023-12-11T18:46:59.623Z+ sleep 1
7832023-12-11T18:47:00.605Z+ retry=2
7842023-12-11T18:47:00.630Z+++ zoneadm list -n
7852023-12-11T18:47:00.630Z+++ grep oxz_ntp
7862023-12-11T18:47:00.630Z++ pfexec svcs -z oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c -Hostate oxide/ntp
7872023-12-11T18:47:00.630Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7882023-12-11T18:47:00.657Z++ true
7892023-12-11T18:47:00.658Z+ [[ '' != online ]]
7902023-12-11T18:47:00.658Z+ [[ 2 -gt 60 ]]
7912023-12-11T18:47:00.658Z+ sleep 1
7922023-12-11T18:47:01.634Z+ retry=3
7932023-12-11T18:47:01.659Z+++ zoneadm list -n
7942023-12-11T18:47:01.659Z+++ grep oxz_ntp
7952023-12-11T18:47:01.660Z++ pfexec svcs -z oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c -Hostate oxide/ntp
7962023-12-11T18:47:01.684Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7972023-12-11T18:47:01.684Z++ true
7982023-12-11T18:47:01.684Z+ [[ '' != online ]]
7992023-12-11T18:47:01.684Z+ [[ 3 -gt 60 ]]
8002023-12-11T18:47:01.685Z+ sleep 1
8012023-12-11T18:47:02.664Z+ retry=4
8022023-12-11T18:47:02.690Z+++ zoneadm list -n
8032023-12-11T18:47:02.690Z+++ grep oxz_ntp
8042023-12-11T18:47:02.690Z++ pfexec svcs -z oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c -Hostate oxide/ntp
8052023-12-11T18:47:02.690Z+ [[ disabled != online ]]
8062023-12-11T18:47:02.690Z+ [[ 4 -gt 60 ]]
8072023-12-11T18:47:02.690Z+ sleep 1
8082023-12-11T18:47:03.688Z+ retry=5
8092023-12-11T18:47:03.714Z+++ zoneadm list -n
8102023-12-11T18:47:03.714Z+++ grep oxz_ntp
8112023-12-11T18:47:03.714Z++ pfexec svcs -z oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c -Hostate oxide/ntp
8122023-12-11T18:47:03.714Z+ [[ disabled != online ]]
8132023-12-11T18:47:03.715Z+ [[ 5 -gt 60 ]]
8142023-12-11T18:47:03.715Z+ sleep 1
8152023-12-11T18:47:04.709Z+ retry=6
8162023-12-11T18:47:04.734Z+++ zoneadm list -n
8172023-12-11T18:47:04.735Z+++ grep oxz_ntp
8182023-12-11T18:47:04.735Z++ pfexec svcs -z oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c -Hostate oxide/ntp
8192023-12-11T18:47:04.735Z+ [[ disabled != online ]]
8202023-12-11T18:47:04.735Z+ [[ 6 -gt 60 ]]
8212023-12-11T18:47:04.735Z+ sleep 1
8222023-12-11T18:47:05.731Z+ retry=7
8232023-12-11T18:47:05.756Z+++ zoneadm list -n
8242023-12-11T18:47:05.756Z+++ grep oxz_ntp
8252023-12-11T18:47:05.756Z++ pfexec svcs -z oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c -Hostate oxide/ntp
8262023-12-11T18:47:05.756Z+ [[ disabled != online ]]
8272023-12-11T18:47:05.756Z+ [[ 7 -gt 60 ]]
8282023-12-11T18:47:05.756Z+ sleep 1
8292023-12-11T18:47:06.753Z+ retry=8
8302023-12-11T18:47:06.777Z+++ zoneadm list -n
8312023-12-11T18:47:06.777Z+++ grep oxz_ntp
8322023-12-11T18:47:06.777Z++ pfexec svcs -z oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c -Hostate oxide/ntp
8332023-12-11T18:47:06.777Z+ [[ disabled != online ]]
8342023-12-11T18:47:06.777Z+ [[ 8 -gt 60 ]]
8352023-12-11T18:47:06.777Z+ sleep 1
8362023-12-11T18:47:07.774Z+ retry=9
8372023-12-11T18:47:07.799Z+++ zoneadm list -n
8382023-12-11T18:47:07.799Z+++ grep oxz_ntp
8392023-12-11T18:47:07.799Z++ pfexec svcs -z oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c -Hostate oxide/ntp
8402023-12-11T18:47:07.799Z+ [[ disabled != online ]]
8412023-12-11T18:47:07.799Z+ [[ 9 -gt 60 ]]
8422023-12-11T18:47:07.799Z+ sleep 1
8432023-12-11T18:47:08.797Z+ retry=10
8442023-12-11T18:47:08.822Z+++ zoneadm list -n
8452023-12-11T18:47:08.822Z+++ grep oxz_ntp
8462023-12-11T18:47:08.822Z++ pfexec svcs -z oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c -Hostate oxide/ntp
8472023-12-11T18:47:08.822Z+ [[ disabled != online ]]
8482023-12-11T18:47:08.822Z+ [[ 10 -gt 60 ]]
8492023-12-11T18:47:08.822Z+ sleep 1
 
8502023-12-11T18:47:09.468Zsock console login: Dec 11 18:47:09 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8512023-12-11T18:47:09.819Z+ retry=11
8522023-12-11T18:47:09.843Z+++ zoneadm list -n
8532023-12-11T18:47:09.843Z+++ grep oxz_ntp
8542023-12-11T18:47:09.844Z++ pfexec svcs -z oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c -Hostate oxide/ntp
8552023-12-11T18:47:09.844Z+ [[ online != online ]]
8562023-12-11T18:47:09.844Z+ echo 'Waited for chrony: 11s'
8572023-12-11T18:47:09.844Z+ export RUST_BACKTRACE=1
8582023-12-11T18:47:09.844ZWaited for chrony: 11s
8592023-12-11T18:47:09.844Z+ RUST_BACKTRACE=1
8602023-12-11T18:47:09.844Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8612023-12-11T18:47:09.844Z++ ./tests/bootstrap
8622023-12-11T18:47:09.868ZUsing external DNS server at 10.151.2.164:53
8632023-12-11T18:47:09.869Z2023-12-11 18:47:10.355422710 UTC: attempting to log into API
8642023-12-11T18:47:24.905Z2023-12-11 18:47:24.829041817 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8652023-12-11T18:47:25.906Z2023-12-11 18:47:25.830048737 UTC: attempting to log into API
8662023-12-11T18:47:40.943Z2023-12-11 18:47:40.862764431 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8672023-12-11T18:47:42.440Z2023-12-11 18:47:42.355209092 UTC: attempting to log into API
8682023-12-11T18:47:57.619Z2023-12-11 18:47:57.559947824 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8692023-12-11T18:47:58.620Z2023-12-11 18:47:58.564466410 UTC: attempting to log into API
8702023-12-11T18:48:13.653Z2023-12-11 18:48:13.637594119 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8712023-12-11T18:48:14.654Z2023-12-11 18:48:14.641326252 UTC: attempting to log into API
8722023-12-11T18:48:29.686Z2023-12-11 18:48:29.676944495 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
8732023-12-11T18:48:29.907ZDec 11 18:48:29 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
8742023-12-11T18:48:30.687Z2023-12-11 18:48:30.677969675 UTC: attempting to log into API
 
8752023-12-11T18:48:37.653ZDec 11 18:48:36 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
8762023-12-11T18:48:44.838ZDec 11 18:48:44 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
8772023-12-11T18:48:45.720Z2023-12-11 18:48:45.711247033 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8782023-12-11T18:48:46.722Z2023-12-11 18:48:46.711524365 UTC: attempting to log into API
 
8792023-12-11T18:48:52.049ZDec 11 18:48:51 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
8802023-12-11T18:48:57.656ZDec 11 18:48:57 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
8812023-12-11T18:49:01.840Z2023-12-11 18:49:01.744776806 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8822023-12-11T18:49:02.758Z2023-12-11 18:49:02.746623825 UTC: attempting to log into API
8832023-12-11T18:49:17.791Z2023-12-11 18:49:17.779397049 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8842023-12-11T18:49:19.205Z2023-12-11 18:49:18.780348407 UTC: attempting to log into API
8852023-12-11T18:49:29.591Z2023-12-11 18:49:29.580256476 UTC: login succeeded
8862023-12-11T18:49:29.624Zwaiting for nexus to come up...
8872023-12-11T18:49:29.692Zcreating IP pool... 10.151.2.175 - 10.151.2.227
8882023-12-11T18:49:29.735Zensuring datasets are ready...
8892023-12-11T18:49:32.479Zlet's roll.
8902023-12-11T18:49:32.503Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
8912023-12-11T18:49:32.504ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
8922023-12-11T18:49:32.504ZOXIDE_TOKEN=oxide-token-d419296047012f9becdf5b80d83355a02c22955b'
8932023-12-11T18:49:32.504Z++ OXIDE_HOST=https://recovery.sys.oxide.test
8942023-12-11T18:49:32.504Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
8952023-12-11T18:49:32.504Z++ OXIDE_TOKEN=oxide-token-d419296047012f9becdf5b80d83355a02c22955b
8962023-12-11T18:49:32.504Z+ export OXIDE_HOST OXIDE_TOKEN
8972023-12-11T18:49:32.504Z+ retry=0
8982023-12-11T18:49:32.504Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8992023-12-11T18:49:32.573Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
9002023-12-11T18:49:32.995Z{"status":"ok"}success
9012023-12-11T18:49:32.996ZProject {
9022023-12-11T18:49:32.996Z description: "some images",
9032023-12-11T18:49:32.997Z id: 1ac3c951-c4c8-4c91-9623-095cf29c9593,
9042023-12-11T18:49:32.997Z name: Name(
9052023-12-11T18:49:32.997Z "images",
9062023-12-11T18:49:32.997Z ),
9072023-12-11T18:49:33.027Z time_created: 2023-12-11T18:49:32.631608Z,
9082023-12-11T18:49:33.027Z time_modified: 2023-12-11T18:49:32.631608Z,
9092023-12-11T18:49:33.027Z}
9102023-12-11T18:49:33.028Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
9112023-12-11T18:55:15.907Zdone!
9122023-12-11T18:55:15.908Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
9132023-12-11T18:55:15.908Zsuccess
9142023-12-11T18:55:15.908ZImage {
9152023-12-11T18:55:15.908Z block_size: ByteCount(
9162023-12-11T18:55:15.908Z 512,
9172023-12-11T18:55:15.908Z ),
9182023-12-11T18:55:15.908Z description: "debian 11 original base image",
9192023-12-11T18:55:15.908Z digest: None,
9202023-12-11T18:55:15.908Z id: 20acedb8-3618-42f1-bdd9-2e398d7abd2b,
9212023-12-11T18:55:15.909Z name: Name(
9222023-12-11T18:55:15.909Z "debian11",
9232023-12-11T18:55:15.909Z ),
9242023-12-11T18:55:15.909Z os: "debian",
9252023-12-11T18:55:15.909Z project_id: None,
9262023-12-11T18:55:15.909Z size: ByteCount(
9272023-12-11T18:55:15.909Z 2147483648,
9282023-12-11T18:55:15.909Z ),
9292023-12-11T18:55:15.910Z time_created: 2023-12-11T18:54:59.683874Z,
9302023-12-11T18:55:15.910Z time_modified: 2023-12-11T18:54:59.931862Z,
9312023-12-11T18:55:15.910Z url: None,
9322023-12-11T18:55:15.910Z version: "11",
9332023-12-11T18:55:15.910Z}
9342023-12-11T18:55:15.910Z+ rm ./tests/bootstrap
9352023-12-11T18:55:15.910Z+ for test_bin in tests/*
9362023-12-11T18:55:15.910Z+ ./tests/bootstrap-8898bd4b8ded3394
9372023-12-11T18:55:15.910Z
9382023-12-11T18:55:15.910Zrunning 0 tests
9392023-12-11T18:55:15.910Z
9402023-12-11T18:55:15.910Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9412023-12-11T18:55:15.911Z
9422023-12-11T18:55:15.911Z+ for test_bin in tests/*
9432023-12-11T18:55:15.911Z+ ./tests/end_to_end_tests-76081d63fda74637
9442023-12-11T18:55:15.911Z
9452023-12-11T18:55:15.911Zrunning 2 tests
9462023-12-11T18:55:15.911Ztest no_spoof::no_spoof_here ... ok
9472023-12-11T18:55:33.468Ztest instance_launch::instance_launch ... ok
9482023-12-11T18:55:33.468Z
9492023-12-11T18:55:33.468Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 33.39s
9502023-12-11T18:55:33.468Z
9512023-12-11T18:55:33.493Z+ _exit_trap
9522023-12-11T18:55:33.493Z+ local status=0
9532023-12-11T18:55:33.493Z+ [[ 0 -eq 0 ]]
9542023-12-11T18:55:33.493Z+ exit 0
9552023-12-11T18:55:33.494Zprocess exited: duration 592935 ms, exit code 0
 
9562023-12-11T18:55:33.632Zfound 56 output files
9572023-12-11T18:55:33.632Zuploading: /var/svc/log/oxide-sled-agent:default.log (748199 bytes)
9582023-12-11T18:55:35.667Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9592023-12-11T18:55:35.667Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b5430a46-177b-4d96-8b94-95f0af63065e/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
9602023-12-11T18:55:36.806Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b5430a46-177b-4d96-8b94-95f0af63065e/root/var/svc/log/oxide-crucible-agent:default.log
9612023-12-11T18:55:36.806Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b5430a46-177b-4d96-8b94-95f0af63065e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9622023-12-11T18:55:37.901Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b5430a46-177b-4d96-8b94-95f0af63065e/root/var/svc/log/oxide-crucible-downstairs:default.log
9632023-12-11T18:55:37.902Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b5430a46-177b-4d96-8b94-95f0af63065e/root/var/svc/log/oxide-crucible-downstairs:downstairs-6beab47c-aab0-47b3-9614-30f596e1f9c9.log (5537 bytes)
9642023-12-11T18:55:39.844Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b5430a46-177b-4d96-8b94-95f0af63065e/root/var/svc/log/oxide-crucible-downstairs:downstairs-6beab47c-aab0-47b3-9614-30f596e1f9c9.log
9652023-12-11T18:55:39.844Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_e7506722-e61d-49f1-83f3-7d2048591da3/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9662023-12-11T18:55:40.945Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_e7506722-e61d-49f1-83f3-7d2048591da3/root/var/svc/log/oxide-crucible-pantry:default.log
9672023-12-11T18:55:40.945Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_901bb9dd-000c-44f6-bc94-0359be972ae4/root/var/svc/log/oxide-external_dns:default.log (26943 bytes)
9682023-12-11T18:55:42.132Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_901bb9dd-000c-44f6-bc94-0359be972ae4/root/var/svc/log/oxide-external_dns:default.log
9692023-12-11T18:55:42.132Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_980e3db8-3c54-4bc8-9afd-a3f1c220fbaf/root/var/svc/log/oxide-nexus:default.log (884591 bytes)
9702023-12-11T18:55:43.261Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_980e3db8-3c54-4bc8-9afd-a3f1c220fbaf/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 884591 -> 885198
9712023-12-11T18:55:44.334Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_980e3db8-3c54-4bc8-9afd-a3f1c220fbaf/root/var/svc/log/oxide-nexus:default.log
9722023-12-11T18:55:44.334Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_99a002d9-5a8f-4e13-95c6-d38321841b7d/root/var/svc/log/oxide-nexus:default.log (769071 bytes)
9732023-12-11T18:55:45.292Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_99a002d9-5a8f-4e13-95c6-d38321841b7d/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 769071 -> 769679
9742023-12-11T18:55:46.363Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_99a002d9-5a8f-4e13-95c6-d38321841b7d/root/var/svc/log/oxide-nexus:default.log
9752023-12-11T18:55:46.364Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c/root/var/svc/log/oxide-ntp:default.log (1383 bytes)
9762023-12-11T18:55:47.459Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c/root/var/svc/log/oxide-ntp:default.log
9772023-12-11T18:55:47.459Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_30d596ff-e164-4d2c-9529-fdd70e26a7d0/root/var/svc/log/oxide-clickhouse:default.log (826943 bytes)
9782023-12-11T18:55:48.318Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_30d596ff-e164-4d2c-9529-fdd70e26a7d0/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 826943 -> 850949
9792023-12-11T18:55:49.389Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_30d596ff-e164-4d2c-9529-fdd70e26a7d0/root/var/svc/log/oxide-clickhouse:default.log
9802023-12-11T18:55:49.389Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_ad2c1469-e227-450e-8b24-2599848687cc/root/var/svc/log/oxide-cockroachdb:default.log (5413 bytes)
9812023-12-11T18:55:50.486Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_ad2c1469-e227-450e-8b24-2599848687cc/root/var/svc/log/oxide-cockroachdb:default.log
9822023-12-11T18:55:50.519Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bfe4a4d8-cade-49ec-a2e0-fb90bb53ab79/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9832023-12-11T18:55:51.603Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bfe4a4d8-cade-49ec-a2e0-fb90bb53ab79/root/var/svc/log/oxide-crucible-agent:default.log
9842023-12-11T18:55:51.603Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bfe4a4d8-cade-49ec-a2e0-fb90bb53ab79/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9852023-12-11T18:55:52.740Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bfe4a4d8-cade-49ec-a2e0-fb90bb53ab79/root/var/svc/log/oxide-crucible-downstairs:default.log
9862023-12-11T18:55:52.740Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c2098b5c-e480-44ba-842f-4f1e20aea505/root/var/svc/log/oxide-crucible-agent:default.log (28258 bytes)
9872023-12-11T18:55:53.880Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c2098b5c-e480-44ba-842f-4f1e20aea505/root/var/svc/log/oxide-crucible-agent:default.log
9882023-12-11T18:55:53.880Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c2098b5c-e480-44ba-842f-4f1e20aea505/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9892023-12-11T18:55:54.976Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c2098b5c-e480-44ba-842f-4f1e20aea505/root/var/svc/log/oxide-crucible-downstairs:default.log
9902023-12-11T18:55:54.976Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c2098b5c-e480-44ba-842f-4f1e20aea505/root/var/svc/log/oxide-crucible-downstairs:downstairs-972c4021-f3ab-44db-b335-c26e403b7e27.log (5537 bytes)
9912023-12-11T18:55:56.071Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c2098b5c-e480-44ba-842f-4f1e20aea505/root/var/svc/log/oxide-crucible-downstairs:downstairs-972c4021-f3ab-44db-b335-c26e403b7e27.log
9922023-12-11T18:55:56.071Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c2098b5c-e480-44ba-842f-4f1e20aea505/root/var/svc/log/oxide-crucible-downstairs:downstairs-b0a3780e-689f-4ea9-bcbd-af5e5de60841.log (6436 bytes)
9932023-12-11T18:55:57.165Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c2098b5c-e480-44ba-842f-4f1e20aea505/root/var/svc/log/oxide-crucible-downstairs:downstairs-b0a3780e-689f-4ea9-bcbd-af5e5de60841.log
9942023-12-11T18:55:57.166Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c2098b5c-e480-44ba-842f-4f1e20aea505/root/var/svc/log/oxide-crucible-downstairs:snapshot-b0a3780e-689f-4ea9-bcbd-af5e5de60841-10d2c586-80aa-4e97-ab5c-713c478e9e4c.log (5175 bytes)
9952023-12-11T18:55:58.268Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c2098b5c-e480-44ba-842f-4f1e20aea505/root/var/svc/log/oxide-crucible-downstairs:snapshot-b0a3780e-689f-4ea9-bcbd-af5e5de60841-10d2c586-80aa-4e97-ab5c-713c478e9e4c.log
9962023-12-11T18:55:58.268Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f50cdd25-ff5d-4df7-b45c-e80ee72485a7/root/var/svc/log/oxide-crucible-agent:default.log (30293 bytes)
9972023-12-11T18:55:59.418Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f50cdd25-ff5d-4df7-b45c-e80ee72485a7/root/var/svc/log/oxide-crucible-agent:default.log
9982023-12-11T18:55:59.418Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f50cdd25-ff5d-4df7-b45c-e80ee72485a7/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9992023-12-11T18:56:00.521Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f50cdd25-ff5d-4df7-b45c-e80ee72485a7/root/var/svc/log/oxide-crucible-downstairs:default.log
10002023-12-11T18:56:00.521Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f50cdd25-ff5d-4df7-b45c-e80ee72485a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-57134dce-1fa1-49cc-ab0a-053366523465.log (2944 bytes)
10012023-12-11T18:56:01.617Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f50cdd25-ff5d-4df7-b45c-e80ee72485a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-57134dce-1fa1-49cc-ab0a-053366523465.log
10022023-12-11T18:56:01.617Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f50cdd25-ff5d-4df7-b45c-e80ee72485a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-cdf761a9-4ae2-4d34-ad52-2338fcb7c99e.log (3051 bytes)
10032023-12-11T18:56:03.126Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f50cdd25-ff5d-4df7-b45c-e80ee72485a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-cdf761a9-4ae2-4d34-ad52-2338fcb7c99e.log
10042023-12-11T18:56:03.126Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f50cdd25-ff5d-4df7-b45c-e80ee72485a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-faf96056-b30d-4585-8e64-e76c69b4f3ee.log (5536 bytes)
10052023-12-11T18:56:04.224Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f50cdd25-ff5d-4df7-b45c-e80ee72485a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-faf96056-b30d-4585-8e64-e76c69b4f3ee.log
10062023-12-11T18:56:04.225Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_ba7ae14a-9687-4cd8-b97a-9174d9a7c5ef/root/var/svc/log/oxide-cockroachdb:default.log (5413 bytes)
10072023-12-11T18:56:05.320Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_ba7ae14a-9687-4cd8-b97a-9174d9a7c5ef/root/var/svc/log/oxide-cockroachdb:default.log
10082023-12-11T18:56:05.323Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_c3fbadc4-668a-44d4-a87e-40b26171d029/root/var/svc/log/oxide-oximeter:default.log (248368 bytes)
10092023-12-11T18:56:05.526Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_c3fbadc4-668a-44d4-a87e-40b26171d029/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 248368 -> 266470
10102023-12-11T18:56:06.597Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_c3fbadc4-668a-44d4-a87e-40b26171d029/root/var/svc/log/oxide-oximeter:default.log
10112023-12-11T18:56:06.597Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_5c62c30b-23c8-48af-a54c-5811820263a4/root/var/svc/log/oxide-cockroachdb:default.log (5386 bytes)
10122023-12-11T18:56:07.699Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_5c62c30b-23c8-48af-a54c-5811820263a4/root/var/svc/log/oxide-cockroachdb:default.log
10132023-12-11T18:56:07.699Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_26a9da2e-7167-43d0-b868-67461e760d9a/root/var/svc/log/oxide-crucible-agent:default.log (20039 bytes)
10142023-12-11T18:56:08.842Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_26a9da2e-7167-43d0-b868-67461e760d9a/root/var/svc/log/oxide-crucible-agent:default.log
10152023-12-11T18:56:08.842Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_26a9da2e-7167-43d0-b868-67461e760d9a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10162023-12-11T18:56:10.339Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_26a9da2e-7167-43d0-b868-67461e760d9a/root/var/svc/log/oxide-crucible-downstairs:default.log
10172023-12-11T18:56:10.339Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_26a9da2e-7167-43d0-b868-67461e760d9a/root/var/svc/log/oxide-crucible-downstairs:downstairs-1c8950b0-1732-4627-a3af-990badc71a1a.log (3051 bytes)
10182023-12-11T18:56:11.438Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_26a9da2e-7167-43d0-b868-67461e760d9a/root/var/svc/log/oxide-crucible-downstairs:downstairs-1c8950b0-1732-4627-a3af-990badc71a1a.log
10192023-12-11T18:56:11.438Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_26a9da2e-7167-43d0-b868-67461e760d9a/root/var/svc/log/oxide-crucible-downstairs:downstairs-adc9dbb6-7ce9-48cf-8402-10eef2523c0b.log (2941 bytes)
10202023-12-11T18:56:12.533Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_26a9da2e-7167-43d0-b868-67461e760d9a/root/var/svc/log/oxide-crucible-downstairs:downstairs-adc9dbb6-7ce9-48cf-8402-10eef2523c0b.log
10212023-12-11T18:56:12.533Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_faaa6142-7dd1-4eef-87e3-a6af2c8bc3a8/root/var/svc/log/oxide-crucible-agent:default.log (9788 bytes)
10222023-12-11T18:56:13.649Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_faaa6142-7dd1-4eef-87e3-a6af2c8bc3a8/root/var/svc/log/oxide-crucible-agent:default.log
10232023-12-11T18:56:13.649Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_faaa6142-7dd1-4eef-87e3-a6af2c8bc3a8/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10242023-12-11T18:56:14.744Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_faaa6142-7dd1-4eef-87e3-a6af2c8bc3a8/root/var/svc/log/oxide-crucible-downstairs:default.log
10252023-12-11T18:56:14.744Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_faaa6142-7dd1-4eef-87e3-a6af2c8bc3a8/root/var/svc/log/oxide-crucible-downstairs:downstairs-dda951d1-38ef-44fd-b2a6-5c5638734385.log (2945 bytes)
10262023-12-11T18:56:15.839Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_faaa6142-7dd1-4eef-87e3-a6af2c8bc3a8/root/var/svc/log/oxide-crucible-downstairs:downstairs-dda951d1-38ef-44fd-b2a6-5c5638734385.log
10272023-12-11T18:56:15.839Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_b8ca3a50-57f8-4874-87a5-ed2c4bded51f/root/var/svc/log/oxide-internal_dns:default.log (116829 bytes)
10282023-12-11T18:56:16.023Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_b8ca3a50-57f8-4874-87a5-ed2c4bded51f/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 116829 -> 120583
10292023-12-11T18:56:17.095Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_b8ca3a50-57f8-4874-87a5-ed2c4bded51f/root/var/svc/log/oxide-internal_dns:default.log
10302023-12-11T18:56:17.095Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_5908292a-a899-4de9-97d1-1df3102fc535/root/var/svc/log/oxide-crucible-pantry:default.log (1227201 bytes)
10312023-12-11T18:56:18.867Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_5908292a-a899-4de9-97d1-1df3102fc535/root/var/svc/log/oxide-crucible-pantry:default.log
10322023-12-11T18:56:18.867Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_f98d40aa-1f98-4b8a-a9a6-11245a64564f/root/var/svc/log/oxide-cockroachdb:default.log (5412 bytes)
10332023-12-11T18:56:19.967Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_f98d40aa-1f98-4b8a-a9a6-11245a64564f/root/var/svc/log/oxide-cockroachdb:default.log
10342023-12-11T18:56:19.967Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_4d007c0d-a3fa-4a83-8e8b-2033ea605519/root/var/svc/log/oxide-internal_dns:default.log (110750 bytes)
10352023-12-11T18:56:20.131Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_4d007c0d-a3fa-4a83-8e8b-2033ea605519/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 110750 -> 114506
10362023-12-11T18:56:21.202Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_4d007c0d-a3fa-4a83-8e8b-2033ea605519/root/var/svc/log/oxide-internal_dns:default.log
10372023-12-11T18:56:21.280Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_03c32c3d-9f40-438f-ae6e-1c1787e67a78/root/var/svc/log/oxide-nexus:default.log (35978903 bytes)
10382023-12-11T18:56:33.301Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_03c32c3d-9f40-438f-ae6e-1c1787e67a78/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 35978903 -> 36054514
10392023-12-11T18:56:34.371Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_03c32c3d-9f40-438f-ae6e-1c1787e67a78/root/var/svc/log/oxide-nexus:default.log
10402023-12-11T18:56:34.396Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_ef7c67e4-c471-4740-a2b5-8ac9dd0e3f1f/root/var/svc/log/oxide-external_dns:default.log (26940 bytes)
10412023-12-11T18:56:34.484Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_ef7c67e4-c471-4740-a2b5-8ac9dd0e3f1f/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 26940 -> 30717
10422023-12-11T18:56:35.555Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_ef7c67e4-c471-4740-a2b5-8ac9dd0e3f1f/root/var/svc/log/oxide-external_dns:default.log
10432023-12-11T18:56:35.555Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69f8c42b-5f56-4300-a8b4-e60ca8a1147f/root/var/svc/log/oxide-crucible-agent:default.log (14113 bytes)
10442023-12-11T18:56:37.109Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69f8c42b-5f56-4300-a8b4-e60ca8a1147f/root/var/svc/log/oxide-crucible-agent:default.log
10452023-12-11T18:56:37.109Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69f8c42b-5f56-4300-a8b4-e60ca8a1147f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10462023-12-11T18:56:38.215Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69f8c42b-5f56-4300-a8b4-e60ca8a1147f/root/var/svc/log/oxide-crucible-downstairs:default.log
10472023-12-11T18:56:38.216Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69f8c42b-5f56-4300-a8b4-e60ca8a1147f/root/var/svc/log/oxide-crucible-downstairs:downstairs-1b2d532a-2f59-4e40-aaff-8590daf4e2ca.log (3050 bytes)
10482023-12-11T18:56:39.310Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69f8c42b-5f56-4300-a8b4-e60ca8a1147f/root/var/svc/log/oxide-crucible-downstairs:downstairs-1b2d532a-2f59-4e40-aaff-8590daf4e2ca.log
10492023-12-11T18:56:39.310Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e6c1a482-0a9e-4de3-a288-389be088495a/root/var/svc/log/oxide-crucible-agent:default.log (17553 bytes)
10502023-12-11T18:56:40.452Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e6c1a482-0a9e-4de3-a288-389be088495a/root/var/svc/log/oxide-crucible-agent:default.log
10512023-12-11T18:56:40.452Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e6c1a482-0a9e-4de3-a288-389be088495a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10522023-12-11T18:56:41.547Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e6c1a482-0a9e-4de3-a288-389be088495a/root/var/svc/log/oxide-crucible-downstairs:default.log
10532023-12-11T18:56:41.548Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e6c1a482-0a9e-4de3-a288-389be088495a/root/var/svc/log/oxide-crucible-downstairs:downstairs-1d7ef9c4-5431-4cd9-b67a-2fc467af01cd.log (6438 bytes)
10542023-12-11T18:56:42.644Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e6c1a482-0a9e-4de3-a288-389be088495a/root/var/svc/log/oxide-crucible-downstairs:downstairs-1d7ef9c4-5431-4cd9-b67a-2fc467af01cd.log
10552023-12-11T18:56:42.668Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e6c1a482-0a9e-4de3-a288-389be088495a/root/var/svc/log/oxide-crucible-downstairs:snapshot-1d7ef9c4-5431-4cd9-b67a-2fc467af01cd-10d2c586-80aa-4e97-ab5c-713c478e9e4c.log (5176 bytes)
10562023-12-11T18:56:43.738Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e6c1a482-0a9e-4de3-a288-389be088495a/root/var/svc/log/oxide-crucible-downstairs:snapshot-1d7ef9c4-5431-4cd9-b67a-2fc467af01cd-10d2c586-80aa-4e97-ab5c-713c478e9e4c.log
10572023-12-11T18:56:43.738Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_70bbb979-a57a-4be5-bfc1-4d0d6a336a0d/root/var/svc/log/oxide-crucible-pantry:default.log (2332 bytes)
10582023-12-11T18:56:44.837Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_70bbb979-a57a-4be5-bfc1-4d0d6a336a0d/root/var/svc/log/oxide-crucible-pantry:default.log
10592023-12-11T18:56:44.837Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_81aa7f9a-bae1-4364-ad60-9efbfaec1c70/root/var/svc/log/oxide-internal_dns:default.log (114799 bytes)
10602023-12-11T18:56:44.997Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_81aa7f9a-bae1-4364-ad60-9efbfaec1c70/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 114799 -> 118554
10612023-12-11T18:56:46.076Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_81aa7f9a-bae1-4364-ad60-9efbfaec1c70/root/var/svc/log/oxide-internal_dns:default.log
10622023-12-11T18:56:46.076Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9826eadc-70a4-4416-bb8b-526e8a5d88e7/root/var/svc/log/oxide-cockroachdb:default.log (5412 bytes)
10632023-12-11T18:56:47.171Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9826eadc-70a4-4416-bb8b-526e8a5d88e7/root/var/svc/log/oxide-cockroachdb:default.log
10642023-12-11T18:56:47.171Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf7714fb-6191-42c7-8eb1-bc5b1a22b77d/root/var/svc/log/oxide-crucible-agent:default.log (17547 bytes)
10652023-12-11T18:56:48.317Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf7714fb-6191-42c7-8eb1-bc5b1a22b77d/root/var/svc/log/oxide-crucible-agent:default.log
10662023-12-11T18:56:48.318Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf7714fb-6191-42c7-8eb1-bc5b1a22b77d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10672023-12-11T18:56:49.420Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf7714fb-6191-42c7-8eb1-bc5b1a22b77d/root/var/svc/log/oxide-crucible-downstairs:default.log
10682023-12-11T18:56:49.421Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf7714fb-6191-42c7-8eb1-bc5b1a22b77d/root/var/svc/log/oxide-crucible-downstairs:downstairs-c2cacaf7-f4f3-4196-9d7a-d9b92a00ee56.log (6435 bytes)
10692023-12-11T18:56:50.566Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf7714fb-6191-42c7-8eb1-bc5b1a22b77d/root/var/svc/log/oxide-crucible-downstairs:downstairs-c2cacaf7-f4f3-4196-9d7a-d9b92a00ee56.log
10702023-12-11T18:56:50.566Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf7714fb-6191-42c7-8eb1-bc5b1a22b77d/root/var/svc/log/oxide-crucible-downstairs:snapshot-c2cacaf7-f4f3-4196-9d7a-d9b92a00ee56-10d2c586-80aa-4e97-ab5c-713c478e9e4c.log (5175 bytes)
10712023-12-11T18:56:51.667Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf7714fb-6191-42c7-8eb1-bc5b1a22b77d/root/var/svc/log/oxide-crucible-downstairs:snapshot-c2cacaf7-f4f3-4196-9d7a-d9b92a00ee56-10d2c586-80aa-4e97-ab5c-713c478e9e4c.log
10722023-12-11T18:56:51.670Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c/root/var/log/chrony/measurements.log (22331 bytes)
10732023-12-11T18:56:51.759Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c/root/var/log/chrony/measurements.log" changed size mid upload: 22331 -> 24934
10742023-12-11T18:56:52.839Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c/root/var/log/chrony/measurements.log
10752023-12-11T18:56:52.839Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c/root/var/log/chrony/statistics.log (14742 bytes)
10762023-12-11T18:56:52.937Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c/root/var/log/chrony/statistics.log" changed size mid upload: 14742 -> 16848
10772023-12-11T18:56:54.033Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c/root/var/log/chrony/statistics.log
10782023-12-11T18:56:54.033Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c/root/var/log/chrony/tracking.log (2245 bytes)
10792023-12-11T18:56:54.089Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c/root/var/log/chrony/tracking.log" changed size mid upload: 2245 -> 2377
10802023-12-11T18:56:55.162Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecdb1628-5687-46dd-9307-db6642e8541c/root/var/log/chrony/tracking.log