01HGW2H42K7TNSJG97G0ZYFN8B: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HGW2HS56Q0MBFVT5YREB0WBT

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-05T04:55:12.297Zjob dependencies complete; ready to run (waiting for 47 m 39 s)
22023-12-05T04:57:54.306Zjob assigned to worker 01HGW52MNH7MKAB7C0DQVC6P4Y (queued for 2 m 42 s)
32023-12-05T04:57:54.352Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-12-05T04:57:54.402Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-12-05T04:57:54.434Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-12-05T04:57:54.467Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-05T04:57:54.501Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-12-05T04:57:54.535ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-05T04:57:54.594Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-05T04:57:54.641ZOxide Helios Version helios-2.0.22318 64-bit
112023-12-05T04:57:54.685ZConfiguring devices.
122023-12-05T04:57:54.723Z##################################################
132023-12-05T04:57:54.762Z#### ############# ##
142023-12-05T04:57:54.803Z### ### ############ ##
152023-12-05T04:57:54.836Z## ### # ## ### ## Oxide Computer Company ##
162023-12-05T04:57:54.870Z## ## ## ### # ### ##
172023-12-05T04:57:54.904Z## # ### #### #### This Station Under ##
182023-12-05T04:57:54.942Z### ### #### # ### Computer Control ##
192023-12-05T04:57:54.979Z#### #### ### ## ##
202023-12-05T04:57:55.016Z##################################################
212023-12-05T04:57:55.058ZHostname: sock
222023-12-05T04:57:55.499ZLoading smf(7) service descriptions:
232023-12-05T04:57:56.281Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-83d41f1b41d1fb89.gz
242023-12-05T04:57:59.153Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-83d41f1b41d1fb89.gz
252023-12-05T04:57:59.153Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-05T04:58:26.150Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-05T04:58:26.150Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8c93950b566420c7.gz
282023-12-05T04:58:49.702Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8c93950b566420c7.gz
292023-12-05T04:58:49.702Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-05T04:58:51.441Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-05T04:58:51.441Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-05T04:58:56.238Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-05T04:58:56.238Zdownloading input: /input/package/work/version.txt
342023-12-05T04:58:56.264Zdownloaded input: /input/package/work/version.txt
352023-12-05T04:58:56.264Zdownloading input: /input/package/work/package.tar.gz
362023-12-05T04:59:19.891Zdownloaded input: /input/package/work/package.tar.gz
372023-12-05T04:59:19.891Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-05T04:59:44.684Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-05T04:59:44.714Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-05T04:59:52.409Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-05T04:59:52.439Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-05T05:00:25.007Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-05T05:00:25.007Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-05T05:00:57.297Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-05T05:00:57.297Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-05T05:01:19.035Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-05T05:01:19.035Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-05T05:01:23.035Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-05T05:01:23.059Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-05T05:01:30.649Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-05T05:01:30.673Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-05T05:01:35.097Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-05T05:01:35.098Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-05T05:01:40.126Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-05T05:01:40.126Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-05T05:01:40.162Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-05T05:01:40.162Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-05T05:01:43.541Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-05T05:01:43.541Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-05T05:01:46.874Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-05T05:01:46.898Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-05T05:01:59.869Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-05T05:01:59.869Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-05T05:02:13.204Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-05T05:02:13.204Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-05T05:02:13.242Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-05T05:02:13.243Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-05T05:02:19.503Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-05T05:02:19.527Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-05T05:02:32.164Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-05T05:02:32.164Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-05T05:03:17.505Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-05T05:03:17.505Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-05T05:03:54.585Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-05T05:03:54.609Zstarting task 0: "setup"
762023-12-05T05:03:54.609Z++ uname -s
772023-12-05T05:03:54.609Z+ kern=SunOS
782023-12-05T05:03:54.609Z+ case "$kern" in
792023-12-05T05:03:54.609Z+ groupadd -g 12345 build
802023-12-05T05:03:54.609Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-05T05:03:56.602Z+ zfs create -o mountpoint=/work rpool/work
822023-12-05T05:03:56.635Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-05T05:03:56.662Z+ home_fs=zfs
842023-12-05T05:03:56.662Z+ [[ zfs == autofs ]]
852023-12-05T05:03:56.662Z+ mkdir -p /home/build
862023-12-05T05:03:56.662Z+ chown build:build /home/build /work
872023-12-05T05:03:58.643Z+ chmod 0700 /home/build /work
882023-12-05T05:03:58.667Zprocess exited: duration 4061 ms, exit code 0
 
892023-12-05T05:03:58.715Zstarting task 1: "authentication"
902023-12-05T05:03:58.760Zprocess exited: duration 44 ms, exit code 0
 
912023-12-05T05:03:58.816Zstarting task 2: "build"
922023-12-05T05:03:58.840Z+ trap _exit_trap EXIT
932023-12-05T05:03:58.840Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-05T05:03:59.018Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-05T05:03:59.018Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-05T05:03:59.019Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-05T05:03:59.019Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-05T05:03:59.019Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-05T05:03:59.022Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-05T05:03:59.046Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-05T05:03:59.046Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-05T05:03:59.046Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-05T05:03:59.046Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-05T05:03:59.046Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-05T05:03:59.046Z+ [[ -d /opt/oxide ]]
1062023-12-05T05:03:59.046Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-05T05:03:59.046Z+ pfexec mkdir /opt/oxide
1082023-12-05T05:03:59.046Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-05T05:03:59.046Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-05T05:03:59.046Z+ cd /opt/oxide-underneath
1112023-12-05T05:03:59.046Z+ pfexec tar ceEp@/f - .
1122023-12-05T05:03:59.046Z+ cd /opt/oxide
1132023-12-05T05:03:59.046Z+ pfexec tar xveEp@/f -
1142023-12-05T05:03:59.140Zx ., 0 bytes, 0 tape blocks
1152023-12-05T05:03:59.140Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-05T05:03:59.140Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-05T05:03:59.140Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-12-05T05:03:59.164Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-05T05:03:59.164Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-05T05:03:59.187Z+ pfexec mkdir /opt/oxide/work
1212023-12-05T05:03:59.187Z+ pfexec chown build:build /opt/oxide/work
1222023-12-05T05:03:59.187Z+ cd /opt/oxide/work
1232023-12-05T05:03:59.187Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-05T05:03:59.187Ztar: blocksize = 10
1252023-12-05T05:04:00.662ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1262023-12-05T05:04:00.662Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1272023-12-05T05:04:00.662Zx out/omicron-sled-agent.tar, 111094784 bytes, 216982 tape blocks
1282023-12-05T05:04:00.662Zx out/target/test, 75 bytes, 1 tape blocks
1292023-12-05T05:04:00.662Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1302023-12-05T05:04:00.662Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1312023-12-05T05:04:00.663Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1322023-12-05T05:04:00.663Zx target/release/omicron-package, 42582368 bytes, 83169 tape blocks
1332023-12-05T05:04:00.665Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1342023-12-05T05:04:00.665Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1352023-12-05T05:04:00.665Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1362023-12-05T05:04:00.665Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1372023-12-05T05:04:00.665Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1382023-12-05T05:04:00.665Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1392023-12-05T05:04:00.665Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1402023-12-05T05:04:00.696Z
1412023-12-05T05:04:00.696Zreal 1.484913951
1422023-12-05T05:04:00.696Zuser 1.383585449
1432023-12-05T05:04:00.696Zsys 0.198615069
1442023-12-05T05:04:00.696Ztrap 0.000062620
1452023-12-05T05:04:00.696Ztflt 0.000000000
1462023-12-05T05:04:00.696Zdflt 0.000000000
1472023-12-05T05:04:00.696Zkflt 0.000000000
1482023-12-05T05:04:00.696Zlock 0.000000000
1492023-12-05T05:04:00.696Zslp 1.354813940
1502023-12-05T05:04:00.697Zlat 0.030663028
1512023-12-05T05:04:00.697Zstop 0.000123721
1522023-12-05T05:04:00.697Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1532023-12-05T05:04:01.759Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1542023-12-05T05:04:01.783Z+ mkdir tests
1552023-12-05T05:04:01.783Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1562023-12-05T05:04:01.783Z+ ptime -m gunzip
1572023-12-05T05:04:01.783Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-83d41f1b41d1fb89
1582023-12-05T05:04:01.917Z
1592023-12-05T05:04:01.941Zreal 0.136999073
1602023-12-05T05:04:01.941Zuser 0.126081053
1612023-12-05T05:04:01.941Zsys 0.010680066
1622023-12-05T05:04:01.941Ztrap 0.000008129
1632023-12-05T05:04:01.941Ztflt 0.000000000
1642023-12-05T05:04:01.941Zdflt 0.000000000
1652023-12-05T05:04:01.941Zkflt 0.000000000
1662023-12-05T05:04:01.941Zlock 0.000000000
1672023-12-05T05:04:01.942Zslp 0.000016210
1682023-12-05T05:04:01.942Zlat 0.000022710
1692023-12-05T05:04:01.942Zstop 0.000125741
1702023-12-05T05:04:01.942Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-83d41f1b41d1fb89
1712023-12-05T05:04:01.942Z+ chmod a+x tests/bootstrap-83d41f1b41d1fb89
1722023-12-05T05:04:01.942Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1732023-12-05T05:04:01.942Z+ ptime -m gunzip
1742023-12-05T05:04:01.942Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1752023-12-05T05:04:04.126Z
1762023-12-05T05:04:04.150Zreal 2.197573494
1772023-12-05T05:04:04.152Zuser 2.055238758
1782023-12-05T05:04:04.152Zsys 0.141963118
1792023-12-05T05:04:04.152Ztrap 0.000147823
1802023-12-05T05:04:04.152Ztflt 0.000000000
1812023-12-05T05:04:04.152Zdflt 0.000000000
1822023-12-05T05:04:04.152Zkflt 0.000000000
1832023-12-05T05:04:04.152Zlock 0.000000000
1842023-12-05T05:04:04.152Zslp 0.000018500
1852023-12-05T05:04:04.152Zlat 0.000026010
1862023-12-05T05:04:04.152Zstop 0.000118821
1872023-12-05T05:04:04.152Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1882023-12-05T05:04:04.152Z+ chmod a+x tests/bootstrap
1892023-12-05T05:04:04.152Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1902023-12-05T05:04:04.152Z+ ptime -m gunzip
1912023-12-05T05:04:04.152Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8c93950b566420c7
1922023-12-05T05:04:06.447Z
1932023-12-05T05:04:06.471Zreal 2.309969907
1942023-12-05T05:04:06.471Zuser 2.167074491
1952023-12-05T05:04:06.471Zsys 0.142473950
1962023-12-05T05:04:06.471Ztrap 0.000181122
1972023-12-05T05:04:06.471Ztflt 0.000000000
1982023-12-05T05:04:06.471Zdflt 0.000000000
1992023-12-05T05:04:06.471Zkflt 0.000000000
2002023-12-05T05:04:06.471Zlock 0.000000000
2012023-12-05T05:04:06.471Zslp 0.000026000
2022023-12-05T05:04:06.471Zlat 0.000022640
2032023-12-05T05:04:06.471Zstop 0.000127002
2042023-12-05T05:04:06.471Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8c93950b566420c7
2052023-12-05T05:04:06.471Z+ chmod a+x tests/end_to_end_tests-8c93950b566420c7
2062023-12-05T05:04:06.471Z+ bmat address ls
2072023-12-05T05:04:06.471ZNAME CIDR FIRST LAST COUNT
2082023-12-05T05:04:06.471Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2092023-12-05T05:04:06.471Z++ bmat address ls -f extra -Ho first,last,count
2102023-12-05T05:04:06.495Z+ set -- 10.151.2.164 10.151.2.227 64
2112023-12-05T05:04:06.495Z+ EXTRA_IP_START=10.151.2.164
2122023-12-05T05:04:06.495Z+ EXTRA_IP_END=10.151.2.227
2132023-12-05T05:04:06.495Z+ EXTRA_IP_COUNT=64
2142023-12-05T05:04:06.495Z+ (( EXTRA_IP_COUNT >= 32 ))
2152023-12-05T05:04:06.495Z+ EXTRA_IP_BASE=10.151.2
2162023-12-05T05:04:06.495Z+ EXTRA_IP_FOCTET=164
2172023-12-05T05:04:06.495Z+ EXTRA_IP_LOCTET=227
2182023-12-05T05:04:06.508Z+ SERVICE_IP_POOL_START=10.151.2.164
2192023-12-05T05:04:06.508Z+ SERVICE_IP_POOL_END=10.151.2.173
2202023-12-05T05:04:06.508Z+ DNS_IP1=10.151.2.164
2212023-12-05T05:04:06.508Z+ DNS_IP2=10.151.2.165
2222023-12-05T05:04:06.508Z+ UPLINK_IP=10.151.2.174
2232023-12-05T05:04:06.508Z+ IPPOOL_START=10.151.2.175
2242023-12-05T05:04:06.508Z+ IPPOOL_END=10.151.2.227
2252023-12-05T05:04:06.508Z++ ipadm show-addr -po type,addr
2262023-12-05T05:04:06.508Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2272023-12-05T05:04:06.508Z+ GATEWAY_IP=10.151.2.100
2282023-12-05T05:04:06.508Z+ [[ -n 10.151.2.100 ]]
2292023-12-05T05:04:06.508Z+ ping -s 10.151.2.100 56 1
2302023-12-05T05:04:06.508ZPING 10.151.2.100: 56 data bytes
2312023-12-05T05:04:06.508Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2322023-12-05T05:04:06.508Z
2332023-12-05T05:04:06.508Z----10.151.2.100 PING Statistics----
2342023-12-05T05:04:06.508Z1 packets transmitted, 1 packets received, 0% packet loss
2352023-12-05T05:04:06.508Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2362023-12-05T05:04:06.508Z++ arp -an
2372023-12-05T05:04:06.508Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2382023-12-05T05:04:06.532Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2392023-12-05T05:04:06.532Z+ routeadm -e ipv4-forwarding -u
2402023-12-05T05:04:06.609Z+ PXA_START=10.151.2.164
2412023-12-05T05:04:06.609Z+ PXA_END=10.151.2.227
2422023-12-05T05:04:06.609Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2432023-12-05T05:04:06.634Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2442023-12-05T05:04:06.761Z+ ZPOOL_VDEV_DIR=/scratch
2452023-12-05T05:04:06.761Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2462023-12-05T05:04:06.818Z+++ dirname ./tools/create_virtual_hardware.sh
2472023-12-05T05:04:06.818Z++ cd ./tools
2482023-12-05T05:04:06.818Z++ pwd
2492023-12-05T05:04:06.818Z+ SOURCE_DIR=/opt/oxide/work/tools
2502023-12-05T05:04:06.818Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2512023-12-05T05:04:06.818Z+ SOFTNPU_MODE=zone
2522023-12-05T05:04:06.818Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2532023-12-05T05:04:06.818Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2542023-12-05T05:04:06.818Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2552023-12-05T05:04:06.818Z++ dladm show-phys -p -o LINK
2562023-12-05T05:04:06.818Z++ head -1
2572023-12-05T05:04:06.850Z+ PHYSICAL_LINK=igb0
2582023-12-05T05:04:06.850Z+ echo 'Using igb0 as physical link'
2592023-12-05T05:04:06.850ZUsing igb0 as physical link
2602023-12-05T05:04:06.850Z+ ensure_run_as_root
2612023-12-05T05:04:06.850Z++ id -u
2622023-12-05T05:04:06.850Z+ [[ 0 -ne 0 ]]
2632023-12-05T05:04:06.850Z+ ensure_zpools
2642023-12-05T05:04:06.850Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2652023-12-05T05:04:06.850Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2662023-12-05T05:04:06.850Z+ readarray -t ZPOOLS
2672023-12-05T05:04:06.850Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2682023-12-05T05:04:06.850Z++ sed 's/[ ",]//g'
2692023-12-05T05:04:06.850Z+ for ZPOOL in "${ZPOOLS[@]}"
2702023-12-05T05:04:06.850Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2712023-12-05T05:04:06.850ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2722023-12-05T05:04:06.850Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2732023-12-05T05:04:06.850Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2742023-12-05T05:04:06.850Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2752023-12-05T05:04:06.874Z0+0 records in
2762023-12-05T05:04:06.874Z0+0 records out
2772023-12-05T05:04:06.874Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2782023-12-05T05:04:06.874Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2792023-12-05T05:04:06.874Z+ set +x
2802023-12-05T05:04:06.874ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2812023-12-05T05:04:06.874Z++ zpool list -o name
2822023-12-05T05:04:06.898Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2832023-12-05T05:04:06.898Z+ [[ -z '' ]]
2842023-12-05T05:04:06.898Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2852023-12-05T05:04:06.926Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2862023-12-05T05:04:06.950Z+ set +x
2872023-12-05T05:04:06.950ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2882023-12-05T05:04:06.950Z+ for ZPOOL in "${ZPOOLS[@]}"
2892023-12-05T05:04:06.950Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2902023-12-05T05:04:06.950ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2912023-12-05T05:04:06.950Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2922023-12-05T05:04:06.950Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2932023-12-05T05:04:06.950Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2942023-12-05T05:04:06.951Z0+0 records in
2952023-12-05T05:04:06.951Z0+0 records out
2962023-12-05T05:04:06.951Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2972023-12-05T05:04:06.951Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2982023-12-05T05:04:06.951Z+ set +x
2992023-12-05T05:04:06.951ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3002023-12-05T05:04:06.951Z++ zpool list -o name
3012023-12-05T05:04:06.951Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3022023-12-05T05:04:06.951Z+ [[ -z '' ]]
3032023-12-05T05:04:06.951Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3042023-12-05T05:04:06.978Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3052023-12-05T05:04:06.978Z+ set +x
3062023-12-05T05:04:07.002ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3072023-12-05T05:04:07.002Z+ for ZPOOL in "${ZPOOLS[@]}"
3082023-12-05T05:04:07.002Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3092023-12-05T05:04:07.002ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3102023-12-05T05:04:07.002Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3112023-12-05T05:04:07.002Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3122023-12-05T05:04:07.002Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3132023-12-05T05:04:07.002Z0+0 records in
3142023-12-05T05:04:07.002Z0+0 records out
3152023-12-05T05:04:07.002Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3162023-12-05T05:04:07.002Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3172023-12-05T05:04:07.002Z+ set +x
3182023-12-05T05:04:07.002ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3192023-12-05T05:04:07.005Z++ zpool list -o name
3202023-12-05T05:04:07.005Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3212023-12-05T05:04:07.005Z+ [[ -z '' ]]
3222023-12-05T05:04:07.005Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3232023-12-05T05:04:07.031Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3242023-12-05T05:04:07.055Z+ set +x
3252023-12-05T05:04:07.055ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3262023-12-05T05:04:07.055Z+ for ZPOOL in "${ZPOOLS[@]}"
3272023-12-05T05:04:07.055Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3282023-12-05T05:04:07.055ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3292023-12-05T05:04:07.055Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3302023-12-05T05:04:07.055Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3312023-12-05T05:04:07.055Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3322023-12-05T05:04:07.055Z0+0 records in
3332023-12-05T05:04:07.055Z0+0 records out
3342023-12-05T05:04:07.055Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3352023-12-05T05:04:07.055Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3362023-12-05T05:04:07.055Z+ set +x
3372023-12-05T05:04:07.055ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3382023-12-05T05:04:07.055Z++ zpool list -o name
3392023-12-05T05:04:07.055Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3402023-12-05T05:04:07.055Z+ [[ -z '' ]]
3412023-12-05T05:04:07.055Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3422023-12-05T05:04:07.079Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3432023-12-05T05:04:07.079Z+ set +x
3442023-12-05T05:04:07.079ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3452023-12-05T05:04:07.079Z+ for ZPOOL in "${ZPOOLS[@]}"
3462023-12-05T05:04:07.079ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-12-05T05:04:07.079Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3482023-12-05T05:04:07.079Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3492023-12-05T05:04:07.079Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3502023-12-05T05:04:07.079Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3512023-12-05T05:04:07.103Z0+0 records in
3522023-12-05T05:04:07.103Z0+0 records out
3532023-12-05T05:04:07.103Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3542023-12-05T05:04:07.103Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3552023-12-05T05:04:07.103Z+ set +x
3562023-12-05T05:04:07.103ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3572023-12-05T05:04:07.103Z++ zpool list -o name
3582023-12-05T05:04:07.103Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3592023-12-05T05:04:07.103Z+ [[ -z '' ]]
3602023-12-05T05:04:07.103Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3612023-12-05T05:04:07.127Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3622023-12-05T05:04:07.127Z+ set +x
3632023-12-05T05:04:07.127ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3642023-12-05T05:04:07.127Z+ for ZPOOL in "${ZPOOLS[@]}"
3652023-12-05T05:04:07.127Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3662023-12-05T05:04:07.128ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3672023-12-05T05:04:07.128Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3682023-12-05T05:04:07.128Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3692023-12-05T05:04:07.128Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3702023-12-05T05:04:07.129Z0+0 records in
3712023-12-05T05:04:07.129Z0+0 records out
3722023-12-05T05:04:07.129Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3732023-12-05T05:04:07.129Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3742023-12-05T05:04:07.129Z+ set +x
3752023-12-05T05:04:07.129ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3762023-12-05T05:04:07.152Z++ zpool list -o name
3772023-12-05T05:04:07.152Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3782023-12-05T05:04:07.152Z+ [[ -z '' ]]
3792023-12-05T05:04:07.152Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3802023-12-05T05:04:07.176Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3812023-12-05T05:04:07.176Z+ set +x
3822023-12-05T05:04:07.176ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3832023-12-05T05:04:07.176Z+ for ZPOOL in "${ZPOOLS[@]}"
3842023-12-05T05:04:07.176Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3852023-12-05T05:04:07.176ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3862023-12-05T05:04:07.176Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3872023-12-05T05:04:07.176Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3882023-12-05T05:04:07.176Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3892023-12-05T05:04:07.200Z0+0 records in
3902023-12-05T05:04:07.200Z0+0 records out
3912023-12-05T05:04:07.200Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-12-05T05:04:07.200Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3932023-12-05T05:04:07.200Z+ set +x
3942023-12-05T05:04:07.200ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3952023-12-05T05:04:07.200Z++ zpool list -o name
3962023-12-05T05:04:07.200Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3972023-12-05T05:04:07.200Z+ [[ -z '' ]]
3982023-12-05T05:04:07.200Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3992023-12-05T05:04:07.224Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4002023-12-05T05:04:07.224Z+ set +x
4012023-12-05T05:04:07.248ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4022023-12-05T05:04:07.248Z+ for ZPOOL in "${ZPOOLS[@]}"
4032023-12-05T05:04:07.248Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4042023-12-05T05:04:07.248ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4052023-12-05T05:04:07.248Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4062023-12-05T05:04:07.248Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4072023-12-05T05:04:07.248Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4082023-12-05T05:04:07.248Z0+0 records in
4092023-12-05T05:04:07.248Z0+0 records out
4102023-12-05T05:04:07.248Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4112023-12-05T05:04:07.248Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4122023-12-05T05:04:07.248Z+ set +x
4132023-12-05T05:04:07.248ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4142023-12-05T05:04:07.249Z++ zpool list -o name
4152023-12-05T05:04:07.249Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4162023-12-05T05:04:07.249Z+ [[ -z '' ]]
4172023-12-05T05:04:07.249Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4182023-12-05T05:04:07.278Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4192023-12-05T05:04:07.278Z+ set +x
4202023-12-05T05:04:07.302ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4212023-12-05T05:04:07.302Z+ for ZPOOL in "${ZPOOLS[@]}"
4222023-12-05T05:04:07.302ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-12-05T05:04:07.302Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4242023-12-05T05:04:07.302Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4252023-12-05T05:04:07.302Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4262023-12-05T05:04:07.302Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4272023-12-05T05:04:07.302Z0+0 records in
4282023-12-05T05:04:07.302Z0+0 records out
4292023-12-05T05:04:07.304Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4302023-12-05T05:04:07.304Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4312023-12-05T05:04:07.304Z+ set +x
4322023-12-05T05:04:07.304ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4332023-12-05T05:04:07.304Z++ zpool list -o name
4342023-12-05T05:04:07.304Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4352023-12-05T05:04:07.304Z+ [[ -z '' ]]
4362023-12-05T05:04:07.304Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4372023-12-05T05:04:07.329Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4382023-12-05T05:04:07.329Z+ set +x
4392023-12-05T05:04:07.355ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4402023-12-05T05:04:07.355Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4412023-12-05T05:04:07.355Z+ readarray -t ZPOOLS
4422023-12-05T05:04:07.355Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4432023-12-05T05:04:07.355Z++ sed 's/[ ",]//g'
4442023-12-05T05:04:07.355Z+ for ZPOOL in "${ZPOOLS[@]}"
4452023-12-05T05:04:07.355Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4462023-12-05T05:04:07.355ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4472023-12-05T05:04:07.355Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4482023-12-05T05:04:07.355Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4492023-12-05T05:04:07.355Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4502023-12-05T05:04:07.355Z0+0 records in
4512023-12-05T05:04:07.355Z0+0 records out
4522023-12-05T05:04:07.355Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4532023-12-05T05:04:07.355Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4542023-12-05T05:04:07.355Z+ set +x
4552023-12-05T05:04:07.355ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4562023-12-05T05:04:07.355Z++ zpool list -o name
4572023-12-05T05:04:07.355Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4582023-12-05T05:04:07.355Z+ [[ -z '' ]]
4592023-12-05T05:04:07.355Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4602023-12-05T05:04:08.730Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4612023-12-05T05:04:08.730Z+ set +x
4622023-12-05T05:04:08.730ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4632023-12-05T05:04:08.730Z+ for ZPOOL in "${ZPOOLS[@]}"
4642023-12-05T05:04:08.730Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4652023-12-05T05:04:08.730ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4662023-12-05T05:04:08.730Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4672023-12-05T05:04:08.730Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4682023-12-05T05:04:08.730Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4692023-12-05T05:04:08.730Z0+0 records in
4702023-12-05T05:04:08.730Z0+0 records out
4712023-12-05T05:04:08.730Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4722023-12-05T05:04:08.730Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4732023-12-05T05:04:08.730Z+ set +x
4742023-12-05T05:04:08.730ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4752023-12-05T05:04:08.730Z++ zpool list -o name
4762023-12-05T05:04:08.730Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4772023-12-05T05:04:08.730Z+ [[ -z '' ]]
4782023-12-05T05:04:08.730Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4792023-12-05T05:04:08.730Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4802023-12-05T05:04:08.730Z+ set +x
4812023-12-05T05:04:08.731ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4822023-12-05T05:04:08.731Z+ [[ zone == \z\o\n\e ]]
4832023-12-05T05:04:08.731Z+ ensure_simulated_links igb0
4842023-12-05T05:04:08.731Z+ local PHYSICAL_LINK=igb0
4852023-12-05T05:04:08.731Z+ INDICES=("0" "1")
4862023-12-05T05:04:08.731Z+ for I in "${INDICES[@]}"
4872023-12-05T05:04:08.731Z++ get_simnet_name_if_exists net0
4882023-12-05T05:04:08.731Z++ dladm show-simnet -p -o LINK net0
4892023-12-05T05:04:08.731Z++ echo ''
4902023-12-05T05:04:08.731Z+ [[ -z '' ]]
4912023-12-05T05:04:08.731Z+ dladm create-simnet -t net0
4922023-12-05T05:04:08.731Z+ dladm create-simnet -t sc0_0
4932023-12-05T05:04:08.731Z+ dladm modify-simnet -t -p net0 sc0_0
4942023-12-05T05:04:08.731Z+ dladm set-linkprop -p mtu=9000 sc0_0
4952023-12-05T05:04:08.731Z+ success 'Simnet net0/sc0_0 exists'
4962023-12-05T05:04:08.731Z+ set +x
4972023-12-05T05:04:08.731ZSimnet net0/sc0_0 exists
4982023-12-05T05:04:08.731Z+ for I in "${INDICES[@]}"
4992023-12-05T05:04:08.731Z++ get_simnet_name_if_exists net1
5002023-12-05T05:04:08.732Z++ dladm show-simnet -p -o LINK net1
5012023-12-05T05:04:08.732Z++ echo ''
5022023-12-05T05:04:08.732Z+ [[ -z '' ]]
5032023-12-05T05:04:08.732Z+ dladm create-simnet -t net1
5042023-12-05T05:04:08.732Z+ dladm create-simnet -t sc1_0
5052023-12-05T05:04:08.732Z+ dladm modify-simnet -t -p net1 sc1_0
5062023-12-05T05:04:08.732Z+ dladm set-linkprop -p mtu=9000 sc1_0
5072023-12-05T05:04:08.732Z+ success 'Simnet net1/sc1_0 exists'
5082023-12-05T05:04:08.732Z+ set +x
5092023-12-05T05:04:08.732ZSimnet net1/sc1_0 exists
5102023-12-05T05:04:08.732Z++ get_vnic_name_if_exists sc0_1
5112023-12-05T05:04:08.732Z++ dladm show-vnic -p -o LINK sc0_1
5122023-12-05T05:04:08.732Z++ echo ''
5132023-12-05T05:04:08.732Z+ [[ -z '' ]]
5142023-12-05T05:04:08.732Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5152023-12-05T05:04:08.732Z+ [[ -v PROMISC_FILT_OFF ]]
5162023-12-05T05:04:08.733Z+ success 'Vnic sc0_1 exists'
5172023-12-05T05:04:08.733Z+ set +x
5182023-12-05T05:04:08.733ZVnic sc0_1 exists
5192023-12-05T05:04:08.733Z+ warn_if_no_proxy_arp
5202023-12-05T05:04:08.733Z+ [[ -v PXA_START ]]
5212023-12-05T05:04:08.733Z+ [[ -v PXA_END ]]
5222023-12-05T05:04:08.733Z+ ensure_softnpu_zone
5232023-12-05T05:04:08.734Z+ zoneadm list
5242023-12-05T05:04:08.734Z+ grep -q sidecar_softnpu
5252023-12-05T05:04:08.734Z+ [[ -f out/npuzone/npuzone ]]
5262023-12-05T05:04:08.734Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5272023-12-05T05:04:08.734Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
5282023-12-05T05:04:12.464Zdownloading softnpu ...done
5292023-12-05T05:04:12.488Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5302023-12-05T05:04:17.069Zdownloading asic_program.so ...done
5312023-12-05T05:04:17.093Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5322023-12-05T05:04:22.892Zdownloading scadm ...done
5332023-12-05T05:04:22.918Zcreating port tfportrear0_0 for sc0_0
5342023-12-05T05:04:22.918Zcreating port tfportqsfp0_0 for sc0_1
5352023-12-05T05:04:22.918Zsoftnpu zone setup
5362023-12-05T05:04:22.918Zconfigure zone
5372023-12-05T05:04:22.918Zinstall zone
5382023-12-05T05:04:23.571Zboot zone
5392023-12-05T05:04:26.680Z[sidecar_softnpu] svcs milestone/network
5402023-12-05T05:04:26.706ZSTATE STIME FMRI
5412023-12-05T05:04:26.706Zonline 5:04:24 svc:/milestone/network:default
5422023-12-05T05:04:26.706Z[sidecar_softnpu] /softnpu/npu start
5432023-12-05T05:04:26.732Z
5442023-12-05T05:04:26.756Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5452023-12-05T05:04:26.756Z+ GATEWAY_IP=10.151.2.100
5462023-12-05T05:04:26.758Z+ echo 'Using 10.151.2.100 as gateway ip'
5472023-12-05T05:04:26.758ZUsing 10.151.2.100 as gateway ip
5482023-12-05T05:04:26.758Z+ [[ ! -v GATEWAY_MAC ]]
5492023-12-05T05:04:26.758Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5502023-12-05T05:04:26.758Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5512023-12-05T05:04:26.758Z+++ echo 18:c0:4d:0d:a0:2a
5522023-12-05T05:04:26.758Z+ COUNT=1
5532023-12-05T05:04:26.758Z+ [[ 1 -ne 1 ]]
5542023-12-05T05:04:26.758Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5552023-12-05T05:04:26.758ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5562023-12-05T05:04:26.758Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-12-05T05:04:26.758Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5582023-12-05T05:04:26.782Z+ PXA_MAC=a8:e1:de:01:70:1d
5592023-12-05T05:04:26.782Z+ [[ -v PXA_START ]]
5602023-12-05T05:04:26.782Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-12-05T05:04:26.782Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5622023-12-05T05:04:26.805Z+ z_scadm dump-state
5632023-12-05T05:04:26.805Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5642023-12-05T05:04:26.829Zlocal v6:
5652023-12-05T05:04:26.829Zlocal v4:
5662023-12-05T05:04:26.829Zrouter v6:
5672023-12-05T05:04:26.829Zrouter v4_idx:
5682023-12-05T05:04:26.829Zrouter v4_routes:
5692023-12-05T05:04:26.829Zresolver v4:
5702023-12-05T05:04:26.829Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5712023-12-05T05:04:26.829Zresolver v6:
5722023-12-05T05:04:26.829Znat v4:
5732023-12-05T05:04:26.829Znat_v6:
5742023-12-05T05:04:26.829Zport_mac:
5752023-12-05T05:04:26.829Zproxy arp:
5762023-12-05T05:04:26.829Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5772023-12-05T05:04:26.853Z+ success 'softnpu zone exists'
5782023-12-05T05:04:26.853Z+ set +x
5792023-12-05T05:04:26.853Zsoftnpu zone exists
5802023-12-05T05:04:26.854Z
5812023-12-05T05:04:26.854Zreal 20.084492371
5822023-12-05T05:04:26.854Zuser 0.701487075
5832023-12-05T05:04:26.854Zsys 1.612891440
5842023-12-05T05:04:26.854Ztrap 0.000102307
5852023-12-05T05:04:26.854Ztflt 0.000144636
5862023-12-05T05:04:26.854Zdflt 0.001229828
5872023-12-05T05:04:26.854Zkflt 0.000000000
5882023-12-05T05:04:26.854Zlock 15:21.175019831
5892023-12-05T05:04:26.854Zslp 59.944394883
5902023-12-05T05:04:26.854Zlat 0.081166981
5912023-12-05T05:04:26.854Zstop 0.366719070
5922023-12-05T05:04:26.854Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5932023-12-05T05:04:26.878Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5942023-12-05T05:04:26.879Z+ SILO_NAME=recovery
5952023-12-05T05:04:26.879Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5962023-12-05T05:04:26.879Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5972023-12-05T05:04:26.879Z+ sed -i~ '
5982023-12-05T05:04:26.879Z /^external_dns_ips/c\
5992023-12-05T05:04:26.879Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6002023-12-05T05:04:26.879Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6012023-12-05T05:04:26.879Z /^first/c\
6022023-12-05T05:04:26.879Zfirst = "10.151.2.164"
6032023-12-05T05:04:26.879Z /^last/c\
6042023-12-05T05:04:26.879Zlast = "10.151.2.173"
6052023-12-05T05:04:26.879Z }
6062023-12-05T05:04:26.879Z /^\[rack_network_config/,/^$/ {
6072023-12-05T05:04:26.879Z /^infra_ip_first/c\
6082023-12-05T05:04:26.879Zinfra_ip_first = "10.151.2.174"
6092023-12-05T05:04:26.879Z /^infra_ip_last/c\
6102023-12-05T05:04:26.879Zinfra_ip_last = "10.151.2.174"
6112023-12-05T05:04:26.879Z }
6122023-12-05T05:04:26.879Z /^\[\[rack_network_config.ports/,/^$/ {
6132023-12-05T05:04:26.879Z /^routes/c\
6142023-12-05T05:04:26.879Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6152023-12-05T05:04:26.881Z /^addresses/c\
6162023-12-05T05:04:26.881Zaddresses = \["10.151.2.174/32"\]
6172023-12-05T05:04:26.881Z }
6182023-12-05T05:04:26.881Z' pkg/config-rss.toml
6192023-12-05T05:04:26.906Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6202023-12-05T05:04:26.906Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6212023-12-05T05:04:26.906Z+++ pkg/config-rss.toml Tue Dec 5 05:04:26 2023
6222023-12-05T05:04:26.906Z@@ -41,7 +41,7 @@
6232023-12-05T05:04:26.906Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6242023-12-05T05:04:26.906Z # must be contained in one of the "internal services" IP Pool ranges listed
6252023-12-05T05:04:26.906Z # below.
6262023-12-05T05:04:26.906Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6272023-12-05T05:04:26.906Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6282023-12-05T05:04:26.906Z
6292023-12-05T05:04:26.906Z # Initial TLS certificates for the external API
6302023-12-05T05:04:26.906Z #
6312023-12-05T05:04:26.906Z@@ -77,8 +77,8 @@
6322023-12-05T05:04:26.906Z #
6332023-12-05T05:04:26.906Z # For more on this and what to put here, see docs/how-to-run.adoc.
6342023-12-05T05:04:26.906Z [[internal_services_ip_pool_ranges]]
6352023-12-05T05:04:26.906Z-first = "192.168.1.20"
6362023-12-05T05:04:26.906Z-last = "192.168.1.29"
6372023-12-05T05:04:26.906Z+first = "10.151.2.164"
6382023-12-05T05:04:26.906Z+last = "10.151.2.173"
6392023-12-05T05:04:26.906Z
6402023-12-05T05:04:26.906Z # TODO - this configuration is subject to change going forward. Ultimately these
6412023-12-05T05:04:26.906Z # parameters should be provided to the control plane via wicket, but we need to
6422023-12-05T05:04:26.906Z@@ -92,8 +92,8 @@
6432023-12-05T05:04:26.906Z # A range of IP addresses used by Boundary Services on the external network. In
6442023-12-05T05:04:26.906Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6452023-12-05T05:04:26.906Z # With softnpu, only one address is used.
6462023-12-05T05:04:26.906Z-infra_ip_first = "192.168.1.30"
6472023-12-05T05:04:26.906Z-infra_ip_last = "192.168.1.30"
6482023-12-05T05:04:26.906Z+infra_ip_first = "10.151.2.174"
6492023-12-05T05:04:26.907Z+infra_ip_last = "10.151.2.174"
6502023-12-05T05:04:26.907Z
6512023-12-05T05:04:26.907Z # Configurations for BGP routers to run on the scrimlets.
6522023-12-05T05:04:26.907Z bgp = []
6532023-12-05T05:04:26.907Z@@ -101,9 +101,9 @@
6542023-12-05T05:04:26.907Z # You can configure multiple uplinks by repeating the following stanza
6552023-12-05T05:04:26.907Z [[rack_network_config.ports]]
6562023-12-05T05:04:26.907Z # Routes associated with this port.
6572023-12-05T05:04:26.907Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6582023-12-05T05:04:26.907Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6592023-12-05T05:04:26.907Z # Addresses associated with this port.
6602023-12-05T05:04:26.907Z-addresses = ["192.168.1.30/32"]
6612023-12-05T05:04:26.907Z+addresses = ["10.151.2.174/32"]
6622023-12-05T05:04:26.907Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6632023-12-05T05:04:26.907Z port = "qsfp0"
6642023-12-05T05:04:26.907Z # The speed of this port.
6652023-12-05T05:04:26.907Z+ true
6662023-12-05T05:04:26.907Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6672023-12-05T05:04:26.907Za pkg/config-rss.toml 7K
6682023-12-05T05:04:26.907Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6692023-12-05T05:04:26.907Z+ TLS_NAME=recovery.sys.oxide.test
6702023-12-05T05:04:26.907Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6712023-12-05T05:04:27.061Z.....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+..+......+............+.+..............+...+....+.....+...+.......+.....+.+..+...+...+......+...+.......+...+........+...+......+.+.................+.......+...+...+..+.+.....+....+......+.....+.......+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.........+...+.....+..........+.....+...+............+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-05T05:04:27.308Z.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.................+.....+.......+.....+..................+...............+...+.+...+..+.........+......+....+.........+.....+................+......+...............+......+....................+.............+..+..........+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6732023-12-05T05:04:27.332Z-----
6742023-12-05T05:04:27.332Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6752023-12-05T05:04:27.356Za pkg/initial-tls-cert.pem 2K
6762023-12-05T05:04:27.356Za pkg/initial-tls-key.pem 4K
6772023-12-05T05:04:27.356Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6782023-12-05T05:04:27.356Z+ rmdir pkg
6792023-12-05T05:04:27.356Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6802023-12-05T05:04:27.356Z+ pfexec mkdir -p /usr/oxide
6812023-12-05T05:04:27.356Z+ pfexec rm -f /usr/oxide/tcpproxy
6822023-12-05T05:04:27.356Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6832023-12-05T05:04:27.643Z+ pfexec chmod +x /usr/oxide/tcpproxy
6842023-12-05T05:04:27.667Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6852023-12-05T05:04:27.667Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6862023-12-05T05:04:27.667Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6872023-12-05T05:04:27.772Z+ OMICRON_NO_UNINSTALL=1
6882023-12-05T05:04:27.772Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6892023-12-05T05:04:27.796ZDec 05 05:04:27.419 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6902023-12-05T05:04:27.796ZDec 05 05:04:27.421 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6912023-12-05T05:04:27.796ZDec 05 05:04:27.421 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6922023-12-05T05:04:27.796ZDec 05 05:04:27.421 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6932023-12-05T05:04:27.796ZDec 05 05:04:27.421 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6942023-12-05T05:04:27.796ZDec 05 05:04:27.421 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6952023-12-05T05:04:27.796ZDec 05 05:04:27.421 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6962023-12-05T05:04:27.796ZDec 05 05:04:27.421 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6972023-12-05T05:04:27.796ZDec 05 05:04:27.421 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6982023-12-05T05:04:27.796ZDec 05 05:04:27.421 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6992023-12-05T05:04:27.796ZDec 05 05:04:27.421 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7002023-12-05T05:04:27.796ZDec 05 05:04:27.421 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7012023-12-05T05:04:27.796ZDec 05 05:04:27.421 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7022023-12-05T05:04:27.796ZDec 05 05:04:27.421 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7032023-12-05T05:04:27.796ZDec 05 05:04:27.421 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7042023-12-05T05:04:27.796ZDec 05 05:04:27.421 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7052023-12-05T05:04:27.797ZDec 05 05:04:27.422 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7062023-12-05T05:04:28.044ZDec 05 05:04:27.674 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7072023-12-05T05:04:28.079ZDec 05 05:04:27.710 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7082023-12-05T05:04:28.147ZDec 05 05:04:27.778 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7092023-12-05T05:04:28.276Z
7102023-12-05T05:04:28.300Zreal 0.498873624
7112023-12-05T05:04:28.300Zuser 0.098571207
7122023-12-05T05:04:28.300Zsys 2.113050975
7132023-12-05T05:04:28.300Ztrap 0.000005840
7142023-12-05T05:04:28.300Ztflt 0.000600369
7152023-12-05T05:04:28.300Zdflt 0.000411367
7162023-12-05T05:04:28.300Zkflt 0.000000000
7172023-12-05T05:04:28.300Zlock 44.651945284
7182023-12-05T05:04:28.300Zslp 0.708610053
7192023-12-05T05:04:28.300Zlat 0.003023896
7202023-12-05T05:04:28.300Zstop 0.212239554
7212023-12-05T05:04:28.301Z+ retry=0
7222023-12-05T05:04:28.301Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232023-12-05T05:04:28.301Z+ [[ 0 -gt 30 ]]
7242023-12-05T05:04:28.301Z+ sleep 1
7252023-12-05T05:04:29.290Z+ retry=1
7262023-12-05T05:04:29.290Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272023-12-05T05:04:29.314Z+ [[ 1 -gt 30 ]]
7282023-12-05T05:04:29.314Z+ sleep 1
7292023-12-05T05:04:30.305Z+ retry=2
7302023-12-05T05:04:30.305Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312023-12-05T05:04:30.330Z+ [[ 2 -gt 30 ]]
7322023-12-05T05:04:30.330Z+ sleep 1
7332023-12-05T05:04:31.319Z+ retry=3
7342023-12-05T05:04:31.319Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352023-12-05T05:04:31.343Z+ [[ 3 -gt 30 ]]
7362023-12-05T05:04:31.343Z+ sleep 1
7372023-12-05T05:04:32.334Z+ retry=4
7382023-12-05T05:04:32.334Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7392023-12-05T05:04:57.511Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7402023-12-05T05:04:57.541Zlocal v6:
7412023-12-05T05:04:57.541Zfd00:99::1
7422023-12-05T05:04:57.541Zfe80::10fd:74ff:fe98:6490
7432023-12-05T05:04:57.541Zlocal v4:
7442023-12-05T05:04:57.541Z10.151.2.174
7452023-12-05T05:04:57.541Zrouter v6:
7462023-12-05T05:04:57.541Zfd00:1122:3344:101::/64 -> fe80::e4b2:cbff:fe5d:200b (1)
7472023-12-05T05:04:57.541Zfd00:1122:3344:2::/64 -> fe80::e4b2:cbff:fe5d:200b (1)
7482023-12-05T05:04:57.541Zfd00:1122:3344:3::/64 -> fe80::e4b2:cbff:fe5d:200b (1)
7492023-12-05T05:04:57.541Zfd00:1122:3344:1::/64 -> fe80::e4b2:cbff:fe5d:200b (1)
7502023-12-05T05:04:57.541Zfdb0:18c0:4d0d:a02a::/64 -> fe80::e4b2:cbff:fe5d:200b (1)
7512023-12-05T05:04:57.541Zrouter v4_idx:
7522023-12-05T05:04:57.541Z0.0.0.0/0 -> 0
7532023-12-05T05:04:57.541Zrouter v4_routes:
7542023-12-05T05:04:57.565Z0 -> 10.151.2.100 (2)
7552023-12-05T05:04:57.565Zresolver v4:
7562023-12-05T05:04:57.565Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7572023-12-05T05:04:57.567Z10.151.2.174 -> e6:f0:0a:c1:f8:1f
7582023-12-05T05:04:57.567Zresolver v6:
7592023-12-05T05:04:57.567Zfe80::e4b2:cbff:fe5d:200b -> e6:b2:cb:5d:20:0b
7602023-12-05T05:04:57.567Zfe80::10fd:74ff:fe98:6490 -> 12:fd:74:98:64:90
7612023-12-05T05:04:57.567Znat v4:
7622023-12-05T05:04:57.567Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:86:04
7632023-12-05T05:04:57.567Znat_v6:
7642023-12-05T05:04:57.567Zport_mac:
7652023-12-05T05:04:57.567Z1: a8:40:25:61:cf:90
7662023-12-05T05:04:57.567Z1000: a8:40:25:61:cf:8f
7672023-12-05T05:04:57.567Zproxy arp:
7682023-12-05T05:04:57.567Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7692023-12-05T05:04:57.567Z+ retry=0
7702023-12-05T05:04:57.567Z+++ zoneadm list -n
7712023-12-05T05:04:57.567Z+++ grep oxz_ntp
7722023-12-05T05:04:57.567Z++ pfexec svcs -z oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4 -Hostate oxide/ntp
7732023-12-05T05:04:57.595Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7742023-12-05T05:04:57.595Z++ true
7752023-12-05T05:04:57.595Z+ [[ '' != online ]]
7762023-12-05T05:04:57.595Z+ [[ 0 -gt 60 ]]
7772023-12-05T05:04:57.595Z+ sleep 1
7782023-12-05T05:04:58.584Z+ retry=1
7792023-12-05T05:04:58.608Z+++ zoneadm list -n
7802023-12-05T05:04:58.608Z+++ grep oxz_ntp
7812023-12-05T05:04:58.608Z++ pfexec svcs -z oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4 -Hostate oxide/ntp
7822023-12-05T05:04:58.632Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7832023-12-05T05:04:58.633Z++ true
7842023-12-05T05:04:58.633Z+ [[ '' != online ]]
7852023-12-05T05:04:58.633Z+ [[ 1 -gt 60 ]]
7862023-12-05T05:04:58.633Z+ sleep 1
7872023-12-05T05:04:59.613Z+ retry=2
7882023-12-05T05:04:59.637Z+++ zoneadm list -n
7892023-12-05T05:04:59.637Z+++ grep oxz_ntp
7902023-12-05T05:04:59.637Z++ pfexec svcs -z oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4 -Hostate oxide/ntp
7912023-12-05T05:04:59.661Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7922023-12-05T05:04:59.661Z++ true
7932023-12-05T05:04:59.661Z+ [[ '' != online ]]
7942023-12-05T05:04:59.661Z+ [[ 2 -gt 60 ]]
7952023-12-05T05:04:59.661Z+ sleep 1
7962023-12-05T05:05:00.642Z+ retry=3
7972023-12-05T05:05:00.669Z+++ zoneadm list -n
7982023-12-05T05:05:00.669Z+++ grep oxz_ntp
7992023-12-05T05:05:00.669Z++ pfexec svcs -z oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4 -Hostate oxide/ntp
8002023-12-05T05:05:00.669Z+ [[ disabled != online ]]
8012023-12-05T05:05:00.670Z+ [[ 3 -gt 60 ]]
8022023-12-05T05:05:00.670Z+ sleep 1
8032023-12-05T05:05:01.665Z+ retry=4
8042023-12-05T05:05:01.689Z+++ zoneadm list -n
8052023-12-05T05:05:01.690Z+++ grep oxz_ntp
8062023-12-05T05:05:01.690Z++ pfexec svcs -z oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4 -Hostate oxide/ntp
8072023-12-05T05:05:01.691Z+ [[ disabled != online ]]
8082023-12-05T05:05:01.691Z+ [[ 4 -gt 60 ]]
8092023-12-05T05:05:01.691Z+ sleep 1
8102023-12-05T05:05:02.686Z+ retry=5
8112023-12-05T05:05:02.711Z+++ zoneadm list -n
8122023-12-05T05:05:02.711Z+++ grep oxz_ntp
8132023-12-05T05:05:02.711Z++ pfexec svcs -z oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4 -Hostate oxide/ntp
8142023-12-05T05:05:02.711Z+ [[ disabled != online ]]
8152023-12-05T05:05:02.711Z+ [[ 5 -gt 60 ]]
8162023-12-05T05:05:02.711Z+ sleep 1
8172023-12-05T05:05:03.708Z+ retry=6
8182023-12-05T05:05:03.733Z+++ zoneadm list -n
8192023-12-05T05:05:03.733Z+++ grep oxz_ntp
8202023-12-05T05:05:03.733Z++ pfexec svcs -z oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4 -Hostate oxide/ntp
8212023-12-05T05:05:03.733Z+ [[ disabled != online ]]
8222023-12-05T05:05:03.733Z+ [[ 6 -gt 60 ]]
8232023-12-05T05:05:03.733Z+ sleep 1
8242023-12-05T05:05:04.731Z+ retry=7
8252023-12-05T05:05:04.755Z+++ zoneadm list -n
8262023-12-05T05:05:04.755Z+++ grep oxz_ntp
8272023-12-05T05:05:04.755Z++ pfexec svcs -z oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4 -Hostate oxide/ntp
8282023-12-05T05:05:04.755Z+ [[ disabled != online ]]
8292023-12-05T05:05:04.755Z+ [[ 7 -gt 60 ]]
8302023-12-05T05:05:04.755Z+ sleep 1
8312023-12-05T05:05:05.754Z+ retry=8
8322023-12-05T05:05:05.778Z+++ zoneadm list -n
8332023-12-05T05:05:05.778Z+++ grep oxz_ntp
8342023-12-05T05:05:05.778Z++ pfexec svcs -z oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4 -Hostate oxide/ntp
8352023-12-05T05:05:05.778Z+ [[ disabled != online ]]
8362023-12-05T05:05:05.778Z+ [[ 8 -gt 60 ]]
8372023-12-05T05:05:05.778Z+ sleep 1
8382023-12-05T05:05:06.775Z+ retry=9
8392023-12-05T05:05:06.799Z+++ zoneadm list -n
8402023-12-05T05:05:06.799Z+++ grep oxz_ntp
8412023-12-05T05:05:06.799Z++ pfexec svcs -z oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4 -Hostate oxide/ntp
8422023-12-05T05:05:06.799Z+ [[ disabled != online ]]
8432023-12-05T05:05:06.799Z+ [[ 9 -gt 60 ]]
8442023-12-05T05:05:06.799Z+ sleep 1
8452023-12-05T05:05:07.798Z+ retry=10
8462023-12-05T05:05:07.832Z+++ zoneadm list -n
8472023-12-05T05:05:07.832Z+++ grep oxz_ntp
8482023-12-05T05:05:07.832Z++ pfexec svcs -z oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4 -Hostate oxide/ntp
8492023-12-05T05:05:07.832Z+ [[ disabled != online ]]
8502023-12-05T05:05:07.832Z+ [[ 10 -gt 60 ]]
8512023-12-05T05:05:07.832Z+ sleep 1
 
8522023-12-05T05:05:07.867Zsock console login: Dec 5 05:05:07 sok ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8532023-12-05T05:05:08.821Z+ retry=11
8542023-12-05T05:05:08.845Z+++ zoneadm list -n
8552023-12-05T05:05:08.845Z+++ grep oxz_ntp
8562023-12-05T05:05:08.845Z++ pfexec svcs -z oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4 -Hostate oxide/ntp
8572023-12-05T05:05:08.845Z+ [[ online != online ]]
8582023-12-05T05:05:08.845Z+ echo 'Waited for chrony: 11s'
8592023-12-05T05:05:08.845Z+ export RUST_BACKTRACE=1
8602023-12-05T05:05:08.845ZWaited for chrony: 11s
8612023-12-05T05:05:08.845Z+ RUST_BACKTRACE=1
8622023-12-05T05:05:08.845Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8632023-12-05T05:05:08.846Z+ ./tests/bootstrap
8642023-12-05T05:05:08.870ZUsing external DNS server at 10.151.2.164:53
8652023-12-05T05:05:08.870Z2023-12-05 05:05:08.499421401 UTC: attempting to log into API
8662023-12-05T05:05:23.905Z2023-12-05 05:05:24.004028522 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8672023-12-05T05:05:24.906Z2023-12-05 05:05:25.004759598 UTC: attempting to log into API
8682023-12-05T05:05:39.939Z2023-12-05 05:05:40.037564990 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8692023-12-05T05:05:40.941Z2023-12-05 05:05:41.039213420 UTC: attempting to log into API
8702023-12-05T05:05:55.974Z2023-12-05 05:05:56.072783315 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8712023-12-05T05:05:56.976Z2023-12-05 05:05:57.074416445 UTC: attempting to log into API
8722023-12-05T05:06:12.009Z2023-12-05 05:06:12.107147917 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8732023-12-05T05:06:13.009Z2023-12-05 05:06:13.107855862 UTC: attempting to log into API
8742023-12-05T05:06:28.043Z2023-12-05 05:06:27.849775935 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8752023-12-05T05:06:29.044Z2023-12-05 05:06:28.854163207 UTC: attempting to log into API
8762023-12-05T05:06:44.077Z2023-12-05 05:06:43.927680021 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8772023-12-05T05:06:45.078Z2023-12-05 05:06:44.930944125 UTC: attempting to log into API
8782023-12-05T05:07:00.111Z2023-12-05 05:07:00.004525446 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8792023-12-05T05:07:01.113Z2023-12-05 05:07:01.008978195 UTC: attempting to log into API
8802023-12-05T05:07:16.153Z2023-12-05 05:07:16.082559512 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8812023-12-05T05:07:17.146Z2023-12-05 05:07:17.085827629 UTC: attempting to log into API
8822023-12-05T05:07:32.184Z2023-12-05 05:07:32.159042956 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8832023-12-05T05:07:33.180Z2023-12-05 05:07:33.162513816 UTC: attempting to log into API
8842023-12-05T05:07:48.211Z2023-12-05 05:07:48.199334310 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8852023-12-05T05:07:49.212Z2023-12-05 05:07:49.200203779 UTC: attempting to log into API
8862023-12-05T05:08:04.245Z2023-12-05 05:08:04.232546668 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8872023-12-05T05:08:05.248Z2023-12-05 05:08:05.234277210 UTC: attempting to log into API
8882023-12-05T05:08:20.286Z2023-12-05 05:08:20.272524135 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8892023-12-05T05:08:21.286Z2023-12-05 05:08:21.273268761 UTC: attempting to log into API
 
8902023-12-05T05:08:33.406ZDec 5 05:08:33 sok ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8912023-12-05T05:08:36.331Z2023-12-05 05:08:36.315934276 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8922023-12-05T05:08:37.334Z2023-12-05 05:08:37.316977648 UTC: attempting to log into API
 
8932023-12-05T05:08:42.194ZDec 5 05:08:41 sok ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
8942023-12-05T05:08:48.646ZDec 5 05:08:48 sok ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
8952023-12-05T05:08:52.364Z2023-12-05 05:08:52.350489523 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8962023-12-05T05:08:53.365Z2023-12-05 05:08:53.351461043 UTC: attempting to log into API
 
8972023-12-05T05:08:54.797ZDec 5 05:08:54 sok ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
8982023-12-05T05:09:00.129ZDec 5 05:08:59 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
8992023-12-05T05:09:08.403Z2023-12-05 05:09:08.385002530 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9002023-12-05T05:09:09.402Z2023-12-05 05:09:09.385832407 UTC: attempting to log into API
9012023-12-05T05:09:24.435Z2023-12-05 05:09:24.418857726 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9022023-12-05T05:09:25.842Z2023-12-05 05:09:25.419738054 UTC: attempting to log into API
9032023-12-05T05:09:26.244Z2023-12-05 05:09:26.204527977 UTC: login succeeded
9042023-12-05T05:09:26.269Zwaiting for nexus to come up...
9052023-12-05T05:09:26.321Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9062023-12-05T05:09:26.373Zensuring datasets are ready...
9072023-12-05T05:09:28.866Zlet's roll.
9082023-12-05T05:09:28.891Z+ rm ./tests/bootstrap
9092023-12-05T05:09:28.935Z+ for test_bin in tests/*
9102023-12-05T05:09:28.935Z+ ./tests/bootstrap-83d41f1b41d1fb89
9112023-12-05T05:09:28.961Z
9122023-12-05T05:09:28.962Zrunning 0 tests
9132023-12-05T05:09:28.962Z
9142023-12-05T05:09:28.962Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9152023-12-05T05:09:28.962Z
9162023-12-05T05:09:28.962Z+ for test_bin in tests/*
9172023-12-05T05:09:28.962Z+ ./tests/end_to_end_tests-8c93950b566420c7
9182023-12-05T05:09:28.962Z
9192023-12-05T05:09:28.962Zrunning 2 tests
9202023-12-05T05:09:29.184Ztest no_spoof::no_spoof_here ... ok
9212023-12-05T05:10:28.956Ztest instance_launch::instance_launch has been running for over 60 seconds
9222023-12-05T05:12:26.138Ztest instance_launch::instance_launch ... ok
9232023-12-05T05:12:26.138Z
9242023-12-05T05:12:26.138Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 177.18s
9252023-12-05T05:12:26.138Z
9262023-12-05T05:12:26.162Z+ _exit_trap
9272023-12-05T05:12:26.162Z+ local status=0
9282023-12-05T05:12:26.162Z+ [[ 0 -eq 0 ]]
9292023-12-05T05:12:26.162Z+ exit 0
9302023-12-05T05:12:26.162Zprocess exited: duration 507328 ms, exit code 0
 
9312023-12-05T05:12:26.647Zfound 47 output files
9322023-12-05T05:12:26.647Zuploading: /var/svc/log/oxide-sled-agent:default.log (769558 bytes)
9332023-12-05T05:12:29.296Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9342023-12-05T05:12:29.296Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7f279962-27c0-4f24-8072-0ded7f5eb637/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
9352023-12-05T05:12:30.402Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7f279962-27c0-4f24-8072-0ded7f5eb637/root/var/svc/log/oxide-crucible-agent:default.log
9362023-12-05T05:12:30.402Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7f279962-27c0-4f24-8072-0ded7f5eb637/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9372023-12-05T05:12:31.527Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7f279962-27c0-4f24-8072-0ded7f5eb637/root/var/svc/log/oxide-crucible-downstairs:default.log
9382023-12-05T05:12:31.527Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4/root/var/svc/log/oxide-ntp:default.log (1260 bytes)
9392023-12-05T05:12:33.061Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4/root/var/svc/log/oxide-ntp:default.log
9402023-12-05T05:12:33.061Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a51678db-3c7e-4874-acaf-9b8c6ea7cb7f/root/var/svc/log/oxide-crucible-agent:default.log (13691 bytes)
9412023-12-05T05:12:35.055Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a51678db-3c7e-4874-acaf-9b8c6ea7cb7f/root/var/svc/log/oxide-crucible-agent:default.log
9422023-12-05T05:12:35.055Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a51678db-3c7e-4874-acaf-9b8c6ea7cb7f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9432023-12-05T05:12:35.870Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a51678db-3c7e-4874-acaf-9b8c6ea7cb7f/root/var/svc/log/oxide-crucible-downstairs:default.log
9442023-12-05T05:12:35.870Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a51678db-3c7e-4874-acaf-9b8c6ea7cb7f/root/var/svc/log/oxide-crucible-downstairs:downstairs-6e5b97b0-7874-4f6e-8697-0f07ae437ebf.log (5535 bytes)
9452023-12-05T05:12:37.029Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a51678db-3c7e-4874-acaf-9b8c6ea7cb7f/root/var/svc/log/oxide-crucible-downstairs:downstairs-6e5b97b0-7874-4f6e-8697-0f07ae437ebf.log
9462023-12-05T05:12:37.029Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_feb76d41-816e-4b57-b5fa-e306dfd86cd3/root/var/svc/log/oxide-crucible-agent:default.log (13685 bytes)
9472023-12-05T05:12:38.167Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_feb76d41-816e-4b57-b5fa-e306dfd86cd3/root/var/svc/log/oxide-crucible-agent:default.log
9482023-12-05T05:12:38.167Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_feb76d41-816e-4b57-b5fa-e306dfd86cd3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9492023-12-05T05:12:39.259Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_feb76d41-816e-4b57-b5fa-e306dfd86cd3/root/var/svc/log/oxide-crucible-downstairs:default.log
9502023-12-05T05:12:39.259Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_feb76d41-816e-4b57-b5fa-e306dfd86cd3/root/var/svc/log/oxide-crucible-downstairs:downstairs-4e09e1b5-93eb-4f00-853c-f202518046cc.log (3051 bytes)
9512023-12-05T05:12:40.372Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_feb76d41-816e-4b57-b5fa-e306dfd86cd3/root/var/svc/log/oxide-crucible-downstairs:downstairs-4e09e1b5-93eb-4f00-853c-f202518046cc.log
9522023-12-05T05:12:40.372Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_e0d104fc-ca1c-4308-a389-5069dc37d95b/root/var/svc/log/oxide-internal_dns:default.log (101107 bytes)
9532023-12-05T05:12:41.654Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_e0d104fc-ca1c-4308-a389-5069dc37d95b/root/var/svc/log/oxide-internal_dns:default.log
9542023-12-05T05:12:41.654Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_903cda51-87eb-4b32-86c4-e883df17b263/root/var/svc/log/oxide-nexus:default.log (533135 bytes)
9552023-12-05T05:12:42.776Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_903cda51-87eb-4b32-86c4-e883df17b263/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 533135 -> 539408
9562023-12-05T05:12:43.847Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_903cda51-87eb-4b32-86c4-e883df17b263/root/var/svc/log/oxide-nexus:default.log
9572023-12-05T05:12:43.847Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_340ad134-f72a-4c54-9416-c27bb599507c/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
9582023-12-05T05:12:44.945Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_340ad134-f72a-4c54-9416-c27bb599507c/root/var/svc/log/oxide-cockroachdb:default.log
9592023-12-05T05:12:44.945Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_a0360a65-7a82-44ef-8fd7-b03c015bf9f8/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
9602023-12-05T05:12:46.043Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_a0360a65-7a82-44ef-8fd7-b03c015bf9f8/root/var/svc/log/oxide-cockroachdb:default.log
9612023-12-05T05:12:46.043Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_4fd38e3a-adb8-4422-8c33-e1197f737dc0/root/var/svc/log/oxide-nexus:default.log (4898449 bytes)
9622023-12-05T05:12:49.528Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_4fd38e3a-adb8-4422-8c33-e1197f737dc0/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4898449 -> 4904725
9632023-12-05T05:12:50.595Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_4fd38e3a-adb8-4422-8c33-e1197f737dc0/root/var/svc/log/oxide-nexus:default.log
9642023-12-05T05:12:50.595Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_bff6cf57-b2fe-4c61-94b1-605a87d7faa9/root/var/svc/log/oxide-nexus:default.log (570826 bytes)
9652023-12-05T05:12:51.134Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_bff6cf57-b2fe-4c61-94b1-605a87d7faa9/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 570826 -> 577709
9662023-12-05T05:12:52.205Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_bff6cf57-b2fe-4c61-94b1-605a87d7faa9/root/var/svc/log/oxide-nexus:default.log
9672023-12-05T05:12:52.206Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_3d1f23a7-05ef-4306-b3bc-9436f6e77520/root/var/svc/log/oxide-oximeter:default.log (153675 bytes)
9682023-12-05T05:12:52.428Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_3d1f23a7-05ef-4306-b3bc-9436f6e77520/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 153675 -> 165452
9692023-12-05T05:12:53.497Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_3d1f23a7-05ef-4306-b3bc-9436f6e77520/root/var/svc/log/oxide-oximeter:default.log
9702023-12-05T05:12:53.497Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_44d685ea-4630-44c8-81f2-44586581d1cd/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9712023-12-05T05:12:54.619Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_44d685ea-4630-44c8-81f2-44586581d1cd/root/var/svc/log/oxide-crucible-agent:default.log
9722023-12-05T05:12:54.619Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_44d685ea-4630-44c8-81f2-44586581d1cd/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9732023-12-05T05:12:55.690Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_44d685ea-4630-44c8-81f2-44586581d1cd/root/var/svc/log/oxide-crucible-downstairs:default.log
9742023-12-05T05:12:55.690Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_c9e88858-15f1-494c-8021-3748893562c7/root/var/svc/log/oxide-crucible-pantry:default.log (2337 bytes)
9752023-12-05T05:12:56.782Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_c9e88858-15f1-494c-8021-3748893562c7/root/var/svc/log/oxide-crucible-pantry:default.log
9762023-12-05T05:12:56.782Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_6cb9a679-dafa-44c0-b151-529f455cbf39/root/var/svc/log/oxide-internal_dns:default.log (103129 bytes)
9772023-12-05T05:12:58.005Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_6cb9a679-dafa-44c0-b151-529f455cbf39/root/var/svc/log/oxide-internal_dns:default.log
9782023-12-05T05:12:58.005Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_aebce85f-8b17-4c9a-aedc-556f9944b882/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
9792023-12-05T05:12:59.099Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_aebce85f-8b17-4c9a-aedc-556f9944b882/root/var/svc/log/oxide-cockroachdb:default.log
9802023-12-05T05:12:59.099Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d2a2c404-d5ae-402d-b852-b79d6b22d2ae/root/var/svc/log/oxide-crucible-agent:default.log (14579 bytes)
9812023-12-05T05:13:00.213Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d2a2c404-d5ae-402d-b852-b79d6b22d2ae/root/var/svc/log/oxide-crucible-agent:default.log
9822023-12-05T05:13:00.237Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d2a2c404-d5ae-402d-b852-b79d6b22d2ae/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9832023-12-05T05:13:01.308Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d2a2c404-d5ae-402d-b852-b79d6b22d2ae/root/var/svc/log/oxide-crucible-downstairs:default.log
9842023-12-05T05:13:01.313Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d2a2c404-d5ae-402d-b852-b79d6b22d2ae/root/var/svc/log/oxide-crucible-downstairs:downstairs-b4013600-169b-46df-bbd8-dec7be6445a7.log (3050 bytes)
9852023-12-05T05:13:02.404Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d2a2c404-d5ae-402d-b852-b79d6b22d2ae/root/var/svc/log/oxide-crucible-downstairs:downstairs-b4013600-169b-46df-bbd8-dec7be6445a7.log
9862023-12-05T05:13:02.404Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_89d65e32-8f5e-4bac-9654-1713b749f8f7/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
9872023-12-05T05:13:03.500Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_89d65e32-8f5e-4bac-9654-1713b749f8f7/root/var/svc/log/oxide-crucible-pantry:default.log
9882023-12-05T05:13:03.500Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_060b4bc0-1885-4f01-baf6-8beb679c449f/root/var/svc/log/oxide-external_dns:default.log (15612 bytes)
9892023-12-05T05:13:03.591Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_060b4bc0-1885-4f01-baf6-8beb679c449f/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15612 -> 19387
9902023-12-05T05:13:04.674Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_060b4bc0-1885-4f01-baf6-8beb679c449f/root/var/svc/log/oxide-external_dns:default.log
9912023-12-05T05:13:04.674Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_8d1cafe1-6ec6-4228-bc6c-ba1b1acc2a92/root/var/svc/log/oxide-external_dns:default.log (15609 bytes)
9922023-12-05T05:13:04.763Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_8d1cafe1-6ec6-4228-bc6c-ba1b1acc2a92/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15609 -> 19384
9932023-12-05T05:13:05.835Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_8d1cafe1-6ec6-4228-bc6c-ba1b1acc2a92/root/var/svc/log/oxide-external_dns:default.log
9942023-12-05T05:13:05.835Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_81b30f58-91a1-4053-83db-a97199571ed1/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
9952023-12-05T05:13:06.980Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_81b30f58-91a1-4053-83db-a97199571ed1/root/var/svc/log/oxide-crucible-agent:default.log
9962023-12-05T05:13:06.980Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_81b30f58-91a1-4053-83db-a97199571ed1/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9972023-12-05T05:13:08.081Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_81b30f58-91a1-4053-83db-a97199571ed1/root/var/svc/log/oxide-crucible-downstairs:default.log
9982023-12-05T05:13:08.081Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_81b30f58-91a1-4053-83db-a97199571ed1/root/var/svc/log/oxide-crucible-downstairs:downstairs-b7c7c70d-2a47-4500-89e7-af9056667fb6.log (3050 bytes)
9992023-12-05T05:13:09.200Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_81b30f58-91a1-4053-83db-a97199571ed1/root/var/svc/log/oxide-crucible-downstairs:downstairs-b7c7c70d-2a47-4500-89e7-af9056667fb6.log
10002023-12-05T05:13:09.201Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_9c3bf8c9-7846-4d2d-af2d-e044e80492c5/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10012023-12-05T05:13:10.294Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_9c3bf8c9-7846-4d2d-af2d-e044e80492c5/root/var/svc/log/oxide-crucible-pantry:default.log
10022023-12-05T05:13:10.294Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_130ef4ad-e11c-4ab3-9a64-2af3dd5dc61f/root/var/svc/log/oxide-clickhouse:default.log (515519 bytes)
10032023-12-05T05:13:11.298Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_130ef4ad-e11c-4ab3-9a64-2af3dd5dc61f/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 515519 -> 584343
10042023-12-05T05:13:12.373Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_130ef4ad-e11c-4ab3-9a64-2af3dd5dc61f/root/var/svc/log/oxide-clickhouse:default.log
10052023-12-05T05:13:12.373Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0077dee3-3523-460f-8941-233ad420d4f1/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
10062023-12-05T05:13:13.472Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0077dee3-3523-460f-8941-233ad420d4f1/root/var/svc/log/oxide-cockroachdb:default.log
10072023-12-05T05:13:13.472Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_262edd9c-1fa9-409a-ab28-0f1ae145a3ab/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10082023-12-05T05:13:14.575Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_262edd9c-1fa9-409a-ab28-0f1ae145a3ab/root/var/svc/log/oxide-cockroachdb:default.log
10092023-12-05T05:13:14.575Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87870395-a0aa-4439-9d55-b4f0069ccb94/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10102023-12-05T05:13:16.488Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87870395-a0aa-4439-9d55-b4f0069ccb94/root/var/svc/log/oxide-crucible-agent:default.log
10112023-12-05T05:13:16.488Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87870395-a0aa-4439-9d55-b4f0069ccb94/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10122023-12-05T05:13:17.184Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87870395-a0aa-4439-9d55-b4f0069ccb94/root/var/svc/log/oxide-crucible-downstairs:default.log
10132023-12-05T05:13:17.184Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_999a9308-e9f0-490c-939e-e6029f15187c/root/var/svc/log/oxide-crucible-agent:default.log (14117 bytes)
10142023-12-05T05:13:18.351Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_999a9308-e9f0-490c-939e-e6029f15187c/root/var/svc/log/oxide-crucible-agent:default.log
10152023-12-05T05:13:18.375Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_999a9308-e9f0-490c-939e-e6029f15187c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10162023-12-05T05:13:19.468Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_999a9308-e9f0-490c-939e-e6029f15187c/root/var/svc/log/oxide-crucible-downstairs:default.log
10172023-12-05T05:13:19.468Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_999a9308-e9f0-490c-939e-e6029f15187c/root/var/svc/log/oxide-crucible-downstairs:downstairs-aaa19f59-33b4-4c6d-87c0-7ac3433135f8.log (5535 bytes)
10182023-12-05T05:13:20.594Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_999a9308-e9f0-490c-939e-e6029f15187c/root/var/svc/log/oxide-crucible-downstairs:downstairs-aaa19f59-33b4-4c6d-87c0-7ac3433135f8.log
10192023-12-05T05:13:20.594Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_feec2b16-334a-412a-ab80-9f1333641ea8/root/var/svc/log/oxide-crucible-agent:default.log (14118 bytes)
10202023-12-05T05:13:21.736Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_feec2b16-334a-412a-ab80-9f1333641ea8/root/var/svc/log/oxide-crucible-agent:default.log
10212023-12-05T05:13:21.736Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_feec2b16-334a-412a-ab80-9f1333641ea8/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10222023-12-05T05:13:22.831Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_feec2b16-334a-412a-ab80-9f1333641ea8/root/var/svc/log/oxide-crucible-downstairs:default.log
10232023-12-05T05:13:22.831Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_feec2b16-334a-412a-ab80-9f1333641ea8/root/var/svc/log/oxide-crucible-downstairs:downstairs-bec46b80-15a2-4e38-a867-5c2a7d8c9c42.log (5534 bytes)
10242023-12-05T05:13:23.946Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_feec2b16-334a-412a-ab80-9f1333641ea8/root/var/svc/log/oxide-crucible-downstairs:downstairs-bec46b80-15a2-4e38-a867-5c2a7d8c9c42.log
10252023-12-05T05:13:23.946Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_30b754fc-b101-4f89-9dcf-4760d2c4f883/root/var/svc/log/oxide-internal_dns:default.log (99072 bytes)
10262023-12-05T05:13:24.786Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_30b754fc-b101-4f89-9dcf-4760d2c4f883/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99072 -> 102828
10272023-12-05T05:13:25.857Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_30b754fc-b101-4f89-9dcf-4760d2c4f883/root/var/svc/log/oxide-internal_dns:default.log
10282023-12-05T05:13:25.857Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4/root/var/log/chrony/measurements.log (9590 bytes)
10292023-12-05T05:13:25.955Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4/root/var/log/chrony/measurements.log" changed size mid upload: 9590 -> 13289
10302023-12-05T05:13:27.026Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4/root/var/log/chrony/measurements.log
10312023-12-05T05:13:27.026Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4/root/var/log/chrony/statistics.log (6318 bytes)
10322023-12-05T05:13:27.095Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4/root/var/log/chrony/statistics.log" changed size mid upload: 6318 -> 8190
10332023-12-05T05:13:28.172Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4/root/var/log/chrony/statistics.log
10342023-12-05T05:13:28.172Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4/root/var/log/chrony/tracking.log (1854 bytes)
10352023-12-05T05:13:28.246Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4/root/var/log/chrony/tracking.log" changed size mid upload: 1854 -> 1986
10362023-12-05T05:13:29.318Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5d87c074-0e57-431e-8efd-e3989605fde4/root/var/log/chrony/tracking.log