01HFCX0MY0FT5CM2CVXV69BYYV: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HFCX192YS9RRW871AJM0V3WF

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-16T21:15:39.293Zjob dependencies complete; ready to run (waiting for 48 m 47 s)
22023-11-16T21:16:25.477Zjob assigned to worker 01HFCZTTBMAN2R6PMMV7Y93WD4 (queued for 46 s)
32023-11-16T21:16:25.607Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-11-16T21:16:25.655Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-11-16T21:16:25.689Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-11-16T21:16:25.722Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-16T21:16:25.779Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-11-16T21:16:25.812ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-16T21:16:25.845Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-16T21:16:25.879ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-16T21:16:25.913ZConfiguring devices.
122023-11-16T21:16:25.946Z##################################################
132023-11-16T21:16:25.980Z#### ############# ##
142023-11-16T21:16:26.014Z### ### ############ ##
152023-11-16T21:16:26.048Z## ### # ## ### ## Oxide Computer Company ##
162023-11-16T21:16:26.081Z## ## ## ### # ### ##
172023-11-16T21:16:26.115Z## # ### #### #### This Station Under ##
182023-11-16T21:16:26.149Z### ### #### # ### Computer Control ##
192023-11-16T21:16:26.183Z#### #### ### ## ##
202023-11-16T21:16:26.216Z##################################################
212023-11-16T21:16:26.249ZHostname: buskin
222023-11-16T21:16:26.283ZLoading smf(7) service descriptions:
232023-11-16T21:16:27.324Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-cd1d99075975e5dc.gz
242023-11-16T21:16:28.734Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-cd1d99075975e5dc.gz
252023-11-16T21:16:28.759Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-16T21:16:48.973Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-16T21:16:49.004Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d1c7f350d876f572.gz
282023-11-16T21:17:06.764Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d1c7f350d876f572.gz
292023-11-16T21:17:06.790Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-16T21:17:07.874Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-16T21:17:07.905Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-16T21:17:10.786Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-16T21:17:10.811Zdownloading input: /input/package/work/version.txt
342023-11-16T21:17:10.836Zdownloaded input: /input/package/work/version.txt
352023-11-16T21:17:10.861Zdownloading input: /input/package/work/package.tar.gz
362023-11-16T21:17:30.967Zdownloaded input: /input/package/work/package.tar.gz
372023-11-16T21:17:30.992Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-16T21:18:02.709Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-16T21:18:02.742Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-16T21:18:12.732Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-16T21:18:12.757Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-16T21:19:01.575Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-16T21:19:01.611Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-16T21:19:39.083Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-16T21:19:39.109Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-16T21:20:07.229Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-16T21:20:07.261Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-16T21:20:11.430Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-16T21:20:11.461Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-16T21:20:19.296Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-16T21:20:19.321Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-16T21:20:24.916Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-16T21:20:24.941Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-16T21:20:30.679Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-16T21:20:30.711Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-16T21:20:30.743Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-16T21:20:30.774Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-16T21:20:34.062Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-16T21:20:34.087Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-16T21:20:38.312Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-16T21:20:38.344Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-16T21:20:51.575Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-16T21:20:51.614Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-16T21:21:08.788Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-16T21:21:08.820Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-16T21:21:08.858Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-16T21:21:08.884Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-16T21:21:16.742Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-16T21:21:16.768Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-16T21:21:32.014Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-16T21:21:32.046Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-16T21:22:15.202Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-16T21:22:15.227Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-16T21:22:54.764Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-16T21:22:54.796Zstarting task 0: "setup"
762023-11-16T21:22:54.837Z++ uname -s
772023-11-16T21:22:54.870Z+ kern=SunOS
782023-11-16T21:22:54.902Z+ case "$kern" in
792023-11-16T21:22:54.934Z+ groupadd -g 12345 build
802023-11-16T21:22:54.966Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-16T21:22:56.850Z+ zfs create -o mountpoint=/work rpool/work
822023-11-16T21:22:56.887Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-16T21:22:56.923Z+ home_fs=zfs
842023-11-16T21:22:56.957Z+ [[ zfs == autofs ]]
852023-11-16T21:22:56.993Z+ mkdir -p /home/build
862023-11-16T21:22:57.030Z+ chown build:build /home/build /work
872023-11-16T21:22:57.879Z+ chmod 0700 /home/build /work
882023-11-16T21:22:57.911Zprocess exited: duration 3054 ms, exit code 0
 
892023-11-16T21:22:57.975Zstarting task 1: "authentication"
902023-11-16T21:22:58.065Zprocess exited: duration 52 ms, exit code 0
 
912023-11-16T21:22:58.137Zstarting task 2: "build"
922023-11-16T21:22:58.251Z+ trap _exit_trap EXIT
932023-11-16T21:22:58.292Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-16T21:22:58.380Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-16T21:22:58.419Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-16T21:22:58.456Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-16T21:22:58.493Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-16T21:22:58.531Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-16T21:22:58.565Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-16T21:22:58.596Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-16T21:22:58.628Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-16T21:22:58.660Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-16T21:22:58.691Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-16T21:22:58.723Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-16T21:22:58.755Z+ [[ -d /opt/oxide ]]
1062023-11-16T21:22:58.786Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-16T21:22:58.818Z+ pfexec mkdir /opt/oxide
1082023-11-16T21:22:58.850Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-16T21:22:58.883Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-16T21:22:58.918Z+ cd /opt/oxide-underneath
1112023-11-16T21:22:58.956Z+ pfexec tar ceEp@/f - .
1122023-11-16T21:22:59.437Z+ cd /opt/oxide
1132023-11-16T21:22:59.471Z+ pfexec tar xveEp@/f -
1142023-11-16T21:22:59.503Zx ., 0 bytes, 0 tape blocks
1152023-11-16T21:22:59.535Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-16T21:22:59.566Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-16T21:22:59.598Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-16T21:22:59.629Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-16T21:22:59.661Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-16T21:22:59.692Z+ pfexec mkdir /opt/oxide/work
1212023-11-16T21:22:59.725Z+ pfexec chown build:build /opt/oxide/work
1222023-11-16T21:22:59.762Z+ cd /opt/oxide/work
1232023-11-16T21:22:59.794Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-16T21:23:00.094ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-16T21:23:00.127Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-11-16T21:23:00.159Zx out/omicron-sled-agent.tar, 111574528 bytes, 217919 tape blocks
1272023-11-16T21:23:00.191Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-16T21:23:00.222Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-11-16T21:23:00.254Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-11-16T21:23:00.312Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-11-16T21:23:00.344Zx target/release/omicron-package, 42418736 bytes, 82850 tape blocks
1322023-11-16T21:23:00.376Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-11-16T21:23:00.408Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-16T21:23:00.440Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-16T21:23:00.472Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-16T21:23:00.503Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-16T21:23:00.539Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-16T21:23:00.578Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-16T21:23:00.610Z
1402023-11-16T21:23:00.641Zreal 1.552800193
1412023-11-16T21:23:00.673Zuser 1.386681777
1422023-11-16T21:23:00.705Zsys 0.286373315
1432023-11-16T21:23:00.736Ztrap 0.000064521
1442023-11-16T21:23:00.768Ztflt 0.000000000
1452023-11-16T21:23:00.800Zdflt 0.000000000
1462023-11-16T21:23:00.831Zkflt 0.000000000
1472023-11-16T21:23:00.863Zlock 0.000000000
1482023-11-16T21:23:00.894Zslp 1.395140438
1492023-11-16T21:23:00.938Zlat 0.035428474
1502023-11-16T21:23:00.970Zstop 0.000117863
1512023-11-16T21:23:01.002Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-16T21:23:01.168Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-16T21:23:01.201Z+ mkdir tests
1542023-11-16T21:23:01.234Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-16T21:23:01.269Z+ ptime -m gunzip
1562023-11-16T21:23:01.304Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-cd1d99075975e5dc
1572023-11-16T21:23:01.364Z
1582023-11-16T21:23:01.402Zreal 0.139326987
1592023-11-16T21:23:01.434Zuser 0.128689893
1602023-11-16T21:23:01.467Zsys 0.010396996
1612023-11-16T21:23:01.500Ztrap 0.000015820
1622023-11-16T21:23:01.534Ztflt 0.000000000
1632023-11-16T21:23:01.566Zdflt 0.000000000
1642023-11-16T21:23:01.599Zkflt 0.000000000
1652023-11-16T21:23:01.631Zlock 0.000000000
1662023-11-16T21:23:01.663Zslp 0.000020180
1672023-11-16T21:23:01.695Zlat 0.000017460
1682023-11-16T21:23:01.728Zstop 0.000120103
1692023-11-16T21:23:01.760Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-cd1d99075975e5dc
1702023-11-16T21:23:01.803Z+ chmod a+x tests/bootstrap-cd1d99075975e5dc
1712023-11-16T21:23:02.251Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-16T21:23:02.298Z+ ptime -m gunzip
1732023-11-16T21:23:02.398Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-16T21:23:03.334Z
1752023-11-16T21:23:03.366Zreal 1.995596991
1762023-11-16T21:23:03.415Zuser 1.858381282
1772023-11-16T21:23:03.447Zsys 0.136866332
1782023-11-16T21:23:03.479Ztrap 0.000128582
1792023-11-16T21:23:03.513Ztflt 0.000000000
1802023-11-16T21:23:03.545Zdflt 0.000000000
1812023-11-16T21:23:03.582Zkflt 0.000000000
1822023-11-16T21:23:03.615Zlock 0.000000000
1832023-11-16T21:23:03.647Zslp 0.000020610
1842023-11-16T21:23:03.680Zlat 0.000017560
1852023-11-16T21:23:03.713Zstop 0.000119093
1862023-11-16T21:23:03.745Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-16T21:23:03.777Z+ chmod a+x tests/bootstrap
1882023-11-16T21:23:03.809Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-16T21:23:03.842Z+ ptime -m gunzip
1902023-11-16T21:23:03.874Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d1c7f350d876f572
1912023-11-16T21:23:05.610Z
1922023-11-16T21:23:05.650Zreal 2.262203833
1932023-11-16T21:23:05.691Zuser 2.124727843
1942023-11-16T21:23:05.731Zsys 0.137085629
1952023-11-16T21:23:06.177Ztrap 0.000168075
1962023-11-16T21:23:06.215Ztflt 0.000000000
1972023-11-16T21:23:06.662Zdflt 0.000000000
1982023-11-16T21:23:06.695Zkflt 0.000000000
1992023-11-16T21:23:06.727Zlock 0.000000000
2002023-11-16T21:23:06.764Zslp 0.000024300
2012023-11-16T21:23:06.798Zlat 0.000016040
2022023-11-16T21:23:06.830Zstop 0.000116993
2032023-11-16T21:23:06.861Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d1c7f350d876f572
2042023-11-16T21:23:06.894Z+ chmod a+x tests/end_to_end_tests-d1c7f350d876f572
2052023-11-16T21:23:06.930Z+ bmat address ls
2062023-11-16T21:23:06.965ZNAME CIDR FIRST LAST COUNT
2072023-11-16T21:23:07.003Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-11-16T21:23:07.039Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-16T21:23:07.081Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-11-16T21:23:07.114Z+ EXTRA_IP_START=10.151.1.164
2112023-11-16T21:23:07.247Z+ EXTRA_IP_END=10.151.1.227
2122023-11-16T21:23:07.289Z+ EXTRA_IP_COUNT=64
2132023-11-16T21:23:07.332Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-16T21:23:07.417Z+ EXTRA_IP_BASE=10.151.1
2152023-11-16T21:23:07.453Z+ EXTRA_IP_FOCTET=164
2162023-11-16T21:23:07.497Z+ EXTRA_IP_LOCTET=227
2172023-11-16T21:23:07.543Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-11-16T21:23:07.598Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-11-16T21:23:07.630Z+ DNS_IP1=10.151.1.164
2202023-11-16T21:23:07.662Z+ DNS_IP2=10.151.1.165
2212023-11-16T21:23:07.695Z+ UPLINK_IP=10.151.1.174
2222023-11-16T21:23:07.732Z+ IPPOOL_START=10.151.1.175
2232023-11-16T21:23:07.768Z+ IPPOOL_END=10.151.1.227
2242023-11-16T21:23:07.801Z++ ipadm show-addr -po type,addr
2252023-11-16T21:23:07.833Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-16T21:23:07.864Z+ GATEWAY_IP=10.151.1.100
2272023-11-16T21:23:07.896Z+ [[ -n 10.151.1.100 ]]
2282023-11-16T21:23:07.929Z+ ping -s 10.151.1.100 56 1
2292023-11-16T21:23:07.964ZPING 10.151.1.100: 56 data bytes
2302023-11-16T21:23:08.416Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.019 ms
2312023-11-16T21:23:08.452Z
2322023-11-16T21:23:08.489Z----10.151.1.100 PING Statistics----
2332023-11-16T21:23:08.521Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-16T21:23:08.553Zround-trip (ms) min/avg/max/stddev = 0.019/0.019/0.019/-nan
2352023-11-16T21:23:08.585Z++ arp -an
2362023-11-16T21:23:08.624Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-11-16T21:23:08.656Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-11-16T21:23:08.688Z+ routeadm -e ipv4-forwarding -u
2392023-11-16T21:23:08.719Z+ PXA_START=10.151.1.164
2402023-11-16T21:23:08.752Z+ PXA_END=10.151.1.227
2412023-11-16T21:23:08.784Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-16T21:23:08.815Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-16T21:23:08.849Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-16T21:23:08.881Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-16T21:23:08.913Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-16T21:23:08.945Z++ cd ./tools
2472023-11-16T21:23:08.979Z++ pwd
2482023-11-16T21:23:09.012Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-16T21:23:09.046Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-16T21:23:09.082Z+ SOFTNPU_MODE=zone
2512023-11-16T21:23:09.120Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-16T21:23:09.157Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-16T21:23:09.194Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-16T21:23:09.241Z++ dladm show-phys -p -o LINK
2552023-11-16T21:23:09.292Z++ head -1
2562023-11-16T21:23:09.333Z+ PHYSICAL_LINK=igb0
2572023-11-16T21:23:09.370Z+ echo 'Using igb0 as physical link'
2582023-11-16T21:23:09.404ZUsing igb0 as physical link
2592023-11-16T21:23:09.436Z+ ensure_run_as_root
2602023-11-16T21:23:09.468Z++ id -u
2612023-11-16T21:23:09.500ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2622023-11-16T21:23:09.532Z+ [[ 0 -ne 0 ]]
2632023-11-16T21:23:09.563ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2642023-11-16T21:23:09.595Z+ ensure_zpools
2652023-11-16T21:23:09.650ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2662023-11-16T21:23:09.682Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2672023-11-16T21:23:09.714ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2682023-11-16T21:23:09.752Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2692023-11-16T21:23:09.791ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2702023-11-16T21:23:09.826Z+ readarray -t ZPOOLS
2712023-11-16T21:23:09.859ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2722023-11-16T21:23:09.892Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2732023-11-16T21:23:09.924ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2742023-11-16T21:23:09.956Z++ sed 's/[ ",]//g'
2752023-11-16T21:23:09.987ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2762023-11-16T21:23:10.019Z+ for ZPOOL in "${ZPOOLS[@]}"
2772023-11-16T21:23:10.051ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2782023-11-16T21:23:10.083Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2792023-11-16T21:23:10.115ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2802023-11-16T21:23:10.147Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2812023-11-16T21:23:10.185ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2822023-11-16T21:23:10.224Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2832023-11-16T21:23:10.260ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2842023-11-16T21:23:10.296Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2852023-11-16T21:23:10.331ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2862023-11-16T21:23:10.370Z0+0 records in
2872023-11-16T21:23:10.403ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2882023-11-16T21:23:10.439Z0+0 records out
2892023-11-16T21:23:10.474ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2902023-11-16T21:23:10.508Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2912023-11-16T21:23:10.546ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
2922023-11-16T21:23:10.587Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2932023-11-16T21:23:10.626ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
2942023-11-16T21:23:11.071Z+ set +x
2952023-11-16T21:23:11.105ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
2962023-11-16T21:23:11.137Z++ zpool list -o name
2972023-11-16T21:23:11.180ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
2982023-11-16T21:23:11.217Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2992023-11-16T21:23:11.254ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3002023-11-16T21:23:11.290Z+ [[ -z '' ]]
3012023-11-16T21:23:11.329ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3022023-11-16T21:23:11.365Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
3032023-11-16T21:23:11.402ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3042023-11-16T21:23:11.437Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
3052023-11-16T21:23:11.481ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3062023-11-16T21:23:11.516Z+ set +x
3072023-11-16T21:23:11.550ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3082023-11-16T21:23:11.589Z+ for ZPOOL in "${ZPOOLS[@]}"
3092023-11-16T21:23:11.622ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3102023-11-16T21:23:11.654Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3112023-11-16T21:23:11.687ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3122023-11-16T21:23:11.761Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3132023-11-16T21:23:11.800ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3142023-11-16T21:23:11.840Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3152023-11-16T21:23:11.878ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3162023-11-16T21:23:11.924Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3172023-11-16T21:23:11.962ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3182023-11-16T21:23:11.996Z0+0 records in
3192023-11-16T21:23:12.034ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3202023-11-16T21:23:12.067Z0+0 records out
3212023-11-16T21:23:12.099ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3222023-11-16T21:23:12.130Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3232023-11-16T21:23:12.162ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3242023-11-16T21:23:12.194Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3252023-11-16T21:23:12.225ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3262023-11-16T21:23:12.257Z+ set +x
3272023-11-16T21:23:12.293ZSimnet net0/sc0_0 exists
3282023-11-16T21:23:12.325Z++ zpool list -o name
3292023-11-16T21:23:12.363ZSimnet net1/sc1_0 exists
3302023-11-16T21:23:12.401Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3312023-11-16T21:23:12.848ZVnic sc0_1 exists
3322023-11-16T21:23:12.884Z+ [[ -z '' ]]
3332023-11-16T21:23:12.922Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
3342023-11-16T21:23:12.989Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3352023-11-16T21:23:13.022Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3362023-11-16T21:23:13.053Z+ set +x
3372023-11-16T21:23:13.085Z+ for ZPOOL in "${ZPOOLS[@]}"
3382023-11-16T21:23:13.116Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3392023-11-16T21:23:13.148Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3402023-11-16T21:23:13.179Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3412023-11-16T21:23:13.210Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3422023-11-16T21:23:13.251Z0+0 records in
3432023-11-16T21:23:13.696Z0+0 records out
3442023-11-16T21:23:13.737Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3452023-11-16T21:23:13.774Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3462023-11-16T21:23:13.831Z+ set +x
3472023-11-16T21:23:13.862Z++ zpool list -o name
3482023-11-16T21:23:13.894Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3492023-11-16T21:23:13.927Z+ [[ -z '' ]]
3502023-11-16T21:23:13.958Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3512023-11-16T21:23:13.995Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3522023-11-16T21:23:14.026Z+ set +x
3532023-11-16T21:23:14.058Z+ for ZPOOL in "${ZPOOLS[@]}"
3542023-11-16T21:23:14.089Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552023-11-16T21:23:14.121Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562023-11-16T21:23:14.152Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572023-11-16T21:23:14.183Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582023-11-16T21:23:14.220Z0+0 records in
3592023-11-16T21:23:14.254Z0+0 records out
3602023-11-16T21:23:14.288Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3612023-11-16T21:23:14.325Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622023-11-16T21:23:14.362Z+ set +x
3632023-11-16T21:23:14.400Z++ zpool list -o name
3642023-11-16T21:23:14.434Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3652023-11-16T21:23:14.469Z+ [[ -z '' ]]
3662023-11-16T21:23:14.506Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3672023-11-16T21:23:14.545Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3682023-11-16T21:23:14.592Z+ set +x
3692023-11-16T21:23:14.629Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-11-16T21:23:14.671Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3712023-11-16T21:23:14.707Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3722023-11-16T21:23:14.746Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3732023-11-16T21:23:14.783Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3742023-11-16T21:23:14.818Z0+0 records in
3752023-11-16T21:23:14.894Z0+0 records out
3762023-11-16T21:23:14.931Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3772023-11-16T21:23:14.965Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3782023-11-16T21:23:14.999Z+ set +x
3792023-11-16T21:23:15.042Z++ zpool list -o name
3802023-11-16T21:23:15.079Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3812023-11-16T21:23:15.116Z+ [[ -z '' ]]
3822023-11-16T21:23:15.150Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3832023-11-16T21:23:15.183Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3842023-11-16T21:23:15.215Z+ set +x
3852023-11-16T21:23:15.247Z+ for ZPOOL in "${ZPOOLS[@]}"
3862023-11-16T21:23:15.279Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-11-16T21:23:15.311Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-11-16T21:23:15.347Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-11-16T21:23:15.382Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-11-16T21:23:15.414Z0+0 records in
3912023-11-16T21:23:15.446Z0+0 records out
3922023-11-16T21:23:15.478Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3932023-11-16T21:23:15.510Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942023-11-16T21:23:15.963Z+ set +x
3952023-11-16T21:23:15.997Z++ zpool list -o name
3962023-11-16T21:23:16.042Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972023-11-16T21:23:16.076Z+ [[ -z '' ]]
3982023-11-16T21:23:16.108Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992023-11-16T21:23:16.141Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002023-11-16T21:23:16.173Z+ set +x
4012023-11-16T21:23:16.204Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-16T21:23:16.238Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4032023-11-16T21:23:16.269Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-11-16T21:23:16.302Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4052023-11-16T21:23:16.333Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4062023-11-16T21:23:16.365Z0+0 records in
4072023-11-16T21:23:16.397Z0+0 records out
4082023-11-16T21:23:16.428Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4092023-11-16T21:23:16.472Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4102023-11-16T21:23:16.513Z+ set +x
4112023-11-16T21:23:16.546Z++ zpool list -o name
4122023-11-16T21:23:16.582Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4132023-11-16T21:23:16.619Z+ [[ -z '' ]]
4142023-11-16T21:23:16.653Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4152023-11-16T21:23:16.690Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4162023-11-16T21:23:16.732Z+ set +x
4172023-11-16T21:23:16.768Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-11-16T21:23:16.800Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4192023-11-16T21:23:16.832Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4202023-11-16T21:23:16.864Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4212023-11-16T21:23:16.903Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4222023-11-16T21:23:16.948Z0+0 records in
4232023-11-16T21:23:16.980Z0+0 records out
4242023-11-16T21:23:17.016Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4252023-11-16T21:23:17.050Zdownloading softnpu ...done
4262023-11-16T21:23:17.082Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4272023-11-16T21:23:17.113Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
4282023-11-16T21:23:17.146Z+ set +x
4292023-11-16T21:23:17.183Z++ zpool list -o name
4302023-11-16T21:23:17.220Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4312023-11-16T21:23:17.253Z+ [[ -z '' ]]
4322023-11-16T21:23:17.287Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4332023-11-16T21:23:17.326Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4342023-11-16T21:23:17.396Z+ set +x
4352023-11-16T21:23:17.437Z+ for ZPOOL in "${ZPOOLS[@]}"
4362023-11-16T21:23:17.918Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4372023-11-16T21:23:18.001Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4382023-11-16T21:23:18.039Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4392023-11-16T21:23:18.073Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4402023-11-16T21:23:18.106Z0+0 records in
4412023-11-16T21:23:18.138Z0+0 records out
4422023-11-16T21:23:18.178Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4432023-11-16T21:23:18.215Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4442023-11-16T21:23:18.247Z+ set +x
4452023-11-16T21:23:18.279Z++ zpool list -o name
4462023-11-16T21:23:18.310Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4472023-11-16T21:23:18.342Z+ [[ -z '' ]]
4482023-11-16T21:23:18.373Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4492023-11-16T21:23:18.405Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4502023-11-16T21:23:18.437Z+ set +x
4512023-11-16T21:23:18.480Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4522023-11-16T21:23:18.525Z+ readarray -t ZPOOLS
4532023-11-16T21:23:18.556Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4542023-11-16T21:23:18.588Z++ sed 's/[ ",]//g'
4552023-11-16T21:23:18.626Z+ for ZPOOL in "${ZPOOLS[@]}"
4562023-11-16T21:23:18.667Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4572023-11-16T21:23:18.705Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4582023-11-16T21:23:18.737Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4592023-11-16T21:23:18.768Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4602023-11-16T21:23:18.801Z0+0 records in
4612023-11-16T21:23:18.836Z0+0 records out
4622023-11-16T21:23:18.868Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4632023-11-16T21:23:18.904Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4642023-11-16T21:23:19.345Z+ set +x
4652023-11-16T21:23:19.377Z++ zpool list -o name
4662023-11-16T21:23:19.409Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4672023-11-16T21:23:19.441Z+ [[ -z '' ]]
4682023-11-16T21:23:19.473Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4692023-11-16T21:23:19.506Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4702023-11-16T21:23:19.544Z+ set +x
4712023-11-16T21:23:19.579Z+ for ZPOOL in "${ZPOOLS[@]}"
4722023-11-16T21:23:19.611Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4732023-11-16T21:23:19.643Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4742023-11-16T21:23:19.674Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4752023-11-16T21:23:19.710Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4762023-11-16T21:23:19.742Z0+0 records in
4772023-11-16T21:23:19.777Z0+0 records out
4782023-11-16T21:23:19.813Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4792023-11-16T21:23:19.848Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4802023-11-16T21:23:19.889Z+ set +x
4812023-11-16T21:23:19.926Z++ zpool list -o name
4822023-11-16T21:23:19.962Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4832023-11-16T21:23:19.997Z+ [[ -z '' ]]
4842023-11-16T21:23:20.063Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4852023-11-16T21:23:20.097Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4862023-11-16T21:23:20.129Z+ set +x
4872023-11-16T21:23:20.168Z+ [[ zone == \z\o\n\e ]]
4882023-11-16T21:23:20.200Z+ ensure_simulated_links igb0
4892023-11-16T21:23:20.232Z+ local PHYSICAL_LINK=igb0
4902023-11-16T21:23:20.264Z+ INDICES=("0" "1")
4912023-11-16T21:23:20.295Z+ for I in "${INDICES[@]}"
4922023-11-16T21:23:20.332Z++ get_simnet_name_if_exists net0
4932023-11-16T21:23:20.366Z++ dladm show-simnet -p -o LINK net0
4942023-11-16T21:23:20.401Z++ echo ''
4952023-11-16T21:23:20.434Z+ [[ -z '' ]]
4962023-11-16T21:23:20.466Z+ dladm create-simnet -t net0
4972023-11-16T21:23:20.499Z+ dladm create-simnet -t sc0_0
4982023-11-16T21:23:20.538Z+ dladm modify-simnet -t -p net0 sc0_0
4992023-11-16T21:23:20.572Z+ dladm set-linkprop -p mtu=9000 sc0_0
5002023-11-16T21:23:20.604Z+ success 'Simnet net0/sc0_0 exists'
5012023-11-16T21:23:20.636Z+ set +x
5022023-11-16T21:23:20.667Z+ for I in "${INDICES[@]}"
5032023-11-16T21:23:20.700Z++ get_simnet_name_if_exists net1
5042023-11-16T21:23:20.738Z++ dladm show-simnet -p -o LINK net1
5052023-11-16T21:23:21.216Z++ echo ''
5062023-11-16T21:23:21.248Z+ [[ -z '' ]]
5072023-11-16T21:23:21.284Z+ dladm create-simnet -t net1
5082023-11-16T21:23:21.315Z+ dladm create-simnet -t sc1_0
5092023-11-16T21:23:21.347Z+ dladm modify-simnet -t -p net1 sc1_0
5102023-11-16T21:23:21.380Z+ dladm set-linkprop -p mtu=9000 sc1_0
5112023-11-16T21:23:21.411Z+ success 'Simnet net1/sc1_0 exists'
5122023-11-16T21:23:21.443Z+ set +x
5132023-11-16T21:23:21.475Z++ get_vnic_name_if_exists sc0_1
5142023-11-16T21:23:21.506Z++ dladm show-vnic -p -o LINK sc0_1
5152023-11-16T21:23:21.537Z++ echo ''
5162023-11-16T21:23:21.570Z+ [[ -z '' ]]
5172023-11-16T21:23:21.611Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5182023-11-16T21:23:21.649Z+ [[ -v PROMISC_FILT_OFF ]]
5192023-11-16T21:23:21.690Z+ success 'Vnic sc0_1 exists'
5202023-11-16T21:23:21.732Z+ set +x
5212023-11-16T21:23:21.765Z+ warn_if_no_proxy_arp
5222023-11-16T21:23:21.798Z+ [[ -v PXA_START ]]
5232023-11-16T21:23:21.830Z+ [[ -v PXA_END ]]
5242023-11-16T21:23:21.862Z+ ensure_softnpu_zone
5252023-11-16T21:23:21.893Z+ zoneadm list
5262023-11-16T21:23:21.925Z+ grep -q sidecar_softnpu
5272023-11-16T21:23:21.956Z+ [[ -f out/npuzone/npuzone ]]
5282023-11-16T21:23:21.988Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5292023-11-16T21:23:22.019Zdownloading asic_program.so ...done
5302023-11-16T21:23:22.056Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-16T21:23:29.252Zdownloading scadm ...done
5322023-11-16T21:23:29.284Zcreating port tfportrear0_0 for sc0_0
5332023-11-16T21:23:29.315Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-16T21:23:29.347Zsoftnpu zone setup
5352023-11-16T21:23:29.404Zconfigure zone
5362023-11-16T21:23:29.437Zinstall zone
5372023-11-16T21:23:30.361Zboot zone
5382023-11-16T21:23:33.471Z[sidecar_softnpu] svcs milestone/network
5392023-11-16T21:23:33.503ZSTATE STIME FMRI
5402023-11-16T21:23:33.536Zonline 21:23:31 svc:/milestone/network:default
5412023-11-16T21:23:33.568Z[sidecar_softnpu] /softnpu/npu start
5422023-11-16T21:23:33.601Z
5432023-11-16T21:23:33.634Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-16T21:23:33.665Z+ GATEWAY_IP=10.151.1.100
5452023-11-16T21:23:33.697ZUsing 10.151.1.100 as gateway ip
5462023-11-16T21:23:33.746Z+ echo 'Using 10.151.1.100 as gateway ip'
5472023-11-16T21:23:33.780Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-16T21:23:33.813Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-11-16T21:23:33.848Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-16T21:23:33.881Z+++ echo 18:c0:4d:0d:9f:b2
5512023-11-16T21:23:33.913Z+ COUNT=1
5522023-11-16T21:23:33.945Z+ [[ 1 -ne 1 ]]
5532023-11-16T21:23:33.976Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-11-16T21:23:34.007ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-11-16T21:23:34.039Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-11-16T21:23:34.071Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-11-16T21:23:34.103Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-16T21:23:34.134Z+ [[ -v PXA_START ]]
5592023-11-16T21:23:34.172Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-11-16T21:23:34.207Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-11-16T21:23:34.239Z+ z_scadm dump-state
5622023-11-16T21:23:34.274Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-16T21:23:34.309Zlocal v6:
5642023-11-16T21:23:34.343Zlocal v4:
5652023-11-16T21:23:34.375Zrouter v6:
5662023-11-16T21:23:34.406Zrouter v4_idx:
5672023-11-16T21:23:34.438Zrouter v4_routes:
5682023-11-16T21:23:34.469Zresolver v4:
5692023-11-16T21:23:34.501Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-11-16T21:23:34.532Zresolver v6:
5712023-11-16T21:23:34.564Znat v4:
5722023-11-16T21:23:34.605Znat_v6:
5732023-11-16T21:23:34.641Zport_mac:
5742023-11-16T21:23:34.674Zproxy arp:
5752023-11-16T21:23:34.710Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-11-16T21:23:34.748Z+ success 'softnpu zone exists'
5772023-11-16T21:23:34.780Z+ set +x
5782023-11-16T21:23:34.812Zsoftnpu zone exists
5792023-11-16T21:23:34.843Z
5802023-11-16T21:23:34.875Zreal 27.662923322
5812023-11-16T21:23:34.906Zuser 0.710513412
5822023-11-16T21:23:34.939Zsys 1.628738261
5832023-11-16T21:23:34.971Ztrap 0.000064578
5842023-11-16T21:23:35.002Ztflt 0.000173395
5852023-11-16T21:23:35.043Zdflt 0.001171935
5862023-11-16T21:23:35.079Zkflt 0.000000000
5872023-11-16T21:23:35.121Zlock 21:28.415963300
5882023-11-16T21:23:35.153Zslp 1:22.967789726
5892023-11-16T21:23:35.186Zlat 0.079476127
5902023-11-16T21:23:35.217Zstop 0.350937921
5912023-11-16T21:23:35.249Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-16T21:23:35.281Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-16T21:23:35.312Z+ SILO_NAME=recovery
5942023-11-16T21:23:35.344Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-16T21:23:35.375Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-16T21:23:35.406Z+ sed -i~ '
5972023-11-16T21:23:35.438Z /^external_dns_ips/c\
5982023-11-16T21:23:35.475Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-11-16T21:23:35.521Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-16T21:23:35.552Z /^first/c\
6012023-11-16T21:23:35.592Zfirst = "10.151.1.164"
6022023-11-16T21:23:35.630Z /^last/c\
6032023-11-16T21:23:35.670Zlast = "10.151.1.173"
6042023-11-16T21:23:35.702Z }
6052023-11-16T21:23:35.734Z /^\[rack_network_config/,/^$/ {
6062023-11-16T21:23:35.765Z /^infra_ip_first/c\
6072023-11-16T21:23:35.797Zinfra_ip_first = "10.151.1.174"
6082023-11-16T21:23:35.828Z /^infra_ip_last/c\
6092023-11-16T21:23:35.860Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6102023-11-16T21:23:35.891Zinfra_ip_last = "10.151.1.174"
6112023-11-16T21:23:35.940Z+++ pkg/config-rss.toml Thu Nov 16 21:23:33 2023
6122023-11-16T21:23:35.974Z }
6132023-11-16T21:23:36.006Z@@ -41,7 +41,7 @@
6142023-11-16T21:23:36.041Z /^\[\[rack_network_config.ports/,/^$/ {
6152023-11-16T21:23:36.079Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6162023-11-16T21:23:36.112Z /^routes/c\
6172023-11-16T21:23:36.144Z # must be contained in one of the "internal services" IP Pool ranges listed
6182023-11-16T21:23:36.176Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6192023-11-16T21:23:36.207Z # below.
6202023-11-16T21:23:36.238Z /^addresses/c\
6212023-11-16T21:23:36.270Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6222023-11-16T21:23:36.302Zaddresses = \["10.151.1.174/32"\]
6232023-11-16T21:23:36.333Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6242023-11-16T21:23:36.368Z }
6252023-11-16T21:23:36.401Z
6262023-11-16T21:23:36.432Z' pkg/config-rss.toml
6272023-11-16T21:23:36.466Z # Initial TLS certificates for the external API
6282023-11-16T21:23:36.502Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6292023-11-16T21:23:36.534Z #
6302023-11-16T21:23:36.566Z+ true
6312023-11-16T21:23:36.619Z@@ -77,8 +77,8 @@
6322023-11-16T21:23:36.651Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6332023-11-16T21:23:36.683Z #
6342023-11-16T21:23:36.714Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6352023-11-16T21:23:36.748Z # For more on this and what to put here, see docs/how-to-run.adoc.
6362023-11-16T21:23:36.780Z+ TLS_NAME=recovery.sys.oxide.test
6372023-11-16T21:23:36.812Z [[internal_services_ip_pool_ranges]]
6382023-11-16T21:23:36.843Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6392023-11-16T21:23:36.875Z-first = "192.168.1.20"
6402023-11-16T21:23:36.909Z.+.+..+..........+...+.....+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+............+.....+.............+...+.....+.........+.............+..................+........+......+......+..........+..............+....+...........+..................+.......+......+..+.+..+.............+.....................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6412023-11-16T21:23:36.949Z-last = "192.168.1.29"
6422023-11-16T21:23:36.982Z...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...............+..+......+.......+........+.......+..+.......+......+..+...+....+...+...+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6432023-11-16T21:23:37.014Z+first = "10.151.1.164"
6442023-11-16T21:23:37.056Z-----
6452023-11-16T21:23:37.088Z+last = "10.151.1.173"
6462023-11-16T21:23:37.120Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6472023-11-16T21:23:37.151Z
6482023-11-16T21:23:37.184Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6492023-11-16T21:23:37.220Z # TODO - this configuration is subject to change going forward. Ultimately these
6502023-11-16T21:23:37.254Z+ rmdir pkg
6512023-11-16T21:23:37.286Z # parameters should be provided to the control plane via wicket, but we need to
6522023-11-16T21:23:37.317Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6532023-11-16T21:23:37.379Z@@ -92,8 +92,8 @@
6542023-11-16T21:23:37.415Z+ pfexec mkdir -p /usr/oxide
6552023-11-16T21:23:37.452Z # A range of IP addresses used by Boundary Services on the external network. In
6562023-11-16T21:23:37.525Z+ pfexec rm -f /usr/oxide/tcpproxy
6572023-11-16T21:23:37.557Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6582023-11-16T21:23:37.588Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6592023-11-16T21:23:37.663Z # With softnpu, only one address is used.
6602023-11-16T21:23:37.697Z+ pfexec chmod +x /usr/oxide/tcpproxy
6612023-11-16T21:23:37.729Z-infra_ip_first = "192.168.1.30"
6622023-11-16T21:23:37.760Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6632023-11-16T21:23:37.792Z-infra_ip_last = "192.168.1.30"
6642023-11-16T21:23:37.831Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6652023-11-16T21:23:37.865Z+infra_ip_first = "10.151.1.174"
6662023-11-16T21:23:37.896Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6672023-11-16T21:23:37.928Z+infra_ip_last = "10.151.1.174"
6682023-11-16T21:23:37.959Z+ OMICRON_NO_UNINSTALL=1
6692023-11-16T21:23:37.991Z
6702023-11-16T21:23:38.022Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6712023-11-16T21:23:38.055Z # Configurations for BGP routers to run on the scrimlets.
6722023-11-16T21:23:38.096ZNov 16 21:23:34.674 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6732023-11-16T21:23:38.130Z bgp = []
6742023-11-16T21:23:38.162ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6752023-11-16T21:23:38.193Z@@ -101,9 +101,9 @@
6762023-11-16T21:23:38.225ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6772023-11-16T21:23:38.257Z # You can configure multiple uplinks by repeating the following stanza
6782023-11-16T21:23:38.294ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6792023-11-16T21:23:38.327Z [[rack_network_config.ports]]
6802023-11-16T21:23:38.358ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6812023-11-16T21:23:38.390Z # Routes associated with this port.
6822023-11-16T21:23:38.423ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6832023-11-16T21:23:38.454Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6842023-11-16T21:23:38.486ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6852023-11-16T21:23:38.521Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6862023-11-16T21:23:38.565ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6872023-11-16T21:23:38.597Z # Addresses associated with this port.
6882023-11-16T21:23:38.628ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6892023-11-16T21:23:38.660Z-addresses = ["192.168.1.30/32"]
6902023-11-16T21:23:38.692ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6912023-11-16T21:23:38.728Z+addresses = ["10.151.1.174/32"]
6922023-11-16T21:23:38.762ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6932023-11-16T21:23:38.798Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6942023-11-16T21:23:38.836ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6952023-11-16T21:23:38.870Z port = "qsfp0"
6962023-11-16T21:23:38.905ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6972023-11-16T21:23:38.940Z # The speed of this port.
6982023-11-16T21:23:38.983ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6992023-11-16T21:23:39.016Za pkg/config-rss.toml 7K
7002023-11-16T21:23:39.049ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7012023-11-16T21:23:39.084Za pkg/initial-tls-cert.pem 2K
7022023-11-16T21:23:39.120ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7032023-11-16T21:23:39.156Za pkg/initial-tls-key.pem 4K
7042023-11-16T21:23:39.191ZNov 16 21:23:34.676 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7052023-11-16T21:23:39.225ZNov 16 21:23:34.902 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-11-16T21:23:39.262ZNov 16 21:23:34.936 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-16T21:23:39.297ZNov 16 21:23:34.996 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-16T21:23:39.334Z
7092023-11-16T21:23:39.368Zreal 0.461771832
7102023-11-16T21:23:39.406Zuser 0.099327488
7112023-11-16T21:23:39.442Zsys 1.841597289
7122023-11-16T21:23:39.478Ztrap 0.000007089
7132023-11-16T21:23:39.515Ztflt 0.000026859
7142023-11-16T21:23:39.549Zdflt 0.001343469
7152023-11-16T21:23:39.585Zkflt 0.000000000
7162023-11-16T21:23:39.621Zlock 41.376712624
7172023-11-16T21:23:39.656Zslp 0.677325858
7182023-11-16T21:23:39.734Zlat 0.002889447
7192023-11-16T21:23:39.768Zstop 0.197969121
7202023-11-16T21:23:39.800Z+ retry=0
7212023-11-16T21:23:39.837Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-16T21:23:39.882Z+ [[ 0 -gt 30 ]]
7232023-11-16T21:23:39.916Z+ sleep 1
7242023-11-16T21:23:39.950Z+ retry=1
7252023-11-16T21:23:39.986Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-16T21:23:40.021Z+ [[ 1 -gt 30 ]]
7272023-11-16T21:23:40.058Z+ sleep 1
7282023-11-16T21:23:40.095Z+ retry=2
7292023-11-16T21:23:40.132Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-16T21:23:40.169Z+ [[ 2 -gt 30 ]]
7312023-11-16T21:23:40.206Z+ sleep 1
7322023-11-16T21:23:40.242Z+ retry=3
7332023-11-16T21:23:40.280Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-11-16T21:23:40.316Z+ [[ 3 -gt 30 ]]
7352023-11-16T21:23:40.354Z+ sleep 1
7362023-11-16T21:23:40.387Z+ retry=4
7372023-11-16T21:23:40.421Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-11-16T21:23:58.838Z+ [[ 4 -gt 30 ]]
7392023-11-16T21:23:58.875Z+ sleep 1
7402023-11-16T21:23:59.836Z+ retry=5
7412023-11-16T21:23:59.869Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7422023-11-16T21:23:59.901Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7432023-11-16T21:23:59.938Zlocal v6:
7442023-11-16T21:23:59.973Zfd00:99::1
7452023-11-16T21:24:00.005Zfe80::c82:6aff:fe11:7cc3
7462023-11-16T21:24:00.037Zlocal v4:
7472023-11-16T21:24:00.070Z10.151.1.174
7482023-11-16T21:24:00.103Zrouter v6:
7492023-11-16T21:24:00.145Zrouter v4_idx:
7502023-11-16T21:24:00.185Z0.0.0.0/0 -> 0
7512023-11-16T21:24:00.631Zrouter v4_routes:
7522023-11-16T21:24:00.672Z0 -> 10.151.1.100 (2)
7532023-11-16T21:24:00.710Zresolver v4:
7542023-11-16T21:24:00.742Z10.151.1.174 -> 5e:ab:a4:1b:ed:bf
7552023-11-16T21:24:00.773Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7562023-11-16T21:24:00.804Zresolver v6:
7572023-11-16T21:24:00.845Zfe80::c82:6aff:fe11:7cc3 -> 0e:82:6a:11:7c:c3
7582023-11-16T21:24:00.880Zfe80::985f:7cff:fee5:9263 -> 9a:5f:7c:e5:92:63
7592023-11-16T21:24:00.912Znat v4:
7602023-11-16T21:24:00.944Znat_v6:
7612023-11-16T21:24:00.976Zport_mac:
7622023-11-16T21:24:01.010Z1000: a8:40:25:54:41:8c
7632023-11-16T21:24:01.041Z1: a8:40:25:54:41:8d
7642023-11-16T21:24:01.074Zproxy arp:
7652023-11-16T21:24:01.126Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7662023-11-16T21:24:01.166Z+ retry=0
7672023-11-16T21:24:01.199Z+++ zoneadm list -n
7682023-11-16T21:24:01.231Z+++ grep oxz_ntp
7692023-11-16T21:24:01.264Z++ pfexec svcs -z -Hostate oxide/ntp
7702023-11-16T21:24:01.297Zsvcs: zone '-Hostate': entity not found
7712023-11-16T21:24:01.329Z++ true
7722023-11-16T21:24:01.361Z+ [[ '' != online ]]
7732023-11-16T21:24:01.403Z+ [[ 0 -gt 60 ]]
7742023-11-16T21:24:01.435Z+ sleep 1
7752023-11-16T21:24:01.466Z+ retry=1
7762023-11-16T21:24:01.498Z+++ zoneadm list -n
7772023-11-16T21:24:01.530Z+++ grep oxz_ntp
7782023-11-16T21:24:01.562Z++ pfexec svcs -z -Hostate oxide/ntp
7792023-11-16T21:24:01.601Zsvcs: zone '-Hostate': entity not found
7802023-11-16T21:24:01.647Z++ true
7812023-11-16T21:24:01.685Z+ [[ '' != online ]]
7822023-11-16T21:24:01.724Z+ [[ 1 -gt 60 ]]
7832023-11-16T21:24:01.760Z+ sleep 1
7842023-11-16T21:24:01.922Z+ retry=2
7852023-11-16T21:24:01.955Z+++ zoneadm list -n
7862023-11-16T21:24:01.988Z+++ grep oxz_ntp
7872023-11-16T21:24:02.021Z++ pfexec svcs -z -Hostate oxide/ntp
7882023-11-16T21:24:02.055Zsvcs: zone '-Hostate': entity not found
7892023-11-16T21:24:02.091Z++ true
7902023-11-16T21:24:02.125Z+ [[ '' != online ]]
7912023-11-16T21:24:02.173Z+ [[ 2 -gt 60 ]]
7922023-11-16T21:24:02.210Z+ sleep 1
7932023-11-16T21:24:02.947Z+ retry=3
7942023-11-16T21:24:02.991Z+++ zoneadm list -n
7952023-11-16T21:24:03.024Z+++ grep oxz_ntp
7962023-11-16T21:24:03.468Z++ pfexec svcs -z -Hostate oxide/ntp
7972023-11-16T21:24:03.500Zsvcs: zone '-Hostate': entity not found
7982023-11-16T21:24:03.538Z++ true
7992023-11-16T21:24:03.574Z+ [[ '' != online ]]
8002023-11-16T21:24:03.606Z+ [[ 3 -gt 60 ]]
8012023-11-16T21:24:03.638Z+ sleep 1
8022023-11-16T21:24:03.980Z+ retry=4
8032023-11-16T21:24:04.016Z+++ zoneadm list -n
8042023-11-16T21:24:04.049Z+++ grep oxz_ntp
8052023-11-16T21:24:04.081Z++ pfexec svcs -z oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1 -Hostate oxide/ntp
8062023-11-16T21:24:04.114Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8072023-11-16T21:24:04.146Z++ true
8082023-11-16T21:24:04.178Z+ [[ '' != online ]]
8092023-11-16T21:24:04.212Z+ [[ 4 -gt 60 ]]
8102023-11-16T21:24:04.244Z+ sleep 1
8112023-11-16T21:24:05.002Z+ retry=5
8122023-11-16T21:24:05.034Z+++ zoneadm list -n
8132023-11-16T21:24:05.066Z+++ grep oxz_ntp
8142023-11-16T21:24:05.098Z++ pfexec svcs -z oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1 -Hostate oxide/ntp
8152023-11-16T21:24:05.131Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8162023-11-16T21:24:05.571Z++ true
8172023-11-16T21:24:05.616Z+ [[ '' != online ]]
8182023-11-16T21:24:05.652Z+ [[ 5 -gt 60 ]]
8192023-11-16T21:24:05.684Z+ sleep 1
8202023-11-16T21:24:06.030Z+ retry=6
8212023-11-16T21:24:06.066Z+++ zoneadm list -n
8222023-11-16T21:24:06.099Z+++ grep oxz_ntp
8232023-11-16T21:24:06.130Z++ pfexec svcs -z oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1 -Hostate oxide/ntp
8242023-11-16T21:24:06.161Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8252023-11-16T21:24:06.193Z++ true
8262023-11-16T21:24:06.225Z+ [[ '' != online ]]
8272023-11-16T21:24:06.257Z+ [[ 6 -gt 60 ]]
8282023-11-16T21:24:06.289Z+ sleep 1
8292023-11-16T21:24:07.061Z+ retry=7
8302023-11-16T21:24:07.092Z+++ zoneadm list -n
8312023-11-16T21:24:07.124Z+++ grep oxz_ntp
8322023-11-16T21:24:07.156Z++ pfexec svcs -z oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1 -Hostate oxide/ntp
8332023-11-16T21:24:07.187Z+ [[ disabled != online ]]
8342023-11-16T21:24:07.219Z+ [[ 7 -gt 60 ]]
8352023-11-16T21:24:07.251Z+ sleep 1
8362023-11-16T21:24:08.086Z+ retry=8
8372023-11-16T21:24:08.126Z+++ zoneadm list -n
8382023-11-16T21:24:08.163Z+++ grep oxz_ntp
8392023-11-16T21:24:08.207Z++ pfexec svcs -z oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1 -Hostate oxide/ntp
8402023-11-16T21:24:08.242Z+ [[ disabled != online ]]
8412023-11-16T21:24:08.276Z+ [[ 8 -gt 60 ]]
8422023-11-16T21:24:08.321Z+ sleep 1
8432023-11-16T21:24:09.106Z+ retry=9
8442023-11-16T21:24:09.138Z+++ zoneadm list -n
8452023-11-16T21:24:09.170Z+++ grep oxz_ntp
8462023-11-16T21:24:09.202Z++ pfexec svcs -z oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1 -Hostate oxide/ntp
8472023-11-16T21:24:09.234Z+ [[ disabled != online ]]
8482023-11-16T21:24:09.266Z+ [[ 9 -gt 60 ]]
8492023-11-16T21:24:09.301Z+ sleep 1
8502023-11-16T21:24:10.131Z+ retry=10
8512023-11-16T21:24:10.169Z+++ zoneadm list -n
8522023-11-16T21:24:10.215Z+++ grep oxz_ntp
8532023-11-16T21:24:10.260Z++ pfexec svcs -z oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1 -Hostate oxide/ntp
8542023-11-16T21:24:10.302Z+ [[ disabled != online ]]
8552023-11-16T21:24:10.335Z+ [[ 10 -gt 60 ]]
8562023-11-16T21:24:10.366Z+ sleep 1
8572023-11-16T21:24:11.153Z+ retry=11
8582023-11-16T21:24:11.186Z+++ zoneadm list -n
8592023-11-16T21:24:11.218Z+++ grep oxz_ntp
8602023-11-16T21:24:11.249Z++ pfexec svcs -z oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1 -Hostate oxide/ntp
8612023-11-16T21:24:11.281Z+ [[ disabled != online ]]
8622023-11-16T21:24:11.313Z+ [[ 11 -gt 60 ]]
8632023-11-16T21:24:11.348Z+ sleep 1
8642023-11-16T21:24:12.173Z+ retry=12
8652023-11-16T21:24:12.205Z+++ zoneadm list -n
8662023-11-16T21:24:12.237Z+++ grep oxz_ntp
8672023-11-16T21:24:12.268Z++ pfexec svcs -z oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1 -Hostate oxide/ntp
8682023-11-16T21:24:12.300Z+ [[ disabled != online ]]
8692023-11-16T21:24:12.364Z+ [[ 12 -gt 60 ]]
8702023-11-16T21:24:12.396Z+ sleep 1
8712023-11-16T21:24:13.197Z+ retry=13
8722023-11-16T21:24:13.228Z+++ zoneadm list -n
8732023-11-16T21:24:13.260Z+++ grep oxz_ntp
8742023-11-16T21:24:13.297Z++ pfexec svcs -z oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1 -Hostate oxide/ntp
8752023-11-16T21:24:13.332Z+ [[ disabled != online ]]
8762023-11-16T21:24:13.364Z+ [[ 13 -gt 60 ]]
8772023-11-16T21:24:13.400Z+ sleep 1
8782023-11-16T21:24:14.222Z+ retry=14
8792023-11-16T21:24:14.263Z+++ zoneadm list -n
8802023-11-16T21:24:14.297Z+++ grep oxz_ntp
8812023-11-16T21:24:14.328Z++ pfexec svcs -z oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1 -Hostate oxide/ntp
8822023-11-16T21:24:14.360Z+ [[ disabled != online ]]
8832023-11-16T21:24:14.391Z+ [[ 14 -gt 60 ]]
8842023-11-16T21:24:14.423Z+ sleep 1
 
8852023-11-16T21:24:15.083Zbuskin console login: Nov 16 21:24:14 bukin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8862023-11-16T21:24:15.245Z+ retry=15
8872023-11-16T21:24:15.281Z+++ zoneadm list -n
8882023-11-16T21:24:15.325Z+++ grep oxz_ntp
8892023-11-16T21:24:15.361Z++ pfexec svcs -z oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1 -Hostate oxide/ntp
8902023-11-16T21:24:15.394Z+ [[ disabled != online ]]
8912023-11-16T21:24:15.429Z+ [[ 15 -gt 60 ]]
8922023-11-16T21:24:15.462Z+ sleep 1
8932023-11-16T21:24:16.263Z+ retry=16
8942023-11-16T21:24:16.295Z+++ zoneadm list -n
8952023-11-16T21:24:16.327Z+++ grep oxz_ntp
8962023-11-16T21:24:16.361Z++ pfexec svcs -z oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1 -Hostate oxide/ntp
8972023-11-16T21:24:16.397Z+ [[ online != online ]]
8982023-11-16T21:24:16.430Z+ echo 'Waited for chrony: 16s'
8992023-11-16T21:24:16.462Z+ export RUST_BACKTRACE=1
9002023-11-16T21:24:16.495Z+ RUST_BACKTRACE=1
9012023-11-16T21:24:16.532ZWaited for chrony: 16s
9022023-11-16T21:24:16.563Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9032023-11-16T21:24:16.595Z+ ./tests/bootstrap
9042023-11-16T21:24:16.633ZUsing external DNS server at 10.151.1.164:53
9052023-11-16T21:24:16.669Z2023-11-16 21:24:16.457747928 UTC: attempting to log into API
9062023-11-16T21:24:31.347Z2023-11-16 21:24:31.320117724 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9072023-11-16T21:24:32.359Z2023-11-16 21:24:32.321286943 UTC: attempting to log into API
9082023-11-16T21:24:47.382Z2023-11-16 21:24:47.354933777 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9092023-11-16T21:24:48.383Z2023-11-16 21:24:48.356054640 UTC: attempting to log into API
9102023-11-16T21:25:03.424Z2023-11-16 21:25:03.390951180 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9112023-11-16T21:25:04.419Z2023-11-16 21:25:04.392018996 UTC: attempting to log into API
9122023-11-16T21:25:19.481Z2023-11-16 21:25:19.454970861 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9132023-11-16T21:25:20.482Z2023-11-16 21:25:20.455770331 UTC: attempting to log into API
 
9142023-11-16T21:25:28.626ZNov 16 21:25:27 bukin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9152023-11-16T21:25:35.293ZNov 16 21:25:34 bukin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9162023-11-16T21:25:35.519Z2023-11-16 21:25:35.506518783 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9172023-11-16T21:25:36.519Z2023-11-16 21:25:36.507649561 UTC: attempting to log into API
 
9182023-11-16T21:25:40.797ZNov 16 21:25:40 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9192023-11-16T21:25:48.123ZNov 16 21:25:47 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9202023-11-16T21:25:51.555Z2023-11-16 21:25:51.540826264 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9212023-11-16T21:25:52.553Z2023-11-16 21:25:52.541947082 UTC: attempting to log into API
 
9222023-11-16T21:25:54.148ZNov 16 21:25:53 bukin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9232023-11-16T21:26:07.587Z2023-11-16 21:26:07.576130179 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9242023-11-16T21:26:08.588Z2023-11-16 21:26:08.577209915 UTC: attempting to log into API
9252023-11-16T21:26:23.622Z2023-11-16 21:26:23.611670470 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9262023-11-16T21:26:24.623Z2023-11-16 21:26:24.612889771 UTC: attempting to log into API
9272023-11-16T21:26:34.677Z2023-11-16 21:26:34.667356920 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: tcp connect error: Connection refused (os error 146)
9282023-11-16T21:26:35.682Z2023-11-16 21:26:35.668737225 UTC: attempting to log into API
9292023-11-16T21:26:36.478Z2023-11-16 21:26:36.468365587 UTC: login succeeded
9302023-11-16T21:26:36.514Zwaiting for nexus to come up...
9312023-11-16T21:26:36.579Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9322023-11-16T21:26:36.632Zensuring datasets are ready...
9332023-11-16T21:26:39.031Zlet's roll.
9342023-11-16T21:26:39.067Z+ rm ./tests/bootstrap
9352023-11-16T21:26:39.102Z+ for test_bin in tests/*
9362023-11-16T21:26:39.139Z+ ./tests/bootstrap-cd1d99075975e5dc
9372023-11-16T21:26:39.174Z
9382023-11-16T21:26:39.211Zrunning 0 tests
9392023-11-16T21:26:39.248Z
9402023-11-16T21:26:39.322Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9412023-11-16T21:26:39.356Z
9422023-11-16T21:26:39.391Z+ for test_bin in tests/*
9432023-11-16T21:26:39.427Z+ ./tests/end_to_end_tests-d1c7f350d876f572
9442023-11-16T21:26:39.460Z
9452023-11-16T21:26:39.494Zrunning 2 tests
9462023-11-16T21:26:39.528Ztest no_spoof::no_spoof_here ... ok
9472023-11-16T21:27:39.105Ztest instance_launch::instance_launch has been running for over 60 seconds
9482023-11-16T21:29:38.189Ztest instance_launch::instance_launch ... ok
9492023-11-16T21:29:38.220Z
9502023-11-16T21:29:38.252Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 179.08s
9512023-11-16T21:29:38.283Z
9522023-11-16T21:29:38.327Z+ _exit_trap
9532023-11-16T21:29:38.358Z+ local status=0
9542023-11-16T21:29:38.390Z+ [[ 0 -eq 0 ]]
9552023-11-16T21:29:38.421Z+ exit 0
9562023-11-16T21:29:38.452Zprocess exited: duration 400015 ms, exit code 0
 
9572023-11-16T21:29:38.544Zfound 47 output files
9582023-11-16T21:29:38.607Zuploading: /var/svc/log/oxide-sled-agent:default.log (687439 bytes)
9592023-11-16T21:29:39.887Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9602023-11-16T21:29:39.912Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_62232a5b-f4fd-468d-941c-1fbcb80c1fc4/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
9612023-11-16T21:29:41.009Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_62232a5b-f4fd-468d-941c-1fbcb80c1fc4/root/var/svc/log/oxide-cockroachdb:default.log
9622023-11-16T21:29:41.034Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_aaa8fba9-6b64-4650-be4a-52a75a7e58d1/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9632023-11-16T21:29:42.131Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_aaa8fba9-6b64-4650-be4a-52a75a7e58d1/root/var/svc/log/oxide-crucible-agent:default.log
9642023-11-16T21:29:42.156Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_aaa8fba9-6b64-4650-be4a-52a75a7e58d1/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9652023-11-16T21:29:43.252Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_aaa8fba9-6b64-4650-be4a-52a75a7e58d1/root/var/svc/log/oxide-crucible-downstairs:default.log
9662023-11-16T21:29:43.277Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e9a1f531-ec50-429d-956f-0de9835a2eb7/root/var/svc/log/oxide-external_dns:default.log (15611 bytes)
9672023-11-16T21:29:44.403Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e9a1f531-ec50-429d-956f-0de9835a2eb7/root/var/svc/log/oxide-external_dns:default.log
9682023-11-16T21:29:44.428Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ec3960ce-4c3a-4fb0-8ea3-27112af0a440/root/var/svc/log/oxide-external_dns:default.log (15609 bytes)
9692023-11-16T21:29:45.571Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ec3960ce-4c3a-4fb0-8ea3-27112af0a440/root/var/svc/log/oxide-external_dns:default.log
9702023-11-16T21:29:45.595Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_45f65c8c-ef47-4b01-be25-dbc68d4d1afb/root/var/svc/log/oxide-nexus:default.log (4805566 bytes)
9712023-11-16T21:29:49.374Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_45f65c8c-ef47-4b01-be25-dbc68d4d1afb/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4805566 -> 4806175
9722023-11-16T21:29:50.466Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_45f65c8c-ef47-4b01-be25-dbc68d4d1afb/root/var/svc/log/oxide-nexus:default.log
9732023-11-16T21:29:50.490Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_973f466f-2747-479e-9c53-a058ba63a495/root/var/svc/log/oxide-nexus:default.log (521023 bytes)
9742023-11-16T21:29:50.796Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_973f466f-2747-479e-9c53-a058ba63a495/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 521023 -> 521632
9752023-11-16T21:29:51.887Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_973f466f-2747-479e-9c53-a058ba63a495/root/var/svc/log/oxide-nexus:default.log
9762023-11-16T21:29:51.911Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_035ca48d-c9fe-4751-81d8-2aee31cd1b40/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9772023-11-16T21:29:53.009Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_035ca48d-c9fe-4751-81d8-2aee31cd1b40/root/var/svc/log/oxide-crucible-agent:default.log
9782023-11-16T21:29:53.033Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_035ca48d-c9fe-4751-81d8-2aee31cd1b40/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9792023-11-16T21:29:54.131Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_035ca48d-c9fe-4751-81d8-2aee31cd1b40/root/var/svc/log/oxide-crucible-downstairs:default.log
9802023-11-16T21:29:54.155Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8fbf5041-11cd-48ea-9cdf-cd6f4e8ff24f/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9812023-11-16T21:29:55.252Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8fbf5041-11cd-48ea-9cdf-cd6f4e8ff24f/root/var/svc/log/oxide-crucible-agent:default.log
9822023-11-16T21:29:55.277Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8fbf5041-11cd-48ea-9cdf-cd6f4e8ff24f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9832023-11-16T21:29:56.374Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8fbf5041-11cd-48ea-9cdf-cd6f4e8ff24f/root/var/svc/log/oxide-crucible-downstairs:default.log
9842023-11-16T21:29:56.398Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_db07cc8a-87d4-4560-8682-9e4eb94510cf/root/var/svc/log/oxide-internal_dns:default.log (106465 bytes)
9852023-11-16T21:29:56.581Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_db07cc8a-87d4-4560-8682-9e4eb94510cf/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 106465 -> 110219
9862023-11-16T21:29:57.671Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_db07cc8a-87d4-4560-8682-9e4eb94510cf/root/var/svc/log/oxide-internal_dns:default.log
9872023-11-16T21:29:57.696Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_176d0528-6b87-4837-ba99-69826c655bbe/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9882023-11-16T21:29:58.794Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_176d0528-6b87-4837-ba99-69826c655bbe/root/var/svc/log/oxide-crucible-agent:default.log
9892023-11-16T21:29:58.818Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_176d0528-6b87-4837-ba99-69826c655bbe/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9902023-11-16T21:29:59.915Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_176d0528-6b87-4837-ba99-69826c655bbe/root/var/svc/log/oxide-crucible-downstairs:default.log
9912023-11-16T21:29:59.939Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_166c5a84-df84-48f1-9a50-b096aaa2f90f/root/var/svc/log/oxide-crucible-agent:default.log (24401 bytes)
9922023-11-16T21:30:01.099Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_166c5a84-df84-48f1-9a50-b096aaa2f90f/root/var/svc/log/oxide-crucible-agent:default.log
9932023-11-16T21:30:01.123Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_166c5a84-df84-48f1-9a50-b096aaa2f90f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9942023-11-16T21:30:01.148Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_166c5a84-df84-48f1-9a50-b096aaa2f90f/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
9952023-11-16T21:30:02.224Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_166c5a84-df84-48f1-9a50-b096aaa2f90f/root/var/svc/log/oxide-crucible-downstairs:default.log
9962023-11-16T21:30:02.250Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_166c5a84-df84-48f1-9a50-b096aaa2f90f/root/var/svc/log/oxide-crucible-downstairs:downstairs-4bcfe797-2661-4e64-b4bf-8f2d2712a06c.log (3051 bytes)
9972023-11-16T21:30:02.275Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_166c5a84-df84-48f1-9a50-b096aaa2f90f/root/var/svc/log/oxide-crucible-downstairs:downstairs-4bcfe797-2661-4e64-b4bf-8f2d2712a06c.log" changed size mid upload: 3051 -> 0
9982023-11-16T21:30:03.350Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_166c5a84-df84-48f1-9a50-b096aaa2f90f/root/var/svc/log/oxide-crucible-downstairs:downstairs-4bcfe797-2661-4e64-b4bf-8f2d2712a06c.log
9992023-11-16T21:30:03.377Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_166c5a84-df84-48f1-9a50-b096aaa2f90f/root/var/svc/log/oxide-crucible-downstairs:downstairs-f43d154c-f6f6-4189-889b-e2912c06eb35.log (5536 bytes)
10002023-11-16T21:30:03.402Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_166c5a84-df84-48f1-9a50-b096aaa2f90f/root/var/svc/log/oxide-crucible-downstairs:downstairs-f43d154c-f6f6-4189-889b-e2912c06eb35.log" changed size mid upload: 5536 -> 0
10012023-11-16T21:30:04.474Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_166c5a84-df84-48f1-9a50-b096aaa2f90f/root/var/svc/log/oxide-crucible-downstairs:downstairs-f43d154c-f6f6-4189-889b-e2912c06eb35.log
10022023-11-16T21:30:04.499Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9569d16f-ea80-42c0-b206-03460d2b71f5/root/var/svc/log/oxide-crucible-agent:default.log (14151 bytes)
10032023-11-16T21:30:04.524Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9569d16f-ea80-42c0-b206-03460d2b71f5/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 14151 -> 0
10042023-11-16T21:30:05.597Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9569d16f-ea80-42c0-b206-03460d2b71f5/root/var/svc/log/oxide-crucible-agent:default.log
10052023-11-16T21:30:05.622Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9569d16f-ea80-42c0-b206-03460d2b71f5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10062023-11-16T21:30:05.648Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9569d16f-ea80-42c0-b206-03460d2b71f5/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10072023-11-16T21:30:06.720Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9569d16f-ea80-42c0-b206-03460d2b71f5/root/var/svc/log/oxide-crucible-downstairs:default.log
10082023-11-16T21:30:06.745Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9569d16f-ea80-42c0-b206-03460d2b71f5/root/var/svc/log/oxide-crucible-downstairs:downstairs-ed6bed9f-2d1b-4673-b76d-765adb08f921.log (5537 bytes)
10092023-11-16T21:30:06.773Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9569d16f-ea80-42c0-b206-03460d2b71f5/root/var/svc/log/oxide-crucible-downstairs:downstairs-ed6bed9f-2d1b-4673-b76d-765adb08f921.log" changed size mid upload: 5537 -> 0
10102023-11-16T21:30:07.841Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9569d16f-ea80-42c0-b206-03460d2b71f5/root/var/svc/log/oxide-crucible-downstairs:downstairs-ed6bed9f-2d1b-4673-b76d-765adb08f921.log
10112023-11-16T21:30:07.866Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_87cd4f4e-2da6-48ea-a050-f8a6d3f09f5c/root/var/svc/log/oxide-clickhouse:default.log (790855 bytes)
10122023-11-16T21:30:07.902Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_87cd4f4e-2da6-48ea-a050-f8a6d3f09f5c/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 790855 -> 155
10132023-11-16T21:30:08.994Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_87cd4f4e-2da6-48ea-a050-f8a6d3f09f5c/root/var/svc/log/oxide-clickhouse:default.log
10142023-11-16T21:30:09.019Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_0e022914-7f6d-49f8-8f74-41e9adde01ea/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
10152023-11-16T21:30:09.043Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_0e022914-7f6d-49f8-8f74-41e9adde01ea/root/var/svc/log/oxide-cockroachdb:default.log" changed size mid upload: 5343 -> 0
10162023-11-16T21:30:10.115Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_0e022914-7f6d-49f8-8f74-41e9adde01ea/root/var/svc/log/oxide-cockroachdb:default.log
10172023-11-16T21:30:10.165Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_d788df3b-18cd-4866-92f6-c29b4e222c24/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10182023-11-16T21:30:10.189Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_d788df3b-18cd-4866-92f6-c29b4e222c24/root/var/svc/log/oxide-cockroachdb:default.log" changed size mid upload: 5369 -> 0
10192023-11-16T21:30:11.249Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_d788df3b-18cd-4866-92f6-c29b4e222c24/root/var/svc/log/oxide-cockroachdb:default.log
10202023-11-16T21:30:11.273Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0e04aaea-ebe6-4fb5-844c-a4c1a3ce1268/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10212023-11-16T21:30:11.298Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0e04aaea-ebe6-4fb5-844c-a4c1a3ce1268/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 3436 -> 0
10222023-11-16T21:30:12.371Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0e04aaea-ebe6-4fb5-844c-a4c1a3ce1268/root/var/svc/log/oxide-crucible-agent:default.log
10232023-11-16T21:30:12.403Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0e04aaea-ebe6-4fb5-844c-a4c1a3ce1268/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10242023-11-16T21:30:12.434Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0e04aaea-ebe6-4fb5-844c-a4c1a3ce1268/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10252023-11-16T21:30:13.465Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0e04aaea-ebe6-4fb5-844c-a4c1a3ce1268/root/var/svc/log/oxide-crucible-downstairs:default.log
10262023-11-16T21:30:13.490Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_d6e54f4e-7b51-4188-a854-9644f18c90a1/root/var/svc/log/oxide-crucible-pantry:default.log (2337 bytes)
10272023-11-16T21:30:13.514Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_d6e54f4e-7b51-4188-a854-9644f18c90a1/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 2337 -> 0
10282023-11-16T21:30:14.586Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_d6e54f4e-7b51-4188-a854-9644f18c90a1/root/var/svc/log/oxide-crucible-pantry:default.log
10292023-11-16T21:30:14.611Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_61e28bc3-9c4f-45ea-b5d6-5140291717af/root/var/svc/log/oxide-internal_dns:default.log (103626 bytes)
10302023-11-16T21:30:14.635Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_61e28bc3-9c4f-45ea-b5d6-5140291717af/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 103626 -> 0
10312023-11-16T21:30:15.707Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_61e28bc3-9c4f-45ea-b5d6-5140291717af/root/var/svc/log/oxide-internal_dns:default.log
10322023-11-16T21:30:15.732Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_574dc0b9-3dd7-468f-a661-b75f31ebd549/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10332023-11-16T21:30:15.758Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_574dc0b9-3dd7-468f-a661-b75f31ebd549/root/var/svc/log/oxide-cockroachdb:default.log" changed size mid upload: 5369 -> 0
10342023-11-16T21:30:16.829Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_574dc0b9-3dd7-468f-a661-b75f31ebd549/root/var/svc/log/oxide-cockroachdb:default.log
10352023-11-16T21:30:16.853Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7e21f336-c0a5-4716-b66d-bc7b5fb9b78b/root/var/svc/log/oxide-crucible-agent:default.log (25259 bytes)
10362023-11-16T21:30:16.877Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7e21f336-c0a5-4716-b66d-bc7b5fb9b78b/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 25259 -> 0
10372023-11-16T21:30:17.950Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7e21f336-c0a5-4716-b66d-bc7b5fb9b78b/root/var/svc/log/oxide-crucible-agent:default.log
10382023-11-16T21:30:17.975Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7e21f336-c0a5-4716-b66d-bc7b5fb9b78b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10392023-11-16T21:30:17.999Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7e21f336-c0a5-4716-b66d-bc7b5fb9b78b/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10402023-11-16T21:30:19.071Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7e21f336-c0a5-4716-b66d-bc7b5fb9b78b/root/var/svc/log/oxide-crucible-downstairs:default.log
10412023-11-16T21:30:19.097Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7e21f336-c0a5-4716-b66d-bc7b5fb9b78b/root/var/svc/log/oxide-crucible-downstairs:downstairs-1b3b6581-9d5e-41ff-ba9e-dc92ec3ba8bf.log (3049 bytes)
10422023-11-16T21:30:19.121Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7e21f336-c0a5-4716-b66d-bc7b5fb9b78b/root/var/svc/log/oxide-crucible-downstairs:downstairs-1b3b6581-9d5e-41ff-ba9e-dc92ec3ba8bf.log" changed size mid upload: 3049 -> 0
10432023-11-16T21:30:20.193Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7e21f336-c0a5-4716-b66d-bc7b5fb9b78b/root/var/svc/log/oxide-crucible-downstairs:downstairs-1b3b6581-9d5e-41ff-ba9e-dc92ec3ba8bf.log
10442023-11-16T21:30:20.218Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7e21f336-c0a5-4716-b66d-bc7b5fb9b78b/root/var/svc/log/oxide-crucible-downstairs:downstairs-bcad4fab-cd8d-448e-9218-46216a1f619b.log (5536 bytes)
10452023-11-16T21:30:20.242Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7e21f336-c0a5-4716-b66d-bc7b5fb9b78b/root/var/svc/log/oxide-crucible-downstairs:downstairs-bcad4fab-cd8d-448e-9218-46216a1f619b.log" changed size mid upload: 5536 -> 0
10462023-11-16T21:30:21.314Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7e21f336-c0a5-4716-b66d-bc7b5fb9b78b/root/var/svc/log/oxide-crucible-downstairs:downstairs-bcad4fab-cd8d-448e-9218-46216a1f619b.log
10472023-11-16T21:30:21.338Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1/root/var/svc/log/oxide-ntp:default.log (1258 bytes)
10482023-11-16T21:30:21.362Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1/root/var/svc/log/oxide-ntp:default.log" changed size mid upload: 1258 -> 0
10492023-11-16T21:30:22.434Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1/root/var/svc/log/oxide-ntp:default.log
10502023-11-16T21:30:22.458Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dce68356-8099-4814-9329-e60b85c58f83/root/var/svc/log/oxide-crucible-agent:default.log (14580 bytes)
10512023-11-16T21:30:22.485Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dce68356-8099-4814-9329-e60b85c58f83/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 14580 -> 0
10522023-11-16T21:30:23.570Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dce68356-8099-4814-9329-e60b85c58f83/root/var/svc/log/oxide-crucible-agent:default.log
10532023-11-16T21:30:23.594Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dce68356-8099-4814-9329-e60b85c58f83/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10542023-11-16T21:30:23.618Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dce68356-8099-4814-9329-e60b85c58f83/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10552023-11-16T21:30:24.694Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dce68356-8099-4814-9329-e60b85c58f83/root/var/svc/log/oxide-crucible-downstairs:default.log
10562023-11-16T21:30:24.719Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dce68356-8099-4814-9329-e60b85c58f83/root/var/svc/log/oxide-crucible-downstairs:downstairs-61579d48-7245-4573-a21b-8d6cad8dd2cd.log (3046 bytes)
10572023-11-16T21:30:24.743Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dce68356-8099-4814-9329-e60b85c58f83/root/var/svc/log/oxide-crucible-downstairs:downstairs-61579d48-7245-4573-a21b-8d6cad8dd2cd.log" changed size mid upload: 3046 -> 0
10582023-11-16T21:30:25.815Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dce68356-8099-4814-9329-e60b85c58f83/root/var/svc/log/oxide-crucible-downstairs:downstairs-61579d48-7245-4573-a21b-8d6cad8dd2cd.log
10592023-11-16T21:30:25.839Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c52f7e8c-9095-4260-8e14-3324cc1724a0/root/var/svc/log/oxide-nexus:default.log (525747 bytes)
10602023-11-16T21:30:25.875Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c52f7e8c-9095-4260-8e14-3324cc1724a0/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 525747 -> 1824
10612023-11-16T21:30:26.966Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c52f7e8c-9095-4260-8e14-3324cc1724a0/root/var/svc/log/oxide-nexus:default.log
10622023-11-16T21:30:26.991Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_144ea5e2-ff06-4129-812c-7bf59756446d/root/var/svc/log/oxide-oximeter:default.log (219235 bytes)
10632023-11-16T21:30:27.087Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_144ea5e2-ff06-4129-812c-7bf59756446d/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 219235 -> 17834
10642023-11-16T21:30:28.177Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_144ea5e2-ff06-4129-812c-7bf59756446d/root/var/svc/log/oxide-oximeter:default.log
10652023-11-16T21:30:28.202Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_33902602-c1f8-4f3e-8103-51f12955aba7/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10662023-11-16T21:30:28.226Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_33902602-c1f8-4f3e-8103-51f12955aba7/root/var/svc/log/oxide-cockroachdb:default.log" changed size mid upload: 5369 -> 0
10672023-11-16T21:30:29.299Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_33902602-c1f8-4f3e-8103-51f12955aba7/root/var/svc/log/oxide-cockroachdb:default.log
10682023-11-16T21:30:29.323Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_ead9bd9e-14b1-4707-878e-59787036f1aa/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10692023-11-16T21:30:29.348Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_ead9bd9e-14b1-4707-878e-59787036f1aa/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 2338 -> 0
10702023-11-16T21:30:30.419Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_ead9bd9e-14b1-4707-878e-59787036f1aa/root/var/svc/log/oxide-crucible-pantry:default.log
10712023-11-16T21:30:30.444Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_f234ceda-0b7b-4c0f-8a0a-b7dac15accc7/root/var/svc/log/oxide-crucible-pantry:default.log (2331 bytes)
10722023-11-16T21:30:30.469Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_f234ceda-0b7b-4c0f-8a0a-b7dac15accc7/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 2331 -> 0
10732023-11-16T21:30:31.540Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_f234ceda-0b7b-4c0f-8a0a-b7dac15accc7/root/var/svc/log/oxide-crucible-pantry:default.log
10742023-11-16T21:30:31.565Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_515124c2-bee3-458b-8868-6abb9e4fc536/root/var/svc/log/oxide-internal_dns:default.log (99081 bytes)
10752023-11-16T21:30:31.629Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_515124c2-bee3-458b-8868-6abb9e4fc536/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99081 -> 7557
10762023-11-16T21:30:32.721Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_515124c2-bee3-458b-8868-6abb9e4fc536/root/var/svc/log/oxide-internal_dns:default.log
10772023-11-16T21:30:32.750Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1/root/var/log/chrony/measurements.log (7809 bytes)
10782023-11-16T21:30:32.834Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1/root/var/log/chrony/measurements.log" changed size mid upload: 7809 -> 8905
10792023-11-16T21:30:33.925Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1/root/var/log/chrony/measurements.log
10802023-11-16T21:30:33.949Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1/root/var/log/chrony/statistics.log (4797 bytes)
10812023-11-16T21:30:33.995Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1/root/var/log/chrony/statistics.log" changed size mid upload: 4797 -> 5733
10822023-11-16T21:30:35.086Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1/root/var/log/chrony/statistics.log
10832023-11-16T21:30:35.110Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1/root/var/log/chrony/tracking.log (1585 bytes)
10842023-11-16T21:30:35.146Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1/root/var/log/chrony/tracking.log" changed size mid upload: 1585 -> 1717
10852023-11-16T21:30:36.244Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_09e68376-571c-4560-8a7f-f4197b303ae1/root/var/log/chrony/tracking.log