01HDWPFJMDHZG3BNY09K03NWQZ: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HDWPGMBGNARGSR0283DBHNZJ

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-29T04:02:16.130Zjob dependencies complete; ready to run (waiting for 52 m 54 s)
22023-10-29T04:18:18.235Zjob assigned to worker 01HDWTDF6S16KVRRS8AS45JATG (queued for 16 m 2 s)
32023-10-29T04:18:18.444Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-29T04:18:18.493Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-10-29T04:18:18.527Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-10-29T04:18:18.560Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-29T04:18:18.594Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-10-29T04:18:18.627ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-29T04:18:18.661Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-29T04:18:18.694ZOxide Helios Version helios-2.0.22241 64-bit
112023-10-29T04:18:18.728ZConfiguring devices.
122023-10-29T04:18:18.761Z##################################################
132023-10-29T04:18:18.794Z#### ############# ##
142023-10-29T04:18:18.828Z### ### ############ ##
152023-10-29T04:18:18.861Z## ### # ## ### ## Oxide Computer Company ##
162023-10-29T04:18:18.894Z## ## ## ### # ### ##
172023-10-29T04:18:18.928Z## # ### #### #### This Station Under ##
182023-10-29T04:18:18.961Z### ### #### # ### Computer Control ##
192023-10-29T04:18:18.995Z#### #### ### ## ##
202023-10-29T04:18:19.028Z##################################################
212023-10-29T04:18:19.062ZHostname: buskin
222023-10-29T04:18:19.112ZLoading smf(7) serice descriptions:
232023-10-29T04:18:20.020Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-14d6b030e12c027b.gz
242023-10-29T04:18:21.363Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-14d6b030e12c027b.gz
252023-10-29T04:18:21.387Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-29T04:18:39.053Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-29T04:18:39.083Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e13f5e2b300997c0.gz
282023-10-29T04:19:01.105Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e13f5e2b300997c0.gz
292023-10-29T04:19:01.131Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-29T04:19:02.190Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-29T04:19:02.215Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-29T04:19:05.028Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-29T04:19:05.062Zdownloading input: /input/package/work/version.txt
342023-10-29T04:19:05.129Zdownloaded input: /input/package/work/version.txt
352023-10-29T04:19:05.154Zdownloading input: /input/package/work/package.tar.gz
362023-10-29T04:19:21.567Zdownloaded input: /input/package/work/package.tar.gz
372023-10-29T04:19:21.600Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-29T04:19:43.918Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-29T04:19:43.943Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-29T04:19:50.457Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-29T04:19:50.481Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-29T04:20:23.140Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-29T04:20:23.165Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-29T04:20:55.225Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-29T04:20:55.250Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-29T04:21:15.673Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-29T04:21:15.697Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-29T04:21:19.876Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-29T04:21:19.903Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-29T04:21:27.247Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-29T04:21:27.271Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-29T04:21:32.017Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-29T04:21:32.049Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-29T04:21:37.193Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-29T04:21:37.238Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-29T04:21:37.271Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-29T04:21:37.295Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-29T04:21:40.639Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-29T04:21:40.663Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-29T04:21:44.047Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-29T04:21:44.071Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-29T04:21:57.337Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-29T04:21:57.362Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-29T04:22:14.671Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-29T04:22:14.696Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-29T04:22:14.751Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-29T04:22:14.774Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-29T04:22:21.319Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-29T04:22:21.769Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-29T04:22:35.123Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-29T04:22:35.146Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-29T04:23:18.016Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-29T04:23:18.040Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-29T04:23:47.279Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-29T04:23:47.304Zstarting task 0: "setup"
762023-10-29T04:23:47.340Z++ uname -s
772023-10-29T04:23:47.365Z+ kern=SunOS
782023-10-29T04:23:47.393Z+ case "$kern" in
792023-10-29T04:23:47.418Z+ groupadd -g 12345 build
802023-10-29T04:23:47.443Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-29T04:23:49.348Z+ zfs create -o mountpoint=/work rpool/work
822023-10-29T04:23:49.373Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-29T04:23:49.397Z+ home_fs=zfs
842023-10-29T04:23:49.422Z+ [[ zfs == autofs ]]
852023-10-29T04:23:49.447Z+ mkdir -p /home/build
862023-10-29T04:23:49.472Z+ chown build:build /home/build /work
872023-10-29T04:23:51.378Z+ chmod 0700 /home/build /work
882023-10-29T04:23:51.403Zprocess exited: duration 4051 ms, exit code 0
 
892023-10-29T04:23:51.453Zstarting task 1: "authentication"
902023-10-29T04:23:51.522Zprocess exited: duration 44 ms, exit code 0
 
912023-10-29T04:23:51.572Zstarting task 2: "build"
922023-10-29T04:23:51.604Z+ trap _exit_trap EXIT
932023-10-29T04:23:51.629Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-29T04:23:51.802Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-29T04:23:51.827Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-29T04:23:51.852Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-29T04:23:51.876Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-29T04:23:51.901Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-29T04:23:51.926Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-29T04:23:51.951Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-29T04:23:51.980Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-29T04:23:52.005Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-29T04:23:52.030Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-29T04:23:52.055Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-29T04:23:52.080Z+ [[ -d /opt/oxide ]]
1062023-10-29T04:23:52.106Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-29T04:23:52.131Z+ pfexec mkdir /opt/oxide
1082023-10-29T04:23:52.155Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-29T04:23:52.180Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-29T04:23:52.204Z+ cd /opt/oxide-underneath
1112023-10-29T04:23:52.229Z+ pfexec tar ceEp@/f - .
1122023-10-29T04:23:52.253Z+ cd /opt/oxide
1132023-10-29T04:23:52.278Z+ pfexec tar xveEp@/f -
1142023-10-29T04:23:52.302Zx ., 0 bytes, 0 tape blocks
1152023-10-29T04:23:52.327Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-29T04:23:52.352Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-29T04:23:52.376Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-10-29T04:23:52.401Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-29T04:23:52.425Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-29T04:23:52.450Z+ pfexec mkdir /opt/oxide/work
1212023-10-29T04:23:52.475Z+ pfexec chown build:build /opt/oxide/work
1222023-10-29T04:23:52.499Z+ cd /opt/oxide/work
1232023-10-29T04:23:52.524Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-29T04:23:53.580ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-29T04:23:53.604Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1262023-10-29T04:23:53.629Zx out/omicron-sled-agent.tar, 122636800 bytes, 239525 tape blocks
1272023-10-29T04:23:53.654Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-29T04:23:53.679Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-10-29T04:23:53.703Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-10-29T04:23:53.729Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-29T04:23:53.753Zx target/release/omicron-package, 41773664 bytes, 81590 tape blocks
1322023-10-29T04:23:53.778Zx tools/create_virtual_hardware.sh, 2974 bytes, 6 tape blocks
1332023-10-29T04:23:53.802Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-29T04:23:53.827Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-29T04:23:53.852Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-29T04:23:53.877Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-29T04:23:53.901Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-29T04:23:53.926Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-29T04:23:53.950Z
1402023-10-29T04:23:53.975Zreal 1.615744096
1412023-10-29T04:23:53.999Zuser 1.449066590
1422023-10-29T04:23:54.035Zsys 0.293828768
1432023-10-29T04:23:54.060Ztrap 0.000053020
1442023-10-29T04:23:54.085Ztflt 0.000000000
1452023-10-29T04:23:54.109Zdflt 0.000000000
1462023-10-29T04:23:54.134Zkflt 0.000000000
1472023-10-29T04:23:54.159Zlock 0.000000000
1482023-10-29T04:23:54.184Zslp 1.450873690
1492023-10-29T04:23:54.209Zlat 0.035581673
1502023-10-29T04:23:54.233Zstop 0.000122362
1512023-10-29T04:23:54.258Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-29T04:23:54.666Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-29T04:23:54.691Z+ mkdir tests
1542023-10-29T04:23:54.718Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-29T04:23:54.742Z+ ptime -m gunzip
1562023-10-29T04:23:54.768Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-14d6b030e12c027b
1572023-10-29T04:23:54.829Z
1582023-10-29T04:23:54.854Zreal 0.140654668
1592023-10-29T04:23:54.878Zuser 0.129567165
1602023-10-29T04:23:54.903Zsys 0.010848359
1612023-10-29T04:23:54.928Ztrap 0.000015630
1622023-10-29T04:23:54.953Ztflt 0.000000000
1632023-10-29T04:23:54.977Zdflt 0.000000000
1642023-10-29T04:23:55.002Zkflt 0.000000000
1652023-10-29T04:23:55.027Zlock 0.000000000
1662023-10-29T04:23:55.060Zslp 0.000019270
1672023-10-29T04:23:55.085Zlat 0.000021560
1682023-10-29T04:23:55.109Zstop 0.000126002
1692023-10-29T04:23:55.134Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-14d6b030e12c027b
1702023-10-29T04:23:55.158Z+ chmod a+x tests/bootstrap-14d6b030e12c027b
1712023-10-29T04:23:55.183Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-29T04:23:55.234Z+ ptime -m gunzip
1732023-10-29T04:23:55.259Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-29T04:23:57.125Z
1752023-10-29T04:23:57.154Zreal 2.275197105
1762023-10-29T04:23:57.180Zuser 2.122973185
1772023-10-29T04:23:57.205Zsys 0.151876144
1782023-10-29T04:23:57.229Ztrap 0.000129563
1792023-10-29T04:23:57.254Ztflt 0.000000000
1802023-10-29T04:23:57.289Zdflt 0.000000000
1812023-10-29T04:23:57.314Zkflt 0.000000000
1822023-10-29T04:23:57.338Zlock 0.000000000
1832023-10-29T04:23:57.363Zslp 0.000019890
1842023-10-29T04:23:57.387Zlat 0.000014990
1852023-10-29T04:23:57.412Zstop 0.000121082
1862023-10-29T04:23:57.436Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-29T04:23:57.461Z+ chmod a+x tests/bootstrap
1882023-10-29T04:23:57.487Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-29T04:23:57.511Z+ ptime -m gunzip
1902023-10-29T04:23:57.535Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e13f5e2b300997c0
1912023-10-29T04:23:59.693Z
1922023-10-29T04:23:59.718Zreal 2.565107908
1932023-10-29T04:23:59.743Zuser 2.415698362
1942023-10-29T04:23:59.768Zsys 0.148968091
1952023-10-29T04:23:59.792Ztrap 0.000203331
1962023-10-29T04:23:59.817Ztflt 0.000000000
1972023-10-29T04:23:59.841Zdflt 0.000000000
1982023-10-29T04:23:59.866Zkflt 0.000000000
1992023-10-29T04:23:59.890Zlock 0.000000000
2002023-10-29T04:23:59.915Zslp 0.000026360
2012023-10-29T04:23:59.939Zlat 0.000026160
2022023-10-29T04:23:59.964Zstop 0.000124832
2032023-10-29T04:24:00.015Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e13f5e2b300997c0
2042023-10-29T04:24:00.040Z+ chmod a+x tests/end_to_end_tests-e13f5e2b300997c0
2052023-10-29T04:24:00.064Z+ bmat address ls
2062023-10-29T04:24:00.089ZNAME CIDR FIRST LAST COUNT
2072023-10-29T04:24:00.114Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-10-29T04:24:00.139Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-29T04:24:00.163Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-10-29T04:24:00.205Z+ EXTRA_IP_START=10.151.1.164
2112023-10-29T04:24:00.230Z+ EXTRA_IP_END=10.151.1.227
2122023-10-29T04:24:00.255Z+ EXTRA_IP_COUNT=64
2132023-10-29T04:24:00.279Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-10-29T04:24:00.304Z+ EXTRA_IP_BASE=10.151.1
2152023-10-29T04:24:00.329Z+ EXTRA_IP_FOCTET=164
2162023-10-29T04:24:00.353Z+ EXTRA_IP_LOCTET=227
2172023-10-29T04:24:00.378Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-10-29T04:24:00.403Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-10-29T04:24:00.427Z+ DNS_IP1=10.151.1.164
2202023-10-29T04:24:00.452Z+ DNS_IP2=10.151.1.165
2212023-10-29T04:24:00.478Z+ UPLINK_IP=10.151.1.174
2222023-10-29T04:24:00.503Z+ IPPOOL_START=10.151.1.175
2232023-10-29T04:24:00.528Z+ IPPOOL_END=10.151.1.227
2242023-10-29T04:24:00.553Z++ ipadm show-addr -po type,addr
2252023-10-29T04:24:00.578Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-10-29T04:24:00.604Z+ GATEWAY_IP=10.151.1.100
2272023-10-29T04:24:00.629Z+ [[ -n 10.151.1.100 ]]
2282023-10-29T04:24:00.654Z+ ping -s 10.151.1.100 56 1
2292023-10-29T04:24:00.692ZPING 10.151.1.100: 56 data bytes
2302023-10-29T04:24:00.717Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.013 ms
2312023-10-29T04:24:00.741Z
2322023-10-29T04:24:00.766Z----10.151.1.100 PING Statistics----
2332023-10-29T04:24:00.792Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-10-29T04:24:00.816Zround-trip (ms) min/avg/max/stddev = 0.013/0.013/0.013/-nan
2352023-10-29T04:24:00.841Z++ arp -an
2362023-10-29T04:24:00.866Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-10-29T04:24:00.891Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-10-29T04:24:00.916Z+ routeadm -e ipv4-forwarding -u
2392023-10-29T04:24:00.942Z+ PXA_START=10.151.1.164
2402023-10-29T04:24:00.967Z+ PXA_END=10.151.1.227
2412023-10-29T04:24:00.992Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-10-29T04:24:01.016Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-10-29T04:24:01.041Z+ ZPOOL_VDEV_DIR=/scratch
2442023-10-29T04:24:01.066Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-10-29T04:24:01.091Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-10-29T04:24:01.116Z++ cd ./tools
2472023-10-29T04:24:01.141Z++ pwd
2482023-10-29T04:24:01.166Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-10-29T04:24:01.192Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-10-29T04:24:01.230Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-10-29T04:24:01.255Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-10-29T04:24:01.281Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-10-29T04:24:01.306Z++ dladm show-phys -p -o LINK
2542023-10-29T04:24:01.332Z++ head -1
2552023-10-29T04:24:01.359Z+ PHYSICAL_LINK=igb0
2562023-10-29T04:24:01.384Z+ echo 'Using igb0 as physical link'
2572023-10-29T04:24:01.410ZUsing igb0 as physical link
2582023-10-29T04:24:01.435Z+ ensure_run_as_root
2592023-10-29T04:24:01.460Z++ id -u
2602023-10-29T04:24:01.485Z+ [[ 0 -ne 0 ]]
2612023-10-29T04:24:01.510Z+ ensure_zpools
2622023-10-29T04:24:01.536Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-10-29T04:24:01.562Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-10-29T04:24:01.589Z+ readarray -t ZPOOLS
2652023-10-29T04:24:01.615Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-10-29T04:24:01.641Z++ sed 's/[ ",]//g'
2672023-10-29T04:24:01.667Z+ for ZPOOL in "${ZPOOLS[@]}"
2682023-10-29T04:24:01.693Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2692023-10-29T04:24:01.723ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2702023-10-29T04:24:01.755Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2712023-10-29T04:24:01.783Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2722023-10-29T04:24:01.812Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2732023-10-29T04:24:01.841Z0+0 records in
2742023-10-29T04:24:01.871ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2752023-10-29T04:24:01.901Z0+0 records out
2762023-10-29T04:24:01.936ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2772023-10-29T04:24:01.967Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2782023-10-29T04:24:01.998ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2792023-10-29T04:24:02.025Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2802023-10-29T04:24:02.056ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2812023-10-29T04:24:02.087Z+ set +x
2822023-10-29T04:24:02.117ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2832023-10-29T04:24:02.145Z++ zpool list -o name
2842023-10-29T04:24:02.170ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2852023-10-29T04:24:02.196Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2862023-10-29T04:24:02.221ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2872023-10-29T04:24:02.257Z+ [[ -z '' ]]
2882023-10-29T04:24:02.285ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2892023-10-29T04:24:02.316Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2902023-10-29T04:24:02.347ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2912023-10-29T04:24:02.375Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2922023-10-29T04:24:02.400ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2932023-10-29T04:24:02.428Z+ set +x
2942023-10-29T04:24:02.459ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2952023-10-29T04:24:02.486Z+ for ZPOOL in "${ZPOOLS[@]}"
2962023-10-29T04:24:02.515ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2972023-10-29T04:24:02.547Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2982023-10-29T04:24:02.581ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-10-29T04:24:02.611Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3002023-10-29T04:24:02.641ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3012023-10-29T04:24:02.671Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3022023-10-29T04:24:02.703ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3032023-10-29T04:24:02.738Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3042023-10-29T04:24:02.772ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3052023-10-29T04:24:02.800Z0+0 records in
3062023-10-29T04:24:02.830ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3072023-10-29T04:24:02.859Z0+0 records out
3082023-10-29T04:24:02.889ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3092023-10-29T04:24:02.918Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3102023-10-29T04:24:02.948ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3112023-10-29T04:24:02.987Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3122023-10-29T04:24:03.014ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3132023-10-29T04:24:03.042Z+ set +x
3142023-10-29T04:24:03.067ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3152023-10-29T04:24:03.093Z++ zpool list -o name
3162023-10-29T04:24:03.118ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3172023-10-29T04:24:03.144Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3182023-10-29T04:24:03.169ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3192023-10-29T04:24:03.194Z+ [[ -z '' ]]
3202023-10-29T04:24:03.218ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3212023-10-29T04:24:03.244Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-10-29T04:24:03.282ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3232023-10-29T04:24:03.307Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3242023-10-29T04:24:03.333ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3252023-10-29T04:24:03.359Z+ set +x
3262023-10-29T04:24:03.384ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3272023-10-29T04:24:03.411Z+ for ZPOOL in "${ZPOOLS[@]}"
3282023-10-29T04:24:03.436ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3292023-10-29T04:24:03.461Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3302023-10-29T04:24:03.488ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3312023-10-29T04:24:03.514Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3322023-10-29T04:24:03.539ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3332023-10-29T04:24:03.564Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3342023-10-29T04:24:03.591ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3352023-10-29T04:24:03.616Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3362023-10-29T04:24:03.641ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3372023-10-29T04:24:03.666Z0+0 records in
3382023-10-29T04:24:03.691ZSimnet net0/sc0_0 exists
3392023-10-29T04:24:03.717Z0+0 records out
3402023-10-29T04:24:03.742ZSimnet net1/sc1_0 exists
3412023-10-29T04:24:03.767Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3422023-10-29T04:24:03.793ZVnic sc0_1 exists
3432023-10-29T04:24:03.819Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3442023-10-29T04:24:03.844Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3452023-10-29T04:24:03.869Z+ set +x
3462023-10-29T04:24:03.895Z++ zpool list -o name
3472023-10-29T04:24:03.920Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3482023-10-29T04:24:03.945Z+ [[ -z '' ]]
3492023-10-29T04:24:03.971Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-10-29T04:24:03.997Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-10-29T04:24:04.023Z+ set +x
3522023-10-29T04:24:04.048Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-10-29T04:24:04.074Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-10-29T04:24:04.099Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-10-29T04:24:04.124Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-10-29T04:24:04.150Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-10-29T04:24:04.176Z0+0 records in
3582023-10-29T04:24:04.203Z0+0 records out
3592023-10-29T04:24:04.228Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3602023-10-29T04:24:04.254Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-10-29T04:24:04.279Z+ set +x
3622023-10-29T04:24:04.306Z++ zpool list -o name
3632023-10-29T04:24:04.331Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-10-29T04:24:04.356Z+ [[ -z '' ]]
3652023-10-29T04:24:04.383Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-10-29T04:24:04.408Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-10-29T04:24:04.435Z+ set +x
3682023-10-29T04:24:04.462Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-10-29T04:24:04.487Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-10-29T04:24:04.513Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-10-29T04:24:04.538Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-10-29T04:24:04.564Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-10-29T04:24:04.591Z0+0 records in
3742023-10-29T04:24:04.618Z0+0 records out
3752023-10-29T04:24:04.644Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3762023-10-29T04:24:04.669Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-10-29T04:24:04.694Z+ set +x
3782023-10-29T04:24:04.719Z++ zpool list -o name
3792023-10-29T04:24:04.744Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-10-29T04:24:04.769Z+ [[ -z '' ]]
3812023-10-29T04:24:04.794Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-10-29T04:24:04.819Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-10-29T04:24:04.845Z+ set +x
3842023-10-29T04:24:04.871Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-10-29T04:24:04.896Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862023-10-29T04:24:04.924Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872023-10-29T04:24:04.949Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882023-10-29T04:24:04.975Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892023-10-29T04:24:05.027Z0+0 records in
3902023-10-29T04:24:05.052Z0+0 records out
3912023-10-29T04:24:05.077Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-10-29T04:24:05.104Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932023-10-29T04:24:05.130Z+ set +x
3942023-10-29T04:24:05.154Z++ zpool list -o name
3952023-10-29T04:24:05.179Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3962023-10-29T04:24:05.204Z+ [[ -z '' ]]
3972023-10-29T04:24:05.229Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3982023-10-29T04:24:05.254Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3992023-10-29T04:24:05.280Z+ set +x
4002023-10-29T04:24:05.304Z+ for ZPOOL in "${ZPOOLS[@]}"
4012023-10-29T04:24:05.331Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4022023-10-29T04:24:05.355Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4032023-10-29T04:24:05.381Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4042023-10-29T04:24:05.405Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4052023-10-29T04:24:05.431Z0+0 records in
4062023-10-29T04:24:05.458Z0+0 records out
4072023-10-29T04:24:05.484Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4082023-10-29T04:24:05.509Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4092023-10-29T04:24:05.534Z+ set +x
4102023-10-29T04:24:05.559Z++ zpool list -o name
4112023-10-29T04:24:05.584Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4122023-10-29T04:24:05.609Z+ [[ -z '' ]]
4132023-10-29T04:24:05.635Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4142023-10-29T04:24:05.660Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4152023-10-29T04:24:05.685Z+ set +x
4162023-10-29T04:24:05.710Z+ for ZPOOL in "${ZPOOLS[@]}"
4172023-10-29T04:24:05.735Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4182023-10-29T04:24:05.760Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192023-10-29T04:24:05.785Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4202023-10-29T04:24:05.810Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4212023-10-29T04:24:05.834Z0+0 records in
4222023-10-29T04:24:05.860Z0+0 records out
4232023-10-29T04:24:05.885Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4242023-10-29T04:24:05.909Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4252023-10-29T04:24:05.934Z+ set +x
4262023-10-29T04:24:05.958Z++ zpool list -o name
4272023-10-29T04:24:05.983Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4282023-10-29T04:24:06.012Z+ [[ -z '' ]]
4292023-10-29T04:24:06.041Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4302023-10-29T04:24:06.066Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4312023-10-29T04:24:06.091Z+ set +x
4322023-10-29T04:24:06.115Z+ for ZPOOL in "${ZPOOLS[@]}"
4332023-10-29T04:24:06.141Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4342023-10-29T04:24:06.165Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4352023-10-29T04:24:06.191Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4362023-10-29T04:24:06.216Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4372023-10-29T04:24:06.241Z0+0 records in
4382023-10-29T04:24:06.266Z0+0 records out
4392023-10-29T04:24:06.293Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4402023-10-29T04:24:06.318Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4412023-10-29T04:24:06.357Z+ set +x
4422023-10-29T04:24:06.383Z++ zpool list -o name
4432023-10-29T04:24:06.408Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4442023-10-29T04:24:06.433Z+ [[ -z '' ]]
4452023-10-29T04:24:06.458Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4462023-10-29T04:24:06.486Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4472023-10-29T04:24:06.511Z+ set +x
4482023-10-29T04:24:06.535Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4492023-10-29T04:24:06.560Z+ readarray -t ZPOOLS
4502023-10-29T04:24:06.590Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4512023-10-29T04:24:06.630Z++ sed 's/[ ",]//g'
4522023-10-29T04:24:06.683Z+ for ZPOOL in "${ZPOOLS[@]}"
4532023-10-29T04:24:06.718Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4542023-10-29T04:24:06.748Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4552023-10-29T04:24:06.776Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4562023-10-29T04:24:06.801Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4572023-10-29T04:24:06.825Z0+0 records in
4582023-10-29T04:24:06.850Z0+0 records out
4592023-10-29T04:24:06.875Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4602023-10-29T04:24:06.900Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4612023-10-29T04:24:06.925Z+ set +x
4622023-10-29T04:24:06.949Z++ zpool list -o name
4632023-10-29T04:24:06.974Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4642023-10-29T04:24:06.999Z+ [[ -z '' ]]
4652023-10-29T04:24:07.024Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-10-29T04:24:07.049Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4672023-10-29T04:24:07.074Z+ set +x
4682023-10-29T04:24:07.099Z+ for ZPOOL in "${ZPOOLS[@]}"
4692023-10-29T04:24:07.129Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4702023-10-29T04:24:07.160Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4712023-10-29T04:24:07.187Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4722023-10-29T04:24:07.219Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4732023-10-29T04:24:07.245Z0+0 records in
4742023-10-29T04:24:07.269Z0+0 records out
4752023-10-29T04:24:07.294Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4762023-10-29T04:24:07.319Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4772023-10-29T04:24:07.344Z+ set +x
4782023-10-29T04:24:07.381Z++ zpool list -o name
4792023-10-29T04:24:07.406Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4802023-10-29T04:24:07.430Z+ [[ -z '' ]]
4812023-10-29T04:24:07.456Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4822023-10-29T04:24:07.480Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4832023-10-29T04:24:07.505Z+ set +x
4842023-10-29T04:24:07.529Z+ ensure_simulated_links igb0
4852023-10-29T04:24:07.555Z+ local PHYSICAL_LINK=igb0
4862023-10-29T04:24:07.588Z+ INDICES=("0" "1")
4872023-10-29T04:24:07.622Z+ for I in "${INDICES[@]}"
4882023-10-29T04:24:07.648Z++ get_simnet_name_if_exists net0
4892023-10-29T04:24:07.673Z++ dladm show-simnet -p -o LINK net0
4902023-10-29T04:24:07.698Z++ echo ''
4912023-10-29T04:24:07.722Z+ [[ -z '' ]]
4922023-10-29T04:24:07.747Z+ dladm create-simnet -t net0
4932023-10-29T04:24:07.772Z+ dladm create-simnet -t sc0_0
4942023-10-29T04:24:07.796Z+ dladm modify-simnet -t -p net0 sc0_0
4952023-10-29T04:24:07.821Z+ dladm set-linkprop -p mtu=9000 sc0_0
4962023-10-29T04:24:07.846Z+ success 'Simnet net0/sc0_0 exists'
4972023-10-29T04:24:07.871Zdownloading softnpu ...done
4982023-10-29T04:24:07.895Z+ set +x
4992023-10-29T04:24:07.920Z+ for I in "${INDICES[@]}"
5002023-10-29T04:24:07.945Z++ get_simnet_name_if_exists net1
5012023-10-29T04:24:07.970Z++ dladm show-simnet -p -o LINK net1
5022023-10-29T04:24:08.406Z++ echo ''
5032023-10-29T04:24:08.431Z+ [[ -z '' ]]
5042023-10-29T04:24:08.459Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5052023-10-29T04:24:08.484Z+ dladm create-simnet -t net1
5062023-10-29T04:24:08.571Z+ dladm create-simnet -t sc1_0
5072023-10-29T04:24:09.005Z+ dladm modify-simnet -t -p net1 sc1_0
5082023-10-29T04:24:09.034Z+ dladm set-linkprop -p mtu=9000 sc1_0
5092023-10-29T04:24:09.059Z+ success 'Simnet net1/sc1_0 exists'
5102023-10-29T04:24:09.084Z+ set +x
5112023-10-29T04:24:09.108Z++ get_vnic_name_if_exists sc0_1
5122023-10-29T04:24:09.133Z++ dladm show-vnic -p -o LINK sc0_1
5132023-10-29T04:24:09.158Z++ echo ''
5142023-10-29T04:24:09.182Z+ [[ -z '' ]]
5152023-10-29T04:24:09.207Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162023-10-29T04:24:09.232Z+ [[ -v PROMISC_FILT_OFF ]]
5172023-10-29T04:24:09.257Z+ success 'Vnic sc0_1 exists'
5182023-10-29T04:24:09.281Z+ set +x
5192023-10-29T04:24:09.306Z+ warn_if_no_proxy_arp
5202023-10-29T04:24:09.335Z+ [[ -v PXA_START ]]
5212023-10-29T04:24:09.361Z+ [[ -v PXA_END ]]
5222023-10-29T04:24:09.387Z+ ensure_softnpu_zone
5232023-10-29T04:24:09.445Z+ zoneadm list
5242023-10-29T04:24:09.477Z+ grep -q sidecar_softnpu
5252023-10-29T04:24:09.502Z+ [[ -f out/npuzone/npuzone ]]
5262023-10-29T04:24:09.526Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch main
5272023-10-29T04:24:11.724Zdownloading asic_program.so ...done
5282023-10-29T04:24:11.770Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5292023-10-29T04:24:17.392Zdownloading scadm ...done
5302023-10-29T04:24:17.417Zcreating port tfportrear0_0 for sc0_0
5312023-10-29T04:24:17.442Zcreating port tfportqsfp0_0 for sc0_1
5322023-10-29T04:24:17.467Zsoftnpu zone setup
5332023-10-29T04:24:18.518Zconfigure zone
5342023-10-29T04:24:18.542Zinstall zone
5352023-10-29T04:24:18.567Zboot zone
5362023-10-29T04:24:21.634Z[sidecar_softnpu] svcs milestone/network
5372023-10-29T04:24:21.659ZSTATE STIME FMRI
5382023-10-29T04:24:21.684Zonline 4:24:19 svc:/milestone/network:default
5392023-10-29T04:24:21.709Z[sidecar_softnpu] /softnpu/npu start
5402023-10-29T04:24:21.734Z
5412023-10-29T04:24:21.769Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5422023-10-29T04:24:21.795Z+ GATEWAY_IP=10.151.1.100
5432023-10-29T04:24:21.819Z+ echo 'Using 10.151.1.100 as gateway ip'
5442023-10-29T04:24:21.844ZUsing 10.151.1.100 as gateway ip
5452023-10-29T04:24:21.869Z+ [[ ! -v GATEWAY_MAC ]]
5462023-10-29T04:24:21.893Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5472023-10-29T04:24:21.918Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5482023-10-29T04:24:21.943Z+++ echo 18:c0:4d:0d:9f:b2
5492023-10-29T04:24:21.967Z+ COUNT=1
5502023-10-29T04:24:21.992Z+ [[ 1 -ne 1 ]]
5512023-10-29T04:24:22.017Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5522023-10-29T04:24:22.042ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5532023-10-29T04:24:22.067Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5542023-10-29T04:24:22.092Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5552023-10-29T04:24:22.118Z+ PXA_MAC=a8:e1:de:01:70:1d
5562023-10-29T04:24:22.143Z+ [[ -v PXA_START ]]
5572023-10-29T04:24:22.168Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5582023-10-29T04:24:22.192Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5592023-10-29T04:24:22.216Z+ z_scadm dump-state
5602023-10-29T04:24:22.242Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5612023-10-29T04:24:22.267Zlocal v6:
5622023-10-29T04:24:22.291Zlocal v4:
5632023-10-29T04:24:22.316Zrouter v6:
5642023-10-29T04:24:22.340Zrouter v4_idx:
5652023-10-29T04:24:22.365Zrouter v4_routes:
5662023-10-29T04:24:22.389Zresolver v4:
5672023-10-29T04:24:22.415Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5682023-10-29T04:24:22.439Zresolver v6:
5692023-10-29T04:24:22.464Znat v4:
5702023-10-29T04:24:22.488Znat_v6:
5712023-10-29T04:24:22.513Zport_mac:
5722023-10-29T04:24:22.539Zproxy arp:
5732023-10-29T04:24:22.565Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5742023-10-29T04:24:22.669Z+ success 'softnpu zone exists'
5752023-10-29T04:24:22.693Z+ set +x
5762023-10-29T04:24:22.718Zsoftnpu zone exists
5772023-10-29T04:24:22.742Z
5782023-10-29T04:24:22.793Zreal 21.756984420
5792023-10-29T04:24:22.818Zuser 0.735836872
5802023-10-29T04:24:22.843Zsys 1.649786707
5812023-10-29T04:24:22.868Ztrap 0.000073776
5822023-10-29T04:24:22.892Ztflt 0.000120516
5832023-10-29T04:24:22.917Zdflt 0.000614151
5842023-10-29T04:24:22.941Zkflt 0.000000000
5852023-10-29T04:24:22.965Zlock 16:48.353925805
5862023-10-29T04:24:22.990Zslp 1:04.657990204
5872023-10-29T04:24:23.014Zlat 0.060427402
5882023-10-29T04:24:23.039Zstop 0.352589618
5892023-10-29T04:24:23.063Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5902023-10-29T04:24:23.088Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5912023-10-29T04:24:23.113Z+ SILO_NAME=recovery
5922023-10-29T04:24:23.138Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-10-29T04:24:23.163Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5942023-10-29T04:24:23.187Z+ sed -i~ '
5952023-10-29T04:24:23.211Z /^external_dns_ips/c\
5962023-10-29T04:24:23.239Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5972023-10-29T04:24:23.265Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5982023-10-29T04:24:23.291Z /^first/c\
5992023-10-29T04:24:23.317Zfirst = "10.151.1.164"
6002023-10-29T04:24:23.342Z /^last/c\
6012023-10-29T04:24:23.367Zlast = "10.151.1.173"
6022023-10-29T04:24:23.393Z }
6032023-10-29T04:24:23.417Z /^\[rack_network_config/,/^$/ {
6042023-10-29T04:24:23.443Z /^infra_ip_first/c\
6052023-10-29T04:24:23.467Zinfra_ip_first = "10.151.1.174"
6062023-10-29T04:24:23.493Z /^infra_ip_last/c\
6072023-10-29T04:24:23.518Zinfra_ip_last = "10.151.1.174"
6082023-10-29T04:24:23.542Z }
6092023-10-29T04:24:23.567Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6102023-10-29T04:24:23.619Z /^\[\[rack_network_config.ports/,/^$/ {
6112023-10-29T04:24:23.644Z+++ pkg/config-rss.toml Sun Oct 29 04:24:21 2023
6122023-10-29T04:24:23.668Z /^routes/c\
6132023-10-29T04:24:23.694Z@@ -41,7 +41,7 @@
6142023-10-29T04:24:23.718Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6152023-10-29T04:24:23.746Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6162023-10-29T04:24:23.771Z /^addresses/c\
6172023-10-29T04:24:23.816Z # must be contained in one of the "internal services" IP Pool ranges listed
6182023-10-29T04:24:23.840Zaddresses = \["10.151.1.174/32"\]
6192023-10-29T04:24:23.866Z # below.
6202023-10-29T04:24:23.891Z }
6212023-10-29T04:24:23.916Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6222023-10-29T04:24:23.940Z' pkg/config-rss.toml
6232023-10-29T04:24:23.966Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6242023-10-29T04:24:23.992Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6252023-10-29T04:24:24.016Z
6262023-10-29T04:24:24.041Z+ true
6272023-10-29T04:24:24.065Z # Initial TLS certificates for the external API
6282023-10-29T04:24:24.090Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6292023-10-29T04:24:24.116Z #
6302023-10-29T04:24:24.140Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6312023-10-29T04:24:24.165Z@@ -77,8 +77,8 @@
6322023-10-29T04:24:24.190Z+ TLS_NAME=recovery.sys.oxide.test
6332023-10-29T04:24:24.216Z #
6342023-10-29T04:24:24.241Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6352023-10-29T04:24:24.266Z # For more on this and what to put here, see docs/how-to-run.adoc.
6362023-10-29T04:24:24.290Z.....+.+............+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...........+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.....+......+.........+.+............+...+............+..+......+.......+...+.........+......+..+.+.....+.......+........+......+.+........+.+......+.....+.........+.+...+..+....+...+.....+.......+........+..........+..+...+.........................+...........+....+......+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6372023-10-29T04:24:24.314Z [[internal_services_ip_pool_ranges]]
6382023-10-29T04:24:24.339Z-first = "192.168.1.20"
6392023-10-29T04:24:24.365Z-last = "192.168.1.29"
6402023-10-29T04:24:24.389Z+first = "10.151.1.164"
6412023-10-29T04:24:24.414Z+last = "10.151.1.173"
6422023-10-29T04:24:24.438Z
6432023-10-29T04:24:24.464Z # TODO - this configuration is subject to change going forward. Ultimately these
6442023-10-29T04:24:24.488Z # parameters should be provided to the control plane via wicket, but we need to
6452023-10-29T04:24:24.513Z@@ -92,8 +92,8 @@
6462023-10-29T04:24:24.538Z # A range of IP addresses used by Boundary Services on the external network. In
6472023-10-29T04:24:24.562Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6482023-10-29T04:24:24.586Z # With softnpu, only one address is used.
6492023-10-29T04:24:24.611Z-infra_ip_first = "192.168.1.30"
6502023-10-29T04:24:24.637Z-infra_ip_last = "192.168.1.30"
6512023-10-29T04:24:24.661Z+infra_ip_first = "10.151.1.174"
6522023-10-29T04:24:24.686Z+infra_ip_last = "10.151.1.174"
6532023-10-29T04:24:24.711Z......+.....+.+...+..+...+...+...+...................+...+..+.......+.....+..........+......+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.+..+....+...........+...+......+.+..............+...+.+.....+.......+........+.+...+........+.........+....+..+....+.........+..+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+....................+.+..+................+..+.......+...+.....+...+.+...........+.......+............+.....+...+......+.......+........+.......+...+.....+...................+..+...+.............+...+..+..........+...+......+...+..+.............+...........+......+..........+......+..+.......+.....+...+...+.............+..............+.+...........+....+......+......+...+.....+..........+...........+......+.......+...............+...........+....+..+...+..........+.....+...+.......+...+..+.+.....+.......+.........+..................+.................+..................+.......+.....+..........+...+...............+..+...+.+.....+.+......+..+...+...+...............+.......+...+...........+...............+......+..........+..+.+...........+.+..............+.+..............+.........+.+.....+......+.......+........+....+...+........+.......+...+...+.....+.......+..............+....+.....+...................+...+............+...+...+..............+.+..+....+........+.+.....+.+...+..+....+............+..+......+.......+..+.............+.....+...+............+.............+.........+...+........+........................+......................+.....+.........+....+......+......+.....+....+..+..........+.........+........+............+.+...+..+...............+.......+..+....+............+..+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6542023-10-29T04:24:24.739Z
6552023-10-29T04:24:24.768Z-----
6562023-10-29T04:24:24.792Z # Configurations for BGP routers to run on the scrimlets.
6572023-10-29T04:24:24.845Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6582023-10-29T04:24:24.870Z bgp = []
6592023-10-29T04:24:24.896Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6602023-10-29T04:24:24.920Z@@ -101,9 +101,9 @@
6612023-10-29T04:24:24.945Z+ rmdir pkg
6622023-10-29T04:24:25.010Z # You can configure multiple uplinks by repeating the following stanza
6632023-10-29T04:24:25.037Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6642023-10-29T04:24:25.062Z [[rack_network_config.ports]]
6652023-10-29T04:24:25.086Z+ pfexec mkdir -p /usr/oxide
6662023-10-29T04:24:25.112Z # Routes associated with this port.
6672023-10-29T04:24:25.137Z+ pfexec rm -f /usr/oxide/tcpproxy
6682023-10-29T04:24:25.164Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6692023-10-29T04:24:25.189Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6702023-10-29T04:24:25.214Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6712023-10-29T04:24:25.239Z+ pfexec chmod +x /usr/oxide/tcpproxy
6722023-10-29T04:24:25.264Z # Addresses associated with this port.
6732023-10-29T04:24:25.288Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6742023-10-29T04:24:25.313Z-addresses = ["192.168.1.30/32"]
6752023-10-29T04:24:25.337Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6762023-10-29T04:24:25.362Z+addresses = ["10.151.1.174/32"]
6772023-10-29T04:24:25.386Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6782023-10-29T04:24:25.411Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6792023-10-29T04:24:25.435Z+ OMICRON_NO_UNINSTALL=1
6802023-10-29T04:24:25.460Z port = "qsfp0"
6812023-10-29T04:24:25.484Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6822023-10-29T04:24:25.509Z # The speed of this port.
6832023-10-29T04:24:25.535ZOct 29 04:24:23.170 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6842023-10-29T04:24:25.560Za pkg/config-rss.toml 7K
6852023-10-29T04:24:25.586ZOct 29 04:24:23.172 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6862023-10-29T04:24:25.610Za pkg/initial-tls-cert.pem 2K
6872023-10-29T04:24:25.635ZOct 29 04:24:23.172 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6882023-10-29T04:24:25.661Za pkg/initial-tls-key.pem 4K
6892023-10-29T04:24:25.686ZOct 29 04:24:23.172 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6902023-10-29T04:24:25.710ZOct 29 04:24:23.172 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6912023-10-29T04:24:25.735ZOct 29 04:24:23.172 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6922023-10-29T04:24:25.760ZOct 29 04:24:23.172 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6932023-10-29T04:24:25.785ZOct 29 04:24:23.172 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6942023-10-29T04:24:25.809ZOct 29 04:24:23.173 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6952023-10-29T04:24:25.834ZOct 29 04:24:23.173 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6962023-10-29T04:24:25.868ZOct 29 04:24:23.173 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6972023-10-29T04:24:25.894ZOct 29 04:24:23.173 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6982023-10-29T04:24:25.919ZOct 29 04:24:23.173 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6992023-10-29T04:24:25.943ZOct 29 04:24:23.173 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7002023-10-29T04:24:25.968ZOct 29 04:24:23.173 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7012023-10-29T04:24:25.993ZOct 29 04:24:23.173 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7022023-10-29T04:24:26.017ZOct 29 04:24:23.173 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7032023-10-29T04:24:26.042ZOct 29 04:24:23.413 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7042023-10-29T04:24:26.069ZOct 29 04:24:23.449 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7052023-10-29T04:24:26.094ZOct 29 04:24:23.522 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7062023-10-29T04:24:26.121Z
7072023-10-29T04:24:26.146Zreal 0.491892822
7082023-10-29T04:24:26.171Zuser 0.099160582
7092023-10-29T04:24:26.196Zsys 1.890787791
7102023-10-29T04:24:26.221Ztrap 0.000000200
7112023-10-29T04:24:26.246Ztflt 0.000004578
7122023-10-29T04:24:26.271Zdflt 0.001124551
7132023-10-29T04:24:26.301Zkflt 0.000000000
7142023-10-29T04:24:26.326Zlock 44.287895724
7152023-10-29T04:24:26.350Zslp 0.708797416
7162023-10-29T04:24:26.375Zlat 0.003029938
7172023-10-29T04:24:26.400Zstop 0.217998060
7182023-10-29T04:24:26.424Z+ retry=0
7192023-10-29T04:24:26.453Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7202023-10-29T04:24:26.487Z+ [[ 0 -gt 30 ]]
7212023-10-29T04:24:26.512Z+ sleep 1
7222023-10-29T04:24:26.538Z+ retry=1
7232023-10-29T04:24:26.564Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7242023-10-29T04:24:26.589Z+ [[ 1 -gt 30 ]]
7252023-10-29T04:24:26.613Z+ sleep 1
7262023-10-29T04:24:26.640Z+ retry=2
7272023-10-29T04:24:26.664Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7282023-10-29T04:24:51.139Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7292023-10-29T04:24:51.169Zlocal v6:
7302023-10-29T04:24:51.193Zfe80::9030:adff:fe8a:574f
7312023-10-29T04:24:51.218Zfd00:99::1
7322023-10-29T04:24:51.246Zlocal v4:
7332023-10-29T04:24:51.271Z10.151.1.174
7342023-10-29T04:24:51.296Zrouter v6:
7352023-10-29T04:24:51.322Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::18b8:26ff:fea1:3f13 (1)
7362023-10-29T04:24:51.347Zfd00:1122:3344:101::/64 -> fe80::18b8:26ff:fea1:3f13 (1)
7372023-10-29T04:24:51.372Zrouter v4_idx:
7382023-10-29T04:24:51.397Z0.0.0.0/0 -> 0
7392023-10-29T04:24:51.422Zrouter v4_routes:
7402023-10-29T04:24:51.447Z0 -> 10.151.1.100 (2)
7412023-10-29T04:24:51.472Zresolver v4:
7422023-10-29T04:24:51.509Z10.151.1.174 -> 36:59:7e:6f:2d:24
7432023-10-29T04:24:51.534Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7442023-10-29T04:24:51.559Zresolver v6:
7452023-10-29T04:24:51.584Zfe80::18b8:26ff:fea1:3f13 -> 1a:b8:26:a1:3f:13
7462023-10-29T04:24:51.609Zfe80::9030:adff:fe8a:574f -> 92:30:ad:8a:57:4f
7472023-10-29T04:24:51.634Znat v4:
7482023-10-29T04:24:51.659Znat_v6:
7492023-10-29T04:24:51.684Zport_mac:
7502023-10-29T04:24:51.709Z1: a8:40:25:0b:f8:a5
7512023-10-29T04:24:51.734Z1000: a8:40:25:0b:f8:a4
7522023-10-29T04:24:51.759Zproxy arp:
7532023-10-29T04:24:51.784Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7542023-10-29T04:24:51.809Z+ retry=0
7552023-10-29T04:24:51.834Z+++ zoneadm list -n
7562023-10-29T04:24:51.859Z+++ grep oxz_ntp
7572023-10-29T04:24:51.885Z++ pfexec svcs -z -Hostate oxide/ntp
7582023-10-29T04:24:51.909Zsvcs: zone '-Hostate': entity not found
7592023-10-29T04:24:51.935Z++ true
7602023-10-29T04:24:51.960Z+ [[ '' != online ]]
7612023-10-29T04:24:51.985Z+ [[ 0 -gt 60 ]]
7622023-10-29T04:24:52.010Z+ sleep 1
7632023-10-29T04:24:52.195Z+ retry=1
7642023-10-29T04:24:52.220Z+++ zoneadm list -n
7652023-10-29T04:24:52.245Z+++ grep oxz_ntp
7662023-10-29T04:24:52.272Z++ pfexec svcs -z -Hostate oxide/ntp
7672023-10-29T04:24:52.297Zsvcs: zone '-Hostate': entity not found
7682023-10-29T04:24:52.321Z++ true
7692023-10-29T04:24:52.346Z+ [[ '' != online ]]
7702023-10-29T04:24:52.371Z+ [[ 1 -gt 60 ]]
7712023-10-29T04:24:52.396Z+ sleep 1
7722023-10-29T04:24:53.216Z+ retry=2
7732023-10-29T04:24:53.241Z+++ zoneadm list -n
7742023-10-29T04:24:53.266Z+++ grep oxz_ntp
7752023-10-29T04:24:53.291Z++ pfexec svcs -z -Hostate oxide/ntp
7762023-10-29T04:24:53.316Zsvcs: zone '-Hostate': entity not found
7772023-10-29T04:24:53.341Z++ true
7782023-10-29T04:24:53.366Z+ [[ '' != online ]]
7792023-10-29T04:24:53.390Z+ [[ 2 -gt 60 ]]
7802023-10-29T04:24:53.416Z+ sleep 1
7812023-10-29T04:24:54.238Z+ retry=3
7822023-10-29T04:24:54.263Z+++ zoneadm list -n
7832023-10-29T04:24:54.287Z+++ grep oxz_ntp
7842023-10-29T04:24:54.312Z++ pfexec svcs -z -Hostate oxide/ntp
7852023-10-29T04:24:54.337Zsvcs: zone '-Hostate': entity not found
7862023-10-29T04:24:54.362Z++ true
7872023-10-29T04:24:54.387Z+ [[ '' != online ]]
7882023-10-29T04:24:54.412Z+ [[ 3 -gt 60 ]]
7892023-10-29T04:24:54.436Z+ sleep 1
7902023-10-29T04:24:55.260Z+ retry=4
7912023-10-29T04:24:55.285Z+++ zoneadm list -n
7922023-10-29T04:24:55.309Z+++ grep oxz_ntp
7932023-10-29T04:24:55.334Z++ pfexec svcs -z -Hostate oxide/ntp
7942023-10-29T04:24:55.359Zsvcs: zone '-Hostate': entity not found
7952023-10-29T04:24:55.384Z++ true
7962023-10-29T04:24:55.408Z+ [[ '' != online ]]
7972023-10-29T04:24:55.433Z+ [[ 4 -gt 60 ]]
7982023-10-29T04:24:55.458Z+ sleep 1
7992023-10-29T04:24:56.282Z+ retry=5
8002023-10-29T04:24:56.308Z+++ zoneadm list -n
8012023-10-29T04:24:56.333Z+++ grep oxz_ntp
8022023-10-29T04:24:56.357Z++ pfexec svcs -z -Hostate oxide/ntp
8032023-10-29T04:24:56.382Zsvcs: zone '-Hostate': entity not found
8042023-10-29T04:24:56.407Z++ true
8052023-10-29T04:24:56.432Z+ [[ '' != online ]]
8062023-10-29T04:24:56.457Z+ [[ 5 -gt 60 ]]
8072023-10-29T04:24:56.481Z+ sleep 1
8082023-10-29T04:24:57.305Z+ retry=6
8092023-10-29T04:24:57.329Z+++ zoneadm list -n
8102023-10-29T04:24:57.354Z+++ grep oxz_ntp
8112023-10-29T04:24:57.379Z++ pfexec svcs -z -Hostate oxide/ntp
8122023-10-29T04:24:57.404Zsvcs: zone '-Hostate': entity not found
8132023-10-29T04:24:57.429Z++ true
8142023-10-29T04:24:57.454Z+ [[ '' != online ]]
8152023-10-29T04:24:57.479Z+ [[ 6 -gt 60 ]]
8162023-10-29T04:24:57.505Z+ sleep 1
8172023-10-29T04:24:58.325Z+ retry=7
8182023-10-29T04:24:58.350Z+++ zoneadm list -n
8192023-10-29T04:24:58.375Z+++ grep oxz_ntp
8202023-10-29T04:24:58.400Z++ pfexec svcs -z -Hostate oxide/ntp
8212023-10-29T04:24:58.424Zsvcs: zone '-Hostate': entity not found
8222023-10-29T04:24:58.449Z++ true
8232023-10-29T04:24:58.475Z+ [[ '' != online ]]
8242023-10-29T04:24:58.499Z+ [[ 7 -gt 60 ]]
8252023-10-29T04:24:58.524Z+ sleep 1
8262023-10-29T04:24:59.347Z+ retry=8
8272023-10-29T04:24:59.372Z+++ zoneadm list -n
8282023-10-29T04:24:59.397Z+++ grep oxz_ntp
8292023-10-29T04:24:59.424Z++ pfexec svcs -z -Hostate oxide/ntp
8302023-10-29T04:24:59.450Zsvcs: zone '-Hostate': entity not found
8312023-10-29T04:24:59.474Z++ true
8322023-10-29T04:24:59.499Z+ [[ '' != online ]]
8332023-10-29T04:24:59.524Z+ [[ 8 -gt 60 ]]
8342023-10-29T04:24:59.550Z+ sleep 1
8352023-10-29T04:25:00.369Z+ retry=9
8362023-10-29T04:25:00.395Z+++ zoneadm list -n
8372023-10-29T04:25:00.420Z+++ grep oxz_ntp
8382023-10-29T04:25:00.445Z++ pfexec svcs -z -Hostate oxide/ntp
8392023-10-29T04:25:00.469Zsvcs: zone '-Hostate': entity not found
8402023-10-29T04:25:00.494Z++ true
8412023-10-29T04:25:00.519Z+ [[ '' != online ]]
8422023-10-29T04:25:00.544Z+ [[ 9 -gt 60 ]]
8432023-10-29T04:25:00.569Z+ sleep 1
8442023-10-29T04:25:01.391Z+ retry=10
8452023-10-29T04:25:01.416Z+++ zoneadm list -n
8462023-10-29T04:25:01.442Z+++ grep oxz_ntp
8472023-10-29T04:25:01.467Z++ pfexec svcs -z oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8 -Hostate oxide/ntp
8482023-10-29T04:25:01.496Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8492023-10-29T04:25:01.521Z++ true
8502023-10-29T04:25:01.547Z+ [[ '' != online ]]
8512023-10-29T04:25:01.575Z+ [[ 10 -gt 60 ]]
8522023-10-29T04:25:01.600Z+ sleep 1
8532023-10-29T04:25:02.423Z+ retry=11
8542023-10-29T04:25:02.449Z+++ zoneadm list -n
8552023-10-29T04:25:02.475Z+++ grep oxz_ntp
8562023-10-29T04:25:02.501Z++ pfexec svcs -z oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8 -Hostate oxide/ntp
8572023-10-29T04:25:02.526Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8582023-10-29T04:25:02.552Z++ true
8592023-10-29T04:25:02.578Z+ [[ '' != online ]]
8602023-10-29T04:25:02.603Z+ [[ 11 -gt 60 ]]
8612023-10-29T04:25:02.629Z+ sleep 1
8622023-10-29T04:25:03.453Z+ retry=12
8632023-10-29T04:25:03.480Z+++ zoneadm list -n
8642023-10-29T04:25:03.506Z+++ grep oxz_ntp
8652023-10-29T04:25:03.532Z++ pfexec svcs -z oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8 -Hostate oxide/ntp
8662023-10-29T04:25:03.558Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8672023-10-29T04:25:03.584Z++ true
8682023-10-29T04:25:03.609Z+ [[ '' != online ]]
8692023-10-29T04:25:03.634Z+ [[ 12 -gt 60 ]]
8702023-10-29T04:25:03.660Z+ sleep 1
8712023-10-29T04:25:04.483Z+ retry=13
8722023-10-29T04:25:04.509Z+++ zoneadm list -n
8732023-10-29T04:25:04.534Z+++ grep oxz_ntp
8742023-10-29T04:25:04.559Z++ pfexec svcs -z oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8 -Hostate oxide/ntp
8752023-10-29T04:25:04.587Z+ [[ disabled != online ]]
8762023-10-29T04:25:04.616Z+ [[ 13 -gt 60 ]]
8772023-10-29T04:25:04.641Z+ sleep 1
8782023-10-29T04:25:05.507Z+ retry=14
8792023-10-29T04:25:05.534Z+++ zoneadm list -n
8802023-10-29T04:25:05.559Z+++ grep oxz_ntp
8812023-10-29T04:25:05.584Z++ pfexec svcs -z oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8 -Hostate oxide/ntp
8822023-10-29T04:25:05.609Z+ [[ disabled != online ]]
8832023-10-29T04:25:05.634Z+ [[ 14 -gt 60 ]]
8842023-10-29T04:25:05.659Z+ sleep 1
8852023-10-29T04:25:06.529Z+ retry=15
8862023-10-29T04:25:06.554Z+++ zoneadm list -n
8872023-10-29T04:25:06.580Z+++ grep oxz_ntp
8882023-10-29T04:25:06.606Z++ pfexec svcs -z oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8 -Hostate oxide/ntp
8892023-10-29T04:25:06.634Z+ [[ disabled != online ]]
8902023-10-29T04:25:06.660Z+ [[ 15 -gt 60 ]]
8912023-10-29T04:25:06.686Z+ sleep 1
8922023-10-29T04:25:07.549Z+ retry=16
8932023-10-29T04:25:07.574Z+++ zoneadm list -n
8942023-10-29T04:25:07.599Z+++ grep oxz_ntp
8952023-10-29T04:25:07.624Z++ pfexec svcs -z oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8 -Hostate oxide/ntp
8962023-10-29T04:25:07.650Z+ [[ disabled != online ]]
8972023-10-29T04:25:07.675Z+ [[ 16 -gt 60 ]]
8982023-10-29T04:25:07.700Z+ sleep 1
8992023-10-29T04:25:08.570Z+ retry=17
9002023-10-29T04:25:08.595Z+++ zoneadm list -n
9012023-10-29T04:25:08.622Z+++ grep oxz_ntp
9022023-10-29T04:25:08.647Z++ pfexec svcs -z oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8 -Hostate oxide/ntp
9032023-10-29T04:25:08.673Z+ [[ disabled != online ]]
9042023-10-29T04:25:08.698Z+ [[ 17 -gt 60 ]]
9052023-10-29T04:25:08.769Z+ sleep 1
9062023-10-29T04:25:09.592Z+ retry=18
9072023-10-29T04:25:09.617Z+++ zoneadm list -n
9082023-10-29T04:25:09.642Z+++ grep oxz_ntp
9092023-10-29T04:25:09.775Z++ pfexec svcs -z oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8 -Hostate oxide/ntp
9102023-10-29T04:25:09.800Z+ [[ disabled != online ]]
9112023-10-29T04:25:09.839Z+ [[ 18 -gt 60 ]]
9122023-10-29T04:25:09.864Z+ sleep 1
9132023-10-29T04:25:10.614Z+ retry=19
9142023-10-29T04:25:10.639Z+++ zoneadm list -n
9152023-10-29T04:25:10.663Z+++ grep oxz_ntp
9162023-10-29T04:25:10.688Z++ pfexec svcs -z oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8 -Hostate oxide/ntp
9172023-10-29T04:25:10.713Z+ [[ disabled != online ]]
9182023-10-29T04:25:10.738Z+ [[ 19 -gt 60 ]]
9192023-10-29T04:25:10.764Z+ sleep 1
 
9202023-10-29T04:25:11.569Zbuskin console login: Oct 29 04:25:10 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9212023-10-29T04:25:11.636Z+ retry=20
9222023-10-29T04:25:11.660Z+++ zoneadm list -n
9232023-10-29T04:25:11.686Z+++ grep oxz_ntp
9242023-10-29T04:25:11.711Z++ pfexec svcs -z oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8 -Hostate oxide/ntp
9252023-10-29T04:25:11.736Z+ [[ disabled != online ]]
9262023-10-29T04:25:11.761Z+ [[ 20 -gt 60 ]]
9272023-10-29T04:25:11.786Z+ sleep 1
9282023-10-29T04:25:12.659Z+ retry=21
9292023-10-29T04:25:12.684Z+++ zoneadm list -n
9302023-10-29T04:25:12.709Z+++ grep oxz_ntp
9312023-10-29T04:25:12.734Z++ pfexec svcs -z oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8 -Hostate oxide/ntp
9322023-10-29T04:25:12.759Z+ [[ online != online ]]
9332023-10-29T04:25:12.784Z+ echo 'Waited for chrony: 21s'
9342023-10-29T04:25:12.809Z+ export RUST_BACKTRACE=1
9352023-10-29T04:25:12.834Z+ RUST_BACKTRACE=1
9362023-10-29T04:25:12.860ZWaited for chrony: 21s
9372023-10-29T04:25:12.884Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9382023-10-29T04:25:12.909Z+ ./tests/bootstrap
9392023-10-29T04:25:12.934ZUsing external DNS server at 10.151.1.164:53
9402023-10-29T04:25:12.959Z2023-10-29 04:25:12.424048083 UTC: attempting to log into API
9412023-10-29T04:25:27.753Z2023-10-29 04:25:27.797991361 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9422023-10-29T04:25:28.749Z2023-10-29 04:25:28.798886606 UTC: attempting to log into API
9432023-10-29T04:25:43.782Z2023-10-29 04:25:43.832343291 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9442023-10-29T04:25:44.785Z2023-10-29 04:25:44.834226754 UTC: attempting to log into API
9452023-10-29T04:25:59.817Z2023-10-29 04:25:59.867255830 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9462023-10-29T04:26:00.819Z2023-10-29 04:26:00.868955200 UTC: attempting to log into API
9472023-10-29T04:26:15.852Z2023-10-29 04:26:15.901729193 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9482023-10-29T04:26:16.853Z2023-10-29 04:26:16.902496726 UTC: attempting to log into API
9492023-10-29T04:26:31.887Z2023-10-29 04:26:31.915135213 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9502023-10-29T04:26:32.892Z2023-10-29 04:26:32.913924502 UTC: attempting to log into API
9512023-10-29T04:26:48.327Z2023-10-29 04:26:47.913805710 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9522023-10-29T04:26:48.922Z2023-10-29 04:26:48.912222731 UTC: attempting to log into API
9532023-10-29T04:27:03.955Z2023-10-29 04:27:03.933990514 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9542023-10-29T04:27:04.956Z2023-10-29 04:27:04.934770377 UTC: attempting to log into API
9552023-10-29T04:27:19.988Z2023-10-29 04:27:19.967110783 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9562023-10-29T04:27:20.990Z2023-10-29 04:27:20.968193881 UTC: attempting to log into API
9572023-10-29T04:27:36.024Z2023-10-29 04:27:36.012148500 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9582023-10-29T04:27:37.024Z2023-10-29 04:27:37.012872072 UTC: attempting to log into API
9592023-10-29T04:27:52.059Z2023-10-29 04:27:52.046158306 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9602023-10-29T04:27:53.057Z2023-10-29 04:27:53.046368719 UTC: attempting to log into API
9612023-10-29T04:28:08.092Z2023-10-29 04:28:08.080438884 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9622023-10-29T04:28:09.093Z2023-10-29 04:28:09.081352411 UTC: attempting to log into API
9632023-10-29T04:28:24.127Z2023-10-29 04:28:24.115690071 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9642023-10-29T04:28:25.129Z2023-10-29 04:28:25.116341963 UTC: attempting to log into API
9652023-10-29T04:28:40.171Z2023-10-29 04:28:40.150586140 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9662023-10-29T04:28:41.165Z2023-10-29 04:28:41.152406403 UTC: attempting to log into API
9672023-10-29T04:28:56.199Z2023-10-29 04:28:56.185722832 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9682023-10-29T04:28:57.204Z2023-10-29 04:28:57.186792271 UTC: attempting to log into API
9692023-10-29T04:29:24.709Z2023-10-29 04:29:12.220351716 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9702023-10-29T04:29:25.619Z2023-10-29 04:29:13.221172750 UTC: attempting to log into API
9712023-10-29T04:29:28.411Z2023-10-29 04:29:28.261500297 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9722023-10-29T04:29:29.324Z2023-10-29 04:29:29.262276081 UTC: attempting to log into API
9732023-10-29T04:29:44.309Z2023-10-29 04:29:44.298367329 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9742023-10-29T04:29:45.310Z2023-10-29 04:29:45.299472558 UTC: attempting to log into API
9752023-10-29T04:30:00.344Z2023-10-29 04:30:00.333108062 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9762023-10-29T04:30:01.357Z2023-10-29 04:30:01.335069076 UTC: attempting to log into API
 
9772023-10-29T04:30:04.781ZOct 29 04:30:02 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9782023-10-29T04:30:09.519ZOct 29 04:30:08 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9792023-10-29T04:30:15.012ZOct 29 04:30:14 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9802023-10-29T04:30:16.387Z2023-10-29 04:30:16.369772600 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9812023-10-29T04:30:17.382Z2023-10-29 04:30:17.370768707 UTC: attempting to log into API
 
9822023-10-29T04:30:20.732ZOct 29 04:30:20 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9832023-10-29T04:30:28.007ZOct 29 04:30:27 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9842023-10-29T04:30:32.418Z2023-10-29 04:30:32.404187288 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9852023-10-29T04:30:33.419Z2023-10-29 04:30:33.405307808 UTC: attempting to log into API
9862023-10-29T04:30:44.217Z2023-10-29 04:30:44.204362816 UTC: login succeeded
9872023-10-29T04:30:44.248Zwaiting for nexus to come up...
9882023-10-29T04:30:44.311Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9892023-10-29T04:30:44.367Zensuring datasets are ready...
9902023-10-29T04:30:46.999Zlet's roll.
9912023-10-29T04:30:47.027Z+ rm ./tests/bootstrap
9922023-10-29T04:30:47.063Z+ for test_bin in tests/*
9932023-10-29T04:30:47.091Z+ ./tests/bootstrap-14d6b030e12c027b
9942023-10-29T04:30:47.116Z
9952023-10-29T04:30:47.141Zrunning 0 tests
9962023-10-29T04:30:47.165Z
9972023-10-29T04:30:47.189Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9982023-10-29T04:30:47.214Z
9992023-10-29T04:30:47.239Z+ for test_bin in tests/*
10002023-10-29T04:30:47.263Z+ ./tests/end_to_end_tests-e13f5e2b300997c0
10012023-10-29T04:30:47.288Z
10022023-10-29T04:30:47.312Zrunning 2 tests
10032023-10-29T04:30:47.337Ztest no_spoof::no_spoof_here ... ok
10042023-10-29T04:31:47.086Ztest instance_launch::instance_launch has been running for over 60 seconds
10052023-10-29T04:33:40.864Ztest instance_launch::instance_launch ... ok
10062023-10-29T04:33:40.890Z
10072023-10-29T04:33:40.924Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 173.01s
10082023-10-29T04:33:40.949Z
10092023-10-29T04:33:40.974Z+ _exit_trap
10102023-10-29T04:33:41.000Z+ local status=0
10112023-10-29T04:33:41.025Z+ [[ 0 -eq 0 ]]
10122023-10-29T04:33:41.050Z+ exit 0
10132023-10-29T04:33:41.075Zprocess exited: duration 588501 ms, exit code 0
 
10142023-10-29T04:33:41.180Zfound 84 output files
10152023-10-29T04:33:41.385Zuploading: /var/svc/log/oxide-sled-agent:default.log (478414 bytes)
10162023-10-29T04:33:42.469Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10172023-10-29T04:33:42.505Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_aca603b7-a792-44e2-8b52-46d38f7c09f3/root/var/svc/log/oxide-clickhouse:default.log (575083 bytes)
10182023-10-29T04:33:44.329Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_aca603b7-a792-44e2-8b52-46d38f7c09f3/root/var/svc/log/oxide-clickhouse:default.log
10192023-10-29T04:33:44.354Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_aca603b7-a792-44e2-8b52-46d38f7c09f3/root/var/svc/log/oxide-clickhouse:default.log.0 (11904 bytes)
10202023-10-29T04:33:45.449Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_aca603b7-a792-44e2-8b52-46d38f7c09f3/root/var/svc/log/oxide-clickhouse:default.log.0
10212023-10-29T04:33:45.473Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2992609-9231-4847-b522-dd9d3d73dc31/root/var/svc/log/oxide-crucible-agent:default.log (10253 bytes)
10222023-10-29T04:33:46.578Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2992609-9231-4847-b522-dd9d3d73dc31/root/var/svc/log/oxide-crucible-agent:default.log
10232023-10-29T04:33:46.613Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2992609-9231-4847-b522-dd9d3d73dc31/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10242023-10-29T04:33:47.711Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2992609-9231-4847-b522-dd9d3d73dc31/root/var/svc/log/oxide-crucible-agent:default.log.0
10252023-10-29T04:33:48.163Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2992609-9231-4847-b522-dd9d3d73dc31/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10262023-10-29T04:33:49.202Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2992609-9231-4847-b522-dd9d3d73dc31/root/var/svc/log/oxide-crucible-downstairs:default.log
10272023-10-29T04:33:49.227Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2992609-9231-4847-b522-dd9d3d73dc31/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10282023-10-29T04:33:50.297Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2992609-9231-4847-b522-dd9d3d73dc31/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10292023-10-29T04:33:50.322Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2992609-9231-4847-b522-dd9d3d73dc31/root/var/svc/log/oxide-crucible-downstairs:downstairs-2c2b4642-22ed-4458-8469-9e21cd230742.log (5515 bytes)
10302023-10-29T04:33:51.407Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2992609-9231-4847-b522-dd9d3d73dc31/root/var/svc/log/oxide-crucible-downstairs:downstairs-2c2b4642-22ed-4458-8469-9e21cd230742.log
10312023-10-29T04:33:51.438Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_73454b7b-a951-4985-a2ff-9dee4b2a84fb/root/var/svc/log/oxide-external_dns:default.log (15493 bytes)
10322023-10-29T04:33:52.542Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_73454b7b-a951-4985-a2ff-9dee4b2a84fb/root/var/svc/log/oxide-external_dns:default.log
10332023-10-29T04:33:52.566Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_73454b7b-a951-4985-a2ff-9dee4b2a84fb/root/var/svc/log/oxide-external_dns:default.log.0 (75 bytes)
10342023-10-29T04:33:53.648Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_73454b7b-a951-4985-a2ff-9dee4b2a84fb/root/var/svc/log/oxide-external_dns:default.log.0
10352023-10-29T04:33:53.679Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_761159ff-5b3d-4b72-9cbb-31baf88045d6/root/var/svc/log/oxide-crucible-agent:default.log (10682 bytes)
10362023-10-29T04:33:54.867Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_761159ff-5b3d-4b72-9cbb-31baf88045d6/root/var/svc/log/oxide-crucible-agent:default.log
10372023-10-29T04:33:54.892Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_761159ff-5b3d-4b72-9cbb-31baf88045d6/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10382023-10-29T04:33:55.978Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_761159ff-5b3d-4b72-9cbb-31baf88045d6/root/var/svc/log/oxide-crucible-agent:default.log.0
10392023-10-29T04:33:56.013Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_761159ff-5b3d-4b72-9cbb-31baf88045d6/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10402023-10-29T04:33:57.474Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_761159ff-5b3d-4b72-9cbb-31baf88045d6/root/var/svc/log/oxide-crucible-downstairs:default.log
10412023-10-29T04:33:57.499Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_761159ff-5b3d-4b72-9cbb-31baf88045d6/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10422023-10-29T04:33:58.988Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_761159ff-5b3d-4b72-9cbb-31baf88045d6/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10432023-10-29T04:33:59.024Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_761159ff-5b3d-4b72-9cbb-31baf88045d6/root/var/svc/log/oxide-crucible-downstairs:downstairs-d3938fea-4402-4046-a90f-3329f9ff17a5.log (3038 bytes)
10442023-10-29T04:34:00.107Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_761159ff-5b3d-4b72-9cbb-31baf88045d6/root/var/svc/log/oxide-crucible-downstairs:downstairs-d3938fea-4402-4046-a90f-3329f9ff17a5.log
10452023-10-29T04:34:00.134Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8/root/var/svc/log/oxide-ntp:default.log (0 bytes)
10462023-10-29T04:34:01.181Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8/root/var/svc/log/oxide-ntp:default.log
10472023-10-29T04:34:01.215Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8/root/var/svc/log/oxide-ntp:default.log.0 (1052 bytes)
10482023-10-29T04:34:02.316Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8/root/var/svc/log/oxide-ntp:default.log.0
10492023-10-29T04:34:02.399Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6b9c9c83-afae-4241-82ae-da761d90258a/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10502023-10-29T04:34:03.525Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6b9c9c83-afae-4241-82ae-da761d90258a/root/var/svc/log/oxide-cockroachdb:default.log
10512023-10-29T04:34:03.555Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6b9c9c83-afae-4241-82ae-da761d90258a/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
10522023-10-29T04:34:04.660Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6b9c9c83-afae-4241-82ae-da761d90258a/root/var/svc/log/oxide-cockroachdb:default.log.0
10532023-10-29T04:34:04.690Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6ee4877d-c073-4eb9-a615-8c2db66ca336/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10542023-10-29T04:34:05.743Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6ee4877d-c073-4eb9-a615-8c2db66ca336/root/var/svc/log/oxide-cockroachdb:default.log
10552023-10-29T04:34:05.781Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6ee4877d-c073-4eb9-a615-8c2db66ca336/root/var/svc/log/oxide-cockroachdb:default.log.0 (5367 bytes)
10562023-10-29T04:34:07.270Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6ee4877d-c073-4eb9-a615-8c2db66ca336/root/var/svc/log/oxide-cockroachdb:default.log.0
10572023-10-29T04:34:07.300Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2cab9542-84ef-41eb-a2ed-9dd23b40cb6c/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10582023-10-29T04:34:08.351Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2cab9542-84ef-41eb-a2ed-9dd23b40cb6c/root/var/svc/log/oxide-crucible-agent:default.log
10592023-10-29T04:34:08.377Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2cab9542-84ef-41eb-a2ed-9dd23b40cb6c/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10602023-10-29T04:34:09.448Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2cab9542-84ef-41eb-a2ed-9dd23b40cb6c/root/var/svc/log/oxide-crucible-agent:default.log.0
10612023-10-29T04:34:09.474Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2cab9542-84ef-41eb-a2ed-9dd23b40cb6c/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10622023-10-29T04:34:10.532Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2cab9542-84ef-41eb-a2ed-9dd23b40cb6c/root/var/svc/log/oxide-crucible-downstairs:default.log
10632023-10-29T04:34:10.567Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2cab9542-84ef-41eb-a2ed-9dd23b40cb6c/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10642023-10-29T04:34:11.662Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2cab9542-84ef-41eb-a2ed-9dd23b40cb6c/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10652023-10-29T04:34:11.687Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_b75b4e9d-9aca-438e-a112-6167816cf070/root/var/svc/log/oxide-internal_dns:default.log (12517 bytes)
10662023-10-29T04:34:13.206Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_b75b4e9d-9aca-438e-a112-6167816cf070/root/var/svc/log/oxide-internal_dns:default.log
10672023-10-29T04:34:13.232Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_b75b4e9d-9aca-438e-a112-6167816cf070/root/var/svc/log/oxide-internal_dns:default.log.0 (102882 bytes)
10682023-10-29T04:34:13.998Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_b75b4e9d-9aca-438e-a112-6167816cf070/root/var/svc/log/oxide-internal_dns:default.log.0
10692023-10-29T04:34:14.023Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_035f4922-dd30-4996-b58f-fc0d4bc0669c/root/var/svc/log/oxide-crucible-agent:default.log (11145 bytes)
10702023-10-29T04:34:15.119Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_035f4922-dd30-4996-b58f-fc0d4bc0669c/root/var/svc/log/oxide-crucible-agent:default.log
10712023-10-29T04:34:15.148Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_035f4922-dd30-4996-b58f-fc0d4bc0669c/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10722023-10-29T04:34:16.229Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_035f4922-dd30-4996-b58f-fc0d4bc0669c/root/var/svc/log/oxide-crucible-agent:default.log.0
10732023-10-29T04:34:16.258Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_035f4922-dd30-4996-b58f-fc0d4bc0669c/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10742023-10-29T04:34:17.306Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_035f4922-dd30-4996-b58f-fc0d4bc0669c/root/var/svc/log/oxide-crucible-downstairs:default.log
10752023-10-29T04:34:17.340Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_035f4922-dd30-4996-b58f-fc0d4bc0669c/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10762023-10-29T04:34:18.820Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_035f4922-dd30-4996-b58f-fc0d4bc0669c/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10772023-10-29T04:34:18.844Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_035f4922-dd30-4996-b58f-fc0d4bc0669c/root/var/svc/log/oxide-crucible-downstairs:downstairs-7245c2ed-cd1a-4bd6-9bd8-fbfd87379f75.log (3038 bytes)
10782023-10-29T04:34:19.926Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_035f4922-dd30-4996-b58f-fc0d4bc0669c/root/var/svc/log/oxide-crucible-downstairs:downstairs-7245c2ed-cd1a-4bd6-9bd8-fbfd87379f75.log
10792023-10-29T04:34:19.950Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0842f08e-2c29-425e-a3c9-11ac3823c2b1/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10802023-10-29T04:34:20.998Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0842f08e-2c29-425e-a3c9-11ac3823c2b1/root/var/svc/log/oxide-crucible-agent:default.log
10812023-10-29T04:34:21.023Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0842f08e-2c29-425e-a3c9-11ac3823c2b1/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10822023-10-29T04:34:22.099Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0842f08e-2c29-425e-a3c9-11ac3823c2b1/root/var/svc/log/oxide-crucible-agent:default.log.0
10832023-10-29T04:34:22.125Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0842f08e-2c29-425e-a3c9-11ac3823c2b1/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10842023-10-29T04:34:23.175Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0842f08e-2c29-425e-a3c9-11ac3823c2b1/root/var/svc/log/oxide-crucible-downstairs:default.log
10852023-10-29T04:34:23.200Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0842f08e-2c29-425e-a3c9-11ac3823c2b1/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10862023-10-29T04:34:24.281Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0842f08e-2c29-425e-a3c9-11ac3823c2b1/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10872023-10-29T04:34:24.778Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ab9e30b-a55e-4017-a723-1ca18e313140/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10882023-10-29T04:34:25.432Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ab9e30b-a55e-4017-a723-1ca18e313140/root/var/svc/log/oxide-crucible-agent:default.log
10892023-10-29T04:34:25.460Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ab9e30b-a55e-4017-a723-1ca18e313140/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10902023-10-29T04:34:40.065Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ab9e30b-a55e-4017-a723-1ca18e313140/root/var/svc/log/oxide-crucible-agent:default.log.0
10912023-10-29T04:34:40.155Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ab9e30b-a55e-4017-a723-1ca18e313140/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10922023-10-29T04:34:41.307Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ab9e30b-a55e-4017-a723-1ca18e313140/root/var/svc/log/oxide-crucible-downstairs:default.log
10932023-10-29T04:34:41.400Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ab9e30b-a55e-4017-a723-1ca18e313140/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10942023-10-29T04:34:42.489Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ab9e30b-a55e-4017-a723-1ca18e313140/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10952023-10-29T04:34:42.524Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5f33cf33-b357-4e4e-813e-893adfcac21e/root/var/svc/log/oxide-crucible-agent:default.log (10680 bytes)
10962023-10-29T04:34:44.013Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5f33cf33-b357-4e4e-813e-893adfcac21e/root/var/svc/log/oxide-crucible-agent:default.log
10972023-10-29T04:34:44.076Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5f33cf33-b357-4e4e-813e-893adfcac21e/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10982023-10-29T04:34:45.155Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5f33cf33-b357-4e4e-813e-893adfcac21e/root/var/svc/log/oxide-crucible-agent:default.log.0
10992023-10-29T04:34:45.185Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5f33cf33-b357-4e4e-813e-893adfcac21e/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11002023-10-29T04:34:46.237Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5f33cf33-b357-4e4e-813e-893adfcac21e/root/var/svc/log/oxide-crucible-downstairs:default.log
11012023-10-29T04:34:46.261Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5f33cf33-b357-4e4e-813e-893adfcac21e/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11022023-10-29T04:34:47.334Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5f33cf33-b357-4e4e-813e-893adfcac21e/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11032023-10-29T04:34:47.359Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5f33cf33-b357-4e4e-813e-893adfcac21e/root/var/svc/log/oxide-crucible-downstairs:downstairs-7056f44f-b4f6-4d7b-aa7b-97a474d587f0.log (5515 bytes)
11042023-10-29T04:34:48.909Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5f33cf33-b357-4e4e-813e-893adfcac21e/root/var/svc/log/oxide-crucible-downstairs:downstairs-7056f44f-b4f6-4d7b-aa7b-97a474d587f0.log
11052023-10-29T04:34:48.933Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f73054eb-d635-465e-bc9d-6eea1f9ed416/root/var/svc/log/oxide-crucible-agent:default.log (21861 bytes)
11062023-10-29T04:34:50.086Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f73054eb-d635-465e-bc9d-6eea1f9ed416/root/var/svc/log/oxide-crucible-agent:default.log
11072023-10-29T04:34:50.111Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f73054eb-d635-465e-bc9d-6eea1f9ed416/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11082023-10-29T04:34:51.174Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f73054eb-d635-465e-bc9d-6eea1f9ed416/root/var/svc/log/oxide-crucible-agent:default.log.0
11092023-10-29T04:34:51.209Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f73054eb-d635-465e-bc9d-6eea1f9ed416/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11102023-10-29T04:34:52.269Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f73054eb-d635-465e-bc9d-6eea1f9ed416/root/var/svc/log/oxide-crucible-downstairs:default.log
11112023-10-29T04:34:52.294Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f73054eb-d635-465e-bc9d-6eea1f9ed416/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11122023-10-29T04:34:53.426Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f73054eb-d635-465e-bc9d-6eea1f9ed416/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11132023-10-29T04:34:53.450Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f73054eb-d635-465e-bc9d-6eea1f9ed416/root/var/svc/log/oxide-crucible-downstairs:downstairs-3bd12f50-76e7-4413-9753-908111a8de66.log (5516 bytes)
11142023-10-29T04:34:54.527Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f73054eb-d635-465e-bc9d-6eea1f9ed416/root/var/svc/log/oxide-crucible-downstairs:downstairs-3bd12f50-76e7-4413-9753-908111a8de66.log
11152023-10-29T04:34:54.551Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f73054eb-d635-465e-bc9d-6eea1f9ed416/root/var/svc/log/oxide-crucible-downstairs:downstairs-b0b22f3a-493e-4573-bc02-ba6c93fa4c8e.log (3039 bytes)
11162023-10-29T04:34:55.625Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f73054eb-d635-465e-bc9d-6eea1f9ed416/root/var/svc/log/oxide-crucible-downstairs:downstairs-b0b22f3a-493e-4573-bc02-ba6c93fa4c8e.log
11172023-10-29T04:34:55.671Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_10fd2f72-17de-4c79-8bed-7ac2954aaf20/root/var/svc/log/oxide-nexus:default.log (4143240 bytes)
11182023-10-29T04:34:59.116Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_10fd2f72-17de-4c79-8bed-7ac2954aaf20/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4143240 -> 4206158
11192023-10-29T04:35:00.197Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_10fd2f72-17de-4c79-8bed-7ac2954aaf20/root/var/svc/log/oxide-nexus:default.log
11202023-10-29T04:35:00.222Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_10fd2f72-17de-4c79-8bed-7ac2954aaf20/root/var/svc/log/oxide-nexus:default.log.0 (75 bytes)
11212023-10-29T04:35:01.297Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_10fd2f72-17de-4c79-8bed-7ac2954aaf20/root/var/svc/log/oxide-nexus:default.log.0
11222023-10-29T04:35:01.323Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_118a2e33-21e8-4be0-b76d-a90f6c122d9b/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11232023-10-29T04:35:02.380Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_118a2e33-21e8-4be0-b76d-a90f6c122d9b/root/var/svc/log/oxide-cockroachdb:default.log
11242023-10-29T04:35:02.407Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_118a2e33-21e8-4be0-b76d-a90f6c122d9b/root/var/svc/log/oxide-cockroachdb:default.log.0 (5344 bytes)
11252023-10-29T04:35:03.486Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_118a2e33-21e8-4be0-b76d-a90f6c122d9b/root/var/svc/log/oxide-cockroachdb:default.log.0
11262023-10-29T04:35:03.517Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_7f2a9dce-0680-45c2-b00c-8f2d56df189b/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11272023-10-29T04:35:04.567Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_7f2a9dce-0680-45c2-b00c-8f2d56df189b/root/var/svc/log/oxide-cockroachdb:default.log
11282023-10-29T04:35:04.597Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_7f2a9dce-0680-45c2-b00c-8f2d56df189b/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
11292023-10-29T04:35:05.679Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_7f2a9dce-0680-45c2-b00c-8f2d56df189b/root/var/svc/log/oxide-cockroachdb:default.log.0
11302023-10-29T04:35:05.710Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8353a570-20b3-41e3-961e-de9f5284d00b/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11312023-10-29T04:35:06.760Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8353a570-20b3-41e3-961e-de9f5284d00b/root/var/svc/log/oxide-crucible-agent:default.log
11322023-10-29T04:35:06.785Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8353a570-20b3-41e3-961e-de9f5284d00b/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11332023-10-29T04:35:07.886Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8353a570-20b3-41e3-961e-de9f5284d00b/root/var/svc/log/oxide-crucible-agent:default.log.0
11342023-10-29T04:35:07.911Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8353a570-20b3-41e3-961e-de9f5284d00b/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11352023-10-29T04:35:08.972Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8353a570-20b3-41e3-961e-de9f5284d00b/root/var/svc/log/oxide-crucible-downstairs:default.log
11362023-10-29T04:35:08.999Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8353a570-20b3-41e3-961e-de9f5284d00b/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11372023-10-29T04:35:10.101Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8353a570-20b3-41e3-961e-de9f5284d00b/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11382023-10-29T04:35:10.125Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_12d6c8b3-942d-41e1-84f1-d5e12acc497a/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11392023-10-29T04:35:11.174Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_12d6c8b3-942d-41e1-84f1-d5e12acc497a/root/var/svc/log/oxide-crucible-pantry:default.log
11402023-10-29T04:35:11.205Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_12d6c8b3-942d-41e1-84f1-d5e12acc497a/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
11412023-10-29T04:35:12.291Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_12d6c8b3-942d-41e1-84f1-d5e12acc497a/root/var/svc/log/oxide-crucible-pantry:default.log.0
11422023-10-29T04:35:12.321Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_571074b7-7a89-4534-9d49-5e667169957f/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11432023-10-29T04:35:13.789Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_571074b7-7a89-4534-9d49-5e667169957f/root/var/svc/log/oxide-crucible-pantry:default.log
11442023-10-29T04:35:13.818Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_571074b7-7a89-4534-9d49-5e667169957f/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
11452023-10-29T04:35:14.947Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_571074b7-7a89-4534-9d49-5e667169957f/root/var/svc/log/oxide-crucible-pantry:default.log.0
11462023-10-29T04:35:14.977Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_8d951f00-b6b7-4aee-b1de-244964f89041/root/var/svc/log/oxide-nexus:default.log (857418 bytes)
11472023-10-29T04:35:17.115Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_8d951f00-b6b7-4aee-b1de-244964f89041/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 857418 -> 920948
11482023-10-29T04:35:17.778Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_8d951f00-b6b7-4aee-b1de-244964f89041/root/var/svc/log/oxide-nexus:default.log
11492023-10-29T04:35:17.804Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_8d951f00-b6b7-4aee-b1de-244964f89041/root/var/svc/log/oxide-nexus:default.log.0 (75 bytes)
11502023-10-29T04:35:18.889Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_8d951f00-b6b7-4aee-b1de-244964f89041/root/var/svc/log/oxide-nexus:default.log.0
11512023-10-29T04:35:18.917Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_a1e4a9dc-7d5e-4019-949a-40a843508a02/root/var/svc/log/oxide-nexus:default.log (476990 bytes)
11522023-10-29T04:35:20.407Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_a1e4a9dc-7d5e-4019-949a-40a843508a02/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 476990 -> 541129
11532023-10-29T04:35:22.977Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_a1e4a9dc-7d5e-4019-949a-40a843508a02/root/var/svc/log/oxide-nexus:default.log
11542023-10-29T04:35:23.009Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_a1e4a9dc-7d5e-4019-949a-40a843508a02/root/var/svc/log/oxide-nexus:default.log.0 (75 bytes)
11552023-10-29T04:35:24.486Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_a1e4a9dc-7d5e-4019-949a-40a843508a02/root/var/svc/log/oxide-nexus:default.log.0
11562023-10-29T04:35:24.511Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_2431fc92-7b4b-456b-b64a-fc0f9fea098b/root/var/svc/log/oxide-external_dns:default.log (15447 bytes)
11572023-10-29T04:35:24.600Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_2431fc92-7b4b-456b-b64a-fc0f9fea098b/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15447 -> 19210
11582023-10-29T04:35:25.672Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_2431fc92-7b4b-456b-b64a-fc0f9fea098b/root/var/svc/log/oxide-external_dns:default.log
11592023-10-29T04:35:25.696Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_2431fc92-7b4b-456b-b64a-fc0f9fea098b/root/var/svc/log/oxide-external_dns:default.log.0 (120 bytes)
11602023-10-29T04:35:26.778Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_2431fc92-7b4b-456b-b64a-fc0f9fea098b/root/var/svc/log/oxide-external_dns:default.log.0
11612023-10-29T04:35:26.809Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_74ad4491-1a80-4b95-9262-2441d8b934fa/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11622023-10-29T04:35:27.862Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_74ad4491-1a80-4b95-9262-2441d8b934fa/root/var/svc/log/oxide-cockroachdb:default.log
11632023-10-29T04:35:27.890Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_74ad4491-1a80-4b95-9262-2441d8b934fa/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
11642023-10-29T04:35:28.978Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_74ad4491-1a80-4b95-9262-2441d8b934fa/root/var/svc/log/oxide-cockroachdb:default.log.0
11652023-10-29T04:35:29.004Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_a4b52c80-9c37-4eb8-ab55-4668ecfd9ab8/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11662023-10-29T04:35:30.050Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_a4b52c80-9c37-4eb8-ab55-4668ecfd9ab8/root/var/svc/log/oxide-crucible-pantry:default.log
11672023-10-29T04:35:30.075Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_a4b52c80-9c37-4eb8-ab55-4668ecfd9ab8/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2331 bytes)
11682023-10-29T04:35:31.146Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_a4b52c80-9c37-4eb8-ab55-4668ecfd9ab8/root/var/svc/log/oxide-crucible-pantry:default.log.0
11692023-10-29T04:35:31.171Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_80193322-65ae-4adc-95cb-e9e7c9150d16/root/var/svc/log/oxide-internal_dns:default.log (12520 bytes)
11702023-10-29T04:35:31.261Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_80193322-65ae-4adc-95cb-e9e7c9150d16/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 12520 -> 20031
11712023-10-29T04:35:32.333Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_80193322-65ae-4adc-95cb-e9e7c9150d16/root/var/svc/log/oxide-internal_dns:default.log
11722023-10-29T04:35:32.364Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_80193322-65ae-4adc-95cb-e9e7c9150d16/root/var/svc/log/oxide-internal_dns:default.log.0 (112171 bytes)
11732023-10-29T04:35:33.780Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_80193322-65ae-4adc-95cb-e9e7c9150d16/root/var/svc/log/oxide-internal_dns:default.log.0
11742023-10-29T04:35:33.804Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_543f6d87-3357-4965-9d0e-5ae60c1e0d35/root/var/svc/log/oxide-internal_dns:default.log (12519 bytes)
11752023-10-29T04:35:33.894Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_543f6d87-3357-4965-9d0e-5ae60c1e0d35/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 12519 -> 20030
11762023-10-29T04:35:35.420Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_543f6d87-3357-4965-9d0e-5ae60c1e0d35/root/var/svc/log/oxide-internal_dns:default.log
11772023-10-29T04:35:35.457Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_543f6d87-3357-4965-9d0e-5ae60c1e0d35/root/var/svc/log/oxide-internal_dns:default.log.0 (106027 bytes)
11782023-10-29T04:35:36.376Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_543f6d87-3357-4965-9d0e-5ae60c1e0d35/root/var/svc/log/oxide-internal_dns:default.log.0
11792023-10-29T04:35:36.408Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_6c04ab66-e7f6-4295-8890-a4b9c2ee9b0d/root/var/svc/log/oxide-oximeter:default.log (160252 bytes)
11802023-10-29T04:35:37.068Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_6c04ab66-e7f6-4295-8890-a4b9c2ee9b0d/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 160252 -> 233280
11812023-10-29T04:35:38.146Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_6c04ab66-e7f6-4295-8890-a4b9c2ee9b0d/root/var/svc/log/oxide-oximeter:default.log
11822023-10-29T04:35:38.180Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_6c04ab66-e7f6-4295-8890-a4b9c2ee9b0d/root/var/svc/log/oxide-oximeter:default.log.0 (75 bytes)
11832023-10-29T04:35:39.273Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_6c04ab66-e7f6-4295-8890-a4b9c2ee9b0d/root/var/svc/log/oxide-oximeter:default.log.0
11842023-10-29T04:35:39.298Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8/root/var/log/chrony/measurements.log (15481 bytes)
11852023-10-29T04:35:39.365Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8/root/var/log/chrony/measurements.log" changed size mid upload: 15481 -> 18495
11862023-10-29T04:35:40.440Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8/root/var/log/chrony/measurements.log
11872023-10-29T04:35:40.465Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8/root/var/log/chrony/statistics.log (9360 bytes)
11882023-10-29T04:35:40.512Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8/root/var/log/chrony/statistics.log" changed size mid upload: 9360 -> 11700
11892023-10-29T04:35:41.583Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8/root/var/log/chrony/statistics.log
11902023-10-29T04:35:41.609Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8/root/var/log/chrony/tracking.log (1854 bytes)
11912023-10-29T04:35:41.634Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8/root/var/log/chrony/tracking.log" changed size mid upload: 1854 -> 2118
11922023-10-29T04:35:42.705Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ecaa1434-f2e5-4812-9b96-0e17948f68c8/root/var/log/chrony/tracking.log