01HBSAX3AJC9GT92PVAP0Z6BV8: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.23",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HBSAZKWXAG6Q6HTH02KY8EAQ

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-03T01:31:09.751Zjob dependencies complete; ready to run (waiting for 2 h 12 m 26 s)
22023-10-03T04:28:52.012Zjob assigned to worker 01HBSWP7AHBYF1NSW49R7QGQ6V (queued for 2 h 57 m 42 s)
32023-10-03T04:28:52.231Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-10-03T04:28:52.286Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-10-03T04:28:52.327Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-10-03T04:28:52.365Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-03T04:28:52.405Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-10-03T04:28:52.448ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-03T04:28:52.488ZOxide Helios Version helios-2.0.22085 64-bit
102023-10-03T04:28:52.522ZConfiguring devices.
112023-10-03T04:28:52.555Z##################################################
122023-10-03T04:28:52.618Z#### ############# ##
132023-10-03T04:28:52.680Z### ### ############ ##
142023-10-03T04:28:52.735Z## ### # ## ### ## Oxide Computer Company ##
152023-10-03T04:28:52.773Z## ## ## ### # ### ##
162023-10-03T04:28:52.814Z## # ### #### #### This Station Under ##
172023-10-03T04:28:52.849Z### ### #### # ### Computer Control ##
182023-10-03T04:28:52.888Z#### #### ### ## ##
192023-10-03T04:28:52.923Z##################################################
202023-10-03T04:28:52.959ZHostname: sock
212023-10-03T04:28:53.001ZLoading smf(7) serice descriptions:
222023-10-03T04:28:53.421Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-57c3f40605e3c78b.gz
232023-10-03T04:28:55.560Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-57c3f40605e3c78b.gz
242023-10-03T04:28:55.586Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
252023-10-03T04:29:14.267Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-03T04:29:14.292Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-88fbefd8fad2952b.gz
272023-10-03T04:29:54.038Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-88fbefd8fad2952b.gz
282023-10-03T04:29:54.065Zdownloading input: /input/ci-tools/work/caboose-util.gz
292023-10-03T04:29:55.559Zdownloaded input: /input/ci-tools/work/caboose-util.gz
302023-10-03T04:29:55.588Zdownloading input: /input/ci-tools/work/tufaceous.gz
312023-10-03T04:30:01.089Zdownloaded input: /input/ci-tools/work/tufaceous.gz
322023-10-03T04:30:01.119Zdownloading input: /input/package/work/version.txt
332023-10-03T04:30:01.146Zdownloaded input: /input/package/work/version.txt
342023-10-03T04:30:01.174Zdownloading input: /input/package/work/package.tar.gz
352023-10-03T04:30:17.697Zdownloaded input: /input/package/work/package.tar.gz
362023-10-03T04:30:17.722Zdownloading input: /input/package/work/global-zone-packages.tar.gz
372023-10-03T04:30:42.133Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
382023-10-03T04:30:42.162Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
392023-10-03T04:30:49.203Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-03T04:30:49.228Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
412023-10-03T04:31:20.549Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
422023-10-03T04:31:20.574Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
432023-10-03T04:31:54.104Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-03T04:31:54.129Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
452023-10-03T04:32:16.214Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-03T04:32:16.240Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
472023-10-03T04:32:20.393Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-03T04:32:20.418Zdownloading input: /input/package/work/zones/crucible.tar.gz
492023-10-03T04:32:28.332Zdownloaded input: /input/package/work/zones/crucible.tar.gz
502023-10-03T04:32:28.383Zdownloading input: /input/package/work/zones/external-dns.tar.gz
512023-10-03T04:32:32.631Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
522023-10-03T04:32:32.655Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
532023-10-03T04:32:37.042Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
542023-10-03T04:32:37.107Zdownloading input: /input/package/work/zones/ntp.tar.gz
552023-10-03T04:32:37.132Zdownloaded input: /input/package/work/zones/ntp.tar.gz
562023-10-03T04:32:37.157Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
572023-10-03T04:32:40.426Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-03T04:32:40.451Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
592023-10-03T04:32:43.750Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-03T04:32:43.774Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
612023-10-03T04:32:56.942Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
622023-10-03T04:32:56.968Zdownloading input: /input/package/work/zones/overlay.tar.gz
632023-10-03T04:32:57.017Zdownloaded input: /input/package/work/zones/overlay.tar.gz
642023-10-03T04:32:57.043Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
652023-10-03T04:33:00.861Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
662023-10-03T04:33:00.887Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
672023-10-03T04:33:13.707Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
682023-10-03T04:33:13.758Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
692023-10-03T04:33:52.747Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
702023-10-03T04:33:52.772Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
712023-10-03T04:34:17.929Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
722023-10-03T04:34:17.953Zstarting task 0: "setup"
732023-10-03T04:34:17.986Z++ uname -s
742023-10-03T04:34:18.049Z+ kern=SunOS
752023-10-03T04:34:18.074Z+ case "$kern" in
762023-10-03T04:34:18.099Z+ groupadd -g 12345 build
772023-10-03T04:34:18.123Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
782023-10-03T04:34:19.995Z+ zfs create -o mountpoint=/work rpool/work
792023-10-03T04:34:20.035Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
802023-10-03T04:34:20.060Z+ home_fs=zfs
812023-10-03T04:34:20.085Z+ [[ zfs == autofs ]]
822023-10-03T04:34:20.110Z+ mkdir -p /home/build
832023-10-03T04:34:20.134Z+ chown build:build /home/build /work
842023-10-03T04:34:21.049Z+ chmod 0700 /home/build /work
852023-10-03T04:34:21.088Zprocess exited: duration 3066 ms, exit code 0
 
862023-10-03T04:34:21.138Zstarting task 1: "authentication"
872023-10-03T04:34:21.207Zprocess exited: duration 44 ms, exit code 0
 
882023-10-03T04:34:21.270Zstarting task 2: "build"
892023-10-03T04:34:21.301Z+ trap _exit_trap EXIT
902023-10-03T04:34:21.326Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
912023-10-03T04:34:21.497Zsecpolicy_fs_linkdir: 0x55 = 0x55
922023-10-03T04:34:21.522Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
932023-10-03T04:34:21.547Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
942023-10-03T04:34:21.572Zsecpolicy_fs_linkdir+3: 0 = 0xe5
952023-10-03T04:34:21.600Zsecpolicy_fs_linkdir+4: 0 = 0xb8
962023-10-03T04:34:21.625Zsecpolicy_fs_linkdir+5: 0 = 0x1
972023-10-03T04:34:21.651Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
982023-10-03T04:34:21.685Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
992023-10-03T04:34:21.717Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1002023-10-03T04:34:21.749Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1012023-10-03T04:34:21.782Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1022023-10-03T04:34:21.807Z+ [[ -d /opt/oxide ]]
1032023-10-03T04:34:21.836Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1042023-10-03T04:34:21.860Z+ pfexec mkdir /opt/oxide
1052023-10-03T04:34:21.885Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1062023-10-03T04:34:21.910Z+ [[ -d /opt/oxide-underneath ]]
1072023-10-03T04:34:21.937Z+ cd /opt/oxide-underneath
1082023-10-03T04:34:21.963Z+ pfexec tar ceEp@/f - .
1092023-10-03T04:34:21.988Z+ cd /opt/oxide
1102023-10-03T04:34:22.013Z+ pfexec tar xveEp@/f -
1112023-10-03T04:34:22.049Zx ., 0 bytes, 0 tape blocks
1122023-10-03T04:34:22.073Zx ./opte, 0 bytes, 0 tape blocks
1132023-10-03T04:34:22.098Zx ./opte/bin, 0 bytes, 0 tape blocks
1142023-10-03T04:34:22.123Zx ./opte/bin/opteadm, 46195888 bytes, 90227 tape blocks
1152023-10-03T04:34:22.152Z+ pfexec rm -rf /opt/oxide-underneath
1162023-10-03T04:34:22.178Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1172023-10-03T04:34:22.203Z+ pfexec mkdir /opt/oxide/work
1182023-10-03T04:34:22.227Z+ pfexec chown build:build /opt/oxide/work
1192023-10-03T04:34:22.252Z+ cd /opt/oxide/work
1202023-10-03T04:34:22.276Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1212023-10-03T04:34:23.232ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1222023-10-03T04:34:23.257Zx out/maghemite.tar, 74957312 bytes, 146401 tape blocks
1232023-10-03T04:34:23.281Zx out/omicron-sled-agent.tar, 112508416 bytes, 219743 tape blocks
1242023-10-03T04:34:23.306Zx out/target/test, 49 bytes, 1 tape blocks
1252023-10-03T04:34:23.330Zx out/npuzone/npuzone, 36941864 bytes, 72153 tape blocks
1262023-10-03T04:34:23.383Zx package-manifest.toml, 21196 bytes, 42 tape blocks
1272023-10-03T04:34:23.408Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1282023-10-03T04:34:23.432Zx target/release/omicron-package, 37915488 bytes, 74054 tape blocks
1292023-10-03T04:34:23.458Zx tools/create_virtual_hardware.sh, 2810 bytes, 6 tape blocks
1302023-10-03T04:34:23.483Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1312023-10-03T04:34:23.507Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1322023-10-03T04:34:23.535Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1332023-10-03T04:34:23.570Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1342023-10-03T04:34:23.603Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1352023-10-03T04:34:23.632Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1362023-10-03T04:34:23.664Z
1372023-10-03T04:34:23.691Zreal 1.589225931
1382023-10-03T04:34:23.723Zuser 1.434302112
1392023-10-03T04:34:23.759Zsys 0.274536758
1402023-10-03T04:34:23.789Ztrap 0.000075370
1412023-10-03T04:34:23.868Ztflt 0.000000000
1422023-10-03T04:34:23.892Zdflt 0.000000000
1432023-10-03T04:34:23.917Zkflt 0.000000000
1442023-10-03T04:34:23.942Zlock 0.000000000
1452023-10-03T04:34:23.966Zslp 1.432745679
1462023-10-03T04:34:23.991Zlat 0.034819834
1472023-10-03T04:34:24.016Zstop 0.000121061
1482023-10-03T04:34:24.041Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1492023-10-03T04:34:24.168Z+ mkdir tests
1502023-10-03T04:34:24.193Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1512023-10-03T04:34:24.217Z+ ptime -m gunzip
1522023-10-03T04:34:24.244Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-57c3f40605e3c78b
1532023-10-03T04:34:24.316Z
1542023-10-03T04:34:24.342Zreal 0.139370297
1552023-10-03T04:34:24.367Zuser 0.127017675
1562023-10-03T04:34:24.391Zsys 0.012120667
1572023-10-03T04:34:24.415Ztrap 0.000011250
1582023-10-03T04:34:24.440Ztflt 0.000000000
1592023-10-03T04:34:24.464Zdflt 0.000000000
1602023-10-03T04:34:24.489Zkflt 0.000000000
1612023-10-03T04:34:24.923Zlock 0.000000000
1622023-10-03T04:34:24.948Zslp 0.000018220
1632023-10-03T04:34:24.981Zlat 0.000015739
1642023-10-03T04:34:25.006Zstop 0.000123021
1652023-10-03T04:34:25.031Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-57c3f40605e3c78b
1662023-10-03T04:34:25.056Z+ chmod a+x tests/bootstrap-57c3f40605e3c78b
1672023-10-03T04:34:25.081Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1682023-10-03T04:34:25.112Z+ ptime -m gunzip
1692023-10-03T04:34:25.137Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1702023-10-03T04:34:26.495Z
1712023-10-03T04:34:26.520Zreal 2.166204821
1722023-10-03T04:34:26.545Zuser 2.041703932
1732023-10-03T04:34:26.569Zsys 0.124107711
1742023-10-03T04:34:26.593Ztrap 0.000153924
1752023-10-03T04:34:26.618Ztflt 0.000000000
1762023-10-03T04:34:26.642Zdflt 0.000000000
1772023-10-03T04:34:26.667Zkflt 0.000000000
1782023-10-03T04:34:26.691Zlock 0.000000000
1792023-10-03T04:34:26.716Zslp 0.000022610
1802023-10-03T04:34:26.740Zlat 0.000022470
1812023-10-03T04:34:26.765Zstop 0.000130282
1822023-10-03T04:34:26.789Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1832023-10-03T04:34:26.813Z+ chmod a+x tests/bootstrap
1842023-10-03T04:34:26.838Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1852023-10-03T04:34:26.863Z+ ptime -m gunzip
1862023-10-03T04:34:26.887Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-88fbefd8fad2952b
1872023-10-03T04:34:28.988Z
1882023-10-03T04:34:29.062Zreal 2.480641171
1892023-10-03T04:34:29.086Zuser 2.329246461
1902023-10-03T04:34:29.111Zsys 0.151005491
1912023-10-03T04:34:29.135Ztrap 0.000171574
1922023-10-03T04:34:29.160Ztflt 0.000000000
1932023-10-03T04:34:29.184Zdflt 0.000000000
1942023-10-03T04:34:29.209Zkflt 0.000000000
1952023-10-03T04:34:29.233Zlock 0.000000000
1962023-10-03T04:34:29.258Zslp 0.000020530
1972023-10-03T04:34:29.282Zlat 0.000019270
1982023-10-03T04:34:29.306Zstop 0.000110641
1992023-10-03T04:34:29.330Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-88fbefd8fad2952b
2002023-10-03T04:34:29.355Z+ chmod a+x tests/end_to_end_tests-88fbefd8fad2952b
2012023-10-03T04:34:29.379Z+ bmat address ls
2022023-10-03T04:34:29.404ZNAME CIDR FIRST LAST COUNT
2032023-10-03T04:34:29.428Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2042023-10-03T04:34:29.453Z++ bmat address ls -f extra -Ho first,last,count
2052023-10-03T04:34:29.478Z+ set -- 10.151.2.164 10.151.2.227 64
2062023-10-03T04:34:29.502Z+ EXTRA_IP_START=10.151.2.164
2072023-10-03T04:34:29.526Z+ EXTRA_IP_END=10.151.2.227
2082023-10-03T04:34:29.551Z+ EXTRA_IP_COUNT=64
2092023-10-03T04:34:29.576Z+ (( EXTRA_IP_COUNT >= 32 ))
2102023-10-03T04:34:29.600Z+ EXTRA_IP_BASE=10.151.2
2112023-10-03T04:34:29.624Z+ EXTRA_IP_FOCTET=164
2122023-10-03T04:34:29.648Z+ EXTRA_IP_LOCTET=227
2132023-10-03T04:34:29.673Z+ SERVICE_IP_POOL_START=10.151.2.164
2142023-10-03T04:34:29.697Z+ SERVICE_IP_POOL_END=10.151.2.173
2152023-10-03T04:34:29.721Z+ DNS_IP1=10.151.2.164
2162023-10-03T04:34:29.745Z+ DNS_IP2=10.151.2.165
2172023-10-03T04:34:29.770Z+ UPLINK_IP=10.151.2.174
2182023-10-03T04:34:29.795Z+ IPPOOL_START=10.151.2.175
2192023-10-03T04:34:29.819Z+ IPPOOL_END=10.151.2.227
2202023-10-03T04:34:29.844Z++ ipadm show-addr -po type,addr
2212023-10-03T04:34:29.870Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2222023-10-03T04:34:29.895Z+ GATEWAY_IP=10.151.2.100
2232023-10-03T04:34:29.920Z+ [[ -n 10.151.2.100 ]]
2242023-10-03T04:34:29.944Z+ ping -s 10.151.2.100 56 1
2252023-10-03T04:34:29.968ZPING 10.151.2.100: 56 data bytes
2262023-10-03T04:34:29.993Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2272023-10-03T04:34:30.017Z
2282023-10-03T04:34:30.042Z----10.151.2.100 PING Statistics----
2292023-10-03T04:34:30.089Z1 packets transmitted, 1 packets received, 0% packet loss
2302023-10-03T04:34:30.114Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2312023-10-03T04:34:30.553Z++ arp -an
2322023-10-03T04:34:30.579Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2332023-10-03T04:34:30.605Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2342023-10-03T04:34:30.631Z+ routeadm -e ipv4-forwarding -u
2352023-10-03T04:34:30.658Z+ PXA_START=10.151.2.164
2362023-10-03T04:34:30.682Z+ PXA_END=10.151.2.227
2372023-10-03T04:34:30.707Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2382023-10-03T04:34:30.732Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2392023-10-03T04:34:30.757Z+ ZPOOL_VDEV_DIR=/scratch
2402023-10-03T04:34:30.781Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2412023-10-03T04:34:30.806Z+++ dirname ./tools/create_virtual_hardware.sh
2422023-10-03T04:34:30.831Z++ cd ./tools
2432023-10-03T04:34:30.856Z++ pwd
2442023-10-03T04:34:30.881Z+ SOURCE_DIR=/opt/oxide/work/tools
2452023-10-03T04:34:30.913Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2462023-10-03T04:34:30.938Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2472023-10-03T04:34:30.962Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2482023-10-03T04:34:30.987Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2492023-10-03T04:34:31.017Z++ dladm show-phys -p -o LINK
2502023-10-03T04:34:31.041Z++ head -1
2512023-10-03T04:34:31.066Z+ PHYSICAL_LINK=igb0
2522023-10-03T04:34:31.139Z+ echo 'Using igb0 as physical link'
2532023-10-03T04:34:31.171ZUsing igb0 as physical link
2542023-10-03T04:34:31.201Z+ ensure_run_as_root
2552023-10-03T04:34:31.225Z++ id -u
2562023-10-03T04:34:31.249Z+ [[ 0 -ne 0 ]]
2572023-10-03T04:34:31.292Z+ ensure_zpools
2582023-10-03T04:34:31.316Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2592023-10-03T04:34:31.342Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2602023-10-03T04:34:31.368Z+ readarray -t ZPOOLS
2612023-10-03T04:34:31.395Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2622023-10-03T04:34:31.419Z++ sed 's/[ ",]//g'
2632023-10-03T04:34:31.444Z+ for ZPOOL in "${ZPOOLS[@]}"
2642023-10-03T04:34:31.468Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2652023-10-03T04:34:31.492ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2662023-10-03T04:34:31.516Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2672023-10-03T04:34:31.540Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2682023-10-03T04:34:31.566Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2692023-10-03T04:34:31.596Z0+0 records in
2702023-10-03T04:34:31.620Z0+0 records out
2712023-10-03T04:34:31.644ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2722023-10-03T04:34:31.668Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2732023-10-03T04:34:31.692Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2742023-10-03T04:34:31.718ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2752023-10-03T04:34:31.743Z+ set +x
2762023-10-03T04:34:31.767ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2772023-10-03T04:34:31.794Z++ zpool list -o name
2782023-10-03T04:34:31.818ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2792023-10-03T04:34:31.842Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2802023-10-03T04:34:31.866ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2812023-10-03T04:34:31.890Z+ [[ -z '' ]]
2822023-10-03T04:34:31.914ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2832023-10-03T04:34:31.938Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-10-03T04:34:31.962ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2852023-10-03T04:34:31.987Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2862023-10-03T04:34:32.012ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2872023-10-03T04:34:32.044Z+ set +x
2882023-10-03T04:34:32.069ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2892023-10-03T04:34:32.094Z+ for ZPOOL in "${ZPOOLS[@]}"
2902023-10-03T04:34:32.119ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2912023-10-03T04:34:32.189Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2922023-10-03T04:34:32.213ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2932023-10-03T04:34:32.237Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2942023-10-03T04:34:32.260ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2952023-10-03T04:34:32.284Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2962023-10-03T04:34:32.308ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2972023-10-03T04:34:32.332Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2982023-10-03T04:34:32.356ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2992023-10-03T04:34:32.379Z0+0 records in
3002023-10-03T04:34:32.405ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3012023-10-03T04:34:32.435Z0+0 records out
3022023-10-03T04:34:32.463ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3032023-10-03T04:34:32.489Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3042023-10-03T04:34:32.513ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3052023-10-03T04:34:32.537Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3062023-10-03T04:34:32.562ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3072023-10-03T04:34:32.586Z+ set +x
3082023-10-03T04:34:32.611ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3092023-10-03T04:34:32.635Z++ zpool list -o name
3102023-10-03T04:34:32.659ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3112023-10-03T04:34:32.684Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3122023-10-03T04:34:32.708ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3132023-10-03T04:34:32.732Z+ [[ -z '' ]]
3142023-10-03T04:34:32.757ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3152023-10-03T04:34:32.781Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3162023-10-03T04:34:32.807ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3172023-10-03T04:34:32.831Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3182023-10-03T04:34:32.864ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3192023-10-03T04:34:32.898Z+ set +x
3202023-10-03T04:34:32.923ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3212023-10-03T04:34:32.948Z+ for ZPOOL in "${ZPOOLS[@]}"
3222023-10-03T04:34:32.972ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3232023-10-03T04:34:32.996Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3242023-10-03T04:34:33.021ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3252023-10-03T04:34:33.045Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3262023-10-03T04:34:33.069ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3272023-10-03T04:34:33.093Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3282023-10-03T04:34:33.117ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3292023-10-03T04:34:33.141Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3302023-10-03T04:34:33.165ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3312023-10-03T04:34:33.213Z0+0 records in
3322023-10-03T04:34:33.237ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3332023-10-03T04:34:33.261Z0+0 records out
3342023-10-03T04:34:33.286ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3352023-10-03T04:34:33.319Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3362023-10-03T04:34:33.356ZSimnet net0/sc0_0 exists
3372023-10-03T04:34:33.412Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3382023-10-03T04:34:33.436ZSimnet net1/sc1_0 exists
3392023-10-03T04:34:33.461Z+ set +x
3402023-10-03T04:34:33.486ZVnic sc0_1 exists
3412023-10-03T04:34:33.510Z++ zpool list -o name
3422023-10-03T04:34:33.538Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3432023-10-03T04:34:33.563Z+ [[ -z '' ]]
3442023-10-03T04:34:33.588Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3452023-10-03T04:34:33.612Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3462023-10-03T04:34:33.637Z+ set +x
3472023-10-03T04:34:33.662Z+ for ZPOOL in "${ZPOOLS[@]}"
3482023-10-03T04:34:33.687Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3492023-10-03T04:34:33.711Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-10-03T04:34:33.746Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3512023-10-03T04:34:33.777Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3522023-10-03T04:34:33.801Z0+0 records in
3532023-10-03T04:34:33.825Z0+0 records out
3542023-10-03T04:34:33.849Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3552023-10-03T04:34:33.872Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3562023-10-03T04:34:33.897Z+ set +x
3572023-10-03T04:34:33.921Z++ zpool list -o name
3582023-10-03T04:34:33.945Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3592023-10-03T04:34:33.969Z+ [[ -z '' ]]
3602023-10-03T04:34:33.993Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3612023-10-03T04:34:34.017Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3622023-10-03T04:34:34.041Z+ set +x
3632023-10-03T04:34:34.065Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-10-03T04:34:34.090Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3652023-10-03T04:34:34.113Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-10-03T04:34:34.137Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3672023-10-03T04:34:34.172Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3682023-10-03T04:34:34.200Z0+0 records in
3692023-10-03T04:34:34.237Z0+0 records out
3702023-10-03T04:34:34.261Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3712023-10-03T04:34:34.285Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3722023-10-03T04:34:34.310Z+ set +x
3732023-10-03T04:34:34.334Z++ zpool list -o name
3742023-10-03T04:34:34.357Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3752023-10-03T04:34:34.381Z+ [[ -z '' ]]
3762023-10-03T04:34:34.405Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3772023-10-03T04:34:34.429Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3782023-10-03T04:34:34.453Z+ set +x
3792023-10-03T04:34:34.477Z+ for ZPOOL in "${ZPOOLS[@]}"
3802023-10-03T04:34:34.501Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3812023-10-03T04:34:34.525Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3822023-10-03T04:34:34.703Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3832023-10-03T04:34:35.172Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3842023-10-03T04:34:35.206Z0+0 records in
3852023-10-03T04:34:35.230Z0+0 records out
3862023-10-03T04:34:35.267Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3872023-10-03T04:34:35.291Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3882023-10-03T04:34:35.319Z+ set +x
3892023-10-03T04:34:35.343Z++ zpool list -o name
3902023-10-03T04:34:35.367Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3912023-10-03T04:34:35.392Z+ [[ -z '' ]]
3922023-10-03T04:34:35.415Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3932023-10-03T04:34:35.439Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3942023-10-03T04:34:35.464Z+ set +x
3952023-10-03T04:34:35.489Z+ for ZPOOL in "${ZPOOLS[@]}"
3962023-10-03T04:34:35.514Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3972023-10-03T04:34:35.538Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-10-03T04:34:35.563Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3992023-10-03T04:34:35.593Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4002023-10-03T04:34:35.629Z0+0 records in
4012023-10-03T04:34:36.064Z0+0 records out
4022023-10-03T04:34:36.097Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4032023-10-03T04:34:36.127Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4042023-10-03T04:34:36.151Z+ set +x
4052023-10-03T04:34:36.175Z++ zpool list -o name
4062023-10-03T04:34:36.200Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4072023-10-03T04:34:36.224Z+ [[ -z '' ]]
4082023-10-03T04:34:36.248Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4092023-10-03T04:34:36.292Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4102023-10-03T04:34:36.317Z+ set +x
4112023-10-03T04:34:36.341Z+ for ZPOOL in "${ZPOOLS[@]}"
4122023-10-03T04:34:36.365Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4132023-10-03T04:34:36.389Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4142023-10-03T04:34:36.414Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4152023-10-03T04:34:36.439Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4162023-10-03T04:34:36.465Z0+0 records in
4172023-10-03T04:34:36.494Z0+0 records out
4182023-10-03T04:34:36.524Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4192023-10-03T04:34:36.557Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4202023-10-03T04:34:36.592Z+ set +x
4212023-10-03T04:34:36.619Z++ zpool list -o name
4222023-10-03T04:34:36.646Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4232023-10-03T04:34:36.670Z+ [[ -z '' ]]
4242023-10-03T04:34:36.693Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4252023-10-03T04:34:36.721Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4262023-10-03T04:34:36.746Z+ set +x
4272023-10-03T04:34:36.770Z+ for ZPOOL in "${ZPOOLS[@]}"
4282023-10-03T04:34:36.794Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4292023-10-03T04:34:36.818Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4302023-10-03T04:34:36.842Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4312023-10-03T04:34:36.866Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4322023-10-03T04:34:36.889Z0+0 records in
4332023-10-03T04:34:36.913Z0+0 records out
4342023-10-03T04:34:36.937Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
4352023-10-03T04:34:36.962Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4362023-10-03T04:34:36.986Z+ set +x
4372023-10-03T04:34:37.016Z++ zpool list -o name
4382023-10-03T04:34:37.045Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4392023-10-03T04:34:37.069Z+ [[ -z '' ]]
4402023-10-03T04:34:37.095Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4412023-10-03T04:34:37.119Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4422023-10-03T04:34:37.144Z+ set +x
4432023-10-03T04:34:37.168Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4442023-10-03T04:34:37.191Z+ readarray -t ZPOOLS
4452023-10-03T04:34:37.215Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4462023-10-03T04:34:37.239Z++ sed 's/[ ",]//g'
4472023-10-03T04:34:37.263Z+ for ZPOOL in "${ZPOOLS[@]}"
4482023-10-03T04:34:37.287Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4492023-10-03T04:34:37.321Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4502023-10-03T04:34:37.345Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4512023-10-03T04:34:37.370Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4522023-10-03T04:34:37.393Z0+0 records in
4532023-10-03T04:34:37.418Z0+0 records out
4542023-10-03T04:34:37.443Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
4552023-10-03T04:34:37.473Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4562023-10-03T04:34:37.498Z+ set +x
4572023-10-03T04:34:37.523Z++ zpool list -o name
4582023-10-03T04:34:37.547Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4592023-10-03T04:34:37.571Z+ [[ -z '' ]]
4602023-10-03T04:34:37.595Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4612023-10-03T04:34:37.621Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4622023-10-03T04:34:37.644Z+ set +x
4632023-10-03T04:34:37.669Z+ for ZPOOL in "${ZPOOLS[@]}"
4642023-10-03T04:34:37.692Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4652023-10-03T04:34:37.716Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-10-03T04:34:37.740Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-10-03T04:34:37.764Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-10-03T04:34:37.788Z0+0 records in
4692023-10-03T04:34:37.812Z0+0 records out
4702023-10-03T04:34:37.836Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4712023-10-03T04:34:37.861Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-10-03T04:34:37.886Z+ set +x
4732023-10-03T04:34:37.910Z++ zpool list -o name
4742023-10-03T04:34:37.936Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4752023-10-03T04:34:37.962Z+ [[ -z '' ]]
4762023-10-03T04:34:37.989Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4772023-10-03T04:34:38.015Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4782023-10-03T04:34:38.039Z+ set +x
4792023-10-03T04:34:38.064Z+ ensure_simulated_links igb0
4802023-10-03T04:34:38.088Z+ local PHYSICAL_LINK=igb0
4812023-10-03T04:34:38.112Z+ INDICES=("0" "1")
4822023-10-03T04:34:38.137Z+ for I in "${INDICES[@]}"
4832023-10-03T04:34:38.161Z++ get_simnet_name_if_exists net0
4842023-10-03T04:34:38.190Z++ dladm show-simnet -p -o LINK net0
4852023-10-03T04:34:38.221Z++ echo ''
4862023-10-03T04:34:38.250Z+ [[ -z '' ]]
4872023-10-03T04:34:38.277Z+ dladm create-simnet -t net0
4882023-10-03T04:34:38.301Z+ dladm create-simnet -t sc0_0
4892023-10-03T04:34:38.345Z+ dladm modify-simnet -t -p net0 sc0_0
4902023-10-03T04:34:38.370Z+ dladm set-linkprop -p mtu=1600 sc0_0
4912023-10-03T04:34:38.393Z+ success 'Simnet net0/sc0_0 exists'
4922023-10-03T04:34:38.417Z+ set +x
4932023-10-03T04:34:38.441Z+ for I in "${INDICES[@]}"
4942023-10-03T04:34:38.468Z++ get_simnet_name_if_exists net1
4952023-10-03T04:34:38.493Z++ dladm show-simnet -p -o LINK net1
4962023-10-03T04:34:38.517Z++ echo ''
4972023-10-03T04:34:38.566Z+ [[ -z '' ]]
4982023-10-03T04:34:38.590Z+ dladm create-simnet -t net1
4992023-10-03T04:34:38.617Z+ dladm create-simnet -t sc1_0
5002023-10-03T04:34:38.651Z+ dladm modify-simnet -t -p net1 sc1_0
5012023-10-03T04:34:38.682Z+ dladm set-linkprop -p mtu=1600 sc1_0
5022023-10-03T04:34:38.712Z+ success 'Simnet net1/sc1_0 exists'
5032023-10-03T04:34:38.738Z+ set +x
5042023-10-03T04:34:38.762Z++ get_vnic_name_if_exists sc0_1
5052023-10-03T04:34:38.786Z++ dladm show-vnic -p -o LINK sc0_1
5062023-10-03T04:34:38.810Z++ echo ''
5072023-10-03T04:34:38.835Z+ [[ -z '' ]]
5082023-10-03T04:34:38.861Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5092023-10-03T04:34:38.885Z+ success 'Vnic sc0_1 exists'
5102023-10-03T04:34:38.908Z+ set +x
5112023-10-03T04:34:38.932Z+ warn_if_no_proxy_arp
5122023-10-03T04:34:38.956Z+ [[ -v PXA_START ]]
5132023-10-03T04:34:38.979Z+ [[ -v PXA_END ]]
5142023-10-03T04:34:39.003Z+ ensure_softnpu_zone
5152023-10-03T04:34:39.028Z+ zoneadm list
5162023-10-03T04:34:39.053Z+ grep -q sidecar_softnpu
5172023-10-03T04:34:39.082Z+ [[ -f out/npuzone/npuzone ]]
5182023-10-03T04:34:39.525Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0
5192023-10-03T04:34:39.966Zdownloading softnpu ...done
5202023-10-03T04:34:42.646Zdownloading asic_program.so ...done
5212023-10-03T04:34:49.655Zdownloading scadm ...done
5222023-10-03T04:34:49.679Zcreating port tfportrear0_0 for sc0_0
5232023-10-03T04:34:49.711Zcreating port tfportqsfp0_0 for sc0_1
5242023-10-03T04:34:49.739Zsoftnpu zone setup
5252023-10-03T04:34:49.764Zconfigure zone
5262023-10-03T04:34:49.789Zinstall zone
5272023-10-03T04:34:50.738Zboot zone
5282023-10-03T04:34:53.844Z[sidecar_softnpu] svcs milestone/network
5292023-10-03T04:34:53.873ZSTATE STIME FMRI
5302023-10-03T04:34:53.900Zonline 4:34:52 svc:/milestone/network:default
5312023-10-03T04:34:53.925Z[sidecar_softnpu] /softnpu/npu start
5322023-10-03T04:34:53.950Z
5332023-10-03T04:34:53.977Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5342023-10-03T04:34:54.001Z+ GATEWAY_IP=10.151.2.100
5352023-10-03T04:34:54.028Z+ echo 'Using 10.151.2.100 as gateway ip'
5362023-10-03T04:34:54.053ZUsing 10.151.2.100 as gateway ip
5372023-10-03T04:34:54.078Z+ [[ ! -v GATEWAY_MAC ]]
5382023-10-03T04:34:54.102Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5392023-10-03T04:34:54.128Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5402023-10-03T04:34:54.152Z+++ echo 18:c0:4d:0d:a0:2a
5412023-10-03T04:34:54.177Z+ COUNT=1
5422023-10-03T04:34:54.201Z+ [[ 1 -ne 1 ]]
5432023-10-03T04:34:54.226Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5442023-10-03T04:34:54.251ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5452023-10-03T04:34:54.276Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5462023-10-03T04:34:54.300Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5472023-10-03T04:34:54.326Z+ PXA_MAC=a8:e1:de:01:70:1d
5482023-10-03T04:34:54.351Z+ [[ -v PXA_START ]]
5492023-10-03T04:34:54.377Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5502023-10-03T04:34:54.402Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5512023-10-03T04:34:54.426Z+ z_scadm dump-state
5522023-10-03T04:34:54.450Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5532023-10-03T04:34:54.475Zlocal v6:
5542023-10-03T04:34:54.499Zlocal v4:
5552023-10-03T04:34:54.537Zrouter v6:
5562023-10-03T04:34:54.561Zrouter v4:
5572023-10-03T04:34:54.586Zresolver v4:
5582023-10-03T04:34:54.610Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5592023-10-03T04:34:54.635Zresolver v6:
5602023-10-03T04:34:54.662Znat v4:
5612023-10-03T04:34:54.687Znat_v6:
5622023-10-03T04:34:54.714Zport_mac:
5632023-10-03T04:34:54.738Zproxy arp:
5642023-10-03T04:34:54.763Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5652023-10-03T04:34:54.787Z+ success 'softnpu zone exists'
5662023-10-03T04:34:54.812Z+ set +x
5672023-10-03T04:34:54.837Zsoftnpu zone exists
5682023-10-03T04:34:54.874Z
5692023-10-03T04:34:54.899Zreal 24.624038740
5702023-10-03T04:34:54.924Zuser 0.715806565
5712023-10-03T04:34:54.949Zsys 1.594273236
5722023-10-03T04:34:54.974Ztrap 0.000062919
5732023-10-03T04:34:54.998Ztflt 0.000188187
5742023-10-03T04:34:55.024Zdflt 0.000832243
5752023-10-03T04:34:55.049Zkflt 0.000000000
5762023-10-03T04:34:55.074Zlock 19:06.537747659
5772023-10-03T04:34:55.098Zslp 1:13.552905820
5782023-10-03T04:34:55.123Zlat 0.052038549
5792023-10-03T04:34:55.147Zstop 0.370003813
5802023-10-03T04:34:55.173Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5812023-10-03T04:34:55.198Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5822023-10-03T04:34:55.222Z+ SILO_NAME=recovery
5832023-10-03T04:34:55.247Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5842023-10-03T04:34:55.272Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5852023-10-03T04:34:55.296Z+ sed -i~ '
5862023-10-03T04:34:55.320Z /^external_dns_ips/c\
5872023-10-03T04:34:55.344Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5882023-10-03T04:34:55.369Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5892023-10-03T04:34:55.393Z /^first/c\
5902023-10-03T04:34:55.417Zfirst = "10.151.2.164"
5912023-10-03T04:34:55.444Z /^last/c\
5922023-10-03T04:34:55.469Zlast = "10.151.2.173"
5932023-10-03T04:34:55.494Z }
5942023-10-03T04:34:55.522Z /^\[rack_network_config/,/^$/ {
5952023-10-03T04:34:55.547Z /^infra_ip_first/c\
5962023-10-03T04:34:55.982Zinfra_ip_first = "10.151.2.174"
5972023-10-03T04:34:56.008Z /^infra_ip_last/c\
5982023-10-03T04:34:56.050Zinfra_ip_last = "10.151.2.174"
5992023-10-03T04:34:56.087Z }
6002023-10-03T04:34:56.124Z /^\[\[rack_network_config.uplinks/,/^$/ {
6012023-10-03T04:34:56.149Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6022023-10-03T04:34:56.194Z /^gateway_ip/c\
6032023-10-03T04:34:56.219Z+++ pkg/config-rss.toml Tue Oct 3 04:34:54 2023
6042023-10-03T04:34:56.245Zgateway_ip = "10.151.2.100"
6052023-10-03T04:34:56.269Z@@ -41,7 +41,7 @@
6062023-10-03T04:34:56.297Z /^uplink_cidr/c\
6072023-10-03T04:34:56.322Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6082023-10-03T04:34:56.348Zuplink_cidr = "10.151.2.174/32"
6092023-10-03T04:34:56.373Z # must be contained in one of the "internal services" IP Pool ranges listed
6102023-10-03T04:34:56.400Z }
6112023-10-03T04:34:56.424Z # below.
6122023-10-03T04:34:56.448Z' pkg/config-rss.toml
6132023-10-03T04:34:56.509Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6142023-10-03T04:34:56.533Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6152023-10-03T04:34:56.557Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6162023-10-03T04:34:56.584Z+ true
6172023-10-03T04:34:56.609Z
6182023-10-03T04:34:56.633Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6192023-10-03T04:34:56.657Z # Initial TLS certificates for the external API
6202023-10-03T04:34:56.682Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6212023-10-03T04:34:56.707Z #
6222023-10-03T04:34:56.731Z+ TLS_NAME=recovery.sys.oxide.test
6232023-10-03T04:34:56.756Z@@ -77,8 +77,8 @@
6242023-10-03T04:34:56.781Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6252023-10-03T04:34:56.805Z #
6262023-10-03T04:34:56.831Z # For more on this and what to put here, see docs/how-to-run.adoc.
6272023-10-03T04:34:56.855Z [[internal_services_ip_pool_ranges]]
6282023-10-03T04:34:56.880Z-first = "192.168.1.20"
6292023-10-03T04:34:56.904Z-last = "192.168.1.29"
6302023-10-03T04:34:56.957Z+first = "10.151.2.164"
6312023-10-03T04:34:56.983Z+last = "10.151.2.173"
6322023-10-03T04:34:57.008Z
6332023-10-03T04:34:57.033Z # TODO - this configuration is subject to change going forward. Ultimately these
6342023-10-03T04:34:57.059Z # parameters should be provided to the control plane via wicket, but we need to
6352023-10-03T04:34:57.085Z@@ -91,13 +91,13 @@
6362023-10-03T04:34:57.112Z # A range of IP addresses used by Boundary Services on the external network. In
6372023-10-03T04:34:57.137Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6382023-10-03T04:34:57.162Z # With softnpu, only one address is used.
6392023-10-03T04:34:57.187Z-infra_ip_first = "192.168.1.30"
6402023-10-03T04:34:57.212Z-infra_ip_last = "192.168.1.30"
6412023-10-03T04:34:57.236Z+infra_ip_first = "10.151.2.174"
6422023-10-03T04:34:57.261Z+infra_ip_last = "10.151.2.174"
6432023-10-03T04:34:57.285Z
6442023-10-03T04:34:57.310Z # You can configure multiple uplinks by repeating the following stanza
6452023-10-03T04:34:57.334Z [[rack_network_config.uplinks]]
6462023-10-03T04:34:57.360Z # The gateway IP for the rack's external network
6472023-10-03T04:34:57.386Z-gateway_ip = "192.168.1.199"
6482023-10-03T04:34:57.410Z+gateway_ip = "10.151.2.100"
6492023-10-03T04:34:57.435Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6502023-10-03T04:34:57.459Z uplink_port = "qsfp0"
6512023-10-03T04:34:57.484Z uplink_port_speed = "40G"
6522023-10-03T04:34:57.508Z@@ -104,7 +104,7 @@
6532023-10-03T04:34:57.533Z uplink_port_fec="none"
6542023-10-03T04:34:57.557Z # For softnpu, an address within the "infra" block above that will be used for
6552023-10-03T04:34:57.581Z # the softnpu uplink port. You can just pick the first address in that pool.
6562023-10-03T04:34:57.606Z-uplink_cidr = "192.168.1.30/32"
6572023-10-03T04:34:58.051Z+uplink_cidr = "10.151.2.174/32"
6582023-10-03T04:34:58.076Z # Switch to use for the uplink. For single-rack deployments this can be
6592023-10-03T04:34:58.101Z # "switch0" (upper slot) or "switch1" (lower slot). For single-node softnpu
6602023-10-03T04:34:58.126Z # and dendrite stub environments, use "switch0"
6612023-10-03T04:34:58.153Za pkg/config-rss.toml 7K
6622023-10-03T04:34:58.178Z..+..........+..+...+....+......+........+..........+.....+.+...+..+...............+.+.....+...+....+..+......+.......+..+...+.+......+......+........+......+.+..+.+......+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.......+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+...+...+......+.....+...+......+..........+.........+...+..................+..+..................+......+.........+....+...+...+..............+...+....+......+.....+...+......................+...+..+.......+.....+..........+......+...........+....+..+.............+......+...+.....+.+...............+.....+......+...+.......+.....+.+..............+....+.............................+.+..+....+......+.........+...+.....+.+.....+.+........+.......+...............+...+...................................+.+..+...................+...+......+....................+....+.....+...+...+.......+.................+.+...............+..+..........+.........+......+..+.+...............+.................+....+......+.....+.......+......+..+.......+.....+...+...+......................+..+................+.........+.....+............+....+.................+.........+....+......+...+......+..+...+.......+..............+....+.........+............+..............+....+..+.+..................+.....+.......+..+.............+.........+...........+...+.......+........+.......+...+............+..+.+.........+........+...............+...+.+.....+.........+.........+...+....+......+.....+.......+......+...............+......+.....+...+...+..........+..+.......+...+...+...+..............+...............+.......+...+...............+..+...+..........+....................+......+.+...+..+....+............+...+..+...+.......+..+...+...+....+..+.+........+............+....+...............+........+.+..+....+.....+...+.......+.....+.+...+...........+.........+....+...+...+..+...+.......+...+...............+.....+.+........+.............+..............+...+.+...........................+............+.....+...+.......+......+......+......+..+...+...+....+.................+..........+...........+.........+.+..............+...+.+......+...+.....................+..............+.+..............+.......+...+..+.+..+..................+...+.......+....................+..........+............+.....+............+...+......+......+.+.....+....+.....+...............+.............+.....+.+..+.......+...+.....+.......+............+........+.......+......+..+.......+......+.....+...+.......+...+..+.........+....+......+...+...+.........+........................+..+............+..........+...........+......+.+............+.....+...+............+.+.........+...........+.......+...+.....+....+...+..+......+......+....+..+.............+......+...+.......................+...+.......+...+....................+.+...........+......................+..+....+..+....+.....+...+..........+.....+.............+..+.............+.....+.........+...+.......+.....+.............+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6632023-10-03T04:34:58.204Z.....+.+...+..+...+......+.......+...+..+...+.......+..+...+....+.....+....+............+...+.........+...+...+.....+......+...+.+.....+.......+.....+.......+........+.......+..+...+.+..+............+...+.......+............+........+.......+.....+...............+.+.........+...+...+........+.......+...+......+...+..+....+......+......+.....+....+...+..+.+........+................+...+........+....+...+..+.+........+.....................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+.......+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*............+......+......+.....+.+...+.....+......+.........+......................+.....+...+......+..........+...+..+..........+...+.....+.......+.....+...+......+............+...+............+.............+.....+.......+...+..+.......+...........+...................+...+..............+.+...+.........+.....+...+.............+...........+......+....+........+.......+........+......+..........+...........+.............+..+.+...........................+.....+.+...........+....+.........+...+..+......+.+...+............+...............+.....+......+...+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6642023-10-03T04:34:58.230Z-----
6652023-10-03T04:34:58.255Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6662023-10-03T04:34:58.279Za pkg/initial-tls-cert.pem 2K
6672023-10-03T04:34:58.307Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6682023-10-03T04:34:58.332Za pkg/initial-tls-key.pem 4K
6692023-10-03T04:34:58.357Z+ rmdir pkg
6702023-10-03T04:34:58.381Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6712023-10-03T04:34:58.406Z+ pfexec mkdir -p /usr/oxide
6722023-10-03T04:34:58.431Z+ pfexec rm -f /usr/oxide/tcpproxy
6732023-10-03T04:34:58.456Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6742023-10-03T04:34:58.481Z+ pfexec chmod +x /usr/oxide/tcpproxy
6752023-10-03T04:34:58.505Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6762023-10-03T04:34:58.557Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6772023-10-03T04:34:58.582Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6782023-10-03T04:34:58.606Z+ OMICRON_NO_UNINSTALL=1
6792023-10-03T04:34:58.634Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6802023-10-03T04:34:58.658ZOct 03 04:34:56.506 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "switch": "softnpu"})
6812023-10-03T04:34:59.092ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6822023-10-03T04:34:59.118ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6832023-10-03T04:34:59.146ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6842023-10-03T04:34:59.174ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6852023-10-03T04:34:59.199ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6862023-10-03T04:34:59.224ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6872023-10-03T04:34:59.248ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/maghemite.tar
6882023-10-03T04:34:59.273ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6892023-10-03T04:34:59.298ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6902023-10-03T04:34:59.322ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6912023-10-03T04:34:59.346ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6922023-10-03T04:34:59.372ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6932023-10-03T04:34:59.397ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6942023-10-03T04:34:59.422ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6952023-10-03T04:34:59.447ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6962023-10-03T04:34:59.472ZOct 03 04:34:56.508 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6972023-10-03T04:34:59.497ZOct 03 04:34:56.755 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
6982023-10-03T04:34:59.521ZOct 03 04:34:56.795 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
6992023-10-03T04:34:59.546ZOct 03 04:34:56.855 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7002023-10-03T04:34:59.570Z
7012023-10-03T04:34:59.595Zreal 0.485681211
7022023-10-03T04:34:59.619Zuser 0.096134427
7032023-10-03T04:34:59.644Zsys 1.959063125
7042023-10-03T04:34:59.668Ztrap 0.000000360
7052023-10-03T04:34:59.693Ztflt 0.000196562
7062023-10-03T04:35:00.131Zdflt 0.000269550
7072023-10-03T04:35:00.157Zkflt 0.000000000
7082023-10-03T04:35:00.183Zlock 43.736945944
7092023-10-03T04:35:00.208Zslp 0.683446668
7102023-10-03T04:35:00.233Zlat 0.002615198
7112023-10-03T04:35:00.258Zstop 0.200320401
7122023-10-03T04:35:00.283Z+ retry=0
7132023-10-03T04:35:00.309Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7142023-10-03T04:35:00.334Z+ [[ 0 -gt 30 ]]
7152023-10-03T04:35:00.359Z+ sleep 1
7162023-10-03T04:35:00.384Z+ retry=1
7172023-10-03T04:35:00.409Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7182023-10-03T04:35:00.433Z+ [[ 1 -gt 30 ]]
7192023-10-03T04:35:00.458Z+ sleep 1
7202023-10-03T04:35:00.483Z+ retry=2
7212023-10-03T04:35:00.509Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-10-03T04:35:18.594Z+ [[ 2 -gt 30 ]]
7232023-10-03T04:35:18.620Z+ sleep 1
7242023-10-03T04:35:19.598Z+ retry=3
7252023-10-03T04:35:19.623Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-10-03T04:35:19.647Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7272023-10-03T04:35:19.671Zlocal v6:
7282023-10-03T04:35:19.696Zfe80::802d:d4ff:fe02:5f2a
7292023-10-03T04:35:19.720Zlocal v4:
7302023-10-03T04:35:19.744Zrouter v6:
7312023-10-03T04:35:19.768Zrouter v4:
7322023-10-03T04:35:19.793Zresolver v4:
7332023-10-03T04:35:19.818Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7342023-10-03T04:35:19.842Zresolver v6:
7352023-10-03T04:35:19.866Zfe80::802d:d4ff:fe02:5f2a -> 82:2d:d4:02:5f:2a
7362023-10-03T04:35:19.890Zfe80::3cbf:31ff:fefb:542d -> 3e:bf:31:fb:54:2d
7372023-10-03T04:35:19.915Znat v4:
7382023-10-03T04:35:19.939Znat_v6:
7392023-10-03T04:35:19.964Zport_mac:
7402023-10-03T04:35:19.987Z1: a8:40:25:54:cd:8b
7412023-10-03T04:35:20.012Z1000: a8:40:25:54:cd:8a
7422023-10-03T04:35:20.036Zproxy arp:
7432023-10-03T04:35:20.061Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7442023-10-03T04:35:20.085Z+ retry=0
7452023-10-03T04:35:20.109Z+++ zoneadm list -n
7462023-10-03T04:35:20.134Z+++ grep oxz_ntp
7472023-10-03T04:35:20.158Z++ pfexec svcs -z -Hostate oxide/ntp
7482023-10-03T04:35:20.182Zsvcs: zone '-Hostate': entity not found
7492023-10-03T04:35:20.206Z++ true
7502023-10-03T04:35:20.230Z+ [[ '' != online ]]
7512023-10-03T04:35:20.254Z+ [[ 0 -gt 60 ]]
7522023-10-03T04:35:20.279Z+ sleep 1
7532023-10-03T04:35:20.659Z+ retry=1
7542023-10-03T04:35:20.685Z+++ zoneadm list -n
7552023-10-03T04:35:20.709Z+++ grep oxz_ntp
7562023-10-03T04:35:20.733Z++ pfexec svcs -z -Hostate oxide/ntp
7572023-10-03T04:35:20.757Zsvcs: zone '-Hostate': entity not found
7582023-10-03T04:35:20.781Z++ true
7592023-10-03T04:35:20.805Z+ [[ '' != online ]]
7602023-10-03T04:35:20.829Z+ [[ 1 -gt 60 ]]
7612023-10-03T04:35:20.854Z+ sleep 1
7622023-10-03T04:35:21.764Z+ retry=2
7632023-10-03T04:35:21.788Z+++ zoneadm list -n
7642023-10-03T04:35:21.814Z+++ grep oxz_ntp
7652023-10-03T04:35:21.838Z++ pfexec svcs -z -Hostate oxide/ntp
7662023-10-03T04:35:21.863Zsvcs: zone '-Hostate': entity not found
7672023-10-03T04:35:21.887Z++ true
7682023-10-03T04:35:21.911Z+ [[ '' != online ]]
7692023-10-03T04:35:21.935Z+ [[ 2 -gt 60 ]]
7702023-10-03T04:35:21.960Z+ sleep 1
7712023-10-03T04:35:22.701Z+ retry=3
7722023-10-03T04:35:22.725Z+++ zoneadm list -n
7732023-10-03T04:35:22.749Z+++ grep oxz_ntp
7742023-10-03T04:35:22.773Z++ pfexec svcs -z -Hostate oxide/ntp
7752023-10-03T04:35:22.799Zsvcs: zone '-Hostate': entity not found
7762023-10-03T04:35:22.824Z++ true
7772023-10-03T04:35:22.848Z+ [[ '' != online ]]
7782023-10-03T04:35:22.873Z+ [[ 3 -gt 60 ]]
7792023-10-03T04:35:22.905Z+ sleep 1
7802023-10-03T04:35:23.723Z+ retry=4
7812023-10-03T04:35:23.747Z+++ zoneadm list -n
7822023-10-03T04:35:23.771Z+++ grep oxz_ntp
7832023-10-03T04:35:23.795Z++ pfexec svcs -z -Hostate oxide/ntp
7842023-10-03T04:35:23.820Zsvcs: zone '-Hostate': entity not found
7852023-10-03T04:35:23.845Z++ true
7862023-10-03T04:35:23.869Z+ [[ '' != online ]]
7872023-10-03T04:35:23.893Z+ [[ 4 -gt 60 ]]
7882023-10-03T04:35:23.917Z+ sleep 1
7892023-10-03T04:35:24.744Z+ retry=5
7902023-10-03T04:35:24.769Z+++ zoneadm list -n
7912023-10-03T04:35:24.793Z+++ grep oxz_ntp
7922023-10-03T04:35:24.817Z++ pfexec svcs -z -Hostate oxide/ntp
7932023-10-03T04:35:24.841Zsvcs: zone '-Hostate': entity not found
7942023-10-03T04:35:24.864Z++ true
7952023-10-03T04:35:24.888Z+ [[ '' != online ]]
7962023-10-03T04:35:24.912Z+ [[ 5 -gt 60 ]]
7972023-10-03T04:35:24.936Z+ sleep 1
7982023-10-03T04:35:25.768Z+ retry=6
7992023-10-03T04:35:25.792Z+++ zoneadm list -n
8002023-10-03T04:35:25.815Z+++ grep oxz_ntp
8012023-10-03T04:35:25.839Z++ pfexec svcs -z -Hostate oxide/ntp
8022023-10-03T04:35:25.863Zsvcs: zone '-Hostate': entity not found
8032023-10-03T04:35:25.887Z++ true
8042023-10-03T04:35:25.911Z+ [[ '' != online ]]
8052023-10-03T04:35:25.935Z+ [[ 6 -gt 60 ]]
8062023-10-03T04:35:25.959Z+ sleep 1
8072023-10-03T04:35:26.789Z+ retry=7
8082023-10-03T04:35:26.813Z+++ zoneadm list -n
8092023-10-03T04:35:26.838Z+++ grep oxz_ntp
8102023-10-03T04:35:26.862Z++ pfexec svcs -z -Hostate oxide/ntp
8112023-10-03T04:35:26.887Zsvcs: zone '-Hostate': entity not found
8122023-10-03T04:35:26.911Z++ true
8132023-10-03T04:35:26.935Z+ [[ '' != online ]]
8142023-10-03T04:35:26.959Z+ [[ 7 -gt 60 ]]
8152023-10-03T04:35:26.984Z+ sleep 1
8162023-10-03T04:35:27.810Z+ retry=8
8172023-10-03T04:35:27.834Z+++ zoneadm list -n
8182023-10-03T04:35:27.858Z+++ grep oxz_ntp
8192023-10-03T04:35:27.882Z++ pfexec svcs -z -Hostate oxide/ntp
8202023-10-03T04:35:27.906Zsvcs: zone '-Hostate': entity not found
8212023-10-03T04:35:27.930Z++ true
8222023-10-03T04:35:27.955Z+ [[ '' != online ]]
8232023-10-03T04:35:27.979Z+ [[ 8 -gt 60 ]]
8242023-10-03T04:35:28.003Z+ sleep 1
8252023-10-03T04:35:28.831Z+ retry=9
8262023-10-03T04:35:28.855Z+++ zoneadm list -n
8272023-10-03T04:35:28.880Z+++ grep oxz_ntp
8282023-10-03T04:35:28.904Z++ pfexec svcs -z -Hostate oxide/ntp
8292023-10-03T04:35:28.928Zsvcs: zone '-Hostate': entity not found
8302023-10-03T04:35:28.952Z++ true
8312023-10-03T04:35:28.976Z+ [[ '' != online ]]
8322023-10-03T04:35:29.000Z+ [[ 9 -gt 60 ]]
8332023-10-03T04:35:29.024Z+ sleep 1
8342023-10-03T04:35:29.854Z+ retry=10
8352023-10-03T04:35:29.878Z+++ zoneadm list -n
8362023-10-03T04:35:29.902Z+++ grep oxz_ntp
8372023-10-03T04:35:29.926Z++ pfexec svcs -z -Hostate oxide/ntp
8382023-10-03T04:35:29.950Zsvcs: zone '-Hostate': entity not found
8392023-10-03T04:35:29.974Z++ true
8402023-10-03T04:35:29.998Z+ [[ '' != online ]]
8412023-10-03T04:35:30.022Z+ [[ 10 -gt 60 ]]
8422023-10-03T04:35:30.047Z+ sleep 1
8432023-10-03T04:35:30.874Z+ retry=11
8442023-10-03T04:35:30.898Z+++ zoneadm list -n
8452023-10-03T04:35:30.922Z+++ grep oxz_ntp
8462023-10-03T04:35:30.947Z++ pfexec svcs -z -Hostate oxide/ntp
8472023-10-03T04:35:30.972Zsvcs: zone '-Hostate': entity not found
8482023-10-03T04:35:30.996Z++ true
8492023-10-03T04:35:31.021Z+ [[ '' != online ]]
8502023-10-03T04:35:31.045Z+ [[ 11 -gt 60 ]]
8512023-10-03T04:35:31.069Z+ sleep 1
8522023-10-03T04:35:31.896Z+ retry=12
8532023-10-03T04:35:31.920Z+++ zoneadm list -n
8542023-10-03T04:35:31.945Z+++ grep oxz_ntp
8552023-10-03T04:35:31.969Z++ pfexec svcs -z -Hostate oxide/ntp
8562023-10-03T04:35:31.994Zsvcs: zone '-Hostate': entity not found
8572023-10-03T04:35:32.018Z++ true
8582023-10-03T04:35:32.042Z+ [[ '' != online ]]
8592023-10-03T04:35:32.067Z+ [[ 12 -gt 60 ]]
8602023-10-03T04:35:32.092Z+ sleep 1
8612023-10-03T04:35:32.916Z+ retry=13
8622023-10-03T04:35:32.940Z+++ zoneadm list -n
8632023-10-03T04:35:32.964Z+++ grep oxz_ntp
8642023-10-03T04:35:32.989Z++ pfexec svcs -z -Hostate oxide/ntp
8652023-10-03T04:35:33.012Zsvcs: zone '-Hostate': entity not found
8662023-10-03T04:35:33.036Z++ true
8672023-10-03T04:35:33.060Z+ [[ '' != online ]]
8682023-10-03T04:35:33.084Z+ [[ 13 -gt 60 ]]
8692023-10-03T04:35:33.108Z+ sleep 1
8702023-10-03T04:35:33.937Z+ retry=14
8712023-10-03T04:35:33.961Z+++ zoneadm list -n
8722023-10-03T04:35:33.985Z+++ grep oxz_ntp
8732023-10-03T04:35:34.009Z++ pfexec svcs -z -Hostate oxide/ntp
8742023-10-03T04:35:34.034Zsvcs: zone '-Hostate': entity not found
8752023-10-03T04:35:34.058Z++ true
8762023-10-03T04:35:34.083Z+ [[ '' != online ]]
8772023-10-03T04:35:34.107Z+ [[ 14 -gt 60 ]]
8782023-10-03T04:35:34.132Z+ sleep 1
8792023-10-03T04:35:34.959Z+ retry=15
8802023-10-03T04:35:34.984Z+++ zoneadm list -n
8812023-10-03T04:35:35.009Z+++ grep oxz_ntp
8822023-10-03T04:35:35.035Z++ pfexec svcs -z oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572 -Hostate oxide/ntp
8832023-10-03T04:35:35.059Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8842023-10-03T04:35:35.084Z++ true
8852023-10-03T04:35:35.108Z+ [[ '' != online ]]
8862023-10-03T04:35:35.133Z+ [[ 15 -gt 60 ]]
8872023-10-03T04:35:35.157Z+ sleep 1
8882023-10-03T04:35:35.989Z+ retry=16
8892023-10-03T04:35:36.014Z+++ zoneadm list -n
8902023-10-03T04:35:36.038Z+++ grep oxz_ntp
8912023-10-03T04:35:36.062Z++ pfexec svcs -z oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572 -Hostate oxide/ntp
8922023-10-03T04:35:36.089Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8932023-10-03T04:35:36.116Z++ true
8942023-10-03T04:35:36.140Z+ [[ '' != online ]]
8952023-10-03T04:35:36.164Z+ [[ 16 -gt 60 ]]
8962023-10-03T04:35:36.188Z+ sleep 1
8972023-10-03T04:35:37.019Z+ retry=17
8982023-10-03T04:35:37.043Z+++ zoneadm list -n
8992023-10-03T04:35:37.067Z+++ grep oxz_ntp
9002023-10-03T04:35:37.091Z++ pfexec svcs -z oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572 -Hostate oxide/ntp
9012023-10-03T04:35:37.116Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9022023-10-03T04:35:37.140Z++ true
9032023-10-03T04:35:37.164Z+ [[ '' != online ]]
9042023-10-03T04:35:37.188Z+ [[ 17 -gt 60 ]]
9052023-10-03T04:35:37.214Z+ sleep 1
9062023-10-03T04:35:38.048Z+ retry=18
9072023-10-03T04:35:38.072Z+++ zoneadm list -n
9082023-10-03T04:35:38.096Z+++ grep oxz_ntp
9092023-10-03T04:35:38.120Z++ pfexec svcs -z oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572 -Hostate oxide/ntp
9102023-10-03T04:35:38.144Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9112023-10-03T04:35:38.168Z++ true
9122023-10-03T04:35:38.192Z+ [[ '' != online ]]
9132023-10-03T04:35:38.216Z+ [[ 18 -gt 60 ]]
9142023-10-03T04:35:38.240Z+ sleep 1
9152023-10-03T04:35:39.079Z+ retry=19
9162023-10-03T04:35:39.104Z+++ zoneadm list -n
9172023-10-03T04:35:39.128Z+++ grep oxz_ntp
9182023-10-03T04:35:39.152Z++ pfexec svcs -z oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572 -Hostate oxide/ntp
9192023-10-03T04:35:39.176Z+ [[ disabled != online ]]
9202023-10-03T04:35:39.201Z+ [[ 19 -gt 60 ]]
9212023-10-03T04:35:39.225Z+ sleep 1
9222023-10-03T04:35:40.101Z+ retry=20
9232023-10-03T04:35:40.126Z+++ zoneadm list -n
9242023-10-03T04:35:40.150Z+++ grep oxz_ntp
9252023-10-03T04:35:40.175Z++ pfexec svcs -z oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572 -Hostate oxide/ntp
9262023-10-03T04:35:40.199Z+ [[ disabled != online ]]
9272023-10-03T04:35:40.224Z+ [[ 20 -gt 60 ]]
9282023-10-03T04:35:40.248Z+ sleep 1
9292023-10-03T04:35:41.123Z+ retry=21
9302023-10-03T04:35:41.147Z+++ zoneadm list -n
9312023-10-03T04:35:41.171Z+++ grep oxz_ntp
9322023-10-03T04:35:41.195Z++ pfexec svcs -z oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572 -Hostate oxide/ntp
9332023-10-03T04:35:41.219Z+ [[ disabled != online ]]
9342023-10-03T04:35:41.243Z+ [[ 21 -gt 60 ]]
9352023-10-03T04:35:41.267Z+ sleep 1
9362023-10-03T04:35:42.144Z+ retry=22
9372023-10-03T04:35:42.170Z+++ zoneadm list -n
9382023-10-03T04:35:42.194Z+++ grep oxz_ntp
9392023-10-03T04:35:42.219Z++ pfexec svcs -z oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572 -Hostate oxide/ntp
9402023-10-03T04:35:42.244Z+ [[ disabled != online ]]
9412023-10-03T04:35:42.268Z+ [[ 22 -gt 60 ]]
9422023-10-03T04:35:42.292Z+ sleep 1
9432023-10-03T04:35:43.166Z+ retry=23
9442023-10-03T04:35:43.190Z+++ zoneadm list -n
9452023-10-03T04:35:43.214Z+++ grep oxz_ntp
9462023-10-03T04:35:43.238Z++ pfexec svcs -z oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572 -Hostate oxide/ntp
9472023-10-03T04:35:43.262Z+ [[ disabled != online ]]
9482023-10-03T04:35:43.286Z+ [[ 23 -gt 60 ]]
9492023-10-03T04:35:43.310Z+ sleep 1
9502023-10-03T04:35:44.189Z+ retry=24
9512023-10-03T04:35:44.214Z+++ zoneadm list -n
9522023-10-03T04:35:44.238Z+++ grep oxz_ntp
9532023-10-03T04:35:44.264Z++ pfexec svcs -z oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572 -Hostate oxide/ntp
9542023-10-03T04:35:44.322Z+ [[ disabled != online ]]
9552023-10-03T04:35:44.346Z+ [[ 24 -gt 60 ]]
9562023-10-03T04:35:44.370Z+ sleep 1
9572023-10-03T04:35:45.211Z+ retry=25
9582023-10-03T04:35:45.236Z+++ zoneadm list -n
9592023-10-03T04:35:45.260Z+++ grep oxz_ntp
9602023-10-03T04:35:45.285Z++ pfexec svcs -z oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572 -Hostate oxide/ntp
9612023-10-03T04:35:45.310Z+ [[ disabled != online ]]
9622023-10-03T04:35:45.334Z+ [[ 25 -gt 60 ]]
9632023-10-03T04:35:45.359Z+ sleep 1
 
9642023-10-03T04:35:45.519Zsock console login: Oct 3 04:35:44 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9652023-10-03T04:35:46.233Z+ retry=26
9662023-10-03T04:35:46.257Z+++ zoneadm list -n
9672023-10-03T04:35:46.281Z+++ grep oxz_ntp
9682023-10-03T04:35:46.305Z++ pfexec svcs -z oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572 -Hostate oxide/ntp
9692023-10-03T04:35:46.329Z+ [[ online != online ]]
9702023-10-03T04:35:46.353Z+ echo 'Waited for chrony: 26s'
9712023-10-03T04:35:46.377Z+ export RUST_BACKTRACE=1
9722023-10-03T04:35:46.401Z+ RUST_BACKTRACE=1
9732023-10-03T04:35:46.425ZWaited for chrony: 26s
9742023-10-03T04:35:46.448Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9752023-10-03T04:35:46.473Z+ ./tests/bootstrap
9762023-10-03T04:35:46.496ZUsing external DNS server at 10.151.2.164:53
9772023-10-03T04:35:46.521Z2023-10-03 04:35:46.253665872 UTC: attempting to log into API
9782023-10-03T04:36:01.324Z2023-10-03 04:36:01.314886778 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9792023-10-03T04:36:02.494Z2023-10-03 04:36:02.315811835 UTC: attempting to log into API
9802023-10-03T04:36:17.364Z2023-10-03 04:36:17.349926651 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9812023-10-03T04:36:18.385Z2023-10-03 04:36:18.351067242 UTC: attempting to log into API
9822023-10-03T04:36:33.394Z2023-10-03 04:36:33.385394922 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9832023-10-03T04:36:34.396Z2023-10-03 04:36:34.387238415 UTC: attempting to log into API
9842023-10-03T04:36:49.430Z2023-10-03 04:36:49.421548725 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9852023-10-03T04:36:50.430Z2023-10-03 04:36:50.422330069 UTC: attempting to log into API
9862023-10-03T04:37:05.474Z2023-10-03 04:37:05.455163754 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9872023-10-03T04:37:06.465Z2023-10-03 04:37:06.456257984 UTC: attempting to log into API
9882023-10-03T04:37:21.498Z2023-10-03 04:37:21.489056700 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9892023-10-03T04:37:22.499Z2023-10-03 04:37:22.490204971 UTC: attempting to log into API
9902023-10-03T04:37:37.532Z2023-10-03 04:37:37.523394195 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9912023-10-03T04:37:38.533Z2023-10-03 04:37:38.524424544 UTC: attempting to log into API
9922023-10-03T04:37:53.568Z2023-10-03 04:37:53.558569325 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9932023-10-03T04:37:54.571Z2023-10-03 04:37:54.560057732 UTC: attempting to log into API
9942023-10-03T04:38:09.615Z2023-10-03 04:38:09.605130257 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9952023-10-03T04:38:10.616Z2023-10-03 04:38:10.606214826 UTC: attempting to log into API
 
9962023-10-03T04:38:10.941ZOct 3 04:38:10 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9972023-10-03T04:38:17.949ZOct 3 04:38:17 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9982023-10-03T04:38:25.651Z2023-10-03 04:38:25.641575191 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9992023-10-03T04:38:25.742ZOct 3 04:38:24 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
10002023-10-03T04:38:26.652Z2023-10-03 04:38:26.642700220 UTC: attempting to log into API
 
10012023-10-03T04:38:33.279ZOct 3 04:38:32 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
10022023-10-03T04:38:39.529ZOct 3 04:38:38 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
10032023-10-03T04:38:41.686Z2023-10-03 04:38:41.677011066 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10042023-10-03T04:38:42.687Z2023-10-03 04:38:42.678145727 UTC: attempting to log into API
10052023-10-03T04:38:57.719Z2023-10-03 04:38:57.711150599 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10062023-10-03T04:38:58.721Z2023-10-03 04:38:58.712285269 UTC: attempting to log into API
10072023-10-03T04:39:13.755Z2023-10-03 04:39:13.746529295 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10082023-10-03T04:39:14.756Z2023-10-03 04:39:14.747355870 UTC: attempting to log into API
10092023-10-03T04:39:19.796Z2023-10-03 04:39:19.788131156 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: tcp connect error: Connection refused (os error 146)
10102023-10-03T04:39:20.798Z2023-10-03 04:39:20.789470720 UTC: attempting to log into API
10112023-10-03T04:39:21.571Z2023-10-03 04:39:21.561401041 UTC: login succeeded
10122023-10-03T04:39:21.602Zwaiting for nexus to come up...
10132023-10-03T04:39:21.665Zcreating IP pool... 10.151.2.175 - 10.151.2.227
10142023-10-03T04:39:21.717Zensuring datasets are ready...
10152023-10-03T04:39:24.125Zlet's roll.
10162023-10-03T04:39:24.149Z+ rm ./tests/bootstrap
10172023-10-03T04:39:24.209Z+ for test_bin in tests/*
10182023-10-03T04:39:24.232Z+ ./tests/bootstrap-57c3f40605e3c78b
10192023-10-03T04:39:24.256Z
10202023-10-03T04:39:24.280Zrunning 0 tests
10212023-10-03T04:39:24.304Z
10222023-10-03T04:39:24.328Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10232023-10-03T04:39:24.357Z
10242023-10-03T04:39:24.381Z+ for test_bin in tests/*
10252023-10-03T04:39:24.405Z+ ./tests/end_to_end_tests-88fbefd8fad2952b
10262023-10-03T04:39:24.430Z
10272023-10-03T04:39:24.454Zrunning 2 tests
10282023-10-03T04:39:24.478Ztest no_spoof::no_spoof_here ... ok
10292023-10-03T04:40:24.231Ztest instance_launch::instance_launch has been running for over 60 seconds
10302023-10-03T04:42:19.686Ztest instance_launch::instance_launch ... ok
10312023-10-03T04:42:19.710Z
10322023-10-03T04:42:19.734Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 175.46s
10332023-10-03T04:42:19.759Z
10342023-10-03T04:42:19.787Z+ _exit_trap
10352023-10-03T04:42:19.812Z+ local status=0
10362023-10-03T04:42:19.836Z+ [[ 0 -eq 0 ]]
10372023-10-03T04:42:19.860Z+ exit 0
10382023-10-03T04:42:19.884Zprocess exited: duration 478400 ms, exit code 0
 
10392023-10-03T04:42:19.961Zfound 47 output files
10402023-10-03T04:42:20.012Zuploading: /var/svc/log/oxide-sled-agent:default.log (435452 bytes)
10412023-10-03T04:42:21.255Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10422023-10-03T04:42:21.320Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5f99b052-3a86-497c-91e0-a4175ab4b33e/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10432023-10-03T04:42:22.404Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5f99b052-3a86-497c-91e0-a4175ab4b33e/root/var/svc/log/oxide-crucible-agent:default.log
10442023-10-03T04:42:22.429Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5f99b052-3a86-497c-91e0-a4175ab4b33e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10452023-10-03T04:42:23.497Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5f99b052-3a86-497c-91e0-a4175ab4b33e/root/var/svc/log/oxide-crucible-downstairs:default.log
10462023-10-03T04:42:23.522Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_59d42cf6-e814-48c3-9e0d-d2ac1c67141a/root/var/svc/log/oxide-internal_dns:default.log (101509 bytes)
10472023-10-03T04:42:24.677Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_59d42cf6-e814-48c3-9e0d-d2ac1c67141a/root/var/svc/log/oxide-internal_dns:default.log
10482023-10-03T04:42:24.704Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_b3fd41d5-80fb-4f3f-ab27-97631f2a4501/root/var/svc/log/oxide-nexus:default.log (855444 bytes)
10492023-10-03T04:42:24.991Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_b3fd41d5-80fb-4f3f-ab27-97631f2a4501/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 855444 -> 856052
10502023-10-03T04:42:26.062Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_b3fd41d5-80fb-4f3f-ab27-97631f2a4501/root/var/svc/log/oxide-nexus:default.log
10512023-10-03T04:42:26.086Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_2699c759-6d54-4058-9cb8-9ff85d45ec15/root/var/svc/log/oxide-clickhouse:default.log (667093 bytes)
10522023-10-03T04:42:26.315Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_2699c759-6d54-4058-9cb8-9ff85d45ec15/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 667093 -> 678823
10532023-10-03T04:42:27.384Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_2699c759-6d54-4058-9cb8-9ff85d45ec15/root/var/svc/log/oxide-clickhouse:default.log
10542023-10-03T04:42:27.408Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0765499b-9f41-42a2-8357-2ee80af6a9af/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10552023-10-03T04:42:28.486Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0765499b-9f41-42a2-8357-2ee80af6a9af/root/var/svc/log/oxide-cockroachdb:default.log
10562023-10-03T04:42:28.511Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_684c89f6-5362-4a53-a350-e2ffe2ac7b79/root/var/svc/log/oxide-crucible-agent:default.log (14298 bytes)
10572023-10-03T04:42:29.602Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_684c89f6-5362-4a53-a350-e2ffe2ac7b79/root/var/svc/log/oxide-crucible-agent:default.log
10582023-10-03T04:42:29.627Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_684c89f6-5362-4a53-a350-e2ffe2ac7b79/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10592023-10-03T04:42:30.695Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_684c89f6-5362-4a53-a350-e2ffe2ac7b79/root/var/svc/log/oxide-crucible-downstairs:default.log
10602023-10-03T04:42:30.719Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_684c89f6-5362-4a53-a350-e2ffe2ac7b79/root/var/svc/log/oxide-crucible-downstairs:downstairs-027ea2c0-fe9b-4f7d-962d-8fae1869592c.log (3494 bytes)
10612023-10-03T04:42:31.787Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_684c89f6-5362-4a53-a350-e2ffe2ac7b79/root/var/svc/log/oxide-crucible-downstairs:downstairs-027ea2c0-fe9b-4f7d-962d-8fae1869592c.log
10622023-10-03T04:42:31.812Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_722a1e81-2300-4a0d-8e02-df2ef0260642/root/var/svc/log/oxide-crucible-agent:default.log (14295 bytes)
10632023-10-03T04:42:32.902Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_722a1e81-2300-4a0d-8e02-df2ef0260642/root/var/svc/log/oxide-crucible-agent:default.log
10642023-10-03T04:42:32.927Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_722a1e81-2300-4a0d-8e02-df2ef0260642/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10652023-10-03T04:42:33.995Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_722a1e81-2300-4a0d-8e02-df2ef0260642/root/var/svc/log/oxide-crucible-downstairs:default.log
10662023-10-03T04:42:34.034Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_722a1e81-2300-4a0d-8e02-df2ef0260642/root/var/svc/log/oxide-crucible-downstairs:downstairs-faf16bb6-ba16-448b-8124-51bb70a56da1.log (3495 bytes)
10672023-10-03T04:42:35.086Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_722a1e81-2300-4a0d-8e02-df2ef0260642/root/var/svc/log/oxide-crucible-downstairs:downstairs-faf16bb6-ba16-448b-8124-51bb70a56da1.log
10682023-10-03T04:42:35.112Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9437cf60-11cd-4f5c-97a3-e2f5be4862a5/root/var/svc/log/oxide-crucible-agent:default.log (14300 bytes)
10692023-10-03T04:42:36.202Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9437cf60-11cd-4f5c-97a3-e2f5be4862a5/root/var/svc/log/oxide-crucible-agent:default.log
10702023-10-03T04:42:36.228Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9437cf60-11cd-4f5c-97a3-e2f5be4862a5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10712023-10-03T04:42:37.302Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9437cf60-11cd-4f5c-97a3-e2f5be4862a5/root/var/svc/log/oxide-crucible-downstairs:default.log
10722023-10-03T04:42:37.332Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9437cf60-11cd-4f5c-97a3-e2f5be4862a5/root/var/svc/log/oxide-crucible-downstairs:downstairs-1b4669af-8adf-48f5-a198-6d78c6f56ab6.log (5966 bytes)
10732023-10-03T04:42:38.398Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9437cf60-11cd-4f5c-97a3-e2f5be4862a5/root/var/svc/log/oxide-crucible-downstairs:downstairs-1b4669af-8adf-48f5-a198-6d78c6f56ab6.log
10742023-10-03T04:42:38.421Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_47431068-7509-462e-80bf-229db744cf06/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10752023-10-03T04:42:39.918Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_47431068-7509-462e-80bf-229db744cf06/root/var/svc/log/oxide-crucible-pantry:default.log
10762023-10-03T04:42:39.991Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_7ce8bd3a-4db0-4142-a0fb-296e3bf1685f/root/var/svc/log/oxide-nexus:default.log (4160902 bytes)
10772023-10-03T04:42:41.999Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_7ce8bd3a-4db0-4142-a0fb-296e3bf1685f/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4160902 -> 4220906
10782023-10-03T04:42:43.089Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_7ce8bd3a-4db0-4142-a0fb-296e3bf1685f/root/var/svc/log/oxide-nexus:default.log
10792023-10-03T04:42:43.113Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_080fff3c-dc49-4faa-a484-caebe22adce0/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10802023-10-03T04:42:44.186Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_080fff3c-dc49-4faa-a484-caebe22adce0/root/var/svc/log/oxide-crucible-agent:default.log
10812023-10-03T04:42:44.210Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_080fff3c-dc49-4faa-a484-caebe22adce0/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10822023-10-03T04:42:45.283Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_080fff3c-dc49-4faa-a484-caebe22adce0/root/var/svc/log/oxide-crucible-downstairs:default.log
10832023-10-03T04:42:45.308Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_78c3a6b5-3ab6-4e78-a9c1-4754baa2938b/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10842023-10-03T04:42:46.381Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_78c3a6b5-3ab6-4e78-a9c1-4754baa2938b/root/var/svc/log/oxide-crucible-agent:default.log
10852023-10-03T04:42:46.405Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_78c3a6b5-3ab6-4e78-a9c1-4754baa2938b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10862023-10-03T04:42:47.478Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_78c3a6b5-3ab6-4e78-a9c1-4754baa2938b/root/var/svc/log/oxide-crucible-downstairs:default.log
10872023-10-03T04:42:47.503Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_e7915780-8adb-49e1-883e-9e6bb6117b1d/root/var/svc/log/oxide-external_dns:default.log (15609 bytes)
10882023-10-03T04:42:47.571Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_e7915780-8adb-49e1-883e-9e6bb6117b1d/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15609 -> 19386
10892023-10-03T04:42:48.643Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_e7915780-8adb-49e1-883e-9e6bb6117b1d/root/var/svc/log/oxide-external_dns:default.log
10902023-10-03T04:42:48.667Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_90f934a9-1410-4ae4-8300-e5818cc0e7cd/root/var/svc/log/oxide-internal_dns:default.log (99481 bytes)
10912023-10-03T04:42:48.784Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_90f934a9-1410-4ae4-8300-e5818cc0e7cd/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99481 -> 103236
10922023-10-03T04:42:49.857Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_90f934a9-1410-4ae4-8300-e5818cc0e7cd/root/var/svc/log/oxide-internal_dns:default.log
10932023-10-03T04:42:49.880Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_b9b66882-90f4-42c3-beb3-e4cb028420bf/root/var/svc/log/oxide-nexus:default.log (456208 bytes)
10942023-10-03T04:42:50.138Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_b9b66882-90f4-42c3-beb3-e4cb028420bf/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 456208 -> 516824
10952023-10-03T04:42:51.212Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_b9b66882-90f4-42c3-beb3-e4cb028420bf/root/var/svc/log/oxide-nexus:default.log
10962023-10-03T04:42:51.253Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b7e660f0-c149-4b89-93cf-9df12cda7d34/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10972023-10-03T04:42:52.327Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b7e660f0-c149-4b89-93cf-9df12cda7d34/root/var/svc/log/oxide-crucible-agent:default.log
10982023-10-03T04:42:52.350Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b7e660f0-c149-4b89-93cf-9df12cda7d34/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10992023-10-03T04:42:53.422Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b7e660f0-c149-4b89-93cf-9df12cda7d34/root/var/svc/log/oxide-crucible-downstairs:default.log
11002023-10-03T04:42:53.446Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_4fcb94b1-52e4-497b-ac17-6539603bcf7d/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
11012023-10-03T04:42:54.517Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_4fcb94b1-52e4-497b-ac17-6539603bcf7d/root/var/svc/log/oxide-crucible-pantry:default.log
11022023-10-03T04:42:54.541Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_76b4f6c9-c3b6-43cf-a071-35fda263562b/root/var/svc/log/oxide-crucible-pantry:default.log (2336 bytes)
11032023-10-03T04:42:55.614Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_76b4f6c9-c3b6-43cf-a071-35fda263562b/root/var/svc/log/oxide-crucible-pantry:default.log
11042023-10-03T04:42:55.637Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572/root/var/svc/log/oxide-ntp:default.log (910 bytes)
11052023-10-03T04:42:56.711Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572/root/var/svc/log/oxide-ntp:default.log
11062023-10-03T04:42:56.735Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_5ba62614-a596-4dc2-952a-9bf630e92527/root/var/svc/log/oxide-oximeter:default.log (188062 bytes)
11072023-10-03T04:42:56.919Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_5ba62614-a596-4dc2-952a-9bf630e92527/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 188062 -> 217952
11082023-10-03T04:42:57.992Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_5ba62614-a596-4dc2-952a-9bf630e92527/root/var/svc/log/oxide-oximeter:default.log
11092023-10-03T04:42:58.016Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_041f4fd5-b926-415d-8a4d-6e35c54e139b/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
11102023-10-03T04:42:59.090Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_041f4fd5-b926-415d-8a4d-6e35c54e139b/root/var/svc/log/oxide-cockroachdb:default.log
11112023-10-03T04:42:59.114Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_e6d47a02-b575-4a58-b9d5-39e65a6a5fd3/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
11122023-10-03T04:43:00.188Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_e6d47a02-b575-4a58-b9d5-39e65a6a5fd3/root/var/svc/log/oxide-cockroachdb:default.log
11132023-10-03T04:43:00.212Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8109b950-1636-4d57-a80f-5251a83e9b1e/root/var/svc/log/oxide-crucible-agent:default.log (14764 bytes)
11142023-10-03T04:43:01.309Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8109b950-1636-4d57-a80f-5251a83e9b1e/root/var/svc/log/oxide-crucible-agent:default.log
11152023-10-03T04:43:01.333Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8109b950-1636-4d57-a80f-5251a83e9b1e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11162023-10-03T04:43:02.408Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8109b950-1636-4d57-a80f-5251a83e9b1e/root/var/svc/log/oxide-crucible-downstairs:default.log
11172023-10-03T04:43:02.435Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8109b950-1636-4d57-a80f-5251a83e9b1e/root/var/svc/log/oxide-crucible-downstairs:downstairs-35aa7fa9-9c9d-41b8-a334-bda5ab1a9ac2.log (5964 bytes)
11182023-10-03T04:43:03.509Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8109b950-1636-4d57-a80f-5251a83e9b1e/root/var/svc/log/oxide-crucible-downstairs:downstairs-35aa7fa9-9c9d-41b8-a334-bda5ab1a9ac2.log
11192023-10-03T04:43:03.555Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9065e406-8d1b-4538-b337-29ccc61106fd/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
11202023-10-03T04:43:04.636Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9065e406-8d1b-4538-b337-29ccc61106fd/root/var/svc/log/oxide-cockroachdb:default.log
11212023-10-03T04:43:04.660Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b865bb30-fe30-4c62-8ada-a0b3807c4047/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
11222023-10-03T04:43:05.735Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b865bb30-fe30-4c62-8ada-a0b3807c4047/root/var/svc/log/oxide-cockroachdb:default.log
11232023-10-03T04:43:05.759Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e0018e51-f156-42ff-8b51-9816e5081b78/root/var/svc/log/oxide-external_dns:default.log (15612 bytes)
11242023-10-03T04:43:05.830Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e0018e51-f156-42ff-8b51-9816e5081b78/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15612 -> 19385
11252023-10-03T04:43:06.904Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e0018e51-f156-42ff-8b51-9816e5081b78/root/var/svc/log/oxide-external_dns:default.log
11262023-10-03T04:43:06.929Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e0f9ddf5-627e-4c61-b74b-671314e193bf/root/var/svc/log/oxide-crucible-agent:default.log (25704 bytes)
11272023-10-03T04:43:08.048Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e0f9ddf5-627e-4c61-b74b-671314e193bf/root/var/svc/log/oxide-crucible-agent:default.log
11282023-10-03T04:43:08.072Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e0f9ddf5-627e-4c61-b74b-671314e193bf/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11292023-10-03T04:43:09.155Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e0f9ddf5-627e-4c61-b74b-671314e193bf/root/var/svc/log/oxide-crucible-downstairs:default.log
11302023-10-03T04:43:09.183Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e0f9ddf5-627e-4c61-b74b-671314e193bf/root/var/svc/log/oxide-crucible-downstairs:downstairs-b5fa758a-6915-453f-8128-8814d99e7574.log (5966 bytes)
11312023-10-03T04:43:10.265Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e0f9ddf5-627e-4c61-b74b-671314e193bf/root/var/svc/log/oxide-crucible-downstairs:downstairs-b5fa758a-6915-453f-8128-8814d99e7574.log
11322023-10-03T04:43:10.289Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e0f9ddf5-627e-4c61-b74b-671314e193bf/root/var/svc/log/oxide-crucible-downstairs:downstairs-e735b68f-b611-4ac4-932b-59c2289df996.log (3495 bytes)
11332023-10-03T04:43:11.363Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e0f9ddf5-627e-4c61-b74b-671314e193bf/root/var/svc/log/oxide-crucible-downstairs:downstairs-e735b68f-b611-4ac4-932b-59c2289df996.log
11342023-10-03T04:43:11.387Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_8aab4d9e-67ba-4c3a-86b0-394a4353825d/root/var/svc/log/oxide-internal_dns:default.log (98667 bytes)
11352023-10-03T04:43:11.535Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_8aab4d9e-67ba-4c3a-86b0-394a4353825d/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 98667 -> 102421
11362023-10-03T04:43:12.611Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_8aab4d9e-67ba-4c3a-86b0-394a4353825d/root/var/svc/log/oxide-internal_dns:default.log
11372023-10-03T04:43:12.635Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572/root/var/log/chrony/measurements.log (9042 bytes)
11382023-10-03T04:43:12.772Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572/root/var/log/chrony/measurements.log" changed size mid upload: 9042 -> 12741
11392023-10-03T04:43:13.850Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572/root/var/log/chrony/measurements.log
11402023-10-03T04:43:13.892Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572/root/var/log/chrony/statistics.log (5850 bytes)
11412023-10-03T04:43:13.933Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572/root/var/log/chrony/statistics.log" changed size mid upload: 5850 -> 7722
11422023-10-03T04:43:15.005Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572/root/var/log/chrony/statistics.log
11432023-10-03T04:43:15.029Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572/root/var/log/chrony/tracking.log (1587 bytes)
11442023-10-03T04:43:15.054Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572/root/var/log/chrony/tracking.log" changed size mid upload: 1587 -> 1719
11452023-10-03T04:43:16.129Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_279d05e7-0f6a-47d5-bfb0-35683234e572/root/var/log/chrony/tracking.log