01HBY3S4W24RPFSMYH3BPTSK0H: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.23",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HBY3WDCC00BCZPSM1MR14102

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-04T20:38:15.973Zjob dependencies complete; ready to run (waiting for 47 m 25 s)
22023-10-04T20:40:43.510Zjob assigned to worker 01HBY6NABDA8Y1HT1W4D9GPER1 (queued for 2 m 27 s)
32023-10-04T20:40:44.043Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-10-04T20:40:44.158Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-10-04T20:40:44.280Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-10-04T20:40:44.359Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-04T20:40:44.469Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-10-04T20:40:44.582ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-04T20:40:45.049ZOxide Helios Version helios-2.0.22085 64-bit
102023-10-04T20:40:45.214ZConfiguring devices.
112023-10-04T20:40:45.289Z##################################################
122023-10-04T20:40:45.590Z#### ############# ##
132023-10-04T20:40:45.793Z### ### ############ ##
142023-10-04T20:40:46.470Z## ### # ## ### ## Oxide Computer Company ##
152023-10-04T20:40:46.622Z## ## ## ### # ### ##
162023-10-04T20:40:47.182Z## # ### #### #### This Station Under ##
172023-10-04T20:40:47.400Z### ### #### # ### Computer Control ##
182023-10-04T20:40:47.580Z#### #### ### ## ##
192023-10-04T20:40:47.786Z##################################################
202023-10-04T20:40:48.491ZHostname: sock
212023-10-04T20:40:48.735ZLoading smf(7) serice descriptions:
222023-10-04T20:40:49.651Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-61cfb22b53ec0472.gz
232023-10-04T20:40:56.268Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-61cfb22b53ec0472.gz
242023-10-04T20:40:56.324Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
252023-10-04T20:43:02.558Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-04T20:43:03.043Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cfdf15182b8b4dac.gz
272023-10-04T20:43:42.140Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cfdf15182b8b4dac.gz
282023-10-04T20:43:42.176Zdownloading input: /input/ci-tools/work/caboose-util.gz
292023-10-04T20:43:46.445Zdownloaded input: /input/ci-tools/work/caboose-util.gz
302023-10-04T20:43:46.473Zdownloading input: /input/ci-tools/work/tufaceous.gz
312023-10-04T20:43:50.639Zdownloaded input: /input/ci-tools/work/tufaceous.gz
322023-10-04T20:43:50.664Zdownloading input: /input/package/work/version.txt
332023-10-04T20:43:50.693Zdownloaded input: /input/package/work/version.txt
342023-10-04T20:43:50.717Zdownloading input: /input/package/work/package.tar.gz
352023-10-04T20:44:11.321Zdownloaded input: /input/package/work/package.tar.gz
362023-10-04T20:44:11.375Zdownloading input: /input/package/work/global-zone-packages.tar.gz
372023-10-04T20:44:34.153Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
382023-10-04T20:44:34.190Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
392023-10-04T20:44:41.177Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-04T20:44:41.202Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
412023-10-04T20:45:13.315Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
422023-10-04T20:45:13.339Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
432023-10-04T20:45:44.854Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-04T20:45:44.877Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
452023-10-04T20:46:05.325Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-04T20:46:05.350Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
472023-10-04T20:46:09.432Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-04T20:46:09.456Zdownloading input: /input/package/work/zones/crucible.tar.gz
492023-10-04T20:46:17.127Zdownloaded input: /input/package/work/zones/crucible.tar.gz
502023-10-04T20:46:17.152Zdownloading input: /input/package/work/zones/external-dns.tar.gz
512023-10-04T20:46:20.998Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
522023-10-04T20:46:21.037Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
532023-10-04T20:46:25.216Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
542023-10-04T20:46:25.241Zdownloading input: /input/package/work/zones/ntp.tar.gz
552023-10-04T20:46:25.293Zdownloaded input: /input/package/work/zones/ntp.tar.gz
562023-10-04T20:46:25.317Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
572023-10-04T20:46:28.675Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-04T20:46:28.699Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
592023-10-04T20:46:31.555Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-04T20:46:31.580Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
612023-10-04T20:46:43.479Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-04T20:46:43.518Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
632023-10-04T20:46:55.813Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-04T20:46:55.837Zdownloading input: /input/package/work/zones/overlay.tar.gz
652023-10-04T20:46:55.863Zdownloaded input: /input/package/work/zones/overlay.tar.gz
662023-10-04T20:46:55.888Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
672023-10-04T20:46:59.141Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-04T20:46:59.166Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
692023-10-04T20:47:10.617Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
702023-10-04T20:47:10.641Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
712023-10-04T20:47:49.371Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
722023-10-04T20:47:49.396Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
732023-10-04T20:48:16.599Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
742023-10-04T20:48:16.626Zstarting task 0: "setup"
752023-10-04T20:48:16.663Z++ uname -s
762023-10-04T20:48:16.692Z+ kern=SunOS
772023-10-04T20:48:16.722Z+ case "$kern" in
782023-10-04T20:48:16.753Z+ groupadd -g 12345 build
792023-10-04T20:48:16.781Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
802023-10-04T20:48:18.669Z+ zfs create -o mountpoint=/work rpool/work
812023-10-04T20:48:18.695Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
822023-10-04T20:48:18.719Z+ home_fs=zfs
832023-10-04T20:48:18.743Z+ [[ zfs == autofs ]]
842023-10-04T20:48:18.768Z+ mkdir -p /home/build
852023-10-04T20:48:18.791Z+ chown build:build /home/build /work
862023-10-04T20:48:20.700Z+ chmod 0700 /home/build /work
872023-10-04T20:48:20.724Zprocess exited: duration 4052 ms, exit code 0
 
882023-10-04T20:48:20.773Zstarting task 1: "authentication"
892023-10-04T20:48:21.260Zprocess exited: duration 43 ms, exit code 0
 
902023-10-04T20:48:21.319Zstarting task 2: "build"
912023-10-04T20:48:21.351Z+ trap _exit_trap EXIT
922023-10-04T20:48:21.376Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
932023-10-04T20:48:21.549Zsecpolicy_fs_linkdir: 0x55 = 0x55
942023-10-04T20:48:21.574Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
952023-10-04T20:48:21.598Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
962023-10-04T20:48:21.623Zsecpolicy_fs_linkdir+3: 0 = 0xe5
972023-10-04T20:48:21.650Zsecpolicy_fs_linkdir+4: 0 = 0xb8
982023-10-04T20:48:21.676Zsecpolicy_fs_linkdir+5: 0 = 0x1
992023-10-04T20:48:21.700Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1002023-10-04T20:48:21.725Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1012023-10-04T20:48:21.751Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1022023-10-04T20:48:21.778Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1032023-10-04T20:48:21.802Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1042023-10-04T20:48:21.827Z+ [[ -d /opt/oxide ]]
1052023-10-04T20:48:21.851Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1062023-10-04T20:48:21.875Z+ pfexec mkdir /opt/oxide
1072023-10-04T20:48:21.899Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1082023-10-04T20:48:21.924Z+ [[ -d /opt/oxide-underneath ]]
1092023-10-04T20:48:21.947Z+ cd /opt/oxide-underneath
1102023-10-04T20:48:21.971Z+ pfexec tar ceEp@/f - .
1112023-10-04T20:48:21.998Z+ cd /opt/oxide
1122023-10-04T20:48:22.022Z+ pfexec tar xveEp@/f -
1132023-10-04T20:48:22.046Zx ., 0 bytes, 0 tape blocks
1142023-10-04T20:48:22.070Zx ./opte, 0 bytes, 0 tape blocks
1152023-10-04T20:48:22.094Zx ./opte/bin, 0 bytes, 0 tape blocks
1162023-10-04T20:48:22.118Zx ./opte/bin/opteadm, 46195888 bytes, 90227 tape blocks
1172023-10-04T20:48:22.143Z+ pfexec rm -rf /opt/oxide-underneath
1182023-10-04T20:48:22.168Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1192023-10-04T20:48:22.192Z+ pfexec mkdir /opt/oxide/work
1202023-10-04T20:48:22.215Z+ pfexec chown build:build /opt/oxide/work
1212023-10-04T20:48:22.239Z+ cd /opt/oxide/work
1222023-10-04T20:48:22.271Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1232023-10-04T20:48:23.342ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1242023-10-04T20:48:23.367Zx out/maghemite.tar, 74957312 bytes, 146401 tape blocks
1252023-10-04T20:48:23.390Zx out/omicron-sled-agent.tar, 113288192 bytes, 221266 tape blocks
1262023-10-04T20:48:23.415Zx out/target/test, 75 bytes, 1 tape blocks
1272023-10-04T20:48:23.440Zx out/npuzone/npuzone, 36941864 bytes, 72153 tape blocks
1282023-10-04T20:48:23.464Zx package-manifest.toml, 21303 bytes, 42 tape blocks
1292023-10-04T20:48:23.490Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1302023-10-04T20:48:23.514Zx target/release/omicron-package, 37946976 bytes, 74116 tape blocks
1312023-10-04T20:48:23.537Zx tools/create_virtual_hardware.sh, 2810 bytes, 6 tape blocks
1322023-10-04T20:48:23.563Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1332023-10-04T20:48:23.587Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1342023-10-04T20:48:23.610Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1352023-10-04T20:48:23.635Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1362023-10-04T20:48:23.659Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1372023-10-04T20:48:23.683Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1382023-10-04T20:48:23.707Z
1392023-10-04T20:48:23.731Zreal 1.587159111
1402023-10-04T20:48:23.794Zuser 1.438704452
1412023-10-04T20:48:23.818Zsys 0.267862379
1422023-10-04T20:48:23.852Ztrap 0.000057460
1432023-10-04T20:48:23.875Ztflt 0.000000000
1442023-10-04T20:48:23.899Zdflt 0.000000000
1452023-10-04T20:48:23.923Zkflt 0.000000000
1462023-10-04T20:48:23.947Zlock 0.000000000
1472023-10-04T20:48:23.970Zslp 1.433702523
1482023-10-04T20:48:23.994Zlat 0.032113176
1492023-10-04T20:48:24.018Zstop 0.000124791
1502023-10-04T20:48:24.042Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1512023-10-04T20:48:24.317Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1522023-10-04T20:48:24.361Z+ mkdir tests
1532023-10-04T20:48:24.385Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1542023-10-04T20:48:24.408Z+ ptime -m gunzip
1552023-10-04T20:48:24.433Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-61cfb22b53ec0472
1562023-10-04T20:48:24.477Z
1572023-10-04T20:48:24.502Zreal 0.141368667
1582023-10-04T20:48:24.526Zuser 0.129458417
1592023-10-04T20:48:24.549Zsys 0.011653362
1602023-10-04T20:48:24.573Ztrap 0.000015200
1612023-10-04T20:48:24.597Ztflt 0.000000000
1622023-10-04T20:48:24.647Zdflt 0.000000000
1632023-10-04T20:48:24.671Zkflt 0.000000000
1642023-10-04T20:48:24.699Zlock 0.000000000
1652023-10-04T20:48:24.723Zslp 0.000024680
1662023-10-04T20:48:24.746Zlat 0.000023301
1672023-10-04T20:48:24.770Zstop 0.000133322
1682023-10-04T20:48:24.794Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-61cfb22b53ec0472
1692023-10-04T20:48:24.819Z+ chmod a+x tests/bootstrap-61cfb22b53ec0472
1702023-10-04T20:48:24.844Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1712023-10-04T20:48:24.868Z+ ptime -m gunzip
1722023-10-04T20:48:24.892Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1732023-10-04T20:48:26.704Z
1742023-10-04T20:48:26.728Zreal 2.213999108
1752023-10-04T20:48:26.752Zuser 2.065276774
1762023-10-04T20:48:26.776Zsys 0.148300504
1772023-10-04T20:48:26.800Ztrap 0.000171286
1782023-10-04T20:48:26.824Ztflt 0.000000000
1792023-10-04T20:48:26.850Zdflt 0.000000000
1802023-10-04T20:48:26.874Zkflt 0.000000000
1812023-10-04T20:48:26.899Zlock 0.000000000
1822023-10-04T20:48:26.923Zslp 0.000023120
1832023-10-04T20:48:26.947Zlat 0.000040700
1842023-10-04T20:48:26.972Zstop 0.000127342
1852023-10-04T20:48:26.996Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1862023-10-04T20:48:27.020Z+ chmod a+x tests/bootstrap
1872023-10-04T20:48:27.052Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1882023-10-04T20:48:27.076Z+ ptime -m gunzip
1892023-10-04T20:48:27.100Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cfdf15182b8b4dac
1902023-10-04T20:48:29.269Z
1912023-10-04T20:48:29.293Zreal 2.553785909
1922023-10-04T20:48:29.317Zuser 2.400170681
1932023-10-04T20:48:29.341Zsys 0.153229579
1942023-10-04T20:48:29.365Ztrap 0.000161854
1952023-10-04T20:48:29.390Ztflt 0.000000000
1962023-10-04T20:48:29.414Zdflt 0.000000000
1972023-10-04T20:48:29.438Zkflt 0.000000000
1982023-10-04T20:48:29.462Zlock 0.000000000
1992023-10-04T20:48:29.486Zslp 0.000018290
2002023-10-04T20:48:29.511Zlat 0.000022911
2012023-10-04T20:48:29.537Zstop 0.000128792
2022023-10-04T20:48:29.589Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cfdf15182b8b4dac
2032023-10-04T20:48:29.613Z+ chmod a+x tests/end_to_end_tests-cfdf15182b8b4dac
2042023-10-04T20:48:29.638Z+ bmat address ls
2052023-10-04T20:48:29.662ZNAME CIDR FIRST LAST COUNT
2062023-10-04T20:48:29.693Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2072023-10-04T20:48:29.717Z++ bmat address ls -f extra -Ho first,last,count
2082023-10-04T20:48:29.741Z+ set -- 10.151.2.164 10.151.2.227 64
2092023-10-04T20:48:29.765Z+ EXTRA_IP_START=10.151.2.164
2102023-10-04T20:48:29.789Z+ EXTRA_IP_END=10.151.2.227
2112023-10-04T20:48:29.814Z+ EXTRA_IP_COUNT=64
2122023-10-04T20:48:29.839Z+ (( EXTRA_IP_COUNT >= 32 ))
2132023-10-04T20:48:29.864Z+ EXTRA_IP_BASE=10.151.2
2142023-10-04T20:48:29.890Z+ EXTRA_IP_FOCTET=164
2152023-10-04T20:48:29.916Z+ EXTRA_IP_LOCTET=227
2162023-10-04T20:48:29.940Z+ SERVICE_IP_POOL_START=10.151.2.164
2172023-10-04T20:48:29.965Z+ SERVICE_IP_POOL_END=10.151.2.173
2182023-10-04T20:48:29.989Z+ DNS_IP1=10.151.2.164
2192023-10-04T20:48:30.013Z+ DNS_IP2=10.151.2.165
2202023-10-04T20:48:30.037Z+ UPLINK_IP=10.151.2.174
2212023-10-04T20:48:30.061Z+ IPPOOL_START=10.151.2.175
2222023-10-04T20:48:30.086Z+ IPPOOL_END=10.151.2.227
2232023-10-04T20:48:30.109Z++ ipadm show-addr -po type,addr
2242023-10-04T20:48:30.133Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2252023-10-04T20:48:30.157Z+ GATEWAY_IP=10.151.2.100
2262023-10-04T20:48:30.181Z+ [[ -n 10.151.2.100 ]]
2272023-10-04T20:48:30.205Z+ ping -s 10.151.2.100 56 1
2282023-10-04T20:48:30.229ZPING 10.151.2.100: 56 data bytes
2292023-10-04T20:48:30.253Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2302023-10-04T20:48:30.281Z
2312023-10-04T20:48:30.306Z----10.151.2.100 PING Statistics----
2322023-10-04T20:48:30.332Z1 packets transmitted, 1 packets received, 0% packet loss
2332023-10-04T20:48:30.358Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2342023-10-04T20:48:30.382Z++ arp -an
2352023-10-04T20:48:30.408Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2362023-10-04T20:48:30.432Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2372023-10-04T20:48:30.456Z+ routeadm -e ipv4-forwarding -u
2382023-10-04T20:48:30.482Z+ PXA_START=10.151.2.164
2392023-10-04T20:48:30.506Z+ PXA_END=10.151.2.227
2402023-10-04T20:48:30.539Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2412023-10-04T20:48:30.563Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2422023-10-04T20:48:30.587Z+ ZPOOL_VDEV_DIR=/scratch
2432023-10-04T20:48:30.612Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2442023-10-04T20:48:30.636Z+++ dirname ./tools/create_virtual_hardware.sh
2452023-10-04T20:48:30.666Z++ cd ./tools
2462023-10-04T20:48:30.691Z++ pwd
2472023-10-04T20:48:30.714Z+ SOURCE_DIR=/opt/oxide/work/tools
2482023-10-04T20:48:30.738Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2492023-10-04T20:48:30.763Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2502023-10-04T20:48:30.788Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2512023-10-04T20:48:30.812Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2522023-10-04T20:48:30.840Z++ dladm show-phys -p -o LINK
2532023-10-04T20:48:30.870Z++ head -1
2542023-10-04T20:48:30.896Z+ PHYSICAL_LINK=igb0
2552023-10-04T20:48:30.919Z+ echo 'Using igb0 as physical link'
2562023-10-04T20:48:30.946ZUsing igb0 as physical link
2572023-10-04T20:48:30.971Z+ ensure_run_as_root
2582023-10-04T20:48:31.004Z++ id -u
2592023-10-04T20:48:31.040Z+ [[ 0 -ne 0 ]]
2602023-10-04T20:48:31.069Z+ ensure_zpools
2612023-10-04T20:48:31.097Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2622023-10-04T20:48:31.126Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2632023-10-04T20:48:31.163Z+ readarray -t ZPOOLS
2642023-10-04T20:48:31.203Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2652023-10-04T20:48:31.244Z++ sed 's/[ ",]//g'
2662023-10-04T20:48:31.271Z+ for ZPOOL in "${ZPOOLS[@]}"
2672023-10-04T20:48:31.297Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2682023-10-04T20:48:31.325ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2692023-10-04T20:48:31.356Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2702023-10-04T20:48:31.390Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2712023-10-04T20:48:31.419Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2722023-10-04T20:48:31.448Z0+0 records in
2732023-10-04T20:48:31.478Z0+0 records out
2742023-10-04T20:48:31.508Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2752023-10-04T20:48:31.595ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2762023-10-04T20:48:31.636Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2772023-10-04T20:48:31.677ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2782023-10-04T20:48:31.711Z+ set +x
2792023-10-04T20:48:31.743ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2802023-10-04T20:48:31.772Z++ zpool list -o name
2812023-10-04T20:48:31.797ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2822023-10-04T20:48:31.822Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2832023-10-04T20:48:31.846ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2842023-10-04T20:48:31.871Z+ [[ -z '' ]]
2852023-10-04T20:48:31.896ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2862023-10-04T20:48:31.920Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2872023-10-04T20:48:31.943ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2882023-10-04T20:48:31.967Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2892023-10-04T20:48:31.991ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2902023-10-04T20:48:32.015Z+ set +x
2912023-10-04T20:48:32.039ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2922023-10-04T20:48:32.063Z+ for ZPOOL in "${ZPOOLS[@]}"
2932023-10-04T20:48:32.097ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2942023-10-04T20:48:32.129Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2952023-10-04T20:48:32.154ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2962023-10-04T20:48:32.178Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2972023-10-04T20:48:32.202ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2982023-10-04T20:48:32.226Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2992023-10-04T20:48:32.250ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3002023-10-04T20:48:32.274Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3012023-10-04T20:48:32.297ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3022023-10-04T20:48:32.322Z0+0 records in
3032023-10-04T20:48:32.346ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3042023-10-04T20:48:32.371Z0+0 records out
3052023-10-04T20:48:32.395ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3062023-10-04T20:48:32.420Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3072023-10-04T20:48:32.444ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3082023-10-04T20:48:32.469Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3092023-10-04T20:48:32.494ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3102023-10-04T20:48:32.530Z+ set +x
3112023-10-04T20:48:32.571ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3122023-10-04T20:48:32.614Z++ zpool list -o name
3132023-10-04T20:48:32.640ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3142023-10-04T20:48:32.664Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3152023-10-04T20:48:32.688ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3162023-10-04T20:48:32.724Z+ [[ -z '' ]]
3172023-10-04T20:48:32.748ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3182023-10-04T20:48:32.774Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3192023-10-04T20:48:32.798ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3202023-10-04T20:48:32.822Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3212023-10-04T20:48:32.846ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3222023-10-04T20:48:32.870Z+ set +x
3232023-10-04T20:48:32.896ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3242023-10-04T20:48:32.920Z+ for ZPOOL in "${ZPOOLS[@]}"
3252023-10-04T20:48:32.944ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3262023-10-04T20:48:33.379Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-10-04T20:48:33.819ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3282023-10-04T20:48:33.861Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-10-04T20:48:33.898ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3302023-10-04T20:48:33.922Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3312023-10-04T20:48:33.946ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3322023-10-04T20:48:33.970Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3332023-10-04T20:48:33.994ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3342023-10-04T20:48:34.018Z0+0 records in
3352023-10-04T20:48:34.042ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3362023-10-04T20:48:34.066Z0+0 records out
3372023-10-04T20:48:34.090ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3382023-10-04T20:48:34.114Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3392023-10-04T20:48:34.138ZSimnet net0/sc0_0 exists
3402023-10-04T20:48:34.174Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3412023-10-04T20:48:34.198ZSimnet net1/sc1_0 exists
3422023-10-04T20:48:34.222Z+ set +x
3432023-10-04T20:48:34.246ZVnic sc0_1 exists
3442023-10-04T20:48:34.270Z++ zpool list -o name
3452023-10-04T20:48:34.312Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3462023-10-04T20:48:34.347Z+ [[ -z '' ]]
3472023-10-04T20:48:34.371Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-10-04T20:48:34.396Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3492023-10-04T20:48:34.419Z+ set +x
3502023-10-04T20:48:34.444Z+ for ZPOOL in "${ZPOOLS[@]}"
3512023-10-04T20:48:34.469Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3522023-10-04T20:48:34.495Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3532023-10-04T20:48:34.520Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3542023-10-04T20:48:34.576Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3552023-10-04T20:48:34.602Z0+0 records in
3562023-10-04T20:48:34.626Z0+0 records out
3572023-10-04T20:48:34.650Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3582023-10-04T20:48:34.680Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3592023-10-04T20:48:34.715Z+ set +x
3602023-10-04T20:48:35.181Z++ zpool list -o name
3612023-10-04T20:48:35.205Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3622023-10-04T20:48:35.230Z+ [[ -z '' ]]
3632023-10-04T20:48:35.255Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3642023-10-04T20:48:35.280Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3652023-10-04T20:48:35.304Z+ set +x
3662023-10-04T20:48:35.329Z+ for ZPOOL in "${ZPOOLS[@]}"
3672023-10-04T20:48:35.360Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3682023-10-04T20:48:35.384Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3692023-10-04T20:48:35.408Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3702023-10-04T20:48:35.432Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3712023-10-04T20:48:35.456Z0+0 records in
3722023-10-04T20:48:35.480Z0+0 records out
3732023-10-04T20:48:35.504Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3742023-10-04T20:48:35.528Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3752023-10-04T20:48:35.552Z+ set +x
3762023-10-04T20:48:35.577Z++ zpool list -o name
3772023-10-04T20:48:35.601Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3782023-10-04T20:48:35.637Z+ [[ -z '' ]]
3792023-10-04T20:48:35.662Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3802023-10-04T20:48:35.704Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3812023-10-04T20:48:35.727Z+ set +x
3822023-10-04T20:48:35.751Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-10-04T20:48:35.775Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3842023-10-04T20:48:35.799Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3852023-10-04T20:48:35.823Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3862023-10-04T20:48:35.848Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3872023-10-04T20:48:35.872Z0+0 records in
3882023-10-04T20:48:35.896Z0+0 records out
3892023-10-04T20:48:35.920Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3902023-10-04T20:48:35.945Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3912023-10-04T20:48:35.969Z+ set +x
3922023-10-04T20:48:36.004Z++ zpool list -o name
3932023-10-04T20:48:36.040Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3942023-10-04T20:48:36.064Z+ [[ -z '' ]]
3952023-10-04T20:48:36.088Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3962023-10-04T20:48:36.129Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3972023-10-04T20:48:36.154Z+ set +x
3982023-10-04T20:48:36.178Z+ for ZPOOL in "${ZPOOLS[@]}"
3992023-10-04T20:48:36.202Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4002023-10-04T20:48:36.226Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4012023-10-04T20:48:36.250Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4022023-10-04T20:48:36.275Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4032023-10-04T20:48:36.300Z0+0 records in
4042023-10-04T20:48:36.326Z0+0 records out
4052023-10-04T20:48:36.353Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4062023-10-04T20:48:36.379Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4072023-10-04T20:48:36.405Z+ set +x
4082023-10-04T20:48:36.430Z++ zpool list -o name
4092023-10-04T20:48:36.466Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4102023-10-04T20:48:36.529Z+ [[ -z '' ]]
4112023-10-04T20:48:36.575Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4122023-10-04T20:48:36.603Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4132023-10-04T20:48:36.632Z+ set +x
4142023-10-04T20:48:36.659Z+ for ZPOOL in "${ZPOOLS[@]}"
4152023-10-04T20:48:36.692Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4162023-10-04T20:48:36.742Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-10-04T20:48:36.771Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4182023-10-04T20:48:36.805Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4192023-10-04T20:48:36.832Z0+0 records in
4202023-10-04T20:48:36.858Z0+0 records out
4212023-10-04T20:48:36.883Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4222023-10-04T20:48:36.907Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4232023-10-04T20:48:36.932Z+ set +x
4242023-10-04T20:48:36.968Z++ zpool list -o name
4252023-10-04T20:48:37.008Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4262023-10-04T20:48:37.048Z+ [[ -z '' ]]
4272023-10-04T20:48:37.086Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4282023-10-04T20:48:37.115Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4292023-10-04T20:48:37.139Z+ set +x
4302023-10-04T20:48:37.163Z+ for ZPOOL in "${ZPOOLS[@]}"
4312023-10-04T20:48:37.188Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4322023-10-04T20:48:37.213Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4332023-10-04T20:48:37.237Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4342023-10-04T20:48:37.261Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4352023-10-04T20:48:37.285Z0+0 records in
4362023-10-04T20:48:37.311Z0+0 records out
4372023-10-04T20:48:37.335Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4382023-10-04T20:48:37.359Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4392023-10-04T20:48:37.383Z+ set +x
4402023-10-04T20:48:37.410Z++ zpool list -o name
4412023-10-04T20:48:37.443Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4422023-10-04T20:48:37.476Z+ [[ -z '' ]]
4432023-10-04T20:48:37.916Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4442023-10-04T20:48:37.940Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4452023-10-04T20:48:37.968Z+ set +x
4462023-10-04T20:48:37.998Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4472023-10-04T20:48:38.023Z+ readarray -t ZPOOLS
4482023-10-04T20:48:38.048Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4492023-10-04T20:48:38.072Zdownloading softnpu ...done
4502023-10-04T20:48:38.097Z++ sed 's/[ ",]//g'
4512023-10-04T20:48:38.121Z+ for ZPOOL in "${ZPOOLS[@]}"
4522023-10-04T20:48:38.145Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4532023-10-04T20:48:38.170Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4542023-10-04T20:48:38.195Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4552023-10-04T20:48:38.220Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4562023-10-04T20:48:38.244Z0+0 records in
4572023-10-04T20:48:38.273Z0+0 records out
4582023-10-04T20:48:38.303Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
4592023-10-04T20:48:38.327Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4602023-10-04T20:48:38.351Z+ set +x
4612023-10-04T20:48:38.375Z++ zpool list -o name
4622023-10-04T20:48:38.399Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4632023-10-04T20:48:38.423Z+ [[ -z '' ]]
4642023-10-04T20:48:38.448Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4652023-10-04T20:48:38.479Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4662023-10-04T20:48:38.508Z+ set +x
4672023-10-04T20:48:38.533Z+ for ZPOOL in "${ZPOOLS[@]}"
4682023-10-04T20:48:38.559Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4692023-10-04T20:48:38.583Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4702023-10-04T20:48:38.607Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4712023-10-04T20:48:38.631Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4722023-10-04T20:48:38.656Z0+0 records in
4732023-10-04T20:48:38.682Z0+0 records out
4742023-10-04T20:48:38.706Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4752023-10-04T20:48:38.738Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4762023-10-04T20:48:38.838Z+ set +x
4772023-10-04T20:48:38.886Z++ zpool list -o name
4782023-10-04T20:48:38.910Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4792023-10-04T20:48:38.934Z+ [[ -z '' ]]
4802023-10-04T20:48:38.958Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4812023-10-04T20:48:38.983Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4822023-10-04T20:48:39.007Z+ set +x
4832023-10-04T20:48:39.032Z+ ensure_simulated_links igb0
4842023-10-04T20:48:39.057Z+ local PHYSICAL_LINK=igb0
4852023-10-04T20:48:39.081Z+ INDICES=("0" "1")
4862023-10-04T20:48:39.111Z+ for I in "${INDICES[@]}"
4872023-10-04T20:48:39.136Z++ get_simnet_name_if_exists net0
4882023-10-04T20:48:39.160Z++ dladm show-simnet -p -o LINK net0
4892023-10-04T20:48:39.185Z++ echo ''
4902023-10-04T20:48:39.209Z+ [[ -z '' ]]
4912023-10-04T20:48:39.233Z+ dladm create-simnet -t net0
4922023-10-04T20:48:39.257Z+ dladm create-simnet -t sc0_0
4932023-10-04T20:48:39.281Z+ dladm modify-simnet -t -p net0 sc0_0
4942023-10-04T20:48:39.305Z+ dladm set-linkprop -p mtu=1600 sc0_0
4952023-10-04T20:48:39.329Z+ success 'Simnet net0/sc0_0 exists'
4962023-10-04T20:48:39.353Z+ set +x
4972023-10-04T20:48:39.378Z+ for I in "${INDICES[@]}"
4982023-10-04T20:48:39.402Z++ get_simnet_name_if_exists net1
4992023-10-04T20:48:39.426Z++ dladm show-simnet -p -o LINK net1
5002023-10-04T20:48:39.451Z++ echo ''
5012023-10-04T20:48:39.475Z+ [[ -z '' ]]
5022023-10-04T20:48:39.499Z+ dladm create-simnet -t net1
5032023-10-04T20:48:39.523Z+ dladm create-simnet -t sc1_0
5042023-10-04T20:48:39.548Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-10-04T20:48:39.574Z+ dladm set-linkprop -p mtu=1600 sc1_0
5062023-10-04T20:48:39.598Z+ success 'Simnet net1/sc1_0 exists'
5072023-10-04T20:48:39.623Z+ set +x
5082023-10-04T20:48:39.649Z++ get_vnic_name_if_exists sc0_1
5092023-10-04T20:48:39.700Z++ dladm show-vnic -p -o LINK sc0_1
5102023-10-04T20:48:39.725Z++ echo ''
5112023-10-04T20:48:39.749Z+ [[ -z '' ]]
5122023-10-04T20:48:39.773Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5132023-10-04T20:48:39.797Z+ success 'Vnic sc0_1 exists'
5142023-10-04T20:48:39.820Z+ set +x
5152023-10-04T20:48:39.861Z+ warn_if_no_proxy_arp
5162023-10-04T20:48:39.886Z+ [[ -v PXA_START ]]
5172023-10-04T20:48:39.909Z+ [[ -v PXA_END ]]
5182023-10-04T20:48:39.934Z+ ensure_softnpu_zone
5192023-10-04T20:48:39.958Z+ zoneadm list
5202023-10-04T20:48:39.982Z+ grep -q sidecar_softnpu
5212023-10-04T20:48:40.006Z+ [[ -f out/npuzone/npuzone ]]
5222023-10-04T20:48:40.031Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0
5232023-10-04T20:48:40.743Zdownloading asic_program.so ...done
5242023-10-04T20:48:47.263Zdownloading scadm ...done
5252023-10-04T20:48:47.295Zcreating port tfportrear0_0 for sc0_0
5262023-10-04T20:48:47.319Zcreating port tfportqsfp0_0 for sc0_1
5272023-10-04T20:48:47.344Zsoftnpu zone setup
5282023-10-04T20:48:47.368Zconfigure zone
5292023-10-04T20:48:47.392Zinstall zone
5302023-10-04T20:48:48.365Zboot zone
5312023-10-04T20:48:51.473Z[sidecar_softnpu] svcs milestone/network
5322023-10-04T20:48:51.499ZSTATE STIME FMRI
5332023-10-04T20:48:51.523Zonline 20:48:49 svc:/milestone/network:default
5342023-10-04T20:48:51.555Z[sidecar_softnpu] /softnpu/npu start
5352023-10-04T20:48:51.580Z
5362023-10-04T20:48:51.610Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5372023-10-04T20:48:51.634Z+ GATEWAY_IP=10.151.2.100
5382023-10-04T20:48:51.658Z+ echo 'Using 10.151.2.100 as gateway ip'
5392023-10-04T20:48:51.682ZUsing 10.151.2.100 as gateway ip
5402023-10-04T20:48:51.706Z+ [[ ! -v GATEWAY_MAC ]]
5412023-10-04T20:48:51.730Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5422023-10-04T20:48:51.754Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5432023-10-04T20:48:51.778Z+++ echo 18:c0:4d:0d:a0:2a
5442023-10-04T20:48:51.802Z+ COUNT=1
5452023-10-04T20:48:51.826Z+ [[ 1 -ne 1 ]]
5462023-10-04T20:48:51.854Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5472023-10-04T20:48:51.879ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5482023-10-04T20:48:51.902Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5492023-10-04T20:48:51.926Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5502023-10-04T20:48:51.950Z+ PXA_MAC=a8:e1:de:01:70:1d
5512023-10-04T20:48:51.974Z+ [[ -v PXA_START ]]
5522023-10-04T20:48:51.998Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5532023-10-04T20:48:52.024Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5542023-10-04T20:48:52.051Z+ z_scadm dump-state
5552023-10-04T20:48:52.076Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5562023-10-04T20:48:52.102Zlocal v6:
5572023-10-04T20:48:52.127Zlocal v4:
5582023-10-04T20:48:52.153Zrouter v6:
5592023-10-04T20:48:52.180Zrouter v4:
5602023-10-04T20:48:52.205Zresolver v4:
5612023-10-04T20:48:52.247Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5622023-10-04T20:48:52.272Zresolver v6:
5632023-10-04T20:48:52.296Znat v4:
5642023-10-04T20:48:52.320Znat_v6:
5652023-10-04T20:48:52.345Zport_mac:
5662023-10-04T20:48:52.370Zproxy arp:
5672023-10-04T20:48:52.394Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5682023-10-04T20:48:52.418Z+ success 'softnpu zone exists'
5692023-10-04T20:48:52.442Z+ set +x
5702023-10-04T20:48:52.466Zsoftnpu zone exists
5712023-10-04T20:48:52.490Z
5722023-10-04T20:48:52.514Zreal 21.984287275
5732023-10-04T20:48:52.539Zuser 0.713520453
5742023-10-04T20:48:52.563Zsys 1.595492490
5752023-10-04T20:48:52.587Ztrap 0.000082557
5762023-10-04T20:48:52.610Ztflt 0.000152844
5772023-10-04T20:48:52.634Zdflt 0.000837722
5782023-10-04T20:48:52.658Zkflt 0.000000000
5792023-10-04T20:48:52.682Zlock 17:05.484348001
5802023-10-04T20:48:52.706Zslp 1:05.496856815
5812023-10-04T20:48:52.730Zlat 0.056900843
5822023-10-04T20:48:52.754Zstop 0.350424424
5832023-10-04T20:48:52.777Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5842023-10-04T20:48:52.801Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5852023-10-04T20:48:52.825Z+ SILO_NAME=recovery
5862023-10-04T20:48:52.849Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5872023-10-04T20:48:52.873Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5882023-10-04T20:48:52.898Z+ sed -i~ '
5892023-10-04T20:48:52.922Z /^external_dns_ips/c\
5902023-10-04T20:48:52.955Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5912023-10-04T20:48:52.979Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5922023-10-04T20:48:53.003Z /^first/c\
5932023-10-04T20:48:53.027Zfirst = "10.151.2.164"
5942023-10-04T20:48:53.051Z /^last/c\
5952023-10-04T20:48:53.076Zlast = "10.151.2.173"
5962023-10-04T20:48:53.100Z }
5972023-10-04T20:48:53.124Z /^\[rack_network_config/,/^$/ {
5982023-10-04T20:48:53.148Z /^infra_ip_first/c\
5992023-10-04T20:48:53.172Zinfra_ip_first = "10.151.2.174"
6002023-10-04T20:48:53.195Z /^infra_ip_last/c\
6012023-10-04T20:48:53.219Zinfra_ip_last = "10.151.2.174"
6022023-10-04T20:48:53.268Z }
6032023-10-04T20:48:53.292Z /^\[\[rack_network_config.uplinks/,/^$/ {
6042023-10-04T20:48:53.316Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6052023-10-04T20:48:53.340Z /^gateway_ip/c\
6062023-10-04T20:48:53.365Z+++ pkg/config-rss.toml Wed Oct 4 20:48:51 2023
6072023-10-04T20:48:53.388Zgateway_ip = "10.151.2.100"
6082023-10-04T20:48:53.412Z@@ -41,7 +41,7 @@
6092023-10-04T20:48:53.436Z /^uplink_cidr/c\
6102023-10-04T20:48:53.460Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6112023-10-04T20:48:53.484Zuplink_cidr = "10.151.2.174/32"
6122023-10-04T20:48:53.508Z # must be contained in one of the "internal services" IP Pool ranges listed
6132023-10-04T20:48:53.533Z }
6142023-10-04T20:48:53.558Z # below.
6152023-10-04T20:48:53.582Z' pkg/config-rss.toml
6162023-10-04T20:48:53.606Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6172023-10-04T20:48:53.630Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6182023-10-04T20:48:53.655Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6192023-10-04T20:48:53.680Z+ true
6202023-10-04T20:48:53.704Z
6212023-10-04T20:48:53.728Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6222023-10-04T20:48:53.752Z # Initial TLS certificates for the external API
6232023-10-04T20:48:53.777Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6242023-10-04T20:48:53.801Z #
6252023-10-04T20:48:53.825Z+ TLS_NAME=recovery.sys.oxide.test
6262023-10-04T20:48:53.849Z@@ -77,8 +77,8 @@
6272023-10-04T20:48:53.873Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6282023-10-04T20:48:53.935Z #
6292023-10-04T20:48:53.959Z..+...+................+..+...+............+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+...+............+.............+..+...+....+...+........+....+..+.........+.+........+......+.........+......+.+.....+................+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+....+...............+...+...........+.+..............+.........+...+............+......+.+.....+...............+.+...+................................+....+......+...+......+...........+............+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6302023-10-04T20:48:53.984Z # For more on this and what to put here, see docs/how-to-run.adoc.
6312023-10-04T20:48:54.010Z.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..+.........+.+.....+.+...+.....+....+..+.+..+..........+.....+...+.......+...+...+..+...+.+.....+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.........+...+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6322023-10-04T20:48:54.035Z [[internal_services_ip_pool_ranges]]
6332023-10-04T20:48:54.059Z-----
6342023-10-04T20:48:54.083Z-first = "192.168.1.20"
6352023-10-04T20:48:54.107Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6362023-10-04T20:48:54.135Z-last = "192.168.1.29"
6372023-10-04T20:48:54.159Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6382023-10-04T20:48:54.183Z+first = "10.151.2.164"
6392023-10-04T20:48:54.207Z+ rmdir pkg
6402023-10-04T20:48:54.230Z+last = "10.151.2.173"
6412023-10-04T20:48:54.254Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6422023-10-04T20:48:54.296Z
6432023-10-04T20:48:54.319Z+ pfexec mkdir -p /usr/oxide
6442023-10-04T20:48:54.343Z # TODO - this configuration is subject to change going forward. Ultimately these
6452023-10-04T20:48:54.367Z+ pfexec rm -f /usr/oxide/tcpproxy
6462023-10-04T20:48:54.391Z # parameters should be provided to the control plane via wicket, but we need to
6472023-10-04T20:48:54.416Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6482023-10-04T20:48:54.439Z@@ -91,13 +91,13 @@
6492023-10-04T20:48:54.463Z+ pfexec chmod +x /usr/oxide/tcpproxy
6502023-10-04T20:48:54.487Z # A range of IP addresses used by Boundary Services on the external network. In
6512023-10-04T20:48:54.511Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6522023-10-04T20:48:54.535Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6532023-10-04T20:48:54.559Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6542023-10-04T20:48:54.583Z # With softnpu, only one address is used.
6552023-10-04T20:48:54.607Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6562023-10-04T20:48:54.631Z-infra_ip_first = "192.168.1.30"
6572023-10-04T20:48:54.680Z+ OMICRON_NO_UNINSTALL=1
6582023-10-04T20:48:54.704Z-infra_ip_last = "192.168.1.30"
6592023-10-04T20:48:54.728Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6602023-10-04T20:48:54.752Z+infra_ip_first = "10.151.2.174"
6612023-10-04T20:48:54.776ZOct 04 20:48:52.552 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6622023-10-04T20:48:54.800Z+infra_ip_last = "10.151.2.174"
6632023-10-04T20:48:54.825ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6642023-10-04T20:48:54.848Z
6652023-10-04T20:48:54.872ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6662023-10-04T20:48:54.896Z # You can configure multiple uplinks by repeating the following stanza
6672023-10-04T20:48:54.920ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6682023-10-04T20:48:54.944Z [[rack_network_config.uplinks]]
6692023-10-04T20:48:54.968ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6702023-10-04T20:48:54.992Z # The gateway IP for the rack's external network
6712023-10-04T20:48:55.016ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6722023-10-04T20:48:55.041Z-gateway_ip = "192.168.1.199"
6732023-10-04T20:48:55.066ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6742023-10-04T20:48:55.091Z+gateway_ip = "10.151.2.100"
6752023-10-04T20:48:55.115ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6762023-10-04T20:48:55.141Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6772023-10-04T20:48:55.165ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6782023-10-04T20:48:55.189Z uplink_port = "qsfp0"
6792023-10-04T20:48:55.213ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/maghemite.tar
6802023-10-04T20:48:55.237Z uplink_port_speed = "40G"
6812023-10-04T20:48:55.264ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6822023-10-04T20:48:55.288Z@@ -104,7 +104,7 @@
6832023-10-04T20:48:55.320ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6842023-10-04T20:48:55.344Z uplink_port_fec="none"
6852023-10-04T20:48:55.368ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6862023-10-04T20:48:55.392Z # For softnpu, an address within the "infra" block above that will be used for
6872023-10-04T20:48:55.416ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6882023-10-04T20:48:55.440Z # the softnpu uplink port. You can just pick the first address in that pool.
6892023-10-04T20:48:55.463ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6902023-10-04T20:48:55.487Z-uplink_cidr = "192.168.1.30/32"
6912023-10-04T20:48:55.511ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6922023-10-04T20:48:55.535Z+uplink_cidr = "10.151.2.174/32"
6932023-10-04T20:48:55.559ZOct 04 20:48:52.554 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6942023-10-04T20:48:55.582Z # Switch to use for the uplink. For single-rack deployments this can be
6952023-10-04T20:48:55.607ZOct 04 20:48:52.810 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
6962023-10-04T20:48:55.631Z # "switch0" (upper slot) or "switch1" (lower slot). For single-node softnpu
6972023-10-04T20:48:55.655ZOct 04 20:48:52.854 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
6982023-10-04T20:48:55.679Z # and dendrite stub environments, use "switch0"
6992023-10-04T20:48:55.703ZOct 04 20:48:52.918 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7002023-10-04T20:48:55.727Za pkg/config-rss.toml 7K
7012023-10-04T20:48:55.750Z
7022023-10-04T20:48:55.774Za pkg/initial-tls-cert.pem 2K
7032023-10-04T20:48:55.798Zreal 0.505857745
7042023-10-04T20:48:55.822Za pkg/initial-tls-key.pem 4K
7052023-10-04T20:48:55.846Zuser 0.097628660
7062023-10-04T20:48:55.870Zsys 1.988130515
7072023-10-04T20:48:55.894Ztrap 0.000010267
7082023-10-04T20:48:55.918Ztflt 0.000494419
7092023-10-04T20:48:55.943Zdflt 0.000450515
7102023-10-04T20:48:55.967Zkflt 0.000003430
7112023-10-04T20:48:55.991Zlock 45.482291017
7122023-10-04T20:48:56.015Zslp 0.708406693
7132023-10-04T20:48:56.046Zlat 0.002927141
7142023-10-04T20:48:56.070Zstop 0.196198581
7152023-10-04T20:48:56.094Z+ retry=0
7162023-10-04T20:48:56.118Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7172023-10-04T20:48:56.144Z+ [[ 0 -gt 30 ]]
7182023-10-04T20:48:56.168Z+ sleep 1
7192023-10-04T20:48:56.193Z+ retry=1
7202023-10-04T20:48:56.216Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7212023-10-04T20:48:56.240Z+ [[ 1 -gt 30 ]]
7222023-10-04T20:48:56.264Z+ sleep 1
7232023-10-04T20:48:56.290Z+ retry=2
7242023-10-04T20:48:56.355Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7252023-10-04T20:49:14.742Z+ [[ 2 -gt 30 ]]
7262023-10-04T20:49:14.766Z+ sleep 1
7272023-10-04T20:49:15.746Z+ retry=3
7282023-10-04T20:49:15.770Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7292023-10-04T20:49:15.794Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7302023-10-04T20:49:15.818Zlocal v6:
7312023-10-04T20:49:15.842Zfd00:99::1
7322023-10-04T20:49:15.866Zfe80::b40e:bcff:feeb:ca92
7332023-10-04T20:49:15.890Zlocal v4:
7342023-10-04T20:49:15.914Z10.151.2.174
7352023-10-04T20:49:15.938Zrouter v6:
7362023-10-04T20:49:15.971Zrouter v4:
7372023-10-04T20:49:15.995Z0.0.0.0/0 -> 10.151.2.100 (2) vid=0
7382023-10-04T20:49:16.045Zresolver v4:
7392023-10-04T20:49:16.069Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7402023-10-04T20:49:16.093Zresolver v6:
7412023-10-04T20:49:16.116Zfe80::b40e:bcff:feeb:ca92 -> b6:0e:bc:eb:ca:92
7422023-10-04T20:49:16.140Zfe80::3808:ecff:fe20:efbb -> 3a:08:ec:20:ef:bb
7432023-10-04T20:49:16.171Znat v4:
7442023-10-04T20:49:16.195Znat_v6:
7452023-10-04T20:49:16.220Zport_mac:
7462023-10-04T20:49:16.244Z1000: a8:40:25:58:45:c5
7472023-10-04T20:49:16.268Z1: a8:40:25:58:45:c6
7482023-10-04T20:49:16.292Zproxy arp:
7492023-10-04T20:49:16.316Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7502023-10-04T20:49:16.340Z+ retry=0
7512023-10-04T20:49:16.364Z+++ zoneadm list -n
7522023-10-04T20:49:16.388Z+++ grep oxz_ntp
7532023-10-04T20:49:16.412Z++ pfexec svcs -z -Hostate oxide/ntp
7542023-10-04T20:49:16.436Zsvcs: zone '-Hostate': entity not found
7552023-10-04T20:49:16.460Z++ true
7562023-10-04T20:49:16.484Z+ [[ '' != online ]]
7572023-10-04T20:49:16.508Z+ [[ 0 -gt 60 ]]
7582023-10-04T20:49:16.532Z+ sleep 1
7592023-10-04T20:49:16.807Z+ retry=1
7602023-10-04T20:49:16.831Z+++ zoneadm list -n
7612023-10-04T20:49:16.856Z+++ grep oxz_ntp
7622023-10-04T20:49:16.880Z++ pfexec svcs -z -Hostate oxide/ntp
7632023-10-04T20:49:16.904Zsvcs: zone '-Hostate': entity not found
7642023-10-04T20:49:16.928Z++ true
7652023-10-04T20:49:16.952Z+ [[ '' != online ]]
7662023-10-04T20:49:16.976Z+ [[ 1 -gt 60 ]]
7672023-10-04T20:49:17.006Z+ sleep 1
7682023-10-04T20:49:17.830Z+ retry=2
7692023-10-04T20:49:17.854Z+++ zoneadm list -n
7702023-10-04T20:49:17.878Z+++ grep oxz_ntp
7712023-10-04T20:49:17.902Z++ pfexec svcs -z -Hostate oxide/ntp
7722023-10-04T20:49:17.926Zsvcs: zone '-Hostate': entity not found
7732023-10-04T20:49:17.950Z++ true
7742023-10-04T20:49:17.974Z+ [[ '' != online ]]
7752023-10-04T20:49:17.998Z+ [[ 2 -gt 60 ]]
7762023-10-04T20:49:18.022Z+ sleep 1
7772023-10-04T20:49:18.851Z+ retry=3
7782023-10-04T20:49:18.875Z+++ zoneadm list -n
7792023-10-04T20:49:18.899Z+++ grep oxz_ntp
7802023-10-04T20:49:18.924Z++ pfexec svcs -z -Hostate oxide/ntp
7812023-10-04T20:49:18.948Zsvcs: zone '-Hostate': entity not found
7822023-10-04T20:49:18.972Z++ true
7832023-10-04T20:49:18.996Z+ [[ '' != online ]]
7842023-10-04T20:49:19.019Z+ [[ 3 -gt 60 ]]
7852023-10-04T20:49:19.043Z+ sleep 1
7862023-10-04T20:49:19.873Z+ retry=4
7872023-10-04T20:49:19.897Z+++ zoneadm list -n
7882023-10-04T20:49:19.922Z+++ grep oxz_ntp
7892023-10-04T20:49:19.946Z++ pfexec svcs -z -Hostate oxide/ntp
7902023-10-04T20:49:19.970Zsvcs: zone '-Hostate': entity not found
7912023-10-04T20:49:19.993Z++ true
7922023-10-04T20:49:20.017Z+ [[ '' != online ]]
7932023-10-04T20:49:20.041Z+ [[ 4 -gt 60 ]]
7942023-10-04T20:49:20.066Z+ sleep 1
7952023-10-04T20:49:20.900Z+ retry=5
7962023-10-04T20:49:20.924Z+++ zoneadm list -n
7972023-10-04T20:49:20.948Z+++ grep oxz_ntp
7982023-10-04T20:49:20.972Z++ pfexec svcs -z -Hostate oxide/ntp
7992023-10-04T20:49:20.997Zsvcs: zone '-Hostate': entity not found
8002023-10-04T20:49:21.021Z++ true
8012023-10-04T20:49:21.045Z+ [[ '' != online ]]
8022023-10-04T20:49:21.069Z+ [[ 5 -gt 60 ]]
8032023-10-04T20:49:21.093Z+ sleep 1
8042023-10-04T20:49:21.920Z+ retry=6
8052023-10-04T20:49:21.943Z+++ zoneadm list -n
8062023-10-04T20:49:21.967Z+++ grep oxz_ntp
8072023-10-04T20:49:21.991Z++ pfexec svcs -z -Hostate oxide/ntp
8082023-10-04T20:49:22.015Zsvcs: zone '-Hostate': entity not found
8092023-10-04T20:49:22.039Z++ true
8102023-10-04T20:49:22.063Z+ [[ '' != online ]]
8112023-10-04T20:49:22.086Z+ [[ 6 -gt 60 ]]
8122023-10-04T20:49:22.110Z+ sleep 1
8132023-10-04T20:49:22.940Z+ retry=7
8142023-10-04T20:49:22.964Z+++ zoneadm list -n
8152023-10-04T20:49:22.987Z+++ grep oxz_ntp
8162023-10-04T20:49:23.012Z++ pfexec svcs -z -Hostate oxide/ntp
8172023-10-04T20:49:23.036Zsvcs: zone '-Hostate': entity not found
8182023-10-04T20:49:23.060Z++ true
8192023-10-04T20:49:23.085Z+ [[ '' != online ]]
8202023-10-04T20:49:23.109Z+ [[ 7 -gt 60 ]]
8212023-10-04T20:49:23.133Z+ sleep 1
8222023-10-04T20:49:23.962Z+ retry=8
8232023-10-04T20:49:23.986Z+++ zoneadm list -n
8242023-10-04T20:49:24.033Z+++ grep oxz_ntp
8252023-10-04T20:49:24.058Z++ pfexec svcs -z -Hostate oxide/ntp
8262023-10-04T20:49:24.082Zsvcs: zone '-Hostate': entity not found
8272023-10-04T20:49:24.105Z++ true
8282023-10-04T20:49:24.129Z+ [[ '' != online ]]
8292023-10-04T20:49:24.155Z+ [[ 8 -gt 60 ]]
8302023-10-04T20:49:24.179Z+ sleep 1
8312023-10-04T20:49:24.983Z+ retry=9
8322023-10-04T20:49:25.008Z+++ zoneadm list -n
8332023-10-04T20:49:25.033Z+++ grep oxz_ntp
8342023-10-04T20:49:25.057Z++ pfexec svcs -z -Hostate oxide/ntp
8352023-10-04T20:49:25.081Zsvcs: zone '-Hostate': entity not found
8362023-10-04T20:49:25.105Z++ true
8372023-10-04T20:49:25.129Z+ [[ '' != online ]]
8382023-10-04T20:49:25.153Z+ [[ 9 -gt 60 ]]
8392023-10-04T20:49:25.177Z+ sleep 1
8402023-10-04T20:49:26.005Z+ retry=10
8412023-10-04T20:49:26.029Z+++ zoneadm list -n
8422023-10-04T20:49:26.055Z+++ grep oxz_ntp
8432023-10-04T20:49:26.079Z++ pfexec svcs -z -Hostate oxide/ntp
8442023-10-04T20:49:26.103Zsvcs: zone '-Hostate': entity not found
8452023-10-04T20:49:26.128Z++ true
8462023-10-04T20:49:26.160Z+ [[ '' != online ]]
8472023-10-04T20:49:26.185Z+ [[ 10 -gt 60 ]]
8482023-10-04T20:49:26.209Z+ sleep 1
8492023-10-04T20:49:27.026Z+ retry=11
8502023-10-04T20:49:27.051Z+++ zoneadm list -n
8512023-10-04T20:49:27.075Z+++ grep oxz_ntp
8522023-10-04T20:49:27.099Z++ pfexec svcs -z -Hostate oxide/ntp
8532023-10-04T20:49:27.123Zsvcs: zone '-Hostate': entity not found
8542023-10-04T20:49:27.147Z++ true
8552023-10-04T20:49:27.172Z+ [[ '' != online ]]
8562023-10-04T20:49:27.196Z+ [[ 11 -gt 60 ]]
8572023-10-04T20:49:27.220Z+ sleep 1
8582023-10-04T20:49:28.047Z+ retry=12
8592023-10-04T20:49:28.072Z+++ zoneadm list -n
8602023-10-04T20:49:28.096Z+++ grep oxz_ntp
8612023-10-04T20:49:28.120Z++ pfexec svcs -z -Hostate oxide/ntp
8622023-10-04T20:49:28.144Zsvcs: zone '-Hostate': entity not found
8632023-10-04T20:49:28.168Z++ true
8642023-10-04T20:49:28.191Z+ [[ '' != online ]]
8652023-10-04T20:49:28.215Z+ [[ 12 -gt 60 ]]
8662023-10-04T20:49:28.239Z+ sleep 1
8672023-10-04T20:49:29.069Z+ retry=13
8682023-10-04T20:49:29.093Z+++ zoneadm list -n
8692023-10-04T20:49:29.117Z+++ grep oxz_ntp
8702023-10-04T20:49:29.141Z++ pfexec svcs -z -Hostate oxide/ntp
8712023-10-04T20:49:29.165Zsvcs: zone '-Hostate': entity not found
8722023-10-04T20:49:29.189Z++ true
8732023-10-04T20:49:29.213Z+ [[ '' != online ]]
8742023-10-04T20:49:29.238Z+ [[ 13 -gt 60 ]]
8752023-10-04T20:49:29.268Z+ sleep 1
8762023-10-04T20:49:30.090Z+ retry=14
8772023-10-04T20:49:30.114Z+++ zoneadm list -n
8782023-10-04T20:49:30.138Z+++ grep oxz_ntp
8792023-10-04T20:49:30.163Z++ pfexec svcs -z -Hostate oxide/ntp
8802023-10-04T20:49:30.186Zsvcs: zone '-Hostate': entity not found
8812023-10-04T20:49:30.210Z++ true
8822023-10-04T20:49:30.235Z+ [[ '' != online ]]
8832023-10-04T20:49:30.258Z+ [[ 14 -gt 60 ]]
8842023-10-04T20:49:30.282Z+ sleep 1
8852023-10-04T20:49:31.113Z+ retry=15
8862023-10-04T20:49:31.137Z+++ zoneadm list -n
8872023-10-04T20:49:31.161Z+++ grep oxz_ntp
8882023-10-04T20:49:31.185Z++ pfexec svcs -z oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76 -Hostate oxide/ntp
8892023-10-04T20:49:31.209Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8902023-10-04T20:49:31.233Z++ true
8912023-10-04T20:49:31.257Z+ [[ '' != online ]]
8922023-10-04T20:49:31.281Z+ [[ 15 -gt 60 ]]
8932023-10-04T20:49:31.304Z+ sleep 1
8942023-10-04T20:49:32.146Z+ retry=16
8952023-10-04T20:49:32.171Z+++ zoneadm list -n
8962023-10-04T20:49:32.195Z+++ grep oxz_ntp
8972023-10-04T20:49:32.219Z++ pfexec svcs -z oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76 -Hostate oxide/ntp
8982023-10-04T20:49:32.244Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8992023-10-04T20:49:32.268Z++ true
9002023-10-04T20:49:32.292Z+ [[ '' != online ]]
9012023-10-04T20:49:32.317Z+ [[ 16 -gt 60 ]]
9022023-10-04T20:49:32.341Z+ sleep 1
9032023-10-04T20:49:33.176Z+ retry=17
9042023-10-04T20:49:33.200Z+++ zoneadm list -n
9052023-10-04T20:49:33.224Z+++ grep oxz_ntp
9062023-10-04T20:49:33.248Z++ pfexec svcs -z oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76 -Hostate oxide/ntp
9072023-10-04T20:49:33.272Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9082023-10-04T20:49:33.296Z++ true
9092023-10-04T20:49:33.319Z+ [[ '' != online ]]
9102023-10-04T20:49:33.343Z+ [[ 17 -gt 60 ]]
9112023-10-04T20:49:33.368Z+ sleep 1
9122023-10-04T20:49:34.205Z+ retry=18
9132023-10-04T20:49:34.228Z+++ zoneadm list -n
9142023-10-04T20:49:34.252Z+++ grep oxz_ntp
9152023-10-04T20:49:34.276Z++ pfexec svcs -z oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76 -Hostate oxide/ntp
9162023-10-04T20:49:34.301Z+ [[ disabled != online ]]
9172023-10-04T20:49:34.325Z+ [[ 18 -gt 60 ]]
9182023-10-04T20:49:34.349Z+ sleep 1
9192023-10-04T20:49:35.227Z+ retry=19
9202023-10-04T20:49:35.251Z+++ zoneadm list -n
9212023-10-04T20:49:35.276Z+++ grep oxz_ntp
9222023-10-04T20:49:35.299Z++ pfexec svcs -z oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76 -Hostate oxide/ntp
9232023-10-04T20:49:35.323Z+ [[ disabled != online ]]
9242023-10-04T20:49:35.347Z+ [[ 19 -gt 60 ]]
9252023-10-04T20:49:35.372Z+ sleep 1
9262023-10-04T20:49:36.248Z+ retry=20
9272023-10-04T20:49:36.273Z+++ zoneadm list -n
9282023-10-04T20:49:36.297Z+++ grep oxz_ntp
9292023-10-04T20:49:36.321Z++ pfexec svcs -z oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76 -Hostate oxide/ntp
9302023-10-04T20:49:36.345Z+ [[ disabled != online ]]
9312023-10-04T20:49:36.369Z+ [[ 20 -gt 60 ]]
9322023-10-04T20:49:36.394Z+ sleep 1
9332023-10-04T20:49:37.268Z+ retry=21
9342023-10-04T20:49:37.294Z+++ zoneadm list -n
9352023-10-04T20:49:37.319Z+++ grep oxz_ntp
9362023-10-04T20:49:37.343Z++ pfexec svcs -z oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76 -Hostate oxide/ntp
9372023-10-04T20:49:37.367Z+ [[ disabled != online ]]
9382023-10-04T20:49:37.393Z+ [[ 21 -gt 60 ]]
9392023-10-04T20:49:37.417Z+ sleep 1
9402023-10-04T20:49:38.291Z+ retry=22
9412023-10-04T20:49:38.316Z+++ zoneadm list -n
9422023-10-04T20:49:38.340Z+++ grep oxz_ntp
9432023-10-04T20:49:38.364Z++ pfexec svcs -z oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76 -Hostate oxide/ntp
9442023-10-04T20:49:38.388Z+ [[ disabled != online ]]
9452023-10-04T20:49:38.412Z+ [[ 22 -gt 60 ]]
9462023-10-04T20:49:38.436Z+ sleep 1
9472023-10-04T20:49:39.312Z+ retry=23
9482023-10-04T20:49:39.337Z+++ zoneadm list -n
9492023-10-04T20:49:39.361Z+++ grep oxz_ntp
9502023-10-04T20:49:39.386Z++ pfexec svcs -z oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76 -Hostate oxide/ntp
9512023-10-04T20:49:39.410Z+ [[ disabled != online ]]
9522023-10-04T20:49:39.435Z+ [[ 23 -gt 60 ]]
9532023-10-04T20:49:39.459Z+ sleep 1
9542023-10-04T20:49:40.333Z+ retry=24
9552023-10-04T20:49:40.358Z+++ zoneadm list -n
9562023-10-04T20:49:40.382Z+++ grep oxz_ntp
9572023-10-04T20:49:40.406Z++ pfexec svcs -z oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76 -Hostate oxide/ntp
9582023-10-04T20:49:40.429Z+ [[ disabled != online ]]
9592023-10-04T20:49:40.453Z+ [[ 24 -gt 60 ]]
9602023-10-04T20:49:40.478Z+ sleep 1
9612023-10-04T20:49:41.355Z+ retry=25
9622023-10-04T20:49:41.379Z+++ zoneadm list -n
9632023-10-04T20:49:41.404Z+++ grep oxz_ntp
9642023-10-04T20:49:41.429Z++ pfexec svcs -z oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76 -Hostate oxide/ntp
9652023-10-04T20:49:41.453Z+ [[ disabled != online ]]
9662023-10-04T20:49:41.478Z+ [[ 25 -gt 60 ]]
9672023-10-04T20:49:41.503Z+ sleep 1
 
9682023-10-04T20:49:41.950Zsock console login: Oct 4 20:49:41 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9692023-10-04T20:49:42.376Z+ retry=26
9702023-10-04T20:49:42.401Z+++ zoneadm list -n
9712023-10-04T20:49:42.427Z+++ grep oxz_ntp
9722023-10-04T20:49:42.452Z++ pfexec svcs -z oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76 -Hostate oxide/ntp
9732023-10-04T20:49:42.477Z+ [[ online != online ]]
9742023-10-04T20:49:42.502Z+ echo 'Waited for chrony: 26s'
9752023-10-04T20:49:42.528ZWaited for chrony: 26s
9762023-10-04T20:49:42.552Z+ export RUST_BACKTRACE=1
9772023-10-04T20:49:42.577Z+ RUST_BACKTRACE=1
9782023-10-04T20:49:42.601Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9792023-10-04T20:49:42.626Z+ ./tests/bootstrap
9802023-10-04T20:49:42.650ZUsing external DNS server at 10.151.2.164:53
9812023-10-04T20:49:42.676Z2023-10-04 20:49:42.440660009 UTC: attempting to log into API
9822023-10-04T20:49:57.464Z2023-10-04 20:49:57.456113828 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9832023-10-04T20:49:58.467Z2023-10-04 20:49:58.457109787 UTC: attempting to log into API
9842023-10-04T20:50:13.498Z2023-10-04 20:50:13.488595253 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9852023-10-04T20:50:14.499Z2023-10-04 20:50:14.489564632 UTC: attempting to log into API
9862023-10-04T20:50:29.533Z2023-10-04 20:50:29.522419645 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9872023-10-04T20:50:30.534Z2023-10-04 20:50:30.523032047 UTC: attempting to log into API
9882023-10-04T20:50:45.566Z2023-10-04 20:50:45.555044993 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9892023-10-04T20:50:46.568Z2023-10-04 20:50:46.556018952 UTC: attempting to log into API
 
9902023-10-04T20:50:47.756ZOct 4 20:50:47 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9912023-10-04T20:50:54.649ZOct 4 20:50:53 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9922023-10-04T20:51:01.429ZOct 4 20:50:59 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9932023-10-04T20:51:01.602Z2023-10-04 20:51:01.593164089 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9942023-10-04T20:51:02.605Z2023-10-04 20:51:02.594619364 UTC: attempting to log into API
 
9952023-10-04T20:51:05.758ZOct 4 20:51:05 sok ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9962023-10-04T20:51:12.006ZOct 4 20:51:11 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9972023-10-04T20:51:17.636Z2023-10-04 20:51:17.628548087 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9982023-10-04T20:51:18.637Z2023-10-04 20:51:18.629581195 UTC: attempting to log into API
9992023-10-04T20:51:33.670Z2023-10-04 20:51:33.662488398 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10002023-10-04T20:51:35.091Z2023-10-04 20:51:34.663475506 UTC: attempting to log into API
10012023-10-04T20:51:45.446Z2023-10-04 20:51:45.438508585 UTC: login succeeded
10022023-10-04T20:51:45.480Zwaiting for nexus to come up...
10032023-10-04T20:51:45.549Zcreating IP pool... 10.151.2.175 - 10.151.2.227
10042023-10-04T20:51:45.672Zensuring datasets are ready...
10052023-10-04T20:51:50.581Zlet's roll.
10062023-10-04T20:51:50.611Z+ rm ./tests/bootstrap
10072023-10-04T20:51:50.641Z+ for test_bin in tests/*
10082023-10-04T20:51:50.669Z+ ./tests/bootstrap-61cfb22b53ec0472
10092023-10-04T20:51:50.694Z
10102023-10-04T20:51:50.720Zrunning 0 tests
10112023-10-04T20:51:50.745Z
10122023-10-04T20:51:50.770Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10132023-10-04T20:51:50.795Z
10142023-10-04T20:51:50.865Z+ for test_bin in tests/*
10152023-10-04T20:51:50.904Z+ ./tests/end_to_end_tests-cfdf15182b8b4dac
10162023-10-04T20:51:50.928Z
10172023-10-04T20:51:50.958Zrunning 2 tests
10182023-10-04T20:51:51.001Ztest no_spoof::no_spoof_here ... ok
10192023-10-04T20:52:50.248Ztest instance_launch::instance_launch has been running for over 60 seconds
10202023-10-04T20:55:16.121Ztest instance_launch::instance_launch ... ok
10212023-10-04T20:55:16.158Z
10222023-10-04T20:55:16.216Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 205.46s
10232023-10-04T20:55:16.261Z
10242023-10-04T20:55:16.295Z+ _exit_trap
10252023-10-04T20:55:16.342Z+ local status=0
10262023-10-04T20:55:16.380Z+ [[ 0 -eq 0 ]]
10272023-10-04T20:55:16.469Z+ exit 0
10282023-10-04T20:55:16.508Zprocess exited: duration 414375 ms, exit code 0
 
10292023-10-04T20:55:16.583Zfound 47 output files
10302023-10-04T20:55:16.644Zuploading: /var/svc/log/oxide-sled-agent:default.log (427298 bytes)
10312023-10-04T20:55:18.035Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10322023-10-04T20:55:18.084Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_265bc3f8-992e-4422-bc98-92f30f98052a/root/var/svc/log/oxide-nexus:default.log (4731695 bytes)
10332023-10-04T20:55:21.659Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_265bc3f8-992e-4422-bc98-92f30f98052a/root/var/svc/log/oxide-nexus:default.log
10342023-10-04T20:55:21.698Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_122f71d7-070f-4351-8837-e6cb92bf24da/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
10352023-10-04T20:55:23.165Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_122f71d7-070f-4351-8837-e6cb92bf24da/root/var/svc/log/oxide-crucible-agent:default.log
10362023-10-04T20:55:23.200Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_122f71d7-070f-4351-8837-e6cb92bf24da/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10372023-10-04T20:55:24.274Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_122f71d7-070f-4351-8837-e6cb92bf24da/root/var/svc/log/oxide-crucible-downstairs:default.log
10382023-10-04T20:55:24.300Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1705a4e7-ff65-4d75-b2fe-5b5b5c7fc751/root/var/svc/log/oxide-crucible-agent:default.log (14758 bytes)
10392023-10-04T20:55:25.919Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1705a4e7-ff65-4d75-b2fe-5b5b5c7fc751/root/var/svc/log/oxide-crucible-agent:default.log
10402023-10-04T20:55:25.962Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1705a4e7-ff65-4d75-b2fe-5b5b5c7fc751/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10412023-10-04T20:55:27.040Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1705a4e7-ff65-4d75-b2fe-5b5b5c7fc751/root/var/svc/log/oxide-crucible-downstairs:default.log
10422023-10-04T20:55:27.067Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1705a4e7-ff65-4d75-b2fe-5b5b5c7fc751/root/var/svc/log/oxide-crucible-downstairs:downstairs-c6c88f56-c5f9-4758-916f-6784fd43a216.log (3494 bytes)
10432023-10-04T20:55:28.187Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1705a4e7-ff65-4d75-b2fe-5b5b5c7fc751/root/var/svc/log/oxide-crucible-downstairs:downstairs-c6c88f56-c5f9-4758-916f-6784fd43a216.log
10442023-10-04T20:55:28.226Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3eb7f097-a7a4-4ba9-be37-d71be526caa8/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10452023-10-04T20:55:29.315Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3eb7f097-a7a4-4ba9-be37-d71be526caa8/root/var/svc/log/oxide-crucible-agent:default.log
10462023-10-04T20:55:29.339Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3eb7f097-a7a4-4ba9-be37-d71be526caa8/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10472023-10-04T20:55:30.414Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3eb7f097-a7a4-4ba9-be37-d71be526caa8/root/var/svc/log/oxide-crucible-downstairs:default.log
10482023-10-04T20:55:30.439Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54ff59e7-bc43-4bba-894f-51650442e3d1/root/var/svc/log/oxide-crucible-agent:default.log (26171 bytes)
10492023-10-04T20:55:31.706Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54ff59e7-bc43-4bba-894f-51650442e3d1/root/var/svc/log/oxide-crucible-agent:default.log
10502023-10-04T20:55:31.730Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54ff59e7-bc43-4bba-894f-51650442e3d1/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10512023-10-04T20:55:32.804Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54ff59e7-bc43-4bba-894f-51650442e3d1/root/var/svc/log/oxide-crucible-downstairs:default.log
10522023-10-04T20:55:33.240Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54ff59e7-bc43-4bba-894f-51650442e3d1/root/var/svc/log/oxide-crucible-downstairs:downstairs-0fe81705-8299-45e9-a9f1-8b24efb50091.log (3491 bytes)
10532023-10-04T20:55:33.916Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54ff59e7-bc43-4bba-894f-51650442e3d1/root/var/svc/log/oxide-crucible-downstairs:downstairs-0fe81705-8299-45e9-a9f1-8b24efb50091.log
10542023-10-04T20:55:33.941Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54ff59e7-bc43-4bba-894f-51650442e3d1/root/var/svc/log/oxide-crucible-downstairs:downstairs-e05556e3-a733-42b9-bc5f-e9c607437070.log (5965 bytes)
10552023-10-04T20:55:35.459Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54ff59e7-bc43-4bba-894f-51650442e3d1/root/var/svc/log/oxide-crucible-downstairs:downstairs-e05556e3-a733-42b9-bc5f-e9c607437070.log
10562023-10-04T20:55:35.492Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b6fd05f7-0522-40f2-aa67-14856d455734/root/var/svc/log/oxide-crucible-pantry:default.log (2337 bytes)
10572023-10-04T20:55:36.564Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b6fd05f7-0522-40f2-aa67-14856d455734/root/var/svc/log/oxide-crucible-pantry:default.log
10582023-10-04T20:55:36.589Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_638b687b-a76d-4e44-886e-0184b1dd28ed/root/var/svc/log/oxide-internal_dns:default.log (105668 bytes)
10592023-10-04T20:55:37.790Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_638b687b-a76d-4e44-886e-0184b1dd28ed/root/var/svc/log/oxide-internal_dns:default.log
10602023-10-04T20:55:37.815Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_3b3f8afa-95ea-451b-83d4-d013bb5fe1b1/root/var/svc/log/oxide-crucible-agent:default.log (14760 bytes)
10612023-10-04T20:55:38.913Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_3b3f8afa-95ea-451b-83d4-d013bb5fe1b1/root/var/svc/log/oxide-crucible-agent:default.log
10622023-10-04T20:55:38.939Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_3b3f8afa-95ea-451b-83d4-d013bb5fe1b1/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10632023-10-04T20:55:40.041Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_3b3f8afa-95ea-451b-83d4-d013bb5fe1b1/root/var/svc/log/oxide-crucible-downstairs:default.log
10642023-10-04T20:55:40.069Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_3b3f8afa-95ea-451b-83d4-d013bb5fe1b1/root/var/svc/log/oxide-crucible-downstairs:downstairs-84d24578-c3c4-4c90-b453-2e3007221770.log (5961 bytes)
10652023-10-04T20:55:41.144Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_3b3f8afa-95ea-451b-83d4-d013bb5fe1b1/root/var/svc/log/oxide-crucible-downstairs:downstairs-84d24578-c3c4-4c90-b453-2e3007221770.log
10662023-10-04T20:55:41.168Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fe0f8fce-c069-416d-b0d2-7fc06c6f9859/root/var/svc/log/oxide-crucible-agent:default.log (14762 bytes)
10672023-10-04T20:55:42.297Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fe0f8fce-c069-416d-b0d2-7fc06c6f9859/root/var/svc/log/oxide-crucible-agent:default.log
10682023-10-04T20:55:42.331Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fe0f8fce-c069-416d-b0d2-7fc06c6f9859/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10692023-10-04T20:55:43.411Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fe0f8fce-c069-416d-b0d2-7fc06c6f9859/root/var/svc/log/oxide-crucible-downstairs:default.log
10702023-10-04T20:55:43.450Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fe0f8fce-c069-416d-b0d2-7fc06c6f9859/root/var/svc/log/oxide-crucible-downstairs:downstairs-e90d8fc5-64d9-4d6f-a2c3-5bac51f92d2c.log (3494 bytes)
10712023-10-04T20:55:44.528Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fe0f8fce-c069-416d-b0d2-7fc06c6f9859/root/var/svc/log/oxide-crucible-downstairs:downstairs-e90d8fc5-64d9-4d6f-a2c3-5bac51f92d2c.log
10722023-10-04T20:55:44.553Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_2c5dc049-1e23-4b48-9076-f51088e6144c/root/var/svc/log/oxide-internal_dns:default.log (107697 bytes)
10732023-10-04T20:55:45.931Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_2c5dc049-1e23-4b48-9076-f51088e6144c/root/var/svc/log/oxide-internal_dns:default.log
10742023-10-04T20:55:45.962Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_be2700f7-37e2-4bba-bc7d-47657fce14d2/root/var/svc/log/oxide-clickhouse:default.log (549073 bytes)
10752023-10-04T20:55:47.316Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_be2700f7-37e2-4bba-bc7d-47657fce14d2/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 549073 -> 598749
10762023-10-04T20:55:48.009Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_be2700f7-37e2-4bba-bc7d-47657fce14d2/root/var/svc/log/oxide-clickhouse:default.log
10772023-10-04T20:55:48.045Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_68e09f4f-3593-4797-aa3b-c07768def21f/root/var/svc/log/oxide-external_dns:default.log (19386 bytes)
10782023-10-04T20:55:49.178Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_68e09f4f-3593-4797-aa3b-c07768def21f/root/var/svc/log/oxide-external_dns:default.log
10792023-10-04T20:55:49.203Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76/root/var/svc/log/oxide-ntp:default.log (780 bytes)
10802023-10-04T20:55:50.292Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76/root/var/svc/log/oxide-ntp:default.log
10812023-10-04T20:55:50.356Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_ebc1c715-1c7e-47da-8aec-cd96b9b84c2f/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10822023-10-04T20:55:51.430Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_ebc1c715-1c7e-47da-8aec-cd96b9b84c2f/root/var/svc/log/oxide-cockroachdb:default.log
10832023-10-04T20:55:51.455Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_364fe51b-0a65-4276-b413-98ffe07a49c6/root/var/svc/log/oxide-nexus:default.log (506402 bytes)
10842023-10-04T20:55:52.198Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_364fe51b-0a65-4276-b413-98ffe07a49c6/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 506402 -> 508226
10852023-10-04T20:55:53.270Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_364fe51b-0a65-4276-b413-98ffe07a49c6/root/var/svc/log/oxide-nexus:default.log
10862023-10-04T20:55:53.295Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_42389fdb-df83-4cff-b3e8-4d411838aa71/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10872023-10-04T20:55:54.373Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_42389fdb-df83-4cff-b3e8-4d411838aa71/root/var/svc/log/oxide-cockroachdb:default.log
10882023-10-04T20:55:54.398Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0a7266c5-6f2c-4987-a84e-401bf4366e32/root/var/svc/log/oxide-crucible-agent:default.log (14299 bytes)
10892023-10-04T20:55:55.491Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0a7266c5-6f2c-4987-a84e-401bf4366e32/root/var/svc/log/oxide-crucible-agent:default.log
10902023-10-04T20:55:55.516Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0a7266c5-6f2c-4987-a84e-401bf4366e32/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10912023-10-04T20:55:56.656Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0a7266c5-6f2c-4987-a84e-401bf4366e32/root/var/svc/log/oxide-crucible-downstairs:default.log
10922023-10-04T20:55:56.745Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0a7266c5-6f2c-4987-a84e-401bf4366e32/root/var/svc/log/oxide-crucible-downstairs:downstairs-75a63791-c502-4f21-b25c-f591e4bf8b0b.log (5966 bytes)
10932023-10-04T20:55:57.876Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0a7266c5-6f2c-4987-a84e-401bf4366e32/root/var/svc/log/oxide-crucible-downstairs:downstairs-75a63791-c502-4f21-b25c-f591e4bf8b0b.log
10942023-10-04T20:55:57.902Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_9c9a5cd9-d711-4cca-af30-3744c0326e08/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10952023-10-04T20:55:59.040Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_9c9a5cd9-d711-4cca-af30-3744c0326e08/root/var/svc/log/oxide-crucible-pantry:default.log
10962023-10-04T20:55:59.065Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_bff180fa-ede6-474f-9f8c-21b9491b64e0/root/var/svc/log/oxide-external_dns:default.log (19384 bytes)
10972023-10-04T20:56:00.286Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_bff180fa-ede6-474f-9f8c-21b9491b64e0/root/var/svc/log/oxide-external_dns:default.log
10982023-10-04T20:56:00.311Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_595bcf82-f606-44d1-95ef-117bb74346be/root/var/svc/log/oxide-internal_dns:default.log (103233 bytes)
10992023-10-04T20:56:01.523Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_595bcf82-f606-44d1-95ef-117bb74346be/root/var/svc/log/oxide-internal_dns:default.log
11002023-10-04T20:56:01.547Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_371e67d7-190f-4943-a359-9015ac849af3/root/var/svc/log/oxide-oximeter:default.log (169714 bytes)
11012023-10-04T20:56:01.758Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_371e67d7-190f-4943-a359-9015ac849af3/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 169714 -> 195794
11022023-10-04T20:56:02.877Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_371e67d7-190f-4943-a359-9015ac849af3/root/var/svc/log/oxide-oximeter:default.log
11032023-10-04T20:56:02.902Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f2685a75-0c73-4d6e-818f-7c2c94c6e578/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
11042023-10-04T20:56:03.975Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f2685a75-0c73-4d6e-818f-7c2c94c6e578/root/var/svc/log/oxide-cockroachdb:default.log
11052023-10-04T20:56:04.000Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f5200763-a22c-4058-87f6-f38a60d54c80/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
11062023-10-04T20:56:05.078Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f5200763-a22c-4058-87f6-f38a60d54c80/root/var/svc/log/oxide-cockroachdb:default.log
11072023-10-04T20:56:05.104Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a12ae1b1-ce8b-4b8b-ad7a-49ce5fdf435a/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
11082023-10-04T20:56:06.173Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a12ae1b1-ce8b-4b8b-ad7a-49ce5fdf435a/root/var/svc/log/oxide-crucible-agent:default.log
11092023-10-04T20:56:06.198Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a12ae1b1-ce8b-4b8b-ad7a-49ce5fdf435a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11102023-10-04T20:56:07.270Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a12ae1b1-ce8b-4b8b-ad7a-49ce5fdf435a/root/var/svc/log/oxide-crucible-downstairs:default.log
11112023-10-04T20:56:07.297Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_1dd8049a-7dcc-492e-9b40-e8598db9fc71/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
11122023-10-04T20:56:08.404Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_1dd8049a-7dcc-492e-9b40-e8598db9fc71/root/var/svc/log/oxide-crucible-pantry:default.log
11132023-10-04T20:56:08.429Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c075754e-7b96-41cc-b7eb-3fc5678ff6f2/root/var/svc/log/oxide-nexus:default.log (489961 bytes)
11142023-10-04T20:56:08.658Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c075754e-7b96-41cc-b7eb-3fc5678ff6f2/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 489961 -> 492999
11152023-10-04T20:56:09.732Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c075754e-7b96-41cc-b7eb-3fc5678ff6f2/root/var/svc/log/oxide-nexus:default.log
11162023-10-04T20:56:09.759Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_16d1de6f-94d7-4e02-9953-5e860482d8e5/root/var/svc/log/oxide-cockroachdb:default.log (5341 bytes)
11172023-10-04T20:56:10.831Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_16d1de6f-94d7-4e02-9953-5e860482d8e5/root/var/svc/log/oxide-cockroachdb:default.log
11182023-10-04T20:56:10.857Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7e89a6f4-5036-4ad6-a28f-1e13fee6785a/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
11192023-10-04T20:56:11.941Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7e89a6f4-5036-4ad6-a28f-1e13fee6785a/root/var/svc/log/oxide-crucible-agent:default.log
11202023-10-04T20:56:11.967Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7e89a6f4-5036-4ad6-a28f-1e13fee6785a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11212023-10-04T20:56:13.051Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7e89a6f4-5036-4ad6-a28f-1e13fee6785a/root/var/svc/log/oxide-crucible-downstairs:default.log
11222023-10-04T20:56:13.078Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76/root/var/log/chrony/measurements.log (8083 bytes)
11232023-10-04T20:56:14.176Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76/root/var/log/chrony/measurements.log
11242023-10-04T20:56:14.202Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76/root/var/log/chrony/statistics.log (4680 bytes)
11252023-10-04T20:56:15.273Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76/root/var/log/chrony/statistics.log
11262023-10-04T20:56:15.297Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76/root/var/log/chrony/tracking.log (1190 bytes)
11272023-10-04T20:56:16.421Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_e3b39b6f-e9e4-40b6-bb07-5e225b38ae76/root/var/log/chrony/tracking.log