01HCG34FEAG7KHK398041KA0WZ: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.23",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HCG3702CXW6DVG4DPTXBFY2F

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-11T20:23:03.987Zjob dependencies complete; ready to run (waiting for 57 m 35 s)
22023-10-11T22:00:56.489Zjob assigned to worker 01HCGC1HHG2MT23TPWC8RGCRR4 (queued for 1 h 37 m 52 s)
32023-10-11T22:00:57.003Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-11T22:00:57.405Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-10-11T22:00:57.769Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-10-11T22:00:58.013Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-11T22:00:58.321Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-10-11T22:00:58.600ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-11T22:00:59.237Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-11T22:00:59.881ZOxide Helios Version helios-2.0.22216 64-bit
112023-10-11T22:01:00.031ZConfiguring devices.
122023-10-11T22:01:00.223Z#################################################
132023-10-11T22:01:00.361Z#### ############# ##
142023-10-11T22:01:00.502Z### ### ############ ##
152023-10-11T22:01:00.580Z## ### # ## ### ## Oxide Computer Company ##
162023-10-11T22:01:00.643Z## ## ## ### # ### ##
172023-10-11T22:01:01.311Z## # ### #### #### This Station Under ##
182023-10-11T22:01:01.616Z### ### #### # ### Computer Control ##
192023-10-11T22:01:01.852Z#### #### ### ## ##
202023-10-11T22:01:02.068Z##################################################
212023-10-11T22:01:12.472ZHostname: buskin
222023-10-11T22:01:13.259ZLoading smf(7) serice descriptions:
232023-10-11T22:01:14.943Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-0b5a6903c7c17be7.gz
242023-10-11T22:01:18.524Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-0b5a6903c7c17be7.gz
252023-10-11T22:01:18.577Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-11T22:03:17.125Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-11T22:03:17.186Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-6da9107f9543f44b.gz
282023-10-11T22:05:17.879Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-6da9107f9543f44b.gz
292023-10-11T22:05:18.403Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-11T22:05:20.662Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-11T22:05:20.696Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-11T22:05:28.638Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-11T22:05:28.665Zdownloading input: /input/package/work/version.txt
342023-10-11T22:05:28.760Zdownloaded input: /input/package/work/version.txt
352023-10-11T22:05:28.814Zdownloading input: /input/package/work/package.tar.gz
362023-10-11T22:06:09.250Zdownloaded input: /input/package/work/package.tar.gz
372023-10-11T22:06:09.278Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-11T22:06:42.997Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-11T22:06:43.233Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-11T22:07:36.447Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-11T22:07:36.477Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-11T22:09:15.203Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-11T22:09:15.424Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-11T22:10:26.190Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-11T22:10:26.630Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-11T22:11:00.605Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-11T22:11:00.645Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-11T22:11:10.597Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-11T22:11:10.681Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-11T22:11:24.356Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-11T22:11:24.420Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-11T22:11:53.423Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-11T22:11:54.715Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-11T22:12:00.757Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-11T22:12:00.798Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-11T22:12:01.317Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-11T22:12:01.349Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-11T22:12:07.094Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-11T22:12:07.120Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-11T22:12:12.906Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-11T22:12:12.943Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-11T22:12:46.486Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-11T22:12:46.638Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-11T22:13:23.639Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-11T22:13:23.674Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-11T22:13:23.736Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-11T22:13:23.775Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-11T22:13:32.681Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-11T22:13:32.706Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-11T22:14:06.772Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-11T22:14:06.892Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-11T22:16:20.721Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-11T22:16:20.963Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-11T22:19:28.674Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-11T22:19:28.703Zstarting task 0: "setup"
762023-10-11T22:19:28.741Z++ uname -s
772023-10-11T22:19:28.772Z+ kern=SunOS
782023-10-11T22:19:28.828Z+ case "$kern" in
792023-10-11T22:19:28.856Z+ groupadd -g 12345 build
802023-10-11T22:19:29.295Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-11T22:19:30.935Z+ zfs create -o mountpoint=/work rpool/work
822023-10-11T22:19:30.987Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-11T22:19:31.067Z+ home_fs=zfs
842023-10-11T22:19:31.108Z+ [[ zfs == autofs ]]
852023-10-11T22:19:31.184Z+ mkdir -p /home/build
862023-10-11T22:19:31.252Z+ chown build:build /home/build /work
872023-10-11T22:19:37.675Z+ chmod 0700 /home/build /work
882023-10-11T22:19:38.573Zprocess exited: duration 4046 ms, exit code 0
 
892023-10-11T22:19:39.313Zstarting task 1: "authentication"
902023-10-11T22:19:39.941Zprocess exited: duration 544 ms, exit code 0
 
912023-10-11T22:19:40.829Zstarting task 2: "build"
922023-10-11T22:19:40.894Z+ trap _exit_trap EXIT
932023-10-11T22:19:40.933Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-11T22:19:41.076Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-11T22:19:41.112Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-11T22:19:41.142Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-11T22:19:41.179Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-11T22:19:41.218Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-11T22:19:41.250Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-11T22:19:41.286Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-11T22:19:41.325Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-11T22:19:41.361Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-11T22:19:41.395Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-11T22:19:41.426Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-11T22:19:41.458Z+ [[ -d /opt/oxide ]]
1062023-10-11T22:19:41.491Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-11T22:19:41.943Z+ pfexec mkdir /opt/oxide
1082023-10-11T22:19:41.977Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-11T22:19:42.005Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-11T22:19:42.030Z+ cd /opt/oxide-underneath
1112023-10-11T22:19:42.054Z+ pfexec tar ceEp@/f - .
1122023-10-11T22:19:42.083Z+ cd /opt/oxide
1132023-10-11T22:19:42.532Z+ pfexec tar xveEp@/f -
1142023-10-11T22:19:42.570Zx ., 0 bytes, 0 tape blocks
1152023-10-11T22:19:42.611Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-11T22:19:42.647Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-11T22:19:42.687Zx ./opte/bin/opteadm, 50138640 bytes, 97928 tape blocks
1182023-10-11T22:19:42.723Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-11T22:19:42.766Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-11T22:19:42.799Z+ pfexec mkdir /opt/oxide/work
1212023-10-11T22:19:42.830Z+ pfexec chown build:build /opt/oxide/work
1222023-10-11T22:19:42.856Z+ cd /opt/oxide/work
1232023-10-11T22:19:42.884Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-11T22:19:42.912ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-11T22:19:42.959Zx out/maghemite.tar, 74957312 bytes, 146401 tape blocks
1262023-10-11T22:19:42.984Zx out/omicron-sled-agent.tar, 113162240 bytes, 221020 tape blocks
1272023-10-11T22:19:43.020Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-11T22:19:43.059Zx out/npuzone/npuzone, 37379944 bytes, 73008 tape blocks
1292023-10-11T22:19:43.093Zx package-manifest.toml, 21303 bytes, 42 tape blocks
1302023-10-11T22:19:43.125Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-11T22:19:43.161Zx target/release/omicron-package, 37896096 bytes, 74016 tape blocks
1322023-10-11T22:19:43.212Zx tools/create_virtual_hardware.sh, 2818 bytes, 6 tape blocks
1332023-10-11T22:19:43.244Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-11T22:19:43.325Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-11T22:19:43.360Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-11T22:19:43.416Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-11T22:19:43.452Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-11T22:19:43.487Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-11T22:19:43.523Z
1402023-10-11T22:19:43.618Zreal 1.600315166
1412023-10-11T22:19:43.660Zuser 1.438414302
1422023-10-11T22:19:43.693Zsys 0.285801793
1432023-10-11T22:19:43.725Ztrap 0.000053710
1442023-10-11T22:19:43.759Ztflt 0.000000000
1452023-10-11T22:19:43.787Zdflt 0.000000000
1462023-10-11T22:19:43.821Zkflt 0.000000000
1472023-10-11T22:19:43.870Zlock 0.000000000
1482023-10-11T22:19:43.899Zslp 1.437236265
1492023-10-11T22:19:43.930Zlat 0.036908427
1502023-10-11T22:19:43.998Zstop 0.000125271
1512023-10-11T22:19:44.031Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-11T22:19:44.073Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-11T22:19:44.111Z+ mkdir tests
1542023-10-11T22:19:44.141Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-11T22:19:44.169Z+ ptime -m gunzip
1562023-10-11T22:19:44.199Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-0b5a6903c7c17be7
1572023-10-11T22:19:44.255Z
1582023-10-11T22:19:44.285Zreal 0.137970403
1592023-10-11T22:19:44.725Zuser 0.126863930
1602023-10-11T22:19:44.765Zsys 0.010859739
1612023-10-11T22:19:45.285Ztrap 0.000012279
1622023-10-11T22:19:45.317Ztflt 0.000000000
1632023-10-11T22:19:45.350Zdflt 0.000000000
1642023-10-11T22:19:45.385Zkflt 0.000000000
1652023-10-11T22:19:45.420Zlock 0.000000000
1662023-10-11T22:19:45.451Zslp 0.000020740
1672023-10-11T22:19:45.482Zlat 0.000024040
1682023-10-11T22:19:45.515Zstop 0.000127201
1692023-10-11T22:19:45.544Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-0b5a6903c7c17be7
1702023-10-11T22:19:45.570Z+ chmod a+x tests/bootstrap-0b5a6903c7c17be7
1712023-10-11T22:19:46.004Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-11T22:19:46.031Z+ ptime -m gunzip
1732023-10-11T22:19:46.056Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-11T22:19:46.193Z
1752023-10-11T22:19:46.226Zreal 2.214263547
1762023-10-11T22:19:46.263Zuser 2.069072889
1772023-10-11T22:19:46.303Zsys 0.144804723
1782023-10-11T22:19:46.345Ztrap 0.000164081
1792023-10-11T22:19:46.397Ztflt 0.000000000
1802023-10-11T22:19:46.440Zdflt 0.000000000
1812023-10-11T22:19:46.475Zkflt 0.000000000
1822023-10-11T22:19:46.508Zlock 0.000000000
1832023-10-11T22:19:46.539Zslp 0.000014180
1842023-10-11T22:19:46.576Zlat 0.000032750
1852023-10-11T22:19:46.633Zstop 0.000119202
1862023-10-11T22:19:46.689Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-11T22:19:46.727Z+ chmod a+x tests/bootstrap
1882023-10-11T22:19:46.760Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-11T22:19:46.799Z+ ptime -m gunzip
1902023-10-11T22:19:46.836Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-6da9107f9543f44b
1912023-10-11T22:19:48.741Z
1922023-10-11T22:19:48.774Zreal 2.538456537
1932023-10-11T22:19:48.813Zuser 2.392334798
1942023-10-11T22:19:48.847Zsys 0.145677914
1952023-10-11T22:19:48.879Ztrap 0.000217192
1962023-10-11T22:19:48.913Ztflt 0.000000000
1972023-10-11T22:19:48.955Zdflt 0.000000000
1982023-10-11T22:19:49.003Zkflt 0.000000000
1992023-10-11T22:19:49.041Zlock 0.000000000
2002023-10-11T22:19:49.076Zslp 0.000021180
2012023-10-11T22:19:49.109Zlat 0.000026790
2022023-10-11T22:19:49.140Zstop 0.000120201
2032023-10-11T22:19:49.165Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-6da9107f9543f44b
2042023-10-11T22:19:49.190Z+ chmod a+x tests/end_to_end_tests-6da9107f9543f44b
2052023-10-11T22:19:49.214Z+ bmat address ls
2062023-10-11T22:19:49.244ZNAME CIDR FIRST LAST COUNT
2072023-10-11T22:19:49.273Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-10-11T22:19:49.301Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-11T22:19:49.330Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-10-11T22:19:49.364Z+ EXTRA_IP_START=10.151.1.164
2112023-10-11T22:19:49.389Z+ EXTRA_IP_END=10.151.1.227
2122023-10-11T22:19:49.419Z+ EXTRA_IP_COUNT=64
2132023-10-11T22:19:49.860Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-10-11T22:19:50.349Z+ EXTRA_IP_BASE=10.151.1
2152023-10-11T22:19:50.376Z+ EXTRA_IP_FOCTET=164
2162023-10-11T22:19:50.408Z+ EXTRA_IP_LOCTET=227
2172023-10-11T22:19:50.444Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-10-11T22:19:50.476Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-10-11T22:19:50.512Z+ DNS_IP1=10.151.1.164
2202023-10-11T22:19:50.545Z+ DNS_IP2=10.151.1.165
2212023-10-11T22:19:50.574Z+ UPLINK_IP=10.151.1.174
2222023-10-11T22:19:50.604Z+ IPPOOL_START=10.151.1.175
2232023-10-11T22:19:50.635Z+ IPPOOL_END=10.151.1.227
2242023-10-11T22:19:50.669Z++ ipadm show-addr -po type,addr
2252023-10-11T22:19:51.114Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-10-11T22:19:51.143Z+ GATEWAY_IP=10.151.1.100
2272023-10-11T22:19:51.173Z+ [[ -n 10.151.1.100 ]]
2282023-10-11T22:19:51.205Z+ ping -s 10.151.1.100 56 1
2292023-10-11T22:19:51.247ZPING 10.151.1.100: 56 data bytes
2302023-10-11T22:19:51.324Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
2312023-10-11T22:19:51.788Z
2322023-10-11T22:19:51.818Z----10.151.1.100 PING Statistics----
2332023-10-11T22:19:51.853Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-10-11T22:19:51.894Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-10-11T22:19:51.930Z++ arp -an
2362023-10-11T22:19:51.971Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-10-11T22:19:52.012Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-10-11T22:19:52.045Z+ routeadm -e ipv4-forwarding -u
2392023-10-11T22:19:52.076Z+ PXA_START=10.151.1.164
2402023-10-11T22:19:52.107Z+ PXA_END=10.151.1.227
2412023-10-11T22:19:52.141Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-10-11T22:19:52.176Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-10-11T22:19:52.206Z+ ZPOOL_VDEV_DIR=/scratch
2442023-10-11T22:19:52.236Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-10-11T22:19:52.285Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-10-11T22:19:52.317Z++ cd ./tools
2472023-10-11T22:19:52.378Z++ pwd
2482023-10-11T22:19:52.416Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-10-11T22:19:52.467Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-10-11T22:19:52.495Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-10-11T22:19:52.540Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-10-11T22:19:52.575Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-10-11T22:19:52.606Z++ dladm show-phys -p -o LINK
2542023-10-11T22:19:52.642Z++ head -1
2552023-10-11T22:19:52.674Z+ PHYSICAL_LINK=igb0
2562023-10-11T22:19:52.709Z+ echo 'Using igb0 as physical link'
2572023-10-11T22:19:52.744ZUsing igb0 as physical link
2582023-10-11T22:19:52.777Z+ ensure_run_as_root
2592023-10-11T22:19:52.806Z++ id -u
2602023-10-11T22:19:52.843Z+ [[ 0 -ne 0 ]]
2612023-10-11T22:19:52.872Z+ ensure_zpools
2622023-10-11T22:19:52.899Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-10-11T22:19:52.928Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-10-11T22:19:52.968Z+ readarray -t ZPOOLS
2652023-10-11T22:19:53.000Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-10-11T22:19:53.029Z++ sed 's/[ ",]//g'
2672023-10-11T22:19:53.057Z+ for ZPOOL in "${ZPOOLS[@]}"
2682023-10-11T22:19:53.083Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2692023-10-11T22:19:53.114ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2702023-10-11T22:19:53.141Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2712023-10-11T22:19:53.174Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2722023-10-11T22:19:53.612Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2732023-10-11T22:19:53.644Z0+0 records in
2742023-10-11T22:19:53.681Z0+0 records out
2752023-10-11T22:19:53.713Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2762023-10-11T22:19:53.741Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2772023-10-11T22:19:53.773Z+ set +x
2782023-10-11T22:19:53.805ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2792023-10-11T22:19:53.835Z++ zpool list -o name
2802023-10-11T22:19:53.867ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2812023-10-11T22:19:53.917Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-10-11T22:19:53.956ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2832023-10-11T22:19:53.991Z+ [[ -z '' ]]
2842023-10-11T22:19:54.016ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2852023-10-11T22:19:54.053Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2862023-10-11T22:19:54.089ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2872023-10-11T22:19:54.114Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2882023-10-11T22:19:54.141ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2892023-10-11T22:19:54.170Z+ set +x
2902023-10-11T22:19:54.204ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2912023-10-11T22:19:54.235Z+ for ZPOOL in "${ZPOOLS[@]}"
2922023-10-11T22:19:54.265ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2932023-10-11T22:19:54.301Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2942023-10-11T22:19:54.744ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2952023-10-11T22:19:54.831Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2962023-10-11T22:19:54.866ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2972023-10-11T22:19:55.354Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2982023-10-11T22:19:55.396ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2992023-10-11T22:19:55.859Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3002023-10-11T22:19:55.901ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3012023-10-11T22:19:55.936Z0+0 records in
3022023-10-11T22:19:56.374ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3032023-10-11T22:19:56.403Z0+0 records out
3042023-10-11T22:19:56.841ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3052023-10-11T22:19:56.869Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3062023-10-11T22:19:56.903ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3072023-10-11T22:19:57.348Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3082023-10-11T22:19:57.386ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3092023-10-11T22:19:57.420Z+ set +x
3102023-10-11T22:19:57.446ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3112023-10-11T22:19:57.886Z++ zpool list -o name
3122023-10-11T22:19:57.927ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3132023-10-11T22:19:57.969Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3142023-10-11T22:19:58.014ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3152023-10-11T22:19:58.048Z+ [[ -z '' ]]
3162023-10-11T22:19:58.085ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3172023-10-11T22:19:58.122Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3182023-10-11T22:19:58.170ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3192023-10-11T22:19:59.446Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3202023-10-11T22:19:59.480ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3212023-10-11T22:19:59.510Z+ set +x
3222023-10-11T22:19:59.543ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3232023-10-11T22:19:59.579Z+ for ZPOOL in "${ZPOOLS[@]}"
3242023-10-11T22:19:59.626ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3252023-10-11T22:19:59.662Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3262023-10-11T22:19:59.790ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3272023-10-11T22:19:59.929Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3282023-10-11T22:19:59.967ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3292023-10-11T22:20:00.007Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-10-11T22:20:00.058ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3312023-10-11T22:20:00.096Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3322023-10-11T22:20:00.127ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3332023-10-11T22:20:00.570Z0+0 records in
3342023-10-11T22:20:00.615ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3352023-10-11T22:20:00.649Z0+0 records out
3362023-10-11T22:20:00.674ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3372023-10-11T22:20:00.698Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3382023-10-11T22:20:00.724ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3392023-10-11T22:20:00.757Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3402023-10-11T22:20:00.813ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3412023-10-11T22:20:00.839Z+ set +x
3422023-10-11T22:20:00.865ZSimnet net0/sc0_0 exists
3432023-10-11T22:20:00.895Z++ zpool list -o name
3442023-10-11T22:20:00.924ZSimnet net1/sc1_0 exists
3452023-10-11T22:20:00.957Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3462023-10-11T22:20:00.991ZVnic sc0_1 exists
3472023-10-11T22:20:01.025Z+ [[ -z '' ]]
3482023-10-11T22:20:01.057Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3492023-10-11T22:20:01.088Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-10-11T22:20:01.113Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-10-11T22:20:01.138Z+ set +x
3522023-10-11T22:20:01.163Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-10-11T22:20:01.189Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-10-11T22:20:01.218Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-10-11T22:20:01.249Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-10-11T22:20:01.283Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-10-11T22:20:01.307Z0+0 records in
3582023-10-11T22:20:01.334Z0+0 records out
3592023-10-11T22:20:01.367Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3602023-10-11T22:20:01.419Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-10-11T22:20:01.456Z+ set +x
3622023-10-11T22:20:01.487Z++ zpool list -o name
3632023-10-11T22:20:01.529Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-10-11T22:20:01.559Z+ [[ -z '' ]]
3652023-10-11T22:20:01.588Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-10-11T22:20:01.615Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-10-11T22:20:01.649Z+ set +x
3682023-10-11T22:20:01.688Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-10-11T22:20:02.122Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-10-11T22:20:02.166Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-10-11T22:20:02.208Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-10-11T22:20:02.651Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-10-11T22:20:02.681Z0+0 records in
3742023-10-11T22:20:02.714Z0+0 records out
3752023-10-11T22:20:02.742Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3762023-10-11T22:20:02.776Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-10-11T22:20:02.808Z+ set +x
3782023-10-11T22:20:02.910Z++ zpool list -o name
3792023-10-11T22:20:02.940Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-10-11T22:20:02.977Z+ [[ -z '' ]]
3812023-10-11T22:20:03.007Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-10-11T22:20:03.038Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-10-11T22:20:03.072Z+ set +x
3842023-10-11T22:20:03.562Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-10-11T22:20:03.601Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862023-10-11T22:20:03.631Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872023-10-11T22:20:03.659Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882023-10-11T22:20:03.697Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892023-10-11T22:20:03.740Z0+0 records in
3902023-10-11T22:20:03.772Z0+0 records out
3912023-10-11T22:20:03.805Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-10-11T22:20:03.838Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932023-10-11T22:20:03.873Z+ set +x
3942023-10-11T22:20:03.954Z++ zpool list -o name
3952023-10-11T22:20:04.009Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3962023-10-11T22:20:04.060Z+ [[ -z '' ]]
3972023-10-11T22:20:04.096Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3982023-10-11T22:20:04.132Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3992023-10-11T22:20:04.164Z+ set +x
4002023-10-11T22:20:04.200Z+ for ZPOOL in "${ZPOOLS[@]}"
4012023-10-11T22:20:04.228Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4022023-10-11T22:20:04.264Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4032023-10-11T22:20:04.297Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4042023-10-11T22:20:04.325Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4052023-10-11T22:20:04.357Z0+0 records in
4062023-10-11T22:20:04.390Z0+0 records out
4072023-10-11T22:20:04.424Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4082023-10-11T22:20:04.463Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4092023-10-11T22:20:04.502Z+ set +x
4102023-10-11T22:20:04.540Z++ zpool list -o name
4112023-10-11T22:20:04.578Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4122023-10-11T22:20:04.609Z+ [[ -z '' ]]
4132023-10-11T22:20:04.638Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4142023-10-11T22:20:04.671Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4152023-10-11T22:20:04.716Z+ set +x
4162023-10-11T22:20:05.184Z+ for ZPOOL in "${ZPOOLS[@]}"
4172023-10-11T22:20:05.212Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4182023-10-11T22:20:05.242Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192023-10-11T22:20:05.881Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4202023-10-11T22:20:05.936Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4212023-10-11T22:20:05.978Z0+0 records in
4222023-10-11T22:20:08.133Z0+0 records out
4232023-10-11T22:20:08.300Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4242023-10-11T22:20:08.662Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4252023-10-11T22:20:09.158Z+ set +x
4262023-10-11T22:20:09.196Z++ zpool list -o name
4272023-10-11T22:20:09.234Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4282023-10-11T22:20:09.279Z+ [[ -z '' ]]
4292023-10-11T22:20:09.347Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4302023-10-11T22:20:09.397Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4312023-10-11T22:20:09.464Z+ set +x
4322023-10-11T22:20:09.492Z+ for ZPOOL in "${ZPOOLS[@]}"
4332023-10-11T22:20:09.533Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4342023-10-11T22:20:09.573Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4352023-10-11T22:20:09.609Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4362023-10-11T22:20:09.645Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4372023-10-11T22:20:09.674Z0+0 records in
4382023-10-11T22:20:09.702Z0+0 records out
4392023-10-11T22:20:09.836Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4402023-10-11T22:20:10.320Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4412023-10-11T22:20:10.405Z+ set +x
4422023-10-11T22:20:10.439Z++ zpool list -o name
4432023-10-11T22:20:10.473Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4442023-10-11T22:20:10.504Z+ [[ -z '' ]]
4452023-10-11T22:20:10.531Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4462023-10-11T22:20:10.560Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4472023-10-11T22:20:10.592Z+ set +x
4482023-10-11T22:20:10.627Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4492023-10-11T22:20:10.663Z+ readarray -t ZPOOLS
4502023-10-11T22:20:10.693Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4512023-10-11T22:20:10.723Z++ sed 's/[ ",]//g'
4522023-10-11T22:20:10.753Z+ for ZPOOL in "${ZPOOLS[@]}"
4532023-10-11T22:20:10.799Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4542023-10-11T22:20:10.826Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4552023-10-11T22:20:10.860Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4562023-10-11T22:20:11.335Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4572023-10-11T22:20:11.474Z0+0 records in
4582023-10-11T22:20:11.538Z0+0 records out
4592023-10-11T22:20:11.981Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4602023-10-11T22:20:12.020Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4612023-10-11T22:20:12.062Z+ set +x
4622023-10-11T22:20:12.094Z++ zpool list -o name
4632023-10-11T22:20:12.123Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4642023-10-11T22:20:12.157Z+ [[ -z '' ]]
4652023-10-11T22:20:12.197Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-10-11T22:20:12.231Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4672023-10-11T22:20:12.259Z+ set +x
4682023-10-11T22:20:12.287Z+ for ZPOOL in "${ZPOOLS[@]}"
4692023-10-11T22:20:12.315Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4702023-10-11T22:20:12.345Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4712023-10-11T22:20:13.214Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4722023-10-11T22:20:13.268Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4732023-10-11T22:20:13.301Z0+0 records in
4742023-10-11T22:20:13.337Z0+0 records out
4752023-10-11T22:20:13.372Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4762023-10-11T22:20:13.403Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4772023-10-11T22:20:13.432Z+ set +x
4782023-10-11T22:20:13.465Z++ zpool list -o name
4792023-10-11T22:20:13.502Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4802023-10-11T22:20:13.557Z+ [[ -z '' ]]
4812023-10-11T22:20:13.595Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4822023-10-11T22:20:13.620Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4832023-10-11T22:20:13.645Z+ set +x
4842023-10-11T22:20:13.669Z+ ensure_simulated_links igb0
4852023-10-11T22:20:13.701Z+ local PHYSICAL_LINK=igb0
4862023-10-11T22:20:13.733Z+ INDICES=("0" "1")
4872023-10-11T22:20:13.758Z+ for I in "${INDICES[@]}"
4882023-10-11T22:20:13.783Zdownloading softnpu ...done
4892023-10-11T22:20:13.809Z++ get_simnet_name_if_exists net0
4902023-10-11T22:20:13.842Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/libsidecar_lite.so.sha256.txt
4912023-10-11T22:20:13.877Z++ dladm show-simnet -p -o LINK net0
4922023-10-11T22:20:13.910Z++ echo ''
4932023-10-11T22:20:14.359Z+ [[ -z '' ]]
4942023-10-11T22:20:14.394Z+ dladm create-simnet -t net0
4952023-10-11T22:20:14.865Z+ dladm create-simnet -t sc0_0
4962023-10-11T22:20:15.032Z+ dladm modify-simnet -t -p net0 sc0_0
4972023-10-11T22:20:15.063Z+ dladm set-linkprop -p mtu=9000 sc0_0
4982023-10-11T22:20:15.087Z+ success 'Simnet net0/sc0_0 exists'
4992023-10-11T22:20:15.112Z+ set +x
5002023-10-11T22:20:15.145Z+ for I in "${INDICES[@]}"
5012023-10-11T22:20:15.174Z++ get_simnet_name_if_exists net1
5022023-10-11T22:20:15.205Z++ dladm show-simnet -p -o LINK net1
5032023-10-11T22:20:15.238Z++ echo ''
5042023-10-11T22:20:15.266Z+ [[ -z '' ]]
5052023-10-11T22:20:15.293Z+ dladm create-simnet -t net1
5062023-10-11T22:20:15.326Z+ dladm create-simnet -t sc1_0
5072023-10-11T22:20:15.358Z+ dladm modify-simnet -t -p net1 sc1_0
5082023-10-11T22:20:15.390Z+ dladm set-linkprop -p mtu=9000 sc1_0
5092023-10-11T22:20:15.428Z+ success 'Simnet net1/sc1_0 exists'
5102023-10-11T22:20:15.899Z+ set +x
5112023-10-11T22:20:16.788Z++ get_vnic_name_if_exists sc0_1
5122023-10-11T22:20:16.819Z++ dladm show-vnic -p -o LINK sc0_1
5132023-10-11T22:20:16.852Z++ echo ''
5142023-10-11T22:20:16.888Z+ [[ -z '' ]]
5152023-10-11T22:20:16.919Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162023-10-11T22:20:16.953Z+ success 'Vnic sc0_1 exists'
5172023-10-11T22:20:16.979Z+ set +x
5182023-10-11T22:20:17.003Z+ warn_if_no_proxy_arp
5192023-10-11T22:20:17.029Z+ [[ -v PXA_START ]]
5202023-10-11T22:20:17.056Z+ [[ -v PXA_END ]]
5212023-10-11T22:20:17.081Z+ ensure_softnpu_zone
5222023-10-11T22:20:17.130Z+ zoneadm list
5232023-10-11T22:20:17.162Z+ grep -q sidecar_softnpu
5242023-10-11T22:20:17.191Z+ [[ -f out/npuzone/npuzone ]]
5252023-10-11T22:20:17.216Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0
5262023-10-11T22:20:42.362Zdownloading asic_program.so ...done
5272023-10-11T22:20:43.740Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/scadm.sha256.txt
5282023-10-11T22:21:18.436Zdownloading scadm ...done
5292023-10-11T22:21:18.645Zcreating port tfportrear0_0 for sc0_0
5302023-10-11T22:21:20.187Zcreating port tfportqsfp0_0 for sc0_1
5312023-10-11T22:21:20.488Zsoftnpu zone setup
5322023-10-11T22:21:20.649Zconfigure zone
5332023-10-11T22:21:21.235Zinstall zone
5342023-10-11T22:21:21.276Zboot zone
5352023-10-11T22:21:21.312Z[sidecar_softnpu] svcs milestone/network
5362023-10-11T22:21:21.362ZSTATE STIME FMRI
5372023-10-11T22:21:21.416Zonline 22:21:17 svc:/milestone/network:default
5382023-10-11T22:21:21.463Z[sidecar_softnpu] /softnpu/npu start
5392023-10-11T22:21:21.513Z
5402023-10-11T22:21:22.778Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5412023-10-11T22:21:22.813Z+ GATEWAY_IP=10.151.1.100
5422023-10-11T22:21:22.848Z+ echo 'Using 10.151.1.100 as gateway ip'
5432023-10-11T22:21:22.883ZUsing 10.151.1.100 as gateway ip
5442023-10-11T22:21:22.923Z+ [[ ! -v GATEWAY_MAC ]]
5452023-10-11T22:21:23.371Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5462023-10-11T22:21:23.409Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5472023-10-11T22:21:23.461Z+++ echo 18:c0:4d:0d:9f:b2
5482023-10-11T22:21:23.500Z+ COUNT=1
5492023-10-11T22:21:23.539Z+ [[ 1 -ne 1 ]]
5502023-10-11T22:21:23.572Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5512023-10-11T22:21:23.612ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5522023-10-11T22:21:24.078Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5532023-10-11T22:21:24.112Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5542023-10-11T22:21:24.154Z+ PXA_MAC=a8:e1:de:01:70:1d
5552023-10-11T22:21:24.187Z+ [[ -v PXA_START ]]
5562023-10-11T22:21:24.230Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5572023-10-11T22:21:24.714Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5582023-10-11T22:21:24.941Z+ z_scadm dump-state
5592023-10-11T22:21:24.975Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5602023-10-11T22:21:25.022Zlocal v6:
5612023-10-11T22:21:25.052Zlocal v4:
5622023-10-11T22:21:25.084Zrouter v6:
5632023-10-11T22:21:25.124Zrouter v4:
5642023-10-11T22:21:25.162Zresolver v4:
5652023-10-11T22:21:25.195Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5662023-10-11T22:21:25.230Zresolver v6:
5672023-10-11T22:21:25.258Znat v4:
5682023-10-11T22:21:25.286Znat_v6:
5692023-10-11T22:21:25.332Zport_mac:
5702023-10-11T22:21:25.374Zproxy arp:
5712023-10-11T22:21:25.453Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5722023-10-11T22:21:25.478Z+ success 'softnpu zone exists'
5732023-10-11T22:21:25.506Z+ set +x
5742023-10-11T22:21:25.534Zsoftnpu zone exists
5752023-10-11T22:21:25.568Z
5762023-10-11T22:21:25.604Zreal 1:29.476068557
5772023-10-11T22:21:25.635Zuser 0.721099316
5782023-10-11T22:21:25.676Zsys 1.649598077
5792023-10-11T22:21:25.716Ztrap 0.000101736
5802023-10-11T22:21:25.747Ztflt 0.000180988
5812023-10-11T22:21:25.784Zdflt 0.000842920
5822023-10-11T22:21:26.662Zkflt 0.000000000
5832023-10-11T22:21:26.689Zlock 1:11:04.429877964
5842023-10-11T22:21:26.716Zslp 4:27.513104810
5852023-10-11T22:21:26.763Zlat 0.067561500
5862023-10-11T22:21:26.814Zstop 0.350320576
5872023-10-11T22:21:26.854Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5882023-10-11T22:21:26.895Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5892023-10-11T22:21:26.920Z+ SILO_NAME=recovery
5902023-10-11T22:21:26.960Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5912023-10-11T22:21:26.988Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5922023-10-11T22:21:27.028Z+ sed -i~ '
5932023-10-11T22:21:27.068Z /^external_dns_ips/c\
5942023-10-11T22:21:27.106Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5952023-10-11T22:21:27.143Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5962023-10-11T22:21:27.608Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
5972023-10-11T22:21:27.649Z /^first/c\
5982023-10-11T22:21:27.693Z+++ pkg/config-rss.toml Wed Oct 11 22:21:18 2023
5992023-10-11T22:21:27.739Zfirst = "10.151.1.164"
6002023-10-11T22:21:27.801Z@@ -41,7 +41,7 @@
6012023-10-11T22:21:27.845Z /^last/c\
6022023-10-11T22:21:27.889Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6032023-10-11T22:21:28.361Zlast = "10.151.1.173"
6042023-10-11T22:21:28.404Z # must be contained in one of the "internal services" IP Pool ranges listed
6052023-10-11T22:21:28.451Z }
6062023-10-11T22:21:28.499Z # below.
6072023-10-11T22:21:28.967Z /^\[rack_network_config/,/^$/ {
6082023-10-11T22:21:29.005Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6092023-10-11T22:21:29.050Z /^infra_ip_first/c\
6102023-10-11T22:21:29.091Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6112023-10-11T22:21:29.130Zinfra_ip_first = "10.151.1.174"
6122023-10-11T22:21:29.164Z
6132023-10-11T22:21:29.205Z /^infra_ip_last/c\
6142023-10-11T22:21:29.691Z # Initial TLS certificates for the external API
6152023-10-11T22:21:29.740Zinfra_ip_last = "10.151.1.174"
6162023-10-11T22:21:29.778Z #
6172023-10-11T22:21:29.827Z }
6182023-10-11T22:21:29.884Z@@ -77,8 +77,8 @@
6192023-10-11T22:21:30.332Z /^\[\[rack_network_config.uplinks/,/^$/ {
6202023-10-11T22:21:30.379Z #
6212023-10-11T22:21:30.440Z /^gateway_ip/c\
6222023-10-11T22:21:30.914Z # For more on this and what to put here, see docs/how-to-run.adoc.
6232023-10-11T22:21:30.960Zgateway_ip = "10.151.1.100"
6242023-10-11T22:21:30.999Z [[internal_services_ip_pool_ranges]]
6252023-10-11T22:21:32.446Z /^uplink_cidr/c\
6262023-10-11T22:21:32.482Z-first = "192.168.1.20"
6272023-10-11T22:21:32.519Zuplink_cidr = "10.151.1.174/32"
6282023-10-11T22:21:32.561Z-last = "192.168.1.29"
6292023-10-11T22:21:33.027Z }
6302023-10-11T22:21:33.061Z+first = "10.151.1.164"
6312023-10-11T22:21:33.099Z' pkg/config-rss.toml
6322023-10-11T22:21:33.138Z+last = "10.151.1.173"
6332023-10-11T22:21:33.189Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6342023-10-11T22:21:33.679Z
6352023-10-11T22:21:33.716Z+ true
6362023-10-11T22:21:33.755Z # TODO - this configuration is subject to change going forward. Ultimately these
6372023-10-11T22:21:34.226Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6382023-10-11T22:21:34.293Z # parameters should be provided to the control plane via wicket, but we need to
6392023-10-11T22:21:34.753Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6402023-10-11T22:21:34.794Z@@ -91,13 +91,13 @@
6412023-10-11T22:21:34.822Z+ TLS_NAME=recovery.sys.oxide.test
6422023-10-11T22:21:34.849Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-10-11T22:21:34.878Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6442023-10-11T22:21:34.952Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6452023-10-11T22:21:35.113Z...........+.+.........+.........+..+.......+...........+......+.+...+............+..+.+..+.......+..+.............+...+.....+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+...+......+......+...+.......+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...............+...+.........+...............+.+...+...............+..+..........+...+.....+....+...+..+......+.+........+...+.......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6462023-10-11T22:21:35.153Z # With softnpu, only one address is used.
6472023-10-11T22:21:35.193Z.+..............+.+.....+....+.........+.....+.......+.....+....+...+..+................+..+.+..+...+.+...+..+...+...+...+....+...........+......+.........................+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...............+...+......+...............+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.........+.+.................+.......+...+..+....+..+.......+...........+.+...+......+...........+...+............+......+...+......+.+.....+.......+...+..............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6482023-10-11T22:21:35.239Z-infra_ip_first = "192.168.1.30"
6492023-10-11T22:21:35.685Z-----
6502023-10-11T22:21:35.721Z-infra_ip_last = "192.168.1.30"
6512023-10-11T22:21:35.768Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6522023-10-11T22:21:35.819Z+infra_ip_first = "10.151.1.174"
6532023-10-11T22:21:35.874Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6542023-10-11T22:21:35.924Z+infra_ip_last = "10.151.1.174"
6552023-10-11T22:21:36.001Z+ rmdir pkg
6562023-10-11T22:21:36.060Z
6572023-10-11T22:21:36.512Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6582023-10-11T22:21:36.559Z # You can configure multiple uplinks by repeating the following stanza
6592023-10-11T22:21:36.603Z+ pfexec mkdir -p /usr/oxide
6602023-10-11T22:21:37.052Z [[rack_network_config.uplinks]]
6612023-10-11T22:21:37.081Z+ pfexec rm -f /usr/oxide/tcpproxy
6622023-10-11T22:21:37.564Z # The gateway IP for the rack's external network
6632023-10-11T22:21:37.612Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6642023-10-11T22:21:37.658Z-gateway_ip = "192.168.1.199"
6652023-10-11T22:21:37.719Z+ pfexec chmod +x /usr/oxide/tcpproxy
6662023-10-11T22:21:37.790Z+gateway_ip = "10.151.1.100"
6672023-10-11T22:21:38.262Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6682023-10-11T22:21:38.316Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6692023-10-11T22:21:38.367Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6702023-10-11T22:21:38.838Z uplink_port = "qsfp0"
6712023-10-11T22:21:40.220Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6722023-10-11T22:21:40.284Z uplink_port_speed = "40G"
6732023-10-11T22:21:40.331Z+ OMICRON_NO_UNINSTALL=1
6742023-10-11T22:21:40.372Z@@ -104,7 +104,7 @@
6752023-10-11T22:21:40.418Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6762023-10-11T22:21:40.473Z uplink_port_fec="none"
6772023-10-11T22:21:40.623ZOct 11 22:21:20.047 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6782023-10-11T22:21:41.518Z # For softnpu, an address within the "infra" block above that will be used for
6792023-10-11T22:21:41.577ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6802023-10-11T22:21:41.641Z # the softnpu uplink port. You can just pick the first address in that pool.
6812023-10-11T22:21:42.102ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6822023-10-11T22:21:42.150Z-uplink_cidr = "192.168.1.30/32"
6832023-10-11T22:21:42.197ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6842023-10-11T22:21:42.238Z+uplink_cidr = "10.151.1.174/32"
6852023-10-11T22:21:42.691ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6862023-10-11T22:21:42.726Z # Switch to use for the uplink. For single-rack deployments this can be
6872023-10-11T22:21:42.763ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6882023-10-11T22:21:43.236Z # "switch0" (upper slot) or "switch1" (lower slot). For single-node softnpu
6892023-10-11T22:21:43.280ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6902023-10-11T22:21:43.772Z # and dendrite stub environments, use "switch0"
6912023-10-11T22:21:44.230ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6922023-10-11T22:21:44.271Za pkg/config-rss.toml 7K
6932023-10-11T22:21:44.309ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6942023-10-11T22:21:44.351Za pkg/initial-tls-cert.pem 2K
6952023-10-11T22:21:44.394ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/maghemite.tar
6962023-10-11T22:21:44.462Za pkg/initial-tls-key.pem 4K
6972023-10-11T22:21:44.925ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6982023-10-11T22:21:45.782ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6992023-10-11T22:21:45.817ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7002023-10-11T22:21:46.272ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7012023-10-11T22:21:46.759ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7022023-10-11T22:21:47.209ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7032023-10-11T22:21:47.251ZOct 11 22:21:20.050 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7042023-10-11T22:21:47.294ZOct 11 22:21:20.275 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7052023-10-11T22:21:47.334ZOct 11 22:21:20.326 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7062023-10-11T22:21:47.381ZOct 11 22:21:20.386 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7072023-10-11T22:21:47.830Z
7082023-10-11T22:21:48.277Zreal 0.475795916
7092023-10-11T22:21:48.725Zuser 0.095708253
7102023-10-11T22:21:48.766Zsys 1.811505129
7112023-10-11T22:21:48.862Ztrap 0.000001719
7122023-10-11T22:21:48.919Ztflt 0.000007289
7132023-10-11T22:21:48.974Zdflt 0.000410226
7142023-10-11T22:21:49.457Zkflt 0.000000000
7152023-10-11T22:21:49.496Zlock 42.906151254
7162023-10-11T22:21:49.955Zslp 0.673098645
7172023-10-11T22:21:49.984Zlat 0.002896512
7182023-10-11T22:21:50.010Zstop 0.201979348
7192023-10-11T22:21:50.161Z+ retry=0
7202023-10-11T22:21:50.196Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7212023-10-11T22:21:51.471Z+ [[ 0 -gt 30 ]]
7222023-10-11T22:21:51.530Z+ sleep 1
7232023-10-11T22:21:51.571Z+ retry=1
7242023-10-11T22:21:51.623Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7252023-10-11T22:21:52.072Z+ [[ 1 -gt 30 ]]
7262023-10-11T22:21:52.104Z+ sleep 1
7272023-10-11T22:21:52.136Z+ retry=2
7282023-10-11T22:21:52.164Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7292023-10-11T22:21:52.191Z+ [[ 2 -gt 30 ]]
7302023-10-11T22:21:52.222Z+ sleep 1
7312023-10-11T22:21:52.255Z+ retry=3
7322023-10-11T22:21:52.295Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7332023-10-11T22:21:52.381Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7342023-10-11T22:21:52.834Zlocal v6:
7352023-10-11T22:21:52.934Zfe80::440a:3aff:fe2b:9407
7362023-10-11T22:21:52.982Zlocal v4:
7372023-10-11T22:21:53.025Zrouter v6:
7382023-10-11T22:21:53.494Zrouter v4:
7392023-10-11T22:21:53.618Zresolver v4:
7402023-10-11T22:21:53.687Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7412023-10-11T22:21:54.150Zresolver v6:
7422023-10-11T22:21:54.193Zfe80::20ea:90ff:fe96:e8c4 -> 22:ea:90:96:e8:c4
7432023-10-11T22:21:54.250Zfe80::440a:3aff:fe2b:9407 -> 46:0a:3a:2b:94:07
7442023-10-11T22:21:54.303Znat v4:
7452023-10-11T22:21:54.776Znat_v6:
7462023-10-11T22:21:54.926Zport_mac:
7472023-10-11T22:21:54.962Z1000: a8:40:25:6d:b7:bc
7482023-10-11T22:21:55.045Z1: a8:40:25:6d:b7:bd
7492023-10-11T22:21:55.074Zproxy arp:
7502023-10-11T22:21:55.104Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7512023-10-11T22:21:55.139Z+ retry=0
7522023-10-11T22:21:55.183Z+++ zoneadm list -n
7532023-10-11T22:21:55.222Z+++ grep oxz_ntp
7542023-10-11T22:21:55.261Z++ pfexec svcs -z -Hostate oxide/ntp
7552023-10-11T22:21:55.708Zsvcs: zone '-Hostate': entity not found
7562023-10-11T22:21:55.739Z++ true
7572023-10-11T22:21:55.771Z+ [[ '' != online ]]
7582023-10-11T22:21:55.814Z+ [[ 0 -gt 60 ]]
7592023-10-11T22:21:56.264Z+ sleep 1
7602023-10-11T22:21:56.299Z+ retry=1
7612023-10-11T22:21:56.335Z+++ zoneadm list -n
7622023-10-11T22:21:56.819Z+++ grep oxz_ntp
7632023-10-11T22:21:56.862Z++ pfexec svcs -z -Hostate oxide/ntp
7642023-10-11T22:21:56.909Zsvcs: zone '-Hostate': entity not found
7652023-10-11T22:21:56.956Z++ true
7662023-10-11T22:21:57.006Z+ [[ '' != online ]]
7672023-10-11T22:21:57.052Z+ [[ 1 -gt 60 ]]
7682023-10-11T22:21:57.100Z+ sleep 1
7692023-10-11T22:21:57.172Z+ retry=2
7702023-10-11T22:21:57.641Z+++ zoneadm list -n
7712023-10-11T22:21:57.698Z+++ grep oxz_ntp
7722023-10-11T22:21:57.752Z++ pfexec svcs -z -Hostate oxide/ntp
7732023-10-11T22:21:57.798Zsvcs: zone '-Hostate': entity not found
7742023-10-11T22:21:57.846Z++ true
7752023-10-11T22:21:58.298Z+ [[ '' != online ]]
7762023-10-11T22:21:58.341Z+ [[ 2 -gt 60 ]]
7772023-10-11T22:21:58.375Z+ sleep 1
7782023-10-11T22:21:58.416Z+ retry=3
7792023-10-11T22:21:58.459Z+++ zoneadm list -n
7802023-10-11T22:21:58.498Z+++ grep oxz_ntp
7812023-10-11T22:21:58.555Z++ pfexec svcs -z -Hostate oxide/ntp
7822023-10-11T22:21:58.611Zsvcs: zone '-Hostate': entity not found
7832023-10-11T22:21:58.661Z++ true
7842023-10-11T22:21:58.721Z+ [[ '' != online ]]
7852023-10-11T22:21:58.781Z+ [[ 3 -gt 60 ]]
7862023-10-11T22:21:58.833Z+ sleep 1
7872023-10-11T22:21:59.326Z+ retry=4
7882023-10-11T22:21:59.374Z+++ zoneadm list -n
7892023-10-11T22:21:59.425Z+++ grep oxz_ntp
7902023-10-11T22:21:59.483Z++ pfexec svcs -z -Hostate oxide/ntp
7912023-10-11T22:21:59.547Zsvcs: zone '-Hostate': entity not found
7922023-10-11T22:21:59.598Z++ true
7932023-10-11T22:22:00.055Z+ [[ '' != online ]]
7942023-10-11T22:22:01.583Z+ [[ 4 -gt 60 ]]
7952023-10-11T22:22:01.623Z+ sleep 1
7962023-10-11T22:22:01.687Z+ retry=5
7972023-10-11T22:22:01.732Z+++ zoneadm list -n
7982023-10-11T22:22:01.787Z+++ grep oxz_ntp
7992023-10-11T22:22:02.259Z++ pfexec svcs -z -Hostate oxide/ntp
8002023-10-11T22:22:02.311Zsvcs: zone '-Hostate': entity not found
8012023-10-11T22:22:02.349Z++ true
8022023-10-11T22:22:02.419Z+ [[ '' != online ]]
8032023-10-11T22:22:02.462Z+ [[ 5 -gt 60 ]]
8042023-10-11T22:22:02.500Z+ sleep 1
8052023-10-11T22:22:02.954Z+ retry=6
8062023-10-11T22:22:02.992Z+++ zoneadm list -n
8072023-10-11T22:22:03.019Z+++ grep oxz_ntp
8082023-10-11T22:22:03.045Z++ pfexec svcs -z -Hostate oxide/ntp
8092023-10-11T22:22:03.072Zsvcs: zone '-Hostate': entity not found
8102023-10-11T22:22:03.106Z++ true
8112023-10-11T22:22:03.137Z+ [[ '' != online ]]
8122023-10-11T22:22:03.171Z+ [[ 6 -gt 60 ]]
8132023-10-11T22:22:03.206Z+ sleep 1
8142023-10-11T22:22:03.253Z+ retry=7
8152023-10-11T22:22:03.302Z+++ zoneadm list -n
8162023-10-11T22:22:03.339Z+++ grep oxz_ntp
8172023-10-11T22:22:03.793Z++ pfexec svcs -z -Hostate oxide/ntp
8182023-10-11T22:22:03.831Zsvcs: zone '-Hostate': entity not found
8192023-10-11T22:22:03.868Z++ true
8202023-10-11T22:22:03.904Z+ [[ '' != online ]]
8212023-10-11T22:22:03.949Z+ [[ 7 -gt 60 ]]
8222023-10-11T22:22:03.993Z+ sleep 1
8232023-10-11T22:22:07.067Z+ retry=8
8242023-10-11T22:22:07.106Z+++ zoneadm list -n
8252023-10-11T22:22:07.155Z+++ grep oxz_ntp
8262023-10-11T22:22:07.195Z++ pfexec svcs -z -Hostate oxide/ntp
8272023-10-11T22:22:07.705Zsvcs: zone '-Hostate': entity not found
 
8282023-10-11T22:22:07.773Zbuskin console login: Oct 11 22:22:07 bukin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8292023-10-11T22:22:07.995Z++ true
8302023-10-11T22:22:08.048Z+ [[ '' != online ]]
8312023-10-11T22:22:08.099Z+ [[ 8 -gt 60 ]]
8322023-10-11T22:22:08.148Z+ sleep 1
8332023-10-11T22:22:08.194Z+ retry=9
8342023-10-11T22:22:08.654Z+++ zoneadm list -n
8352023-10-11T22:22:08.694Z+++ grep oxz_ntp
8362023-10-11T22:22:09.139Z++ pfexec svcs -z -Hostate oxide/ntp
8372023-10-11T22:22:09.178Zsvcs: zone '-Hostate': entity not found
8382023-10-11T22:22:11.794Z++ true
8392023-10-11T22:22:11.893Z+ [[ '' != online ]]
8402023-10-11T22:22:11.966Z+ [[ 9 -gt 60 ]]
8412023-10-11T22:22:12.026Z+ sleep 1
8422023-10-11T22:22:12.075Z+ retry=10
8432023-10-11T22:22:12.115Z+++ zoneadm list -n
8442023-10-11T22:22:12.167Z+++ grep oxz_ntp
8452023-10-11T22:22:12.205Z++ pfexec svcs -z -Hostate oxide/ntp
8462023-10-11T22:22:12.652Zsvcs: zone '-Hostate': entity not found
8472023-10-11T22:22:12.693Z++ true
8482023-10-11T22:22:13.951Z+ [[ '' != online ]]
8492023-10-11T22:22:14.045Z+ [[ 10 -gt 60 ]]
8502023-10-11T22:22:14.183Z+ sleep 1
8512023-10-11T22:22:14.320Z+ retry=11
8522023-10-11T22:22:14.814Z+++ zoneadm list -n
8532023-10-11T22:22:14.887Z+++ grep oxz_ntp
8542023-10-11T22:22:14.969Z++ pfexec svcs -z -Hostate oxide/ntp
8552023-10-11T22:22:15.517Zsvcs: zone '-Hostate': entity not found
8562023-10-11T22:22:15.549Z++ true
8572023-10-11T22:22:15.584Z+ [[ '' != online ]]
8582023-10-11T22:22:15.627Z+ [[ 11 -gt 60 ]]
8592023-10-11T22:22:15.693Z+ sleep 1
8602023-10-11T22:22:16.194Z+ retry=12
8612023-10-11T22:22:16.244Z+++ zoneadm list -n
8622023-10-11T22:22:16.277Z+++ grep oxz_ntp
8632023-10-11T22:22:16.314Z++ pfexec svcs -z -Hostate oxide/ntp
8642023-10-11T22:22:16.344Zsvcs: zone '-Hostate': entity not found
8652023-10-11T22:22:16.375Z++ true
8662023-10-11T22:22:16.403Z+ [[ '' != online ]]
8672023-10-11T22:22:16.433Z+ [[ 12 -gt 60 ]]
8682023-10-11T22:22:16.469Z+ sleep 1
8692023-10-11T22:22:16.927Z+ retry=13
8702023-10-11T22:22:18.204Z+++ zoneadm list -n
8712023-10-11T22:22:18.675Z+++ grep oxz_ntp
8722023-10-11T22:22:18.739Z++ pfexec svcs -z -Hostate oxide/ntp
8732023-10-11T22:22:18.784Zsvcs: zone '-Hostate': entity not found
8742023-10-11T22:22:18.822Z++ true
8752023-10-11T22:22:18.848Z+ [[ '' != online ]]
8762023-10-11T22:22:18.875Z+ [[ 13 -gt 60 ]]
8772023-10-11T22:22:18.901Z+ sleep 1
8782023-10-11T22:22:18.925Z+ retry=14
8792023-10-11T22:22:18.953Z+++ zoneadm list -n
8802023-10-11T22:22:18.980Z+++ grep oxz_ntp
8812023-10-11T22:22:19.011Z++ pfexec svcs -z oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c -Hostate oxide/ntp
8822023-10-11T22:22:19.048Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8832023-10-11T22:22:19.095Z++ true
8842023-10-11T22:22:19.124Z+ [[ '' != online ]]
8852023-10-11T22:22:19.154Z+ [[ 14 -gt 60 ]]
8862023-10-11T22:22:19.193Z+ sleep 1
8872023-10-11T22:22:19.227Z+ retry=15
8882023-10-11T22:22:19.261Z+++ zoneadm list -n
8892023-10-11T22:22:19.294Z+++ grep oxz_ntp
8902023-10-11T22:22:19.336Z++ pfexec svcs -z oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c -Hostate oxide/ntp
8912023-10-11T22:22:19.495Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8922023-10-11T22:22:19.528Z++ true
8932023-10-11T22:22:19.556Z+ [[ '' != online ]]
8942023-10-11T22:22:19.607Z+ [[ 15 -gt 60 ]]
8952023-10-11T22:22:19.687Z+ sleep 1
8962023-10-11T22:22:19.733Z+ retry=16
8972023-10-11T22:22:19.764Z+++ zoneadm list -n
8982023-10-11T22:22:19.800Z+++ grep oxz_ntp
8992023-10-11T22:22:20.274Z++ pfexec svcs -z oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c -Hostate oxide/ntp
9002023-10-11T22:22:20.309ZWaited for chrony: 25s
9012023-10-11T22:22:20.339Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9022023-10-11T22:22:20.369Z++ true
9032023-10-11T22:22:20.409Z+ [[ '' != online ]]
9042023-10-11T22:22:20.446Z+ [[ 16 -gt 60 ]]
9052023-10-11T22:22:20.889Z+ sleep 1
9062023-10-11T22:22:20.943Z+ retry=17
9072023-10-11T22:22:20.985Z+++ zoneadm list -n
9082023-10-11T22:22:21.026Z+++ grep oxz_ntp
9092023-10-11T22:22:21.495Z++ pfexec svcs -z oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c -Hostate oxide/ntp
9102023-10-11T22:22:21.751Z+ [[ disabled != online ]]
9112023-10-11T22:22:21.810Z+ [[ 17 -gt 60 ]]
9122023-10-11T22:22:21.849Z+ sleep 1
9132023-10-11T22:22:21.885Z+ retry=18
9142023-10-11T22:22:21.923Z+++ zoneadm list -n
9152023-10-11T22:22:22.399Z+++ grep oxz_ntp
9162023-10-11T22:22:22.495Z++ pfexec svcs -z oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c -Hostate oxide/ntp
9172023-10-11T22:22:22.571Z+ [[ disabled != online ]]
9182023-10-11T22:22:22.684Z+ [[ 18 -gt 60 ]]
9192023-10-11T22:22:23.211Z+ sleep 1
9202023-10-11T22:22:23.261Z+ retry=19
9212023-10-11T22:22:23.304Z+++ zoneadm list -n
9222023-10-11T22:22:23.967Z+++ grep oxz_ntp
9232023-10-11T22:22:24.018Z++ pfexec svcs -z oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c -Hostate oxide/ntp
9242023-10-11T22:22:24.059Z+ [[ disabled != online ]]
9252023-10-11T22:22:24.094Z+ [[ 19 -gt 60 ]]
9262023-10-11T22:22:24.576Z+ sleep 1
9272023-10-11T22:22:24.607Z+ retry=20
9282023-10-11T22:22:24.707Z+++ zoneadm list -n
9292023-10-11T22:22:24.930Z+++ grep oxz_ntp
9302023-10-11T22:22:24.968Z++ pfexec svcs -z oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c -Hostate oxide/ntp
9312023-10-11T22:22:25.458Z+ [[ disabled != online ]]
9322023-10-11T22:22:25.501Z+ [[ 20 -gt 60 ]]
9332023-10-11T22:22:25.528Z+ sleep 1
9342023-10-11T22:22:25.584Z+ retry=21
9352023-10-11T22:22:25.652Z+++ zoneadm list -n
9362023-10-11T22:22:25.705Z+++ grep oxz_ntp
9372023-10-11T22:22:25.741Z++ pfexec svcs -z oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c -Hostate oxide/ntp
9382023-10-11T22:22:25.848Z+ [[ disabled != online ]]
9392023-10-11T22:22:26.336Z+ [[ 21 -gt 60 ]]
9402023-10-11T22:22:26.434Z+ sleep 1
9412023-10-11T22:22:26.527Z+ retry=22
9422023-10-11T22:22:27.034Z+++ zoneadm list -n
9432023-10-11T22:22:27.185Z+++ grep oxz_ntp
9442023-10-11T22:22:37.607Z++ pfexec svcs -z oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c -Hostate oxide/ntp
9452023-10-11T22:22:38.035Z+ [[ disabled != online ]]
9462023-10-11T22:22:38.222Z+ [[ 22 -gt 60 ]]
9472023-10-11T22:22:38.406Z+ sleep 1
9482023-10-11T22:22:38.654Z+ retry=23
9492023-10-11T22:22:38.862Z+++ zoneadm list -n
9502023-10-11T22:22:38.996Z+++ grep oxz_ntp
9512023-10-11T22:22:39.204Z++ pfexec svcs -z oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c -Hostate oxide/ntp
9522023-10-11T22:22:39.373Z+ [[ disabled != online ]]
9532023-10-11T22:22:39.517Z+ [[ 23 -gt 60 ]]
9542023-10-11T22:22:39.879Z+ sleep 1
9552023-10-11T22:22:40.073Z+ retry=24
9562023-10-11T22:22:40.230Z+++ zoneadm list -n
9572023-10-11T22:22:40.319Z+++ grep oxz_ntp
9582023-10-11T22:22:40.438Z++ pfexec svcs -z oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c -Hostate oxide/ntp
9592023-10-11T22:22:40.574Z+ [[ disabled != online ]]
9602023-10-11T22:22:40.956Z+ [[ 24 -gt 60 ]]
9612023-10-11T22:22:41.866Z+ sleep 1
9622023-10-11T22:22:42.419Z+ retry=25
9632023-10-11T22:22:42.703Z+++ zoneadm list -n
9642023-10-11T22:22:42.921Z+++ grep oxz_ntp
9652023-10-11T22:22:43.270Z++ pfexec svcs -z oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c -Hostate oxide/ntp
9662023-10-11T22:22:43.970Z+ [[ online != online ]]
9672023-10-11T22:22:44.058Z+ echo 'Waited for chrony: 25s'
9682023-10-11T22:22:44.208Z+ export RUST_BACKTRACE=1
9692023-10-11T22:22:44.428Z+ RUST_BACKTRACE=1
9702023-10-11T22:22:44.521Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9712023-10-11T22:22:44.787Z+ ./tests/bootstrap
9722023-10-11T22:22:44.893ZUsing external DNS server at 10.151.1.164:53
9732023-10-11T22:22:45.099Z2023-10-11 22:22:08.717659818 UTC: attempting to log into API
9742023-10-11T22:22:45.134Z2023-10-11 22:22:23.629627935 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9752023-10-11T22:22:45.383Z2023-10-11 22:22:24.630695041 UTC: attempting to log into API
9762023-10-11T22:22:45.613Z2023-10-11 22:22:39.663982716 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9772023-10-11T22:22:45.797Z2023-10-11 22:22:40.665413087 UTC: attempting to log into API
9782023-10-11T22:22:55.871Z2023-10-11 22:22:55.698712113 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9792023-10-11T22:23:04.932Z2023-10-11 22:22:56.700296797 UTC: attempting to log into API
9802023-10-11T22:23:11.982Z2023-10-11 22:23:11.743460092 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9812023-10-11T22:23:12.714Z2023-10-11 22:23:12.744467907 UTC: attempting to log into API
 
9822023-10-11T22:23:26.378ZOct 11 22:23:25 bukin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9832023-10-11T22:23:27.755Z2023-10-11 22:23:27.763302284 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9842023-10-11T22:23:28.752Z2023-10-11 22:23:28.761437934 UTC: attempting to log into API
 
9852023-10-11T22:23:32.916ZOct 11 22:23:32 bukin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9862023-10-11T22:23:40.750ZOct 11 22:23:40 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9872023-10-11T22:23:43.789Z2023-10-11 22:23:43.757992574 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9882023-10-11T22:23:44.816Z2023-10-11 22:23:44.759013579 UTC: attempting to log into API
 
9892023-10-11T22:23:46.776ZOct 11 22:23:46 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9902023-10-11T22:23:53.883ZOct 11 22:23:53 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9912023-10-11T22:23:59.821Z2023-10-11 22:23:59.792670450 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9922023-10-11T22:24:00.829Z2023-10-11 22:24:00.793390381 UTC: attempting to log into API
9932023-10-11T22:24:16.871Z2023-10-11 22:24:15.826393713 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9942023-10-11T22:24:16.958Z2023-10-11 22:24:16.827330516 UTC: attempting to log into API
9952023-10-11T22:24:31.891Z2023-10-11 22:24:31.861834056 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9962023-10-11T22:24:32.890Z2023-10-11 22:24:32.862900591 UTC: attempting to log into API
9972023-10-11T22:24:33.733Z2023-10-11 22:24:33.640764394 UTC: login succeeded
9982023-10-11T22:24:33.767Zwaiting for nexus to come up...
9992023-10-11T22:24:33.798Zcreating IP pool... 10.151.1.175 - 10.151.1.227
10002023-10-11T22:24:33.832Zensuring datasets are ready...
10012023-10-11T22:24:36.962Zlet's roll.
10022023-10-11T22:24:37.000Z+ rm ./tests/bootstrap
10032023-10-11T22:24:37.041Z+ for test_bin in tests/*
10042023-10-11T22:24:37.535Z+ ./tests/bootstrap-0b5a6903c7c17be7
10052023-10-11T22:24:37.605Z
10062023-10-11T22:24:37.638Zrunning 0 tests
10072023-10-11T22:24:37.679Z
10082023-10-11T22:24:37.714Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10092023-10-11T22:24:37.744Z
10102023-10-11T22:24:37.780Z+ for test_bin in tests/*
10112023-10-11T22:24:37.809Z+ ./tests/end_to_end_tests-6da9107f9543f44b
10122023-10-11T22:24:37.844Z
10132023-10-11T22:24:37.879Zrunning 2 tests
10142023-10-11T22:24:37.919Ztest no_spoof::no_spoof_here ... ok
10152023-10-11T22:25:36.566Ztest instance_launch::instance_launch has been running for over 60 seconds
10162023-10-11T22:27:30.628Ztest instance_launch::instance_launch ... ok
10172023-10-11T22:27:30.661Z
10182023-10-11T22:27:31.102Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 174.07s
10192023-10-11T22:27:31.159Z
10202023-10-11T22:27:31.620Z+ _exit_trap
10212023-10-11T22:27:31.659Z+ local status=0
10222023-10-11T22:27:31.702Z+ [[ 0 -eq 0 ]]
10232023-10-11T22:27:31.738Z+ exit 0
10242023-10-11T22:27:31.775Zprocess exited: duration 469760 ms, exit code 0
 
10252023-10-11T22:27:31.874Zfound 47 output files
10262023-10-11T22:27:31.943Zuploading: /var/svc/log/oxide-sled-agent:default.log (395536 bytes)
10272023-10-11T22:27:33.870Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 395536 -> 396099
10282023-10-11T22:27:35.407Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10292023-10-11T22:27:35.488Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0a2b3a41-8217-4efb-80bd-f26bc57c98ff/root/var/svc/log/oxide-cockroachdb:default.log (5342 bytes)
10302023-10-11T22:27:37.154Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0a2b3a41-8217-4efb-80bd-f26bc57c98ff/root/var/svc/log/oxide-cockroachdb:default.log
10312023-10-11T22:27:37.186Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_33ea0540-6cdf-4a41-803e-744a0f0add1f/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10322023-10-11T22:27:38.283Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_33ea0540-6cdf-4a41-803e-744a0f0add1f/root/var/svc/log/oxide-crucible-agent:default.log
10332023-10-11T22:27:38.798Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_33ea0540-6cdf-4a41-803e-744a0f0add1f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10342023-10-11T22:27:39.411Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_33ea0540-6cdf-4a41-803e-744a0f0add1f/root/var/svc/log/oxide-crucible-downstairs:default.log
10352023-10-11T22:27:39.894Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_30aebbbc-6885-4aea-a6e9-f4a0fcaa926b/root/var/svc/log/oxide-internal_dns:default.log (102315 bytes)
10362023-10-11T22:27:41.458Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_30aebbbc-6885-4aea-a6e9-f4a0fcaa926b/root/var/svc/log/oxide-internal_dns:default.log
10372023-10-11T22:27:41.492Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84f5e880-b9cc-4522-a0fe-ed8738d8fb1f/root/var/svc/log/oxide-crucible-agent:default.log (14583 bytes)
10382023-10-11T22:27:43.816Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84f5e880-b9cc-4522-a0fe-ed8738d8fb1f/root/var/svc/log/oxide-crucible-agent:default.log
10392023-10-11T22:27:43.923Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84f5e880-b9cc-4522-a0fe-ed8738d8fb1f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10402023-10-11T22:27:45.172Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84f5e880-b9cc-4522-a0fe-ed8738d8fb1f/root/var/svc/log/oxide-crucible-downstairs:default.log
10412023-10-11T22:27:45.231Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84f5e880-b9cc-4522-a0fe-ed8738d8fb1f/root/var/svc/log/oxide-crucible-downstairs:downstairs-ba55d214-1d1d-47e3-a0ef-f0b2ffa88c65.log (3491 bytes)
10422023-10-11T22:27:46.766Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84f5e880-b9cc-4522-a0fe-ed8738d8fb1f/root/var/svc/log/oxide-crucible-downstairs:downstairs-ba55d214-1d1d-47e3-a0ef-f0b2ffa88c65.log
10432023-10-11T22:27:46.796Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_5d26dc76-9740-4ab3-a2f3-8334477b4cdd/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10442023-10-11T22:27:47.888Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_5d26dc76-9740-4ab3-a2f3-8334477b4cdd/root/var/svc/log/oxide-crucible-pantry:default.log
10452023-10-11T22:27:47.937Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_aeee75ad-14ab-4034-a96c-9dee68c5064f/root/var/svc/log/oxide-crucible-pantry:default.log (2337 bytes)
10462023-10-11T22:27:49.039Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_aeee75ad-14ab-4034-a96c-9dee68c5064f/root/var/svc/log/oxide-crucible-pantry:default.log
10472023-10-11T22:27:49.086Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e51f930f-3bc1-4c68-96cb-ed93f51a779b/root/var/svc/log/oxide-external_dns:default.log (15610 bytes)
10482023-10-11T22:27:49.256Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e51f930f-3bc1-4c68-96cb-ed93f51a779b/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15610 -> 16869
10492023-10-11T22:27:50.811Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e51f930f-3bc1-4c68-96cb-ed93f51a779b/root/var/svc/log/oxide-external_dns:default.log
10502023-10-11T22:27:50.845Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c7b6d97b-d2f9-4335-a3e9-466aa351f48d/root/var/svc/log/oxide-nexus:default.log (533872 bytes)
10512023-10-11T22:27:52.273Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c7b6d97b-d2f9-4335-a3e9-466aa351f48d/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 533872 -> 535089
10522023-10-11T22:27:53.387Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c7b6d97b-d2f9-4335-a3e9-466aa351f48d/root/var/svc/log/oxide-nexus:default.log
10532023-10-11T22:27:53.460Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb77703c-1786-4e30-ad5e-b29c535361dd/root/var/svc/log/oxide-crucible-agent:default.log (13684 bytes)
10542023-10-11T22:27:55.285Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb77703c-1786-4e30-ad5e-b29c535361dd/root/var/svc/log/oxide-crucible-agent:default.log
10552023-10-11T22:27:55.338Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb77703c-1786-4e30-ad5e-b29c535361dd/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10562023-10-11T22:27:57.123Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb77703c-1786-4e30-ad5e-b29c535361dd/root/var/svc/log/oxide-crucible-downstairs:default.log
10572023-10-11T22:27:57.572Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb77703c-1786-4e30-ad5e-b29c535361dd/root/var/svc/log/oxide-crucible-downstairs:downstairs-4d7fe510-b97e-4cbe-810e-33422f817a0a.log (3494 bytes)
10582023-10-11T22:28:00.022Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb77703c-1786-4e30-ad5e-b29c535361dd/root/var/svc/log/oxide-crucible-downstairs:downstairs-4d7fe510-b97e-4cbe-810e-33422f817a0a.log
10592023-10-11T22:28:00.058Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_e1555a85-e210-4fcf-891b-285614384450/root/var/svc/log/oxide-external_dns:default.log (15614 bytes)
10602023-10-11T22:28:00.890Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_e1555a85-e210-4fcf-891b-285614384450/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15614 -> 19391
10612023-10-11T22:28:02.884Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_e1555a85-e210-4fcf-891b-285614384450/root/var/svc/log/oxide-external_dns:default.log
10622023-10-11T22:28:02.963Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_812313bc-d0ca-456e-8f58-f0e31fe5ed4a/root/var/svc/log/oxide-nexus:default.log (4350323 bytes)
10632023-10-11T22:28:08.469Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_812313bc-d0ca-456e-8f58-f0e31fe5ed4a/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4350323 -> 4410326
10642023-10-11T22:28:09.589Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_812313bc-d0ca-456e-8f58-f0e31fe5ed4a/root/var/svc/log/oxide-nexus:default.log
10652023-10-11T22:28:09.623Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_711a5703-7e21-4c04-8a58-6a8659cdce49/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10662023-10-11T22:28:10.702Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_711a5703-7e21-4c04-8a58-6a8659cdce49/root/var/svc/log/oxide-crucible-agent:default.log
10672023-10-11T22:28:10.738Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_711a5703-7e21-4c04-8a58-6a8659cdce49/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10682023-10-11T22:28:11.820Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_711a5703-7e21-4c04-8a58-6a8659cdce49/root/var/svc/log/oxide-crucible-downstairs:default.log
10692023-10-11T22:28:11.858Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_1b43c4d6-7764-4d32-a634-10fcb676d242/root/var/svc/log/oxide-oximeter:default.log (187487 bytes)
10702023-10-11T22:28:19.047Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_1b43c4d6-7764-4d32-a634-10fcb676d242/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 187487 -> 226068
10712023-10-11T22:28:20.531Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_1b43c4d6-7764-4d32-a634-10fcb676d242/root/var/svc/log/oxide-oximeter:default.log
10722023-10-11T22:28:20.565Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_70994f51-87b1-44ae-b62b-f5f48ab581ce/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10732023-10-11T22:28:21.396Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_70994f51-87b1-44ae-b62b-f5f48ab581ce/root/var/svc/log/oxide-cockroachdb:default.log
10742023-10-11T22:28:21.443Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_aa67f950-9078-412b-a717-9c589ef1f6fb/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
10752023-10-11T22:28:22.546Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_aa67f950-9078-412b-a717-9c589ef1f6fb/root/var/svc/log/oxide-cockroachdb:default.log
10762023-10-11T22:28:22.581Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_3bd621fa-1214-49e2-a96d-7e882108bb9d/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10772023-10-11T22:28:24.526Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_3bd621fa-1214-49e2-a96d-7e882108bb9d/root/var/svc/log/oxide-crucible-agent:default.log
10782023-10-11T22:28:24.567Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_3bd621fa-1214-49e2-a96d-7e882108bb9d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10792023-10-11T22:28:26.076Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_3bd621fa-1214-49e2-a96d-7e882108bb9d/root/var/svc/log/oxide-crucible-downstairs:default.log
10802023-10-11T22:28:26.107Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_86322a6c-72f5-4609-a0d9-bc542ee14e75/root/var/svc/log/oxide-crucible-agent:default.log (24828 bytes)
10812023-10-11T22:28:27.852Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_86322a6c-72f5-4609-a0d9-bc542ee14e75/root/var/svc/log/oxide-crucible-agent:default.log
10822023-10-11T22:28:27.884Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_86322a6c-72f5-4609-a0d9-bc542ee14e75/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10832023-10-11T22:28:28.986Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_86322a6c-72f5-4609-a0d9-bc542ee14e75/root/var/svc/log/oxide-crucible-downstairs:default.log
10842023-10-11T22:28:29.015Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_86322a6c-72f5-4609-a0d9-bc542ee14e75/root/var/svc/log/oxide-crucible-downstairs:downstairs-06a29677-7492-480d-9656-3144c13f245a.log (3491 bytes)
10852023-10-11T22:28:30.098Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_86322a6c-72f5-4609-a0d9-bc542ee14e75/root/var/svc/log/oxide-crucible-downstairs:downstairs-06a29677-7492-480d-9656-3144c13f245a.log
10862023-10-11T22:28:30.128Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_86322a6c-72f5-4609-a0d9-bc542ee14e75/root/var/svc/log/oxide-crucible-downstairs:downstairs-2a71b340-8e11-46f2-bcf0-c645b8d0536f.log (5066 bytes)
10872023-10-11T22:28:31.324Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_86322a6c-72f5-4609-a0d9-bc542ee14e75/root/var/svc/log/oxide-crucible-downstairs:downstairs-2a71b340-8e11-46f2-bcf0-c645b8d0536f.log
10882023-10-11T22:28:31.357Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_aab1a11e-997e-4222-bc4b-fd36a182da26/root/var/svc/log/oxide-crucible-agent:default.log (13690 bytes)
10892023-10-11T22:28:32.521Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_aab1a11e-997e-4222-bc4b-fd36a182da26/root/var/svc/log/oxide-crucible-agent:default.log
10902023-10-11T22:28:32.629Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_aab1a11e-997e-4222-bc4b-fd36a182da26/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10912023-10-11T22:28:34.141Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_aab1a11e-997e-4222-bc4b-fd36a182da26/root/var/svc/log/oxide-crucible-downstairs:default.log
10922023-10-11T22:28:34.167Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_aab1a11e-997e-4222-bc4b-fd36a182da26/root/var/svc/log/oxide-crucible-downstairs:downstairs-f0530155-e2ac-4648-a630-c66b0e980f84.log (5064 bytes)
10932023-10-11T22:28:35.241Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_aab1a11e-997e-4222-bc4b-fd36a182da26/root/var/svc/log/oxide-crucible-downstairs:downstairs-f0530155-e2ac-4648-a630-c66b0e980f84.log
10942023-10-11T22:28:35.281Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_f140fa3d-c58f-462f-9e8a-25b34f66f59e/root/var/svc/log/oxide-crucible-pantry:default.log (2331 bytes)
10952023-10-11T22:28:36.374Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_f140fa3d-c58f-462f-9e8a-25b34f66f59e/root/var/svc/log/oxide-crucible-pantry:default.log
10962023-10-11T22:28:36.405Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_62893e28-e943-4a38-94fd-5cf521eff62f/root/var/svc/log/oxide-internal_dns:default.log (99561 bytes)
10972023-10-11T22:28:36.803Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_62893e28-e943-4a38-94fd-5cf521eff62f/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99561 -> 103317
10982023-10-11T22:28:37.883Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_62893e28-e943-4a38-94fd-5cf521eff62f/root/var/svc/log/oxide-internal_dns:default.log
10992023-10-11T22:28:37.907Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_7d28c35c-6dc8-4197-91a6-0d7f923a9de3/root/var/svc/log/oxide-internal_dns:default.log (99875 bytes)
11002023-10-11T22:28:38.201Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_7d28c35c-6dc8-4197-91a6-0d7f923a9de3/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99875 -> 103630
11012023-10-11T22:28:39.292Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_7d28c35c-6dc8-4197-91a6-0d7f923a9de3/root/var/svc/log/oxide-internal_dns:default.log
11022023-10-11T22:28:39.328Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c/root/var/svc/log/oxide-ntp:default.log (1118 bytes)
11032023-10-11T22:28:40.408Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c/root/var/svc/log/oxide-ntp:default.log
11042023-10-11T22:28:40.937Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_dee63862-e271-4474-9da6-48f35240610f/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
11052023-10-11T22:28:41.514Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_dee63862-e271-4474-9da6-48f35240610f/root/var/svc/log/oxide-cockroachdb:default.log
11062023-10-11T22:28:41.552Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_27cc08d9-b7bf-43be-ab12-a33556e5fe80/root/var/svc/log/oxide-crucible-agent:default.log (14579 bytes)
11072023-10-11T22:28:43.140Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_27cc08d9-b7bf-43be-ab12-a33556e5fe80/root/var/svc/log/oxide-crucible-agent:default.log
11082023-10-11T22:28:43.577Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_27cc08d9-b7bf-43be-ab12-a33556e5fe80/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11092023-10-11T22:28:44.265Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_27cc08d9-b7bf-43be-ab12-a33556e5fe80/root/var/svc/log/oxide-crucible-downstairs:default.log
11102023-10-11T22:28:44.304Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_27cc08d9-b7bf-43be-ab12-a33556e5fe80/root/var/svc/log/oxide-crucible-downstairs:downstairs-e055b7ac-b1e2-4a32-bd31-719451934963.log (5063 bytes)
11112023-10-11T22:28:45.398Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_27cc08d9-b7bf-43be-ab12-a33556e5fe80/root/var/svc/log/oxide-crucible-downstairs:downstairs-e055b7ac-b1e2-4a32-bd31-719451934963.log
11122023-10-11T22:28:45.434Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_acf9e617-2dd3-4366-b1d9-367020073dd2/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
11132023-10-11T22:28:48.215Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_acf9e617-2dd3-4366-b1d9-367020073dd2/root/var/svc/log/oxide-crucible-agent:default.log
11142023-10-11T22:28:48.246Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_acf9e617-2dd3-4366-b1d9-367020073dd2/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11152023-10-11T22:28:48.281Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_acf9e617-2dd3-4366-b1d9-367020073dd2/root/var/svc/log/oxide-crucible-downstairs:default.log
11162023-10-11T22:28:48.777Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_b32498ac-613c-4c19-a15a-5ac58e031f9f/root/var/svc/log/oxide-nexus:default.log (522744 bytes)
11172023-10-11T22:28:49.688Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_b32498ac-613c-4c19-a15a-5ac58e031f9f/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 522744 -> 643980
11182023-10-11T22:28:50.760Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_b32498ac-613c-4c19-a15a-5ac58e031f9f/root/var/svc/log/oxide-nexus:default.log
11192023-10-11T22:28:50.793Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_c465d793-688d-4b87-bb4b-daa6b3228094/root/var/svc/log/oxide-clickhouse:default.log (674381 bytes)
11202023-10-11T22:28:52.457Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_c465d793-688d-4b87-bb4b-daa6b3228094/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 674381 -> 941211
11212023-10-11T22:28:53.534Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_c465d793-688d-4b87-bb4b-daa6b3228094/root/var/svc/log/oxide-clickhouse:default.log
11222023-10-11T22:28:53.561Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_583f17c9-4971-4c55-9dec-24c7261fb115/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
11232023-10-11T22:28:55.102Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_583f17c9-4971-4c55-9dec-24c7261fb115/root/var/svc/log/oxide-cockroachdb:default.log
11242023-10-11T22:28:55.149Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c/root/var/log/chrony/measurements.log (9179 bytes)
11252023-10-11T22:28:55.243Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c/root/var/log/chrony/measurements.log" changed size mid upload: 9179 -> 11782
11262023-10-11T22:28:56.343Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c/root/var/log/chrony/measurements.log
11272023-10-11T22:28:56.372Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c/root/var/log/chrony/statistics.log (5382 bytes)
11282023-10-11T22:28:56.415Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c/root/var/log/chrony/statistics.log" changed size mid upload: 5382 -> 7254
11292023-10-11T22:28:57.499Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c/root/var/log/chrony/statistics.log
11302023-10-11T22:28:57.524Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c/root/var/log/chrony/tracking.log (1456 bytes)
11312023-10-11T22:28:57.562Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c/root/var/log/chrony/tracking.log" changed size mid upload: 1456 -> 1720
11322023-10-11T22:28:58.652Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_9cdb2ecd-63be-4421-b4db-d30277101d9c/root/var/log/chrony/tracking.log