01HBW7TQ8HQPD67WGFPV5BNTG3: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.23",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HBW7WKF3K8C079F21993W0B1

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-04T03:15:17.697Zjob dependencies complete; ready to run (waiting for 52 m 56 s)
22023-10-04T03:16:00.060Zjob assigned to worker 01HBWAXKMRAJR55HBNV76SF2H0 (queued for 42 s)
32023-10-04T03:16:00.147Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-10-04T03:16:00.205Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-10-04T03:16:00.240Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-10-04T03:16:00.273Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-04T03:16:00.306Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-10-04T03:16:00.352ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-04T03:16:00.384ZOxide Helios Version helios-2.0.22085 64-bit
102023-10-04T03:16:00.418ZConfiguring devices.
112023-10-04T03:16:00.451Z#################################################
122023-10-04T03:16:00.485Z#### ############# ##
132023-10-04T03:16:00.518Z### ### ############ ##
142023-10-04T03:16:00.552Z## ### # ## ### ## Oxide Computer Company ##
152023-10-04T03:16:00.585Z## ## ## ### # ### ##
162023-10-04T03:16:00.622Z## # ### #### #### This Station Under ##
172023-10-04T03:16:00.652Z### ### #### # ### Computer Control ##
182023-10-04T03:16:00.686Z#### #### ### ## ##
192023-10-04T03:16:00.735Z##################################################
202023-10-04T03:16:00.778ZHostname: sock
212023-10-04T03:16:00.812ZLoading smf(7) serice descriptions:
222023-10-04T03:16:01.574Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-e8befc34dc38cbc1.gz
232023-10-04T03:16:03.222Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-e8befc34dc38cbc1.gz
242023-10-04T03:16:03.248Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
252023-10-04T03:16:45.320Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-04T03:16:45.350Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-b03cef630464e50a.gz
272023-10-04T03:17:40.185Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-b03cef630464e50a.gz
282023-10-04T03:17:40.621Zdownloading input: /input/ci-tools/work/caboose-util.gz
292023-10-04T03:17:41.444Zdownloaded input: /input/ci-tools/work/caboose-util.gz
302023-10-04T03:17:41.472Zdownloading input: /input/ci-tools/work/tufaceous.gz
312023-10-04T03:17:46.178Zdownloaded input: /input/ci-tools/work/tufaceous.gz
322023-10-04T03:17:46.209Zdownloading input: /input/package/work/version.txt
332023-10-04T03:17:46.246Zdownloaded input: /input/package/work/version.txt
342023-10-04T03:17:46.282Zdownloading input: /input/package/work/package.tar.gz
352023-10-04T03:18:16.494Zdownloaded input: /input/package/work/package.tar.gz
362023-10-04T03:18:17.424Zdownloading input: /input/package/work/global-zone-packages.tar.gz
372023-10-04T03:19:23.253Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
382023-10-04T03:19:23.282Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
392023-10-04T03:19:44.172Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-04T03:19:44.203Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
412023-10-04T03:21:35.749Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
422023-10-04T03:21:35.773Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
432023-10-04T03:22:23.554Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-04T03:22:23.584Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
452023-10-04T03:23:21.966Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-04T03:23:22.003Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
472023-10-04T03:23:26.992Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-04T03:23:27.020Zdownloading input: /input/package/work/zones/crucible.tar.gz
492023-10-04T03:23:35.801Zdownloaded input: /input/package/work/zones/crucible.tar.gz
502023-10-04T03:23:35.828Zdownloading input: /input/package/work/zones/external-dns.tar.gz
512023-10-04T03:23:40.594Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
522023-10-04T03:23:40.620Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
532023-10-04T03:23:45.596Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
542023-10-04T03:23:45.620Zdownloading input: /input/package/work/zones/ntp.tar.gz
552023-10-04T03:23:45.644Zdownloaded input: /input/package/work/zones/ntp.tar.gz
562023-10-04T03:23:45.668Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
572023-10-04T03:23:50.380Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-04T03:23:50.405Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
592023-10-04T03:24:10.145Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-04T03:24:10.593Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
612023-10-04T03:24:35.836Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-04T03:24:38.317Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
632023-10-04T03:24:54.749Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-04T03:24:54.776Zdownloading input: /input/package/work/zones/overlay.tar.gz
652023-10-04T03:24:54.800Zdownloaded input: /input/package/work/zones/overlay.tar.gz
662023-10-04T03:24:54.826Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
672023-10-04T03:24:58.146Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-04T03:24:58.170Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
692023-10-04T03:25:20.211Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
702023-10-04T03:25:20.236Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
712023-10-04T03:26:36.327Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
722023-10-04T03:26:36.366Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
732023-10-04T03:27:42.130Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
742023-10-04T03:27:42.169Zstarting task 0: "setup"
752023-10-04T03:27:42.214Z++ uname -s
762023-10-04T03:27:42.247Z+ kern=SunOS
772023-10-04T03:27:42.280Z+ case "$kern" in
782023-10-04T03:27:42.337Z+ groupadd -g 12345 build
792023-10-04T03:27:42.373Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
802023-10-04T03:27:44.228Z+ zfs create -o mountpoint=/work rpool/work
812023-10-04T03:27:44.265Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
822023-10-04T03:27:44.311Z+ home_fs=zfs
832023-10-04T03:27:44.348Z+ [[ zfs == autofs ]]
842023-10-04T03:27:44.466Z+ mkdir -p /home/build
852023-10-04T03:27:44.502Z+ chown build:build /home/build /work
862023-10-04T03:27:46.248Z+ chmod 0700 /home/build /work
872023-10-04T03:27:46.295Zprocess exited: duration 4048 ms, exit code 0
 
882023-10-04T03:27:46.360Zstarting task 1: "authentication"
892023-10-04T03:27:46.449Zprocess exited: duration 56 ms, exit code 0
 
902023-10-04T03:27:46.522Zstarting task 2: "build"
912023-10-04T03:27:46.676Z+ trap _exit_trap EXIT
922023-10-04T03:27:46.711Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
932023-10-04T03:27:46.763Zsecpolicy_fs_linkdir: 0x55 = 0x55
942023-10-04T03:27:46.794Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
952023-10-04T03:27:46.830Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
962023-10-04T03:27:46.873Zsecpolicy_fs_linkdir+3: 0 = 0xe5
972023-10-04T03:27:46.910Zsecpolicy_fs_linkdir+4: 0 = 0xb8
982023-10-04T03:27:46.965Zsecpolicy_fs_linkdir+5: 0 = 0x1
992023-10-04T03:27:47.004Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1002023-10-04T03:27:47.038Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1012023-10-04T03:27:47.088Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1022023-10-04T03:27:47.130Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1032023-10-04T03:27:47.164Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1042023-10-04T03:27:47.204Z+ [[ -d /opt/oxide ]]
1052023-10-04T03:27:47.252Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1062023-10-04T03:27:47.294Z+ pfexec mkdir /opt/oxide
1072023-10-04T03:27:47.327Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1082023-10-04T03:27:47.378Z+ [[ -d /opt/oxide-underneath ]]
1092023-10-04T03:27:47.421Z+ cd /opt/oxide-underneath
1102023-10-04T03:27:47.457Z+ pfexec tar ceEp@/f - .
1112023-10-04T03:27:47.489Z+ cd /opt/oxide
1122023-10-04T03:27:47.522Z+ pfexec tar xveEp@/f -
1132023-10-04T03:27:47.561Zx ., 0 bytes, 0 tape blocks
1142023-10-04T03:27:47.607Zx ./opte, 0 bytes, 0 tape blocks
1152023-10-04T03:27:47.691Zx ./opte/bin, 0 bytes, 0 tape blocks
1162023-10-04T03:27:47.727Zx ./opte/bin/opteadm, 46195888 bytes, 90227 tape blocks
1172023-10-04T03:27:47.760Z+ pfexec rm -rf /opt/oxide-underneath
1182023-10-04T03:27:47.792Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1192023-10-04T03:27:47.830Z+ pfexec mkdir /opt/oxide/work
1202023-10-04T03:27:47.872Z+ pfexec chown build:build /opt/oxide/work
1212023-10-04T03:27:48.355Z+ cd /opt/oxide/work
1222023-10-04T03:27:48.392Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1232023-10-04T03:27:48.513ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1242023-10-04T03:27:48.962Zx out/maghemite.tar, 74957312 bytes, 146401 tape blocks
1252023-10-04T03:27:48.994Zx out/omicron-sled-agent.tar, 112248832 bytes, 219236 tape blocks
1262023-10-04T03:27:49.020Zx out/target/test, 75 bytes, 1 tape blocks
1272023-10-04T03:27:49.052Zx out/npuzone/npuzone, 36941864 bytes, 72153 tape blocks
1282023-10-04T03:27:49.085Zx package-manifest.toml, 21303 bytes, 42 tape blocks
1292023-10-04T03:27:49.109Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1302023-10-04T03:27:49.135Zx target/release/omicron-package, 37947056 bytes, 74116 tape blocks
1312023-10-04T03:27:49.163Zx tools/create_virtual_hardware.sh, 2810 bytes, 6 tape blocks
1322023-10-04T03:27:49.190Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1332023-10-04T03:27:49.219Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1342023-10-04T03:27:49.255Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1352023-10-04T03:27:49.298Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1362023-10-04T03:27:49.361Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1372023-10-04T03:27:49.412Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1382023-10-04T03:27:49.458Z
1392023-10-04T03:27:49.491Zreal 1.592558676
1402023-10-04T03:27:49.519Zuser 1.434838319
1412023-10-04T03:27:49.552Zsys 0.281413017
1422023-10-04T03:27:49.587Ztrap 0.000067430
1432023-10-04T03:27:49.618Ztflt 0.000000000
1442023-10-04T03:27:50.060Zdflt 0.000000000
1452023-10-04T03:27:50.098Zkflt 0.000000000
1462023-10-04T03:27:50.144Zlock 0.000000000
1472023-10-04T03:27:50.178Zslp 1.433856411
1482023-10-04T03:27:50.203Zlat 0.032862136
1492023-10-04T03:27:50.227Zstop 0.000124552
1502023-10-04T03:27:50.258Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1512023-10-04T03:27:50.707Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1522023-10-04T03:27:50.734Z+ mkdir tests
1532023-10-04T03:27:50.759Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1542023-10-04T03:27:50.784Z+ ptime -m gunzip
1552023-10-04T03:27:50.808Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-e8befc34dc38cbc1
1562023-10-04T03:27:50.832Z
1572023-10-04T03:27:50.859Zreal 0.140571851
1582023-10-04T03:27:50.913Zuser 0.129690634
1592023-10-04T03:27:50.945Zsys 0.010635871
1602023-10-04T03:27:50.973Ztrap 0.000011389
1612023-10-04T03:27:51.002Ztflt 0.000000000
1622023-10-04T03:27:51.030Zdflt 0.000000000
1632023-10-04T03:27:51.065Zkflt 0.000000000
1642023-10-04T03:27:51.098Zlock 0.000000000
1652023-10-04T03:27:51.546Zslp 0.000019500
1662023-10-04T03:27:51.584Zlat 0.000021230
1672023-10-04T03:27:51.785Zstop 0.000128723
1682023-10-04T03:27:51.811Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-e8befc34dc38cbc1
1692023-10-04T03:27:51.845Z+ chmod a+x tests/bootstrap-e8befc34dc38cbc1
1702023-10-04T03:27:51.874Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1712023-10-04T03:27:51.907Z+ ptime -m gunzip
1722023-10-04T03:27:52.022Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1732023-10-04T03:27:52.059Z
1742023-10-04T03:27:52.113Zreal 2.232632213
1752023-10-04T03:27:52.560Zuser 2.092427105
1762023-10-04T03:27:53.009Zsys 0.139808689
1772023-10-04T03:27:53.065Ztrap 0.000171374
1782023-10-04T03:27:53.099Ztflt 0.000000000
1792023-10-04T03:27:53.125Zdflt 0.000000000
1802023-10-04T03:27:53.152Zkflt 0.000000000
1812023-10-04T03:27:53.180Zlock 0.000000000
1822023-10-04T03:27:53.217Zslp 0.000012580
1832023-10-04T03:27:53.251Zlat 0.000029590
1842023-10-04T03:27:53.278Zstop 0.000125422
1852023-10-04T03:27:53.307Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1862023-10-04T03:27:53.335Z+ chmod a+x tests/bootstrap
1872023-10-04T03:27:53.368Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1882023-10-04T03:27:53.820Z+ ptime -m gunzip
1892023-10-04T03:27:54.702Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-b03cef630464e50a
1902023-10-04T03:27:54.737Z
1912023-10-04T03:27:54.769Zreal 2.538360332
1922023-10-04T03:27:55.214Zuser 2.384418860
1932023-10-04T03:27:55.251Zsys 0.153495110
1942023-10-04T03:27:55.697Ztrap 0.000182366
1952023-10-04T03:27:55.722Ztflt 0.000000000
1962023-10-04T03:27:55.760Zdflt 0.000000000
1972023-10-04T03:27:55.791Zkflt 0.000000000
1982023-10-04T03:27:55.817Zlock 0.000000000
1992023-10-04T03:27:55.843Zslp 0.000022630
2002023-10-04T03:27:55.868Zlat 0.000047591
2012023-10-04T03:27:55.893Zstop 0.000127742
2022023-10-04T03:27:55.918Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-b03cef630464e50a
2032023-10-04T03:27:55.942Z+ chmod a+x tests/end_to_end_tests-b03cef630464e50a
2042023-10-04T03:27:55.967Z+ bmat address ls
2052023-10-04T03:27:55.992ZNAME CIDR FIRST LAST COUNT
2062023-10-04T03:27:56.016Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2072023-10-04T03:27:56.041Z++ bmat address ls -f extra -Ho first,last,count
2082023-10-04T03:27:56.080Z+ set -- 10.151.2.164 10.151.2.227 64
2092023-10-04T03:27:56.598Z+ EXTRA_IP_START=10.151.2.164
2102023-10-04T03:27:56.635Z+ EXTRA_IP_END=10.151.2.227
2112023-10-04T03:27:56.674Z+ EXTRA_IP_COUNT=64
2122023-10-04T03:27:56.708Z+ (( EXTRA_IP_COUNT >= 32 ))
2132023-10-04T03:27:56.745Z+ EXTRA_IP_BASE=10.151.2
2142023-10-04T03:27:56.786Z+ EXTRA_IP_FOCTET=164
2152023-10-04T03:27:56.816Z+ EXTRA_IP_LOCTET=227
2162023-10-04T03:27:56.848Z+ SERVICE_IP_POOL_START=10.151.2.164
2172023-10-04T03:27:56.877Z+ SERVICE_IP_POOL_END=10.151.2.173
2182023-10-04T03:27:56.907Z+ DNS_IP1=10.151.2.164
2192023-10-04T03:27:56.939Z+ DNS_IP2=10.151.2.165
2202023-10-04T03:27:56.970Z+ UPLINK_IP=10.151.2.174
2212023-10-04T03:27:57.049Z+ IPPOOL_START=10.151.2.175
2222023-10-04T03:27:57.088Z+ IPPOOL_END=10.151.2.227
2232023-10-04T03:27:57.120Z++ ipadm show-addr -po type,addr
2242023-10-04T03:27:57.152Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2252023-10-04T03:27:57.183Z+ GATEWAY_IP=10.151.2.100
2262023-10-04T03:27:57.211Z+ [[ -n 10.151.2.100 ]]
2272023-10-04T03:27:57.289Z+ ping -s 10.151.2.100 56 1
2282023-10-04T03:27:57.320ZPING 10.151.2.100: 56 data bytes
2292023-10-04T03:27:57.347Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2302023-10-04T03:27:57.371Z
2312023-10-04T03:27:57.400Z----10.151.2.100 PING Statistics----
2322023-10-04T03:27:57.428Z1 packets transmitted, 1 packets received, 0% packet loss
2332023-10-04T03:27:57.455Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2342023-10-04T03:27:57.480Z++ arp -an
2352023-10-04T03:27:57.508Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2362023-10-04T03:27:57.543Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2372023-10-04T03:27:57.570Z+ routeadm -e ipv4-forwarding -u
2382023-10-04T03:27:57.610Z+ PXA_START=10.151.2.164
2392023-10-04T03:27:57.636Z+ PXA_END=10.151.2.227
2402023-10-04T03:27:57.662Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2412023-10-04T03:27:57.688Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2422023-10-04T03:27:57.711Z+ ZPOOL_VDEV_DIR=/scratch
2432023-10-04T03:27:57.737Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2442023-10-04T03:27:57.761Z+++ dirname ./tools/create_virtual_hardware.sh
2452023-10-04T03:27:57.785Z++ cd ./tools
2462023-10-04T03:27:57.809Z++ pwd
2472023-10-04T03:27:57.833Z+ SOURCE_DIR=/opt/oxide/work/tools
2482023-10-04T03:27:57.857Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2492023-10-04T03:27:57.881Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2502023-10-04T03:27:57.906Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2512023-10-04T03:27:57.930Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2522023-10-04T03:27:57.957Z++ dladm show-phys -p -o LINK
2532023-10-04T03:27:57.985Z++ head -1
2542023-10-04T03:27:58.010Z+ PHYSICAL_LINK=igb0
2552023-10-04T03:27:58.039Z+ echo 'Using igb0 as physical link'
2562023-10-04T03:27:58.072ZUsing igb0 as physical link
2572023-10-04T03:27:58.096Z+ ensure_run_as_root
2582023-10-04T03:27:58.120Z++ id -u
2592023-10-04T03:27:58.147Z+ [[ 0 -ne 0 ]]
2602023-10-04T03:27:58.171Z+ ensure_zpools
2612023-10-04T03:27:58.196Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2622023-10-04T03:27:58.219Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2632023-10-04T03:27:58.243Z+ readarray -t ZPOOLS
2642023-10-04T03:27:58.267Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2652023-10-04T03:27:58.291Z++ sed 's/[ ",]//g'
2662023-10-04T03:27:58.330Z+ for ZPOOL in "${ZPOOLS[@]}"
2672023-10-04T03:27:58.356Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2682023-10-04T03:27:58.380ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2692023-10-04T03:27:58.410Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2702023-10-04T03:27:58.443Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2712023-10-04T03:27:58.472Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2722023-10-04T03:27:58.503Z0+0 records in
2732023-10-04T03:27:58.529Z0+0 records out
2742023-10-04T03:27:58.553ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2752023-10-04T03:27:58.578Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2762023-10-04T03:27:58.603ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2772023-10-04T03:27:58.627Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-10-04T03:27:58.651ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2792023-10-04T03:27:58.675Z+ set +x
2802023-10-04T03:27:58.700ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2812023-10-04T03:27:58.724Z++ zpool list -o name
2822023-10-04T03:27:58.749ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2832023-10-04T03:27:58.773Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2842023-10-04T03:27:58.797ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2852023-10-04T03:27:58.821Z+ [[ -z '' ]]
2862023-10-04T03:27:58.863ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2872023-10-04T03:27:58.890Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2882023-10-04T03:27:58.924ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2892023-10-04T03:27:58.978Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2902023-10-04T03:27:59.003ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2912023-10-04T03:27:59.028Z+ set +x
2922023-10-04T03:27:59.052ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2932023-10-04T03:27:59.078Z+ for ZPOOL in "${ZPOOLS[@]}"
2942023-10-04T03:27:59.104ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2952023-10-04T03:27:59.131Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2962023-10-04T03:27:59.160ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2972023-10-04T03:27:59.189Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2982023-10-04T03:27:59.219ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-10-04T03:27:59.247Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3002023-10-04T03:27:59.277ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3012023-10-04T03:27:59.314Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3022023-10-04T03:27:59.386ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3032023-10-04T03:27:59.430Z0+0 records in
3042023-10-04T03:27:59.466ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3052023-10-04T03:27:59.503Z0+0 records out
3062023-10-04T03:27:59.536ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3072023-10-04T03:27:59.569Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3082023-10-04T03:27:59.601ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3092023-10-04T03:27:59.630Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3102023-10-04T03:27:59.657ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3112023-10-04T03:27:59.686Z+ set +x
3122023-10-04T03:27:59.717ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3132023-10-04T03:27:59.743Z++ zpool list -o name
3142023-10-04T03:27:59.771ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3152023-10-04T03:27:59.804Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3162023-10-04T03:27:59.838ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3172023-10-04T03:27:59.874Z+ [[ -z '' ]]
3182023-10-04T03:27:59.902ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3192023-10-04T03:27:59.929Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3202023-10-04T03:27:59.957ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3212023-10-04T03:27:59.986Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3222023-10-04T03:28:00.017ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3232023-10-04T03:28:00.047Z+ set +x
3242023-10-04T03:28:00.076ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3252023-10-04T03:28:00.105Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-10-04T03:28:00.136ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3272023-10-04T03:28:00.170Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3282023-10-04T03:28:00.199ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3292023-10-04T03:28:00.228Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3302023-10-04T03:28:00.267ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3312023-10-04T03:28:00.313Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3322023-10-04T03:28:00.353ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3332023-10-04T03:28:00.391Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3342023-10-04T03:28:00.454ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3352023-10-04T03:28:00.484Z0+0 records in
3362023-10-04T03:28:00.510ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3372023-10-04T03:28:00.535Z0+0 records out
3382023-10-04T03:28:00.574ZSimnet net0/sc0_0 exists
3392023-10-04T03:28:00.602Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3402023-10-04T03:28:00.627ZSimnet net1/sc1_0 exists
3412023-10-04T03:28:00.654Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3422023-10-04T03:28:00.682ZVnic sc0_1 exists
3432023-10-04T03:28:00.711Z+ set +x
3442023-10-04T03:28:00.738Z++ zpool list -o name
3452023-10-04T03:28:00.775Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3462023-10-04T03:28:01.224Z+ [[ -z '' ]]
3472023-10-04T03:28:01.273Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-10-04T03:28:01.310Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3492023-10-04T03:28:01.345Z+ set +x
3502023-10-04T03:28:01.380Z+ for ZPOOL in "${ZPOOLS[@]}"
3512023-10-04T03:28:01.413Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3522023-10-04T03:28:01.445Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3532023-10-04T03:28:01.514Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3542023-10-04T03:28:01.590Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3552023-10-04T03:28:01.635Z0+0 records in
3562023-10-04T03:28:01.676Z0+0 records out
3572023-10-04T03:28:01.711Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3582023-10-04T03:28:01.749Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3592023-10-04T03:28:01.782Z+ set +x
3602023-10-04T03:28:01.813Z++ zpool list -o name
3612023-10-04T03:28:01.848Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3622023-10-04T03:28:01.880Z+ [[ -z '' ]]
3632023-10-04T03:28:01.914Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3642023-10-04T03:28:01.946Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3652023-10-04T03:28:01.977Z+ set +x
3662023-10-04T03:28:02.010Z+ for ZPOOL in "${ZPOOLS[@]}"
3672023-10-04T03:28:02.042Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3682023-10-04T03:28:02.067Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3692023-10-04T03:28:02.093Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3702023-10-04T03:28:02.118Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3712023-10-04T03:28:02.144Z0+0 records in
3722023-10-04T03:28:02.175Z0+0 records out
3732023-10-04T03:28:02.201Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3742023-10-04T03:28:02.227Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3752023-10-04T03:28:02.253Z+ set +x
3762023-10-04T03:28:02.280Z++ zpool list -o name
3772023-10-04T03:28:02.306Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3782023-10-04T03:28:02.340Z+ [[ -z '' ]]
3792023-10-04T03:28:02.376Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3802023-10-04T03:28:02.403Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3812023-10-04T03:28:02.430Z+ set +x
3822023-10-04T03:28:02.455Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-10-04T03:28:02.481Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3842023-10-04T03:28:02.506Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3852023-10-04T03:28:02.557Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3862023-10-04T03:28:02.582Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3872023-10-04T03:28:02.607Z0+0 records in
3882023-10-04T03:28:02.633Z0+0 records out
3892023-10-04T03:28:02.658Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3902023-10-04T03:28:02.683Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3912023-10-04T03:28:02.709Z+ set +x
3922023-10-04T03:28:02.735Z++ zpool list -o name
3932023-10-04T03:28:03.175Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3942023-10-04T03:28:03.202Z+ [[ -z '' ]]
3952023-10-04T03:28:03.230Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3962023-10-04T03:28:03.258Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3972023-10-04T03:28:03.282Z+ set +x
3982023-10-04T03:28:03.313Z+ for ZPOOL in "${ZPOOLS[@]}"
3992023-10-04T03:28:03.344Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4002023-10-04T03:28:03.374Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4012023-10-04T03:28:03.405Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4022023-10-04T03:28:03.433Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4032023-10-04T03:28:03.461Z0+0 records in
4042023-10-04T03:28:03.500Z0+0 records out
4052023-10-04T03:28:03.529Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4062023-10-04T03:28:03.556Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4072023-10-04T03:28:03.622Z+ set +x
4082023-10-04T03:28:03.649Z++ zpool list -o name
4092023-10-04T03:28:03.678Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4102023-10-04T03:28:03.734Z+ [[ -z '' ]]
4112023-10-04T03:28:03.767Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4122023-10-04T03:28:03.800Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4132023-10-04T03:28:03.830Z+ set +x
4142023-10-04T03:28:03.860Z+ for ZPOOL in "${ZPOOLS[@]}"
4152023-10-04T03:28:03.898Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4162023-10-04T03:28:03.930Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-10-04T03:28:03.959Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4182023-10-04T03:28:03.992Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4192023-10-04T03:28:04.020Z0+0 records in
4202023-10-04T03:28:04.050Z0+0 records out
4212023-10-04T03:28:04.094Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4222023-10-04T03:28:04.141Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4232023-10-04T03:28:04.178Z+ set +x
4242023-10-04T03:28:04.217Z++ zpool list -o name
4252023-10-04T03:28:04.259Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4262023-10-04T03:28:04.713Z+ [[ -z '' ]]
4272023-10-04T03:28:04.744Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4282023-10-04T03:28:04.776Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4292023-10-04T03:28:04.812Z+ set +x
4302023-10-04T03:28:04.845Z+ for ZPOOL in "${ZPOOLS[@]}"
4312023-10-04T03:28:04.871Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4322023-10-04T03:28:04.897Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4332023-10-04T03:28:04.923Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4342023-10-04T03:28:04.950Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4352023-10-04T03:28:04.978Z0+0 records in
4362023-10-04T03:28:05.006Z0+0 records out
4372023-10-04T03:28:05.033Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4382023-10-04T03:28:05.059Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4392023-10-04T03:28:05.087Z+ set +x
4402023-10-04T03:28:05.122Z++ zpool list -o name
4412023-10-04T03:28:05.153Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4422023-10-04T03:28:05.180Z+ [[ -z '' ]]
4432023-10-04T03:28:05.210Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4442023-10-04T03:28:05.656Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4452023-10-04T03:28:05.690Z+ set +x
4462023-10-04T03:28:05.765Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4472023-10-04T03:28:05.800Z+ readarray -t ZPOOLS
4482023-10-04T03:28:05.838Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4492023-10-04T03:28:05.866Z++ sed 's/[ ",]//g'
4502023-10-04T03:28:05.895Z+ for ZPOOL in "${ZPOOLS[@]}"
4512023-10-04T03:28:05.926Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4522023-10-04T03:28:05.959Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4532023-10-04T03:28:05.989Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4542023-10-04T03:28:06.020Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4552023-10-04T03:28:06.049Z0+0 records in
4562023-10-04T03:28:06.078Z0+0 records out
4572023-10-04T03:28:06.112Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4582023-10-04T03:28:06.140Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4592023-10-04T03:28:06.171Z+ set +x
4602023-10-04T03:28:06.202Z++ zpool list -o name
4612023-10-04T03:28:06.234Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4622023-10-04T03:28:06.264Z+ [[ -z '' ]]
4632023-10-04T03:28:06.307Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4642023-10-04T03:28:06.342Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4652023-10-04T03:28:06.375Z+ set +x
4662023-10-04T03:28:06.405Z+ for ZPOOL in "${ZPOOLS[@]}"
4672023-10-04T03:28:06.435Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4682023-10-04T03:28:06.475Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4692023-10-04T03:28:06.510Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4702023-10-04T03:28:06.609Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4712023-10-04T03:28:06.637Z0+0 records in
4722023-10-04T03:28:06.668Z0+0 records out
4732023-10-04T03:28:06.703Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4742023-10-04T03:28:06.733Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4752023-10-04T03:28:06.762Z+ set +x
4762023-10-04T03:28:06.819Z++ zpool list -o name
4772023-10-04T03:28:06.848Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4782023-10-04T03:28:06.881Z+ [[ -z '' ]]
4792023-10-04T03:28:06.912Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4802023-10-04T03:28:06.961Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4812023-10-04T03:28:06.998Z+ set +x
4822023-10-04T03:28:07.035Z+ ensure_simulated_links igb0
4832023-10-04T03:28:07.072Z+ local PHYSICAL_LINK=igb0
4842023-10-04T03:28:07.108Z+ INDICES=("0" "1")
4852023-10-04T03:28:07.549Z+ for I in "${INDICES[@]}"
4862023-10-04T03:28:07.588Z++ get_simnet_name_if_exists net0
4872023-10-04T03:28:07.619Zdownloading softnpu ...done
4882023-10-04T03:28:07.647Z++ dladm show-simnet -p -o LINK net0
4892023-10-04T03:28:07.675Z++ echo ''
4902023-10-04T03:28:07.703Z+ [[ -z '' ]]
4912023-10-04T03:28:07.731Z+ dladm create-simnet -t net0
4922023-10-04T03:28:07.760Z+ dladm create-simnet -t sc0_0
4932023-10-04T03:28:07.790Z+ dladm modify-simnet -t -p net0 sc0_0
4942023-10-04T03:28:07.819Z+ dladm set-linkprop -p mtu=1600 sc0_0
4952023-10-04T03:28:07.886Z+ success 'Simnet net0/sc0_0 exists'
4962023-10-04T03:28:07.912Z+ set +x
4972023-10-04T03:28:07.937Z+ for I in "${INDICES[@]}"
4982023-10-04T03:28:07.962Z++ get_simnet_name_if_exists net1
4992023-10-04T03:28:07.989Z++ dladm show-simnet -p -o LINK net1
5002023-10-04T03:28:08.020Z++ echo ''
5012023-10-04T03:28:08.051Z+ [[ -z '' ]]
5022023-10-04T03:28:08.080Z+ dladm create-simnet -t net1
5032023-10-04T03:28:08.107Z+ dladm create-simnet -t sc1_0
5042023-10-04T03:28:08.133Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-10-04T03:28:08.162Z+ dladm set-linkprop -p mtu=1600 sc1_0
5062023-10-04T03:28:08.191Z+ success 'Simnet net1/sc1_0 exists'
5072023-10-04T03:28:08.218Z+ set +x
5082023-10-04T03:28:08.245Z++ get_vnic_name_if_exists sc0_1
5092023-10-04T03:28:08.273Z++ dladm show-vnic -p -o LINK sc0_1
5102023-10-04T03:28:08.303Z++ echo ''
5112023-10-04T03:28:08.337Z+ [[ -z '' ]]
5122023-10-04T03:28:08.368Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5132023-10-04T03:28:08.394Z+ success 'Vnic sc0_1 exists'
5142023-10-04T03:28:08.422Z+ set +x
5152023-10-04T03:28:08.454Z+ warn_if_no_proxy_arp
5162023-10-04T03:28:08.483Z+ [[ -v PXA_START ]]
5172023-10-04T03:28:08.510Z+ [[ -v PXA_END ]]
5182023-10-04T03:28:08.540Z+ ensure_softnpu_zone
5192023-10-04T03:28:08.564Z+ zoneadm list
5202023-10-04T03:28:08.588Z+ grep -q sidecar_softnpu
5212023-10-04T03:28:08.612Z+ [[ -f out/npuzone/npuzone ]]
5222023-10-04T03:28:08.636Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0
5232023-10-04T03:28:10.521Zdownloading asic_program.so ...done
5242023-10-04T03:28:19.402Zdownloading scadm ...done
5252023-10-04T03:28:19.834Zcreating port tfportrear0_0 for sc0_0
5262023-10-04T03:28:19.861Zcreating port tfportqsfp0_0 for sc0_1
5272023-10-04T03:28:19.893Zsoftnpu zone setup
5282023-10-04T03:28:19.930Zconfigure zone
5292023-10-04T03:28:20.402Zinstall zone
5302023-10-04T03:28:20.443Zboot zone
5312023-10-04T03:28:23.554Z[sidecar_softnpu] svcs milestone/network
5322023-10-04T03:28:23.581ZSTATE STIME FMRI
5332023-10-04T03:28:23.609Zonline 3:28:21 svc:/milestone/network:default
5342023-10-04T03:28:23.658Z[sidecar_softnpu] /softnpu/npu start
5352023-10-04T03:28:23.684Z
5362023-10-04T03:28:23.708Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5372023-10-04T03:28:23.732Z+ GATEWAY_IP=10.151.2.100
5382023-10-04T03:28:23.759Z+ echo 'Using 10.151.2.100 as gateway ip'
5392023-10-04T03:28:23.785ZUsing 10.151.2.100 as gateway ip
5402023-10-04T03:28:23.810Z+ [[ ! -v GATEWAY_MAC ]]
5412023-10-04T03:28:23.836Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5422023-10-04T03:28:23.860Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5432023-10-04T03:28:23.885Z+++ echo 18:c0:4d:0d:a0:2a
5442023-10-04T03:28:23.910Z+ COUNT=1
5452023-10-04T03:28:23.935Z+ [[ 1 -ne 1 ]]
5462023-10-04T03:28:23.960Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5472023-10-04T03:28:23.987ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5482023-10-04T03:28:24.015Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5492023-10-04T03:28:24.051Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5502023-10-04T03:28:24.524Z+ PXA_MAC=a8:e1:de:01:70:1d
5512023-10-04T03:28:24.572Z+ [[ -v PXA_START ]]
5522023-10-04T03:28:25.029Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5532023-10-04T03:28:25.053Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5542023-10-04T03:28:25.080Z+ z_scadm dump-state
5552023-10-04T03:28:25.108Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5562023-10-04T03:28:25.141Zlocal v6:
5572023-10-04T03:28:25.183Zlocal v4:
5582023-10-04T03:28:25.218Zrouter v6:
5592023-10-04T03:28:25.252Zrouter v4:
5602023-10-04T03:28:25.294Zresolver v4:
5612023-10-04T03:28:25.759Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5622023-10-04T03:28:26.226Zresolver v6:
5632023-10-04T03:28:26.261Znat v4:
5642023-10-04T03:28:26.299Znat_v6:
5652023-10-04T03:28:26.333Zport_mac:
5662023-10-04T03:28:26.821Zproxy arp:
5672023-10-04T03:28:26.868Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5682023-10-04T03:28:26.977Z+ success 'softnpu zone exists'
5692023-10-04T03:28:27.001Z+ set +x
5702023-10-04T03:28:27.025Zsoftnpu zone exists
5712023-10-04T03:28:27.052Z
5722023-10-04T03:28:27.077Zreal 27.697828543
5732023-10-04T03:28:27.101Zuser 0.706496695
5742023-10-04T03:28:27.125Zsys 1.633777412
5752023-10-04T03:28:27.149Ztrap 0.000096851
5762023-10-04T03:28:27.176Ztflt 0.000170886
5772023-10-04T03:28:27.216Zdflt 0.000359675
5782023-10-04T03:28:27.247Zkflt 0.000000000
5792023-10-04T03:28:27.275Zlock 21:38.093006816
5802023-10-04T03:28:27.303Zslp 1:22.499682128
5812023-10-04T03:28:27.745Zlat 0.069870891
5822023-10-04T03:28:27.772Zstop 0.372699020
5832023-10-04T03:28:27.797Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5842023-10-04T03:28:27.893Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5852023-10-04T03:28:27.944Z+ SILO_NAME=recovery
5862023-10-04T03:28:27.971Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5872023-10-04T03:28:28.001Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5882023-10-04T03:28:28.039Z+ sed -i~ '
5892023-10-04T03:28:28.072Z /^external_dns_ips/c\
5902023-10-04T03:28:28.515Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5912023-10-04T03:28:28.542Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5922023-10-04T03:28:28.566Z /^first/c\
5932023-10-04T03:28:28.595Zfirst = "10.151.2.164"
5942023-10-04T03:28:28.619Z /^last/c\
5952023-10-04T03:28:28.653Zlast = "10.151.2.173"
5962023-10-04T03:28:28.685Z }
5972023-10-04T03:28:28.717Z /^\[rack_network_config/,/^$/ {
5982023-10-04T03:28:28.742Z /^infra_ip_first/c\
5992023-10-04T03:28:28.774Zinfra_ip_first = "10.151.2.174"
6002023-10-04T03:28:28.800Z /^infra_ip_last/c\
6012023-10-04T03:28:28.826Zinfra_ip_last = "10.151.2.174"
6022023-10-04T03:28:28.851Z }
6032023-10-04T03:28:28.894Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6042023-10-04T03:28:28.921Z /^\[\[rack_network_config.uplinks/,/^$/ {
6052023-10-04T03:28:28.947Z+++ pkg/config-rss.toml Wed Oct 4 03:28:23 2023
6062023-10-04T03:28:28.971Z /^gateway_ip/c\
6072023-10-04T03:28:28.996Z@@ -41,7 +41,7 @@
6082023-10-04T03:28:29.021Zgateway_ip = "10.151.2.100"
6092023-10-04T03:28:29.047Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6102023-10-04T03:28:29.076Z /^uplink_cidr/c\
6112023-10-04T03:28:29.108Z # must be contained in one of the "internal services" IP Pool ranges listed
6122023-10-04T03:28:29.139Zuplink_cidr = "10.151.2.174/32"
6132023-10-04T03:28:29.168Z # below.
6142023-10-04T03:28:29.619Z }
6152023-10-04T03:28:29.653Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6162023-10-04T03:28:29.686Z' pkg/config-rss.toml
6172023-10-04T03:28:29.722Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6182023-10-04T03:28:29.754Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-10-04T03:28:29.780Z
6202023-10-04T03:28:29.815Z+ true
6212023-10-04T03:28:29.847Z # Initial TLS certificates for the external API
6222023-10-04T03:28:29.871Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6232023-10-04T03:28:29.929Z #
6242023-10-04T03:28:29.954Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6252023-10-04T03:28:29.980Z@@ -77,8 +77,8 @@
6262023-10-04T03:28:30.005Z+ TLS_NAME=recovery.sys.oxide.test
6272023-10-04T03:28:30.033Z #
6282023-10-04T03:28:30.061Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6292023-10-04T03:28:30.510Z # For more on this and what to put here, see docs/how-to-run.adoc.
6302023-10-04T03:28:30.541Z.+.........+......+..+...+....+...+..+......+...+............+....+.........+.....+.........+.+.........+.....+......+.+.....+....+...........+......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...+.......+...+..+.......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+.................+.+.....+....+..+...................+..+.......+......+......+........+.+.....+.+...+...+.....+.........................+..+......+............+....+.........+.........+..+......+.........+.........+....+...+..+...+...+......+...+.+.................+...+................+......+...............+............+........+....+.........+.........+..+.+...+.....+..........+....................+.+.....+.+..............+....+...............+......+....................+...+......+.+.....+...+...+...+....+...+.......................+....+............+...+...........+.......+........+....+..+.........+......+....+.........+........+................................................+.+......+............+...........+.+.........+..+....+........+...+.......+...+...+............+.................+...+....+...+........+...+......+...............+....+...........+.............+......+.....+....+...+....................+...+....+......+..+.......+.....+.+......+.....+..................+..........+..+.+...+.....+.........+.+.........+..+....+..+...+...............+.+........+.....................+....+..+......+.......+..+.+..............+...+............+....+..+..........+..+............+.........+................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6312023-10-04T03:28:30.567Z [[internal_services_ip_pool_ranges]]
6322023-10-04T03:28:30.601Z...........+.+......+..............+....+........+.......+..+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+........+.+.....+....+...........+.+..+.+..............+.............+..+.+..+....+.....+.+..+............+...+..........+........+....+...+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+...+...............+..+.......+........+............+......+..................+......+.+..+..........+.....................+...........+.........+.+.....+....+..+...+.+...+.....+.......+.....+.+........+.......+..............+...+.............+............+..+...+......+......+.+...............+........+..........+...........+.............+.....+.+.....+.......+..+.+..+...+.+...+...........+..........+............+.....+....+..................+..+...+...+.........+..........+...+...+..+.......+.....+.+..+................+.....+................+......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6332023-10-04T03:28:31.053Z-first = "192.168.1.20"
6342023-10-04T03:28:31.090Z-----
6352023-10-04T03:28:31.125Z-last = "192.168.1.29"
6362023-10-04T03:28:31.159Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6372023-10-04T03:28:31.191Z+first = "10.151.2.164"
6382023-10-04T03:28:31.229Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6392023-10-04T03:28:31.699Z+last = "10.151.2.173"
6402023-10-04T03:28:32.169Z+ rmdir pkg
6412023-10-04T03:28:32.322Z
6422023-10-04T03:28:32.347Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6432023-10-04T03:28:32.374Z # TODO - this configuration is subject to change going forward. Ultimately these
6442023-10-04T03:28:32.404Z+ pfexec mkdir -p /usr/oxide
6452023-10-04T03:28:32.431Z # parameters should be provided to the control plane via wicket, but we need to
6462023-10-04T03:28:32.457Z+ pfexec rm -f /usr/oxide/tcpproxy
6472023-10-04T03:28:32.487Z@@ -91,13 +91,13 @@
6482023-10-04T03:28:32.514Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6492023-10-04T03:28:32.539Z # A range of IP addresses used by Boundary Services on the external network. In
6502023-10-04T03:28:32.565Z+ pfexec chmod +x /usr/oxide/tcpproxy
6512023-10-04T03:28:32.593Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6522023-10-04T03:28:32.638Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6532023-10-04T03:28:32.675Z # With softnpu, only one address is used.
6542023-10-04T03:28:32.708Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6552023-10-04T03:28:32.742Z-infra_ip_first = "192.168.1.30"
6562023-10-04T03:28:32.771Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6572023-10-04T03:28:32.801Z-infra_ip_last = "192.168.1.30"
6582023-10-04T03:28:32.831Z+ OMICRON_NO_UNINSTALL=1
6592023-10-04T03:28:32.860Z+infra_ip_first = "10.151.2.174"
6602023-10-04T03:28:32.894Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6612023-10-04T03:28:32.923Z+infra_ip_last = "10.151.2.174"
6622023-10-04T03:28:32.953ZOct 04 03:28:24.993 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6632023-10-04T03:28:32.983Z
6642023-10-04T03:28:33.019ZOct 04 03:28:24.994 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6652023-10-04T03:28:33.103Z # You can configure multiple uplinks by repeating the following stanza
6662023-10-04T03:28:33.542ZOct 04 03:28:24.995 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/maghemite.tar
6672023-10-04T03:28:33.573Z [[rack_network_config.uplinks]]
6682023-10-04T03:28:34.451ZOct 04 03:28:24.995 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6692023-10-04T03:28:34.480Z # The gateway IP for the rack's external network
6702023-10-04T03:28:34.515ZOct 04 03:28:24.995 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6712023-10-04T03:28:34.544Z-gateway_ip = "192.168.1.199"
6722023-10-04T03:28:34.571ZOct 04 03:28:24.995 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6732023-10-04T03:28:34.597Z+gateway_ip = "10.151.2.100"
6742023-10-04T03:28:34.629ZOct 04 03:28:24.995 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6752023-10-04T03:28:34.672Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6762023-10-04T03:28:34.701ZOct 04 03:28:24.995 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6772023-10-04T03:28:34.726Z uplink_port = "qsfp0"
6782023-10-04T03:28:34.750ZOct 04 03:28:24.995 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6792023-10-04T03:28:34.774Z uplink_port_speed = "40G"
6802023-10-04T03:28:34.799ZOct 04 03:28:24.995 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6812023-10-04T03:28:34.822Z@@ -104,7 +104,7 @@
6822023-10-04T03:28:34.848ZOct 04 03:28:24.995 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6832023-10-04T03:28:34.872Z uplink_port_fec="none"
6842023-10-04T03:28:34.896ZOct 04 03:28:24.995 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6852023-10-04T03:28:34.923Z # For softnpu, an address within the "infra" block above that will be used for
6862023-10-04T03:28:34.955ZOct 04 03:28:24.995 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6872023-10-04T03:28:34.991Z # the softnpu uplink port. You can just pick the first address in that pool.
6882023-10-04T03:28:35.020ZOct 04 03:28:24.995 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6892023-10-04T03:28:35.044Z-uplink_cidr = "192.168.1.30/32"
6902023-10-04T03:28:35.069ZOct 04 03:28:24.995 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6912023-10-04T03:28:35.094Z+uplink_cidr = "10.151.2.174/32"
6922023-10-04T03:28:35.118ZOct 04 03:28:24.995 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6932023-10-04T03:28:35.172Z # Switch to use for the uplink. For single-rack deployments this can be
6942023-10-04T03:28:35.198ZOct 04 03:28:24.995 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6952023-10-04T03:28:35.222Z # "switch0" (upper slot) or "switch1" (lower slot). For single-node softnpu
6962023-10-04T03:28:35.246ZOct 04 03:28:25.218 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
6972023-10-04T03:28:35.270Z # and dendrite stub environments, use "switch0"
6982023-10-04T03:28:35.296ZOct 04 03:28:25.263 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
6992023-10-04T03:28:35.320Za pkg/config-rss.toml 7K
7002023-10-04T03:28:35.344ZOct 04 03:28:25.322 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7012023-10-04T03:28:35.370Za pkg/initial-tls-cert.pem 2K
7022023-10-04T03:28:35.396Z
7032023-10-04T03:28:35.425Za pkg/initial-tls-key.pem 4K
7042023-10-04T03:28:35.860Zreal 0.465921213
7052023-10-04T03:28:35.899Zuser 0.097463040
7062023-10-04T03:28:35.938Zsys 1.893152602
7072023-10-04T03:28:35.976Ztrap 0.000000250
7082023-10-04T03:28:36.012Ztflt 0.000022328
7092023-10-04T03:28:36.035Zdflt 0.000427039
7102023-10-04T03:28:36.062Zkflt 0.000000000
7112023-10-04T03:28:36.086Zlock 41.863359560
7122023-10-04T03:28:36.112Zslp 0.677557420
7132023-10-04T03:28:36.138Zlat 0.002860118
7142023-10-04T03:28:36.163Zstop 0.202890744
7152023-10-04T03:28:36.212Z+ retry=0
7162023-10-04T03:28:36.236Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7172023-10-04T03:28:36.260Z+ [[ 0 -gt 30 ]]
7182023-10-04T03:28:36.284Z+ sleep 1
7192023-10-04T03:28:36.309Z+ retry=1
7202023-10-04T03:28:36.335Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7212023-10-04T03:28:36.367Z+ [[ 1 -gt 30 ]]
7222023-10-04T03:28:36.408Z+ sleep 1
7232023-10-04T03:28:36.447Z+ retry=2
7242023-10-04T03:28:36.485Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7252023-10-04T03:28:47.229Z+ [[ 2 -gt 30 ]]
7262023-10-04T03:28:47.264Z+ sleep 1
7272023-10-04T03:28:48.227Z+ retry=3
7282023-10-04T03:28:48.668Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7292023-10-04T03:28:48.700Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7302023-10-04T03:28:48.726Zlocal v6:
7312023-10-04T03:28:48.758Zfe80::b01f:eaff:feb5:7627
7322023-10-04T03:28:48.788Zlocal v4:
7332023-10-04T03:28:48.815Zrouter v6:
7342023-10-04T03:28:48.841Zrouter v4:
7352023-10-04T03:28:48.867Zresolver v4:
7362023-10-04T03:28:48.907Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7372023-10-04T03:28:48.931Zresolver v6:
7382023-10-04T03:28:48.957Zfe80::b01f:eaff:feb5:7627 -> b2:1f:ea:b5:76:27
7392023-10-04T03:28:48.988Zfe80::103d:51ff:fe1b:ecee -> 12:3d:51:1b:ec:ee
7402023-10-04T03:28:49.016Znat v4:
7412023-10-04T03:28:49.040Znat_v6:
7422023-10-04T03:28:49.064Zport_mac:
7432023-10-04T03:28:49.088Z1000: a8:40:25:25:00:3f
7442023-10-04T03:28:49.112Z1: a8:40:25:25:00:40
7452023-10-04T03:28:49.136Zproxy arp:
7462023-10-04T03:28:49.160Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7472023-10-04T03:28:49.184Z+ retry=0
7482023-10-04T03:28:49.217Z+++ zoneadm list -n
7492023-10-04T03:28:49.255Z+++ grep oxz_ntp
7502023-10-04T03:28:49.279Z++ pfexec svcs -z -Hostate oxide/ntp
7512023-10-04T03:28:49.303Zsvcs: zone '-Hostate': entity not found
7522023-10-04T03:28:49.331Z++ true
7532023-10-04T03:28:49.354Z+ [[ '' != online ]]
7542023-10-04T03:28:49.380Z+ [[ 0 -gt 60 ]]
7552023-10-04T03:28:49.413Z+ sleep 1
7562023-10-04T03:28:49.447Z+ retry=1
7572023-10-04T03:28:49.477Z+++ zoneadm list -n
7582023-10-04T03:28:49.503Z+++ grep oxz_ntp
7592023-10-04T03:28:49.527Z++ pfexec svcs -z -Hostate oxide/ntp
7602023-10-04T03:28:49.550Zsvcs: zone '-Hostate': entity not found
7612023-10-04T03:28:49.577Z++ true
7622023-10-04T03:28:49.601Z+ [[ '' != online ]]
7632023-10-04T03:28:49.625Z+ [[ 1 -gt 60 ]]
7642023-10-04T03:28:49.649Z+ sleep 1
7652023-10-04T03:28:50.315Z+ retry=2
7662023-10-04T03:28:50.345Z+++ zoneadm list -n
7672023-10-04T03:28:50.375Z+++ grep oxz_ntp
7682023-10-04T03:28:50.403Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-10-04T03:28:50.430Zsvcs: zone '-Hostate': entity not found
7702023-10-04T03:28:50.455Z++ true
7712023-10-04T03:28:50.483Z+ [[ '' != online ]]
7722023-10-04T03:28:50.532Z+ [[ 2 -gt 60 ]]
7732023-10-04T03:28:50.576Z+ sleep 1
7742023-10-04T03:28:51.339Z+ retry=3
7752023-10-04T03:28:51.372Z+++ zoneadm list -n
7762023-10-04T03:28:51.403Z+++ grep oxz_ntp
7772023-10-04T03:28:51.432Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-10-04T03:28:51.467Zsvcs: zone '-Hostate': entity not found
7792023-10-04T03:28:51.949Z++ true
7802023-10-04T03:28:51.979Z+ [[ '' != online ]]
7812023-10-04T03:28:52.010Z+ [[ 3 -gt 60 ]]
7822023-10-04T03:28:52.046Z+ sleep 1
7832023-10-04T03:28:52.359Z+ retry=4
7842023-10-04T03:28:52.392Z+++ zoneadm list -n
7852023-10-04T03:28:52.419Z+++ grep oxz_ntp
7862023-10-04T03:28:52.455Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-10-04T03:28:52.494Zsvcs: zone '-Hostate': entity not found
7882023-10-04T03:28:52.524Z++ true
7892023-10-04T03:28:52.550Z+ [[ '' != online ]]
7902023-10-04T03:28:52.574Z+ [[ 4 -gt 60 ]]
7912023-10-04T03:28:52.598Z+ sleep 1
7922023-10-04T03:28:53.385Z+ retry=5
7932023-10-04T03:28:53.416Z+++ zoneadm list -n
7942023-10-04T03:28:53.864Z+++ grep oxz_ntp
7952023-10-04T03:28:53.894Z++ pfexec svcs -z -Hostate oxide/ntp
7962023-10-04T03:28:53.928Zsvcs: zone '-Hostate': entity not found
7972023-10-04T03:28:53.958Z++ true
7982023-10-04T03:28:53.992Z+ [[ '' != online ]]
7992023-10-04T03:28:54.026Z+ [[ 5 -gt 60 ]]
8002023-10-04T03:28:54.051Z+ sleep 1
8012023-10-04T03:28:54.407Z+ retry=6
8022023-10-04T03:28:54.438Z+++ zoneadm list -n
8032023-10-04T03:28:54.466Z+++ grep oxz_ntp
8042023-10-04T03:28:54.502Z++ pfexec svcs -z -Hostate oxide/ntp
8052023-10-04T03:28:54.536Zsvcs: zone '-Hostate': entity not found
8062023-10-04T03:28:54.616Z++ true
8072023-10-04T03:28:54.650Z+ [[ '' != online ]]
8082023-10-04T03:28:54.689Z+ [[ 6 -gt 60 ]]
8092023-10-04T03:28:54.724Z+ sleep 1
8102023-10-04T03:28:55.427Z+ retry=7
8112023-10-04T03:28:55.455Z+++ zoneadm list -n
8122023-10-04T03:28:55.485Z+++ grep oxz_ntp
8132023-10-04T03:28:55.515Z++ pfexec svcs -z -Hostate oxide/ntp
8142023-10-04T03:28:55.545Zsvcs: zone '-Hostate': entity not found
8152023-10-04T03:28:55.577Z++ true
8162023-10-04T03:28:55.617Z+ [[ '' != online ]]
8172023-10-04T03:28:55.649Z+ [[ 7 -gt 60 ]]
8182023-10-04T03:28:55.675Z+ sleep 1
8192023-10-04T03:28:56.465Z+ retry=8
8202023-10-04T03:28:56.500Z+++ zoneadm list -n
8212023-10-04T03:28:56.584Z+++ grep oxz_ntp
8222023-10-04T03:28:56.608Z++ pfexec svcs -z -Hostate oxide/ntp
8232023-10-04T03:28:56.634Zsvcs: zone '-Hostate': entity not found
8242023-10-04T03:28:56.658Z++ true
8252023-10-04T03:28:56.683Z+ [[ '' != online ]]
8262023-10-04T03:28:56.707Z+ [[ 8 -gt 60 ]]
8272023-10-04T03:28:56.742Z+ sleep 1
8282023-10-04T03:28:57.475Z+ retry=9
8292023-10-04T03:28:57.505Z+++ zoneadm list -n
8302023-10-04T03:28:57.543Z+++ grep oxz_ntp
8312023-10-04T03:28:57.571Z++ pfexec svcs -z -Hostate oxide/ntp
8322023-10-04T03:28:57.598Zsvcs: zone '-Hostate': entity not found
8332023-10-04T03:28:57.626Z++ true
8342023-10-04T03:28:57.656Z+ [[ '' != online ]]
8352023-10-04T03:28:57.684Z+ [[ 9 -gt 60 ]]
8362023-10-04T03:28:57.718Z+ sleep 1
8372023-10-04T03:28:58.486Z+ retry=10
8382023-10-04T03:28:58.514Z+++ zoneadm list -n
8392023-10-04T03:28:58.538Z+++ grep oxz_ntp
8402023-10-04T03:28:58.562Z++ pfexec svcs -z -Hostate oxide/ntp
8412023-10-04T03:28:58.586Zsvcs: zone '-Hostate': entity not found
8422023-10-04T03:28:58.610Z++ true
8432023-10-04T03:28:58.635Z+ [[ '' != online ]]
8442023-10-04T03:28:58.664Z+ [[ 10 -gt 60 ]]
8452023-10-04T03:28:59.152Z+ sleep 1
8462023-10-04T03:28:59.514Z+ retry=11
8472023-10-04T03:28:59.542Z+++ zoneadm list -n
8482023-10-04T03:28:59.571Z+++ grep oxz_ntp
8492023-10-04T03:28:59.599Z++ pfexec svcs -z -Hostate oxide/ntp
8502023-10-04T03:29:00.038Zsvcs: zone '-Hostate': entity not found
8512023-10-04T03:29:00.062Z++ true
8522023-10-04T03:29:00.087Z+ [[ '' != online ]]
8532023-10-04T03:29:00.113Z+ [[ 11 -gt 60 ]]
8542023-10-04T03:29:00.143Z+ sleep 1
8552023-10-04T03:29:00.531Z+ retry=12
8562023-10-04T03:29:00.563Z+++ zoneadm list -n
8572023-10-04T03:29:00.595Z+++ grep oxz_ntp
8582023-10-04T03:29:00.622Z++ pfexec svcs -z -Hostate oxide/ntp
8592023-10-04T03:29:00.646Zsvcs: zone '-Hostate': entity not found
8602023-10-04T03:29:00.670Z++ true
8612023-10-04T03:29:00.694Z+ [[ '' != online ]]
8622023-10-04T03:29:00.718Z+ [[ 12 -gt 60 ]]
8632023-10-04T03:29:00.743Z+ sleep 1
8642023-10-04T03:29:01.608Z+ retry=13
8652023-10-04T03:29:01.633Z+++ zoneadm list -n
8662023-10-04T03:29:01.659Z+++ grep oxz_ntp
8672023-10-04T03:29:01.684Z++ pfexec svcs -z oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078 -Hostate oxide/ntp
8682023-10-04T03:29:01.709Zsvcs: Could not bind to repository server for zone oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078: repository server unavailable
8692023-10-04T03:29:01.734Z++ true
8702023-10-04T03:29:01.761Z+ [[ '' != online ]]
8712023-10-04T03:29:01.785Z+ [[ 13 -gt 60 ]]
8722023-10-04T03:29:01.811Z+ sleep 1
8732023-10-04T03:29:02.589Z+ retry=14
8742023-10-04T03:29:02.625Z+++ zoneadm list -n
8752023-10-04T03:29:02.651Z+++ grep oxz_ntp
8762023-10-04T03:29:02.678Z++ pfexec svcs -z oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078 -Hostate oxide/ntp
8772023-10-04T03:29:02.704Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8782023-10-04T03:29:02.729Z++ true
8792023-10-04T03:29:02.755Z+ [[ '' != online ]]
8802023-10-04T03:29:02.780Z+ [[ 14 -gt 60 ]]
8812023-10-04T03:29:02.806Z+ sleep 1
8822023-10-04T03:29:03.643Z+ retry=15
8832023-10-04T03:29:03.669Z+++ zoneadm list -n
8842023-10-04T03:29:03.693Z+++ grep oxz_ntp
8852023-10-04T03:29:03.718Z++ pfexec svcs -z oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078 -Hostate oxide/ntp
8862023-10-04T03:29:03.742Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8872023-10-04T03:29:03.767Z++ true
8882023-10-04T03:29:03.795Z+ [[ '' != online ]]
8892023-10-04T03:29:03.820Z+ [[ 15 -gt 60 ]]
8902023-10-04T03:29:03.849Z+ sleep 1
8912023-10-04T03:29:04.684Z+ retry=16
8922023-10-04T03:29:04.723Z+++ zoneadm list -n
8932023-10-04T03:29:04.754Z+++ grep oxz_ntp
8942023-10-04T03:29:04.791Z++ pfexec svcs -z oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078 -Hostate oxide/ntp
8952023-10-04T03:29:04.822Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8962023-10-04T03:29:04.857Z++ true
8972023-10-04T03:29:04.888Z+ [[ '' != online ]]
8982023-10-04T03:29:04.926Z+ [[ 16 -gt 60 ]]
8992023-10-04T03:29:04.961Z+ sleep 1
9002023-10-04T03:29:05.675Z+ retry=17
9012023-10-04T03:29:05.741Z+++ zoneadm list -n
9022023-10-04T03:29:05.768Z+++ grep oxz_ntp
9032023-10-04T03:29:05.808Z++ pfexec svcs -z oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078 -Hostate oxide/ntp
9042023-10-04T03:29:05.860Z+ [[ disabled != online ]]
9052023-10-04T03:29:05.898Z+ [[ 17 -gt 60 ]]
9062023-10-04T03:29:05.929Z+ sleep 1
9072023-10-04T03:29:06.713Z+ retry=18
9082023-10-04T03:29:06.742Z+++ zoneadm list -n
9092023-10-04T03:29:06.831Z+++ grep oxz_ntp
9102023-10-04T03:29:06.871Z++ pfexec svcs -z oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078 -Hostate oxide/ntp
9112023-10-04T03:29:06.901Z+ [[ disabled != online ]]
9122023-10-04T03:29:06.931Z+ [[ 18 -gt 60 ]]
9132023-10-04T03:29:06.963Z+ sleep 1
9142023-10-04T03:29:07.729Z+ retry=19
9152023-10-04T03:29:07.768Z+++ zoneadm list -n
9162023-10-04T03:29:07.803Z+++ grep oxz_ntp
9172023-10-04T03:29:07.887Z++ pfexec svcs -z oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078 -Hostate oxide/ntp
9182023-10-04T03:29:07.929Z+ [[ disabled != online ]]
9192023-10-04T03:29:07.967Z+ [[ 19 -gt 60 ]]
9202023-10-04T03:29:08.006Z+ sleep 1
9212023-10-04T03:29:08.740Z+ retry=20
9222023-10-04T03:29:08.771Z+++ zoneadm list -n
9232023-10-04T03:29:08.801Z+++ grep oxz_ntp
9242023-10-04T03:29:08.829Z++ pfexec svcs -z oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078 -Hostate oxide/ntp
9252023-10-04T03:29:08.862Z+ [[ disabled != online ]]
9262023-10-04T03:29:08.909Z+ [[ 20 -gt 60 ]]
9272023-10-04T03:29:08.943Z+ sleep 1
9282023-10-04T03:29:09.765Z+ retry=21
9292023-10-04T03:29:09.790Z+++ zoneadm list -n
9302023-10-04T03:29:09.818Z+++ grep oxz_ntp
9312023-10-04T03:29:09.849Z++ pfexec svcs -z oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078 -Hostate oxide/ntp
9322023-10-04T03:29:09.879Z+ [[ disabled != online ]]
9332023-10-04T03:29:09.909Z+ [[ 21 -gt 60 ]]
9342023-10-04T03:29:09.979Z+ sleep 1
9352023-10-04T03:29:10.786Z+ retry=22
9362023-10-04T03:29:10.823Z+++ zoneadm list -n
9372023-10-04T03:29:10.856Z+++ grep oxz_ntp
9382023-10-04T03:29:10.888Z++ pfexec svcs -z oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078 -Hostate oxide/ntp
9392023-10-04T03:29:10.918Z+ [[ disabled != online ]]
9402023-10-04T03:29:10.952Z+ [[ 22 -gt 60 ]]
9412023-10-04T03:29:10.995Z+ sleep 1
9422023-10-04T03:29:11.809Z+ retry=23
9432023-10-04T03:29:11.840Z+++ zoneadm list -n
9442023-10-04T03:29:11.869Z+++ grep oxz_ntp
9452023-10-04T03:29:11.899Z++ pfexec svcs -z oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078 -Hostate oxide/ntp
9462023-10-04T03:29:11.927Z+ [[ disabled != online ]]
9472023-10-04T03:29:11.957Z+ [[ 23 -gt 60 ]]
9482023-10-04T03:29:11.986Z+ sleep 1
9492023-10-04T03:29:12.831Z+ retry=24
9502023-10-04T03:29:12.861Z+++ zoneadm list -n
9512023-10-04T03:29:12.891Z+++ grep oxz_ntp
9522023-10-04T03:29:12.919Z++ pfexec svcs -z oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078 -Hostate oxide/ntp
9532023-10-04T03:29:12.953Z+ [[ disabled != online ]]
9542023-10-04T03:29:12.988Z+ [[ 24 -gt 60 ]]
9552023-10-04T03:29:13.028Z+ sleep 1
 
9562023-10-04T03:29:13.282Zsock console login: Oct 4 03:29:12 sok ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9572023-10-04T03:29:13.853Z+ retry=25
9582023-10-04T03:29:13.885Z+++ zoneadm list -n
9592023-10-04T03:29:13.912Z+++ grep oxz_ntp
9602023-10-04T03:29:13.936Z++ pfexec svcs -z oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078 -Hostate oxide/ntp
9612023-10-04T03:29:14.240Z+ [[ online != online ]]
9622023-10-04T03:29:14.274Z+ echo 'Waited for chrony: 25s'
9632023-10-04T03:29:14.712ZWaited for chrony: 25s
9642023-10-04T03:29:14.747Z+ export RUST_BACKTRACE=1
9652023-10-04T03:29:14.784Z+ RUST_BACKTRACE=1
9662023-10-04T03:29:14.861Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9672023-10-04T03:29:14.890Z+ ./tests/bootstrap
9682023-10-04T03:29:15.321ZUsing external DNS server at 10.151.2.164:53
9692023-10-04T03:29:15.348Z2023-10-04 03:29:13.513047654 UTC: attempting to log into API
9702023-10-04T03:29:28.938Z2023-10-04 03:29:29.014612922 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9712023-10-04T03:29:29.947Z2023-10-04 03:29:30.015648751 UTC: attempting to log into API
9722023-10-04T03:29:44.973Z2023-10-04 03:29:45.049665200 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9732023-10-04T03:29:45.986Z2023-10-04 03:29:46.051502448 UTC: attempting to log into API
9742023-10-04T03:30:01.008Z2023-10-04 03:30:01.083828448 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9752023-10-04T03:30:01.838ZOct 4 03:30:01 sock cron[7960]: Solaris_audit getaddrinfo(sock) failed[node name or service name not known]: Error 0
9762023-10-04T03:30:01.875ZOct 4 03:30:01 sock cron[7960]: Solaris_audit mapping sock to fe80::103d:51ff:fe1b:ecee: Error 0
 
9772023-10-04T03:30:02.008Z2023-10-04 03:30:02.085030401 UTC: attempting to log into API
9782023-10-04T03:30:17.041Z2023-10-04 03:30:17.117815322 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9792023-10-04T03:30:18.051Z2023-10-04 03:30:18.118892881 UTC: attempting to log into API
9802023-10-04T03:30:33.075Z2023-10-04 03:30:33.129581882 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9812023-10-04T03:30:34.076Z2023-10-04 03:30:34.127883040 UTC: attempting to log into API
9822023-10-04T03:30:49.121Z2023-10-04 03:30:49.120799340 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9832023-10-04T03:30:50.513Z2023-10-04 03:30:50.119079248 UTC: attempting to log into API
9842023-10-04T03:31:05.155Z2023-10-04 03:31:05.131735959 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9852023-10-04T03:31:06.145Z2023-10-04 03:31:06.132778577 UTC: attempting to log into API
9862023-10-04T03:31:21.178Z2023-10-04 03:31:21.165788644 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9872023-10-04T03:31:22.179Z2023-10-04 03:31:22.166668339 UTC: attempting to log into API
9882023-10-04T03:31:37.213Z2023-10-04 03:31:37.202543659 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9892023-10-04T03:31:38.214Z2023-10-04 03:31:38.203501825 UTC: attempting to log into API
9902023-10-04T03:31:53.247Z2023-10-04 03:31:53.236329734 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9912023-10-04T03:31:54.254Z2023-10-04 03:31:54.237594378 UTC: attempting to log into API
9922023-10-04T03:32:09.286Z2023-10-04 03:32:09.271064421 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9932023-10-04T03:32:10.310Z2023-10-04 03:32:10.272231182 UTC: attempting to log into API
9942023-10-04T03:32:25.314Z2023-10-04 03:32:25.304681072 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9952023-10-04T03:32:26.322Z2023-10-04 03:32:26.305779912 UTC: attempting to log into API
9962023-10-04T03:32:41.348Z2023-10-04 03:32:41.338446682 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9972023-10-04T03:32:42.349Z2023-10-04 03:32:42.339499812 UTC: attempting to log into API
 
9982023-10-04T03:32:44.972ZOct 4 03:32:44 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
9992023-10-04T03:32:51.586ZOct 4 03:32:51 sok ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
10002023-10-04T03:32:57.383Z2023-10-04 03:32:57.372970204 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10012023-10-04T03:32:58.386Z2023-10-04 03:32:58.374746210 UTC: attempting to log into API
 
10022023-10-04T03:32:59.908ZOct 4 03:32:59 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
10032023-10-04T03:33:06.464ZOct 4 03:33:05 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
10042023-10-04T03:33:12.968ZOct 4 03:33:12 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
10052023-10-04T03:33:14.650Z2023-10-04 03:33:13.408571431 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10062023-10-04T03:33:14.846Z2023-10-04 03:33:14.410622374 UTC: attempting to log into API
10072023-10-04T03:33:29.455Z2023-10-04 03:33:29.443921993 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10082023-10-04T03:33:30.976Z2023-10-04 03:33:30.444989622 UTC: attempting to log into API
10092023-10-04T03:33:31.226Z2023-10-04 03:33:31.216178883 UTC: login succeeded
10102023-10-04T03:33:31.259Zwaiting for nexus to come up...
10112023-10-04T03:33:31.333Zcreating IP pool... 10.151.2.175 - 10.151.2.227
10122023-10-04T03:33:31.372Zensuring datasets are ready...
10132023-10-04T03:33:33.554Zlet's roll.
10142023-10-04T03:33:33.580Z+ rm ./tests/bootstrap
10152023-10-04T03:33:33.627Z+ for test_bin in tests/*
10162023-10-04T03:33:33.659Z+ ./tests/bootstrap-e8befc34dc38cbc1
10172023-10-04T03:33:33.689Z
10182023-10-04T03:33:33.774Zrunning 0 tests
10192023-10-04T03:33:33.805Z
10202023-10-04T03:33:33.845Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10212023-10-04T03:33:33.889Z
10222023-10-04T03:33:33.951Z+ for test_bin in tests/*
10232023-10-04T03:33:34.080Z+ ./tests/end_to_end_tests-b03cef630464e50a
10242023-10-04T03:33:34.158Z
10252023-10-04T03:33:34.246Zrunning 2 tests
10262023-10-04T03:33:34.468Ztest no_spoof::no_spoof_here ... ok
10272023-10-04T03:34:33.647Ztest instance_launch::instance_launch has been running for over 60 seconds
10282023-10-04T03:36:26.108Ztest instance_launch::instance_launch ... ok
10292023-10-04T03:36:26.138Z
10302023-10-04T03:36:26.166Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 172.46s
10312023-10-04T03:36:26.191Z
10322023-10-04T03:36:26.216Z+ _exit_trap
10332023-10-04T03:36:26.240Z+ local status=0
10342023-10-04T03:36:26.265Z+ [[ 0 -eq 0 ]]
10352023-10-04T03:36:26.290Z+ exit 0
10362023-10-04T03:36:26.315Zprocess exited: duration 519566 ms, exit code 0
 
10372023-10-04T03:36:26.409Zfound 51 output files
10382023-10-04T03:36:26.474Zuploading: /var/svc/log/oxide-sled-agent:default.log (423196 bytes)
10392023-10-04T03:36:29.866Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10402023-10-04T03:36:29.891Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_8ae1d36c-d1c5-436e-a4b2-9bf73e501913/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10412023-10-04T03:36:30.979Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_8ae1d36c-d1c5-436e-a4b2-9bf73e501913/root/var/svc/log/oxide-cockroachdb:default.log
10422023-10-04T03:36:31.014Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8014969b-d87d-4850-99f9-08e4a6482d60/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10432023-10-04T03:36:32.945Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8014969b-d87d-4850-99f9-08e4a6482d60/root/var/svc/log/oxide-crucible-agent:default.log
10442023-10-04T03:36:32.969Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8014969b-d87d-4850-99f9-08e4a6482d60/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10452023-10-04T03:36:34.059Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8014969b-d87d-4850-99f9-08e4a6482d60/root/var/svc/log/oxide-crucible-downstairs:default.log
10462023-10-04T03:36:34.089Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_11eeeb5f-9e93-4dbf-94ee-9e2ba70ac8ac/root/var/svc/log/oxide-crucible-agent:default.log (14332 bytes)
10472023-10-04T03:36:35.193Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_11eeeb5f-9e93-4dbf-94ee-9e2ba70ac8ac/root/var/svc/log/oxide-crucible-agent:default.log
10482023-10-04T03:36:35.217Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_11eeeb5f-9e93-4dbf-94ee-9e2ba70ac8ac/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10492023-10-04T03:36:36.318Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_11eeeb5f-9e93-4dbf-94ee-9e2ba70ac8ac/root/var/svc/log/oxide-crucible-downstairs:default.log
10502023-10-04T03:36:36.360Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_11eeeb5f-9e93-4dbf-94ee-9e2ba70ac8ac/root/var/svc/log/oxide-crucible-downstairs:downstairs-23848c0a-5f97-4555-82fb-6c5d3e06afe9.log (3493 bytes)
10512023-10-04T03:36:38.693Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_11eeeb5f-9e93-4dbf-94ee-9e2ba70ac8ac/root/var/svc/log/oxide-crucible-downstairs:downstairs-23848c0a-5f97-4555-82fb-6c5d3e06afe9.log
10522023-10-04T03:36:38.721Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_a554899d-6739-40f1-81e8-0fd7e9cab2e3/root/var/svc/log/oxide-external_dns:default.log (15613 bytes)
10532023-10-04T03:36:39.814Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_a554899d-6739-40f1-81e8-0fd7e9cab2e3/root/var/svc/log/oxide-external_dns:default.log
10542023-10-04T03:36:39.838Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078/root/var/svc/log/oxide-ntp:default.log (137 bytes)
10552023-10-04T03:36:40.918Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078/root/var/svc/log/oxide-ntp:default.log
10562023-10-04T03:36:40.954Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078/root/var/svc/log/oxide-ntp:default.log.0 (912 bytes)
10572023-10-04T03:36:43.185Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078/root/var/svc/log/oxide-ntp:default.log.0
10582023-10-04T03:36:45.224Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_ce09de3c-a76c-4455-9255-0653083262d1/root/var/svc/log/oxide-clickhouse:default.log (594478 bytes)
10592023-10-04T03:37:11.117Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_ce09de3c-a76c-4455-9255-0653083262d1/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 594478 -> 738659
10602023-10-04T03:37:13.851Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_ce09de3c-a76c-4455-9255-0653083262d1/root/var/svc/log/oxide-clickhouse:default.log
10612023-10-04T03:37:14.357Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_f3c2490d-b439-42e1-aaf7-7777c8f2d8f6/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10622023-10-04T03:37:15.693Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_f3c2490d-b439-42e1-aaf7-7777c8f2d8f6/root/var/svc/log/oxide-crucible-pantry:default.log
10632023-10-04T03:37:15.787Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_675f4e05-0b20-4fe8-afee-c8b9263333ed/root/var/svc/log/oxide-internal_dns:default.log (93573 bytes)
10642023-10-04T03:37:16.091Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_675f4e05-0b20-4fe8-afee-c8b9263333ed/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93573 -> 97326
10652023-10-04T03:37:17.192Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_675f4e05-0b20-4fe8-afee-c8b9263333ed/root/var/svc/log/oxide-internal_dns:default.log
10662023-10-04T03:37:17.220Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_675f4e05-0b20-4fe8-afee-c8b9263333ed/root/var/svc/log/oxide-internal_dns:default.log.0 (10964 bytes)
10672023-10-04T03:37:18.380Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_675f4e05-0b20-4fe8-afee-c8b9263333ed/root/var/svc/log/oxide-internal_dns:default.log.0
10682023-10-04T03:37:18.403Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_a5ef4f5c-6322-4d23-87bc-d5eda5bd603a/root/var/svc/log/oxide-nexus:default.log (537473 bytes)
10692023-10-04T03:37:19.289Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_a5ef4f5c-6322-4d23-87bc-d5eda5bd603a/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 537473 -> 598687
10702023-10-04T03:37:20.780Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_a5ef4f5c-6322-4d23-87bc-d5eda5bd603a/root/var/svc/log/oxide-nexus:default.log
10712023-10-04T03:37:20.803Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_b62050b7-bdb4-4758-b7fe-c9540f194342/root/var/svc/log/oxide-oximeter:default.log (169947 bytes)
10722023-10-04T03:37:21.283Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_b62050b7-bdb4-4758-b7fe-c9540f194342/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 169947 -> 215871
10732023-10-04T03:37:22.365Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_b62050b7-bdb4-4758-b7fe-c9540f194342/root/var/svc/log/oxide-oximeter:default.log
10742023-10-04T03:37:22.393Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_1dc39348-9ad4-452f-938d-b431954ef7fe/root/var/svc/log/oxide-cockroachdb:default.log (5344 bytes)
10752023-10-04T03:37:23.470Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_1dc39348-9ad4-452f-938d-b431954ef7fe/root/var/svc/log/oxide-cockroachdb:default.log
10762023-10-04T03:37:23.499Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_45f23105-f79e-4967-b954-e80b6aedf7ba/root/var/svc/log/oxide-crucible-agent:default.log (13867 bytes)
10772023-10-04T03:37:24.618Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_45f23105-f79e-4967-b954-e80b6aedf7ba/root/var/svc/log/oxide-crucible-agent:default.log
10782023-10-04T03:37:24.647Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_45f23105-f79e-4967-b954-e80b6aedf7ba/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10792023-10-04T03:37:26.173Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_45f23105-f79e-4967-b954-e80b6aedf7ba/root/var/svc/log/oxide-crucible-downstairs:default.log
10802023-10-04T03:37:26.196Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_45f23105-f79e-4967-b954-e80b6aedf7ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-6fd59a91-f6fd-4d83-ba46-a2e5be18465e.log (3490 bytes)
10812023-10-04T03:37:27.271Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_45f23105-f79e-4967-b954-e80b6aedf7ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-6fd59a91-f6fd-4d83-ba46-a2e5be18465e.log
10822023-10-04T03:37:27.296Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_ba7933f6-ea13-4233-982a-b099979ba76e/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10832023-10-04T03:37:28.370Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_ba7933f6-ea13-4233-982a-b099979ba76e/root/var/svc/log/oxide-crucible-pantry:default.log
10842023-10-04T03:37:28.408Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_00118fe3-c1a9-46a6-a271-5d9e3a0b8a4f/root/var/svc/log/oxide-crucible-agent:default.log (14765 bytes)
10852023-10-04T03:37:29.526Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_00118fe3-c1a9-46a6-a271-5d9e3a0b8a4f/root/var/svc/log/oxide-crucible-agent:default.log
10862023-10-04T03:37:29.550Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_00118fe3-c1a9-46a6-a271-5d9e3a0b8a4f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10872023-10-04T03:37:30.623Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_00118fe3-c1a9-46a6-a271-5d9e3a0b8a4f/root/var/svc/log/oxide-crucible-downstairs:default.log
10882023-10-04T03:37:30.647Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_00118fe3-c1a9-46a6-a271-5d9e3a0b8a4f/root/var/svc/log/oxide-crucible-downstairs:downstairs-e7b90ca6-21d4-4704-9f01-64c697a9ff1b.log (5965 bytes)
10892023-10-04T03:37:31.744Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_00118fe3-c1a9-46a6-a271-5d9e3a0b8a4f/root/var/svc/log/oxide-crucible-downstairs:downstairs-e7b90ca6-21d4-4704-9f01-64c697a9ff1b.log
10902023-10-04T03:37:31.768Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_38493c3d-0602-4469-9cab-96914d36ae3c/root/var/svc/log/oxide-crucible-agent:default.log (14765 bytes)
10912023-10-04T03:37:32.892Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_38493c3d-0602-4469-9cab-96914d36ae3c/root/var/svc/log/oxide-crucible-agent:default.log
10922023-10-04T03:37:32.915Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_38493c3d-0602-4469-9cab-96914d36ae3c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10932023-10-04T03:37:33.989Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_38493c3d-0602-4469-9cab-96914d36ae3c/root/var/svc/log/oxide-crucible-downstairs:default.log
10942023-10-04T03:37:34.014Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_38493c3d-0602-4469-9cab-96914d36ae3c/root/var/svc/log/oxide-crucible-downstairs:downstairs-534f7dd0-81c3-46b3-b9dd-d0c32359ae92.log (5965 bytes)
10952023-10-04T03:37:35.116Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_38493c3d-0602-4469-9cab-96914d36ae3c/root/var/svc/log/oxide-crucible-downstairs:downstairs-534f7dd0-81c3-46b3-b9dd-d0c32359ae92.log
10962023-10-04T03:37:35.145Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_73767b22-ea59-4e7c-8347-14068d336c9a/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10972023-10-04T03:37:36.218Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_73767b22-ea59-4e7c-8347-14068d336c9a/root/var/svc/log/oxide-crucible-agent:default.log
10982023-10-04T03:37:36.248Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_73767b22-ea59-4e7c-8347-14068d336c9a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10992023-10-04T03:37:37.324Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_73767b22-ea59-4e7c-8347-14068d336c9a/root/var/svc/log/oxide-crucible-downstairs:default.log
11002023-10-04T03:37:37.348Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_da4b824e-a10a-4c05-9ac1-b255f3efb9bc/root/var/svc/log/oxide-external_dns:default.log (15607 bytes)
11012023-10-04T03:37:37.441Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_da4b824e-a10a-4c05-9ac1-b255f3efb9bc/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15607 -> 19382
11022023-10-04T03:37:38.515Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_da4b824e-a10a-4c05-9ac1-b255f3efb9bc/root/var/svc/log/oxide-external_dns:default.log
11032023-10-04T03:37:38.539Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_92fc5a8d-b1d9-499c-aa13-a53df961c39f/root/var/svc/log/oxide-internal_dns:default.log (93572 bytes)
11042023-10-04T03:37:38.844Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_92fc5a8d-b1d9-499c-aa13-a53df961c39f/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93572 -> 97326
11052023-10-04T03:37:39.920Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_92fc5a8d-b1d9-499c-aa13-a53df961c39f/root/var/svc/log/oxide-internal_dns:default.log
11062023-10-04T03:37:39.944Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_92fc5a8d-b1d9-499c-aa13-a53df961c39f/root/var/svc/log/oxide-internal_dns:default.log.0 (6000 bytes)
11072023-10-04T03:37:41.044Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_92fc5a8d-b1d9-499c-aa13-a53df961c39f/root/var/svc/log/oxide-internal_dns:default.log.0
11082023-10-04T03:37:41.067Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_7eff07a3-ba8c-4961-8e69-36f44a320721/root/var/svc/log/oxide-nexus:default.log (4355085 bytes)
11092023-10-04T03:37:44.861Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_7eff07a3-ba8c-4961-8e69-36f44a320721/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4355085 -> 4418134
11102023-10-04T03:37:45.944Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_7eff07a3-ba8c-4961-8e69-36f44a320721/root/var/svc/log/oxide-nexus:default.log
11112023-10-04T03:37:45.967Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_e6e0c0db-0148-4a94-a88d-a04eadd0434d/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
11122023-10-04T03:37:47.042Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_e6e0c0db-0148-4a94-a88d-a04eadd0434d/root/var/svc/log/oxide-cockroachdb:default.log
11132023-10-04T03:37:47.066Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_18333bc9-b777-4854-9d97-8df72e7fcc40/root/var/svc/log/oxide-crucible-agent:default.log (14764 bytes)
11142023-10-04T03:37:48.194Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_18333bc9-b777-4854-9d97-8df72e7fcc40/root/var/svc/log/oxide-crucible-agent:default.log
11152023-10-04T03:37:48.218Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_18333bc9-b777-4854-9d97-8df72e7fcc40/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11162023-10-04T03:37:49.303Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_18333bc9-b777-4854-9d97-8df72e7fcc40/root/var/svc/log/oxide-crucible-downstairs:default.log
11172023-10-04T03:37:49.328Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_18333bc9-b777-4854-9d97-8df72e7fcc40/root/var/svc/log/oxide-crucible-downstairs:downstairs-40bbf143-e3a8-4a3d-8d65-798582ac11b5.log (5961 bytes)
11182023-10-04T03:37:50.425Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_18333bc9-b777-4854-9d97-8df72e7fcc40/root/var/svc/log/oxide-crucible-downstairs:downstairs-40bbf143-e3a8-4a3d-8d65-798582ac11b5.log
11192023-10-04T03:37:50.452Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_f48bd1fc-2a67-481f-8b73-d886279b1a02/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
11202023-10-04T03:37:51.605Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_f48bd1fc-2a67-481f-8b73-d886279b1a02/root/var/svc/log/oxide-crucible-pantry:default.log
11212023-10-04T03:37:51.630Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c1082ae3-e529-48ae-96a8-b78d56b6098a/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
11222023-10-04T03:37:52.622Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c1082ae3-e529-48ae-96a8-b78d56b6098a/root/var/svc/log/oxide-cockroachdb:default.log
11232023-10-04T03:37:52.647Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_16e72fae-e475-42c4-a84d-dafe6736efd0/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
11242023-10-04T03:37:53.715Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_16e72fae-e475-42c4-a84d-dafe6736efd0/root/var/svc/log/oxide-crucible-agent:default.log
11252023-10-04T03:37:53.746Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_16e72fae-e475-42c4-a84d-dafe6736efd0/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11262023-10-04T03:37:55.239Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_16e72fae-e475-42c4-a84d-dafe6736efd0/root/var/svc/log/oxide-crucible-downstairs:default.log
11272023-10-04T03:37:55.263Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c4256cb6-1bdc-4343-8beb-a3cc46aec877/root/var/svc/log/oxide-crucible-agent:default.log (14760 bytes)
11282023-10-04T03:37:56.377Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c4256cb6-1bdc-4343-8beb-a3cc46aec877/root/var/svc/log/oxide-crucible-agent:default.log
11292023-10-04T03:37:56.416Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c4256cb6-1bdc-4343-8beb-a3cc46aec877/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11302023-10-04T03:37:57.491Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c4256cb6-1bdc-4343-8beb-a3cc46aec877/root/var/svc/log/oxide-crucible-downstairs:default.log
11312023-10-04T03:37:57.519Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c4256cb6-1bdc-4343-8beb-a3cc46aec877/root/var/svc/log/oxide-crucible-downstairs:downstairs-fc98518f-9632-484f-b4d7-2cb361f3fec0.log (3490 bytes)
11322023-10-04T03:37:58.594Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c4256cb6-1bdc-4343-8beb-a3cc46aec877/root/var/svc/log/oxide-crucible-downstairs:downstairs-fc98518f-9632-484f-b4d7-2cb361f3fec0.log
11332023-10-04T03:37:58.618Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ff433732-5304-4cd4-b553-8c0cb72ed3f1/root/var/svc/log/oxide-internal_dns:default.log (93566 bytes)
11342023-10-04T03:37:58.912Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ff433732-5304-4cd4-b553-8c0cb72ed3f1/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93566 -> 97320
11352023-10-04T03:38:00.404Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ff433732-5304-4cd4-b553-8c0cb72ed3f1/root/var/svc/log/oxide-internal_dns:default.log
11362023-10-04T03:38:00.432Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ff433732-5304-4cd4-b553-8c0cb72ed3f1/root/var/svc/log/oxide-internal_dns:default.log.0 (5908 bytes)
11372023-10-04T03:38:01.556Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ff433732-5304-4cd4-b553-8c0cb72ed3f1/root/var/svc/log/oxide-internal_dns:default.log.0
11382023-10-04T03:38:01.580Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_b7f2121c-9ff1-49a7-9ddd-0ed818f46cfd/root/var/svc/log/oxide-nexus:default.log (499789 bytes)
11392023-10-04T03:38:02.928Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_b7f2121c-9ff1-49a7-9ddd-0ed818f46cfd/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 499789 -> 563429
11402023-10-04T03:38:04.002Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_b7f2121c-9ff1-49a7-9ddd-0ed818f46cfd/root/var/svc/log/oxide-nexus:default.log
11412023-10-04T03:38:04.026Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_72f22dce-4230-42d8-a6af-a1a8b5ac54a7/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
11422023-10-04T03:38:05.101Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_72f22dce-4230-42d8-a6af-a1a8b5ac54a7/root/var/svc/log/oxide-cockroachdb:default.log
11432023-10-04T03:38:05.127Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078/root/var/log/chrony/measurements.log (14385 bytes)
11442023-10-04T03:38:05.231Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078/root/var/log/chrony/measurements.log" changed size mid upload: 14385 -> 20139
11452023-10-04T03:38:06.306Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078/root/var/log/chrony/measurements.log
11462023-10-04T03:38:06.330Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078/root/var/log/chrony/statistics.log (9477 bytes)
11472023-10-04T03:38:06.402Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078/root/var/log/chrony/statistics.log" changed size mid upload: 9477 -> 13338
11482023-10-04T03:38:07.475Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078/root/var/log/chrony/statistics.log
11492023-10-04T03:38:07.501Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078/root/var/log/chrony/tracking.log (1720 bytes)
11502023-10-04T03:38:07.538Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078/root/var/log/chrony/tracking.log" changed size mid upload: 1720 -> 1984
11512023-10-04T03:38:08.624Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_ccb5059b-9e79-4763-b76d-658c61c41078/root/var/log/chrony/tracking.log