01HD1ZC0XNYM3Y9YA1742RZGBY: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.23",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HD1ZCPZEK5P0MEZYG54ETZD7

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-18T18:54:51.352Zjob dependencies complete; ready to run (waiting for 49 m 50 s)
22023-10-18T18:55:34.621Zjob assigned to worker 01HD2283HARTBQ0FB0CBBWNPJD (queued for 43 s)
32023-10-18T18:55:34.650Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-18T18:55:34.696Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-10-18T18:55:34.750Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-10-18T18:55:34.786Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-18T18:55:34.823Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-10-18T18:55:34.856ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-18T18:55:34.890Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-18T18:55:34.923ZOxide Helios Version helios-2.0.22216 64-bit
112023-10-18T18:55:34.957ZConfiguring devices.
122023-10-18T18:55:34.990Z##################################################
132023-10-18T18:55:35.023Z#### ############# ##
142023-10-18T18:55:35.057Z### ### ############ ##
152023-10-18T18:55:35.090Z## ### # ## ### ## Oxide Computer Company ##
162023-10-18T18:55:35.127Z## ## ## ### # ### ##
172023-10-18T18:55:35.160Z## # ### #### #### This Station Under ##
182023-10-18T18:55:35.192Z### ### #### # ### Computer Control ##
192023-10-18T18:55:35.225Z#### #### ### ## ##
202023-10-18T18:55:35.259Z##################################################
212023-10-18T18:55:35.292ZHostname: buskin
222023-10-18T18:55:35.337ZLoading smf(7) service descriptions:
232023-10-18T18:55:35.993Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-0d925b23e976736e.gz
242023-10-18T18:55:38.695Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-0d925b23e976736e.gz
252023-10-18T18:55:38.726Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-18T18:55:57.688Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-18T18:55:57.712Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-9d1a76d5368d1fd3.gz
282023-10-18T18:56:35.645Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-9d1a76d5368d1fd3.gz
292023-10-18T18:56:35.675Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-18T18:56:37.168Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-18T18:56:37.205Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-18T18:56:47.753Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-18T18:56:47.784Zdownloading input: /input/package/work/version.txt
342023-10-18T18:56:47.821Zdownloaded input: /input/package/work/version.txt
352023-10-18T18:56:47.848Zdownloading input: /input/package/work/package.tar.gz
362023-10-18T18:57:19.602Zdownloaded input: /input/package/work/package.tar.gz
372023-10-18T18:57:19.630Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-18T18:58:16.293Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-18T18:58:16.344Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-18T18:58:24.641Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-18T18:58:24.667Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-18T18:59:12.252Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-18T18:59:12.288Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-18T18:59:57.332Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-18T18:59:57.357Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-18T19:00:31.226Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-18T19:00:31.251Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-18T19:00:35.191Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-18T19:00:35.216Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-18T19:00:46.961Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-18T19:00:46.985Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-18T19:00:50.945Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-18T19:00:50.987Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-18T19:00:57.404Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-18T19:00:57.429Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-18T19:00:57.453Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-18T19:00:57.477Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-18T19:01:00.392Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-18T19:01:00.418Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-18T19:01:04.848Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-18T19:01:04.879Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-18T19:01:20.094Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-18T19:01:20.118Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-18T19:01:31.750Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-18T19:01:31.801Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-18T19:01:31.825Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-18T19:01:31.849Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-18T19:01:35.009Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-18T19:01:35.034Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-18T19:01:46.422Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-18T19:01:46.445Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-18T19:02:23.947Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-18T19:02:23.975Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-18T19:02:56.146Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-18T19:02:56.177Zstarting task 0: "setup"
762023-10-18T19:02:56.211Z++ uname -s
772023-10-18T19:02:56.238Z+ kern=SunOS
782023-10-18T19:02:56.270Z+ case "$kern" in
792023-10-18T19:02:56.294Z+ groupadd -g 12345 build
802023-10-18T19:02:56.318Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-18T19:02:58.220Z+ zfs create -o mountpoint=/work rpool/work
822023-10-18T19:02:58.248Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-18T19:02:58.272Z+ home_fs=zfs
842023-10-18T19:02:58.305Z+ [[ zfs == autofs ]]
852023-10-18T19:02:58.333Z+ mkdir -p /home/build
862023-10-18T19:02:58.360Z+ chown build:build /home/build /work
872023-10-18T19:03:00.264Z+ chmod 0700 /home/build /work
882023-10-18T19:03:00.290Zprocess exited: duration 4058 ms, exit code 0
 
892023-10-18T19:03:00.340Zstarting task 1: "authentication"
902023-10-18T19:03:00.412Zprocess exited: duration 45 ms, exit code 0
 
912023-10-18T19:03:00.462Zstarting task 2: "build"
922023-10-18T19:03:00.495Z+ trap _exit_trap EXIT
932023-10-18T19:03:00.519Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-18T19:03:00.699Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-18T19:03:00.732Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-18T19:03:00.766Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-18T19:03:00.794Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-18T19:03:00.824Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-18T19:03:00.849Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-18T19:03:00.873Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-18T19:03:00.898Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-18T19:03:00.922Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-18T19:03:00.947Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-18T19:03:00.971Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-18T19:03:00.996Z+ [[ -d /opt/oxide ]]
1062023-10-18T19:03:01.020Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-18T19:03:01.063Z+ pfexec mkdir /opt/oxide
1082023-10-18T19:03:01.515Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-18T19:03:01.547Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-18T19:03:01.578Z+ cd /opt/oxide-underneath
1112023-10-18T19:03:01.609Z+ pfexec tar ceEp@/f - .
1122023-10-18T19:03:01.635Z+ cd /opt/oxide
1132023-10-18T19:03:01.662Z+ pfexec tar xveEp@/f -
1142023-10-18T19:03:01.687Zx ., 0 bytes, 0 tape blocks
1152023-10-18T19:03:01.711Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-18T19:03:01.736Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-18T19:03:01.766Zx ./opte/bin/opteadm, 50138640 bytes, 97928 tape blocks
1182023-10-18T19:03:01.792Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-18T19:03:01.816Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-18T19:03:01.841Z+ pfexec mkdir /opt/oxide/work
1212023-10-18T19:03:01.866Z+ pfexec chown build:build /opt/oxide/work
1222023-10-18T19:03:01.890Z+ cd /opt/oxide/work
1232023-10-18T19:03:01.916Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-18T19:03:02.470ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-18T19:03:02.510Zx out/maghemite.tar, 74957312 bytes, 146401 tape blocks
1262023-10-18T19:03:02.543Zx out/omicron-sled-agent.tar, 113838592 bytes, 222341 tape blocks
1272023-10-18T19:03:02.570Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-18T19:03:02.594Zx out/npuzone/npuzone, 37379944 bytes, 73008 tape blocks
1292023-10-18T19:03:02.620Zx package-manifest.toml, 21303 bytes, 42 tape blocks
1302023-10-18T19:03:02.645Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-18T19:03:02.670Zx target/release/omicron-package, 38063936 bytes, 74344 tape blocks
1322023-10-18T19:03:02.694Zx tools/create_virtual_hardware.sh, 2818 bytes, 6 tape blocks
1332023-10-18T19:03:02.723Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-18T19:03:02.751Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-18T19:03:02.776Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-18T19:03:02.801Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-18T19:03:02.839Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-18T19:03:02.869Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-18T19:03:02.894Z
1402023-10-18T19:03:02.919Zreal 1.609655845
1412023-10-18T19:03:02.951Zuser 1.446838371
1422023-10-18T19:03:02.981Zsys 0.284937757
1432023-10-18T19:03:03.018Ztrap 0.000071050
1442023-10-18T19:03:03.045Ztflt 0.000000000
1452023-10-18T19:03:03.070Zdflt 0.000000000
1462023-10-18T19:03:03.095Zkflt 0.000000000
1472023-10-18T19:03:03.121Zlock 0.000000000
1482023-10-18T19:03:03.556Zslp 1.446986929
1492023-10-18T19:03:03.586Zlat 0.038406211
1502023-10-18T19:03:03.627Zstop 0.000122020
1512023-10-18T19:03:03.663Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-18T19:03:03.692Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-18T19:03:03.721Z+ mkdir tests
1542023-10-18T19:03:03.770Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-18T19:03:03.816Z+ ptime -m gunzip
1562023-10-18T19:03:03.854Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-0d925b23e976736e
1572023-10-18T19:03:03.891Z
1582023-10-18T19:03:03.933Zreal 0.140125507
1592023-10-18T19:03:03.973Zuser 0.128529882
1602023-10-18T19:03:04.004Zsys 0.011351452
1612023-10-18T19:03:04.036Ztrap 0.000008519
1622023-10-18T19:03:04.061Ztflt 0.000000000
1632023-10-18T19:03:04.088Zdflt 0.000000000
1642023-10-18T19:03:04.113Zkflt 0.000000000
1652023-10-18T19:03:04.138Zlock 0.000000000
1662023-10-18T19:03:04.164Zslp 0.000019400
1672023-10-18T19:03:04.191Zlat 0.000020059
1682023-10-18T19:03:04.217Zstop 0.000129301
1692023-10-18T19:03:04.243Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-0d925b23e976736e
1702023-10-18T19:03:04.692Z+ chmod a+x tests/bootstrap-0d925b23e976736e
1712023-10-18T19:03:04.721Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-18T19:03:04.747Z+ ptime -m gunzip
1732023-10-18T19:03:04.774Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-18T19:03:05.867Z
1752023-10-18T19:03:06.312Zreal 2.212315818
1762023-10-18T19:03:06.348Zuser 2.075462912
1772023-10-18T19:03:06.380Zsys 0.136425320
1782023-10-18T19:03:06.414Ztrap 0.000177393
1792023-10-18T19:03:06.503Ztflt 0.000000000
1802023-10-18T19:03:06.534Zdflt 0.000000000
1812023-10-18T19:03:06.562Zkflt 0.000000000
1822023-10-18T19:03:06.592Zlock 0.000000000
1832023-10-18T19:03:06.620Zslp 0.000025040
1842023-10-18T19:03:06.647Zlat 0.000029599
1852023-10-18T19:03:06.672Zstop 0.000128911
1862023-10-18T19:03:06.696Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-18T19:03:06.724Z+ chmod a+x tests/bootstrap
1882023-10-18T19:03:06.749Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-18T19:03:06.772Z+ ptime -m gunzip
1902023-10-18T19:03:06.799Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-9d1a76d5368d1fd3
1912023-10-18T19:03:08.395Z
1922023-10-18T19:03:08.419Zreal 2.516622966
1932023-10-18T19:03:08.445Zuser 2.359232911
1942023-10-18T19:03:08.883Zsys 0.157006440
1952023-10-18T19:03:08.917Ztrap 0.000165512
1962023-10-18T19:03:10.265Ztflt 0.000000000
1972023-10-18T19:03:10.293Zdflt 0.000000000
1982023-10-18T19:03:10.328Zkflt 0.000000000
1992023-10-18T19:03:10.364Zlock 0.000000000
2002023-10-18T19:03:10.389Zslp 0.000021520
2012023-10-18T19:03:10.414Zlat 0.000018089
2022023-10-18T19:03:10.438Zstop 0.000118022
2032023-10-18T19:03:10.463Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-9d1a76d5368d1fd3
2042023-10-18T19:03:10.487Z+ chmod a+x tests/end_to_end_tests-9d1a76d5368d1fd3
2052023-10-18T19:03:10.512Z+ bmat address ls
2062023-10-18T19:03:10.536ZNAME CIDR FIRST LAST COUNT
2072023-10-18T19:03:10.560Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-10-18T19:03:10.585Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-18T19:03:10.625Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-10-18T19:03:10.651Z+ EXTRA_IP_START=10.151.1.164
2112023-10-18T19:03:10.677Z+ EXTRA_IP_END=10.151.1.227
2122023-10-18T19:03:11.245Z+ EXTRA_IP_COUNT=64
2132023-10-18T19:03:11.269Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-10-18T19:03:11.293Z+ EXTRA_IP_BASE=10.151.1
2152023-10-18T19:03:11.318Z+ EXTRA_IP_FOCTET=164
2162023-10-18T19:03:11.342Z+ EXTRA_IP_LOCTET=227
2172023-10-18T19:03:11.366Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-10-18T19:03:11.391Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-10-18T19:03:11.415Z+ DNS_IP1=10.151.1.164
2202023-10-18T19:03:11.440Z+ DNS_IP2=10.151.1.165
2212023-10-18T19:03:11.464Z+ UPLINK_IP=10.151.1.174
2222023-10-18T19:03:11.490Z+ IPPOOL_START=10.151.1.175
2232023-10-18T19:03:11.514Z+ IPPOOL_END=10.151.1.227
2242023-10-18T19:03:11.538Z++ ipadm show-addr -po type,addr
2252023-10-18T19:03:11.563Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-10-18T19:03:11.587Z+ GATEWAY_IP=10.151.1.100
2272023-10-18T19:03:11.612Z+ [[ -n 10.151.1.100 ]]
2282023-10-18T19:03:11.707Z+ ping -s 10.151.1.100 56 1
2292023-10-18T19:03:11.758ZPING 10.151.1.100: 56 data bytes
2302023-10-18T19:03:11.787Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.012 ms
2312023-10-18T19:03:11.816Z
2322023-10-18T19:03:11.847Z----10.151.1.100 PING Statistics----
2332023-10-18T19:03:11.948Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-10-18T19:03:11.996Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2352023-10-18T19:03:12.025Z++ arp -an
2362023-10-18T19:03:12.056Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-10-18T19:03:12.085Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-10-18T19:03:12.117Z+ routeadm -e ipv4-forwarding -u
2392023-10-18T19:03:12.142Z+ PXA_START=10.151.1.164
2402023-10-18T19:03:12.201Z+ PXA_END=10.151.1.227
2412023-10-18T19:03:12.235Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-10-18T19:03:12.269Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-10-18T19:03:12.306Z+ ZPOOL_VDEV_DIR=/scratch
2442023-10-18T19:03:12.339Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-10-18T19:03:12.365Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-10-18T19:03:12.389Z++ cd ./tools
2472023-10-18T19:03:12.413Z++ pwd
2482023-10-18T19:03:12.436Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-10-18T19:03:12.461Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-10-18T19:03:12.485Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-10-18T19:03:12.510Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-10-18T19:03:12.535Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-10-18T19:03:12.566Z++ dladm show-phys -p -o LINK
2542023-10-18T19:03:12.593Z++ head -1
2552023-10-18T19:03:12.619Z+ PHYSICAL_LINK=igb0
2562023-10-18T19:03:12.651Z+ echo 'Using igb0 as physical link'
2572023-10-18T19:03:12.682ZUsing igb0 as physical link
2582023-10-18T19:03:13.123Z+ ensure_run_as_root
2592023-10-18T19:03:13.153Z++ id -u
2602023-10-18T19:03:13.193ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2612023-10-18T19:03:13.230Z+ [[ 0 -ne 0 ]]
2622023-10-18T19:03:13.257ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2632023-10-18T19:03:13.282Z+ ensure_zpools
2642023-10-18T19:03:13.306ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2652023-10-18T19:03:13.332Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2662023-10-18T19:03:13.357ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2672023-10-18T19:03:13.384Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2682023-10-18T19:03:13.409ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2692023-10-18T19:03:13.433Z+ readarray -t ZPOOLS
2702023-10-18T19:03:13.457ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2712023-10-18T19:03:13.491Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2722023-10-18T19:03:13.524ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2732023-10-18T19:03:13.551Z++ sed 's/[ ",]//g'
2742023-10-18T19:03:13.582ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2752023-10-18T19:03:13.627Z+ for ZPOOL in "${ZPOOLS[@]}"
2762023-10-18T19:03:13.667ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2772023-10-18T19:03:13.692Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2782023-10-18T19:03:13.716ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2792023-10-18T19:03:13.742Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2802023-10-18T19:03:13.767ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2812023-10-18T19:03:13.796Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2822023-10-18T19:03:13.820ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2832023-10-18T19:03:13.844Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2842023-10-18T19:03:13.868ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2852023-10-18T19:03:13.892Z0+0 records in
2862023-10-18T19:03:13.922ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2872023-10-18T19:03:13.958Z0+0 records out
2882023-10-18T19:03:14.000ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2892023-10-18T19:03:14.036Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2902023-10-18T19:03:14.061ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
2912023-10-18T19:03:14.086Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2922023-10-18T19:03:14.110ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
2932023-10-18T19:03:14.134Z+ set +x
2942023-10-18T19:03:14.158ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
2952023-10-18T19:03:14.182Z++ zpool list -o name
2962023-10-18T19:03:14.206ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
2972023-10-18T19:03:14.232Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2982023-10-18T19:03:14.256ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
2992023-10-18T19:03:14.280Z+ [[ -z '' ]]
3002023-10-18T19:03:14.304ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3012023-10-18T19:03:14.328Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
3022023-10-18T19:03:14.352ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3032023-10-18T19:03:14.379Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
3042023-10-18T19:03:14.404ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3052023-10-18T19:03:14.428Z+ set +x
3062023-10-18T19:03:14.452ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3072023-10-18T19:03:14.481Z+ for ZPOOL in "${ZPOOLS[@]}"
3082023-10-18T19:03:14.507ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3092023-10-18T19:03:14.534Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3102023-10-18T19:03:14.563ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3112023-10-18T19:03:14.597Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3122023-10-18T19:03:14.641ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3132023-10-18T19:03:14.681Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3142023-10-18T19:03:14.706ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3152023-10-18T19:03:14.735Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3162023-10-18T19:03:14.768ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3172023-10-18T19:03:14.793Z0+0 records in
3182023-10-18T19:03:14.817ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3192023-10-18T19:03:14.842Z0+0 records out
3202023-10-18T19:03:14.866ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3212023-10-18T19:03:15.323Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3222023-10-18T19:03:15.347ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3232023-10-18T19:03:15.374Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3242023-10-18T19:03:15.400ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3252023-10-18T19:03:15.434Z+ set +x
3262023-10-18T19:03:15.473ZSimnet net0/sc0_0 exists
3272023-10-18T19:03:15.498Z++ zpool list -o name
3282023-10-18T19:03:15.526ZSimnet net1/sc1_0 exists
3292023-10-18T19:03:15.550Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3302023-10-18T19:03:15.575ZVnic sc0_1 exists
3312023-10-18T19:03:15.708Z+ [[ -z '' ]]
3322023-10-18T19:03:15.732Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3332023-10-18T19:03:15.757Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3342023-10-18T19:03:15.781Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3352023-10-18T19:03:15.806Z+ set +x
3362023-10-18T19:03:15.831Z+ for ZPOOL in "${ZPOOLS[@]}"
3372023-10-18T19:03:15.858Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3382023-10-18T19:03:15.884Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3392023-10-18T19:03:15.913Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3402023-10-18T19:03:15.942Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3412023-10-18T19:03:15.970Z0+0 records in
3422023-10-18T19:03:15.995Z0+0 records out
3432023-10-18T19:03:16.020Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3442023-10-18T19:03:16.055Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3452023-10-18T19:03:16.081Z+ set +x
3462023-10-18T19:03:16.105Z++ zpool list -o name
3472023-10-18T19:03:16.129Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3482023-10-18T19:03:16.157Z+ [[ -z '' ]]
3492023-10-18T19:03:16.181Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-10-18T19:03:16.205Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-10-18T19:03:16.229Z+ set +x
3522023-10-18T19:03:16.253Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-10-18T19:03:16.277Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-10-18T19:03:16.301Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-10-18T19:03:16.325Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-10-18T19:03:16.349Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-10-18T19:03:16.377Z0+0 records in
3582023-10-18T19:03:16.403Z0+0 records out
3592023-10-18T19:03:16.427Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3602023-10-18T19:03:16.451Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-10-18T19:03:16.475Z+ set +x
3622023-10-18T19:03:16.499Z++ zpool list -o name
3632023-10-18T19:03:16.532Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-10-18T19:03:16.564Z+ [[ -z '' ]]
3652023-10-18T19:03:16.596Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-10-18T19:03:16.621Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-10-18T19:03:16.645Z+ set +x
3682023-10-18T19:03:16.669Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-10-18T19:03:16.693Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-10-18T19:03:16.717Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-10-18T19:03:16.741Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-10-18T19:03:16.765Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-10-18T19:03:16.792Z0+0 records in
3742023-10-18T19:03:16.820Z0+0 records out
3752023-10-18T19:03:16.844Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3762023-10-18T19:03:16.869Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-10-18T19:03:16.898Z+ set +x
3782023-10-18T19:03:16.936Z++ zpool list -o name
3792023-10-18T19:03:16.970Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-10-18T19:03:17.420Z+ [[ -z '' ]]
3812023-10-18T19:03:17.445Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-10-18T19:03:17.493Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-10-18T19:03:17.522Z+ set +x
3842023-10-18T19:03:17.546Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-10-18T19:03:17.571Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862023-10-18T19:03:17.595Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872023-10-18T19:03:17.619Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882023-10-18T19:03:17.643Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892023-10-18T19:03:17.667Z0+0 records in
3902023-10-18T19:03:17.691Z0+0 records out
3912023-10-18T19:03:17.717Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-10-18T19:03:17.741Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932023-10-18T19:03:17.765Z+ set +x
3942023-10-18T19:03:17.793Z++ zpool list -o name
3952023-10-18T19:03:17.817Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3962023-10-18T19:03:17.844Z+ [[ -z '' ]]
3972023-10-18T19:03:17.868Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3982023-10-18T19:03:17.898Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3992023-10-18T19:03:17.934Z+ set +x
4002023-10-18T19:03:17.964Z+ for ZPOOL in "${ZPOOLS[@]}"
4012023-10-18T19:03:17.995Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4022023-10-18T19:03:18.022Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4032023-10-18T19:03:18.050Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4042023-10-18T19:03:18.086Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4052023-10-18T19:03:18.149Z0+0 records in
4062023-10-18T19:03:18.176Z0+0 records out
4072023-10-18T19:03:18.203Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4082023-10-18T19:03:18.229Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4092023-10-18T19:03:18.257Z+ set +x
4102023-10-18T19:03:18.285Z++ zpool list -o name
4112023-10-18T19:03:18.313Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4122023-10-18T19:03:18.347Z+ [[ -z '' ]]
4132023-10-18T19:03:18.376Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4142023-10-18T19:03:18.408Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4152023-10-18T19:03:18.439Z+ set +x
4162023-10-18T19:03:18.473Zdownloading softnpu ...done
4172023-10-18T19:03:18.505Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-10-18T19:03:18.546Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4192023-10-18T19:03:18.582Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4202023-10-18T19:03:18.622Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4212023-10-18T19:03:18.657Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4222023-10-18T19:03:18.689Z0+0 records in
4232023-10-18T19:03:18.726Z0+0 records out
4242023-10-18T19:03:18.759Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/4ba9c18928edece80f83153b9d78b76cf093da19/libsidecar_lite.so.sha256.txt
4252023-10-18T19:03:18.793Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4262023-10-18T19:03:18.819Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4272023-10-18T19:03:18.850Z+ set +x
4282023-10-18T19:03:18.882Z++ zpool list -o name
4292023-10-18T19:03:18.913Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4302023-10-18T19:03:18.947Z+ [[ -z '' ]]
4312023-10-18T19:03:18.978Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4322023-10-18T19:03:19.005Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4332023-10-18T19:03:19.043Z+ set +x
4342023-10-18T19:03:19.079Z+ for ZPOOL in "${ZPOOLS[@]}"
4352023-10-18T19:03:19.110Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4362023-10-18T19:03:19.137Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4372023-10-18T19:03:19.173Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4382023-10-18T19:03:19.198Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4392023-10-18T19:03:19.223Z0+0 records in
4402023-10-18T19:03:19.247Z0+0 records out
4412023-10-18T19:03:19.277Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4422023-10-18T19:03:19.742Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4432023-10-18T19:03:19.766Z+ set +x
4442023-10-18T19:03:19.790Z++ zpool list -o name
4452023-10-18T19:03:19.817Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4462023-10-18T19:03:19.846Z+ [[ -z '' ]]
4472023-10-18T19:03:19.879Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4482023-10-18T19:03:19.906Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4492023-10-18T19:03:19.932Z+ set +x
4502023-10-18T19:03:19.958Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4512023-10-18T19:03:19.992Z+ readarray -t ZPOOLS
4522023-10-18T19:03:20.017Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4532023-10-18T19:03:20.042Z++ sed 's/[ ",]//g'
4542023-10-18T19:03:20.070Z+ for ZPOOL in "${ZPOOLS[@]}"
4552023-10-18T19:03:20.102Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4562023-10-18T19:03:20.130Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4572023-10-18T19:03:20.155Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4582023-10-18T19:03:20.198Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4592023-10-18T19:03:20.229Z0+0 records in
4602023-10-18T19:03:20.260Z0+0 records out
4612023-10-18T19:03:20.289Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4622023-10-18T19:03:20.315Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4632023-10-18T19:03:20.339Z+ set +x
4642023-10-18T19:03:20.363Z++ zpool list -o name
4652023-10-18T19:03:20.387Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4662023-10-18T19:03:20.412Z+ [[ -z '' ]]
4672023-10-18T19:03:20.436Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4682023-10-18T19:03:20.461Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4692023-10-18T19:03:20.485Z+ set +x
4702023-10-18T19:03:20.509Z+ for ZPOOL in "${ZPOOLS[@]}"
4712023-10-18T19:03:20.537Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4722023-10-18T19:03:20.568Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4732023-10-18T19:03:20.658Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4742023-10-18T19:03:20.685Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4752023-10-18T19:03:20.714Z0+0 records in
4762023-10-18T19:03:20.748Z0+0 records out
4772023-10-18T19:03:20.782Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4782023-10-18T19:03:20.816Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4792023-10-18T19:03:20.848Z+ set +x
4802023-10-18T19:03:20.886Z++ zpool list -o name
4812023-10-18T19:03:20.918Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4822023-10-18T19:03:20.942Z+ [[ -z '' ]]
4832023-10-18T19:03:20.966Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4842023-10-18T19:03:20.991Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4852023-10-18T19:03:21.015Z+ set +x
4862023-10-18T19:03:21.039Z+ ensure_simulated_links igb0
4872023-10-18T19:03:21.077Z+ local PHYSICAL_LINK=igb0
4882023-10-18T19:03:21.102Z+ INDICES=("0" "1")
4892023-10-18T19:03:21.128Z+ for I in "${INDICES[@]}"
4902023-10-18T19:03:21.154Z++ get_simnet_name_if_exists net0
4912023-10-18T19:03:21.179Z++ dladm show-simnet -p -o LINK net0
4922023-10-18T19:03:21.224Z++ echo ''
4932023-10-18T19:03:21.248Z+ [[ -z '' ]]
4942023-10-18T19:03:21.289Z+ dladm create-simnet -t net0
4952023-10-18T19:03:21.319Z+ dladm create-simnet -t sc0_0
4962023-10-18T19:03:21.352Z+ dladm modify-simnet -t -p net0 sc0_0
4972023-10-18T19:03:21.383Z+ dladm set-linkprop -p mtu=9000 sc0_0
4982023-10-18T19:03:21.411Z+ success 'Simnet net0/sc0_0 exists'
4992023-10-18T19:03:21.434Z+ set +x
5002023-10-18T19:03:21.458Z+ for I in "${INDICES[@]}"
5012023-10-18T19:03:21.482Z++ get_simnet_name_if_exists net1
5022023-10-18T19:03:21.506Z++ dladm show-simnet -p -o LINK net1
5032023-10-18T19:03:21.530Z++ echo ''
5042023-10-18T19:03:21.555Z+ [[ -z '' ]]
5052023-10-18T19:03:21.581Z+ dladm create-simnet -t net1
5062023-10-18T19:03:21.605Z+ dladm create-simnet -t sc1_0
5072023-10-18T19:03:21.629Z+ dladm modify-simnet -t -p net1 sc1_0
5082023-10-18T19:03:21.653Z+ dladm set-linkprop -p mtu=9000 sc1_0
5092023-10-18T19:03:21.677Z+ success 'Simnet net1/sc1_0 exists'
5102023-10-18T19:03:21.701Z+ set +x
5112023-10-18T19:03:21.726Z++ get_vnic_name_if_exists sc0_1
5122023-10-18T19:03:21.750Z++ dladm show-vnic -p -o LINK sc0_1
5132023-10-18T19:03:21.778Z++ echo ''
5142023-10-18T19:03:21.806Z+ [[ -z '' ]]
5152023-10-18T19:03:21.837Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162023-10-18T19:03:21.877Z+ success 'Vnic sc0_1 exists'
5172023-10-18T19:03:21.916Z+ set +x
5182023-10-18T19:03:21.960Z+ warn_if_no_proxy_arp
5192023-10-18T19:03:21.985Z+ [[ -v PXA_START ]]
5202023-10-18T19:03:22.009Z+ [[ -v PXA_END ]]
5212023-10-18T19:03:22.035Z+ ensure_softnpu_zone
5222023-10-18T19:03:22.060Z+ zoneadm list
5232023-10-18T19:03:22.087Z+ grep -q sidecar_softnpu
5242023-10-18T19:03:22.111Z+ [[ -f out/npuzone/npuzone ]]
5252023-10-18T19:03:22.551Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0
5262023-10-18T19:03:24.668Zdownloading asic_program.so ...done
5272023-10-18T19:03:24.715Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/4ba9c18928edece80f83153b9d78b76cf093da19/scadm.sha256.txt
5282023-10-18T19:03:30.772Zdownloading scadm ...done
5292023-10-18T19:03:30.799Zcreating port tfportrear0_0 for sc0_0
5302023-10-18T19:03:30.837Zcreating port tfportqsfp0_0 for sc0_1
5312023-10-18T19:03:30.861Zsoftnpu zone setup
5322023-10-18T19:03:30.885Zconfigure zone
5332023-10-18T19:03:30.909Zinstall zone
5342023-10-18T19:03:32.851Zboot zone
5352023-10-18T19:03:34.974Z[sidecar_softnpu] svcs milestone/network
5362023-10-18T19:03:35.001ZSTATE STIME FMRI
5372023-10-18T19:03:35.025Zonline 19:03:33 svc:/milestone/network:default
5382023-10-18T19:03:35.049Z[sidecar_softnpu] /softnpu/npu start
5392023-10-18T19:03:35.076Z
5402023-10-18T19:03:35.101Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5412023-10-18T19:03:35.125Z+ GATEWAY_IP=10.151.1.100
5422023-10-18T19:03:35.149Z+ echo 'Using 10.151.1.100 as gateway ip'
5432023-10-18T19:03:35.173ZUsing 10.151.1.100 as gateway ip
5442023-10-18T19:03:35.198Z+ [[ ! -v GATEWAY_MAC ]]
5452023-10-18T19:03:35.222Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5462023-10-18T19:03:35.250Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5472023-10-18T19:03:35.710Z+++ echo 18:c0:4d:0d:9f:b2
5482023-10-18T19:03:35.734Z+ COUNT=1
5492023-10-18T19:03:35.758Z+ [[ 1 -ne 1 ]]
5502023-10-18T19:03:35.894Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5512023-10-18T19:03:35.918ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5522023-10-18T19:03:35.945Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5532023-10-18T19:03:35.977Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5542023-10-18T19:03:36.006Z+ PXA_MAC=a8:e1:de:01:70:1d
5552023-10-18T19:03:36.034Z+ [[ -v PXA_START ]]
5562023-10-18T19:03:36.065Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5572023-10-18T19:03:36.520Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5582023-10-18T19:03:36.545Z+ z_scadm dump-state
5592023-10-18T19:03:36.569Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5602023-10-18T19:03:36.600Zlocal v6:
5612023-10-18T19:03:36.624Zlocal v4:
5622023-10-18T19:03:36.649Zrouter v6:
5632023-10-18T19:03:36.673Zrouter v4:
5642023-10-18T19:03:36.701Zresolver v4:
5652023-10-18T19:03:36.726Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5662023-10-18T19:03:36.750Zresolver v6:
5672023-10-18T19:03:36.774Znat v4:
5682023-10-18T19:03:36.799Znat_v6:
5692023-10-18T19:03:36.824Zport_mac:
5702023-10-18T19:03:36.848Zproxy arp:
5712023-10-18T19:03:36.874Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5722023-10-18T19:03:36.944Z+ success 'softnpu zone exists'
5732023-10-18T19:03:36.970Z+ set +x
5742023-10-18T19:03:36.994Zsoftnpu zone exists
5752023-10-18T19:03:37.018Z
5762023-10-18T19:03:37.042Zreal 26.404952740
5772023-10-18T19:03:37.066Zuser 0.711078179
5782023-10-18T19:03:37.090Zsys 1.573649565
5792023-10-18T19:03:37.531Ztrap 0.000080099
5802023-10-18T19:03:37.557Ztflt 0.000154917
5812023-10-18T19:03:37.581Zdflt 0.001169986
5822023-10-18T19:03:37.607Zkflt 0.000000000
5832023-10-18T19:03:37.631Zlock 20:37.493799791
5842023-10-18T19:03:37.655Zslp 1:18.601162252
5852023-10-18T19:03:37.679Zlat 0.064242903
5862023-10-18T19:03:37.703Zstop 0.370873890
5872023-10-18T19:03:37.727Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5882023-10-18T19:03:37.751Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5892023-10-18T19:03:37.775Z+ SILO_NAME=recovery
5902023-10-18T19:03:37.799Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5912023-10-18T19:03:37.826Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5922023-10-18T19:03:37.850Z+ sed -i~ '
5932023-10-18T19:03:37.874Z /^external_dns_ips/c\
5942023-10-18T19:03:37.898Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5952023-10-18T19:03:37.922Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5962023-10-18T19:03:37.969Z /^first/c\
5972023-10-18T19:03:37.995Zfirst = "10.151.1.164"
5982023-10-18T19:03:38.019Z /^last/c\
5992023-10-18T19:03:38.043Zlast = "10.151.1.173"
6002023-10-18T19:03:38.067Z }
6012023-10-18T19:03:38.090Z /^\[rack_network_config/,/^$/ {
6022023-10-18T19:03:38.114Z /^infra_ip_first/c\
6032023-10-18T19:03:38.138Zinfra_ip_first = "10.151.1.174"
6042023-10-18T19:03:38.162Z /^infra_ip_last/c\
6052023-10-18T19:03:38.186Zinfra_ip_last = "10.151.1.174"
6062023-10-18T19:03:38.210Z }
6072023-10-18T19:03:38.234Z /^\[\[rack_network_config.uplinks/,/^$/ {
6082023-10-18T19:03:38.258Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6092023-10-18T19:03:38.282Z /^gateway_ip/c\
6102023-10-18T19:03:38.306Z+++ pkg/config-rss.toml Wed Oct 18 19:03:35 2023
6112023-10-18T19:03:38.330Zgateway_ip = "10.151.1.100"
6122023-10-18T19:03:38.355Z@@ -41,7 +41,7 @@
6132023-10-18T19:03:38.383Z /^uplink_cidr/c\
6142023-10-18T19:03:38.410Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6152023-10-18T19:03:38.434Zuplink_cidr = "10.151.1.174/32"
6162023-10-18T19:03:38.458Z # must be contained in one of the "internal services" IP Pool ranges listed
6172023-10-18T19:03:38.482Z }
6182023-10-18T19:03:38.506Z # below.
6192023-10-18T19:03:38.530Z' pkg/config-rss.toml
6202023-10-18T19:03:38.561Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6212023-10-18T19:03:38.588Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6222023-10-18T19:03:38.614Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6232023-10-18T19:03:38.637Z+ true
6242023-10-18T19:03:38.663Z
6252023-10-18T19:03:38.692Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6262023-10-18T19:03:38.716Z # Initial TLS certificates for the external API
6272023-10-18T19:03:38.744Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6282023-10-18T19:03:38.768Z #
6292023-10-18T19:03:38.792Z+ TLS_NAME=recovery.sys.oxide.test
6302023-10-18T19:03:38.816Z@@ -77,8 +77,8 @@
6312023-10-18T19:03:38.840Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6322023-10-18T19:03:38.865Z #
6332023-10-18T19:03:38.890Z....+.....+...+...+...+.+.....+......+..........+........+....+...+...+......+...+..+..........+...+........+....+...+..+.+........+.......+..+.......+......+........+.......+...+..+.......+..+......+....+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.+.........+..+...+.......+...+........+...+.+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.........+......+..+..........+..+......+..........+..+..........+..+...+...+............+......+.......+........+......+....+.....+.+........................+.........+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6342023-10-18T19:03:38.923Z # For more on this and what to put here, see docs/how-to-run.adoc.
6352023-10-18T19:03:38.956Z [[internal_services_ip_pool_ranges]]
6362023-10-18T19:03:39.038Z-first = "192.168.1.20"
6372023-10-18T19:03:39.071Z-last = "192.168.1.29"
6382023-10-18T19:03:39.510Z+first = "10.151.1.164"
6392023-10-18T19:03:39.543Z+last = "10.151.1.173"
6402023-10-18T19:03:39.569Z
6412023-10-18T19:03:39.597Z # TODO - this configuration is subject to change going forward. Ultimately these
6422023-10-18T19:03:39.632Z...+............+...+..........+.................+.+.....+.+......+........+.+...+..+.+..+.......+.........+............+.....+.......+..+....+..+....+........+.+..+............+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.....+...+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.......+...+..+.......+......+..+...............+.......+........+...+...+....+.....................+...........+.............+...+.........+...+............+......+....................+.......+...............+...+.....+............+.......+......+...+.....+....+..+...+............+...+..........+...........+...............+................+.....+....+..+...+.+.........+............+...+.....+......+....+.....+.+........+..........+...+...+..+...+......+...+...+..............................+....+...+.....+.......+........+......+...................+...+...............+..+............+...+.+..+..........+..............+....+.....+......+......+....+.....+......+.......+..+......+..........+.........+.....+......+.......+...........+....+...+..+.+...........+...+.+..............+............+.+...+.........+...+..+...+....+.....+.........+....+.....+.+......+...+.....+.............+.....+...+......+.+.....+..........+...+..............+................+......+......+...+..+............+.............+..+.....................+....+......+............+.........+...+...........+...................+..+...............+.........+....+..+.+..+......+.+......+........+................+...+..........................+......+...............+......+...+............+....+.........+..+...+....+......+......+...+.....+................+...........+...+.......+.........+......+......+.........+.....+......+.......+......+......+..................+...............+......+...........+.+.....+.........+.+.....+.......+...........+....+..+..........+.....+....+..+..........+...+.....+....+.....+.......+........+.......+......+.....+...+.+......+...+..+...............+...............+............+......+.......+...+......+.....+....+..+............+...+............................+.........+.....+.+..+...+................+...+...+...............+...+..+...+.+..............+......+...............+...+.+......+........+...+......................+............+.........+.........+...+...........................+...+..+.+..+.........+...+.............+.....+.......+...+...+..........................+.........+............+.......+..+.+...+..+..................................+...+.....+................+...........+.............+...+...+.....+......+...............+.............+...+.........+......+...+......+...........+............+...+.........+.+...+...........+.+..................+.....+....+........+.......+...+...+.....+.+.....+.+..+.+..+........................+....+............+..+...............+...+....+...+.....+.+..+..........+..+......+.+.....+...+...+....+........+................+.....+......+.+.....+...............+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6432023-10-18T19:03:39.660Z # parameters should be provided to the control plane via wicket, but we need to
6442023-10-18T19:03:39.684Z-----
6452023-10-18T19:03:39.709Z@@ -91,13 +91,13 @@
6462023-10-18T19:03:39.733Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6472023-10-18T19:03:39.757Z # A range of IP addresses used by Boundary Services on the external network. In
6482023-10-18T19:03:39.781Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6492023-10-18T19:03:39.806Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6502023-10-18T19:03:39.830Z+ rmdir pkg
6512023-10-18T19:03:39.853Z # With softnpu, only one address is used.
6522023-10-18T19:03:39.877Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6532023-10-18T19:03:39.904Z-infra_ip_first = "192.168.1.30"
6542023-10-18T19:03:39.931Z+ pfexec mkdir -p /usr/oxide
6552023-10-18T19:03:39.956Z-infra_ip_last = "192.168.1.30"
6562023-10-18T19:03:39.981Z+ pfexec rm -f /usr/oxide/tcpproxy
6572023-10-18T19:03:40.007Z+infra_ip_first = "10.151.1.174"
6582023-10-18T19:03:40.058Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6592023-10-18T19:03:40.082Z+infra_ip_last = "10.151.1.174"
6602023-10-18T19:03:40.106Z+ pfexec chmod +x /usr/oxide/tcpproxy
6612023-10-18T19:03:40.130Z
6622023-10-18T19:03:40.154Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6632023-10-18T19:03:40.178Z # You can configure multiple uplinks by repeating the following stanza
6642023-10-18T19:03:40.202Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6652023-10-18T19:03:40.226Z [[rack_network_config.uplinks]]
6662023-10-18T19:03:40.251Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6672023-10-18T19:03:40.275Z # The gateway IP for the rack's external network
6682023-10-18T19:03:40.301Z+ OMICRON_NO_UNINSTALL=1
6692023-10-18T19:03:40.327Z-gateway_ip = "192.168.1.199"
6702023-10-18T19:03:40.354Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6712023-10-18T19:03:40.380Z+gateway_ip = "10.151.1.100"
6722023-10-18T19:03:40.407ZOct 18 19:03:37.886 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6732023-10-18T19:03:40.434Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6742023-10-18T19:03:40.467ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6752023-10-18T19:03:40.497Z uplink_port = "qsfp0"
6762023-10-18T19:03:40.521ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6772023-10-18T19:03:40.545Z uplink_port_speed = "40G"
6782023-10-18T19:03:40.569ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6792023-10-18T19:03:40.621Z@@ -104,7 +104,7 @@
6802023-10-18T19:03:40.645ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6812023-10-18T19:03:40.669Z uplink_port_fec="none"
6822023-10-18T19:03:40.693ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6832023-10-18T19:03:40.717Z # For softnpu, an address within the "infra" block above that will be used for
6842023-10-18T19:03:40.742ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6852023-10-18T19:03:40.771Z # the softnpu uplink port. You can just pick the first address in that pool.
6862023-10-18T19:03:40.796ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6872023-10-18T19:03:40.822Z-uplink_cidr = "192.168.1.30/32"
6882023-10-18T19:03:40.850ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6892023-10-18T19:03:40.878Z+uplink_cidr = "10.151.1.174/32"
6902023-10-18T19:03:40.909ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/maghemite.tar
6912023-10-18T19:03:40.941Z # Switch to use for the uplink. For single-rack deployments this can be
6922023-10-18T19:03:40.976ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6932023-10-18T19:03:41.010Z # "switch0" (upper slot) or "switch1" (lower slot). For single-node softnpu
6942023-10-18T19:03:41.044ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6952023-10-18T19:03:41.087Z # and dendrite stub environments, use "switch0"
6962023-10-18T19:03:41.111ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6972023-10-18T19:03:41.135Za pkg/config-rss.toml 7K
6982023-10-18T19:03:41.159ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6992023-10-18T19:03:41.183Za pkg/initial-tls-cert.pem 2K
7002023-10-18T19:03:41.207ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7012023-10-18T19:03:41.231Za pkg/initial-tls-key.pem 4K
7022023-10-18T19:03:41.255ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7032023-10-18T19:03:41.279ZOct 18 19:03:37.889 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7042023-10-18T19:03:41.303ZOct 18 19:03:38.123 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7052023-10-18T19:03:41.327ZOct 18 19:03:38.172 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7062023-10-18T19:03:41.351ZOct 18 19:03:38.232 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7072023-10-18T19:03:41.378Z
7082023-10-18T19:03:41.404Zreal 0.482618906
7092023-10-18T19:03:41.429Zuser 0.095876496
7102023-10-18T19:03:41.461Zsys 1.933083699
7112023-10-18T19:03:41.487Ztrap 0.000000280
7122023-10-18T19:03:41.512Ztflt 0.000032328
7132023-10-18T19:03:41.537Zdflt 0.000802764
7142023-10-18T19:03:41.563Zkflt 0.000000000
7152023-10-18T19:03:41.590Zlock 43.379674862
7162023-10-18T19:03:41.617Zslp 0.683703201
7172023-10-18T19:03:41.642Zlat 0.002902103
7182023-10-18T19:03:41.667Zstop 0.219086956
7192023-10-18T19:03:41.691Z+ retry=0
7202023-10-18T19:03:41.715Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7212023-10-18T19:03:41.739Z+ [[ 0 -gt 30 ]]
7222023-10-18T19:03:41.764Z+ sleep 1
7232023-10-18T19:03:41.788Z+ retry=1
7242023-10-18T19:03:41.812Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7252023-10-18T19:03:41.836Z+ [[ 1 -gt 30 ]]
7262023-10-18T19:03:41.861Z+ sleep 1
7272023-10-18T19:03:41.892Z+ retry=2
7282023-10-18T19:03:41.917Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7292023-10-18T19:03:59.679Z+ [[ 2 -gt 30 ]]
7302023-10-18T19:03:59.703Z+ sleep 1
7312023-10-18T19:04:00.691Z+ retry=3
7322023-10-18T19:04:00.719Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7332023-10-18T19:04:00.745Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7342023-10-18T19:04:00.769Zlocal v6:
7352023-10-18T19:04:00.793Zfe80::802d:3fff:fea7:4684
7362023-10-18T19:04:00.817Zfd00:99::1
7372023-10-18T19:04:00.842Zlocal v4:
7382023-10-18T19:04:00.867Z10.151.1.174
7392023-10-18T19:04:00.891Zrouter v6:
7402023-10-18T19:04:00.915Zrouter v4:
7412023-10-18T19:04:00.939Zexpected [port, address], found: [2, 0, 64, 1, 97, a, 0, 0]
7422023-10-18T19:04:00.964Z0.0.0.0/0 -> ?
7432023-10-18T19:04:00.988Zresolver v4:
7442023-10-18T19:04:01.012Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7452023-10-18T19:04:01.038Zresolver v6:
7462023-10-18T19:04:01.062Zfe80::802d:3fff:fea7:4684 -> 82:2d:3f:a7:46:84
7472023-10-18T19:04:01.089Zfe80::c88c:acff:fe4c:709d -> ca:8c:ac:4c:70:9d
7482023-10-18T19:04:01.114Znat v4:
7492023-10-18T19:04:01.142Znat_v6:
7502023-10-18T19:04:01.169Zport_mac:
7512023-10-18T19:04:01.194Z1000: a8:40:25:2a:88:4d
7522023-10-18T19:04:01.224Z1: a8:40:25:2a:88:4e
7532023-10-18T19:04:01.248Zproxy arp:
7542023-10-18T19:04:01.275Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7552023-10-18T19:04:01.300Z+ retry=0
7562023-10-18T19:04:01.326Z+++ zoneadm list -n
7572023-10-18T19:04:01.350Z+++ grep oxz_ntp
7582023-10-18T19:04:01.375Z++ pfexec svcs -z -Hostate oxide/ntp
7592023-10-18T19:04:01.400Zsvcs: zone '-Hostate': entity not found
7602023-10-18T19:04:01.424Z++ true
7612023-10-18T19:04:01.449Z+ [[ '' != online ]]
7622023-10-18T19:04:01.473Z+ [[ 0 -gt 60 ]]
7632023-10-18T19:04:01.498Z+ sleep 1
7642023-10-18T19:04:01.744Z+ retry=1
7652023-10-18T19:04:01.769Z+++ zoneadm list -n
7662023-10-18T19:04:01.794Z+++ grep oxz_ntp
7672023-10-18T19:04:01.818Z++ pfexec svcs -z -Hostate oxide/ntp
7682023-10-18T19:04:01.845Zsvcs: zone '-Hostate': entity not found
7692023-10-18T19:04:01.870Z++ true
7702023-10-18T19:04:01.895Z+ [[ '' != online ]]
7712023-10-18T19:04:01.920Z+ [[ 1 -gt 60 ]]
7722023-10-18T19:04:01.945Z+ sleep 1
7732023-10-18T19:04:02.769Z+ retry=2
7742023-10-18T19:04:02.794Z+++ zoneadm list -n
7752023-10-18T19:04:02.819Z+++ grep oxz_ntp
7762023-10-18T19:04:02.894Z++ pfexec svcs -z -Hostate oxide/ntp
7772023-10-18T19:04:02.922Zsvcs: zone '-Hostate': entity not found
7782023-10-18T19:04:02.947Z++ true
7792023-10-18T19:04:02.972Z+ [[ '' != online ]]
7802023-10-18T19:04:02.999Z+ [[ 2 -gt 60 ]]
7812023-10-18T19:04:03.025Z+ sleep 1
7822023-10-18T19:04:03.791Z+ retry=3
7832023-10-18T19:04:03.815Z+++ zoneadm list -n
7842023-10-18T19:04:03.840Z+++ grep oxz_ntp
7852023-10-18T19:04:03.865Z++ pfexec svcs -z -Hostate oxide/ntp
7862023-10-18T19:04:03.891Zsvcs: zone '-Hostate': entity not found
7872023-10-18T19:04:03.917Z++ true
7882023-10-18T19:04:03.942Z+ [[ '' != online ]]
7892023-10-18T19:04:03.967Z+ [[ 3 -gt 60 ]]
7902023-10-18T19:04:03.991Z+ sleep 1
7912023-10-18T19:04:04.817Z+ retry=4
7922023-10-18T19:04:04.847Z+++ zoneadm list -n
7932023-10-18T19:04:04.875Z+++ grep oxz_ntp
7942023-10-18T19:04:04.899Z++ pfexec svcs -z -Hostate oxide/ntp
7952023-10-18T19:04:04.925Zsvcs: zone '-Hostate': entity not found
7962023-10-18T19:04:04.950Z++ true
7972023-10-18T19:04:04.975Z+ [[ '' != online ]]
7982023-10-18T19:04:04.999Z+ [[ 4 -gt 60 ]]
7992023-10-18T19:04:05.024Z+ sleep 1
8002023-10-18T19:04:05.842Z+ retry=5
8012023-10-18T19:04:05.866Z+++ zoneadm list -n
8022023-10-18T19:04:05.890Z+++ grep oxz_ntp
8032023-10-18T19:04:05.914Z++ pfexec svcs -z -Hostate oxide/ntp
8042023-10-18T19:04:05.938Zsvcs: zone '-Hostate': entity not found
8052023-10-18T19:04:05.962Z++ true
8062023-10-18T19:04:05.988Z+ [[ '' != online ]]
8072023-10-18T19:04:06.012Z+ [[ 5 -gt 60 ]]
8082023-10-18T19:04:06.037Z+ sleep 1
8092023-10-18T19:04:06.863Z+ retry=6
8102023-10-18T19:04:06.888Z+++ zoneadm list -n
8112023-10-18T19:04:06.913Z+++ grep oxz_ntp
8122023-10-18T19:04:06.939Z++ pfexec svcs -z -Hostate oxide/ntp
8132023-10-18T19:04:06.965Zsvcs: zone '-Hostate': entity not found
8142023-10-18T19:04:06.990Z++ true
8152023-10-18T19:04:07.015Z+ [[ '' != online ]]
8162023-10-18T19:04:07.040Z+ [[ 6 -gt 60 ]]
8172023-10-18T19:04:07.064Z+ sleep 1
8182023-10-18T19:04:07.885Z+ retry=7
8192023-10-18T19:04:07.909Z+++ zoneadm list -n
8202023-10-18T19:04:07.933Z+++ grep oxz_ntp
8212023-10-18T19:04:07.958Z++ pfexec svcs -z -Hostate oxide/ntp
8222023-10-18T19:04:07.982Zsvcs: zone '-Hostate': entity not found
8232023-10-18T19:04:08.006Z++ true
8242023-10-18T19:04:08.030Z+ [[ '' != online ]]
8252023-10-18T19:04:08.054Z+ [[ 7 -gt 60 ]]
8262023-10-18T19:04:08.081Z+ sleep 1
8272023-10-18T19:04:08.908Z+ retry=8
8282023-10-18T19:04:08.932Z+++ zoneadm list -n
8292023-10-18T19:04:08.963Z+++ grep oxz_ntp
8302023-10-18T19:04:08.990Z++ pfexec svcs -z -Hostate oxide/ntp
8312023-10-18T19:04:09.017Zsvcs: zone '-Hostate': entity not found
8322023-10-18T19:04:09.044Z++ true
8332023-10-18T19:04:09.069Z+ [[ '' != online ]]
8342023-10-18T19:04:09.093Z+ [[ 8 -gt 60 ]]
8352023-10-18T19:04:09.117Z+ sleep 1
8362023-10-18T19:04:09.931Z+ retry=9
8372023-10-18T19:04:09.956Z+++ zoneadm list -n
8382023-10-18T19:04:09.984Z+++ grep oxz_ntp
8392023-10-18T19:04:10.008Z++ pfexec svcs -z -Hostate oxide/ntp
8402023-10-18T19:04:10.033Zsvcs: zone '-Hostate': entity not found
8412023-10-18T19:04:10.057Z++ true
8422023-10-18T19:04:10.081Z+ [[ '' != online ]]
8432023-10-18T19:04:10.105Z+ [[ 9 -gt 60 ]]
8442023-10-18T19:04:10.130Z+ sleep 1
8452023-10-18T19:04:10.952Z+ retry=10
8462023-10-18T19:04:10.976Z+++ zoneadm list -n
8472023-10-18T19:04:11.001Z+++ grep oxz_ntp
8482023-10-18T19:04:11.025Z++ pfexec svcs -z -Hostate oxide/ntp
8492023-10-18T19:04:11.050Zsvcs: zone '-Hostate': entity not found
8502023-10-18T19:04:11.075Z++ true
8512023-10-18T19:04:11.101Z+ [[ '' != online ]]
8522023-10-18T19:04:11.125Z+ [[ 10 -gt 60 ]]
8532023-10-18T19:04:11.149Z+ sleep 1
8542023-10-18T19:04:11.974Z+ retry=11
8552023-10-18T19:04:11.998Z+++ zoneadm list -n
8562023-10-18T19:04:12.022Z+++ grep oxz_ntp
8572023-10-18T19:04:12.046Z++ pfexec svcs -z -Hostate oxide/ntp
8582023-10-18T19:04:12.070Zsvcs: zone '-Hostate': entity not found
8592023-10-18T19:04:12.094Z++ true
8602023-10-18T19:04:12.118Z+ [[ '' != online ]]
8612023-10-18T19:04:12.143Z+ [[ 11 -gt 60 ]]
8622023-10-18T19:04:12.171Z+ sleep 1
8632023-10-18T19:04:12.996Z+ retry=12
8642023-10-18T19:04:13.021Z+++ zoneadm list -n
8652023-10-18T19:04:13.046Z+++ grep oxz_ntp
8662023-10-18T19:04:13.072Z++ pfexec svcs -z -Hostate oxide/ntp
8672023-10-18T19:04:13.096Zsvcs: zone '-Hostate': entity not found
8682023-10-18T19:04:13.121Z++ true
8692023-10-18T19:04:13.147Z+ [[ '' != online ]]
8702023-10-18T19:04:13.172Z+ [[ 12 -gt 60 ]]
8712023-10-18T19:04:13.197Z+ sleep 1
8722023-10-18T19:04:14.018Z+ retry=13
8732023-10-18T19:04:14.042Z+++ zoneadm list -n
8742023-10-18T19:04:14.066Z+++ grep oxz_ntp
8752023-10-18T19:04:14.090Z++ pfexec svcs -z -Hostate oxide/ntp
8762023-10-18T19:04:14.114Zsvcs: zone '-Hostate': entity not found
8772023-10-18T19:04:14.138Z++ true
8782023-10-18T19:04:14.162Z+ [[ '' != online ]]
8792023-10-18T19:04:14.186Z+ [[ 13 -gt 60 ]]
8802023-10-18T19:04:14.210Z+ sleep 1
8812023-10-18T19:04:15.043Z+ retry=14
8822023-10-18T19:04:15.067Z+++ zoneadm list -n
8832023-10-18T19:04:15.093Z+++ grep oxz_ntp
8842023-10-18T19:04:15.117Z++ pfexec svcs -z oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2 -Hostate oxide/ntp
8852023-10-18T19:04:15.142Zsvcs: Could not bind to repository server for zone oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2: repository server unavailable
8862023-10-18T19:04:15.166Z++ true
8872023-10-18T19:04:15.190Z+ [[ '' != online ]]
8882023-10-18T19:04:15.213Z+ [[ 14 -gt 60 ]]
8892023-10-18T19:04:15.237Z+ sleep 1
8902023-10-18T19:04:16.077Z+ retry=15
8912023-10-18T19:04:16.101Z+++ zoneadm list -n
8922023-10-18T19:04:16.125Z+++ grep oxz_ntp
8932023-10-18T19:04:16.149Z++ pfexec svcs -z oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2 -Hostate oxide/ntp
8942023-10-18T19:04:16.188Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8952023-10-18T19:04:16.212Z++ true
8962023-10-18T19:04:16.237Z+ [[ '' != online ]]
8972023-10-18T19:04:16.264Z+ [[ 15 -gt 60 ]]
8982023-10-18T19:04:16.295Z+ sleep 1
8992023-10-18T19:04:17.101Z+ retry=16
9002023-10-18T19:04:17.128Z+++ zoneadm list -n
9012023-10-18T19:04:17.152Z+++ grep oxz_ntp
9022023-10-18T19:04:17.176Z++ pfexec svcs -z oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2 -Hostate oxide/ntp
9032023-10-18T19:04:17.201Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9042023-10-18T19:04:17.231Z++ true
9052023-10-18T19:04:17.259Z+ [[ '' != online ]]
9062023-10-18T19:04:17.283Z+ [[ 16 -gt 60 ]]
9072023-10-18T19:04:17.306Z+ sleep 1
9082023-10-18T19:04:18.131Z+ retry=17
9092023-10-18T19:04:18.156Z+++ zoneadm list -n
9102023-10-18T19:04:18.187Z+++ grep oxz_ntp
9112023-10-18T19:04:18.212Z++ pfexec svcs -z oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2 -Hostate oxide/ntp
9122023-10-18T19:04:18.236Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9132023-10-18T19:04:18.260Z++ true
9142023-10-18T19:04:18.284Z+ [[ '' != online ]]
9152023-10-18T19:04:18.308Z+ [[ 17 -gt 60 ]]
9162023-10-18T19:04:18.332Z+ sleep 1
9172023-10-18T19:04:19.158Z+ retry=18
9182023-10-18T19:04:19.182Z+++ zoneadm list -n
9192023-10-18T19:04:19.206Z+++ grep oxz_ntp
9202023-10-18T19:04:19.230Z++ pfexec svcs -z oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2 -Hostate oxide/ntp
9212023-10-18T19:04:19.254Z+ [[ disabled != online ]]
9222023-10-18T19:04:19.278Z+ [[ 18 -gt 60 ]]
9232023-10-18T19:04:19.302Z+ sleep 1
9242023-10-18T19:04:20.181Z+ retry=19
9252023-10-18T19:04:20.207Z+++ zoneadm list -n
9262023-10-18T19:04:20.232Z+++ grep oxz_ntp
9272023-10-18T19:04:20.256Z++ pfexec svcs -z oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2 -Hostate oxide/ntp
9282023-10-18T19:04:20.280Z+ [[ disabled != online ]]
9292023-10-18T19:04:20.305Z+ [[ 19 -gt 60 ]]
9302023-10-18T19:04:20.329Z+ sleep 1
9312023-10-18T19:04:21.216Z+ retry=20
9322023-10-18T19:04:21.241Z+++ zoneadm list -n
9332023-10-18T19:04:21.265Z+++ grep oxz_ntp
9342023-10-18T19:04:21.289Z++ pfexec svcs -z oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2 -Hostate oxide/ntp
9352023-10-18T19:04:21.313Z+ [[ disabled != online ]]
9362023-10-18T19:04:21.337Z+ [[ 20 -gt 60 ]]
9372023-10-18T19:04:21.362Z+ sleep 1
9382023-10-18T19:04:22.233Z+ retry=21
9392023-10-18T19:04:22.258Z+++ zoneadm list -n
9402023-10-18T19:04:22.282Z+++ grep oxz_ntp
9412023-10-18T19:04:22.306Z++ pfexec svcs -z oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2 -Hostate oxide/ntp
9422023-10-18T19:04:22.329Z+ [[ disabled != online ]]
9432023-10-18T19:04:22.353Z+ [[ 21 -gt 60 ]]
9442023-10-18T19:04:22.377Z+ sleep 1
9452023-10-18T19:04:23.258Z+ retry=22
9462023-10-18T19:04:23.282Z+++ zoneadm list -n
9472023-10-18T19:04:23.307Z+++ grep oxz_ntp
9482023-10-18T19:04:23.332Z++ pfexec svcs -z oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2 -Hostate oxide/ntp
9492023-10-18T19:04:23.356Z+ [[ disabled != online ]]
9502023-10-18T19:04:23.381Z+ [[ 22 -gt 60 ]]
9512023-10-18T19:04:23.405Z+ sleep 1
9522023-10-18T19:04:24.279Z+ retry=23
9532023-10-18T19:04:24.304Z+++ zoneadm list -n
9542023-10-18T19:04:24.331Z+++ grep oxz_ntp
9552023-10-18T19:04:24.355Z++ pfexec svcs -z oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2 -Hostate oxide/ntp
9562023-10-18T19:04:24.793Z+ [[ disabled != online ]]
9572023-10-18T19:04:24.817Z+ [[ 23 -gt 60 ]]
9582023-10-18T19:04:24.842Z+ sleep 1
9592023-10-18T19:04:25.304Z+ retry=24
9602023-10-18T19:04:25.334Z+++ zoneadm list -n
9612023-10-18T19:04:25.364Z+++ grep oxz_ntp
9622023-10-18T19:04:25.394Z++ pfexec svcs -z oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2 -Hostate oxide/ntp
9632023-10-18T19:04:25.823Z+ [[ disabled != online ]]
9642023-10-18T19:04:25.852Z+ [[ 24 -gt 60 ]]
 
9652023-10-18T19:04:26.646Zbuskin console login: Oct 18 19:04:26 bukin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9662023-10-18T19:04:26.812Z+ sleep 1
9672023-10-18T19:04:26.839Z+ retry=25
9682023-10-18T19:04:26.875Z+++ zoneadm list -n
9692023-10-18T19:04:26.903Z+++ grep oxz_ntp
9702023-10-18T19:04:26.928Z++ pfexec svcs -z oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2 -Hostate oxide/ntp
9712023-10-18T19:04:26.953Z+ [[ disabled != online ]]
9722023-10-18T19:04:26.978Z+ [[ 25 -gt 60 ]]
9732023-10-18T19:04:27.002Z+ sleep 1
9742023-10-18T19:04:27.362Z+ retry=26
9752023-10-18T19:04:27.387Z+++ zoneadm list -n
9762023-10-18T19:04:27.411Z+++ grep oxz_ntp
9772023-10-18T19:04:27.436Z++ pfexec svcs -z oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2 -Hostate oxide/ntp
9782023-10-18T19:04:27.461Z+ [[ online != online ]]
9792023-10-18T19:04:27.485Z+ echo 'Waited for chrony: 26s'
9802023-10-18T19:04:27.509Z+ export RUST_BACKTRACE=1
9812023-10-18T19:04:27.534ZWaited for chrony: 26s
9822023-10-18T19:04:27.559Z+ RUST_BACKTRACE=1
9832023-10-18T19:04:27.583Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9842023-10-18T19:04:27.607Z+ ./tests/bootstrap
9852023-10-18T19:04:27.632ZUsing external DNS server at 10.151.1.164:53
9862023-10-18T19:04:27.657Z2023-10-18 19:04:27.643314725 UTC: attempting to log into API
9872023-10-18T19:04:42.426Z2023-10-18 19:04:42.449573678 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9882023-10-18T19:04:43.428Z2023-10-18 19:04:43.449379551 UTC: attempting to log into API
9892023-10-18T19:04:58.479Z2023-10-18 19:04:58.464961056 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9902023-10-18T19:04:59.465Z2023-10-18 19:04:59.467463329 UTC: attempting to log into API
9912023-10-18T19:05:14.520Z2023-10-18 19:05:14.528415994 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9922023-10-18T19:05:15.521Z2023-10-18 19:05:15.530356889 UTC: attempting to log into API
9932023-10-18T19:05:30.566Z2023-10-18 19:05:30.581749514 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9942023-10-18T19:05:31.567Z2023-10-18 19:05:31.583342812 UTC: attempting to log into API
 
9952023-10-18T19:05:35.971ZOct 18 19:05:35 bukin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9962023-10-18T19:05:42.187ZOct 18 19:05:42 bukin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9972023-10-18T19:05:46.599Z2023-10-18 19:05:46.614491558 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9982023-10-18T19:05:47.616Z2023-10-18 19:05:47.615308299 UTC: attempting to log into API
 
9992023-10-18T19:05:48.957ZOct 18 19:05:48 bukin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
10002023-10-18T19:05:55.954ZOct 18 19:05:55 bukin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
10012023-10-18T19:06:02.185ZOct 18 19:06:01 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
10022023-10-18T19:06:02.634Z2023-10-18 19:06:02.641206519 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10032023-10-18T19:06:03.634Z2023-10-18 19:06:03.641398582 UTC: attempting to log into API
10042023-10-18T19:06:18.675Z2023-10-18 19:06:18.676577716 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10052023-10-18T19:06:20.088Z2023-10-18 19:06:19.678098124 UTC: attempting to log into API
10062023-10-18T19:06:25.441Z2023-10-18 19:06:25.448221870 UTC: login succeeded
10072023-10-18T19:06:25.475Zwaiting for nexus to come up...
10082023-10-18T19:06:25.536Zcreating IP pool... 10.151.1.175 - 10.151.1.227
10092023-10-18T19:06:25.585Zensuring datasets are ready...
10102023-10-18T19:06:27.819Zlet's roll.
10112023-10-18T19:06:27.854Z+ rm ./tests/bootstrap
10122023-10-18T19:06:27.900Z+ for test_bin in tests/*
10132023-10-18T19:06:27.935Z+ ./tests/bootstrap-0d925b23e976736e
10142023-10-18T19:06:28.041Z
10152023-10-18T19:06:28.521Zrunning 0 tests
10162023-10-18T19:06:28.964Z
10172023-10-18T19:06:28.993Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10182023-10-18T19:06:29.097Z
10192023-10-18T19:06:29.141Z+ for test_bin in tests/*
10202023-10-18T19:06:29.589Z+ ./tests/end_to_end_tests-9d1a76d5368d1fd3
10212023-10-18T19:06:29.638Z
10222023-10-18T19:06:29.682Zrunning 2 tests
10232023-10-18T19:06:29.727Ztest no_spoof::no_spoof_here ... ok
10242023-10-18T19:07:27.926Ztest instance_launch::instance_launch has been running for over 60 seconds
10252023-10-18T19:09:25.730Ztest instance_launch::instance_launch ... ok
10262023-10-18T19:09:25.759Z
10272023-10-18T19:09:25.791Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 177.82s
10282023-10-18T19:09:25.818Z
10292023-10-18T19:09:25.844Z+ _exit_trap
10302023-10-18T19:09:25.871Z+ local status=0
10312023-10-18T19:09:25.895Z+ [[ 0 -eq 0 ]]
10322023-10-18T19:09:25.920Z+ exit 0
10332023-10-18T19:09:25.944Zprocess exited: duration 385244 ms, exit code 0
 
10342023-10-18T19:09:26.014Zfound 47 output files
10352023-10-18T19:09:26.071Zuploading: /var/svc/log/oxide-sled-agent:default.log (429596 bytes)
10362023-10-18T19:09:27.809Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10372023-10-18T19:09:27.837Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_72a63904-0ea8-4221-a9b7-739697956876/root/var/svc/log/oxide-crucible-agent:default.log (14580 bytes)
10382023-10-18T19:09:28.952Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_72a63904-0ea8-4221-a9b7-739697956876/root/var/svc/log/oxide-crucible-agent:default.log
10392023-10-18T19:09:28.977Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_72a63904-0ea8-4221-a9b7-739697956876/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10402023-10-18T19:09:30.056Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_72a63904-0ea8-4221-a9b7-739697956876/root/var/svc/log/oxide-crucible-downstairs:default.log
10412023-10-18T19:09:30.096Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_72a63904-0ea8-4221-a9b7-739697956876/root/var/svc/log/oxide-crucible-downstairs:downstairs-f905f5a0-0f35-4bb1-b380-dfc2f2e2a440.log (5536 bytes)
10422023-10-18T19:09:31.173Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_72a63904-0ea8-4221-a9b7-739697956876/root/var/svc/log/oxide-crucible-downstairs:downstairs-f905f5a0-0f35-4bb1-b380-dfc2f2e2a440.log
10432023-10-18T19:09:31.197Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d8fe84fd-86f7-4c6e-9cf7-1a54479a5af1/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10442023-10-18T19:09:32.300Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d8fe84fd-86f7-4c6e-9cf7-1a54479a5af1/root/var/svc/log/oxide-crucible-agent:default.log
10452023-10-18T19:09:32.324Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d8fe84fd-86f7-4c6e-9cf7-1a54479a5af1/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10462023-10-18T19:09:33.400Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d8fe84fd-86f7-4c6e-9cf7-1a54479a5af1/root/var/svc/log/oxide-crucible-downstairs:default.log
10472023-10-18T19:09:33.430Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_ac2dd6a3-b68d-47a0-a4b6-6f08028ee5de/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10482023-10-18T19:09:34.513Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_ac2dd6a3-b68d-47a0-a4b6-6f08028ee5de/root/var/svc/log/oxide-crucible-pantry:default.log
10492023-10-18T19:09:34.539Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_c7b3ecb5-e5a7-4172-98ba-eba512294ba6/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10502023-10-18T19:09:35.638Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_c7b3ecb5-e5a7-4172-98ba-eba512294ba6/root/var/svc/log/oxide-crucible-pantry:default.log
10512023-10-18T19:09:35.662Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_a9210067-077f-46ad-a09c-552966f0e588/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10522023-10-18T19:09:36.707Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_a9210067-077f-46ad-a09c-552966f0e588/root/var/svc/log/oxide-cockroachdb:default.log
10532023-10-18T19:09:36.732Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_200054fe-b1c1-4a86-8b45-ce98c56decf3/root/var/svc/log/oxide-crucible-agent:default.log (13684 bytes)
10542023-10-18T19:09:37.827Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_200054fe-b1c1-4a86-8b45-ce98c56decf3/root/var/svc/log/oxide-crucible-agent:default.log
10552023-10-18T19:09:37.851Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_200054fe-b1c1-4a86-8b45-ce98c56decf3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10562023-10-18T19:09:38.922Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_200054fe-b1c1-4a86-8b45-ce98c56decf3/root/var/svc/log/oxide-crucible-downstairs:default.log
10572023-10-18T19:09:38.947Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_200054fe-b1c1-4a86-8b45-ce98c56decf3/root/var/svc/log/oxide-crucible-downstairs:downstairs-a9159041-9a0e-4354-b0bc-5e01b8d20999.log (5536 bytes)
10582023-10-18T19:09:40.019Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_200054fe-b1c1-4a86-8b45-ce98c56decf3/root/var/svc/log/oxide-crucible-downstairs:downstairs-a9159041-9a0e-4354-b0bc-5e01b8d20999.log
10592023-10-18T19:09:40.043Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_642024b5-320a-4c87-ad5f-37072f053522/root/var/svc/log/oxide-crucible-pantry:default.log (2337 bytes)
10602023-10-18T19:09:41.113Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_642024b5-320a-4c87-ad5f-37072f053522/root/var/svc/log/oxide-crucible-pantry:default.log
10612023-10-18T19:09:41.138Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_cff2ea66-2497-4a5d-be30-d5e7bc0cadd1/root/var/svc/log/oxide-internal_dns:default.log (97852 bytes)
10622023-10-18T19:09:42.325Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_cff2ea66-2497-4a5d-be30-d5e7bc0cadd1/root/var/svc/log/oxide-internal_dns:default.log
10632023-10-18T19:09:42.349Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_ef0902d1-bcdd-4635-86ea-ed9aa27633e9/root/var/svc/log/oxide-internal_dns:default.log (100292 bytes)
10642023-10-18T19:09:43.545Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_ef0902d1-bcdd-4635-86ea-ed9aa27633e9/root/var/svc/log/oxide-internal_dns:default.log
10652023-10-18T19:09:43.570Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_9ef19dab-6bbf-46b2-8469-110e366e7cce/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10662023-10-18T19:09:44.644Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_9ef19dab-6bbf-46b2-8469-110e366e7cce/root/var/svc/log/oxide-cockroachdb:default.log
10672023-10-18T19:09:44.669Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f3e7b26a-ca3f-42d6-8322-7922a1c952cd/root/var/svc/log/oxide-crucible-agent:default.log (13683 bytes)
10682023-10-18T19:09:45.763Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f3e7b26a-ca3f-42d6-8322-7922a1c952cd/root/var/svc/log/oxide-crucible-agent:default.log
10692023-10-18T19:09:45.787Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f3e7b26a-ca3f-42d6-8322-7922a1c952cd/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10702023-10-18T19:09:46.858Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f3e7b26a-ca3f-42d6-8322-7922a1c952cd/root/var/svc/log/oxide-crucible-downstairs:default.log
10712023-10-18T19:09:46.882Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f3e7b26a-ca3f-42d6-8322-7922a1c952cd/root/var/svc/log/oxide-crucible-downstairs:downstairs-9b7bdffd-1876-49ca-a975-5660eae7aa76.log (3050 bytes)
10722023-10-18T19:09:47.954Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f3e7b26a-ca3f-42d6-8322-7922a1c952cd/root/var/svc/log/oxide-crucible-downstairs:downstairs-9b7bdffd-1876-49ca-a975-5660eae7aa76.log
10732023-10-18T19:09:47.994Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_56a508df-c205-471c-a3e3-f95a5237dd74/root/var/svc/log/oxide-nexus:default.log (4357229 bytes)
10742023-10-18T19:09:49.224Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_56a508df-c205-471c-a3e3-f95a5237dd74/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4357229 -> 4358444
10752023-10-18T19:09:50.296Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_56a508df-c205-471c-a3e3-f95a5237dd74/root/var/svc/log/oxide-nexus:default.log
10762023-10-18T19:09:50.321Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2/root/var/svc/log/oxide-ntp:default.log (846 bytes)
10772023-10-18T19:09:51.392Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2/root/var/svc/log/oxide-ntp:default.log
10782023-10-18T19:09:51.417Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_238630f8-3c05-4c14-b4cb-ce46f1a0e6ca/root/var/svc/log/oxide-clickhouse:default.log (663036 bytes)
10792023-10-18T19:09:52.188Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_238630f8-3c05-4c14-b4cb-ce46f1a0e6ca/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 663036 -> 741355
10802023-10-18T19:09:53.259Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_238630f8-3c05-4c14-b4cb-ce46f1a0e6ca/root/var/svc/log/oxide-clickhouse:default.log
10812023-10-18T19:09:53.283Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8d96930c-8975-454e-bd05-30810a291357/root/var/svc/log/oxide-crucible-agent:default.log (14114 bytes)
10822023-10-18T19:09:54.378Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8d96930c-8975-454e-bd05-30810a291357/root/var/svc/log/oxide-crucible-agent:default.log
10832023-10-18T19:09:54.402Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8d96930c-8975-454e-bd05-30810a291357/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10842023-10-18T19:09:55.473Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8d96930c-8975-454e-bd05-30810a291357/root/var/svc/log/oxide-crucible-downstairs:default.log
10852023-10-18T19:09:55.498Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8d96930c-8975-454e-bd05-30810a291357/root/var/svc/log/oxide-crucible-downstairs:downstairs-9aac8ce0-5f2e-4ab5-9881-17751249ba4a.log (3050 bytes)
10862023-10-18T19:09:56.578Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8d96930c-8975-454e-bd05-30810a291357/root/var/svc/log/oxide-crucible-downstairs:downstairs-9aac8ce0-5f2e-4ab5-9881-17751249ba4a.log
10872023-10-18T19:09:56.603Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_706d81ae-7d7c-45b0-b690-7ba859a77143/root/var/svc/log/oxide-external_dns:default.log (15614 bytes)
10882023-10-18T19:09:57.697Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_706d81ae-7d7c-45b0-b690-7ba859a77143/root/var/svc/log/oxide-external_dns:default.log
10892023-10-18T19:09:57.722Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_976ff27d-0007-4f8f-a0a5-a8ee6c074378/root/var/svc/log/oxide-oximeter:default.log (182225 bytes)
10902023-10-18T19:09:57.916Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_976ff27d-0007-4f8f-a0a5-a8ee6c074378/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 182225 -> 207149
10912023-10-18T19:09:58.989Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_976ff27d-0007-4f8f-a0a5-a8ee6c074378/root/var/svc/log/oxide-oximeter:default.log
10922023-10-18T19:09:59.014Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_866d6a30-2a5c-4c69-a2f3-905ce9e2bf3d/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
10932023-10-18T19:10:00.089Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_866d6a30-2a5c-4c69-a2f3-905ce9e2bf3d/root/var/svc/log/oxide-cockroachdb:default.log
10942023-10-18T19:10:00.116Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_cd7d5750-e0f0-4856-bf50-aa47af9573c4/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10952023-10-18T19:10:01.187Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_cd7d5750-e0f0-4856-bf50-aa47af9573c4/root/var/svc/log/oxide-cockroachdb:default.log
10962023-10-18T19:10:01.212Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_080ef9aa-1a90-405c-a87a-53d13296bef8/root/var/svc/log/oxide-crucible-agent:default.log (13684 bytes)
10972023-10-18T19:10:02.308Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_080ef9aa-1a90-405c-a87a-53d13296bef8/root/var/svc/log/oxide-crucible-agent:default.log
10982023-10-18T19:10:02.334Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_080ef9aa-1a90-405c-a87a-53d13296bef8/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10992023-10-18T19:10:03.413Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_080ef9aa-1a90-405c-a87a-53d13296bef8/root/var/svc/log/oxide-crucible-downstairs:default.log
11002023-10-18T19:10:03.439Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_080ef9aa-1a90-405c-a87a-53d13296bef8/root/var/svc/log/oxide-crucible-downstairs:downstairs-9d24c559-1db5-4423-b14c-e889e123caed.log (5537 bytes)
11012023-10-18T19:10:04.515Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_080ef9aa-1a90-405c-a87a-53d13296bef8/root/var/svc/log/oxide-crucible-downstairs:downstairs-9d24c559-1db5-4423-b14c-e889e123caed.log
11022023-10-18T19:10:04.544Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d8e0f1ba-1841-45fe-a471-0ea2e8339451/root/var/svc/log/oxide-crucible-agent:default.log (14149 bytes)
11032023-10-18T19:10:05.648Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d8e0f1ba-1841-45fe-a471-0ea2e8339451/root/var/svc/log/oxide-crucible-agent:default.log
11042023-10-18T19:10:05.675Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d8e0f1ba-1841-45fe-a471-0ea2e8339451/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11052023-10-18T19:10:06.748Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d8e0f1ba-1841-45fe-a471-0ea2e8339451/root/var/svc/log/oxide-crucible-downstairs:default.log
11062023-10-18T19:10:06.773Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d8e0f1ba-1841-45fe-a471-0ea2e8339451/root/var/svc/log/oxide-crucible-downstairs:downstairs-a579d3a0-ce83-4601-8554-cf4e42280ef4.log (3051 bytes)
11072023-10-18T19:10:07.845Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d8e0f1ba-1841-45fe-a471-0ea2e8339451/root/var/svc/log/oxide-crucible-downstairs:downstairs-a579d3a0-ce83-4601-8554-cf4e42280ef4.log
11082023-10-18T19:10:07.869Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_654051ea-06c0-40b7-92ad-b8951f4eeea0/root/var/svc/log/oxide-internal_dns:default.log (104765 bytes)
11092023-10-18T19:10:08.006Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_654051ea-06c0-40b7-92ad-b8951f4eeea0/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 104765 -> 108516
11102023-10-18T19:10:09.078Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_654051ea-06c0-40b7-92ad-b8951f4eeea0/root/var/svc/log/oxide-internal_dns:default.log
11112023-10-18T19:10:09.102Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_d1dabd3b-5f6d-400a-80a3-c92cf32b581c/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
11122023-10-18T19:10:10.193Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_d1dabd3b-5f6d-400a-80a3-c92cf32b581c/root/var/svc/log/oxide-cockroachdb:default.log
11132023-10-18T19:10:10.219Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6d88225e-df6d-43d0-bcb0-db81945927d1/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
11142023-10-18T19:10:11.290Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6d88225e-df6d-43d0-bcb0-db81945927d1/root/var/svc/log/oxide-crucible-agent:default.log
11152023-10-18T19:10:11.314Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6d88225e-df6d-43d0-bcb0-db81945927d1/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11162023-10-18T19:10:12.386Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6d88225e-df6d-43d0-bcb0-db81945927d1/root/var/svc/log/oxide-crucible-downstairs:default.log
11172023-10-18T19:10:12.416Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_d9a282ee-1c6f-47b6-b344-b8c012aa2947/root/var/svc/log/oxide-external_dns:default.log (15611 bytes)
11182023-10-18T19:10:12.480Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_d9a282ee-1c6f-47b6-b344-b8c012aa2947/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15611 -> 19389
11192023-10-18T19:10:13.552Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_d9a282ee-1c6f-47b6-b344-b8c012aa2947/root/var/svc/log/oxide-external_dns:default.log
11202023-10-18T19:10:13.578Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_87ece1d3-faf7-41f1-8668-61ba1adc40f2/root/var/svc/log/oxide-nexus:default.log (795340 bytes)
11212023-10-18T19:10:13.842Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_87ece1d3-faf7-41f1-8668-61ba1adc40f2/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 795340 -> 856561
11222023-10-18T19:10:14.915Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_87ece1d3-faf7-41f1-8668-61ba1adc40f2/root/var/svc/log/oxide-nexus:default.log
11232023-10-18T19:10:14.944Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_19e63e7f-01c5-4e32-9821-bc1cbefeb7dc/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
11242023-10-18T19:10:16.014Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_19e63e7f-01c5-4e32-9821-bc1cbefeb7dc/root/var/svc/log/oxide-crucible-agent:default.log
11252023-10-18T19:10:16.038Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_19e63e7f-01c5-4e32-9821-bc1cbefeb7dc/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11262023-10-18T19:10:17.110Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_19e63e7f-01c5-4e32-9821-bc1cbefeb7dc/root/var/svc/log/oxide-crucible-downstairs:default.log
11272023-10-18T19:10:17.134Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_84a0d292-94d7-4aa1-85a8-d3ee31a38c28/root/var/svc/log/oxide-nexus:default.log (479546 bytes)
11282023-10-18T19:10:17.907Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_84a0d292-94d7-4aa1-85a8-d3ee31a38c28/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 479546 -> 540760
11292023-10-18T19:10:18.978Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_84a0d292-94d7-4aa1-85a8-d3ee31a38c28/root/var/svc/log/oxide-nexus:default.log
11302023-10-18T19:10:19.003Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2/root/var/log/chrony/measurements.log (6165 bytes)
11312023-10-18T19:10:19.048Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2/root/var/log/chrony/measurements.log" changed size mid upload: 6165 -> 6987
11322023-10-18T19:10:20.123Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2/root/var/log/chrony/measurements.log
11332023-10-18T19:10:20.150Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2/root/var/log/chrony/statistics.log (3276 bytes)
11342023-10-18T19:10:20.175Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2/root/var/log/chrony/statistics.log" changed size mid upload: 3276 -> 3978
11352023-10-18T19:10:21.245Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2/root/var/log/chrony/statistics.log
11362023-10-18T19:10:21.269Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2/root/var/log/chrony/tracking.log (1321 bytes)
11372023-10-18T19:10:21.294Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2/root/var/log/chrony/tracking.log" changed size mid upload: 1321 -> 1453
11382023-10-18T19:10:22.365Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_69fc16fb-e742-4133-b0e7-838eb03387d2/root/var/log/chrony/tracking.log