01HDFGCYQ3CGM6HYH41GV40X9H: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HDFGDHVWMGAT3B675ZCGQCMR

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-24T01:03:42.632Zjob dependencies complete; ready to run (waiting for 51 m 0 s)
22023-10-24T01:04:32.165Zjob assigned to worker 01HDFKB4YSPP633W5EHN6W5PFB (queued for 49 s)
32023-10-24T01:04:32.400Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-24T01:04:32.447Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-10-24T01:04:32.483Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-10-24T01:04:32.516Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-24T01:04:32.550Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-10-24T01:04:32.583ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-24T01:04:32.617Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-24T01:04:32.650ZOxide Helios Version helios-2.0.22241 64-bit
112023-10-24T01:04:32.684ZConfiguring devices.
122023-10-24T01:04:32.718Z##################################################
132023-10-24T01:04:32.751Z#### ############# ##
142023-10-24T01:04:32.787Z### ### ############ ##
152023-10-24T01:04:32.835Z## ### # ## ### ## Oxide Computer Company ##
162023-10-24T01:04:32.873Z## ## ## ### # ### ##
172023-10-24T01:04:32.908Z## # ### #### #### This Station Under ##
182023-10-24T01:04:32.941Z### ### #### # ### Computer Control ##
192023-10-24T01:04:32.975Z#### #### ### ## ##
202023-10-24T01:04:33.008Z##################################################
212023-10-24T01:04:33.064ZHostname: buskin
222023-10-24T01:04:33.097ZLoading smf(7) serice descriptions:
232023-10-24T01:04:33.388Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622.gz
242023-10-24T01:04:34.820Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622.gz
252023-10-24T01:04:34.845Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-24T01:04:52.034Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-24T01:04:52.059Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280.gz
282023-10-24T01:05:12.643Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280.gz
292023-10-24T01:05:12.669Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-24T01:05:14.349Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-24T01:05:14.376Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-24T01:05:17.025Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-24T01:05:17.050Zdownloading input: /input/package/work/version.txt
342023-10-24T01:05:17.075Zdownloaded input: /input/package/work/version.txt
352023-10-24T01:05:17.099Zdownloading input: /input/package/work/package.tar.gz
362023-10-24T01:05:39.703Zdownloaded input: /input/package/work/package.tar.gz
372023-10-24T01:05:39.738Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-24T01:06:11.543Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-24T01:06:11.569Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-24T01:06:19.642Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-24T01:06:19.670Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-24T01:06:59.773Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-24T01:06:59.798Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-24T01:07:36.704Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-24T01:07:36.734Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-24T01:08:01.158Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-24T01:08:01.183Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-24T01:08:06.183Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-24T01:08:06.207Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-24T01:08:14.865Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-24T01:08:15.301Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-24T01:08:20.026Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-24T01:08:20.050Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-24T01:08:24.692Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-24T01:08:24.719Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-24T01:08:24.777Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-24T01:08:24.806Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-24T01:08:29.378Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-24T01:08:29.403Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-24T01:08:33.466Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-24T01:08:33.492Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-24T01:08:48.040Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-24T01:08:48.064Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-24T01:09:03.179Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-24T01:09:03.206Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-24T01:09:03.269Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-24T01:09:03.294Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-24T01:09:07.606Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-24T01:09:07.630Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-24T01:09:20.157Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-24T01:09:20.186Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-24T01:10:13.189Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-24T01:10:13.213Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-24T01:10:49.646Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-24T01:10:49.671Zstarting task 0: "setup"
762023-10-24T01:10:49.704Z++ uname -s
772023-10-24T01:10:49.729Z+ kern=SunOS
782023-10-24T01:10:49.753Z+ case "$kern" in
792023-10-24T01:10:49.778Z+ groupadd -g 12345 build
802023-10-24T01:10:49.803Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-24T01:10:51.714Z+ zfs create -o mountpoint=/work rpool/work
822023-10-24T01:10:51.740Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-24T01:10:51.773Z+ home_fs=zfs
842023-10-24T01:10:51.797Z+ [[ zfs == autofs ]]
852023-10-24T01:10:51.822Z+ mkdir -p /home/build
862023-10-24T01:10:51.847Z+ chown build:build /home/build /work
872023-10-24T01:10:53.744Z+ chmod 0700 /home/build /work
882023-10-24T01:10:53.769Zprocess exited: duration 4051 ms, exit code 0
 
892023-10-24T01:10:53.820Zstarting task 1: "authentication"
902023-10-24T01:10:54.245Zprocess exited: duration 111 ms, exit code 0
 
912023-10-24T01:10:54.351Zstarting task 2: "build"
922023-10-24T01:10:54.408Z+ trap _exit_trap EXIT
932023-10-24T01:10:54.460Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-24T01:10:54.655Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-24T01:10:54.734Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-24T01:10:54.810Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-24T01:10:54.872Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-24T01:10:54.940Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-24T01:10:54.971Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-24T01:10:55.026Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-24T01:10:55.102Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-24T01:10:55.155Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-24T01:10:55.209Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-24T01:10:55.248Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-24T01:10:55.281Z+ [[ -d /opt/oxide ]]
1062023-10-24T01:10:55.326Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-24T01:10:55.374Z+ pfexec mkdir /opt/oxide
1082023-10-24T01:10:55.404Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-24T01:10:55.450Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-24T01:10:55.490Z+ cd /opt/oxide-underneath
1112023-10-24T01:10:55.519Z+ pfexec tar ceEp@/f - .
1122023-10-24T01:10:55.548Z+ cd /opt/oxide
1132023-10-24T01:10:55.603Z+ pfexec tar xveEp@/f -
1142023-10-24T01:10:56.047Zx ., 0 bytes, 0 tape blocks
1152023-10-24T01:10:56.078Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-24T01:10:56.108Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-24T01:10:56.141Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-10-24T01:10:56.168Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-24T01:10:56.250Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-24T01:10:56.277Z+ pfexec mkdir /opt/oxide/work
1212023-10-24T01:10:56.307Z+ pfexec chown build:build /opt/oxide/work
1222023-10-24T01:10:56.340Z+ cd /opt/oxide/work
1232023-10-24T01:10:56.377Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-24T01:10:56.410Ztar: blocksize = 10
1252023-10-24T01:10:56.440ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1262023-10-24T01:10:56.468Zx out/mg-ddm-gz.tar, 63638528 bytes, 124294 tape blocks
1272023-10-24T01:10:56.497Zx out/omicron-sled-agent.tar, 115110912 bytes, 224826 tape blocks
1282023-10-24T01:10:56.527Zx out/target/test, 75 bytes, 1 tape blocks
1292023-10-24T01:10:56.557Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1302023-10-24T01:10:56.587Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1312023-10-24T01:10:56.621Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1322023-10-24T01:10:56.653Zx target/release/omicron-package, 38195920 bytes, 74602 tape blocks
1332023-10-24T01:10:56.681Zx tools/create_virtual_hardware.sh, 2986 bytes, 6 tape blocks
1342023-10-24T01:10:56.718Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1352023-10-24T01:10:56.744Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1362023-10-24T01:10:56.770Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1372023-10-24T01:10:56.795Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1382023-10-24T01:10:56.820Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1392023-10-24T01:10:56.846Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1402023-10-24T01:10:56.873Z
1412023-10-24T01:10:56.899Zreal 1.559353056
1422023-10-24T01:10:56.925Zuser 1.404092378
1432023-10-24T01:10:56.954Zsys 0.302209631
1442023-10-24T01:10:56.980Ztrap 0.000053062
1452023-10-24T01:10:57.005Ztflt 0.000000000
1462023-10-24T01:10:57.057Zdflt 0.000000000
1472023-10-24T01:10:57.085Zkflt 0.000000000
1482023-10-24T01:10:57.116Zlock 0.000000000
1492023-10-24T01:10:57.554Zslp 1.372286883
1502023-10-24T01:10:57.579Zlat 0.037847162
1512023-10-24T01:10:57.610Zstop 0.000124122
1522023-10-24T01:10:58.053Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1532023-10-24T01:10:58.090Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1542023-10-24T01:10:58.133Z+ mkdir tests
1552023-10-24T01:10:58.163Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1562023-10-24T01:10:58.198Z+ ptime -m gunzip
1572023-10-24T01:10:58.232Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622
1582023-10-24T01:10:58.269Z
1592023-10-24T01:10:58.309Zreal 0.140218376
1602023-10-24T01:10:58.370Zuser 0.129443147
1612023-10-24T01:10:58.420Zsys 0.010529854
1622023-10-24T01:10:58.445Ztrap 0.000011920
1632023-10-24T01:10:58.475Ztflt 0.000000000
1642023-10-24T01:10:58.502Zdflt 0.000000000
1652023-10-24T01:10:58.529Zkflt 0.000000000
1662023-10-24T01:10:58.554Zlock 0.000000000
1672023-10-24T01:10:58.578Zslp 0.000018860
1682023-10-24T01:10:58.602Zlat 0.000024050
1692023-10-24T01:10:58.627Zstop 0.000132612
1702023-10-24T01:10:58.651Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622
1712023-10-24T01:10:58.676Z+ chmod a+x tests/bootstrap-49e31e75b930b622
1722023-10-24T01:10:58.706Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1732023-10-24T01:10:59.153Z+ ptime -m gunzip
1742023-10-24T01:10:59.187Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1752023-10-24T01:10:59.805Z
1762023-10-24T01:10:59.832Zreal 2.243864173
1772023-10-24T01:10:59.856Zuser 2.102499064
1782023-10-24T01:10:59.882Zsys 0.140963702
1792023-10-24T01:10:59.906Ztrap 0.000170441
1802023-10-24T01:10:59.930Ztflt 0.000000000
1812023-10-24T01:10:59.955Zdflt 0.000000000
1822023-10-24T01:10:59.981Zkflt 0.000000000
1832023-10-24T01:11:00.017Zlock 0.000000000
1842023-10-24T01:11:00.048Zslp 0.000012080
1852023-10-24T01:11:00.086Zlat 0.000024480
1862023-10-24T01:11:00.116Zstop 0.000131653
1872023-10-24T01:11:00.141Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1882023-10-24T01:11:00.165Z+ chmod a+x tests/bootstrap
1892023-10-24T01:11:00.190Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1902023-10-24T01:11:00.214Z+ ptime -m gunzip
1912023-10-24T01:11:00.242Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280
1922023-10-24T01:11:02.354Z
1932023-10-24T01:11:02.379Zreal 2.537800511
1942023-10-24T01:11:02.404Zuser 2.394133550
1952023-10-24T01:11:02.853Zsys 0.143279254
1962023-10-24T01:11:02.878Ztrap 0.000162053
1972023-10-24T01:11:02.905Ztflt 0.000000000
1982023-10-24T01:11:02.937Zdflt 0.000000000
1992023-10-24T01:11:02.967Zkflt 0.000000000
2002023-10-24T01:11:03.004Zlock 0.000000000
2012023-10-24T01:11:03.035Zslp 0.000021870
2022023-10-24T01:11:03.065Zlat 0.000025389
2032023-10-24T01:11:03.096Zstop 0.000118012
2042023-10-24T01:11:03.144Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280
2052023-10-24T01:11:03.176Z+ chmod a+x tests/end_to_end_tests-a91cf9af3457d280
2062023-10-24T01:11:03.203Z+ bmat address ls
2072023-10-24T01:11:03.230ZNAME CIDR FIRST LAST COUNT
2082023-10-24T01:11:03.256Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2092023-10-24T01:11:03.292Z++ bmat address ls -f extra -Ho first,last,count
2102023-10-24T01:11:03.325Z+ set -- 10.151.1.164 10.151.1.227 64
2112023-10-24T01:11:03.354Z+ EXTRA_IP_START=10.151.1.164
2122023-10-24T01:11:03.385Z+ EXTRA_IP_END=10.151.1.227
2132023-10-24T01:11:03.561Z+ EXTRA_IP_COUNT=64
2142023-10-24T01:11:03.589Z+ (( EXTRA_IP_COUNT >= 32 ))
2152023-10-24T01:11:03.613Z+ EXTRA_IP_BASE=10.151.1
2162023-10-24T01:11:03.640Z+ EXTRA_IP_FOCTET=164
2172023-10-24T01:11:03.665Z+ EXTRA_IP_LOCTET=227
2182023-10-24T01:11:03.693Z+ SERVICE_IP_POOL_START=10.151.1.164
2192023-10-24T01:11:03.719Z+ SERVICE_IP_POOL_END=10.151.1.173
2202023-10-24T01:11:03.753Z+ DNS_IP1=10.151.1.164
2212023-10-24T01:11:04.216Z+ DNS_IP2=10.151.1.165
2222023-10-24T01:11:04.252Z+ UPLINK_IP=10.151.1.174
2232023-10-24T01:11:04.285Z+ IPPOOL_START=10.151.1.175
2242023-10-24T01:11:04.318Z+ IPPOOL_END=10.151.1.227
2252023-10-24T01:11:04.354Z++ ipadm show-addr -po type,addr
2262023-10-24T01:11:04.384Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2272023-10-24T01:11:04.419Z+ GATEWAY_IP=10.151.1.100
2282023-10-24T01:11:04.453Z+ [[ -n 10.151.1.100 ]]
2292023-10-24T01:11:04.486Z+ ping -s 10.151.1.100 56 1
2302023-10-24T01:11:04.524ZPING 10.151.1.100: 56 data bytes
2312023-10-24T01:11:04.558Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.012 ms
2322023-10-24T01:11:04.610Z
2332023-10-24T01:11:04.646Z----10.151.1.100 PING Statistics----
2342023-10-24T01:11:04.675Z1 packets transmitted, 1 packets received, 0% packet loss
2352023-10-24T01:11:04.715Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2362023-10-24T01:11:04.753Z++ arp -an
2372023-10-24T01:11:05.216Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2382023-10-24T01:11:05.245Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2392023-10-24T01:11:05.281Z+ routeadm -e ipv4-forwarding -u
2402023-10-24T01:11:05.312Z+ PXA_START=10.151.1.164
2412023-10-24T01:11:05.339Z+ PXA_END=10.151.1.227
2422023-10-24T01:11:05.363Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2432023-10-24T01:11:05.388Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2442023-10-24T01:11:05.413Z+ ZPOOL_VDEV_DIR=/scratch
2452023-10-24T01:11:05.439Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2462023-10-24T01:11:05.873Z+++ dirname ./tools/create_virtual_hardware.sh
2472023-10-24T01:11:05.908Z++ cd ./tools
2482023-10-24T01:11:06.347Z++ pwd
2492023-10-24T01:11:06.380Z+ SOURCE_DIR=/opt/oxide/work/tools
2502023-10-24T01:11:06.412Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2512023-10-24T01:11:06.444Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-10-24T01:11:06.478Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-10-24T01:11:06.511Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-10-24T01:11:06.975Z++ dladm show-phys -p -o LINK
2552023-10-24T01:11:07.000Z++ head -1
2562023-10-24T01:11:07.024Z+ PHYSICAL_LINK=igb0
2572023-10-24T01:11:07.049Z+ echo 'Using igb0 as physical link'
2582023-10-24T01:11:07.081ZUsing igb0 as physical link
2592023-10-24T01:11:07.110Z+ ensure_run_as_root
2602023-10-24T01:11:07.136Z++ id -u
2612023-10-24T01:11:07.160ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2622023-10-24T01:11:07.186Z+ [[ 0 -ne 0 ]]
2632023-10-24T01:11:07.622ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2642023-10-24T01:11:07.655Z+ ensure_zpools
2652023-10-24T01:11:07.680ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2662023-10-24T01:11:07.704Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2672023-10-24T01:11:07.750ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2682023-10-24T01:11:07.775Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2692023-10-24T01:11:07.799ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2702023-10-24T01:11:07.824Z+ readarray -t ZPOOLS
2712023-10-24T01:11:07.848ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2722023-10-24T01:11:07.873Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2732023-10-24T01:11:07.903ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2742023-10-24T01:11:08.342Z++ sed 's/[ ",]//g'
2752023-10-24T01:11:08.369ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2762023-10-24T01:11:08.862Z+ for ZPOOL in "${ZPOOLS[@]}"
2772023-10-24T01:11:08.891ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2782023-10-24T01:11:09.337Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2792023-10-24T01:11:09.379ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2802023-10-24T01:11:09.413Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2812023-10-24T01:11:09.452ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2822023-10-24T01:11:09.489Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2832023-10-24T01:11:09.519ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2842023-10-24T01:11:09.546Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2852023-10-24T01:11:09.571ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2862023-10-24T01:11:09.596Z0+0 records in
2872023-10-24T01:11:09.621ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2882023-10-24T01:11:09.646Z0+0 records out
2892023-10-24T01:11:09.672ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2902023-10-24T01:11:09.699Z0 bytes (0 iB) transferred in 0.000007 secs (0 iB/sec)
2912023-10-24T01:11:09.733ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
2922023-10-24T01:11:09.762Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2932023-10-24T01:11:09.823ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
2942023-10-24T01:11:10.287Z+ set +x
2952023-10-24T01:11:10.313ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
2962023-10-24T01:11:10.339Z++ zpool list -o name
2972023-10-24T01:11:10.776ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
2982023-10-24T01:11:10.803Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2992023-10-24T01:11:10.863ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3002023-10-24T01:11:10.887Z+ [[ -z '' ]]
3012023-10-24T01:11:10.912ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3022023-10-24T01:11:10.936Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
3032023-10-24T01:11:10.961ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3042023-10-24T01:11:10.986Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
3052023-10-24T01:11:11.011ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3062023-10-24T01:11:11.041Z+ set +x
3072023-10-24T01:11:11.083ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3082023-10-24T01:11:11.124Z+ for ZPOOL in "${ZPOOLS[@]}"
3092023-10-24T01:11:11.158ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3102023-10-24T01:11:11.200Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3112023-10-24T01:11:11.232ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3122023-10-24T01:11:11.267Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3132023-10-24T01:11:11.294ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3142023-10-24T01:11:11.318Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3152023-10-24T01:11:11.343ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3162023-10-24T01:11:11.368Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3172023-10-24T01:11:11.392ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3182023-10-24T01:11:11.416Z0+0 records in
3192023-10-24T01:11:11.440ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3202023-10-24T01:11:11.465Z0+0 records out
3212023-10-24T01:11:11.489ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3222023-10-24T01:11:11.513Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3232023-10-24T01:11:11.548ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3242023-10-24T01:11:11.578Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3252023-10-24T01:11:11.609ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3262023-10-24T01:11:11.636Z+ set +x
3272023-10-24T01:11:11.665ZSimnet net0/sc0_0 exists
3282023-10-24T01:11:11.692Z++ zpool list -o name
3292023-10-24T01:11:11.720ZSimnet net1/sc1_0 exists
3302023-10-24T01:11:11.751Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3312023-10-24T01:11:11.784ZVnic sc0_1 exists
3322023-10-24T01:11:11.826Z+ [[ -z '' ]]
3332023-10-24T01:11:11.858Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3342023-10-24T01:11:11.961Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3352023-10-24T01:11:11.994Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3362023-10-24T01:11:12.024Z+ set +x
3372023-10-24T01:11:12.054Z+ for ZPOOL in "${ZPOOLS[@]}"
3382023-10-24T01:11:12.081Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3392023-10-24T01:11:12.114Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3402023-10-24T01:11:12.146Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3412023-10-24T01:11:12.177Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3422023-10-24T01:11:12.203Z0+0 records in
3432023-10-24T01:11:12.227Z0+0 records out
3442023-10-24T01:11:12.255Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3452023-10-24T01:11:12.286Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3462023-10-24T01:11:12.319Z+ set +x
3472023-10-24T01:11:12.351Z++ zpool list -o name
3482023-10-24T01:11:12.433Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3492023-10-24T01:11:12.465Z+ [[ -z '' ]]
3502023-10-24T01:11:12.496Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3512023-10-24T01:11:12.522Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3522023-10-24T01:11:12.548Z+ set +x
3532023-10-24T01:11:12.582Z+ for ZPOOL in "${ZPOOLS[@]}"
3542023-10-24T01:11:13.047Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552023-10-24T01:11:13.077Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562023-10-24T01:11:13.107Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572023-10-24T01:11:13.136Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582023-10-24T01:11:13.164Z0+0 records in
3592023-10-24T01:11:13.201Z0+0 records out
3602023-10-24T01:11:13.235Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3612023-10-24T01:11:13.265Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622023-10-24T01:11:13.292Z+ set +x
3632023-10-24T01:11:13.317Z++ zpool list -o name
3642023-10-24T01:11:13.346Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3652023-10-24T01:11:13.379Z+ [[ -z '' ]]
3662023-10-24T01:11:13.409Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3672023-10-24T01:11:13.434Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3682023-10-24T01:11:13.680Z+ set +x
3692023-10-24T01:11:13.709Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-10-24T01:11:13.734Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3712023-10-24T01:11:13.763Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3722023-10-24T01:11:13.789Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3732023-10-24T01:11:13.819Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3742023-10-24T01:11:13.846Z0+0 records in
3752023-10-24T01:11:13.875Z0+0 records out
3762023-10-24T01:11:13.909Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3772023-10-24T01:11:13.942Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3782023-10-24T01:11:13.975Z+ set +x
3792023-10-24T01:11:14.014Z++ zpool list -o name
3802023-10-24T01:11:14.105Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3812023-10-24T01:11:14.133Z+ [[ -z '' ]]
3822023-10-24T01:11:14.169Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3832023-10-24T01:11:14.200Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3842023-10-24T01:11:14.229Z+ set +x
3852023-10-24T01:11:14.254Z+ for ZPOOL in "${ZPOOLS[@]}"
3862023-10-24T01:11:14.279Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-10-24T01:11:14.304Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-10-24T01:11:14.329Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-10-24T01:11:14.362Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-10-24T01:11:14.394Z0+0 records in
3912023-10-24T01:11:14.431Z0+0 records out
3922023-10-24T01:11:14.468Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3932023-10-24T01:11:14.514Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942023-10-24T01:11:14.564Z+ set +x
3952023-10-24T01:11:14.611Z++ zpool list -o name
3962023-10-24T01:11:14.646Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972023-10-24T01:11:14.682Z+ [[ -z '' ]]
3982023-10-24T01:11:14.708Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992023-10-24T01:11:14.734Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002023-10-24T01:11:14.759Z+ set +x
4012023-10-24T01:11:14.785Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-10-24T01:11:14.810Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4032023-10-24T01:11:14.840Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-10-24T01:11:14.865Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4052023-10-24T01:11:14.890Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4062023-10-24T01:11:14.917Z0+0 records in
4072023-10-24T01:11:14.945Z0+0 records out
4082023-10-24T01:11:14.971Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4092023-10-24T01:11:14.996Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4102023-10-24T01:11:15.021Z+ set +x
4112023-10-24T01:11:15.048Z++ zpool list -o name
4122023-10-24T01:11:15.074Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4132023-10-24T01:11:15.099Z+ [[ -z '' ]]
4142023-10-24T01:11:15.183Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4152023-10-24T01:11:15.625Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4162023-10-24T01:11:15.651Z+ set +x
4172023-10-24T01:11:15.676Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-10-24T01:11:15.701Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4192023-10-24T01:11:15.726Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4202023-10-24T01:11:15.758Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4212023-10-24T01:11:15.798Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4222023-10-24T01:11:15.831Z0+0 records in
4232023-10-24T01:11:15.859Z0+0 records out
4242023-10-24T01:11:15.891Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4252023-10-24T01:11:15.935Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4262023-10-24T01:11:15.973Z+ set +x
4272023-10-24T01:11:16.005Z++ zpool list -o name
4282023-10-24T01:11:16.040Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4292023-10-24T01:11:16.072Z+ [[ -z '' ]]
4302023-10-24T01:11:16.099Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4312023-10-24T01:11:16.124Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4322023-10-24T01:11:16.154Z+ set +x
4332023-10-24T01:11:16.228Z+ for ZPOOL in "${ZPOOLS[@]}"
4342023-10-24T01:11:16.271Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4352023-10-24T01:11:16.295Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-10-24T01:11:16.320Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4372023-10-24T01:11:16.348Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4382023-10-24T01:11:16.374Z0+0 records in
4392023-10-24T01:11:16.399Z0+0 records out
4402023-10-24T01:11:16.427Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4412023-10-24T01:11:16.451Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4422023-10-24T01:11:16.476Z+ set +x
4432023-10-24T01:11:16.501Z++ zpool list -o name
4442023-10-24T01:11:16.526Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4452023-10-24T01:11:16.550Z+ [[ -z '' ]]
4462023-10-24T01:11:16.575Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4472023-10-24T01:11:16.600Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4482023-10-24T01:11:17.045Z+ set +x
4492023-10-24T01:11:17.074Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4502023-10-24T01:11:17.522Z+ readarray -t ZPOOLS
4512023-10-24T01:11:17.562Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4522023-10-24T01:11:17.588Z++ sed 's/[ ",]//g'
4532023-10-24T01:11:17.613Z+ for ZPOOL in "${ZPOOLS[@]}"
4542023-10-24T01:11:17.639Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4552023-10-24T01:11:17.667Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4562023-10-24T01:11:18.133Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4572023-10-24T01:11:18.158Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4582023-10-24T01:11:18.188Z0+0 records in
4592023-10-24T01:11:18.217Z0+0 records out
4602023-10-24T01:11:18.246Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4612023-10-24T01:11:18.279Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4622023-10-24T01:11:18.308Z+ set +x
4632023-10-24T01:11:18.370Z++ zpool list -o name
4642023-10-24T01:11:18.494Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4652023-10-24T01:11:18.551Z+ [[ -z '' ]]
4662023-10-24T01:11:18.578Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4672023-10-24T01:11:18.603Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4682023-10-24T01:11:18.628Z+ set +x
4692023-10-24T01:11:18.653Z+ for ZPOOL in "${ZPOOLS[@]}"
4702023-10-24T01:11:18.678Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4712023-10-24T01:11:18.706Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4722023-10-24T01:11:18.732Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4732023-10-24T01:11:18.758Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4742023-10-24T01:11:18.788Z0+0 records in
4752023-10-24T01:11:18.818Z0+0 records out
4762023-10-24T01:11:18.847Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4772023-10-24T01:11:18.872Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4782023-10-24T01:11:18.897Z+ set +x
4792023-10-24T01:11:18.922Z++ zpool list -o name
4802023-10-24T01:11:18.946Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4812023-10-24T01:11:18.971Z+ [[ -z '' ]]
4822023-10-24T01:11:18.996Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4832023-10-24T01:11:19.024Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4842023-10-24T01:11:19.049Z+ set +x
4852023-10-24T01:11:19.074Z+ ensure_simulated_links igb0
4862023-10-24T01:11:19.102Z+ local PHYSICAL_LINK=igb0
4872023-10-24T01:11:19.131Z+ INDICES=("0" "1")
4882023-10-24T01:11:19.158Z+ for I in "${INDICES[@]}"
4892023-10-24T01:11:19.187Z++ get_simnet_name_if_exists net0
4902023-10-24T01:11:19.214Z++ dladm show-simnet -p -o LINK net0
4912023-10-24T01:11:19.244Z++ echo ''
4922023-10-24T01:11:19.278Z+ [[ -z '' ]]
4932023-10-24T01:11:19.321Z+ dladm create-simnet -t net0
4942023-10-24T01:11:19.361Z+ dladm create-simnet -t sc0_0
4952023-10-24T01:11:19.401Z+ dladm modify-simnet -t -p net0 sc0_0
4962023-10-24T01:11:19.427Z+ dladm set-linkprop -p mtu=9000 sc0_0
4972023-10-24T01:11:19.452Z+ success 'Simnet net0/sc0_0 exists'
4982023-10-24T01:11:19.495Z+ set +x
4992023-10-24T01:11:19.520Z+ for I in "${INDICES[@]}"
5002023-10-24T01:11:19.552Z++ get_simnet_name_if_exists net1
5012023-10-24T01:11:19.577Z++ dladm show-simnet -p -o LINK net1
5022023-10-24T01:11:19.610Z++ echo ''
5032023-10-24T01:11:19.636Z+ [[ -z '' ]]
5042023-10-24T01:11:19.662Z+ dladm create-simnet -t net1
5052023-10-24T01:11:19.689Z+ dladm create-simnet -t sc1_0
5062023-10-24T01:11:19.729Z+ dladm modify-simnet -t -p net1 sc1_0
5072023-10-24T01:11:19.759Z+ dladm set-linkprop -p mtu=9000 sc1_0
5082023-10-24T01:11:19.785Z+ success 'Simnet net1/sc1_0 exists'
5092023-10-24T01:11:19.810Z+ set +x
5102023-10-24T01:11:19.834Z++ get_vnic_name_if_exists sc0_1
5112023-10-24T01:11:19.859Z++ dladm show-vnic -p -o LINK sc0_1
5122023-10-24T01:11:19.884Z++ echo ''
5132023-10-24T01:11:19.909Z+ [[ -z '' ]]
5142023-10-24T01:11:19.935Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5152023-10-24T01:11:19.963Z+ [[ -v PROMISC_FILT_OFF ]]
5162023-10-24T01:11:19.989Z+ success 'Vnic sc0_1 exists'
5172023-10-24T01:11:20.017Z+ set +x
5182023-10-24T01:11:20.043Z+ warn_if_no_proxy_arp
5192023-10-24T01:11:20.068Z+ [[ -v PXA_START ]]
5202023-10-24T01:11:20.103Z+ [[ -v PXA_END ]]
5212023-10-24T01:11:20.130Z+ ensure_softnpu_zone
5222023-10-24T01:11:20.562Z+ zoneadm list
5232023-10-24T01:11:20.594Z+ grep -q sidecar_softnpu
5242023-10-24T01:11:20.624Z+ [[ -f out/npuzone/npuzone ]]
5252023-10-24T01:11:20.657Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch omicron-tracking
5262023-10-24T01:11:21.500Zdownloading softnpu ...done
5272023-10-24T01:11:21.626Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/libsidecar_lite.so.sha256.txt
5282023-10-24T01:11:27.353Zdownloading asic_program.so ...done
5292023-10-24T01:11:27.521Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/scadm.sha256.txt
5302023-10-24T01:11:34.206Zdownloading scadm ...done
5312023-10-24T01:11:34.231Zcreating port tfportrear0_0 for sc0_0
5322023-10-24T01:11:34.256Zcreating port tfportqsfp0_0 for sc0_1
5332023-10-24T01:11:34.281Zsoftnpu zone setup
5342023-10-24T01:11:34.306Zconfigure zone
5352023-10-24T01:11:34.739Zinstall zone
5362023-10-24T01:11:35.293Zboot zone
5372023-10-24T01:11:38.420Z[sidecar_softnpu] svcs milestone/network
5382023-10-24T01:11:38.444ZSTATE STIME FMRI
5392023-10-24T01:11:38.469Zonline 1:11:37 svc:/milestone/network:default
5402023-10-24T01:11:38.493Z[sidecar_softnpu] /softnpu/npu start
5412023-10-24T01:11:38.518Z
5422023-10-24T01:11:38.547Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5432023-10-24T01:11:38.571Z+ GATEWAY_IP=10.151.1.100
5442023-10-24T01:11:38.596Z+ echo 'Using 10.151.1.100 as gateway ip'
5452023-10-24T01:11:38.629ZUsing 10.151.1.100 as gateway ip
5462023-10-24T01:11:38.655Z+ [[ ! -v GATEWAY_MAC ]]
5472023-10-24T01:11:38.680Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5482023-10-24T01:11:38.704Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5492023-10-24T01:11:38.728Z+++ echo 18:c0:4d:0d:9f:b2
5502023-10-24T01:11:38.752Z+ COUNT=1
5512023-10-24T01:11:38.777Z+ [[ 1 -ne 1 ]]
5522023-10-24T01:11:38.802Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5532023-10-24T01:11:38.826ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5542023-10-24T01:11:38.851Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5552023-10-24T01:11:38.877Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-10-24T01:11:38.901Z+ PXA_MAC=a8:e1:de:01:70:1d
5572023-10-24T01:11:38.926Z+ [[ -v PXA_START ]]
5582023-10-24T01:11:38.950Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5592023-10-24T01:11:38.975Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-10-24T01:11:39.004Z+ z_scadm dump-state
5612023-10-24T01:11:39.029Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5622023-10-24T01:11:39.054Zlocal v6:
5632023-10-24T01:11:39.080Zlocal v4:
5642023-10-24T01:11:39.106Zrouter v6:
5652023-10-24T01:11:39.131Zrouter v4:
5662023-10-24T01:11:39.155Zresolver v4:
5672023-10-24T01:11:39.181Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5682023-10-24T01:11:39.206Zresolver v6:
5692023-10-24T01:11:39.231Znat v4:
5702023-10-24T01:11:39.257Znat_v6:
5712023-10-24T01:11:39.283Zport_mac:
5722023-10-24T01:11:39.308Zproxy arp:
5732023-10-24T01:11:39.333Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5742023-10-24T01:11:39.358Z+ success 'softnpu zone exists'
5752023-10-24T01:11:39.382Z+ set +x
5762023-10-24T01:11:39.406Zsoftnpu zone exists
5772023-10-24T01:11:39.431Z
5782023-10-24T01:11:39.460Zreal 35.870616939
5792023-10-24T01:11:39.489Zuser 0.716833081
5802023-10-24T01:11:39.515Zsys 1.629607266
5812023-10-24T01:11:39.541Ztrap 0.000089825
5822023-10-24T01:11:39.579Ztflt 0.000127446
5832023-10-24T01:11:39.604Zdflt 0.001517574
5842023-10-24T01:11:39.628Zkflt 0.000002280
5852023-10-24T01:11:39.653Zlock 28:10.728993408
5862023-10-24T01:11:39.678Zslp 1:46.876561890
5872023-10-24T01:11:39.702Zlat 0.083519701
5882023-10-24T01:11:39.727Zstop 0.374473543
5892023-10-24T01:11:39.751Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5902023-10-24T01:11:39.776Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5912023-10-24T01:11:39.800Z+ SILO_NAME=recovery
5922023-10-24T01:11:39.825Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-10-24T01:11:39.849Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5942023-10-24T01:11:39.873Z+ sed -i~ '
5952023-10-24T01:11:39.898Z /^external_dns_ips/c\
5962023-10-24T01:11:39.924Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5972023-10-24T01:11:39.949Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5982023-10-24T01:11:39.973Z /^first/c\
5992023-10-24T01:11:39.998Zfirst = "10.151.1.164"
6002023-10-24T01:11:40.022Z /^last/c\
6012023-10-24T01:11:40.047Zlast = "10.151.1.173"
6022023-10-24T01:11:40.071Z }
6032023-10-24T01:11:40.096Z /^\[rack_network_config/,/^$/ {
6042023-10-24T01:11:40.120Z /^infra_ip_first/c\
6052023-10-24T01:11:40.144Zinfra_ip_first = "10.151.1.174"
6062023-10-24T01:11:40.169Z /^infra_ip_last/c\
6072023-10-24T01:11:40.194Zinfra_ip_last = "10.151.1.174"
6082023-10-24T01:11:40.219Z }
6092023-10-24T01:11:40.244Z /^\[\[rack_network_config.ports/,/^$/ {
6102023-10-24T01:11:40.268Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6112023-10-24T01:11:40.293Z /^routes/c\
6122023-10-24T01:11:40.319Z+++ pkg/config-rss.toml Tue Oct 24 01:11:39 2023
6132023-10-24T01:11:40.344Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6142023-10-24T01:11:40.369Z@@ -41,7 +41,7 @@
6152023-10-24T01:11:40.401Z /^addresses/c\
6162023-10-24T01:11:40.426Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6172023-10-24T01:11:40.450Zaddresses = \["10.151.1.174/32"\]
6182023-10-24T01:11:40.474Z # must be contained in one of the "internal services" IP Pool ranges listed
6192023-10-24T01:11:40.498Z }
6202023-10-24T01:11:40.523Z # below.
6212023-10-24T01:11:40.547Z' pkg/config-rss.toml
6222023-10-24T01:11:40.572Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6232023-10-24T01:11:40.621Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6242023-10-24T01:11:40.645Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6252023-10-24T01:11:40.670Z+ true
6262023-10-24T01:11:40.695Z
6272023-10-24T01:11:40.720Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6282023-10-24T01:11:40.745Z # Initial TLS certificates for the external API
6292023-10-24T01:11:40.769Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6302023-10-24T01:11:40.797Z #
6312023-10-24T01:11:40.825Z+ TLS_NAME=recovery.sys.oxide.test
6322023-10-24T01:11:40.851Z@@ -77,8 +77,8 @@
6332023-10-24T01:11:40.876Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6342023-10-24T01:11:40.902Z #
6352023-10-24T01:11:40.930Z..+.....+.........+....+...+.....+...+......+.........+....+..+.+...+..+....+...+......+...............+.........+...+..+......+.+.....+.+..+.................................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+....+..+.......+..+.+.....+..................+....+..+..........+........+.+......+...+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+......+...................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6362023-10-24T01:11:40.956Z # For more on this and what to put here, see docs/how-to-run.adoc.
6372023-10-24T01:11:40.981Z.....+...+.+...+..+.+...+..+.........+.+...+.........+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+...+..+.............+..+....+...+...+..+.+.........+..................+...+.....+.+...+..+...+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+.+.........+.....+...............+......+....+.....+................+..+.......+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6382023-10-24T01:11:41.005Z [[internal_services_ip_pool_ranges]]
6392023-10-24T01:11:41.030Z-----
6402023-10-24T01:11:41.056Z-first = "192.168.1.20"
6412023-10-24T01:11:41.080Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6422023-10-24T01:11:41.107Z-last = "192.168.1.29"
6432023-10-24T01:11:41.132Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6442023-10-24T01:11:41.157Z+first = "10.151.1.164"
6452023-10-24T01:11:41.182Z+ rmdir pkg
6462023-10-24T01:11:41.206Z+last = "10.151.1.173"
6472023-10-24T01:11:41.233Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6482023-10-24T01:11:41.258Z
6492023-10-24T01:11:41.283Z+ pfexec mkdir -p /usr/oxide
6502023-10-24T01:11:41.309Z # TODO - this configuration is subject to change going forward. Ultimately these
6512023-10-24T01:11:41.337Z+ pfexec rm -f /usr/oxide/tcpproxy
6522023-10-24T01:11:41.367Z # parameters should be provided to the control plane via wicket, but we need to
6532023-10-24T01:11:41.392Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6542023-10-24T01:11:41.417Z@@ -92,8 +92,8 @@
6552023-10-24T01:11:41.441Z+ pfexec chmod +x /usr/oxide/tcpproxy
6562023-10-24T01:11:41.465Z # A range of IP addresses used by Boundary Services on the external network. In
6572023-10-24T01:11:41.490Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6582023-10-24T01:11:41.514Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6592023-10-24T01:11:41.539Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6602023-10-24T01:11:41.564Z # With softnpu, only one address is used.
6612023-10-24T01:11:41.588Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6622023-10-24T01:11:41.613Z-infra_ip_first = "192.168.1.30"
6632023-10-24T01:11:41.646Z+ OMICRON_NO_UNINSTALL=1
6642023-10-24T01:11:41.672Z-infra_ip_last = "192.168.1.30"
6652023-10-24T01:11:41.696Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6662023-10-24T01:11:41.721Z+infra_ip_first = "10.151.1.174"
6672023-10-24T01:11:41.745ZOct 24 01:11:39.971 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6682023-10-24T01:11:41.770Z+infra_ip_last = "10.151.1.174"
6692023-10-24T01:11:41.797ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6702023-10-24T01:11:41.823Z
6712023-10-24T01:11:41.852ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6722023-10-24T01:11:41.879Z # Configurations for BGP routers to run on the scrimlets.
6732023-10-24T01:11:41.907ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6742023-10-24T01:11:41.932Z bgp = []
6752023-10-24T01:11:41.957ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6762023-10-24T01:11:41.982Z@@ -101,9 +101,9 @@
6772023-10-24T01:11:42.007ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6782023-10-24T01:11:42.032Z # You can configure multiple uplinks by repeating the following stanza
6792023-10-24T01:11:42.056ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6802023-10-24T01:11:42.081Z [[rack_network_config.ports]]
6812023-10-24T01:11:42.105ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6822023-10-24T01:11:42.130Z # Routes associated with this port.
6832023-10-24T01:11:42.154ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6842023-10-24T01:11:42.181Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6852023-10-24T01:11:42.206ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6862023-10-24T01:11:42.230Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6872023-10-24T01:11:42.255ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6882023-10-24T01:11:42.280Z # Addresses associated with this port.
6892023-10-24T01:11:42.708ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6902023-10-24T01:11:42.733Z-addresses = ["192.168.1.30/32"]
6912023-10-24T01:11:42.757ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6922023-10-24T01:11:42.782Z+addresses = ["10.151.1.174/32"]
6932023-10-24T01:11:42.807ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6942023-10-24T01:11:42.831Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6952023-10-24T01:11:42.856ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6962023-10-24T01:11:42.880Z port = "qsfp0"
6972023-10-24T01:11:42.909ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6982023-10-24T01:11:42.933Z # The speed of this port.
6992023-10-24T01:11:42.958ZOct 24 01:11:39.973 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7002023-10-24T01:11:42.983Za pkg/config-rss.toml 7K
7012023-10-24T01:11:43.008ZOct 24 01:11:40.204 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7022023-10-24T01:11:43.032Za pkg/initial-tls-cert.pem 2K
7032023-10-24T01:11:43.056ZOct 24 01:11:40.240 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7042023-10-24T01:11:43.081Za pkg/initial-tls-key.pem 4K
7052023-10-24T01:11:43.105ZOct 24 01:11:40.301 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7062023-10-24T01:11:43.130Z
7072023-10-24T01:11:43.155Zreal 0.469638382
7082023-10-24T01:11:43.180Zuser 0.096723294
7092023-10-24T01:11:43.207Zsys 1.830204798
7102023-10-24T01:11:43.231Ztrap 0.000000289
7112023-10-24T01:11:43.257Ztflt 0.000018277
7122023-10-24T01:11:43.281Zdflt 0.000294339
7132023-10-24T01:11:43.306Zkflt 0.000000000
7142023-10-24T01:11:43.332Zlock 42.272807878
7152023-10-24T01:11:43.356Zslp 0.679930661
7162023-10-24T01:11:43.382Zlat 0.002926445
7172023-10-24T01:11:43.409Zstop 0.205493731
7182023-10-24T01:11:43.434Z+ retry=0
7192023-10-24T01:11:43.484Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7202023-10-24T01:11:43.510Z+ [[ 0 -gt 30 ]]
7212023-10-24T01:11:43.536Z+ sleep 1
7222023-10-24T01:11:43.563Z+ retry=1
7232023-10-24T01:11:43.592Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7242023-10-24T01:11:43.618Z+ [[ 1 -gt 30 ]]
7252023-10-24T01:11:43.651Z+ sleep 1
7262023-10-24T01:11:43.697Z+ retry=2
7272023-10-24T01:11:43.722Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7282023-10-24T01:12:07.166Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7292023-10-24T01:12:07.195Zlocal v6:
7302023-10-24T01:12:07.220Zfd00:99::1
7312023-10-24T01:12:07.245Zfe80::b8e1:eff:fe02:ae39
7322023-10-24T01:12:07.279Zlocal v4:
7332023-10-24T01:12:07.758Z10.151.1.174
7342023-10-24T01:12:07.783Zrouter v6:
7352023-10-24T01:12:07.808Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::e49b:d2ff:fe9d:726d (1) vid=0
7362023-10-24T01:12:07.838Zfd00:1122:3344:101::/64 -> fe80::e49b:d2ff:fe9d:726d (1) vid=0
7372023-10-24T01:12:07.868Zrouter v4:
7382023-10-24T01:12:07.896Z0.0.0.0/0 -> 10.151.1.100 (2) vid=0
7392023-10-24T01:12:07.926Zresolver v4:
7402023-10-24T01:12:07.952Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7412023-10-24T01:12:07.986Z10.151.1.174 -> 2a:54:fd:69:11:67
7422023-10-24T01:12:08.021Zresolver v6:
7432023-10-24T01:12:08.054Zfe80::e49b:d2ff:fe9d:726d -> e6:9b:d2:9d:72:6d
7442023-10-24T01:12:08.085Zfe80::b8e1:eff:fe02:ae39 -> ba:e1:0e:02:ae:39
7452023-10-24T01:12:08.119Znat v4:
7462023-10-24T01:12:08.151Znat_v6:
7472023-10-24T01:12:08.183Zport_mac:
7482023-10-24T01:12:08.216Z1000: a8:40:25:55:4d:eb
7492023-10-24T01:12:08.250Z1: a8:40:25:55:4d:ec
7502023-10-24T01:12:08.281Zproxy arp:
7512023-10-24T01:12:08.317Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7522023-10-24T01:12:08.355Z+ retry=0
7532023-10-24T01:12:08.787Z+++ zoneadm list -n
7542023-10-24T01:12:09.310Z+++ grep oxz_ntp
7552023-10-24T01:12:09.336Z++ pfexec svcs -z -Hostate oxide/ntp
7562023-10-24T01:12:09.365Zsvcs: zone '-Hostate': entity not found
7572023-10-24T01:12:09.390Z++ true
7582023-10-24T01:12:09.415Z+ [[ '' != online ]]
7592023-10-24T01:12:09.458Z+ [[ 0 -gt 60 ]]
7602023-10-24T01:12:09.485Z+ sleep 1
7612023-10-24T01:12:09.513Z+ retry=1
7622023-10-24T01:12:09.541Z+++ zoneadm list -n
7632023-10-24T01:12:09.569Z+++ grep oxz_ntp
7642023-10-24T01:12:09.600Z++ pfexec svcs -z -Hostate oxide/ntp
7652023-10-24T01:12:09.629Zsvcs: zone '-Hostate': entity not found
7662023-10-24T01:12:09.657Z++ true
7672023-10-24T01:12:09.686Z+ [[ '' != online ]]
7682023-10-24T01:12:09.722Z+ [[ 1 -gt 60 ]]
7692023-10-24T01:12:09.767Z+ sleep 1
7702023-10-24T01:12:09.808Z+ retry=2
7712023-10-24T01:12:09.837Z+++ zoneadm list -n
7722023-10-24T01:12:09.871Z+++ grep oxz_ntp
7732023-10-24T01:12:09.906Z++ pfexec svcs -z -Hostate oxide/ntp
7742023-10-24T01:12:09.942Zsvcs: zone '-Hostate': entity not found
7752023-10-24T01:12:09.974Z++ true
7762023-10-24T01:12:10.002Z+ [[ '' != online ]]
7772023-10-24T01:12:10.038Z+ [[ 2 -gt 60 ]]
7782023-10-24T01:12:10.072Z+ sleep 1
7792023-10-24T01:12:10.266Z+ retry=3
7802023-10-24T01:12:10.291Z+++ zoneadm list -n
7812023-10-24T01:12:10.315Z+++ grep oxz_ntp
7822023-10-24T01:12:10.340Z++ pfexec svcs -z -Hostate oxide/ntp
7832023-10-24T01:12:10.366Zsvcs: zone '-Hostate': entity not found
7842023-10-24T01:12:10.391Z++ true
7852023-10-24T01:12:10.418Z+ [[ '' != online ]]
7862023-10-24T01:12:10.448Z+ [[ 3 -gt 60 ]]
7872023-10-24T01:12:10.491Z+ sleep 1
7882023-10-24T01:12:11.289Z+ retry=4
7892023-10-24T01:12:11.315Z+++ zoneadm list -n
7902023-10-24T01:12:11.339Z+++ grep oxz_ntp
7912023-10-24T01:12:11.372Z++ pfexec svcs -z -Hostate oxide/ntp
7922023-10-24T01:12:11.406Zsvcs: zone '-Hostate': entity not found
7932023-10-24T01:12:11.441Z++ true
7942023-10-24T01:12:11.467Z+ [[ '' != online ]]
7952023-10-24T01:12:11.517Z+ [[ 4 -gt 60 ]]
7962023-10-24T01:12:11.543Z+ sleep 1
7972023-10-24T01:12:12.312Z+ retry=5
7982023-10-24T01:12:12.345Z+++ zoneadm list -n
7992023-10-24T01:12:12.384Z+++ grep oxz_ntp
8002023-10-24T01:12:12.417Z++ pfexec svcs -z -Hostate oxide/ntp
8012023-10-24T01:12:12.444Zsvcs: zone '-Hostate': entity not found
8022023-10-24T01:12:12.472Z++ true
8032023-10-24T01:12:12.503Z+ [[ '' != online ]]
8042023-10-24T01:12:12.550Z+ [[ 5 -gt 60 ]]
8052023-10-24T01:12:12.589Z+ sleep 1
8062023-10-24T01:12:13.338Z+ retry=6
8072023-10-24T01:12:13.377Z+++ zoneadm list -n
8082023-10-24T01:12:13.488Z+++ grep oxz_ntp
8092023-10-24T01:12:13.969Z++ pfexec svcs -z -Hostate oxide/ntp
8102023-10-24T01:12:13.997Zsvcs: zone '-Hostate': entity not found
8112023-10-24T01:12:14.030Z++ true
8122023-10-24T01:12:14.066Z+ [[ '' != online ]]
8132023-10-24T01:12:14.949Z+ [[ 6 -gt 60 ]]
8142023-10-24T01:12:14.976Z+ sleep 1
8152023-10-24T01:12:15.013Z+ retry=7
8162023-10-24T01:12:15.047Z+++ zoneadm list -n
8172023-10-24T01:12:15.078Z+++ grep oxz_ntp
8182023-10-24T01:12:15.111Z++ pfexec svcs -z -Hostate oxide/ntp
8192023-10-24T01:12:15.140Zsvcs: zone '-Hostate': entity not found
8202023-10-24T01:12:15.168Z++ true
8212023-10-24T01:12:15.196Z+ [[ '' != online ]]
8222023-10-24T01:12:15.235Z+ [[ 7 -gt 60 ]]
8232023-10-24T01:12:15.270Z+ sleep 1
8242023-10-24T01:12:15.381Z+ retry=8
8252023-10-24T01:12:15.420Z+++ zoneadm list -n
8262023-10-24T01:12:15.460Z+++ grep oxz_ntp
8272023-10-24T01:12:15.494Z++ pfexec svcs -z -Hostate oxide/ntp
8282023-10-24T01:12:15.521Zsvcs: zone '-Hostate': entity not found
8292023-10-24T01:12:15.547Z++ true
8302023-10-24T01:12:15.572Z+ [[ '' != online ]]
8312023-10-24T01:12:15.596Z+ [[ 8 -gt 60 ]]
8322023-10-24T01:12:15.622Z+ sleep 1
8332023-10-24T01:12:16.402Z+ retry=9
8342023-10-24T01:12:16.439Z+++ zoneadm list -n
8352023-10-24T01:12:16.478Z+++ grep oxz_ntp
8362023-10-24T01:12:17.724Z++ pfexec svcs -z oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216 -Hostate oxide/ntp
8372023-10-24T01:12:17.750Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8382023-10-24T01:12:17.782Z++ true
8392023-10-24T01:12:17.818Z+ [[ '' != online ]]
8402023-10-24T01:12:17.848Z+ [[ 9 -gt 60 ]]
8412023-10-24T01:12:17.873Z+ sleep 1
8422023-10-24T01:12:17.902Z+ retry=10
8432023-10-24T01:12:17.930Z+++ zoneadm list -n
8442023-10-24T01:12:17.961Z+++ grep oxz_ntp
8452023-10-24T01:12:17.986Z++ pfexec svcs -z oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216 -Hostate oxide/ntp
8462023-10-24T01:12:18.832Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8472023-10-24T01:12:18.859Z++ true
8482023-10-24T01:12:18.885Z+ [[ '' != online ]]
8492023-10-24T01:12:18.911Z+ [[ 10 -gt 60 ]]
8502023-10-24T01:12:18.938Z+ sleep 1
8512023-10-24T01:12:18.984Z+ retry=11
8522023-10-24T01:12:19.016Z+++ zoneadm list -n
8532023-10-24T01:12:19.132Z+++ grep oxz_ntp
8542023-10-24T01:12:19.165Z++ pfexec svcs -z oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216 -Hostate oxide/ntp
8552023-10-24T01:12:19.194Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8562023-10-24T01:12:19.225Z++ true
8572023-10-24T01:12:19.255Z+ [[ '' != online ]]
8582023-10-24T01:12:19.297Z+ [[ 11 -gt 60 ]]
8592023-10-24T01:12:19.342Z+ sleep 1
8602023-10-24T01:12:19.505Z+ retry=12
8612023-10-24T01:12:19.536Z+++ zoneadm list -n
8622023-10-24T01:12:19.562Z+++ grep oxz_ntp
8632023-10-24T01:12:19.588Z++ pfexec svcs -z oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216 -Hostate oxide/ntp
8642023-10-24T01:12:19.615Z+ [[ disabled != online ]]
8652023-10-24T01:12:19.643Z+ [[ 12 -gt 60 ]]
8662023-10-24T01:12:19.671Z+ sleep 1
8672023-10-24T01:12:20.516Z+ retry=13
8682023-10-24T01:12:20.543Z+++ zoneadm list -n
8692023-10-24T01:12:20.569Z+++ grep oxz_ntp
8702023-10-24T01:12:20.593Z++ pfexec svcs -z oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216 -Hostate oxide/ntp
8712023-10-24T01:12:20.618Z+ [[ disabled != online ]]
8722023-10-24T01:12:20.667Z+ [[ 13 -gt 60 ]]
8732023-10-24T01:12:20.704Z+ sleep 1
8742023-10-24T01:12:21.537Z+ retry=14
8752023-10-24T01:12:21.568Z+++ zoneadm list -n
8762023-10-24T01:12:21.596Z+++ grep oxz_ntp
8772023-10-24T01:12:21.633Z++ pfexec svcs -z oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216 -Hostate oxide/ntp
8782023-10-24T01:12:21.664Z+ [[ disabled != online ]]
8792023-10-24T01:12:21.695Z+ [[ 14 -gt 60 ]]
8802023-10-24T01:12:21.726Z+ sleep 1
8812023-10-24T01:12:22.565Z+ retry=15
8822023-10-24T01:12:22.605Z+++ zoneadm list -n
8832023-10-24T01:12:22.647Z+++ grep oxz_ntp
8842023-10-24T01:12:22.684Z++ pfexec svcs -z oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216 -Hostate oxide/ntp
8852023-10-24T01:12:22.719Z+ [[ disabled != online ]]
8862023-10-24T01:12:22.753Z+ [[ 15 -gt 60 ]]
8872023-10-24T01:12:22.791Z+ sleep 1
8882023-10-24T01:12:23.587Z+ retry=16
8892023-10-24T01:12:23.624Z+++ zoneadm list -n
8902023-10-24T01:12:23.656Z+++ grep oxz_ntp
8912023-10-24T01:12:23.685Z++ pfexec svcs -z oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216 -Hostate oxide/ntp
8922023-10-24T01:12:23.715Z+ [[ disabled != online ]]
8932023-10-24T01:12:23.742Z+ [[ 16 -gt 60 ]]
8942023-10-24T01:12:23.774Z+ sleep 1
8952023-10-24T01:12:24.601Z+ retry=17
8962023-10-24T01:12:24.626Z+++ zoneadm list -n
8972023-10-24T01:12:24.652Z+++ grep oxz_ntp
8982023-10-24T01:12:24.678Z++ pfexec svcs -z oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216 -Hostate oxide/ntp
8992023-10-24T01:12:24.703Z+ [[ disabled != online ]]
9002023-10-24T01:12:24.731Z+ [[ 17 -gt 60 ]]
9012023-10-24T01:12:24.760Z+ sleep 1
9022023-10-24T01:12:25.629Z+ retry=18
9032023-10-24T01:12:25.660Z+++ zoneadm list -n
9042023-10-24T01:12:25.701Z+++ grep oxz_ntp
9052023-10-24T01:12:25.731Z++ pfexec svcs -z oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216 -Hostate oxide/ntp
9062023-10-24T01:12:25.761Z+ [[ disabled != online ]]
9072023-10-24T01:12:25.802Z+ [[ 18 -gt 60 ]]
9082023-10-24T01:12:26.281Z+ sleep 1
9092023-10-24T01:12:26.653Z+ retry=19
9102023-10-24T01:12:26.695Z+++ zoneadm list -n
9112023-10-24T01:12:26.731Z+++ grep oxz_ntp
9122023-10-24T01:12:26.768Z++ pfexec svcs -z oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216 -Hostate oxide/ntp
9132023-10-24T01:12:26.806Z+ [[ disabled != online ]]
9142023-10-24T01:12:26.845Z+ [[ 19 -gt 60 ]]
9152023-10-24T01:12:27.289Z+ sleep 1
 
9162023-10-24T01:12:27.640Zbuskin console login: Oct 24 01:12:27 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9172023-10-24T01:12:27.681Z+ retry=20
9182023-10-24T01:12:28.156Z+++ zoneadm list -n
9192023-10-24T01:12:28.184Z+++ grep oxz_ntp
9202023-10-24T01:12:28.210Z++ pfexec svcs -z oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216 -Hostate oxide/ntp
9212023-10-24T01:12:28.237Z+ [[ online != online ]]
9222023-10-24T01:12:28.675Z+ echo 'Waited for chrony: 20s'
9232023-10-24T01:12:28.706ZWaited for chrony: 20s
9242023-10-24T01:12:28.754Z+ export RUST_BACKTRACE=1
9252023-10-24T01:12:28.783Z+ RUST_BACKTRACE=1
9262023-10-24T01:12:28.817Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9272023-10-24T01:12:28.958Z+ ./tests/bootstrap
9282023-10-24T01:12:29.005ZUsing external DNS server at 10.151.1.164:53
9292023-10-24T01:12:29.045Z2023-10-24 01:12:28.189459975 UTC: attempting to log into API
9302023-10-24T01:12:42.771Z2023-10-24 01:12:42.443184945 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9312023-10-24T01:12:43.808Z2023-10-24 01:12:43.444058862 UTC: attempting to log into API
9322023-10-24T01:12:58.796Z2023-10-24 01:12:58.478557149 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9332023-10-24T01:12:59.793Z2023-10-24 01:12:59.479543028 UTC: attempting to log into API
9342023-10-24T01:13:14.829Z2023-10-24 01:13:14.512597447 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9352023-10-24T01:13:15.828Z2023-10-24 01:13:15.513289420 UTC: attempting to log into API
9362023-10-24T01:13:30.866Z2023-10-24 01:13:30.548324836 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9372023-10-24T01:13:31.862Z2023-10-24 01:13:31.548871127 UTC: attempting to log into API
9382023-10-24T01:13:46.897Z2023-10-24 01:13:46.606131612 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9392023-10-24T01:13:47.898Z2023-10-24 01:13:47.610556916 UTC: attempting to log into API
9402023-10-24T01:14:02.932Z2023-10-24 01:14:02.684186378 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9412023-10-24T01:14:03.933Z2023-10-24 01:14:03.687871397 UTC: attempting to log into API
9422023-10-24T01:14:18.965Z2023-10-24 01:14:18.761699664 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9432023-10-24T01:14:19.970Z2023-10-24 01:14:19.765239734 UTC: attempting to log into API
9442023-10-24T01:14:51.787Z2023-10-24 01:14:34.839780933 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9452023-10-24T01:14:52.173Z2023-10-24 01:14:35.843134889 UTC: attempting to log into API
9462023-10-24T01:14:52.293Z2023-10-24 01:14:50.917701903 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9472023-10-24T01:14:52.438Z2023-10-24 01:14:51.922147719 UTC: attempting to log into API
9482023-10-24T01:15:07.071Z2023-10-24 01:15:06.997244262 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9492023-10-24T01:15:08.073Z2023-10-24 01:15:08.001900172 UTC: attempting to log into API
9502023-10-24T01:15:23.106Z2023-10-24 01:15:23.075682212 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9512023-10-24T01:15:24.106Z2023-10-24 01:15:24.079308056 UTC: attempting to log into API
9522023-10-24T01:15:39.144Z2023-10-24 01:15:39.127907614 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9532023-10-24T01:15:40.143Z2023-10-24 01:15:40.129878042 UTC: attempting to log into API
9542023-10-24T01:15:55.209Z2023-10-24 01:15:55.196645139 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9552023-10-24T01:15:56.210Z2023-10-24 01:15:56.197721370 UTC: attempting to log into API
9562023-10-24T01:16:11.253Z2023-10-24 01:16:11.239955981 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9572023-10-24T01:16:12.253Z2023-10-24 01:16:12.240811347 UTC: attempting to log into API
9582023-10-24T01:16:27.323Z2023-10-24 01:16:27.310187551 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9592023-10-24T01:16:28.325Z2023-10-24 01:16:28.311374514 UTC: attempting to log into API
9602023-10-24T01:16:43.408Z2023-10-24 01:16:43.346858092 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9612023-10-24T01:16:52.587Z2023-10-24 01:16:44.347911663 UTC: attempting to log into API
 
9622023-10-24T01:16:52.670ZOct 24 01:16:48 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9632023-10-24T01:16:55.310ZOct 24 01:16:54 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9642023-10-24T01:16:59.395Z2023-10-24 01:16:59.381720049 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9652023-10-24T01:17:00.395Z2023-10-24 01:17:00.382872699 UTC: attempting to log into API
 
9662023-10-24T01:17:02.073ZOct 24 01:17:01 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
9672023-10-24T01:17:09.308ZOct 24 01:17:08 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9682023-10-24T01:17:15.430Z2023-10-24 01:17:15.415694457 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9692023-10-24T01:17:15.783ZOct 24 01:17:15 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9702023-10-24T01:17:16.437Z2023-10-24 01:17:16.416818228 UTC: attempting to log into API
9712023-10-24T01:17:31.464Z2023-10-24 01:17:31.451898188 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9722023-10-24T01:17:32.465Z2023-10-24 01:17:32.452784434 UTC: attempting to log into API
9732023-10-24T01:17:47.498Z2023-10-24 01:17:47.485161339 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9742023-10-24T01:17:48.499Z2023-10-24 01:17:48.486257420 UTC: attempting to log into API
9752023-10-24T01:18:03.535Z2023-10-24 01:18:03.518532957 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9762023-10-24T01:18:04.533Z2023-10-24 01:18:04.519623928 UTC: attempting to log into API
9772023-10-24T01:18:15.320Z2023-10-24 01:18:15.307495587 UTC: login succeeded
9782023-10-24T01:18:15.353Zwaiting for nexus to come up...
9792023-10-24T01:18:15.416Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9802023-10-24T01:18:15.467Zensuring datasets are ready...
9812023-10-24T01:18:17.900Zlet's roll.
9822023-10-24T01:18:17.928Z+ rm ./tests/bootstrap
9832023-10-24T01:18:17.994Z+ for test_bin in tests/*
9842023-10-24T01:18:18.020Z+ ./tests/bootstrap-49e31e75b930b622
9852023-10-24T01:18:18.045Z
9862023-10-24T01:18:18.070Zrunning 0 tests
9872023-10-24T01:18:18.094Z
9882023-10-24T01:18:18.118Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9892023-10-24T01:18:18.142Z
9902023-10-24T01:18:18.166Z+ for test_bin in tests/*
9912023-10-24T01:18:18.190Z+ ./tests/end_to_end_tests-a91cf9af3457d280
9922023-10-24T01:18:18.215Z
9932023-10-24T01:18:18.240Zrunning 2 tests
9942023-10-24T01:18:18.265Ztest no_spoof::no_spoof_here ... ok
9952023-10-24T01:19:18.030Ztest instance_launch::instance_launch has been running for over 60 seconds
9962023-10-24T01:21:11.315Ztest instance_launch::instance_launch ... ok
9972023-10-24T01:21:11.340Z
9982023-10-24T01:21:11.365Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 173.29s
9992023-10-24T01:21:11.391Z
10002023-10-24T01:21:11.415Z+ _exit_trap
10012023-10-24T01:21:11.440Z+ local status=0
10022023-10-24T01:21:11.464Z+ [[ 0 -eq 0 ]]
10032023-10-24T01:21:11.495Z+ exit 0
10042023-10-24T01:21:11.523Zprocess exited: duration 616926 ms, exit code 0
 
10052023-10-24T01:21:11.642Zfound 51 output files
10062023-10-24T01:21:11.832Zuploading: /var/svc/log/oxide-sled-agent:default.log (486976 bytes)
10072023-10-24T01:21:12.907Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10082023-10-24T01:21:12.935Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2157347f-a20e-4841-8b0d-1a67e7a423c1/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
10092023-10-24T01:21:14.008Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2157347f-a20e-4841-8b0d-1a67e7a423c1/root/var/svc/log/oxide-cockroachdb:default.log
10102023-10-24T01:21:14.033Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_2790d10e-25b0-4245-8b12-7910827b52d4/root/var/svc/log/oxide-internal_dns:default.log (93570 bytes)
10112023-10-24T01:21:15.230Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_2790d10e-25b0-4245-8b12-7910827b52d4/root/var/svc/log/oxide-internal_dns:default.log
10122023-10-24T01:21:15.254Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_2790d10e-25b0-4245-8b12-7910827b52d4/root/var/svc/log/oxide-internal_dns:default.log.0 (4286 bytes)
10132023-10-24T01:21:16.350Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_2790d10e-25b0-4245-8b12-7910827b52d4/root/var/svc/log/oxide-internal_dns:default.log.0
10142023-10-24T01:21:16.375Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_cacd2320-aa2e-4537-92de-15bb1dca8a99/root/var/svc/log/oxide-internal_dns:default.log (93571 bytes)
10152023-10-24T01:21:16.543Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_cacd2320-aa2e-4537-92de-15bb1dca8a99/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93571 -> 97322
10162023-10-24T01:21:17.614Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_cacd2320-aa2e-4537-92de-15bb1dca8a99/root/var/svc/log/oxide-internal_dns:default.log
10172023-10-24T01:21:17.639Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_cacd2320-aa2e-4537-92de-15bb1dca8a99/root/var/svc/log/oxide-internal_dns:default.log.0 (9558 bytes)
10182023-10-24T01:21:18.734Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_cacd2320-aa2e-4537-92de-15bb1dca8a99/root/var/svc/log/oxide-internal_dns:default.log.0
10192023-10-24T01:21:18.765Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_146855ef-6459-4a56-b8fc-c558864fb006/root/var/svc/log/oxide-nexus:default.log (2039524 bytes)
10202023-10-24T01:21:20.294Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_146855ef-6459-4a56-b8fc-c558864fb006/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 2039524 -> 2097711
10212023-10-24T01:21:21.366Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_146855ef-6459-4a56-b8fc-c558864fb006/root/var/svc/log/oxide-nexus:default.log
10222023-10-24T01:21:21.392Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_89e35822-b255-4c01-b5eb-b4c72a068954/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10232023-10-24T01:21:22.480Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_89e35822-b255-4c01-b5eb-b4c72a068954/root/var/svc/log/oxide-cockroachdb:default.log
10242023-10-24T01:21:22.506Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0db13e5c-51e9-49b5-93b6-9b50dd1ddb5f/root/var/svc/log/oxide-crucible-agent:default.log (13687 bytes)
10252023-10-24T01:21:23.613Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0db13e5c-51e9-49b5-93b6-9b50dd1ddb5f/root/var/svc/log/oxide-crucible-agent:default.log
10262023-10-24T01:21:23.637Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0db13e5c-51e9-49b5-93b6-9b50dd1ddb5f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10272023-10-24T01:21:24.711Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0db13e5c-51e9-49b5-93b6-9b50dd1ddb5f/root/var/svc/log/oxide-crucible-downstairs:default.log
10282023-10-24T01:21:24.735Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0db13e5c-51e9-49b5-93b6-9b50dd1ddb5f/root/var/svc/log/oxide-crucible-downstairs:downstairs-e6368670-206b-4c7b-9db8-fda46bf0beb7.log (3050 bytes)
10292023-10-24T01:21:26.216Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0db13e5c-51e9-49b5-93b6-9b50dd1ddb5f/root/var/svc/log/oxide-crucible-downstairs:downstairs-e6368670-206b-4c7b-9db8-fda46bf0beb7.log
10302023-10-24T01:21:26.647Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6dea1c6e-e6f8-48b7-a975-9c2942f34c41/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
10312023-10-24T01:21:26.946Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6dea1c6e-e6f8-48b7-a975-9c2942f34c41/root/var/svc/log/oxide-crucible-agent:default.log
10322023-10-24T01:21:26.971Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6dea1c6e-e6f8-48b7-a975-9c2942f34c41/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10332023-10-24T01:21:28.039Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6dea1c6e-e6f8-48b7-a975-9c2942f34c41/root/var/svc/log/oxide-crucible-downstairs:default.log
10342023-10-24T01:21:28.064Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6dea1c6e-e6f8-48b7-a975-9c2942f34c41/root/var/svc/log/oxide-crucible-downstairs:downstairs-2d4f04aa-91ed-445c-8d45-4b2c05e4c183.log (5536 bytes)
10352023-10-24T01:21:29.134Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6dea1c6e-e6f8-48b7-a975-9c2942f34c41/root/var/svc/log/oxide-crucible-downstairs:downstairs-2d4f04aa-91ed-445c-8d45-4b2c05e4c183.log
10362023-10-24T01:21:29.165Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_9e70e08d-4e15-451c-8fb0-8c2d46453cbd/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10372023-10-24T01:21:30.709Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_9e70e08d-4e15-451c-8fb0-8c2d46453cbd/root/var/svc/log/oxide-crucible-pantry:default.log
10382023-10-24T01:21:30.734Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4537ec1e-5350-4112-b93a-4a5390706e59/root/var/svc/log/oxide-internal_dns:default.log (93570 bytes)
10392023-10-24T01:21:31.026Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4537ec1e-5350-4112-b93a-4a5390706e59/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93570 -> 97325
10402023-10-24T01:21:32.097Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4537ec1e-5350-4112-b93a-4a5390706e59/root/var/svc/log/oxide-internal_dns:default.log
10412023-10-24T01:21:32.121Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4537ec1e-5350-4112-b93a-4a5390706e59/root/var/svc/log/oxide-internal_dns:default.log.0 (7537 bytes)
10422023-10-24T01:21:33.222Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4537ec1e-5350-4112-b93a-4a5390706e59/root/var/svc/log/oxide-internal_dns:default.log.0
10432023-10-24T01:21:33.249Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c5873794-687d-4c71-9491-ed223bf60da8/root/var/svc/log/oxide-nexus:default.log (944276 bytes)
10442023-10-24T01:21:35.612Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c5873794-687d-4c71-9491-ed223bf60da8/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 944276 -> 1003676
10452023-10-24T01:21:36.682Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c5873794-687d-4c71-9491-ed223bf60da8/root/var/svc/log/oxide-nexus:default.log
10462023-10-24T01:21:36.707Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_87594456-2962-4a8f-bddc-f584970881a7/root/var/svc/log/oxide-oximeter:default.log (207626 bytes)
10472023-10-24T01:21:37.085Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_87594456-2962-4a8f-bddc-f584970881a7/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 207626 -> 228328
10482023-10-24T01:21:38.157Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_87594456-2962-4a8f-bddc-f584970881a7/root/var/svc/log/oxide-oximeter:default.log
10492023-10-24T01:21:38.184Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_2b112e60-c5bc-4e56-972c-df70442f2976/root/var/svc/log/oxide-clickhouse:default.log (766460 bytes)
10502023-10-24T01:21:39.075Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_2b112e60-c5bc-4e56-972c-df70442f2976/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 766460 -> 844202
10512023-10-24T01:21:40.145Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_2b112e60-c5bc-4e56-972c-df70442f2976/root/var/svc/log/oxide-clickhouse:default.log
10522023-10-24T01:21:40.171Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_20c0094c-b8dc-458e-872d-8b5a51384999/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10532023-10-24T01:21:41.247Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_20c0094c-b8dc-458e-872d-8b5a51384999/root/var/svc/log/oxide-crucible-agent:default.log
10542023-10-24T01:21:41.273Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_20c0094c-b8dc-458e-872d-8b5a51384999/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10552023-10-24T01:21:42.348Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_20c0094c-b8dc-458e-872d-8b5a51384999/root/var/svc/log/oxide-crucible-downstairs:default.log
10562023-10-24T01:21:42.373Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5c0d283a-16d2-43aa-a223-7a644412ec77/root/var/svc/log/oxide-crucible-agent:default.log (24366 bytes)
10572023-10-24T01:21:43.487Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5c0d283a-16d2-43aa-a223-7a644412ec77/root/var/svc/log/oxide-crucible-agent:default.log
10582023-10-24T01:21:43.513Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5c0d283a-16d2-43aa-a223-7a644412ec77/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10592023-10-24T01:21:44.581Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5c0d283a-16d2-43aa-a223-7a644412ec77/root/var/svc/log/oxide-crucible-downstairs:default.log
10602023-10-24T01:21:44.606Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5c0d283a-16d2-43aa-a223-7a644412ec77/root/var/svc/log/oxide-crucible-downstairs:downstairs-21634402-5e8a-4d38-b357-e2b6c35f1f4c.log (3050 bytes)
10612023-10-24T01:21:45.675Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5c0d283a-16d2-43aa-a223-7a644412ec77/root/var/svc/log/oxide-crucible-downstairs:downstairs-21634402-5e8a-4d38-b357-e2b6c35f1f4c.log
10622023-10-24T01:21:45.699Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5c0d283a-16d2-43aa-a223-7a644412ec77/root/var/svc/log/oxide-crucible-downstairs:downstairs-b3f1884c-f588-405b-a526-f558d3fa1034.log (5536 bytes)
10632023-10-24T01:21:46.777Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_5c0d283a-16d2-43aa-a223-7a644412ec77/root/var/svc/log/oxide-crucible-downstairs:downstairs-b3f1884c-f588-405b-a526-f558d3fa1034.log
10642023-10-24T01:21:46.802Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_97177e0b-645b-4e36-bc68-8a8092cf6702/root/var/svc/log/oxide-external_dns:default.log (15613 bytes)
10652023-10-24T01:21:46.869Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_97177e0b-645b-4e36-bc68-8a8092cf6702/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15613 -> 19390
10662023-10-24T01:21:47.938Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_97177e0b-645b-4e36-bc68-8a8092cf6702/root/var/svc/log/oxide-external_dns:default.log
10672023-10-24T01:21:47.963Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_d4c7559c-d360-4d93-a4f7-cffc84b3c92b/root/var/svc/log/oxide-external_dns:default.log (15602 bytes)
10682023-10-24T01:21:48.029Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_d4c7559c-d360-4d93-a4f7-cffc84b3c92b/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15602 -> 19380
10692023-10-24T01:21:49.101Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_d4c7559c-d360-4d93-a4f7-cffc84b3c92b/root/var/svc/log/oxide-external_dns:default.log
10702023-10-24T01:21:49.127Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_6cad7b90-24b3-49c0-9587-a1b977e9751d/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10712023-10-24T01:21:50.219Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_6cad7b90-24b3-49c0-9587-a1b977e9751d/root/var/svc/log/oxide-cockroachdb:default.log
10722023-10-24T01:21:50.654Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1aaa54ff-0fa1-4bba-ab70-0c4d8131034f/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
10732023-10-24T01:21:51.318Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1aaa54ff-0fa1-4bba-ab70-0c4d8131034f/root/var/svc/log/oxide-crucible-agent:default.log
10742023-10-24T01:21:51.342Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1aaa54ff-0fa1-4bba-ab70-0c4d8131034f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10752023-10-24T01:21:52.415Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1aaa54ff-0fa1-4bba-ab70-0c4d8131034f/root/var/svc/log/oxide-crucible-downstairs:default.log
10762023-10-24T01:21:52.440Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_66b00766-f623-4a26-8bc4-f6f6603471e5/root/var/svc/log/oxide-crucible-agent:default.log (13683 bytes)
10772023-10-24T01:21:53.554Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_66b00766-f623-4a26-8bc4-f6f6603471e5/root/var/svc/log/oxide-crucible-agent:default.log
10782023-10-24T01:21:53.582Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_66b00766-f623-4a26-8bc4-f6f6603471e5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10792023-10-24T01:21:54.707Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_66b00766-f623-4a26-8bc4-f6f6603471e5/root/var/svc/log/oxide-crucible-downstairs:default.log
10802023-10-24T01:21:54.731Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_66b00766-f623-4a26-8bc4-f6f6603471e5/root/var/svc/log/oxide-crucible-downstairs:downstairs-ec99a654-773b-49f1-883b-8f8ad2346542.log (3049 bytes)
10812023-10-24T01:21:55.806Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_66b00766-f623-4a26-8bc4-f6f6603471e5/root/var/svc/log/oxide-crucible-downstairs:downstairs-ec99a654-773b-49f1-883b-8f8ad2346542.log
10822023-10-24T01:21:55.830Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_273caebc-758e-4958-96c7-1c7c2d77d352/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10832023-10-24T01:21:56.902Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_273caebc-758e-4958-96c7-1c7c2d77d352/root/var/svc/log/oxide-cockroachdb:default.log
10842023-10-24T01:21:56.926Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_02de416f-d570-4919-a5de-66748771264c/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10852023-10-24T01:21:57.998Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_02de416f-d570-4919-a5de-66748771264c/root/var/svc/log/oxide-crucible-pantry:default.log
10862023-10-24T01:21:58.023Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_db1bd18a-e2d3-4307-83e0-7802865783ee/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10872023-10-24T01:21:59.096Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_db1bd18a-e2d3-4307-83e0-7802865783ee/root/var/svc/log/oxide-cockroachdb:default.log
10882023-10-24T01:21:59.121Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d13b971e-3b1c-406a-8f5e-e830b6165490/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10892023-10-24T01:22:00.208Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d13b971e-3b1c-406a-8f5e-e830b6165490/root/var/svc/log/oxide-crucible-agent:default.log
10902023-10-24T01:22:00.233Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d13b971e-3b1c-406a-8f5e-e830b6165490/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10912023-10-24T01:22:01.308Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_d13b971e-3b1c-406a-8f5e-e830b6165490/root/var/svc/log/oxide-crucible-downstairs:default.log
10922023-10-24T01:22:01.334Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216/root/var/svc/log/oxide-ntp:default.log (0 bytes)
10932023-10-24T01:22:02.381Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216/root/var/svc/log/oxide-ntp:default.log
10942023-10-24T01:22:02.407Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216/root/var/svc/log/oxide-ntp:default.log.0 (1324 bytes)
10952023-10-24T01:22:03.477Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216/root/var/svc/log/oxide-ntp:default.log.0
10962023-10-24T01:22:03.503Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e8c4183d-eaa2-4b1d-beb6-61cca6329970/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
10972023-10-24T01:22:04.597Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e8c4183d-eaa2-4b1d-beb6-61cca6329970/root/var/svc/log/oxide-crucible-agent:default.log
10982023-10-24T01:22:04.623Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e8c4183d-eaa2-4b1d-beb6-61cca6329970/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10992023-10-24T01:22:05.692Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e8c4183d-eaa2-4b1d-beb6-61cca6329970/root/var/svc/log/oxide-crucible-downstairs:default.log
11002023-10-24T01:22:05.717Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e8c4183d-eaa2-4b1d-beb6-61cca6329970/root/var/svc/log/oxide-crucible-downstairs:downstairs-87645e9e-f22e-449a-8693-39f2badd257d.log (5536 bytes)
11012023-10-24T01:22:06.815Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e8c4183d-eaa2-4b1d-beb6-61cca6329970/root/var/svc/log/oxide-crucible-downstairs:downstairs-87645e9e-f22e-449a-8693-39f2badd257d.log
11022023-10-24T01:22:06.839Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_24caf15a-5e4c-474c-a0f9-f7fd7ccde57e/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
11032023-10-24T01:22:07.909Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_24caf15a-5e4c-474c-a0f9-f7fd7ccde57e/root/var/svc/log/oxide-crucible-pantry:default.log
11042023-10-24T01:22:07.934Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d581e938-d7fe-430d-82e6-3bae8868d1a1/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
11052023-10-24T01:22:09.017Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d581e938-d7fe-430d-82e6-3bae8868d1a1/root/var/svc/log/oxide-crucible-agent:default.log
11062023-10-24T01:22:09.044Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d581e938-d7fe-430d-82e6-3bae8868d1a1/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11072023-10-24T01:22:10.117Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d581e938-d7fe-430d-82e6-3bae8868d1a1/root/var/svc/log/oxide-crucible-downstairs:default.log
11082023-10-24T01:22:10.152Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c7ab4c01-242a-4a6e-8db1-52fa3718c68b/root/var/svc/log/oxide-nexus:default.log (2844771 bytes)
11092023-10-24T01:22:12.368Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c7ab4c01-242a-4a6e-8db1-52fa3718c68b/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 2844771 -> 2906594
11102023-10-24T01:22:13.466Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c7ab4c01-242a-4a6e-8db1-52fa3718c68b/root/var/svc/log/oxide-nexus:default.log
11112023-10-24T01:22:13.490Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216/root/var/log/chrony/measurements.log (18632 bytes)
11122023-10-24T01:22:13.555Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216/root/var/log/chrony/measurements.log" changed size mid upload: 18632 -> 20687
11132023-10-24T01:22:14.629Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216/root/var/log/chrony/measurements.log
11142023-10-24T01:22:14.655Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216/root/var/log/chrony/statistics.log (11700 bytes)
11152023-10-24T01:22:15.112Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216/root/var/log/chrony/statistics.log" changed size mid upload: 11700 -> 13455
11162023-10-24T01:22:15.781Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216/root/var/log/chrony/statistics.log
11172023-10-24T01:22:15.806Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216/root/var/log/chrony/tracking.log (2250 bytes)
11182023-10-24T01:22:15.830Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216/root/var/log/chrony/tracking.log" changed size mid upload: 2250 -> 2382
11192023-10-24T01:22:16.898Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_91d69a37-b756-4d9f-adcf-ad0498603216/root/var/log/chrony/tracking.log