01HGKDKC20K25FN1W9H7AZMK5E: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HGKDKV784M8BYTACJVBXS6W7

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-01T20:13:04.822Zjob dependencies complete; ready to run (waiting for 45 m 19 s)
22023-12-01T20:13:57.074Zjob assigned to worker 01HGKG70WBSEZRHN35ZHEPPBPB (queued for 52 s)
32023-12-01T20:13:57.211Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-12-01T20:13:57.295Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-12-01T20:13:57.324Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-12-01T20:13:57.361Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-01T20:13:57.391Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-12-01T20:13:57.425ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-01T20:13:57.458Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-01T20:13:57.492ZOxide Helios Version helios-2.0.22318 64-bit
112023-12-01T20:13:57.525ZConfiguring devices.
122023-12-01T20:13:57.558Z#################################################
132023-12-01T20:13:57.592Z#### ############# ##
142023-12-01T20:13:57.626Z### ### ############ ##
152023-12-01T20:13:57.659Z## ### # ## ### ## Oxide Computer Company ##
162023-12-01T20:13:57.692Z## ## ## ### # ### ##
172023-12-01T20:13:57.725Z## # ### #### #### This Station Under ##
182023-12-01T20:13:57.759Z### ### #### # ### Computer Control ##
192023-12-01T20:13:57.792Z#### #### ### ## ##
202023-12-01T20:13:57.826Z##################################################
212023-12-01T20:13:57.859ZHostname: sock
222023-12-01T20:13:57.892ZLoading smf(7) serice descriptions:
232023-12-01T20:13:58.795Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-7a35cfb1c9365afc.gz
242023-12-01T20:14:00.188Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-7a35cfb1c9365afc.gz
252023-12-01T20:14:00.188Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-01T20:14:19.769Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-01T20:14:19.800Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-be479211b7b081a6.gz
282023-12-01T20:14:37.721Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-be479211b7b081a6.gz
292023-12-01T20:14:37.747Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-01T20:14:38.705Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-01T20:14:38.755Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-01T20:14:42.606Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-01T20:14:42.631Zdownloading input: /input/package/work/version.txt
342023-12-01T20:14:42.655Zdownloaded input: /input/package/work/version.txt
352023-12-01T20:14:42.655Zdownloading input: /input/package/work/package.tar.gz
362023-12-01T20:15:01.560Zdownloaded input: /input/package/work/package.tar.gz
372023-12-01T20:15:01.560Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-01T20:15:28.389Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-01T20:15:28.390Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-01T20:15:35.119Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-01T20:15:35.155Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-01T20:16:17.026Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-01T20:16:17.026Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-01T20:16:49.993Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-01T20:16:50.017Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-01T20:17:13.205Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-01T20:17:13.205Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-01T20:17:17.319Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-01T20:17:17.319Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-01T20:17:26.220Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-01T20:17:26.245Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-01T20:17:30.646Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-01T20:17:30.646Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-01T20:17:35.335Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-01T20:17:35.359Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-01T20:17:35.409Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-01T20:17:35.409Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-01T20:17:38.762Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-01T20:17:38.786Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-01T20:17:42.024Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-01T20:17:42.024Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-01T20:17:54.903Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-01T20:17:54.903Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-01T20:18:08.201Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-01T20:18:08.201Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-01T20:18:08.238Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-01T20:18:08.238Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-01T20:18:14.280Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-01T20:18:14.305Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-01T20:18:25.983Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-01T20:18:25.983Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-01T20:19:08.819Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-01T20:19:08.844Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-01T20:19:38.381Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-01T20:19:38.406Zstarting task 0: "setup"
762023-12-01T20:19:38.408Z++ uname -s
772023-12-01T20:19:38.408Z+ kern=SunOS
782023-12-01T20:19:38.408Z+ case "$kern" in
792023-12-01T20:19:38.408Z+ groupadd -g 12345 build
802023-12-01T20:19:38.408Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-01T20:19:40.399Z+ zfs create -o mountpoint=/work rpool/work
822023-12-01T20:19:40.437Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-01T20:19:40.461Z+ home_fs=zfs
842023-12-01T20:19:40.461Z+ [[ zfs == autofs ]]
852023-12-01T20:19:40.461Z+ mkdir -p /home/build
862023-12-01T20:19:40.461Z+ chown build:build /home/build /work
872023-12-01T20:19:42.445Z+ chmod 0700 /home/build /work
882023-12-01T20:19:42.469Zprocess exited: duration 4067 ms, exit code 0
 
892023-12-01T20:19:42.519Zstarting task 1: "authentication"
902023-12-01T20:19:42.562Zprocess exited: duration 43 ms, exit code 0
 
912023-12-01T20:19:42.612Zstarting task 2: "build"
922023-12-01T20:19:42.636Z+ trap _exit_trap EXIT
932023-12-01T20:19:42.636Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-01T20:19:42.817Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-01T20:19:42.817Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-01T20:19:42.817Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-01T20:19:42.817Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-01T20:19:42.817Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-01T20:19:42.817Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-01T20:19:42.817Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-01T20:19:42.817Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-01T20:19:42.842Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-01T20:19:42.842Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-01T20:19:42.842Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-01T20:19:42.842Z+ [[ -d /opt/oxide ]]
1062023-12-01T20:19:42.842Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-01T20:19:42.842Z+ pfexec mkdir /opt/oxide
1082023-12-01T20:19:42.842Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-01T20:19:42.842Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-01T20:19:42.842Z+ cd /opt/oxide-underneath
1112023-12-01T20:19:42.842Z+ pfexec tar ceEp@/f - .
1122023-12-01T20:19:42.842Z+ cd /opt/oxide
1132023-12-01T20:19:42.842Z+ pfexec tar xveEp@/f -
1142023-12-01T20:19:42.942Zx ., 0 bytes, 0 tape blocks
1152023-12-01T20:19:42.942Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-01T20:19:42.942Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-01T20:19:42.942Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-12-01T20:19:42.967Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-01T20:19:42.967Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-01T20:19:42.992Z+ pfexec mkdir /opt/oxide/work
1212023-12-01T20:19:42.992Z+ pfexec chown build:build /opt/oxide/work
1222023-12-01T20:19:42.992Z+ cd /opt/oxide/work
1232023-12-01T20:19:42.992Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-01T20:19:44.526ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-12-01T20:19:44.526Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-12-01T20:19:44.526Zx out/omicron-sled-agent.tar, 111069184 bytes, 216932 tape blocks
1272023-12-01T20:19:44.526Zx out/target/test, 75 bytes, 1 tape blocks
1282023-12-01T20:19:44.526Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-12-01T20:19:44.526Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-12-01T20:19:44.526Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-12-01T20:19:44.526Zx target/release/omicron-package, 42577472 bytes, 83160 tape blocks
1322023-12-01T20:19:44.526Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-12-01T20:19:44.526Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-12-01T20:19:44.526Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-12-01T20:19:44.528Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-12-01T20:19:44.528Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-12-01T20:19:44.528Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-12-01T20:19:44.528Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-12-01T20:19:44.553Z
1402023-12-01T20:19:44.553Zreal 1.542935096
1412023-12-01T20:19:44.553Zuser 1.382478840
1422023-12-01T20:19:44.553Zsys 0.279408984
1432023-12-01T20:19:44.553Ztrap 0.000068238
1442023-12-01T20:19:44.553Ztflt 0.000000000
1452023-12-01T20:19:44.553Zdflt 0.000000000
1462023-12-01T20:19:44.555Zkflt 0.000000000
1472023-12-01T20:19:44.555Zlock 0.000000000
1482023-12-01T20:19:44.555Zslp 1.388426346
1492023-12-01T20:19:44.555Zlat 0.033274711
1502023-12-01T20:19:44.555Zstop 0.000121251
1512023-12-01T20:19:44.555Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-12-01T20:19:45.673Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-12-01T20:19:45.697Z+ mkdir tests
1542023-12-01T20:19:45.697Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-12-01T20:19:45.697Z+ ptime -m gunzip
1562023-12-01T20:19:45.697Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-7a35cfb1c9365afc
1572023-12-01T20:19:45.836Z
1582023-12-01T20:19:45.861Zreal 0.141162996
1592023-12-01T20:19:45.861Zuser 0.128118719
1602023-12-01T20:19:45.861Zsys 0.012796202
1612023-12-01T20:19:45.861Ztrap 0.000014650
1622023-12-01T20:19:45.861Ztflt 0.000000000
1632023-12-01T20:19:45.861Zdflt 0.000000000
1642023-12-01T20:19:45.861Zkflt 0.000000000
1652023-12-01T20:19:45.861Zlock 0.000000000
1662023-12-01T20:19:45.861Zslp 0.000018090
1672023-12-01T20:19:45.861Zlat 0.000021720
1682023-12-01T20:19:45.861Zstop 0.000125842
1692023-12-01T20:19:45.861Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-7a35cfb1c9365afc
1702023-12-01T20:19:45.861Z+ chmod a+x tests/bootstrap-7a35cfb1c9365afc
1712023-12-01T20:19:45.861Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-12-01T20:19:45.861Z+ ptime -m gunzip
1732023-12-01T20:19:45.861Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-12-01T20:19:48.058Z
1752023-12-01T20:19:48.083Zreal 2.211517619
1762023-12-01T20:19:48.083Zuser 2.050644931
1772023-12-01T20:19:48.083Zsys 0.160532032
1782023-12-01T20:19:48.083Ztrap 0.000138682
1792023-12-01T20:19:48.083Ztflt 0.000000000
1802023-12-01T20:19:48.083Zdflt 0.000000000
1812023-12-01T20:19:48.083Zkflt 0.000000000
1822023-12-01T20:19:48.083Zlock 0.000000000
1832023-12-01T20:19:48.083Zslp 0.000019180
1842023-12-01T20:19:48.083Zlat 0.000015749
1852023-12-01T20:19:48.083Zstop 0.000114482
1862023-12-01T20:19:48.083Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-12-01T20:19:48.083Z+ chmod a+x tests/bootstrap
1882023-12-01T20:19:48.083Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-12-01T20:19:48.083Z+ ptime -m gunzip
1902023-12-01T20:19:48.083Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-be479211b7b081a6
1912023-12-01T20:19:50.320Z
1922023-12-01T20:19:50.345Zreal 2.250312167
1932023-12-01T20:19:50.345Zuser 2.115622156
1942023-12-01T20:19:50.345Zsys 0.134279724
1952023-12-01T20:19:50.345Ztrap 0.000160672
1962023-12-01T20:19:50.345Ztflt 0.000000000
1972023-12-01T20:19:50.345Zdflt 0.000000000
1982023-12-01T20:19:50.345Zkflt 0.000000000
1992023-12-01T20:19:50.345Zlock 0.000000000
2002023-12-01T20:19:50.345Zslp 0.000021780
2012023-12-01T20:19:50.345Zlat 0.000022509
2022023-12-01T20:19:50.345Zstop 0.000128812
2032023-12-01T20:19:50.345Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-be479211b7b081a6
2042023-12-01T20:19:50.345Z+ chmod a+x tests/end_to_end_tests-be479211b7b081a6
2052023-12-01T20:19:50.345Z+ bmat address ls
2062023-12-01T20:19:50.345ZNAME CIDR FIRST LAST COUNT
2072023-12-01T20:19:50.345Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-12-01T20:19:50.345Z++ bmat address ls -f extra -Ho first,last,count
2092023-12-01T20:19:50.370Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-12-01T20:19:50.370Z+ EXTRA_IP_START=10.151.2.164
2112023-12-01T20:19:50.370Z+ EXTRA_IP_END=10.151.2.227
2122023-12-01T20:19:50.370Z+ EXTRA_IP_COUNT=64
2132023-12-01T20:19:50.370Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-12-01T20:19:50.370Z+ EXTRA_IP_BASE=10.151.2
2152023-12-01T20:19:50.370Z+ EXTRA_IP_FOCTET=164
2162023-12-01T20:19:50.370Z+ EXTRA_IP_LOCTET=227
2172023-12-01T20:19:50.370Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-12-01T20:19:50.370Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-12-01T20:19:50.370Z+ DNS_IP1=10.151.2.164
2202023-12-01T20:19:50.370Z+ DNS_IP2=10.151.2.165
2212023-12-01T20:19:50.370Z+ UPLINK_IP=10.151.2.174
2222023-12-01T20:19:50.370Z+ IPPOOL_START=10.151.2.175
2232023-12-01T20:19:50.370Z+ IPPOOL_END=10.151.2.227
2242023-12-01T20:19:50.370Z++ ipadm show-addr -po type,addr
2252023-12-01T20:19:50.370Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-12-01T20:19:50.370Z+ GATEWAY_IP=10.151.2.100
2272023-12-01T20:19:50.370Z+ [[ -n 10.151.2.100 ]]
2282023-12-01T20:19:50.370Z+ ping -s 10.151.2.100 56 1
2292023-12-01T20:19:50.370ZPING 10.151.2.100: 56 data bytes
2302023-12-01T20:19:50.370Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.012 ms
2312023-12-01T20:19:50.370Z
2322023-12-01T20:19:50.370Z----10.151.2.100 PING Statistics----
2332023-12-01T20:19:50.370Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-12-01T20:19:50.370Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2352023-12-01T20:19:50.370Z++ arp -an
2362023-12-01T20:19:50.370Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-12-01T20:19:50.395Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-12-01T20:19:50.395Z+ routeadm -e ipv4-forwarding -u
2392023-12-01T20:19:50.482Z+ PXA_START=10.151.2.164
2402023-12-01T20:19:50.482Z+ PXA_END=10.151.2.227
2412023-12-01T20:19:50.506Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-12-01T20:19:50.506Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-12-01T20:19:50.619Z+ ZPOOL_VDEV_DIR=/scratch
2442023-12-01T20:19:50.619Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-12-01T20:19:50.644Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-12-01T20:19:50.644Z++ cd ./tools
2472023-12-01T20:19:50.644Z++ pwd
2482023-12-01T20:19:50.644Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-12-01T20:19:50.644Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-12-01T20:19:50.644Z+ SOFTNPU_MODE=zone
2512023-12-01T20:19:50.644Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-12-01T20:19:50.644Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-12-01T20:19:50.644Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-12-01T20:19:50.644Z++ dladm show-phys -p -o LINK
2552023-12-01T20:19:50.644Z++ head -1
2562023-12-01T20:19:50.703Z+ PHYSICAL_LINK=igb0
2572023-12-01T20:19:50.728Z+ echo 'Using igb0 as physical link'
2582023-12-01T20:19:50.728ZUsing igb0 as physical link
2592023-12-01T20:19:50.728Z+ ensure_run_as_root
2602023-12-01T20:19:50.728Z++ id -u
2612023-12-01T20:19:50.728Z+ [[ 0 -ne 0 ]]
2622023-12-01T20:19:50.728Z+ ensure_zpools
2632023-12-01T20:19:50.728Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-12-01T20:19:50.728Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-12-01T20:19:50.728Z+ readarray -t ZPOOLS
2662023-12-01T20:19:50.728Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-12-01T20:19:50.728Z++ sed 's/[ ",]//g'
2682023-12-01T20:19:50.728Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-12-01T20:19:50.728Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-12-01T20:19:50.728ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-12-01T20:19:50.728Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-12-01T20:19:50.728Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-12-01T20:19:50.729Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-12-01T20:19:50.753Z0+0 records in
2752023-12-01T20:19:50.753Z0+0 records out
2762023-12-01T20:19:50.753Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2772023-12-01T20:19:50.753Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-12-01T20:19:50.753Z+ set +x
2792023-12-01T20:19:50.753ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-12-01T20:19:50.753Z++ zpool list -o name
2812023-12-01T20:19:50.753Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-12-01T20:19:50.778Z+ [[ -z '' ]]
2832023-12-01T20:19:50.778Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-12-01T20:19:50.803Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-12-01T20:19:50.803Z+ set +x
2862023-12-01T20:19:50.803ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-12-01T20:19:50.803Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-12-01T20:19:50.803Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-12-01T20:19:50.803ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-12-01T20:19:50.803Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-12-01T20:19:50.804Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-12-01T20:19:50.804Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-12-01T20:19:50.804Z0+0 records in
2942023-12-01T20:19:50.804Z0+0 records out
2952023-12-01T20:19:50.805Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-12-01T20:19:50.805Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-12-01T20:19:50.805Z+ set +x
2982023-12-01T20:19:50.805ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-12-01T20:19:50.805Z++ zpool list -o name
3002023-12-01T20:19:50.805Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-12-01T20:19:50.829Z+ [[ -z '' ]]
3022023-12-01T20:19:50.829Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-12-01T20:19:50.854Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-12-01T20:19:50.854Z+ set +x
3052023-12-01T20:19:50.854ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-12-01T20:19:50.854Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-12-01T20:19:50.854Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-12-01T20:19:50.854ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3092023-12-01T20:19:50.854Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-12-01T20:19:50.854Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-12-01T20:19:50.854Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-12-01T20:19:50.854Z0+0 records in
3132023-12-01T20:19:50.854Z0+0 records out
3142023-12-01T20:19:50.854Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3152023-12-01T20:19:50.854Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-12-01T20:19:50.854Z+ set +x
3172023-12-01T20:19:50.854ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-12-01T20:19:50.854Z++ zpool list -o name
3192023-12-01T20:19:50.854Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-12-01T20:19:50.854Z+ [[ -z '' ]]
3212023-12-01T20:19:50.855Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-12-01T20:19:50.879Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-12-01T20:19:50.879Z+ set +x
3242023-12-01T20:19:50.879ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-12-01T20:19:50.879Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-12-01T20:19:50.879Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-12-01T20:19:50.879ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-12-01T20:19:50.879Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-12-01T20:19:50.879Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-12-01T20:19:50.879Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-12-01T20:19:50.879Z0+0 records in
3322023-12-01T20:19:50.879Z0+0 records out
3332023-12-01T20:19:50.879Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3342023-12-01T20:19:50.879Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-12-01T20:19:50.879Z+ set +x
3362023-12-01T20:19:50.879ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-12-01T20:19:50.904Z++ zpool list -o name
3382023-12-01T20:19:50.904Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-12-01T20:19:50.904Z+ [[ -z '' ]]
3402023-12-01T20:19:50.904Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-12-01T20:19:50.929Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-12-01T20:19:50.929Z+ set +x
3432023-12-01T20:19:50.929ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-12-01T20:19:50.929Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-12-01T20:19:50.929Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-12-01T20:19:50.929ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-12-01T20:19:50.929Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-12-01T20:19:50.929Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-12-01T20:19:50.929Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-12-01T20:19:50.929Z0+0 records in
3512023-12-01T20:19:50.929Z0+0 records out
3522023-12-01T20:19:50.929Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3532023-12-01T20:19:50.929Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-12-01T20:19:50.929Z+ set +x
3552023-12-01T20:19:50.929ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-12-01T20:19:50.929Z++ zpool list -o name
3572023-12-01T20:19:50.929Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-12-01T20:19:50.954Z+ [[ -z '' ]]
3592023-12-01T20:19:50.954Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-12-01T20:19:50.979Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-12-01T20:19:50.979Z+ set +x
3622023-12-01T20:19:50.979ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-12-01T20:19:50.979Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-12-01T20:19:50.979Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-12-01T20:19:50.979ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-12-01T20:19:50.979Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-12-01T20:19:50.979Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-12-01T20:19:50.979Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-12-01T20:19:50.979Z0+0 records in
3702023-12-01T20:19:50.980Z0+0 records out
3712023-12-01T20:19:50.980Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3722023-12-01T20:19:50.980Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-12-01T20:19:50.980Z+ set +x
3742023-12-01T20:19:50.980ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-12-01T20:19:50.980Z++ zpool list -o name
3762023-12-01T20:19:50.980Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-12-01T20:19:50.980Z+ [[ -z '' ]]
3782023-12-01T20:19:50.980Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-12-01T20:19:51.004Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-12-01T20:19:51.004Z+ set +x
3812023-12-01T20:19:51.004ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-12-01T20:19:51.004Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-12-01T20:19:51.004Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-12-01T20:19:51.005ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-12-01T20:19:51.006Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-12-01T20:19:51.006Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-12-01T20:19:51.006Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-12-01T20:19:51.030Z0+0 records in
3892023-12-01T20:19:51.030Z0+0 records out
3902023-12-01T20:19:51.030Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-12-01T20:19:51.030Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-12-01T20:19:51.030Z+ set +x
3932023-12-01T20:19:51.030ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-12-01T20:19:51.030Z++ zpool list -o name
3952023-12-01T20:19:51.032Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-12-01T20:19:51.032Z+ [[ -z '' ]]
3972023-12-01T20:19:51.032Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-12-01T20:19:51.056Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-12-01T20:19:51.056Z+ set +x
4002023-12-01T20:19:51.056ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-12-01T20:19:51.056Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-12-01T20:19:51.056Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-12-01T20:19:51.056ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-12-01T20:19:51.056Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-12-01T20:19:51.056Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-12-01T20:19:51.056Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-12-01T20:19:51.056Z0+0 records in
4082023-12-01T20:19:51.056Z0+0 records out
4092023-12-01T20:19:51.056Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-12-01T20:19:51.056Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-12-01T20:19:51.056Z+ set +x
4122023-12-01T20:19:51.056ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-12-01T20:19:51.056Z++ zpool list -o name
4142023-12-01T20:19:51.056Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-12-01T20:19:51.081Z+ [[ -z '' ]]
4162023-12-01T20:19:51.081Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-12-01T20:19:51.105Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-12-01T20:19:51.105Z+ set +x
4192023-12-01T20:19:51.105ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-12-01T20:19:51.105Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-12-01T20:19:51.105Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-12-01T20:19:51.105ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-12-01T20:19:51.105Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-12-01T20:19:51.106Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-12-01T20:19:51.106Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-12-01T20:19:51.106Z0+0 records in
4272023-12-01T20:19:51.106Z0+0 records out
4282023-12-01T20:19:51.106Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4292023-12-01T20:19:51.106Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-12-01T20:19:51.106Z+ set +x
4312023-12-01T20:19:51.106ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-12-01T20:19:51.106Z++ zpool list -o name
4332023-12-01T20:19:51.106Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-12-01T20:19:51.106Z+ [[ -z '' ]]
4352023-12-01T20:19:51.106Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-12-01T20:19:51.131Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-12-01T20:19:51.131Z+ set +x
4382023-12-01T20:19:51.156ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-12-01T20:19:51.156Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-12-01T20:19:51.156Z+ readarray -t ZPOOLS
4412023-12-01T20:19:51.156Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-12-01T20:19:51.156Z++ sed 's/[ ",]//g'
4432023-12-01T20:19:51.156Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-12-01T20:19:51.156Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-12-01T20:19:51.157ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-12-01T20:19:51.157Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-12-01T20:19:51.157Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-12-01T20:19:51.157Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-12-01T20:19:51.157Z0+0 records in
4502023-12-01T20:19:51.157Z0+0 records out
4512023-12-01T20:19:51.157Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4522023-12-01T20:19:51.157Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-12-01T20:19:51.157Z+ set +x
4542023-12-01T20:19:51.157ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-12-01T20:19:51.157Z++ zpool list -o name
4562023-12-01T20:19:51.157Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-12-01T20:19:51.157Z+ [[ -z '' ]]
4582023-12-01T20:19:51.157Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-12-01T20:19:51.182Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-12-01T20:19:51.182Z+ set +x
4612023-12-01T20:19:51.182ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-12-01T20:19:51.182Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-12-01T20:19:51.182Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-12-01T20:19:51.182ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-12-01T20:19:51.182Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-12-01T20:19:51.182Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-12-01T20:19:51.182Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-12-01T20:19:51.206Z0+0 records in
4692023-12-01T20:19:51.207Z0+0 records out
4702023-12-01T20:19:51.208Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4712023-12-01T20:19:51.208Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-12-01T20:19:51.208Z+ set +x
4732023-12-01T20:19:51.208ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-12-01T20:19:51.208Z++ zpool list -o name
4752023-12-01T20:19:51.208Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-12-01T20:19:51.208Z+ [[ -z '' ]]
4772023-12-01T20:19:51.208Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-12-01T20:19:51.233Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-12-01T20:19:51.233Z+ set +x
4802023-12-01T20:19:51.233ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-12-01T20:19:51.233Z+ [[ zone == \z\o\n\e ]]
4822023-12-01T20:19:51.233Z+ ensure_simulated_links igb0
4832023-12-01T20:19:51.233Z+ local PHYSICAL_LINK=igb0
4842023-12-01T20:19:51.233Z+ INDICES=("0" "1")
4852023-12-01T20:19:51.233Z+ for I in "${INDICES[@]}"
4862023-12-01T20:19:51.233Z++ get_simnet_name_if_exists net0
4872023-12-01T20:19:51.233Z++ dladm show-simnet -p -o LINK net0
4882023-12-01T20:19:51.258Z++ echo ''
4892023-12-01T20:19:51.258Z+ [[ -z '' ]]
4902023-12-01T20:19:51.258Z+ dladm create-simnet -t net0
4912023-12-01T20:19:51.258Z+ dladm create-simnet -t sc0_0
4922023-12-01T20:19:51.258Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-12-01T20:19:51.258Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-12-01T20:19:51.258Z+ success 'Simnet net0/sc0_0 exists'
4952023-12-01T20:19:51.258Z+ set +x
4962023-12-01T20:19:51.258ZSimnet net0/sc0_0 exists
4972023-12-01T20:19:51.258Z+ for I in "${INDICES[@]}"
4982023-12-01T20:19:51.258Z++ get_simnet_name_if_exists net1
4992023-12-01T20:19:51.258Z++ dladm show-simnet -p -o LINK net1
5002023-12-01T20:19:51.283Z++ echo ''
5012023-12-01T20:19:51.283Z+ [[ -z '' ]]
5022023-12-01T20:19:51.283Z+ dladm create-simnet -t net1
5032023-12-01T20:19:51.283Z+ dladm create-simnet -t sc1_0
5042023-12-01T20:19:51.283Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-12-01T20:19:51.283Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-12-01T20:19:51.283Z+ success 'Simnet net1/sc1_0 exists'
5072023-12-01T20:19:51.283Z+ set +x
5082023-12-01T20:19:51.283ZSimnet net1/sc1_0 exists
5092023-12-01T20:19:51.283Z++ get_vnic_name_if_exists sc0_1
5102023-12-01T20:19:51.283Z++ dladm show-vnic -p -o LINK sc0_1
5112023-12-01T20:19:51.283Z++ echo ''
5122023-12-01T20:19:51.283Z+ [[ -z '' ]]
5132023-12-01T20:19:51.283Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-12-01T20:19:51.308Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-12-01T20:19:51.308Z+ success 'Vnic sc0_1 exists'
5162023-12-01T20:19:51.308Z+ set +x
5172023-12-01T20:19:51.308ZVnic sc0_1 exists
5182023-12-01T20:19:51.308Z+ warn_if_no_proxy_arp
5192023-12-01T20:19:51.308Z+ [[ -v PXA_START ]]
5202023-12-01T20:19:51.308Z+ [[ -v PXA_END ]]
5212023-12-01T20:19:51.308Z+ ensure_softnpu_zone
5222023-12-01T20:19:51.308Z+ zoneadm list
5232023-12-01T20:19:51.308Z+ grep -q sidecar_softnpu
5242023-12-01T20:19:51.308Z+ [[ -f out/npuzone/npuzone ]]
5252023-12-01T20:19:51.308Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5262023-12-01T20:19:51.332Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
5272023-12-01T20:20:01.265Zdownloading softnpu ...done
5282023-12-01T20:20:01.289Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5292023-12-01T20:20:07.522Zdownloading asic_program.so ...done
5302023-12-01T20:20:07.549Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-12-01T20:20:12.752Zdownloading scadm ...done
5322023-12-01T20:20:12.777Zcreating port tfportrear0_0 for sc0_0
5332023-12-01T20:20:12.777Zcreating port tfportqsfp0_0 for sc0_1
5342023-12-01T20:20:12.777Zsoftnpu zone setup
5352023-12-01T20:20:12.777Zconfigure zone
5362023-12-01T20:20:12.801Zinstall zone
5372023-12-01T20:20:13.814Zboot zone
5382023-12-01T20:20:16.925Z[sidecar_softnpu] svcs milestone/network
5392023-12-01T20:20:16.952ZSTATE STIME FMRI
5402023-12-01T20:20:16.952Zonline 20:20:14 svc:/milestone/network:default
5412023-12-01T20:20:16.952Z[sidecar_softnpu] /softnpu/npu start
5422023-12-01T20:20:16.978Z
5432023-12-01T20:20:17.003Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-12-01T20:20:17.003Z+ GATEWAY_IP=10.151.2.100
5452023-12-01T20:20:17.003Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-12-01T20:20:17.003ZUsing 10.151.2.100 as gateway ip
5472023-12-01T20:20:17.003Z+ [[ ! -v GATEWAY_MAC ]]
5482023-12-01T20:20:17.003Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-12-01T20:20:17.003Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-12-01T20:20:17.003Z+++ echo 18:c0:4d:0d:a0:2a
5512023-12-01T20:20:17.003Z+ COUNT=1
5522023-12-01T20:20:17.003Z+ [[ 1 -ne 1 ]]
5532023-12-01T20:20:17.003Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-12-01T20:20:17.003ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-12-01T20:20:17.003Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-12-01T20:20:17.003Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-12-01T20:20:17.028Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-12-01T20:20:17.029Z+ [[ -v PXA_START ]]
5592023-12-01T20:20:17.029Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-12-01T20:20:17.029Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-12-01T20:20:17.058Z+ z_scadm dump-state
5622023-12-01T20:20:17.058Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-12-01T20:20:17.083Zlocal v6:
5642023-12-01T20:20:17.083Zlocal v4:
5652023-12-01T20:20:17.083Zrouter v6:
5662023-12-01T20:20:17.083Zrouter v4_idx:
5672023-12-01T20:20:17.083Zrouter v4_routes:
5682023-12-01T20:20:17.083Zresolver v4:
5692023-12-01T20:20:17.083Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-12-01T20:20:17.084Zresolver v6:
5712023-12-01T20:20:17.084Znat v4:
5722023-12-01T20:20:17.084Znat_v6:
5732023-12-01T20:20:17.084Zport_mac:
5742023-12-01T20:20:17.084Zproxy arp:
5752023-12-01T20:20:17.084Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-12-01T20:20:17.084Z+ success 'softnpu zone exists'
5772023-12-01T20:20:17.084Z+ set +x
5782023-12-01T20:20:17.084Zsoftnpu zone exists
5792023-12-01T20:20:17.084Z
5802023-12-01T20:20:17.084Zreal 26.455534037
5812023-12-01T20:20:17.084Zuser 0.732436546
5822023-12-01T20:20:17.084Zsys 1.626332400
5832023-12-01T20:20:17.084Ztrap 0.000071627
5842023-12-01T20:20:17.084Ztflt 0.000120669
5852023-12-01T20:20:17.084Zdflt 0.001376345
5862023-12-01T20:20:17.084Zkflt 0.000000000
5872023-12-01T20:20:17.085Zlock 20:29.745131592
5882023-12-01T20:20:17.085Zslp 1:19.211594082
5892023-12-01T20:20:17.085Zlat 0.084174525
5902023-12-01T20:20:17.085Zstop 0.364997955
5912023-12-01T20:20:17.085Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-12-01T20:20:17.115Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-12-01T20:20:17.115Z+ SILO_NAME=recovery
5942023-12-01T20:20:17.142Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-01T20:20:17.142Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-12-01T20:20:17.142Z+ sed -i~ '
5972023-12-01T20:20:17.142Z /^external_dns_ips/c\
5982023-12-01T20:20:17.142Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-12-01T20:20:17.142Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-12-01T20:20:17.142Z /^first/c\
6012023-12-01T20:20:17.142Zfirst = "10.151.2.164"
6022023-12-01T20:20:17.142Z /^last/c\
6032023-12-01T20:20:17.142Zlast = "10.151.2.173"
6042023-12-01T20:20:17.142Z }
6052023-12-01T20:20:17.142Z /^\[rack_network_config/,/^$/ {
6062023-12-01T20:20:17.142Z /^infra_ip_first/c\
6072023-12-01T20:20:17.142Zinfra_ip_first = "10.151.2.174"
6082023-12-01T20:20:17.142Z /^infra_ip_last/c\
6092023-12-01T20:20:17.142Zinfra_ip_last = "10.151.2.174"
6102023-12-01T20:20:17.142Z }
6112023-12-01T20:20:17.142Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-12-01T20:20:17.142Z /^routes/c\
6132023-12-01T20:20:17.142Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6142023-12-01T20:20:17.142Z /^addresses/c\
6152023-12-01T20:20:17.142Zaddresses = \["10.151.2.174/32"\]
6162023-12-01T20:20:17.144Z }
6172023-12-01T20:20:17.144Z' pkg/config-rss.toml
6182023-12-01T20:20:17.144Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-12-01T20:20:17.144Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-12-01T20:20:17.144Z+++ pkg/config-rss.toml Fri Dec 1 20:20:16 2023
6212023-12-01T20:20:17.144Z@@ -41,7 +41,7 @@
6222023-12-01T20:20:17.144Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-12-01T20:20:17.144Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-12-01T20:20:17.144Z # below.
6252023-12-01T20:20:17.144Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-12-01T20:20:17.144Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6272023-12-01T20:20:17.144Z
6282023-12-01T20:20:17.144Z # Initial TLS certificates for the external API
6292023-12-01T20:20:17.144Z #
6302023-12-01T20:20:17.144Z@@ -77,8 +77,8 @@
6312023-12-01T20:20:17.144Z #
6322023-12-01T20:20:17.144Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-12-01T20:20:17.144Z [[internal_services_ip_pool_ranges]]
6342023-12-01T20:20:17.144Z-first = "192.168.1.20"
6352023-12-01T20:20:17.144Z-last = "192.168.1.29"
6362023-12-01T20:20:17.144Z+first = "10.151.2.164"
6372023-12-01T20:20:17.144Z+last = "10.151.2.173"
6382023-12-01T20:20:17.144Z
6392023-12-01T20:20:17.144Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-12-01T20:20:17.144Z # parameters should be provided to the control plane via wicket, but we need to
6412023-12-01T20:20:17.144Z@@ -92,8 +92,8 @@
6422023-12-01T20:20:17.144Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-12-01T20:20:17.145Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-12-01T20:20:17.145Z # With softnpu, only one address is used.
6452023-12-01T20:20:17.145Z-infra_ip_first = "192.168.1.30"
6462023-12-01T20:20:17.145Z-infra_ip_last = "192.168.1.30"
6472023-12-01T20:20:17.145Z+infra_ip_first = "10.151.2.174"
6482023-12-01T20:20:17.145Z+infra_ip_last = "10.151.2.174"
6492023-12-01T20:20:17.145Z
6502023-12-01T20:20:17.145Z # Configurations for BGP routers to run on the scrimlets.
6512023-12-01T20:20:17.145Z bgp = []
6522023-12-01T20:20:17.145Z@@ -101,9 +101,9 @@
6532023-12-01T20:20:17.145Z # You can configure multiple uplinks by repeating the following stanza
6542023-12-01T20:20:17.145Z [[rack_network_config.ports]]
6552023-12-01T20:20:17.145Z # Routes associated with this port.
6562023-12-01T20:20:17.145Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-12-01T20:20:17.145Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6582023-12-01T20:20:17.145Z # Addresses associated with this port.
6592023-12-01T20:20:17.145Z-addresses = ["192.168.1.30/32"]
6602023-12-01T20:20:17.145Z+addresses = ["10.151.2.174/32"]
6612023-12-01T20:20:17.145Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-12-01T20:20:17.145Z port = "qsfp0"
6632023-12-01T20:20:17.146Z # The speed of this port.
6642023-12-01T20:20:17.146Z+ true
6652023-12-01T20:20:17.146Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-12-01T20:20:17.146Za pkg/config-rss.toml 7K
6672023-12-01T20:20:17.146Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-12-01T20:20:17.146Z+ TLS_NAME=recovery.sys.oxide.test
6692023-12-01T20:20:17.147Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-12-01T20:20:17.283Z............+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+.......+........+.+.....+......+.+...............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+........+.+......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-12-01T20:20:18.642Z....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+...+......+.+.....+..................+..........+......+..+...+....+...+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+..+..........+.....+......+...+.......+..+.+.........+.....+......+.+..............+......+.+............+...+....................+.........+....+..............+.+......+.................................+........+.......+...+...+.........+........+....+...........+....+...........+.........+....+.....+..........+..............................+..+...+............+.........+....+............+.....+..................+......+...+.+.....+....+...............+...+.....+...+....+.....+.+......+...+......+......+...............+..............................+......+..+......+.......+..................+...............+..+...+............+.......+..+...+......+......+...+.......+.....+......+.............+.....................+.....+...+......+....+.....+.+.........+..+.........+............+....+...+........+...+............+.........+.+..+............+....+...............+..+..........+..+.......+...........+....+..............+...............+........................+.......+...+...............+..+.+.........+......+...+.....+...............+....+........+...........................+..........+...............+...+..+.......+.....+.+........+............+....+..............+.......+.....+.......+......+..+...+.......+..+..........+...+........+.+.........+..+.........+.............+...........+.......+..+...+....+...+..+..........+........+...+....+......+...+......+.........+..+.........+.+...............+.................+.+......+.............................+....+.................+.+.....+...+.......+.......................+....+..+.............+........+..........+............+...+........+........................+.......+..................+..+......+.........+....+.................+.......+...+..+.+...........+...+............+..........+...+...+........+...........................+...+.+......+..+.......+.........+......+........+.+..+.......+...+..+..........+..+.........+.......+.....+............+...+.......+........+.......+....................+.+......+...............+.....+......+.......+...+..................+.....+.......+..............................+........+.+.....+.+........+.+...+..+.+........+....+.....+...+............+.......+...+...+.................................+.....+............+...+....+......+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-01T20:20:18.666Z-----
6732023-12-01T20:20:18.666Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-12-01T20:20:18.691Za pkg/initial-tls-cert.pem 2K
6752023-12-01T20:20:18.691Za pkg/initial-tls-key.pem 4K
6762023-12-01T20:20:18.691Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-12-01T20:20:18.691Z+ rmdir pkg
6782023-12-01T20:20:18.691Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-12-01T20:20:18.691Z+ pfexec mkdir -p /usr/oxide
6802023-12-01T20:20:18.691Z+ pfexec rm -f /usr/oxide/tcpproxy
6812023-12-01T20:20:18.692Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6822023-12-01T20:20:19.046Z+ pfexec chmod +x /usr/oxide/tcpproxy
6832023-12-01T20:20:19.070Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6842023-12-01T20:20:19.070Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6852023-12-01T20:20:19.070Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6862023-12-01T20:20:19.167Z+ OMICRON_NO_UNINSTALL=1
6872023-12-01T20:20:19.167Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6882023-12-01T20:20:19.191ZDec 01 20:20:18.660 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6892023-12-01T20:20:19.191ZDec 01 20:20:18.662 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6902023-12-01T20:20:19.193ZDec 01 20:20:18.662 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6912023-12-01T20:20:19.193ZDec 01 20:20:18.662 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6922023-12-01T20:20:19.193ZDec 01 20:20:18.662 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6932023-12-01T20:20:19.193ZDec 01 20:20:18.662 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6942023-12-01T20:20:19.193ZDec 01 20:20:18.662 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6952023-12-01T20:20:19.193ZDec 01 20:20:18.662 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6962023-12-01T20:20:19.193ZDec 01 20:20:18.662 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6972023-12-01T20:20:19.193ZDec 01 20:20:18.662 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6982023-12-01T20:20:19.193ZDec 01 20:20:18.662 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6992023-12-01T20:20:19.193ZDec 01 20:20:18.662 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7002023-12-01T20:20:19.193ZDec 01 20:20:18.663 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
7012023-12-01T20:20:19.193ZDec 01 20:20:18.663 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7022023-12-01T20:20:19.193ZDec 01 20:20:18.663 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7032023-12-01T20:20:19.193ZDec 01 20:20:18.663 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7042023-12-01T20:20:19.193ZDec 01 20:20:18.663 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7052023-12-01T20:20:19.417ZDec 01 20:20:18.896 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-12-01T20:20:19.453ZDec 01 20:20:18.932 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-12-01T20:20:19.514ZDec 01 20:20:18.993 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-12-01T20:20:19.643Z
7092023-12-01T20:20:19.668Zreal 0.471702752
7102023-12-01T20:20:19.668Zuser 0.097793827
7112023-12-01T20:20:19.668Zsys 1.853605801
7122023-12-01T20:20:19.668Ztrap 0.000001740
7132023-12-01T20:20:19.668Ztflt 0.000891885
7142023-12-01T20:20:19.668Zdflt 0.000486543
7152023-12-01T20:20:19.668Zkflt 0.000013920
7162023-12-01T20:20:19.668Zlock 42.379307286
7172023-12-01T20:20:19.668Zslp 0.689527258
7182023-12-01T20:20:19.668Zlat 0.003175956
7192023-12-01T20:20:19.668Zstop 0.220739733
7202023-12-01T20:20:19.668Z+ retry=0
7212023-12-01T20:20:19.668Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-12-01T20:20:19.669Z+ [[ 0 -gt 30 ]]
7232023-12-01T20:20:19.669Z+ sleep 1
7242023-12-01T20:20:20.659Z+ retry=1
7252023-12-01T20:20:20.659Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-12-01T20:20:20.683Z+ [[ 1 -gt 30 ]]
7272023-12-01T20:20:20.683Z+ sleep 1
7282023-12-01T20:20:21.673Z+ retry=2
7292023-12-01T20:20:21.673Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-12-01T20:20:21.697Z+ [[ 2 -gt 30 ]]
7312023-12-01T20:20:21.697Z+ sleep 1
7322023-12-01T20:20:22.687Z+ retry=3
7332023-12-01T20:20:22.687Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-12-01T20:20:22.712Z+ [[ 3 -gt 30 ]]
7352023-12-01T20:20:22.712Z+ sleep 1
7362023-12-01T20:20:23.701Z+ retry=4
7372023-12-01T20:20:23.702Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-12-01T20:20:43.616Z+ [[ 4 -gt 30 ]]
7392023-12-01T20:20:43.616Z+ sleep 1
7402023-12-01T20:20:44.619Z+ retry=5
7412023-12-01T20:20:44.619Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7422023-12-01T20:20:44.644Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7432023-12-01T20:20:44.669Zlocal v6:
7442023-12-01T20:20:44.669Zfe80::4cf9:5eff:fe63:8a9
7452023-12-01T20:20:44.669Zfd00:99::1
7462023-12-01T20:20:44.669Zlocal v4:
7472023-12-01T20:20:44.669Z10.151.2.174
7482023-12-01T20:20:44.669Zrouter v6:
7492023-12-01T20:20:44.669Zrouter v4_idx:
7502023-12-01T20:20:44.669Z0.0.0.0/0 -> 0
7512023-12-01T20:20:44.669Zrouter v4_routes:
7522023-12-01T20:20:44.669Z0 -> 10.151.2.100 (2)
7532023-12-01T20:20:44.669Zresolver v4:
7542023-12-01T20:20:44.669Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7552023-12-01T20:20:44.669Zresolver v6:
7562023-12-01T20:20:44.669Zfe80::4cf9:5eff:fe63:8a9 -> 4e:f9:5e:63:08:a9
7572023-12-01T20:20:44.669Zfe80::e487:afff:fe8d:f75c -> e6:87:af:8d:f7:5c
7582023-12-01T20:20:44.669Znat v4:
7592023-12-01T20:20:44.669Znat_v6:
7602023-12-01T20:20:44.670Zport_mac:
7612023-12-01T20:20:44.670Z1: a8:40:25:75:96:d2
7622023-12-01T20:20:44.670Z1000: a8:40:25:75:96:d1
7632023-12-01T20:20:44.670Zproxy arp:
7642023-12-01T20:20:44.670Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7652023-12-01T20:20:44.670Z+ retry=0
7662023-12-01T20:20:44.670Z+++ zoneadm list -n
7672023-12-01T20:20:44.670Z+++ grep oxz_ntp
7682023-12-01T20:20:44.695Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-12-01T20:20:44.695Zsvcs: zone '-Hostate': entity not found
7702023-12-01T20:20:44.695Z++ true
7712023-12-01T20:20:44.695Z+ [[ '' != online ]]
7722023-12-01T20:20:44.695Z+ [[ 0 -gt 60 ]]
7732023-12-01T20:20:44.695Z+ sleep 1
7742023-12-01T20:20:45.680Z+ retry=1
7752023-12-01T20:20:45.705Z+++ zoneadm list -n
7762023-12-01T20:20:45.705Z+++ grep oxz_ntp
7772023-12-01T20:20:45.705Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-12-01T20:20:45.705Zsvcs: zone '-Hostate': entity not found
7792023-12-01T20:20:45.705Z++ true
7802023-12-01T20:20:45.705Z+ [[ '' != online ]]
7812023-12-01T20:20:45.705Z+ [[ 1 -gt 60 ]]
7822023-12-01T20:20:45.705Z+ sleep 1
7832023-12-01T20:20:46.704Z+ retry=2
7842023-12-01T20:20:46.729Z+++ zoneadm list -n
7852023-12-01T20:20:46.730Z+++ grep oxz_ntp
7862023-12-01T20:20:46.730Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-12-01T20:20:46.730Zsvcs: zone '-Hostate': entity not found
7882023-12-01T20:20:46.730Z++ true
7892023-12-01T20:20:46.730Z+ [[ '' != online ]]
7902023-12-01T20:20:46.730Z+ [[ 2 -gt 60 ]]
7912023-12-01T20:20:46.730Z+ sleep 1
7922023-12-01T20:20:47.726Z+ retry=3
7932023-12-01T20:20:47.751Z+++ zoneadm list -n
7942023-12-01T20:20:47.751Z+++ grep oxz_ntp
7952023-12-01T20:20:47.751Z++ pfexec svcs -z -Hostate oxide/ntp
7962023-12-01T20:20:47.751Zsvcs: zone '-Hostate': entity not found
7972023-12-01T20:20:47.751Z++ true
7982023-12-01T20:20:47.751Z+ [[ '' != online ]]
7992023-12-01T20:20:47.751Z+ [[ 3 -gt 60 ]]
8002023-12-01T20:20:47.751Z+ sleep 1
8012023-12-01T20:20:48.751Z+ retry=4
8022023-12-01T20:20:48.776Z+++ zoneadm list -n
8032023-12-01T20:20:48.776Z+++ grep oxz_ntp
8042023-12-01T20:20:48.776Z++ pfexec svcs -z -Hostate oxide/ntp
8052023-12-01T20:20:48.776Zsvcs: zone '-Hostate': entity not found
8062023-12-01T20:20:48.776Z++ true
8072023-12-01T20:20:48.776Z+ [[ '' != online ]]
8082023-12-01T20:20:48.776Z+ [[ 4 -gt 60 ]]
8092023-12-01T20:20:48.776Z+ sleep 1
8102023-12-01T20:20:49.771Z+ retry=5
8112023-12-01T20:20:49.796Z+++ zoneadm list -n
8122023-12-01T20:20:49.796Z+++ grep oxz_ntp
8132023-12-01T20:20:49.796Z++ pfexec svcs -z -Hostate oxide/ntp
8142023-12-01T20:20:49.796Zsvcs: zone '-Hostate': entity not found
8152023-12-01T20:20:49.796Z++ true
8162023-12-01T20:20:49.796Z+ [[ '' != online ]]
8172023-12-01T20:20:49.796Z+ [[ 5 -gt 60 ]]
8182023-12-01T20:20:49.796Z+ sleep 1
8192023-12-01T20:20:50.794Z+ retry=6
8202023-12-01T20:20:50.819Z+++ zoneadm list -n
8212023-12-01T20:20:50.819Z+++ grep oxz_ntp
8222023-12-01T20:20:50.819Z++ pfexec svcs -z -Hostate oxide/ntp
8232023-12-01T20:20:50.819Zsvcs: zone '-Hostate': entity not found
8242023-12-01T20:20:50.819Z++ true
8252023-12-01T20:20:50.819Z+ [[ '' != online ]]
8262023-12-01T20:20:50.819Z+ [[ 6 -gt 60 ]]
8272023-12-01T20:20:50.819Z+ sleep 1
8282023-12-01T20:20:51.816Z+ retry=7
8292023-12-01T20:20:51.841Z+++ zoneadm list -n
8302023-12-01T20:20:51.841Z+++ grep oxz_ntp
8312023-12-01T20:20:51.841Z++ pfexec svcs -z oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18 -Hostate oxide/ntp
8322023-12-01T20:20:51.869Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8332023-12-01T20:20:51.869Z++ true
8342023-12-01T20:20:51.877Z+ [[ '' != online ]]
8352023-12-01T20:20:51.877Z+ [[ 7 -gt 60 ]]
8362023-12-01T20:20:51.877Z+ sleep 1
8372023-12-01T20:20:52.863Z+ retry=8
8382023-12-01T20:20:52.888Z+++ zoneadm list -n
8392023-12-01T20:20:52.888Z+++ grep oxz_ntp
8402023-12-01T20:20:52.888Z++ pfexec svcs -z oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18 -Hostate oxide/ntp
8412023-12-01T20:20:52.912Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8422023-12-01T20:20:52.912Z++ true
8432023-12-01T20:20:52.912Z+ [[ '' != online ]]
8442023-12-01T20:20:52.912Z+ [[ 8 -gt 60 ]]
8452023-12-01T20:20:52.912Z+ sleep 1
8462023-12-01T20:20:53.893Z+ retry=9
8472023-12-01T20:20:53.918Z+++ zoneadm list -n
8482023-12-01T20:20:53.918Z+++ grep oxz_ntp
8492023-12-01T20:20:53.918Z++ pfexec svcs -z oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18 -Hostate oxide/ntp
8502023-12-01T20:20:53.943Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8512023-12-01T20:20:53.943Z++ true
8522023-12-01T20:20:53.943Z+ [[ '' != online ]]
8532023-12-01T20:20:53.943Z+ [[ 9 -gt 60 ]]
8542023-12-01T20:20:53.943Z+ sleep 1
8552023-12-01T20:20:54.923Z+ retry=10
8562023-12-01T20:20:54.949Z+++ zoneadm list -n
8572023-12-01T20:20:54.950Z+++ grep oxz_ntp
8582023-12-01T20:20:54.950Z++ pfexec svcs -z oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18 -Hostate oxide/ntp
8592023-12-01T20:20:54.950Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8602023-12-01T20:20:54.950Z++ true
8612023-12-01T20:20:54.974Z+ [[ '' != online ]]
8622023-12-01T20:20:54.974Z+ [[ 10 -gt 60 ]]
8632023-12-01T20:20:54.974Z+ sleep 1
8642023-12-01T20:20:55.953Z+ retry=11
8652023-12-01T20:20:55.977Z+++ zoneadm list -n
8662023-12-01T20:20:55.977Z+++ grep oxz_ntp
8672023-12-01T20:20:55.977Z++ pfexec svcs -z oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18 -Hostate oxide/ntp
8682023-12-01T20:20:55.977Z+ [[ disabled != online ]]
8692023-12-01T20:20:55.977Z+ [[ 11 -gt 60 ]]
8702023-12-01T20:20:55.977Z+ sleep 1
8712023-12-01T20:20:57.025Z+ retry=12
8722023-12-01T20:20:57.049Z+++ zoneadm list -n
8732023-12-01T20:20:57.049Z+++ grep oxz_ntp
8742023-12-01T20:20:57.051Z++ pfexec svcs -z oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18 -Hostate oxide/ntp
8752023-12-01T20:20:57.051Z+ [[ disabled != online ]]
8762023-12-01T20:20:57.051Z+ [[ 12 -gt 60 ]]
8772023-12-01T20:20:57.051Z+ sleep 1
8782023-12-01T20:20:57.998Z+ retry=13
8792023-12-01T20:20:58.023Z+++ zoneadm list -n
8802023-12-01T20:20:58.023Z+++ grep oxz_ntp
8812023-12-01T20:20:58.023Z++ pfexec svcs -z oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18 -Hostate oxide/ntp
8822023-12-01T20:20:58.023Z+ [[ disabled != online ]]
8832023-12-01T20:20:58.023Z+ [[ 13 -gt 60 ]]
8842023-12-01T20:20:58.023Z+ sleep 1
8852023-12-01T20:20:59.020Z+ retry=14
8862023-12-01T20:20:59.045Z+++ zoneadm list -n
8872023-12-01T20:20:59.045Z+++ grep oxz_ntp
8882023-12-01T20:20:59.045Z++ pfexec svcs -z oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18 -Hostate oxide/ntp
8892023-12-01T20:20:59.045Z+ [[ disabled != online ]]
8902023-12-01T20:20:59.045Z+ [[ 14 -gt 60 ]]
8912023-12-01T20:20:59.045Z+ sleep 1
8922023-12-01T20:21:00.043Z+ retry=15
8932023-12-01T20:21:00.068Z+++ zoneadm list -n
8942023-12-01T20:21:00.068Z+++ grep oxz_ntp
8952023-12-01T20:21:00.068Z++ pfexec svcs -z oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18 -Hostate oxide/ntp
8962023-12-01T20:21:00.068Z+ [[ disabled != online ]]
8972023-12-01T20:21:00.068Z+ [[ 15 -gt 60 ]]
8982023-12-01T20:21:00.069Z+ sleep 1
8992023-12-01T20:21:01.065Z+ retry=16
9002023-12-01T20:21:01.090Z+++ zoneadm list -n
9012023-12-01T20:21:01.090Z+++ grep oxz_ntp
9022023-12-01T20:21:01.090Z++ pfexec svcs -z oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18 -Hostate oxide/ntp
9032023-12-01T20:21:01.090Z+ [[ disabled != online ]]
9042023-12-01T20:21:01.090Z+ [[ 16 -gt 60 ]]
9052023-12-01T20:21:01.090Z+ sleep 1
9062023-12-01T20:21:02.089Z+ retry=17
9072023-12-01T20:21:02.114Z+++ zoneadm list -n
9082023-12-01T20:21:02.115Z+++ grep oxz_ntp
9092023-12-01T20:21:02.115Z++ pfexec svcs -z oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18 -Hostate oxide/ntp
9102023-12-01T20:21:02.115Z+ [[ disabled != online ]]
9112023-12-01T20:21:02.115Z+ [[ 17 -gt 60 ]]
9122023-12-01T20:21:02.115Z+ sleep 1
9132023-12-01T20:21:03.110Z+ retry=18
9142023-12-01T20:21:03.137Z+++ zoneadm list -n
9152023-12-01T20:21:03.137Z+++ grep oxz_ntp
9162023-12-01T20:21:03.137Z++ pfexec svcs -z oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18 -Hostate oxide/ntp
9172023-12-01T20:21:03.137Z+ [[ disabled != online ]]
9182023-12-01T20:21:03.137Z+ [[ 18 -gt 60 ]]
9192023-12-01T20:21:03.137Z+ sleep 1
 
9202023-12-01T20:21:03.297Zsock console login: Dec 1 20:21:02 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9212023-12-01T20:21:04.133Z+ retry=19
9222023-12-01T20:21:04.158Z+++ zoneadm list -n
9232023-12-01T20:21:04.158Z+++ grep oxz_ntp
9242023-12-01T20:21:04.158Z++ pfexec svcs -z oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18 -Hostate oxide/ntp
9252023-12-01T20:21:04.158Z+ [[ online != online ]]
9262023-12-01T20:21:04.158Z+ echo 'Waited for chrony: 19s'
9272023-12-01T20:21:04.158Z+ export RUST_BACKTRACE=1
9282023-12-01T20:21:04.158ZWaited for chrony: 19s
9292023-12-01T20:21:04.158Z+ RUST_BACKTRACE=1
9302023-12-01T20:21:04.158Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9312023-12-01T20:21:04.158Z+ ./tests/bootstrap
9322023-12-01T20:21:04.184ZUsing external DNS server at 10.151.2.164:53
9332023-12-01T20:21:04.184Z2023-12-01 20:21:03.658856023 UTC: attempting to log into API
9342023-12-01T20:21:19.237Z2023-12-01 20:21:18.933770981 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9352023-12-01T20:21:20.218Z2023-12-01 20:21:19.934895996 UTC: attempting to log into API
9362023-12-01T20:21:35.251Z2023-12-01 20:21:34.967985134 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9372023-12-01T20:21:36.253Z2023-12-01 20:21:35.969608058 UTC: attempting to log into API
9382023-12-01T20:21:51.299Z2023-12-01 20:21:51.016043923 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9392023-12-01T20:21:52.301Z2023-12-01 20:21:52.017622507 UTC: attempting to log into API
9402023-12-01T20:22:07.354Z2023-12-01 20:22:07.070712025 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9412023-12-01T20:22:08.355Z2023-12-01 20:22:08.071499874 UTC: attempting to log into API
 
9422023-12-01T20:22:14.037ZDec 1 20:22:13 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9432023-12-01T20:22:21.760ZDec 1 20:22:21 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9442023-12-01T20:22:23.397Z2023-12-01 20:22:23.647512089 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9452023-12-01T20:22:24.398Z2023-12-01 20:22:24.645854902 UTC: attempting to log into API
 
9462023-12-01T20:22:28.439ZDec 1 20:22:28 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9472023-12-01T20:22:34.048ZDec 1 20:22:33 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9482023-12-01T20:22:39.431Z2023-12-01 20:22:39.638334052 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9492023-12-01T20:22:40.432Z2023-12-01 20:22:40.636520353 UTC: attempting to log into API
 
9502023-12-01T20:22:42.269ZDec 1 20:22:41 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9512023-12-01T20:22:55.464Z2023-12-01 20:22:55.627633847 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9522023-12-01T20:22:56.465Z2023-12-01 20:22:56.625982901 UTC: attempting to log into API
9532023-12-01T20:23:02.252Z2023-12-01 20:23:02.397383771 UTC: login succeeded
9542023-12-01T20:23:02.287Zwaiting for nexus to come up...
9552023-12-01T20:23:02.348Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9562023-12-01T20:23:02.403Zensuring datasets are ready...
9572023-12-01T20:23:04.858Zlet's roll.
9582023-12-01T20:23:04.882Z+ rm ./tests/bootstrap
9592023-12-01T20:23:04.926Z+ for test_bin in tests/*
9602023-12-01T20:23:04.926Z+ ./tests/bootstrap-7a35cfb1c9365afc
9612023-12-01T20:23:04.950Z
9622023-12-01T20:23:04.950Zrunning 0 tests
9632023-12-01T20:23:04.950Z
9642023-12-01T20:23:04.950Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9652023-12-01T20:23:04.950Z
9662023-12-01T20:23:04.950Z+ for test_bin in tests/*
9672023-12-01T20:23:04.950Z+ ./tests/end_to_end_tests-be479211b7b081a6
9682023-12-01T20:23:04.950Z
9692023-12-01T20:23:04.950Zrunning 2 tests
9702023-12-01T20:23:05.170Ztest no_spoof::no_spoof_here ... ok
9712023-12-01T20:24:05.098Ztest instance_launch::instance_launch has been running for over 60 seconds
9722023-12-01T20:25:58.479Ztest instance_launch::instance_launch ... ok
9732023-12-01T20:25:58.479Z
9742023-12-01T20:25:58.479Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 173.53s
9752023-12-01T20:25:58.479Z
9762023-12-01T20:25:58.504Z+ _exit_trap
9772023-12-01T20:25:58.504Z+ local status=0
9782023-12-01T20:25:58.504Z+ [[ 0 -eq 0 ]]
9792023-12-01T20:25:58.504Z+ exit 0
9802023-12-01T20:25:58.504Zprocess exited: duration 375875 ms, exit code 0
 
9812023-12-01T20:25:58.611Zfound 47 output files
9822023-12-01T20:25:58.611Zuploading: /var/svc/log/oxide-sled-agent:default.log (741123 bytes)
9832023-12-01T20:26:00.548Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9842023-12-01T20:26:00.548Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_40302a57-dabd-4048-8471-abe54c94a5c6/root/var/svc/log/oxide-clickhouse:default.log (657986 bytes)
9852023-12-01T20:26:00.812Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_40302a57-dabd-4048-8471-abe54c94a5c6/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 657986 -> 682466
9862023-12-01T20:26:01.887Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_40302a57-dabd-4048-8471-abe54c94a5c6/root/var/svc/log/oxide-clickhouse:default.log
9872023-12-01T20:26:01.887Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_3a2217f1-0118-4f38-b99e-bf7adc2f6ffe/root/var/svc/log/oxide-crucible-pantry:default.log (2337 bytes)
9882023-12-01T20:26:02.985Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_3a2217f1-0118-4f38-b99e-bf7adc2f6ffe/root/var/svc/log/oxide-crucible-pantry:default.log
9892023-12-01T20:26:02.985Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_e08fa54d-6d16-42e5-aea4-5192d0525092/root/var/svc/log/oxide-nexus:default.log (523884 bytes)
9902023-12-01T20:26:04.293Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_e08fa54d-6d16-42e5-aea4-5192d0525092/root/var/svc/log/oxide-nexus:default.log
9912023-12-01T20:26:04.293Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_1389029a-bfc7-42f8-811e-32f540038fce/root/var/svc/log/oxide-oximeter:default.log (173367 bytes)
9922023-12-01T20:26:04.479Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_1389029a-bfc7-42f8-811e-32f540038fce/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 173367 -> 179726
9932023-12-01T20:26:05.551Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_1389029a-bfc7-42f8-811e-32f540038fce/root/var/svc/log/oxide-oximeter:default.log
9942023-12-01T20:26:05.551Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0ea1736a-a549-4c40-8f9f-3107197abd8d/root/var/svc/log/oxide-cockroachdb:default.log (5366 bytes)
9952023-12-01T20:26:06.669Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0ea1736a-a549-4c40-8f9f-3107197abd8d/root/var/svc/log/oxide-cockroachdb:default.log
9962023-12-01T20:26:06.669Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eecf1538-63c4-4d9d-833e-25c14fc54f0d/root/var/svc/log/oxide-crucible-agent:default.log (14584 bytes)
9972023-12-01T20:26:07.788Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eecf1538-63c4-4d9d-833e-25c14fc54f0d/root/var/svc/log/oxide-crucible-agent:default.log
9982023-12-01T20:26:07.790Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eecf1538-63c4-4d9d-833e-25c14fc54f0d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9992023-12-01T20:26:08.884Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eecf1538-63c4-4d9d-833e-25c14fc54f0d/root/var/svc/log/oxide-crucible-downstairs:default.log
10002023-12-01T20:26:08.884Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eecf1538-63c4-4d9d-833e-25c14fc54f0d/root/var/svc/log/oxide-crucible-downstairs:downstairs-99f7e420-14af-43c2-9d28-b4a47a3c08b7.log (5536 bytes)
10012023-12-01T20:26:09.981Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eecf1538-63c4-4d9d-833e-25c14fc54f0d/root/var/svc/log/oxide-crucible-downstairs:downstairs-99f7e420-14af-43c2-9d28-b4a47a3c08b7.log
10022023-12-01T20:26:09.981Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_2c9cf47c-e152-4c6a-bbef-63f2f7e95e4c/root/var/svc/log/oxide-nexus:default.log (4701376 bytes)
10032023-12-01T20:26:12.070Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_2c9cf47c-e152-4c6a-bbef-63f2f7e95e4c/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4701376 -> 4702594
10042023-12-01T20:26:13.142Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_2c9cf47c-e152-4c6a-bbef-63f2f7e95e4c/root/var/svc/log/oxide-nexus:default.log
10052023-12-01T20:26:13.142Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_47d8ca09-98d0-4c1a-a8ec-2d9972ae9a03/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10062023-12-01T20:26:14.238Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_47d8ca09-98d0-4c1a-a8ec-2d9972ae9a03/root/var/svc/log/oxide-crucible-agent:default.log
10072023-12-01T20:26:14.238Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_47d8ca09-98d0-4c1a-a8ec-2d9972ae9a03/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10082023-12-01T20:26:15.334Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_47d8ca09-98d0-4c1a-a8ec-2d9972ae9a03/root/var/svc/log/oxide-crucible-downstairs:default.log
10092023-12-01T20:26:15.334Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_3d47be57-2a0a-42dc-9510-643f0ed22660/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10102023-12-01T20:26:16.431Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_3d47be57-2a0a-42dc-9510-643f0ed22660/root/var/svc/log/oxide-crucible-pantry:default.log
10112023-12-01T20:26:16.431Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_4f53e2d1-634d-425b-b55b-d7c13a752df6/root/var/svc/log/oxide-crucible-pantry:default.log (2332 bytes)
10122023-12-01T20:26:17.526Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_4f53e2d1-634d-425b-b55b-d7c13a752df6/root/var/svc/log/oxide-crucible-pantry:default.log
10132023-12-01T20:26:17.527Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_fd8ac9d3-b0b0-4757-8159-6f9ec8c6c3e6/root/var/svc/log/oxide-internal_dns:default.log (104761 bytes)
10142023-12-01T20:26:18.749Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_fd8ac9d3-b0b0-4757-8159-6f9ec8c6c3e6/root/var/svc/log/oxide-internal_dns:default.log
10152023-12-01T20:26:18.749Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_caaa156e-7be2-4f1c-9121-8ba076f7b9de/root/var/svc/log/oxide-nexus:default.log (519110 bytes)
10162023-12-01T20:26:19.510Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_caaa156e-7be2-4f1c-9121-8ba076f7b9de/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 519110 -> 523877
10172023-12-01T20:26:20.581Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_caaa156e-7be2-4f1c-9121-8ba076f7b9de/root/var/svc/log/oxide-nexus:default.log
10182023-12-01T20:26:20.581Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_4fed016d-381c-4e52-9d55-ce484c17ffa4/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10192023-12-01T20:26:21.678Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_4fed016d-381c-4e52-9d55-ce484c17ffa4/root/var/svc/log/oxide-cockroachdb:default.log
10202023-12-01T20:26:21.678Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_b9d28670-a492-4007-b362-0b1f1097a943/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10212023-12-01T20:26:22.774Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_b9d28670-a492-4007-b362-0b1f1097a943/root/var/svc/log/oxide-cockroachdb:default.log
10222023-12-01T20:26:22.774Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_611a2554-917d-4261-9cf1-9b759f4385a6/root/var/svc/log/oxide-crucible-agent:default.log (14119 bytes)
10232023-12-01T20:26:23.892Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_611a2554-917d-4261-9cf1-9b759f4385a6/root/var/svc/log/oxide-crucible-agent:default.log
10242023-12-01T20:26:23.892Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_611a2554-917d-4261-9cf1-9b759f4385a6/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10252023-12-01T20:26:24.989Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_611a2554-917d-4261-9cf1-9b759f4385a6/root/var/svc/log/oxide-crucible-downstairs:default.log
10262023-12-01T20:26:24.989Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_611a2554-917d-4261-9cf1-9b759f4385a6/root/var/svc/log/oxide-crucible-downstairs:downstairs-d8cbae51-b997-4e57-a354-6dd2561bb732.log (5537 bytes)
10272023-12-01T20:26:26.085Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_611a2554-917d-4261-9cf1-9b759f4385a6/root/var/svc/log/oxide-crucible-downstairs:downstairs-d8cbae51-b997-4e57-a354-6dd2561bb732.log
10282023-12-01T20:26:26.085Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_11854ffa-1764-4c42-8bea-aae473a41329/root/var/svc/log/oxide-crucible-agent:default.log (13684 bytes)
10292023-12-01T20:26:27.205Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_11854ffa-1764-4c42-8bea-aae473a41329/root/var/svc/log/oxide-crucible-agent:default.log
10302023-12-01T20:26:27.205Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_11854ffa-1764-4c42-8bea-aae473a41329/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10312023-12-01T20:26:28.314Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_11854ffa-1764-4c42-8bea-aae473a41329/root/var/svc/log/oxide-crucible-downstairs:default.log
10322023-12-01T20:26:28.314Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_11854ffa-1764-4c42-8bea-aae473a41329/root/var/svc/log/oxide-crucible-downstairs:downstairs-539caf17-3e33-4219-ae42-aa015da34b9e.log (3051 bytes)
10332023-12-01T20:26:29.409Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_11854ffa-1764-4c42-8bea-aae473a41329/root/var/svc/log/oxide-crucible-downstairs:downstairs-539caf17-3e33-4219-ae42-aa015da34b9e.log
10342023-12-01T20:26:29.409Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_e34866b5-41eb-452d-8612-8675bdd8b86d/root/var/svc/log/oxide-external_dns:default.log (15614 bytes)
10352023-12-01T20:26:30.537Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_e34866b5-41eb-452d-8612-8675bdd8b86d/root/var/svc/log/oxide-external_dns:default.log
10362023-12-01T20:26:30.537Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_09d20e91-71d5-4b38-a3fb-479799388ca4/root/var/svc/log/oxide-internal_dns:default.log (119875 bytes)
10372023-12-01T20:26:31.748Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_09d20e91-71d5-4b38-a3fb-479799388ca4/root/var/svc/log/oxide-internal_dns:default.log
10382023-12-01T20:26:31.748Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_0e656f00-b76e-4229-ad24-524aaf2667f9/root/var/svc/log/oxide-cockroachdb:default.log (5342 bytes)
10392023-12-01T20:26:32.845Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_0e656f00-b76e-4229-ad24-524aaf2667f9/root/var/svc/log/oxide-cockroachdb:default.log
10402023-12-01T20:26:32.845Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4d9f8886-62e3-4da7-b343-880dcec31cd3/root/var/svc/log/oxide-crucible-agent:default.log (13680 bytes)
10412023-12-01T20:26:33.970Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4d9f8886-62e3-4da7-b343-880dcec31cd3/root/var/svc/log/oxide-crucible-agent:default.log
10422023-12-01T20:26:33.970Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4d9f8886-62e3-4da7-b343-880dcec31cd3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10432023-12-01T20:26:35.067Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4d9f8886-62e3-4da7-b343-880dcec31cd3/root/var/svc/log/oxide-crucible-downstairs:default.log
10442023-12-01T20:26:35.067Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4d9f8886-62e3-4da7-b343-880dcec31cd3/root/var/svc/log/oxide-crucible-downstairs:downstairs-78d6f1d6-acb2-4020-be1a-9b1ff809e954.log (3051 bytes)
10452023-12-01T20:26:36.163Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4d9f8886-62e3-4da7-b343-880dcec31cd3/root/var/svc/log/oxide-crucible-downstairs:downstairs-78d6f1d6-acb2-4020-be1a-9b1ff809e954.log
10462023-12-01T20:26:36.163Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_77bd84d6-c60c-4f2f-ad75-394271827f0d/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10472023-12-01T20:26:37.259Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_77bd84d6-c60c-4f2f-ad75-394271827f0d/root/var/svc/log/oxide-crucible-agent:default.log
10482023-12-01T20:26:37.259Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_77bd84d6-c60c-4f2f-ad75-394271827f0d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10492023-12-01T20:26:38.354Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_77bd84d6-c60c-4f2f-ad75-394271827f0d/root/var/svc/log/oxide-crucible-downstairs:default.log
10502023-12-01T20:26:38.354Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a05b6447-1de1-4773-a08b-af26700afacc/root/var/svc/log/oxide-crucible-agent:default.log (14112 bytes)
10512023-12-01T20:26:39.472Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a05b6447-1de1-4773-a08b-af26700afacc/root/var/svc/log/oxide-crucible-agent:default.log
10522023-12-01T20:26:39.472Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a05b6447-1de1-4773-a08b-af26700afacc/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10532023-12-01T20:26:40.568Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a05b6447-1de1-4773-a08b-af26700afacc/root/var/svc/log/oxide-crucible-downstairs:default.log
10542023-12-01T20:26:40.568Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a05b6447-1de1-4773-a08b-af26700afacc/root/var/svc/log/oxide-crucible-downstairs:downstairs-8621dec3-6040-44ab-abbf-30dea2bce91f.log (3051 bytes)
10552023-12-01T20:26:41.663Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a05b6447-1de1-4773-a08b-af26700afacc/root/var/svc/log/oxide-crucible-downstairs:downstairs-8621dec3-6040-44ab-abbf-30dea2bce91f.log
10562023-12-01T20:26:41.663Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_722a35b6-2241-4996-989a-64f7bb6224c1/root/var/svc/log/oxide-internal_dns:default.log (110295 bytes)
10572023-12-01T20:26:42.873Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_722a35b6-2241-4996-989a-64f7bb6224c1/root/var/svc/log/oxide-internal_dns:default.log
10582023-12-01T20:26:42.873Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e57cf3df-bd68-4b6a-959d-456e679e6729/root/var/svc/log/oxide-crucible-agent:default.log (13681 bytes)
10592023-12-01T20:26:43.992Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e57cf3df-bd68-4b6a-959d-456e679e6729/root/var/svc/log/oxide-crucible-agent:default.log
10602023-12-01T20:26:43.992Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e57cf3df-bd68-4b6a-959d-456e679e6729/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10612023-12-01T20:26:45.091Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e57cf3df-bd68-4b6a-959d-456e679e6729/root/var/svc/log/oxide-crucible-downstairs:default.log
10622023-12-01T20:26:45.091Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e57cf3df-bd68-4b6a-959d-456e679e6729/root/var/svc/log/oxide-crucible-downstairs:downstairs-3f8b77aa-e82b-417c-b83e-97a087a61906.log (5536 bytes)
10632023-12-01T20:26:46.188Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e57cf3df-bd68-4b6a-959d-456e679e6729/root/var/svc/log/oxide-crucible-downstairs:downstairs-3f8b77aa-e82b-417c-b83e-97a087a61906.log
10642023-12-01T20:26:46.188Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c0050ed1-1b73-432e-b04b-3862259ecfb0/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10652023-12-01T20:26:47.284Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c0050ed1-1b73-432e-b04b-3862259ecfb0/root/var/svc/log/oxide-cockroachdb:default.log
10662023-12-01T20:26:47.284Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cb88aa9f-3994-4865-8316-423b2415891b/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10672023-12-01T20:26:48.380Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cb88aa9f-3994-4865-8316-423b2415891b/root/var/svc/log/oxide-crucible-agent:default.log
10682023-12-01T20:26:48.381Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cb88aa9f-3994-4865-8316-423b2415891b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10692023-12-01T20:26:49.476Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cb88aa9f-3994-4865-8316-423b2415891b/root/var/svc/log/oxide-crucible-downstairs:default.log
10702023-12-01T20:26:49.477Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_1bed9890-7282-49b4-8df2-ae1240e7fb19/root/var/svc/log/oxide-external_dns:default.log (15617 bytes)
10712023-12-01T20:26:49.570Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_1bed9890-7282-49b4-8df2-ae1240e7fb19/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15617 -> 19394
10722023-12-01T20:26:50.641Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_1bed9890-7282-49b4-8df2-ae1240e7fb19/root/var/svc/log/oxide-external_dns:default.log
10732023-12-01T20:26:50.641Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18/root/var/svc/log/oxide-ntp:default.log (1185 bytes)
10742023-12-01T20:26:51.737Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18/root/var/svc/log/oxide-ntp:default.log
10752023-12-01T20:26:51.737Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18/root/var/log/chrony/measurements.log (13974 bytes)
10762023-12-01T20:26:51.829Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18/root/var/log/chrony/measurements.log" changed size mid upload: 13974 -> 16440
10772023-12-01T20:26:52.903Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18/root/var/log/chrony/measurements.log
10782023-12-01T20:26:52.903Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18/root/var/log/chrony/statistics.log (8073 bytes)
10792023-12-01T20:26:52.976Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18/root/var/log/chrony/statistics.log" changed size mid upload: 8073 -> 10179
10802023-12-01T20:26:54.047Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18/root/var/log/chrony/statistics.log
10812023-12-01T20:26:54.047Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18/root/var/log/chrony/tracking.log (1717 bytes)
10822023-12-01T20:26:54.095Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18/root/var/log/chrony/tracking.log" changed size mid upload: 1717 -> 1849
10832023-12-01T20:26:55.179Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a3ace03-761f-4140-ac61-ace72435cb18/root/var/log/chrony/tracking.log