01HEK63M7P2H54B1ADJFBFEMG8: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HEK65M7AF3C6NM7A3XGFN24X

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-06T21:45:00.608Zjob dependencies complete; ready to run (waiting for 58 m 44 s)
22023-11-06T23:38:14.116Zjob assigned to worker 01HEKFZ1V9Y2429N7461BZJNG7 (queued for 1 h 53 m 13 s)
32023-11-06T23:38:14.283Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-11-06T23:38:14.349Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-11-06T23:38:14.398Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-11-06T23:38:14.437Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-06T23:38:14.483Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-11-06T23:38:14.524ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-06T23:38:14.557Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-06T23:38:14.598ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-06T23:38:14.634ZConfiguring devices.
122023-11-06T23:38:14.668Z##################################################
132023-11-06T23:38:14.701Z#### ############# ##
142023-11-06T23:38:14.845Z### ### ############ ##
152023-11-06T23:38:14.999Z## ### # ## ### ## Oxide Computer Company ##
162023-11-06T23:38:15.510Z## ## ## ### # ### ##
172023-11-06T23:38:15.562Z## # ### #### #### This Station Under ##
182023-11-06T23:38:15.603Z### ### #### # ### Computer Control ##
192023-11-06T23:38:15.635Z#### #### ### ## ##
202023-11-06T23:38:15.675Z##################################################
212023-11-06T23:38:15.713ZHostname: buskin
222023-11-06T23:38:15.747ZLoading smf(7) serice descriptions:
232023-11-06T23:38:16.043Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-346a6c0cadffef35.gz
242023-11-06T23:38:22.638Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-346a6c0cadffef35.gz
252023-11-06T23:38:22.679Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-06T23:38:57.467Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-06T23:38:57.495Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ad8d578e5459ec9d.gz
282023-11-06T23:39:34.042Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ad8d578e5459ec9d.gz
292023-11-06T23:39:34.076Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-06T23:39:37.243Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-06T23:39:37.268Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-06T23:39:41.063Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-06T23:39:41.088Zdownloading input: /input/package/work/version.txt
342023-11-06T23:39:41.120Zdownloaded input: /input/package/work/version.txt
352023-11-06T23:39:41.553Zdownloading input: /input/package/work/package.tar.gz
362023-11-06T23:40:11.329Zdownloaded input: /input/package/work/package.tar.gz
372023-11-06T23:40:11.354Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-06T23:40:45.551Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-06T23:40:45.580Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-06T23:41:01.729Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-06T23:41:01.762Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-06T23:41:58.212Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-06T23:41:58.240Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-06T23:42:54.954Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-06T23:42:54.979Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-06T23:44:04.291Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-06T23:44:04.764Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-06T23:44:09.811Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-06T23:44:09.847Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-06T23:44:22.896Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-06T23:44:22.954Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-06T23:44:31.003Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-06T23:44:31.027Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-06T23:44:36.667Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-06T23:44:36.698Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-06T23:44:36.727Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-06T23:44:36.754Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-06T23:44:40.418Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-06T23:44:40.442Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-06T23:44:45.749Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-06T23:44:45.778Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-06T23:45:05.253Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-06T23:45:05.347Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-06T23:45:21.225Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-06T23:45:21.250Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-06T23:45:21.806Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-06T23:45:21.840Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-06T23:45:26.501Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-06T23:45:26.525Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-06T23:45:40.918Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-06T23:45:40.943Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-06T23:46:39.912Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-06T23:46:41.177Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-06T23:47:17.897Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-06T23:47:17.924Zstarting task 0: "setup"
762023-11-06T23:47:17.959Z++ uname -s
772023-11-06T23:47:17.987Z+ kern=SunOS
782023-11-06T23:47:18.012Z+ case "$kern" in
792023-11-06T23:47:18.036Z+ groupadd -g 12345 build
802023-11-06T23:47:18.060Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-06T23:47:19.968Z+ zfs create -o mountpoint=/work rpool/work
822023-11-06T23:47:19.992Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-06T23:47:20.016Z+ home_fs=zfs
842023-11-06T23:47:20.040Z+ [[ zfs == autofs ]]
852023-11-06T23:47:20.064Z+ mkdir -p /home/build
862023-11-06T23:47:20.088Z+ chown build:build /home/build /work
872023-11-06T23:47:21.994Z+ chmod 0700 /home/build /work
882023-11-06T23:47:22.021Zprocess exited: duration 4048 ms, exit code 0
 
892023-11-06T23:47:22.072Zstarting task 1: "authentication"
902023-11-06T23:47:22.145Zprocess exited: duration 44 ms, exit code 0
 
912023-11-06T23:47:22.199Zstarting task 2: "build"
922023-11-06T23:47:23.582Z+ trap _exit_trap EXIT
932023-11-06T23:47:23.615Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-06T23:47:23.675Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-06T23:47:23.714Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-06T23:47:23.744Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-06T23:47:23.771Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-06T23:47:23.799Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-06T23:47:23.826Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-06T23:47:23.850Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-06T23:47:23.874Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-06T23:47:23.898Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-06T23:47:23.922Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-06T23:47:23.946Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-06T23:47:23.969Z+ [[ -d /opt/oxide ]]
1062023-11-06T23:47:24.402Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-06T23:47:24.426Z+ pfexec mkdir /opt/oxide
1082023-11-06T23:47:24.451Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-06T23:47:24.475Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-06T23:47:24.501Z+ cd /opt/oxide-underneath
1112023-11-06T23:47:24.530Z+ pfexec tar ceEp@/f - .
1122023-11-06T23:47:24.570Z+ cd /opt/oxide
1132023-11-06T23:47:24.594Z+ pfexec tar xveEp@/f -
1142023-11-06T23:47:24.618Zx ., 0 bytes, 0 tape blocks
1152023-11-06T23:47:24.642Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-06T23:47:24.679Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-06T23:47:24.720Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-06T23:47:24.759Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-06T23:47:24.786Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-06T23:47:24.812Z+ pfexec mkdir /opt/oxide/work
1212023-11-06T23:47:24.837Z+ pfexec chown build:build /opt/oxide/work
1222023-11-06T23:47:24.861Z+ cd /opt/oxide/work
1232023-11-06T23:47:24.887Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-06T23:47:25.456ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-06T23:47:25.504Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1262023-11-06T23:47:25.535Zx out/omicron-sled-agent.tar, 123253248 bytes, 240729 tape blocks
1272023-11-06T23:47:25.561Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-06T23:47:25.606Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-11-06T23:47:25.647Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-11-06T23:47:25.678Zx smf/sled-agent/non-gimlet/config.toml, 3102 bytes, 7 tape blocks
1312023-11-06T23:47:25.713Zx target/release/omicron-package, 42682512 bytes, 83365 tape blocks
1322023-11-06T23:47:25.755Zx tools/create_virtual_hardware.sh, 3067 bytes, 6 tape blocks
1332023-11-06T23:47:25.782Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-06T23:47:25.820Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-06T23:47:25.846Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-06T23:47:25.880Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-06T23:47:25.916Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-06T23:47:25.947Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-06T23:47:25.973Z
1402023-11-06T23:47:25.998Zreal 1.618778501
1412023-11-06T23:47:26.033Zuser 1.452955437
1422023-11-06T23:47:26.076Zsys 0.289739379
1432023-11-06T23:47:26.109Ztrap 0.000071921
1442023-11-06T23:47:26.196Ztflt 0.000000000
1452023-11-06T23:47:26.221Zdflt 0.000000000
1462023-11-06T23:47:26.249Zkflt 0.000000000
1472023-11-06T23:47:26.277Zlock 0.000000000
1482023-11-06T23:47:26.334Zslp 1.457856019
1492023-11-06T23:47:26.370Zlat 0.034532808
1502023-11-06T23:47:26.411Zstop 0.000124510
1512023-11-06T23:47:26.443Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-06T23:47:26.514Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-06T23:47:26.545Z+ mkdir tests
1542023-11-06T23:47:26.992Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-06T23:47:27.023Z+ ptime -m gunzip
1562023-11-06T23:47:27.051Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-346a6c0cadffef35
1572023-11-06T23:47:27.075Z
1582023-11-06T23:47:27.099Zreal 0.141820081
1592023-11-06T23:47:27.124Zuser 0.129590007
1602023-11-06T23:47:27.150Zsys 0.012002309
1612023-11-06T23:47:27.175Ztrap 0.000012439
1622023-11-06T23:47:27.216Ztflt 0.000000000
1632023-11-06T23:47:27.247Zdflt 0.000000000
1642023-11-06T23:47:27.274Zkflt 0.000000000
1652023-11-06T23:47:27.298Zlock 0.000000000
1662023-11-06T23:47:27.323Zslp 0.000021820
1672023-11-06T23:47:27.354Zlat 0.000016650
1682023-11-06T23:47:27.378Zstop 0.000119592
1692023-11-06T23:47:27.403Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-346a6c0cadffef35
1702023-11-06T23:47:27.429Z+ chmod a+x tests/bootstrap-346a6c0cadffef35
1712023-11-06T23:47:27.470Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-06T23:47:27.498Z+ ptime -m gunzip
1732023-11-06T23:47:27.524Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-06T23:47:28.993Z
1752023-11-06T23:47:29.021Zreal 2.295653110
1762023-11-06T23:47:29.045Zuser 2.149336116
1772023-11-06T23:47:29.069Zsys 0.145899287
1782023-11-06T23:47:29.096Ztrap 0.000191963
1792023-11-06T23:47:29.126Ztflt 0.000000000
1802023-11-06T23:47:29.153Zdflt 0.000000000
1812023-11-06T23:47:29.182Zkflt 0.000000000
1822023-11-06T23:47:29.208Zlock 0.000000000
1832023-11-06T23:47:29.238Zslp 0.000023080
1842023-11-06T23:47:29.263Zlat 0.000022479
1852023-11-06T23:47:29.286Zstop 0.000115871
1862023-11-06T23:47:29.311Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-06T23:47:29.335Z+ chmod a+x tests/bootstrap
1882023-11-06T23:47:29.359Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-06T23:47:29.383Z+ ptime -m gunzip
1902023-11-06T23:47:29.821Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ad8d578e5459ec9d
1912023-11-06T23:47:31.606Z
1922023-11-06T23:47:31.630Zreal 2.610345798
1932023-11-06T23:47:31.654Zuser 2.451608353
1942023-11-06T23:47:31.678Zsys 0.158282689
1952023-11-06T23:47:32.111Ztrap 0.000204801
1962023-11-06T23:47:32.135Ztflt 0.000000000
1972023-11-06T23:47:32.159Zdflt 0.000000000
1982023-11-06T23:47:32.183Zkflt 0.000000000
1992023-11-06T23:47:32.208Zlock 0.000000000
2002023-11-06T23:47:32.237Zslp 0.000024220
2012023-11-06T23:47:32.263Zlat 0.000023440
2022023-11-06T23:47:32.288Zstop 0.000134651
2032023-11-06T23:47:32.318Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ad8d578e5459ec9d
2042023-11-06T23:47:32.342Z+ chmod a+x tests/end_to_end_tests-ad8d578e5459ec9d
2052023-11-06T23:47:32.369Z+ bmat address ls
2062023-11-06T23:47:32.399ZNAME CIDR FIRST LAST COUNT
2072023-11-06T23:47:32.424Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-11-06T23:47:32.448Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-06T23:47:32.472Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-11-06T23:47:32.496Z+ EXTRA_IP_START=10.151.1.164
2112023-11-06T23:47:32.520Z+ EXTRA_IP_END=10.151.1.227
2122023-11-06T23:47:32.544Z+ EXTRA_IP_COUNT=64
2132023-11-06T23:47:32.568Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-06T23:47:32.592Z+ EXTRA_IP_BASE=10.151.1
2152023-11-06T23:47:32.615Z+ EXTRA_IP_FOCTET=164
2162023-11-06T23:47:32.639Z+ EXTRA_IP_LOCTET=227
2172023-11-06T23:47:32.669Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-11-06T23:47:32.712Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-11-06T23:47:32.744Z+ DNS_IP1=10.151.1.164
2202023-11-06T23:47:32.770Z+ DNS_IP2=10.151.1.165
2212023-11-06T23:47:32.800Z+ UPLINK_IP=10.151.1.174
2222023-11-06T23:47:32.849Z+ IPPOOL_START=10.151.1.175
2232023-11-06T23:47:32.873Z+ IPPOOL_END=10.151.1.227
2242023-11-06T23:47:32.897Z++ ipadm show-addr -po type,addr
2252023-11-06T23:47:32.921Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-06T23:47:32.945Z+ GATEWAY_IP=10.151.1.100
2272023-11-06T23:47:32.968Z+ [[ -n 10.151.1.100 ]]
2282023-11-06T23:47:32.992Z+ ping -s 10.151.1.100 56 1
2292023-11-06T23:47:33.015ZPING 10.151.1.100: 56 data bytes
2302023-11-06T23:47:33.039Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.012 ms
2312023-11-06T23:47:33.064Z
2322023-11-06T23:47:33.088Z----10.151.1.100 PING Statistics----
2332023-11-06T23:47:33.112Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-06T23:47:33.135Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2352023-11-06T23:47:33.159Z++ arp -an
2362023-11-06T23:47:33.184Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-11-06T23:47:33.213Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-11-06T23:47:33.238Z+ routeadm -e ipv4-forwarding -u
2392023-11-06T23:47:33.262Z+ PXA_START=10.151.1.164
2402023-11-06T23:47:33.286Z+ PXA_END=10.151.1.227
2412023-11-06T23:47:33.309Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-06T23:47:33.333Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-06T23:47:33.357Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-06T23:47:33.382Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-06T23:47:33.405Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-06T23:47:33.429Z++ cd ./tools
2472023-11-06T23:47:33.453Z++ pwd
2482023-11-06T23:47:33.477Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-06T23:47:33.505Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-06T23:47:33.535Z+ SOFTNPU_MODE=zone
2512023-11-06T23:47:33.565Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-06T23:47:33.592Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-06T23:47:33.621Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-06T23:47:33.659Z++ dladm show-phys -p -o LINK
2552023-11-06T23:47:33.700Z++ head -1
2562023-11-06T23:47:33.737Z+ PHYSICAL_LINK=igb0
2572023-11-06T23:47:33.769Z+ echo 'Using igb0 as physical link'
2582023-11-06T23:47:33.800ZUsing igb0 as physical link
2592023-11-06T23:47:33.834Z+ ensure_run_as_root
2602023-11-06T23:47:33.901Z++ id -u
2612023-11-06T23:47:33.928ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2622023-11-06T23:47:33.959Z+ [[ 0 -ne 0 ]]
2632023-11-06T23:47:33.984ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2642023-11-06T23:47:34.008Z+ ensure_zpools
2652023-11-06T23:47:34.033ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2662023-11-06T23:47:34.058Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2672023-11-06T23:47:34.082ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2682023-11-06T23:47:34.106Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2692023-11-06T23:47:34.132ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2702023-11-06T23:47:34.159Z+ readarray -t ZPOOLS
2712023-11-06T23:47:34.183ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2722023-11-06T23:47:34.208Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2732023-11-06T23:47:34.246ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2742023-11-06T23:47:34.281Z++ sed 's/[ ",]//g'
2752023-11-06T23:47:34.307ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2762023-11-06T23:47:34.334Z+ for ZPOOL in "${ZPOOLS[@]}"
2772023-11-06T23:47:34.364ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2782023-11-06T23:47:34.391Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2792023-11-06T23:47:34.415ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2802023-11-06T23:47:34.439Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2812023-11-06T23:47:34.463ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2822023-11-06T23:47:34.486Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2832023-11-06T23:47:34.510ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2842023-11-06T23:47:34.548Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2852023-11-06T23:47:34.575ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2862023-11-06T23:47:34.600Z0+0 records in
2872023-11-06T23:47:34.625ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2882023-11-06T23:47:34.648Z0+0 records out
2892023-11-06T23:47:34.673ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2902023-11-06T23:47:34.698Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2912023-11-06T23:47:34.724ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
2922023-11-06T23:47:34.748Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2932023-11-06T23:47:34.784ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
2942023-11-06T23:47:35.221Z+ set +x
2952023-11-06T23:47:35.247ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
2962023-11-06T23:47:35.272Z++ zpool list -o name
2972023-11-06T23:47:35.296ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
2982023-11-06T23:47:35.320Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2992023-11-06T23:47:35.344ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3002023-11-06T23:47:35.368Z+ [[ -z '' ]]
3012023-11-06T23:47:35.392ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3022023-11-06T23:47:35.416Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
3032023-11-06T23:47:35.440ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3042023-11-06T23:47:35.464Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
3052023-11-06T23:47:35.495ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3062023-11-06T23:47:35.526Z+ set +x
3072023-11-06T23:47:35.551ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3082023-11-06T23:47:35.575Z+ for ZPOOL in "${ZPOOLS[@]}"
3092023-11-06T23:47:35.599ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3102023-11-06T23:47:35.627Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3112023-11-06T23:47:36.197ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3122023-11-06T23:47:36.221Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3132023-11-06T23:47:36.245ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3142023-11-06T23:47:36.275Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3152023-11-06T23:47:36.302ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3162023-11-06T23:47:36.330Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3172023-11-06T23:47:36.357ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3182023-11-06T23:47:36.384Z0+0 records in
3192023-11-06T23:47:36.410ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3202023-11-06T23:47:36.447Z0+0 records out
3212023-11-06T23:47:36.477ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3222023-11-06T23:47:36.504Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3232023-11-06T23:47:36.528ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3242023-11-06T23:47:36.553Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3252023-11-06T23:47:36.577ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3262023-11-06T23:47:36.601Z+ set +x
3272023-11-06T23:47:36.625ZSimnet net0/sc0_0 exists
3282023-11-06T23:47:36.648Z++ zpool list -o name
3292023-11-06T23:47:36.672ZSimnet net1/sc1_0 exists
3302023-11-06T23:47:36.697Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3312023-11-06T23:47:36.721ZVnic sc0_1 exists
3322023-11-06T23:47:36.745Z+ [[ -z '' ]]
3332023-11-06T23:47:36.770Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3342023-11-06T23:47:36.802Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3352023-11-06T23:47:36.830Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3362023-11-06T23:47:36.858Z+ set +x
3372023-11-06T23:47:36.886Z+ for ZPOOL in "${ZPOOLS[@]}"
3382023-11-06T23:47:36.914Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3392023-11-06T23:47:36.944Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3402023-11-06T23:47:36.968Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3412023-11-06T23:47:36.994Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3422023-11-06T23:47:37.020Z0+0 records in
3432023-11-06T23:47:37.044Z0+0 records out
3442023-11-06T23:47:37.068Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3452023-11-06T23:47:37.091Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3462023-11-06T23:47:37.117Z+ set +x
3472023-11-06T23:47:37.142Z++ zpool list -o name
3482023-11-06T23:47:37.173Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3492023-11-06T23:47:37.197Z+ [[ -z '' ]]
3502023-11-06T23:47:37.227Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3512023-11-06T23:47:37.261Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3522023-11-06T23:47:37.288Z+ set +x
3532023-11-06T23:47:37.315Z+ for ZPOOL in "${ZPOOLS[@]}"
3542023-11-06T23:47:37.345Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552023-11-06T23:47:37.381Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562023-11-06T23:47:37.821Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572023-11-06T23:47:37.850Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582023-11-06T23:47:37.877Z0+0 records in
3592023-11-06T23:47:37.903Z0+0 records out
3602023-11-06T23:47:37.928Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3612023-11-06T23:47:37.952Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622023-11-06T23:47:37.978Z+ set +x
3632023-11-06T23:47:38.033Z++ zpool list -o name
3642023-11-06T23:47:38.057Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3652023-11-06T23:47:38.083Z+ [[ -z '' ]]
3662023-11-06T23:47:38.108Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3672023-11-06T23:47:38.196Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3682023-11-06T23:47:38.223Z+ set +x
3692023-11-06T23:47:38.253Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-11-06T23:47:38.284Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3712023-11-06T23:47:38.313Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3722023-11-06T23:47:38.337Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3732023-11-06T23:47:38.361Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3742023-11-06T23:47:38.385Z0+0 records in
3752023-11-06T23:47:38.410Z0+0 records out
3762023-11-06T23:47:38.434Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3772023-11-06T23:47:38.457Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3782023-11-06T23:47:38.481Z+ set +x
3792023-11-06T23:47:38.505Z++ zpool list -o name
3802023-11-06T23:47:38.529Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3812023-11-06T23:47:38.554Z+ [[ -z '' ]]
3822023-11-06T23:47:39.015Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3832023-11-06T23:47:39.054Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3842023-11-06T23:47:39.079Z+ set +x
3852023-11-06T23:47:39.115Z+ for ZPOOL in "${ZPOOLS[@]}"
3862023-11-06T23:47:39.144Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-11-06T23:47:39.170Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-11-06T23:47:39.198Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-11-06T23:47:39.222Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-11-06T23:47:39.246Z0+0 records in
3912023-11-06T23:47:39.270Z0+0 records out
3922023-11-06T23:47:39.294Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3932023-11-06T23:47:39.323Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942023-11-06T23:47:39.351Z+ set +x
3952023-11-06T23:47:39.378Z++ zpool list -o name
3962023-11-06T23:47:39.415Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972023-11-06T23:47:39.445Z+ [[ -z '' ]]
3982023-11-06T23:47:39.470Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992023-11-06T23:47:39.493Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002023-11-06T23:47:39.517Z+ set +x
4012023-11-06T23:47:39.549Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-06T23:47:39.587Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4032023-11-06T23:47:39.622Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-11-06T23:47:39.651Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4052023-11-06T23:47:39.675Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4062023-11-06T23:47:39.705Z0+0 records in
4072023-11-06T23:47:39.729Z0+0 records out
4082023-11-06T23:47:39.755Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4092023-11-06T23:47:39.790Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4102023-11-06T23:47:39.815Z+ set +x
4112023-11-06T23:47:39.844Z++ zpool list -o name
4122023-11-06T23:47:39.871Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4132023-11-06T23:47:39.897Z+ [[ -z '' ]]
4142023-11-06T23:47:39.922Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4152023-11-06T23:47:39.946Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4162023-11-06T23:47:39.977Z+ set +x
4172023-11-06T23:47:40.005Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-11-06T23:47:40.440Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4192023-11-06T23:47:40.474Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4202023-11-06T23:47:40.504Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4212023-11-06T23:47:40.528Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4222023-11-06T23:47:40.552Z0+0 records in
4232023-11-06T23:47:40.576Z0+0 records out
4242023-11-06T23:47:40.599Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4252023-11-06T23:47:40.623Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4262023-11-06T23:47:40.647Z+ set +x
4272023-11-06T23:47:40.677Z++ zpool list -o name
4282023-11-06T23:47:40.702Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4292023-11-06T23:47:40.730Z+ [[ -z '' ]]
4302023-11-06T23:47:40.755Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4312023-11-06T23:47:40.785Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4322023-11-06T23:47:40.811Z+ set +x
4332023-11-06T23:47:40.835Z+ for ZPOOL in "${ZPOOLS[@]}"
4342023-11-06T23:47:40.863Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4352023-11-06T23:47:40.899Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-11-06T23:47:41.390Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4372023-11-06T23:47:41.528Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4382023-11-06T23:47:41.555Z0+0 records in
4392023-11-06T23:47:41.579Z0+0 records out
4402023-11-06T23:47:41.603Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4412023-11-06T23:47:41.630Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4422023-11-06T23:47:41.678Z+ set +x
4432023-11-06T23:47:41.705Z++ zpool list -o name
4442023-11-06T23:47:41.730Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4452023-11-06T23:47:41.755Z+ [[ -z '' ]]
4462023-11-06T23:47:41.785Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4472023-11-06T23:47:41.818Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4482023-11-06T23:47:41.844Z+ set +x
4492023-11-06T23:47:41.880Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4502023-11-06T23:47:41.913Z+ readarray -t ZPOOLS
4512023-11-06T23:47:42.341Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4522023-11-06T23:47:42.369Z++ sed 's/[ ",]//g'
4532023-11-06T23:47:42.395Z+ for ZPOOL in "${ZPOOLS[@]}"
4542023-11-06T23:47:42.423Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4552023-11-06T23:47:42.451Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4562023-11-06T23:47:42.478Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4572023-11-06T23:47:42.526Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4582023-11-06T23:47:42.558Z0+0 records in
4592023-11-06T23:47:42.589Z0+0 records out
4602023-11-06T23:47:42.617Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4612023-11-06T23:47:42.641Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4622023-11-06T23:47:42.668Z+ set +x
4632023-11-06T23:47:42.692Z++ zpool list -o name
4642023-11-06T23:47:42.748Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4652023-11-06T23:47:42.781Z+ [[ -z '' ]]
4662023-11-06T23:47:42.817Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4672023-11-06T23:47:42.845Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4682023-11-06T23:47:42.870Z+ set +x
4692023-11-06T23:47:42.899Z+ for ZPOOL in "${ZPOOLS[@]}"
4702023-11-06T23:47:42.925Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4712023-11-06T23:47:42.951Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4722023-11-06T23:47:42.977Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4732023-11-06T23:47:43.002Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4742023-11-06T23:47:43.026Z0+0 records in
4752023-11-06T23:47:43.050Z0+0 records out
4762023-11-06T23:47:43.074Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4772023-11-06T23:47:43.098Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4782023-11-06T23:47:43.121Z+ set +x
4792023-11-06T23:47:43.145Z++ zpool list -o name
4802023-11-06T23:47:43.179Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4812023-11-06T23:47:43.203Z+ [[ -z '' ]]
4822023-11-06T23:47:43.232Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4832023-11-06T23:47:43.262Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4842023-11-06T23:47:43.296Z+ set +x
4852023-11-06T23:47:43.336Z+ [[ zone == \z\o\n\e ]]
4862023-11-06T23:47:43.364Z+ ensure_simulated_links igb0
4872023-11-06T23:47:43.393Z+ local PHYSICAL_LINK=igb0
4882023-11-06T23:47:43.419Z+ INDICES=("0" "1")
4892023-11-06T23:47:43.448Z+ for I in "${INDICES[@]}"
4902023-11-06T23:47:43.473Z++ get_simnet_name_if_exists net0
4912023-11-06T23:47:43.497Z++ dladm show-simnet -p -o LINK net0
4922023-11-06T23:47:43.520Z++ echo ''
4932023-11-06T23:47:43.544Z+ [[ -z '' ]]
4942023-11-06T23:47:43.568Z+ dladm create-simnet -t net0
4952023-11-06T23:47:43.591Z+ dladm create-simnet -t sc0_0
4962023-11-06T23:47:43.615Z+ dladm modify-simnet -t -p net0 sc0_0
4972023-11-06T23:47:43.650Z+ dladm set-linkprop -p mtu=9000 sc0_0
4982023-11-06T23:47:43.677Z+ success 'Simnet net0/sc0_0 exists'
4992023-11-06T23:47:44.132Z+ set +x
5002023-11-06T23:47:44.160Z+ for I in "${INDICES[@]}"
5012023-11-06T23:47:44.600Z++ get_simnet_name_if_exists net1
5022023-11-06T23:47:44.625Z++ dladm show-simnet -p -o LINK net1
5032023-11-06T23:47:44.650Z++ echo ''
5042023-11-06T23:47:44.683Z+ [[ -z '' ]]
5052023-11-06T23:47:44.737Z+ dladm create-simnet -t net1
5062023-11-06T23:47:44.768Z+ dladm create-simnet -t sc1_0
5072023-11-06T23:47:44.795Z+ dladm modify-simnet -t -p net1 sc1_0
5082023-11-06T23:47:44.819Z+ dladm set-linkprop -p mtu=9000 sc1_0
5092023-11-06T23:47:44.843Z+ success 'Simnet net1/sc1_0 exists'
5102023-11-06T23:47:44.867Z+ set +x
5112023-11-06T23:47:44.891Z++ get_vnic_name_if_exists sc0_1
5122023-11-06T23:47:44.916Z++ dladm show-vnic -p -o LINK sc0_1
5132023-11-06T23:47:44.942Z++ echo ''
5142023-11-06T23:47:44.972Z+ [[ -z '' ]]
5152023-11-06T23:47:44.996Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162023-11-06T23:47:45.430Z+ [[ -v PROMISC_FILT_OFF ]]
5172023-11-06T23:47:45.471Z+ success 'Vnic sc0_1 exists'
5182023-11-06T23:47:45.496Z+ set +x
5192023-11-06T23:47:45.520Z+ warn_if_no_proxy_arp
5202023-11-06T23:47:45.543Z+ [[ -v PXA_START ]]
5212023-11-06T23:47:45.567Z+ [[ -v PXA_END ]]
5222023-11-06T23:47:45.591Z+ ensure_softnpu_zone
5232023-11-06T23:47:45.622Z+ zoneadm list
5242023-11-06T23:47:45.650Z+ grep -q sidecar_softnpu
5252023-11-06T23:47:45.679Z+ [[ -f out/npuzone/npuzone ]]
5262023-11-06T23:47:45.708Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch main
5272023-11-06T23:47:45.741Zdownloading softnpu ...done
5282023-11-06T23:47:45.769Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5292023-11-06T23:47:50.781Zdownloading asic_program.so ...done
5302023-11-06T23:47:50.966Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-06T23:47:57.849Zdownloading scadm ...done
5322023-11-06T23:47:57.876Zcreating port tfportrear0_0 for sc0_0
5332023-11-06T23:47:58.360Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-06T23:47:58.384Zsoftnpu zone setup
5352023-11-06T23:47:58.410Zconfigure zone
5362023-11-06T23:47:58.435Zinstall zone
5372023-11-06T23:47:58.927Zboot zone
5382023-11-06T23:48:02.034Z[sidecar_softnpu] svcs milestone/network
5392023-11-06T23:48:02.061ZSTATE STIME FMRI
5402023-11-06T23:48:02.085Zonline 23:48:00 svc:/milestone/network:default
5412023-11-06T23:48:02.111Z[sidecar_softnpu] /softnpu/npu start
5422023-11-06T23:48:02.136Z
5432023-11-06T23:48:02.185Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-06T23:48:02.214Z+ GATEWAY_IP=10.151.1.100
5452023-11-06T23:48:02.238Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-11-06T23:48:02.262ZUsing 10.151.1.100 as gateway ip
5472023-11-06T23:48:02.291Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-06T23:48:02.318Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-11-06T23:48:02.347Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-06T23:48:02.376Z+++ echo 18:c0:4d:0d:9f:b2
5512023-11-06T23:48:02.408Z+ COUNT=1
5522023-11-06T23:48:02.437Z+ [[ 1 -ne 1 ]]
5532023-11-06T23:48:02.465Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-11-06T23:48:02.492ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-11-06T23:48:02.516Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-11-06T23:48:02.541Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-11-06T23:48:02.567Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-06T23:48:02.601Z+ [[ -v PXA_START ]]
5592023-11-06T23:48:02.640Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-11-06T23:48:02.669Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-11-06T23:48:02.694Z+ z_scadm dump-state
5622023-11-06T23:48:02.718Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-06T23:48:02.755Zlocal v6:
5642023-11-06T23:48:02.779Zlocal v4:
5652023-11-06T23:48:02.815Zrouter v6:
5662023-11-06T23:48:02.845Zrouter v4_idx:
5672023-11-06T23:48:02.872Zrouter v4_routes:
5682023-11-06T23:48:02.898Zresolver v4:
5692023-11-06T23:48:02.924Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-11-06T23:48:02.951Zresolver v6:
5712023-11-06T23:48:02.977Znat v4:
5722023-11-06T23:48:03.002Znat_v6:
5732023-11-06T23:48:03.037Zport_mac:
5742023-11-06T23:48:03.070Zproxy arp:
5752023-11-06T23:48:03.102Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-11-06T23:48:03.128Z+ success 'softnpu zone exists'
5772023-11-06T23:48:03.160Z+ set +x
5782023-11-06T23:48:03.186Zsoftnpu zone exists
5792023-11-06T23:48:03.217Z
5802023-11-06T23:48:03.244Zreal 30.254546342
5812023-11-06T23:48:03.271Zuser 0.701483625
5822023-11-06T23:48:03.296Zsys 1.599964073
5832023-11-06T23:48:03.322Ztrap 0.000102862
5842023-11-06T23:48:03.347Ztflt 0.000096406
5852023-11-06T23:48:03.372Zdflt 0.000845820
5862023-11-06T23:48:03.397Zkflt 0.000000000
5872023-11-06T23:48:03.429Zlock 23:43.045964482
5882023-11-06T23:48:03.455Zslp 1:29.814445158
5892023-11-06T23:48:03.486Zlat 0.073401967
5902023-11-06T23:48:03.519Zstop 0.364501643
5912023-11-06T23:48:03.551Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-06T23:48:03.580Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-06T23:48:03.607Z+ SILO_NAME=recovery
5942023-11-06T23:48:03.631Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-06T23:48:03.659Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-06T23:48:03.685Z+ sed -i~ '
5972023-11-06T23:48:04.118Z /^external_dns_ips/c\
5982023-11-06T23:48:04.142Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-11-06T23:48:04.171Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-06T23:48:04.198Z /^first/c\
6012023-11-06T23:48:04.628Zfirst = "10.151.1.164"
6022023-11-06T23:48:04.654Z /^last/c\
6032023-11-06T23:48:04.688Zlast = "10.151.1.173"
6042023-11-06T23:48:04.723Z }
6052023-11-06T23:48:04.750Z /^\[rack_network_config/,/^$/ {
6062023-11-06T23:48:04.775Z /^infra_ip_first/c\
6072023-11-06T23:48:04.799Zinfra_ip_first = "10.151.1.174"
6082023-11-06T23:48:04.824Z /^infra_ip_last/c\
6092023-11-06T23:48:04.880Zinfra_ip_last = "10.151.1.174"
6102023-11-06T23:48:04.906Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6112023-11-06T23:48:04.931Z }
6122023-11-06T23:48:04.963Z+++ pkg/config-rss.toml Mon Nov 6 23:48:02 2023
6132023-11-06T23:48:04.994Z /^\[\[rack_network_config.ports/,/^$/ {
6142023-11-06T23:48:05.021Z@@ -41,7 +41,7 @@
6152023-11-06T23:48:05.049Z /^routes/c\
6162023-11-06T23:48:05.075Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6172023-11-06T23:48:05.106Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6182023-11-06T23:48:05.132Z # must be contained in one of the "internal services" IP Pool ranges listed
6192023-11-06T23:48:05.166Z /^addresses/c\
6202023-11-06T23:48:05.208Z # below.
6212023-11-06T23:48:05.260Zaddresses = \["10.151.1.174/32"\]
6222023-11-06T23:48:05.294Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6232023-11-06T23:48:05.321Z }
6242023-11-06T23:48:05.346Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6252023-11-06T23:48:05.371Z' pkg/config-rss.toml
6262023-11-06T23:48:05.395Z
6272023-11-06T23:48:05.421Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6282023-11-06T23:48:05.446Z # Initial TLS certificates for the external API
6292023-11-06T23:48:05.471Z+ true
6302023-11-06T23:48:05.495Z #
6312023-11-06T23:48:05.519Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6322023-11-06T23:48:05.544Z@@ -77,8 +77,8 @@
6332023-11-06T23:48:05.568Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6342023-11-06T23:48:05.592Z #
6352023-11-06T23:48:05.617Z+ TLS_NAME=recovery.sys.oxide.test
6362023-11-06T23:48:05.641Z # For more on this and what to put here, see docs/how-to-run.adoc.
6372023-11-06T23:48:05.667Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6382023-11-06T23:48:05.692Z [[internal_services_ip_pool_ranges]]
6392023-11-06T23:48:05.724Z..+...+..........+..+.+............+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+...+..+....+.....+..........+......+.....+.............+...+...+...........+.+..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6402023-11-06T23:48:06.194Z-first = "192.168.1.20"
6412023-11-06T23:48:07.330Z.+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+...................+...+.........+.....+...+.......+.........+..+...+.+...............+........+.+..+...+.+...+...+........+...............+....+...+..+.......+...+..+..........+...+.....+......+.......+..+.+............+.........+.....+.+..+...+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+............+..........+...+..+.............+..+..........+...+.....+...............+......+..........+......+...............+............+.....+.+..+.......+.....+...+...+.+.......................+.......+.....+....+..+..........+............+...+...+..+...+....+.....+...............+...+..........+...+...........+...................+......+...+..+.....................+.......+............+.....+...+.+.....+.+......+..............+.+..+.+............+...+.....+......+.+.....+...+.+.....................+..............+......+..................+.+...+...+..+.......+........+...+....+........+......+.+......+...............+...+.........+...............+.....+...+...+......+.+.........+...........+....+.....+...+......+......+..................+...............+.+..............+.+.....+.....................+...+.............+........+.+......+...............+...+..+.........+......+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6422023-11-06T23:48:07.360Z-last = "192.168.1.29"
6432023-11-06T23:48:07.393Z-----
6442023-11-06T23:48:07.419Z+first = "10.151.1.164"
6452023-11-06T23:48:07.449Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6462023-11-06T23:48:07.490Z+last = "10.151.1.173"
6472023-11-06T23:48:07.535Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6482023-11-06T23:48:07.560Z
6492023-11-06T23:48:07.585Z+ rmdir pkg
6502023-11-06T23:48:07.614Z # TODO - this configuration is subject to change going forward. Ultimately these
6512023-11-06T23:48:07.639Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6522023-11-06T23:48:07.663Z # parameters should be provided to the control plane via wicket, but we need to
6532023-11-06T23:48:07.686Z+ pfexec mkdir -p /usr/oxide
6542023-11-06T23:48:07.711Z@@ -92,8 +92,8 @@
6552023-11-06T23:48:07.735Z+ pfexec rm -f /usr/oxide/tcpproxy
6562023-11-06T23:48:07.759Z # A range of IP addresses used by Boundary Services on the external network. In
6572023-11-06T23:48:07.783Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6582023-11-06T23:48:07.813Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6592023-11-06T23:48:07.838Z+ pfexec chmod +x /usr/oxide/tcpproxy
6602023-11-06T23:48:07.869Z # With softnpu, only one address is used.
6612023-11-06T23:48:07.895Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6622023-11-06T23:48:07.919Z-infra_ip_first = "192.168.1.30"
6632023-11-06T23:48:07.957Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6642023-11-06T23:48:07.983Z-infra_ip_last = "192.168.1.30"
6652023-11-06T23:48:08.008Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6662023-11-06T23:48:08.032Z+infra_ip_first = "10.151.1.174"
6672023-11-06T23:48:08.056Z+ OMICRON_NO_UNINSTALL=1
6682023-11-06T23:48:08.080Z+infra_ip_last = "10.151.1.174"
6692023-11-06T23:48:08.103Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6702023-11-06T23:48:08.127Z
6712023-11-06T23:48:08.152ZNov 06 23:48:04.272 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6722023-11-06T23:48:08.175Z # Configurations for BGP routers to run on the scrimlets.
6732023-11-06T23:48:08.199ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6742023-11-06T23:48:08.227Z bgp = []
6752023-11-06T23:48:08.255ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6762023-11-06T23:48:08.290Z@@ -101,9 +101,9 @@
6772023-11-06T23:48:08.317ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6782023-11-06T23:48:08.347Z # You can configure multiple uplinks by repeating the following stanza
6792023-11-06T23:48:08.375ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6802023-11-06T23:48:08.399Z [[rack_network_config.ports]]
6812023-11-06T23:48:08.423ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6822023-11-06T23:48:08.446Z # Routes associated with this port.
6832023-11-06T23:48:08.470ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6842023-11-06T23:48:08.500Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6852023-11-06T23:48:08.527ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6862023-11-06T23:48:08.551Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6872023-11-06T23:48:08.575ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6882023-11-06T23:48:08.598Z # Addresses associated with this port.
6892023-11-06T23:48:08.622ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6902023-11-06T23:48:08.649Z-addresses = ["192.168.1.30/32"]
6912023-11-06T23:48:08.676ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6922023-11-06T23:48:08.702Z+addresses = ["10.151.1.174/32"]
6932023-11-06T23:48:08.731ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6942023-11-06T23:48:08.755Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6952023-11-06T23:48:08.779ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6962023-11-06T23:48:08.802Z port = "qsfp0"
6972023-11-06T23:48:08.832ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6982023-11-06T23:48:08.857Z # The speed of this port.
6992023-11-06T23:48:08.882ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7002023-11-06T23:48:08.910Za pkg/config-rss.toml 7K
7012023-11-06T23:48:08.937ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7022023-11-06T23:48:08.982Za pkg/initial-tls-cert.pem 2K
7032023-11-06T23:48:09.418ZNov 06 23:48:04.274 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7042023-11-06T23:48:09.449Za pkg/initial-tls-key.pem 4K
7052023-11-06T23:48:09.478ZNov 06 23:48:04.521 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-11-06T23:48:09.501ZNov 06 23:48:04.556 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-06T23:48:09.525ZNov 06 23:48:04.626 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-06T23:48:09.548Z
7092023-11-06T23:48:09.573Zreal 0.493249887
7102023-11-06T23:48:09.597Zuser 0.098161854
7112023-11-06T23:48:09.621Zsys 2.011477488
7122023-11-06T23:48:09.644Ztrap 0.000010057
7132023-11-06T23:48:09.668Ztflt 0.000033268
7142023-11-06T23:48:09.698Zdflt 0.001473656
7152023-11-06T23:48:09.730Zkflt 0.000009709
7162023-11-06T23:48:09.762Zlock 44.345856403
7172023-11-06T23:48:09.789Zslp 0.703771172
7182023-11-06T23:48:09.813Zlat 0.003243619
7192023-11-06T23:48:09.837Zstop 0.212342366
7202023-11-06T23:48:09.861Z+ retry=0
7212023-11-06T23:48:09.884Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-06T23:48:09.910Z+ [[ 0 -gt 30 ]]
7232023-11-06T23:48:09.935Z+ sleep 1
7242023-11-06T23:48:09.959Z+ retry=1
7252023-11-06T23:48:10.025Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-06T23:48:10.051Z+ [[ 1 -gt 30 ]]
7272023-11-06T23:48:10.092Z+ sleep 1
7282023-11-06T23:48:10.120Z+ retry=2
7292023-11-06T23:48:10.144Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-06T23:48:49.894Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7312023-11-06T23:48:50.313Zlocal v6:
7322023-11-06T23:48:50.399Zfe80::102b:f3ff:fe5f:6b81
7332023-11-06T23:48:50.456Zfd00:99::1
7342023-11-06T23:48:50.514Zlocal v4:
7352023-11-06T23:48:50.582Z10.151.1.174
7362023-11-06T23:48:50.628Zrouter v6:
7372023-11-06T23:48:50.689Zfd00:1122:3344:101::/64 -> fe80::aca2:11ff:fe78:76e5 (1)
7382023-11-06T23:48:50.750Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::aca2:11ff:fe78:76e5 (1)
7392023-11-06T23:48:50.814Zrouter v4_idx:
7402023-11-06T23:48:50.881Z0.0.0.0/0 -> 0
7412023-11-06T23:48:50.928Zrouter v4_routes:
7422023-11-06T23:48:50.972Z0 -> 10.151.1.100 (2)
7432023-11-06T23:48:51.029Zresolver v4:
7442023-11-06T23:48:51.140Z10.151.1.174 -> fa:46:bc:b9:69:c6
7452023-11-06T23:48:51.200Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7462023-11-06T23:48:51.367Zresolver v6:
 
7472023-11-06T23:48:51.380Zbuskin console login: Nov 6 23:48:51 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
7482023-11-06T23:48:51.594Zfe80::aca2:11ff:fe78:76e5 -> ae:a2:11:78:76:e5
7492023-11-06T23:48:51.652Zfe80::102b:f3ff:fe5f:6b81 -> 12:2b:f3:5f:6b:81
7502023-11-06T23:48:51.709Znat v4:
7512023-11-06T23:48:51.783Znat_v6:
7522023-11-06T23:48:51.823Zport_mac:
7532023-11-06T23:48:51.857Z1000: a8:40:25:42:61:83
7542023-11-06T23:48:51.895Z1: a8:40:25:42:61:84
7552023-11-06T23:48:51.962Zproxy arp:
7562023-11-06T23:48:51.988Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7572023-11-06T23:48:52.021Z+ retry=0
7582023-11-06T23:48:52.056Z+++ zoneadm list -n
7592023-11-06T23:48:52.097Z+++ grep oxz_ntp
7602023-11-06T23:48:52.141Z++ pfexec svcs -z -Hostate oxide/ntp
7612023-11-06T23:48:52.201Zsvcs: zone '-Hostate': entity not found
7622023-11-06T23:48:52.264Z++ true
7632023-11-06T23:48:52.291Z+ [[ '' != online ]]
7642023-11-06T23:48:52.346Z+ [[ 0 -gt 60 ]]
7652023-11-06T23:48:52.419Z+ sleep 1
7662023-11-06T23:48:52.454Z+ retry=1
7672023-11-06T23:48:52.494Z+++ zoneadm list -n
7682023-11-06T23:48:52.518Z+++ grep oxz_ntp
7692023-11-06T23:48:52.564Z++ pfexec svcs -z -Hostate oxide/ntp
7702023-11-06T23:48:52.587Zsvcs: zone '-Hostate': entity not found
7712023-11-06T23:48:52.613Z++ true
7722023-11-06T23:48:52.637Z+ [[ '' != online ]]
7732023-11-06T23:48:52.661Z+ [[ 1 -gt 60 ]]
7742023-11-06T23:48:52.697Z+ sleep 1
7752023-11-06T23:48:52.729Z+ retry=2
7762023-11-06T23:48:52.753Z+++ zoneadm list -n
7772023-11-06T23:48:52.777Z+++ grep oxz_ntp
7782023-11-06T23:48:52.800Z++ pfexec svcs -z -Hostate oxide/ntp
7792023-11-06T23:48:52.826Zsvcs: zone '-Hostate': entity not found
7802023-11-06T23:48:52.852Z++ true
7812023-11-06T23:48:52.877Z+ [[ '' != online ]]
7822023-11-06T23:48:52.903Z+ [[ 2 -gt 60 ]]
7832023-11-06T23:48:52.927Z+ sleep 1
7842023-11-06T23:48:52.952Z+ retry=3
7852023-11-06T23:48:52.984Z+++ zoneadm list -n
7862023-11-06T23:48:54.234Z+++ grep oxz_ntp
7872023-11-06T23:48:54.263Z++ pfexec svcs -z -Hostate oxide/ntp
7882023-11-06T23:48:54.287Zsvcs: zone '-Hostate': entity not found
7892023-11-06T23:48:54.312Z++ true
7902023-11-06T23:48:54.336Z+ [[ '' != online ]]
7912023-11-06T23:48:54.360Z+ [[ 3 -gt 60 ]]
7922023-11-06T23:48:54.384Z+ sleep 1
7932023-11-06T23:48:54.412Z+ retry=4
7942023-11-06T23:48:54.436Z+++ zoneadm list -n
7952023-11-06T23:48:54.460Z+++ grep oxz_ntp
7962023-11-06T23:48:54.486Z++ pfexec svcs -z -Hostate oxide/ntp
7972023-11-06T23:48:54.511Zsvcs: zone '-Hostate': entity not found
7982023-11-06T23:48:54.535Z++ true
7992023-11-06T23:48:54.559Z+ [[ '' != online ]]
8002023-11-06T23:48:54.582Z+ [[ 4 -gt 60 ]]
8012023-11-06T23:48:54.606Z+ sleep 1
8022023-11-06T23:48:54.632Z+ retry=5
8032023-11-06T23:48:54.661Z+++ zoneadm list -n
8042023-11-06T23:48:54.693Z+++ grep oxz_ntp
8052023-11-06T23:48:54.720Z++ pfexec svcs -z -Hostate oxide/ntp
8062023-11-06T23:48:54.744Zsvcs: zone '-Hostate': entity not found
8072023-11-06T23:48:54.767Z++ true
8082023-11-06T23:48:54.794Z+ [[ '' != online ]]
8092023-11-06T23:48:54.818Z+ [[ 5 -gt 60 ]]
8102023-11-06T23:48:54.843Z+ sleep 1
8112023-11-06T23:48:54.868Z+ retry=6
8122023-11-06T23:48:54.900Z+++ zoneadm list -n
8132023-11-06T23:48:55.333Z+++ grep oxz_ntp
8142023-11-06T23:48:55.368Z++ pfexec svcs -z -Hostate oxide/ntp
8152023-11-06T23:48:55.396Zsvcs: zone '-Hostate': entity not found
8162023-11-06T23:48:55.420Z++ true
8172023-11-06T23:48:55.444Z+ [[ '' != online ]]
8182023-11-06T23:48:55.467Z+ [[ 6 -gt 60 ]]
8192023-11-06T23:48:55.497Z+ sleep 1
8202023-11-06T23:48:55.524Z+ retry=7
8212023-11-06T23:48:55.548ZWaited for chrony: 20s
8222023-11-06T23:48:55.572Z+++ zoneadm list -n
8232023-11-06T23:48:55.598Z+++ grep oxz_ntp
8242023-11-06T23:48:55.627Z++ pfexec svcs -z -Hostate oxide/ntp
8252023-11-06T23:48:55.659Zsvcs: zone '-Hostate': entity not found
8262023-11-06T23:48:55.684Z++ true
8272023-11-06T23:48:55.708Z+ [[ '' != online ]]
8282023-11-06T23:48:55.732Z+ [[ 7 -gt 60 ]]
8292023-11-06T23:48:55.756Z+ sleep 1
8302023-11-06T23:48:55.779Z+ retry=8
8312023-11-06T23:48:55.803Z+++ zoneadm list -n
8322023-11-06T23:48:55.831Z+++ grep oxz_ntp
8332023-11-06T23:48:55.866Z++ pfexec svcs -z -Hostate oxide/ntp
8342023-11-06T23:48:55.891Zsvcs: zone '-Hostate': entity not found
8352023-11-06T23:48:55.915Z++ true
8362023-11-06T23:48:55.939Z+ [[ '' != online ]]
8372023-11-06T23:48:55.965Z+ [[ 8 -gt 60 ]]
8382023-11-06T23:48:55.991Z+ sleep 1
8392023-11-06T23:48:56.016Z+ retry=9
8402023-11-06T23:48:56.085Z+++ zoneadm list -n
8412023-11-06T23:48:56.111Z+++ grep oxz_ntp
8422023-11-06T23:48:56.134Z++ pfexec svcs -z -Hostate oxide/ntp
8432023-11-06T23:48:56.159Zsvcs: zone '-Hostate': entity not found
8442023-11-06T23:48:56.183Z++ true
8452023-11-06T23:48:56.207Z+ [[ '' != online ]]
8462023-11-06T23:48:56.231Z+ [[ 9 -gt 60 ]]
8472023-11-06T23:48:56.256Z+ sleep 1
8482023-11-06T23:48:56.283Z+ retry=10
8492023-11-06T23:48:56.308Z+++ zoneadm list -n
8502023-11-06T23:48:56.332Z+++ grep oxz_ntp
8512023-11-06T23:48:56.355Z++ pfexec svcs -z oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5 -Hostate oxide/ntp
8522023-11-06T23:48:56.379Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8532023-11-06T23:48:56.404Z++ true
8542023-11-06T23:48:56.431Z+ [[ '' != online ]]
8552023-11-06T23:48:56.897Z+ [[ 10 -gt 60 ]]
8562023-11-06T23:48:56.924Z+ sleep 1
8572023-11-06T23:48:56.952Z+ retry=11
8582023-11-06T23:48:56.991Z+++ zoneadm list -n
8592023-11-06T23:48:57.016Z+++ grep oxz_ntp
8602023-11-06T23:48:57.040Z++ pfexec svcs -z oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5 -Hostate oxide/ntp
8612023-11-06T23:48:57.064Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8622023-11-06T23:48:57.116Z++ true
8632023-11-06T23:48:57.141Z+ [[ '' != online ]]
8642023-11-06T23:48:57.164Z+ [[ 11 -gt 60 ]]
8652023-11-06T23:48:57.188Z+ sleep 1
8662023-11-06T23:48:57.212Z+ retry=12
8672023-11-06T23:48:57.236Z+++ zoneadm list -n
8682023-11-06T23:48:57.262Z+++ grep oxz_ntp
8692023-11-06T23:48:57.287Z++ pfexec svcs -z oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5 -Hostate oxide/ntp
8702023-11-06T23:48:57.310Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8712023-11-06T23:48:57.334Z++ true
8722023-11-06T23:48:57.362Z+ [[ '' != online ]]
8732023-11-06T23:48:57.396Z+ [[ 12 -gt 60 ]]
8742023-11-06T23:48:57.429Z+ sleep 1
8752023-11-06T23:48:57.863Z+ retry=13
8762023-11-06T23:48:57.898Z+++ zoneadm list -n
8772023-11-06T23:48:57.931Z+++ grep oxz_ntp
8782023-11-06T23:48:57.963Z++ pfexec svcs -z oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5 -Hostate oxide/ntp
8792023-11-06T23:48:57.989Z+ [[ disabled != online ]]
8802023-11-06T23:48:58.013Z+ [[ 13 -gt 60 ]]
8812023-11-06T23:48:58.037Z+ sleep 1
8822023-11-06T23:48:58.061Z+ retry=14
8832023-11-06T23:48:58.084Z+++ zoneadm list -n
8842023-11-06T23:48:58.108Z+++ grep oxz_ntp
8852023-11-06T23:48:58.147Z++ pfexec svcs -z oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5 -Hostate oxide/ntp
8862023-11-06T23:48:58.182Z+ [[ disabled != online ]]
8872023-11-06T23:48:58.213Z+ [[ 14 -gt 60 ]]
8882023-11-06T23:48:58.245Z+ sleep 1
8892023-11-06T23:48:58.272Z+ retry=15
8902023-11-06T23:48:58.298Z+++ zoneadm list -n
8912023-11-06T23:48:58.322Z+++ grep oxz_ntp
8922023-11-06T23:48:58.346Z++ pfexec svcs -z oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5 -Hostate oxide/ntp
8932023-11-06T23:48:58.375Z+ [[ disabled != online ]]
8942023-11-06T23:48:58.402Z+ [[ 15 -gt 60 ]]
8952023-11-06T23:48:58.426Z+ sleep 1
8962023-11-06T23:48:58.450Z+ retry=16
8972023-11-06T23:48:58.475Z+++ zoneadm list -n
8982023-11-06T23:48:58.499Z+++ grep oxz_ntp
8992023-11-06T23:48:58.523Z++ pfexec svcs -z oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5 -Hostate oxide/ntp
9002023-11-06T23:48:58.546Z+ [[ disabled != online ]]
9012023-11-06T23:48:58.570Z+ [[ 16 -gt 60 ]]
9022023-11-06T23:48:58.594Z+ sleep 1
9032023-11-06T23:48:58.619Z+ retry=17
9042023-11-06T23:48:58.646Z+++ zoneadm list -n
9052023-11-06T23:48:58.673Z+++ grep oxz_ntp
9062023-11-06T23:48:58.698Z++ pfexec svcs -z oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5 -Hostate oxide/ntp
9072023-11-06T23:48:58.726Z+ [[ disabled != online ]]
9082023-11-06T23:48:58.753Z+ [[ 17 -gt 60 ]]
9092023-11-06T23:48:58.778Z+ sleep 1
9102023-11-06T23:48:58.804Z+ retry=18
9112023-11-06T23:48:58.831Z+++ zoneadm list -n
9122023-11-06T23:48:58.855Z+++ grep oxz_ntp
9132023-11-06T23:48:58.879Z++ pfexec svcs -z oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5 -Hostate oxide/ntp
9142023-11-06T23:48:58.902Z+ [[ disabled != online ]]
9152023-11-06T23:48:58.926Z+ [[ 18 -gt 60 ]]
9162023-11-06T23:48:58.952Z+ sleep 1
9172023-11-06T23:48:58.980Z+ retry=19
9182023-11-06T23:48:59.006Z+++ zoneadm list -n
9192023-11-06T23:48:59.031Z+++ grep oxz_ntp
9202023-11-06T23:48:59.055Z++ pfexec svcs -z oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5 -Hostate oxide/ntp
9212023-11-06T23:48:59.080Z+ [[ disabled != online ]]
9222023-11-06T23:48:59.106Z+ [[ 19 -gt 60 ]]
9232023-11-06T23:48:59.129Z+ sleep 1
9242023-11-06T23:48:59.202Z+ retry=20
9252023-11-06T23:48:59.266Z+++ zoneadm list -n
9262023-11-06T23:48:59.299Z+++ grep oxz_ntp
9272023-11-06T23:48:59.327Z++ pfexec svcs -z oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5 -Hostate oxide/ntp
9282023-11-06T23:48:59.379Z+ [[ online != online ]]
9292023-11-06T23:48:59.420Z+ echo 'Waited for chrony: 20s'
9302023-11-06T23:48:59.457Z+ export RUST_BACKTRACE=1
9312023-11-06T23:48:59.500Z+ RUST_BACKTRACE=1
9322023-11-06T23:48:59.525Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9332023-11-06T23:48:59.549Z+ ./tests/bootstrap
9342023-11-06T23:48:59.573ZUsing external DNS server at 10.151.1.164:53
9352023-11-06T23:48:59.598Z2023-11-06 23:48:52.525126108 UTC: attempting to log into API
9362023-11-06T23:49:08.584Z2023-11-06 23:49:07.039081945 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9372023-11-06T23:49:09.746Z2023-11-06 23:49:08.040917769 UTC: attempting to log into API
9382023-11-06T23:49:33.994Z2023-11-06 23:49:23.073749172 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9392023-11-06T23:49:34.416Z2023-11-06 23:49:24.074714923 UTC: attempting to log into API
9402023-11-06T23:49:39.108Z2023-11-06 23:49:39.107450925 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9412023-11-06T23:49:40.112Z2023-11-06 23:49:40.108511928 UTC: attempting to log into API
9422023-11-06T23:49:55.144Z2023-11-06 23:49:55.142161132 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9432023-11-06T23:49:56.145Z2023-11-06 23:49:56.144001956 UTC: attempting to log into API
9442023-11-06T23:50:11.181Z2023-11-06 23:50:11.171041870 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9452023-11-06T23:50:12.179Z2023-11-06 23:50:12.170633893 UTC: attempting to log into API
9462023-11-06T23:50:27.965Z2023-11-06 23:50:27.201618323 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9472023-11-06T23:50:28.213Z2023-11-06 23:50:28.202730067 UTC: attempting to log into API
9482023-11-06T23:50:43.249Z2023-11-06 23:50:43.236389818 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9492023-11-06T23:50:44.248Z2023-11-06 23:50:44.237373360 UTC: attempting to log into API
9502023-11-06T23:50:59.282Z2023-11-06 23:50:59.270907750 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9512023-11-06T23:51:00.289Z2023-11-06 23:51:00.272102505 UTC: attempting to log into API
9522023-11-06T23:51:15.315Z2023-11-06 23:51:15.304787010 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9532023-11-06T23:51:16.318Z2023-11-06 23:51:16.306508973 UTC: attempting to log into API
9542023-11-06T23:51:31.350Z2023-11-06 23:51:31.340157774 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9552023-11-06T23:51:32.353Z2023-11-06 23:51:32.341429890 UTC: attempting to log into API
 
9562023-11-06T23:51:39.978ZNov 6 23:51:39 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9572023-11-06T23:51:51.414ZNov 6 23:51:46 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9582023-11-06T23:51:51.733Z2023-11-06 23:51:47.377262432 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9592023-11-06T23:51:51.809Z2023-11-06 23:51:48.378461318 UTC: attempting to log into API
 
9602023-11-06T23:51:54.113ZNov 6 23:51:53 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9612023-11-06T23:51:59.373ZNov 6 23:51:59 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9622023-11-06T23:52:03.426Z2023-11-06 23:52:03.412174509 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9632023-11-06T23:52:04.424Z2023-11-06 23:52:04.414087095 UTC: attempting to log into API
 
9642023-11-06T23:52:07.126ZNov 6 23:52:06 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9652023-11-06T23:52:19.458Z2023-11-06 23:52:19.446963815 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9662023-11-06T23:52:20.462Z2023-11-06 23:52:20.447955016 UTC: attempting to log into API
9672023-11-06T23:52:21.251Z2023-11-06 23:52:21.235966634 UTC: login succeeded
9682023-11-06T23:52:21.281Zwaiting for nexus to come up...
9692023-11-06T23:52:21.358Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9702023-11-06T23:52:21.413Zensuring datasets are ready...
9712023-11-06T23:52:24.037Zlet's roll.
9722023-11-06T23:52:24.061Z+ rm ./tests/bootstrap
9732023-11-06T23:52:24.108Z+ for test_bin in tests/*
9742023-11-06T23:52:24.152Z+ ./tests/bootstrap-346a6c0cadffef35
9752023-11-06T23:52:24.184Z
9762023-11-06T23:52:24.218Zrunning 0 tests
9772023-11-06T23:52:24.255Z
9782023-11-06T23:52:24.296Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9792023-11-06T23:52:24.333Z
9802023-11-06T23:52:24.368Z+ for test_bin in tests/*
9812023-11-06T23:52:24.401Z+ ./tests/end_to_end_tests-ad8d578e5459ec9d
9822023-11-06T23:52:24.433Z
9832023-11-06T23:52:24.462Zrunning 2 tests
9842023-11-06T23:52:24.507Ztest no_spoof::no_spoof_here ... ok
9852023-11-06T23:53:24.127Ztest instance_launch::instance_launch has been running for over 60 seconds
9862023-11-06T23:55:21.985Ztest instance_launch::instance_launch ... ok
9872023-11-06T23:55:22.023Z
9882023-11-06T23:55:22.486Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 177.57s
9892023-11-06T23:55:22.531Z
9902023-11-06T23:55:22.569Z+ _exit_trap
9912023-11-06T23:55:22.600Z+ local status=0
9922023-11-06T23:55:23.040Z+ [[ 0 -eq 0 ]]
9932023-11-06T23:55:23.069Z+ exit 0
9942023-11-06T23:55:23.106Zprocess exited: duration 478228 ms, exit code 0
 
9952023-11-06T23:55:23.222Zfound 47 output files
9962023-11-06T23:55:23.308Zuploading: /var/svc/log/oxide-sled-agent:default.log (663034 bytes)
9972023-11-06T23:55:25.287Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 663034 -> 663434
9982023-11-06T23:55:26.421Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9992023-11-06T23:55:26.464Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1efc32ac-580c-408f-9f6b-3d9f4f706d0c/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
10002023-11-06T23:55:28.014Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1efc32ac-580c-408f-9f6b-3d9f4f706d0c/root/var/svc/log/oxide-crucible-agent:default.log
10012023-11-06T23:55:28.465Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1efc32ac-580c-408f-9f6b-3d9f4f706d0c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10022023-11-06T23:55:29.977Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1efc32ac-580c-408f-9f6b-3d9f4f706d0c/root/var/svc/log/oxide-crucible-downstairs:default.log
10032023-11-06T23:55:30.014Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1efc32ac-580c-408f-9f6b-3d9f4f706d0c/root/var/svc/log/oxide-crucible-downstairs:downstairs-babc8b04-236b-4203-baf8-7ae9f2ae6496.log (5537 bytes)
10042023-11-06T23:55:31.125Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1efc32ac-580c-408f-9f6b-3d9f4f706d0c/root/var/svc/log/oxide-crucible-downstairs:downstairs-babc8b04-236b-4203-baf8-7ae9f2ae6496.log
10052023-11-06T23:55:31.156Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c495ea32-1144-4c27-a185-19bdd385d9e1/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10062023-11-06T23:55:32.247Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c495ea32-1144-4c27-a185-19bdd385d9e1/root/var/svc/log/oxide-crucible-agent:default.log
10072023-11-06T23:55:32.288Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c495ea32-1144-4c27-a185-19bdd385d9e1/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10082023-11-06T23:55:33.370Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c495ea32-1144-4c27-a185-19bdd385d9e1/root/var/svc/log/oxide-crucible-downstairs:default.log
10092023-11-06T23:55:33.408Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_7209febe-d979-41ad-9679-0b0d99bd3edc/root/var/svc/log/oxide-nexus:default.log (4390641 bytes)
10102023-11-06T23:55:37.817Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_7209febe-d979-41ad-9679-0b0d99bd3edc/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4390641 -> 4391249
10112023-11-06T23:55:38.917Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_7209febe-d979-41ad-9679-0b0d99bd3edc/root/var/svc/log/oxide-nexus:default.log
10122023-11-06T23:55:38.949Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_96c50585-9958-453a-928f-793c08e9021a/root/var/svc/log/oxide-nexus:default.log (619743 bytes)
10132023-11-06T23:55:40.207Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_96c50585-9958-453a-928f-793c08e9021a/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 619743 -> 620959
10142023-11-06T23:55:41.489Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_96c50585-9958-453a-928f-793c08e9021a/root/var/svc/log/oxide-nexus:default.log
10152023-11-06T23:55:41.519Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_12a8b70f-af8b-41cf-82e1-3dc35dd2773d/root/var/svc/log/oxide-external_dns:default.log (15610 bytes)
10162023-11-06T23:55:42.468Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_12a8b70f-af8b-41cf-82e1-3dc35dd2773d/root/var/svc/log/oxide-external_dns:default.log
10172023-11-06T23:55:42.908Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_adbb50a3-4cf7-42ec-a937-b1ccd10c42bb/root/var/svc/log/oxide-internal_dns:default.log (99893 bytes)
10182023-11-06T23:55:44.327Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_adbb50a3-4cf7-42ec-a937-b1ccd10c42bb/root/var/svc/log/oxide-internal_dns:default.log
10192023-11-06T23:55:44.352Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_99a55caa-46ee-4dcb-8efa-eec1086507f4/root/var/svc/log/oxide-oximeter:default.log (203493 bytes)
10202023-11-06T23:55:44.561Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_99a55caa-46ee-4dcb-8efa-eec1086507f4/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 203493 -> 221205
10212023-11-06T23:55:45.640Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_99a55caa-46ee-4dcb-8efa-eec1086507f4/root/var/svc/log/oxide-oximeter:default.log
10222023-11-06T23:55:45.666Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_7ff9fd71-f620-49ff-b1b2-49714d8a744e/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10232023-11-06T23:55:46.756Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_7ff9fd71-f620-49ff-b1b2-49714d8a744e/root/var/svc/log/oxide-cockroachdb:default.log
10242023-11-06T23:55:46.793Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_747aaa55-fda3-4b4a-8b46-3bec4077a9a5/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10252023-11-06T23:55:47.874Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_747aaa55-fda3-4b4a-8b46-3bec4077a9a5/root/var/svc/log/oxide-crucible-pantry:default.log
10262023-11-06T23:55:47.902Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_79d1beb7-67e6-4c3d-808c-7fdec20a642f/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10272023-11-06T23:55:48.977Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_79d1beb7-67e6-4c3d-808c-7fdec20a642f/root/var/svc/log/oxide-crucible-pantry:default.log
10282023-11-06T23:55:49.002Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_5821ca0e-91b7-41ae-be95-91b92679cf01/root/var/svc/log/oxide-cockroachdb:default.log (5344 bytes)
10292023-11-06T23:55:50.078Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_5821ca0e-91b7-41ae-be95-91b92679cf01/root/var/svc/log/oxide-cockroachdb:default.log
10302023-11-06T23:55:50.107Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0953c62e-ab75-480e-8430-73e26f85442c/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10312023-11-06T23:55:51.599Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0953c62e-ab75-480e-8430-73e26f85442c/root/var/svc/log/oxide-crucible-agent:default.log
10322023-11-06T23:55:51.624Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0953c62e-ab75-480e-8430-73e26f85442c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10332023-11-06T23:55:52.696Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0953c62e-ab75-480e-8430-73e26f85442c/root/var/svc/log/oxide-crucible-downstairs:default.log
10342023-11-06T23:55:52.720Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_5e2d9b2a-3b6c-4f46-a5e7-fbe3ae8ceb3f/root/var/svc/log/oxide-internal_dns:default.log (104837 bytes)
10352023-11-06T23:55:53.941Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_5e2d9b2a-3b6c-4f46-a5e7-fbe3ae8ceb3f/root/var/svc/log/oxide-internal_dns:default.log
10362023-11-06T23:55:53.969Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_b4e07ae4-2ad5-4de8-a00c-555f1d18d4d4/root/var/svc/log/oxide-nexus:default.log (890218 bytes)
10372023-11-06T23:55:54.756Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_b4e07ae4-2ad5-4de8-a00c-555f1d18d4d4/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 890218 -> 892044
10382023-11-06T23:55:55.836Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_b4e07ae4-2ad5-4de8-a00c-555f1d18d4d4/root/var/svc/log/oxide-nexus:default.log
10392023-11-06T23:55:55.861Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_3d6728a2-6100-4e59-b33f-405a54c66596/root/var/svc/log/oxide-clickhouse:default.log (779375 bytes)
10402023-11-06T23:55:56.148Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_3d6728a2-6100-4e59-b33f-405a54c66596/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 779375 -> 895607
10412023-11-06T23:55:57.239Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_3d6728a2-6100-4e59-b33f-405a54c66596/root/var/svc/log/oxide-clickhouse:default.log
10422023-11-06T23:55:57.267Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_5c40973f-4645-474f-a5f1-ff8813dab731/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10432023-11-06T23:55:58.343Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_5c40973f-4645-474f-a5f1-ff8813dab731/root/var/svc/log/oxide-cockroachdb:default.log
10442023-11-06T23:55:58.367Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_5acd3579-1108-4616-b491-5cd98dfda48c/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10452023-11-06T23:55:59.451Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_5acd3579-1108-4616-b491-5cd98dfda48c/root/var/svc/log/oxide-crucible-agent:default.log
10462023-11-06T23:55:59.478Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_5acd3579-1108-4616-b491-5cd98dfda48c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10472023-11-06T23:56:00.559Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_5acd3579-1108-4616-b491-5cd98dfda48c/root/var/svc/log/oxide-crucible-downstairs:default.log
10482023-11-06T23:56:00.629Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c6ccf119-d081-4357-a12a-ccf704591da1/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
10492023-11-06T23:56:01.725Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c6ccf119-d081-4357-a12a-ccf704591da1/root/var/svc/log/oxide-crucible-agent:default.log
10502023-11-06T23:56:01.768Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c6ccf119-d081-4357-a12a-ccf704591da1/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10512023-11-06T23:56:02.878Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c6ccf119-d081-4357-a12a-ccf704591da1/root/var/svc/log/oxide-crucible-downstairs:default.log
10522023-11-06T23:56:02.910Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c6ccf119-d081-4357-a12a-ccf704591da1/root/var/svc/log/oxide-crucible-downstairs:downstairs-45e8ada5-8285-4799-9f6f-7143f0ecb4ea.log (3051 bytes)
10532023-11-06T23:56:03.970Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c6ccf119-d081-4357-a12a-ccf704591da1/root/var/svc/log/oxide-crucible-downstairs:downstairs-45e8ada5-8285-4799-9f6f-7143f0ecb4ea.log
10542023-11-06T23:56:04.025Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_fb5de800-f8ff-41d5-99ff-7e8f3708e93e/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10552023-11-06T23:56:05.138Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_fb5de800-f8ff-41d5-99ff-7e8f3708e93e/root/var/svc/log/oxide-crucible-pantry:default.log
10562023-11-06T23:56:05.184Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_c739215f-bc69-4443-a0b2-d817a1b88a38/root/var/svc/log/oxide-external_dns:default.log (15612 bytes)
10572023-11-06T23:56:05.256Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_c739215f-bc69-4443-a0b2-d817a1b88a38/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15612 -> 16871
10582023-11-06T23:56:06.391Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_c739215f-bc69-4443-a0b2-d817a1b88a38/root/var/svc/log/oxide-external_dns:default.log
10592023-11-06T23:56:06.416Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_912ef9e0-ff85-40ea-8f0c-a24658de764e/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
10602023-11-06T23:56:07.485Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_912ef9e0-ff85-40ea-8f0c-a24658de764e/root/var/svc/log/oxide-cockroachdb:default.log
10612023-11-06T23:56:07.518Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5699dec2-ece9-47af-bc5b-da5fbfc8d84f/root/var/svc/log/oxide-crucible-agent:default.log (14118 bytes)
10622023-11-06T23:56:08.748Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5699dec2-ece9-47af-bc5b-da5fbfc8d84f/root/var/svc/log/oxide-crucible-agent:default.log
10632023-11-06T23:56:08.777Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5699dec2-ece9-47af-bc5b-da5fbfc8d84f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10642023-11-06T23:56:09.841Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5699dec2-ece9-47af-bc5b-da5fbfc8d84f/root/var/svc/log/oxide-crucible-downstairs:default.log
10652023-11-06T23:56:09.894Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5699dec2-ece9-47af-bc5b-da5fbfc8d84f/root/var/svc/log/oxide-crucible-downstairs:downstairs-d91fc639-0930-4d47-984b-bf5cddec6e0a.log (5536 bytes)
10662023-11-06T23:56:10.991Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5699dec2-ece9-47af-bc5b-da5fbfc8d84f/root/var/svc/log/oxide-crucible-downstairs:downstairs-d91fc639-0930-4d47-984b-bf5cddec6e0a.log
10672023-11-06T23:56:11.024Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ce084777-80a7-4f13-83b8-ecfd86bfff2d/root/var/svc/log/oxide-crucible-agent:default.log (14580 bytes)
10682023-11-06T23:56:12.142Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ce084777-80a7-4f13-83b8-ecfd86bfff2d/root/var/svc/log/oxide-crucible-agent:default.log
10692023-11-06T23:56:12.177Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ce084777-80a7-4f13-83b8-ecfd86bfff2d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10702023-11-06T23:56:13.843Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ce084777-80a7-4f13-83b8-ecfd86bfff2d/root/var/svc/log/oxide-crucible-downstairs:default.log
10712023-11-06T23:56:13.971Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ce084777-80a7-4f13-83b8-ecfd86bfff2d/root/var/svc/log/oxide-crucible-downstairs:downstairs-a904c7a5-8392-4367-8f8c-406aff2752a4.log (3050 bytes)
10722023-11-06T23:56:27.514Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ce084777-80a7-4f13-83b8-ecfd86bfff2d/root/var/svc/log/oxide-crucible-downstairs:downstairs-a904c7a5-8392-4367-8f8c-406aff2752a4.log
10732023-11-06T23:56:27.676Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_9d7f6186-41eb-4f0f-8ab3-a438b3f09267/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10742023-11-06T23:56:29.194Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_9d7f6186-41eb-4f0f-8ab3-a438b3f09267/root/var/svc/log/oxide-cockroachdb:default.log
10752023-11-06T23:56:29.314Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2c92a24b-087e-475b-b5d3-a4269f347790/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10762023-11-06T23:56:30.428Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2c92a24b-087e-475b-b5d3-a4269f347790/root/var/svc/log/oxide-crucible-agent:default.log
10772023-11-06T23:56:30.542Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2c92a24b-087e-475b-b5d3-a4269f347790/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10782023-11-06T23:56:31.668Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2c92a24b-087e-475b-b5d3-a4269f347790/root/var/svc/log/oxide-crucible-downstairs:default.log
10792023-11-06T23:56:31.762Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_71af4c59-f24d-4a80-b32f-6e56e0fb05d8/root/var/svc/log/oxide-internal_dns:default.log (99568 bytes)
10802023-11-06T23:56:31.896Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_71af4c59-f24d-4a80-b32f-6e56e0fb05d8/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99568 -> 103324
10812023-11-06T23:56:33.045Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_71af4c59-f24d-4a80-b32f-6e56e0fb05d8/root/var/svc/log/oxide-internal_dns:default.log
10822023-11-06T23:56:33.116Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5/root/var/svc/log/oxide-ntp:default.log (1248 bytes)
10832023-11-06T23:56:34.265Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5/root/var/svc/log/oxide-ntp:default.log
10842023-11-06T23:56:34.371Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b9b4460c-28f5-4ec9-a283-b8ac697c39d6/root/var/svc/log/oxide-crucible-agent:default.log (25292 bytes)
10852023-11-06T23:56:41.107Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b9b4460c-28f5-4ec9-a283-b8ac697c39d6/root/var/svc/log/oxide-crucible-agent:default.log
10862023-11-06T23:56:41.252Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b9b4460c-28f5-4ec9-a283-b8ac697c39d6/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10872023-11-06T23:56:42.352Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b9b4460c-28f5-4ec9-a283-b8ac697c39d6/root/var/svc/log/oxide-crucible-downstairs:default.log
10882023-11-06T23:56:42.400Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b9b4460c-28f5-4ec9-a283-b8ac697c39d6/root/var/svc/log/oxide-crucible-downstairs:downstairs-1df9b571-1c3c-41e9-af61-c9415f0ac1f9.log (3051 bytes)
10892023-11-06T23:56:43.458Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b9b4460c-28f5-4ec9-a283-b8ac697c39d6/root/var/svc/log/oxide-crucible-downstairs:downstairs-1df9b571-1c3c-41e9-af61-c9415f0ac1f9.log
10902023-11-06T23:56:43.487Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b9b4460c-28f5-4ec9-a283-b8ac697c39d6/root/var/svc/log/oxide-crucible-downstairs:downstairs-5c07c082-ae15-4553-9735-4ef1c9741084.log (5537 bytes)
10912023-11-06T23:56:44.581Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b9b4460c-28f5-4ec9-a283-b8ac697c39d6/root/var/svc/log/oxide-crucible-downstairs:downstairs-5c07c082-ae15-4553-9735-4ef1c9741084.log
10922023-11-06T23:56:44.610Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5/root/var/log/chrony/measurements.log (18221 bytes)
10932023-11-06T23:56:44.708Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5/root/var/log/chrony/measurements.log" changed size mid upload: 18221 -> 22879
10942023-11-06T23:56:45.798Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5/root/var/log/chrony/measurements.log
10952023-11-06T23:56:45.825Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5/root/var/log/chrony/statistics.log (9711 bytes)
10962023-11-06T23:56:45.922Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5/root/var/log/chrony/statistics.log" changed size mid upload: 9711 -> 13104
10972023-11-06T23:56:47.021Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5/root/var/log/chrony/statistics.log
10982023-11-06T23:56:47.052Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5/root/var/log/chrony/tracking.log (1721 bytes)
10992023-11-06T23:56:47.082Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5/root/var/log/chrony/tracking.log" changed size mid upload: 1721 -> 1985
11002023-11-06T23:56:48.167Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f0808ded-e680-4347-9de6-3bbbb1cf9de5/root/var/log/chrony/tracking.log