01HEJPEBVRVSGTTYDP2GGX5HTB: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HEJPF19VCMW521ERJHKCTXM7

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-06T17:03:23.944Zjob dependencies complete; ready to run (waiting for 51 m 37 s)
22023-11-06T17:04:10.408Zjob assigned to worker 01HEJSDQYTJ758QF7GWFZD510W (queued for 46 s)
32023-11-06T17:04:10.661Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-11-06T17:04:10.705Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-11-06T17:04:10.740Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-11-06T17:04:10.778Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-06T17:04:10.818Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-11-06T17:04:10.850ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-06T17:04:10.883Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-06T17:04:10.918ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-06T17:04:10.950ZConfiguring devices.
122023-11-06T17:04:10.986Z#################################################
132023-11-06T17:04:11.017Z#### ############# ##
142023-11-06T17:04:11.051Z### ### ############ ##
152023-11-06T17:04:11.084Z## ### # ## ### ## Oxide Computer Company ##
162023-11-06T17:04:11.117Z## ## ## ### # ### ##
172023-11-06T17:04:11.151Z## # ### #### #### This Station Under ##
182023-11-06T17:04:11.211Z### ### #### # ### Computer Control ##
192023-11-06T17:04:11.252Z#### #### ### ## ##
202023-11-06T17:04:11.285Z##################################################
212023-11-06T17:04:11.318ZHostname: sock
222023-11-06T17:04:11.351ZLoading smf(7) serice descriptions:
232023-11-06T17:04:12.455Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-00f20c85209291d2.gz
242023-11-06T17:04:15.242Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-00f20c85209291d2.gz
252023-11-06T17:04:15.314Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-06T17:04:38.818Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-06T17:04:38.848Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2a5726111815475a.gz
282023-11-06T17:05:09.514Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2a5726111815475a.gz
292023-11-06T17:05:09.540Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-06T17:05:11.276Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-06T17:05:11.301Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-06T17:05:17.100Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-06T17:05:17.125Zdownloading input: /input/package/work/version.txt
342023-11-06T17:05:17.166Zdownloaded input: /input/package/work/version.txt
352023-11-06T17:05:17.193Zdownloading input: /input/package/work/package.tar.gz
362023-11-06T17:05:55.913Zdownloaded input: /input/package/work/package.tar.gz
372023-11-06T17:05:55.938Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-06T17:06:30.180Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-06T17:06:30.204Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-06T17:06:38.274Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-06T17:06:38.298Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-06T17:07:21.163Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-06T17:07:21.603Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-06T17:08:10.035Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-06T17:08:10.060Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-06T17:08:45.910Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-06T17:08:45.934Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-06T17:08:51.525Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-06T17:08:51.556Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-06T17:09:01.860Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-06T17:09:01.891Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-06T17:09:06.976Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-06T17:09:07.000Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-06T17:09:11.258Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-06T17:09:11.282Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-06T17:09:11.306Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-06T17:09:11.330Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-06T17:09:14.811Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-06T17:09:14.835Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-06T17:09:18.114Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-06T17:09:18.146Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-06T17:09:30.668Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-06T17:09:30.692Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-06T17:09:44.024Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-06T17:09:44.048Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-06T17:09:44.072Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-06T17:09:44.096Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-06T17:09:47.597Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-06T17:09:47.621Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-06T17:09:59.397Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-06T17:09:59.422Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-06T17:10:47.796Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-06T17:10:47.843Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-06T17:11:23.980Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-06T17:11:24.005Zstarting task 0: "setup"
762023-11-06T17:11:24.038Z++ uname -s
772023-11-06T17:11:24.062Z+ kern=SunOS
782023-11-06T17:11:24.087Z+ case "$kern" in
792023-11-06T17:11:24.112Z+ groupadd -g 12345 build
802023-11-06T17:11:24.136Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-06T17:11:26.047Z+ zfs create -o mountpoint=/work rpool/work
822023-11-06T17:11:26.072Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-06T17:11:26.096Z+ home_fs=zfs
842023-11-06T17:11:26.120Z+ [[ zfs == autofs ]]
852023-11-06T17:11:26.144Z+ mkdir -p /home/build
862023-11-06T17:11:26.169Z+ chown build:build /home/build /work
872023-11-06T17:11:28.072Z+ chmod 0700 /home/build /work
882023-11-06T17:11:28.096Zprocess exited: duration 4045 ms, exit code 0
 
892023-11-06T17:11:28.144Zstarting task 1: "authentication"
902023-11-06T17:11:28.213Zprocess exited: duration 44 ms, exit code 0
 
912023-11-06T17:11:28.262Zstarting task 2: "build"
922023-11-06T17:11:28.301Z+ trap _exit_trap EXIT
932023-11-06T17:11:28.328Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-06T17:11:28.491Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-06T17:11:28.515Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-06T17:11:28.539Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-06T17:11:28.563Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-06T17:11:28.588Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-06T17:11:28.611Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-06T17:11:28.635Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-06T17:11:28.659Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-06T17:11:28.683Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-06T17:11:28.707Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-06T17:11:28.732Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-06T17:11:28.755Z+ [[ -d /opt/oxide ]]
1062023-11-06T17:11:28.790Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-06T17:11:28.814Z+ pfexec mkdir /opt/oxide
1082023-11-06T17:11:28.837Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-06T17:11:28.862Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-06T17:11:28.885Z+ cd /opt/oxide-underneath
1112023-11-06T17:11:28.909Z+ pfexec tar ceEp@/f - .
1122023-11-06T17:11:28.933Z+ cd /opt/oxide
1132023-11-06T17:11:28.964Z+ pfexec tar xveEp@/f -
1142023-11-06T17:11:28.988Zx ., 0 bytes, 0 tape blocks
1152023-11-06T17:11:29.014Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-06T17:11:29.038Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-06T17:11:29.061Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-06T17:11:29.085Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-06T17:11:29.109Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-06T17:11:29.133Z+ pfexec mkdir /opt/oxide/work
1212023-11-06T17:11:29.157Z+ pfexec chown build:build /opt/oxide/work
1222023-11-06T17:11:29.181Z+ cd /opt/oxide/work
1232023-11-06T17:11:29.206Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-06T17:11:30.255ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-06T17:11:30.282Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1262023-11-06T17:11:30.306Zx out/omicron-sled-agent.tar, 122518016 bytes, 239293 tape blocks
1272023-11-06T17:11:30.330Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-06T17:11:30.354Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-11-06T17:11:30.378Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-11-06T17:11:30.402Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-11-06T17:11:30.426Zx target/release/omicron-package, 41743424 bytes, 81531 tape blocks
1322023-11-06T17:11:30.450Zx tools/create_virtual_hardware.sh, 2974 bytes, 6 tape blocks
1332023-11-06T17:11:30.475Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-06T17:11:30.502Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-06T17:11:30.527Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-06T17:11:30.551Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-06T17:11:30.576Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-06T17:11:30.600Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-06T17:11:30.623Z
1402023-11-06T17:11:30.648Zreal 1.602581118
1412023-11-06T17:11:30.672Zuser 1.444701971
1422023-11-06T17:11:30.696Zsys 0.281983948
1432023-11-06T17:11:30.720Ztrap 0.000051601
1442023-11-06T17:11:30.744Ztflt 0.000000000
1452023-11-06T17:11:30.768Zdflt 0.000000000
1462023-11-06T17:11:30.792Zkflt 0.000000000
1472023-11-06T17:11:30.843Zlock 0.000000000
1482023-11-06T17:11:30.867Zslp 1.441328149
1492023-11-06T17:11:30.891Zlat 0.034954617
1502023-11-06T17:11:30.920Zstop 0.000118721
1512023-11-06T17:11:30.945Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-06T17:11:31.309Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-06T17:11:31.334Z+ mkdir tests
1542023-11-06T17:11:31.358Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-06T17:11:31.382Z+ ptime -m gunzip
1562023-11-06T17:11:31.407Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-00f20c85209291d2
1572023-11-06T17:11:31.465Z
1582023-11-06T17:11:31.489Zreal 0.141145205
1592023-11-06T17:11:31.513Zuser 0.129688312
1602023-11-06T17:11:31.537Zsys 0.011218987
1612023-11-06T17:11:31.561Ztrap 0.000013000
1622023-11-06T17:11:31.585Ztflt 0.000000000
1632023-11-06T17:11:31.609Zdflt 0.000000000
1642023-11-06T17:11:31.633Zkflt 0.000000000
1652023-11-06T17:11:31.657Zlock 0.000000000
1662023-11-06T17:11:31.681Zslp 0.000018900
1672023-11-06T17:11:31.705Zlat 0.000022370
1682023-11-06T17:11:31.729Zstop 0.000121042
1692023-11-06T17:11:31.752Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-00f20c85209291d2
1702023-11-06T17:11:31.777Z+ chmod a+x tests/bootstrap-00f20c85209291d2
1712023-11-06T17:11:31.802Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-06T17:11:31.826Z+ ptime -m gunzip
1732023-11-06T17:11:31.867Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-06T17:11:33.791Z
1752023-11-06T17:11:33.819Zreal 2.312791643
1762023-11-06T17:11:33.842Zuser 2.164973559
1772023-11-06T17:11:33.867Zsys 0.147423956
1782023-11-06T17:11:33.918Ztrap 0.000170364
1792023-11-06T17:11:34.357Ztflt 0.000000000
1802023-11-06T17:11:34.392Zdflt 0.000000000
1812023-11-06T17:11:34.428Zkflt 0.000000000
1822023-11-06T17:11:34.458Zlock 0.000000000
1832023-11-06T17:11:34.484Zslp 0.000021070
1842023-11-06T17:11:34.514Zlat 0.000021480
1852023-11-06T17:11:34.544Zstop 0.000123302
1862023-11-06T17:11:34.569Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-06T17:11:34.593Z+ chmod a+x tests/bootstrap
1882023-11-06T17:11:34.617Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-06T17:11:34.642Z+ ptime -m gunzip
1902023-11-06T17:11:34.667Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2a5726111815475a
1912023-11-06T17:11:36.419Z
1922023-11-06T17:11:36.443Zreal 2.618413440
1932023-11-06T17:11:36.467Zuser 2.457666665
1942023-11-06T17:11:36.491Zsys 0.160297647
1952023-11-06T17:11:36.515Ztrap 0.000213304
1962023-11-06T17:11:36.539Ztflt 0.000000000
1972023-11-06T17:11:36.563Zdflt 0.000000000
1982023-11-06T17:11:36.588Zkflt 0.000000000
1992023-11-06T17:11:36.611Zlock 0.000000000
2002023-11-06T17:11:36.638Zslp 0.000020880
2012023-11-06T17:11:36.663Zlat 0.000023210
2022023-11-06T17:11:36.687Zstop 0.000129711
2032023-11-06T17:11:36.714Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2a5726111815475a
2042023-11-06T17:11:36.744Z+ chmod a+x tests/end_to_end_tests-2a5726111815475a
2052023-11-06T17:11:36.772Z+ bmat address ls
2062023-11-06T17:11:36.796ZNAME CIDR FIRST LAST COUNT
2072023-11-06T17:11:36.821Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-11-06T17:11:36.845Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-06T17:11:36.871Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-11-06T17:11:36.896Z+ EXTRA_IP_START=10.151.2.164
2112023-11-06T17:11:36.920Z+ EXTRA_IP_END=10.151.2.227
2122023-11-06T17:11:36.943Z+ EXTRA_IP_COUNT=64
2132023-11-06T17:11:36.974Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-06T17:11:37.007Z+ EXTRA_IP_BASE=10.151.2
2152023-11-06T17:11:37.033Z+ EXTRA_IP_FOCTET=164
2162023-11-06T17:11:37.057Z+ EXTRA_IP_LOCTET=227
2172023-11-06T17:11:37.084Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-11-06T17:11:37.109Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-11-06T17:11:37.132Z+ DNS_IP1=10.151.2.164
2202023-11-06T17:11:37.156Z+ DNS_IP2=10.151.2.165
2212023-11-06T17:11:37.189Z+ UPLINK_IP=10.151.2.174
2222023-11-06T17:11:37.212Z+ IPPOOL_START=10.151.2.175
2232023-11-06T17:11:37.236Z+ IPPOOL_END=10.151.2.227
2242023-11-06T17:11:37.260Z++ ipadm show-addr -po type,addr
2252023-11-06T17:11:37.284Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-06T17:11:37.308Z+ GATEWAY_IP=10.151.2.100
2272023-11-06T17:11:37.332Z+ [[ -n 10.151.2.100 ]]
2282023-11-06T17:11:37.356Z+ ping -s 10.151.2.100 56 1
2292023-11-06T17:11:37.380ZPING 10.151.2.100: 56 data bytes
2302023-11-06T17:11:37.407Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2312023-11-06T17:11:37.432Z
2322023-11-06T17:11:37.458Z----10.151.2.100 PING Statistics----
2332023-11-06T17:11:37.528Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-06T17:11:37.553Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-11-06T17:11:37.577Z++ arp -an
2362023-11-06T17:11:37.601Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-11-06T17:11:37.625Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-11-06T17:11:37.649Z+ routeadm -e ipv4-forwarding -u
2392023-11-06T17:11:37.674Z+ PXA_START=10.151.2.164
2402023-11-06T17:11:37.698Z+ PXA_END=10.151.2.227
2412023-11-06T17:11:37.722Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-06T17:11:37.745Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-06T17:11:37.770Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-06T17:11:37.797Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-06T17:11:37.821Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-06T17:11:37.844Z++ cd ./tools
2472023-11-06T17:11:37.868Z++ pwd
2482023-11-06T17:11:37.892Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-06T17:11:37.925Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-06T17:11:37.951Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-11-06T17:11:37.975Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-11-06T17:11:38.000Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-11-06T17:11:38.031Z++ dladm show-phys -p -o LINK
2542023-11-06T17:11:38.056Z++ head -1
2552023-11-06T17:11:38.080Z+ PHYSICAL_LINK=igb0
2562023-11-06T17:11:38.106Z+ echo 'Using igb0 as physical link'
2572023-11-06T17:11:38.130ZUsing igb0 as physical link
2582023-11-06T17:11:38.154Z+ ensure_run_as_root
2592023-11-06T17:11:38.179Z++ id -u
2602023-11-06T17:11:38.202Z+ [[ 0 -ne 0 ]]
2612023-11-06T17:11:38.227Z+ ensure_zpools
2622023-11-06T17:11:38.251Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-11-06T17:11:38.276Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-11-06T17:11:38.300Z+ readarray -t ZPOOLS
2652023-11-06T17:11:38.325Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-11-06T17:11:38.356Z++ sed 's/[ ",]//g'
2672023-11-06T17:11:38.393Z+ for ZPOOL in "${ZPOOLS[@]}"
2682023-11-06T17:11:38.421Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2692023-11-06T17:11:38.447ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2702023-11-06T17:11:38.471Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2712023-11-06T17:11:38.495Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2722023-11-06T17:11:38.520Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2732023-11-06T17:11:38.543Z0+0 records in
2742023-11-06T17:11:38.568Z0+0 records out
2752023-11-06T17:11:38.592Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2762023-11-06T17:11:38.616Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2772023-11-06T17:11:38.640ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2782023-11-06T17:11:38.664Z+ set +x
2792023-11-06T17:11:38.688ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2802023-11-06T17:11:38.712Z++ zpool list -o name
2812023-11-06T17:11:38.736ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2822023-11-06T17:11:38.760Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2832023-11-06T17:11:38.784ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2842023-11-06T17:11:38.809Z+ [[ -z '' ]]
2852023-11-06T17:11:38.840ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2862023-11-06T17:11:38.867Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2872023-11-06T17:11:38.891ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2882023-11-06T17:11:38.915Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2892023-11-06T17:11:38.940ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2902023-11-06T17:11:38.963Z+ set +x
2912023-11-06T17:11:38.987ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2922023-11-06T17:11:39.012Z+ for ZPOOL in "${ZPOOLS[@]}"
2932023-11-06T17:11:39.060ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2942023-11-06T17:11:39.084Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2952023-11-06T17:11:39.108ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2962023-11-06T17:11:39.132Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2972023-11-06T17:11:39.156ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2982023-11-06T17:11:39.180Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2992023-11-06T17:11:39.204ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3002023-11-06T17:11:39.228Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3012023-11-06T17:11:39.257ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3022023-11-06T17:11:39.286Z0+0 records in
3032023-11-06T17:11:39.311ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3042023-11-06T17:11:39.336Z0+0 records out
3052023-11-06T17:11:39.360ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3062023-11-06T17:11:39.384Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3072023-11-06T17:11:39.408ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3082023-11-06T17:11:39.432Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3092023-11-06T17:11:39.456ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3102023-11-06T17:11:39.480Z+ set +x
3112023-11-06T17:11:39.504ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3122023-11-06T17:11:39.528Z++ zpool list -o name
3132023-11-06T17:11:39.552ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3142023-11-06T17:11:39.576Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3152023-11-06T17:11:39.600ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3162023-11-06T17:11:39.624Z+ [[ -z '' ]]
3172023-11-06T17:11:39.648ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3182023-11-06T17:11:39.672Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3192023-11-06T17:11:40.109ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3202023-11-06T17:11:40.135Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3212023-11-06T17:11:40.166ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3222023-11-06T17:11:40.196Z+ set +x
3232023-11-06T17:11:40.221ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3242023-11-06T17:11:40.244Z+ for ZPOOL in "${ZPOOLS[@]}"
3252023-11-06T17:11:40.269ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3262023-11-06T17:11:40.292Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-11-06T17:11:40.318ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3282023-11-06T17:11:40.342Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-11-06T17:11:40.366ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3302023-11-06T17:11:40.390Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3312023-11-06T17:11:40.415ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3322023-11-06T17:11:40.439Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3332023-11-06T17:11:40.463ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3342023-11-06T17:11:40.487Z0+0 records in
3352023-11-06T17:11:40.511ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3362023-11-06T17:11:40.536Z0+0 records out
3372023-11-06T17:11:40.560ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3382023-11-06T17:11:40.592Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3392023-11-06T17:11:41.056ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3402023-11-06T17:11:41.148Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3412023-11-06T17:11:41.174ZSimnet net0/sc0_0 exists
3422023-11-06T17:11:41.199Z+ set +x
3432023-11-06T17:11:41.629ZSimnet net1/sc1_0 exists
3442023-11-06T17:11:41.653Z++ zpool list -o name
3452023-11-06T17:11:41.677ZVnic sc0_1 exists
3462023-11-06T17:11:41.710Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3472023-11-06T17:11:41.735Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3482023-11-06T17:11:41.759Z+ [[ -z '' ]]
3492023-11-06T17:11:41.782Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-11-06T17:11:41.806Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-11-06T17:11:41.830Z+ set +x
3522023-11-06T17:11:41.854Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-11-06T17:11:41.878Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-11-06T17:11:41.909Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-11-06T17:11:41.942Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-11-06T17:11:41.970Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-11-06T17:11:41.994Z0+0 records in
3582023-11-06T17:11:42.019Z0+0 records out
3592023-11-06T17:11:42.043Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3602023-11-06T17:11:42.080Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-11-06T17:11:42.104Z+ set +x
3622023-11-06T17:11:42.131Z++ zpool list -o name
3632023-11-06T17:11:42.172Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-11-06T17:11:42.196Z+ [[ -z '' ]]
3652023-11-06T17:11:42.221Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-11-06T17:11:42.245Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-11-06T17:11:42.269Z+ set +x
3682023-11-06T17:11:42.293Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-11-06T17:11:42.317Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-11-06T17:11:42.347Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-11-06T17:11:42.392Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-11-06T17:11:42.420Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-11-06T17:11:42.446Z0+0 records in
3742023-11-06T17:11:42.473Z0+0 records out
3752023-11-06T17:11:42.497Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3762023-11-06T17:11:42.521Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-11-06T17:11:42.597Z+ set +x
3782023-11-06T17:11:42.624Z++ zpool list -o name
3792023-11-06T17:11:42.652Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-11-06T17:11:42.676Z+ [[ -z '' ]]
3812023-11-06T17:11:42.700Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-11-06T17:11:42.724Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-11-06T17:11:42.748Z+ set +x
3842023-11-06T17:11:42.776Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-11-06T17:11:42.811Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862023-11-06T17:11:42.841Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872023-11-06T17:11:42.866Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882023-11-06T17:11:42.889Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892023-11-06T17:11:42.913Z0+0 records in
3902023-11-06T17:11:42.937Z0+0 records out
3912023-11-06T17:11:42.961Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-11-06T17:11:42.985Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932023-11-06T17:11:43.008Z+ set +x
3942023-11-06T17:11:43.032Z++ zpool list -o name
3952023-11-06T17:11:43.056Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3962023-11-06T17:11:43.085Z+ [[ -z '' ]]
3972023-11-06T17:11:43.111Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3982023-11-06T17:11:43.135Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3992023-11-06T17:11:43.159Z+ set +x
4002023-11-06T17:11:43.203Z+ for ZPOOL in "${ZPOOLS[@]}"
4012023-11-06T17:11:43.644Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4022023-11-06T17:11:43.673Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4032023-11-06T17:11:43.699Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4042023-11-06T17:11:43.722Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4052023-11-06T17:11:43.746Z0+0 records in
4062023-11-06T17:11:43.770Z0+0 records out
4072023-11-06T17:11:43.794Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4082023-11-06T17:11:43.819Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4092023-11-06T17:11:43.843Z+ set +x
4102023-11-06T17:11:43.867Z++ zpool list -o name
4112023-11-06T17:11:43.891Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4122023-11-06T17:11:43.918Z+ [[ -z '' ]]
4132023-11-06T17:11:43.943Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4142023-11-06T17:11:43.973Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4152023-11-06T17:11:44.004Z+ set +x
4162023-11-06T17:11:44.031Z+ for ZPOOL in "${ZPOOLS[@]}"
4172023-11-06T17:11:44.055Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4182023-11-06T17:11:44.488Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192023-11-06T17:11:44.525Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4202023-11-06T17:11:44.552Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4212023-11-06T17:11:44.577Z0+0 records in
4222023-11-06T17:11:44.601Z0+0 records out
4232023-11-06T17:11:44.625Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4242023-11-06T17:11:44.648Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4252023-11-06T17:11:44.672Z+ set +x
4262023-11-06T17:11:44.696Z++ zpool list -o name
4272023-11-06T17:11:44.721Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4282023-11-06T17:11:44.745Z+ [[ -z '' ]]
4292023-11-06T17:11:44.769Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4302023-11-06T17:11:44.796Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4312023-11-06T17:11:44.821Z+ set +x
4322023-11-06T17:11:44.845Z+ for ZPOOL in "${ZPOOLS[@]}"
4332023-11-06T17:11:44.876Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4342023-11-06T17:11:44.902Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4352023-11-06T17:11:44.926Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4362023-11-06T17:11:44.954Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4372023-11-06T17:11:44.978Z0+0 records in
4382023-11-06T17:11:45.002Z0+0 records out
4392023-11-06T17:11:45.026Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4402023-11-06T17:11:45.060Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4412023-11-06T17:11:45.084Z+ set +x
4422023-11-06T17:11:45.108Z++ zpool list -o name
4432023-11-06T17:11:45.132Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4442023-11-06T17:11:45.156Z+ [[ -z '' ]]
4452023-11-06T17:11:45.180Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4462023-11-06T17:11:45.204Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4472023-11-06T17:11:45.252Z+ set +x
4482023-11-06T17:11:45.276Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4492023-11-06T17:11:45.302Z+ readarray -t ZPOOLS
4502023-11-06T17:11:45.330Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4512023-11-06T17:11:45.357Z++ sed 's/[ ",]//g'
4522023-11-06T17:11:45.384Z+ for ZPOOL in "${ZPOOLS[@]}"
4532023-11-06T17:11:45.424Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4542023-11-06T17:11:45.900Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4552023-11-06T17:11:45.928Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4562023-11-06T17:11:45.954Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4572023-11-06T17:11:45.983Z0+0 records in
4582023-11-06T17:11:46.017Z0+0 records out
4592023-11-06T17:11:46.043Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4602023-11-06T17:11:46.070Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4612023-11-06T17:11:46.100Z+ set +x
4622023-11-06T17:11:46.129Z++ zpool list -o name
4632023-11-06T17:11:46.163Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4642023-11-06T17:11:46.194Z+ [[ -z '' ]]
4652023-11-06T17:11:46.224Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-11-06T17:11:46.338Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4672023-11-06T17:11:46.372Z+ set +x
4682023-11-06T17:11:46.402Z+ for ZPOOL in "${ZPOOLS[@]}"
4692023-11-06T17:11:46.429Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4702023-11-06T17:11:46.455Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4712023-11-06T17:11:46.479Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4722023-11-06T17:11:46.503Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4732023-11-06T17:11:46.526Z0+0 records in
4742023-11-06T17:11:46.550Z0+0 records out
4752023-11-06T17:11:46.574Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4762023-11-06T17:11:47.002Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4772023-11-06T17:11:47.026Z+ set +x
4782023-11-06T17:11:47.050Z++ zpool list -o name
4792023-11-06T17:11:47.074Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4802023-11-06T17:11:47.098Z+ [[ -z '' ]]
4812023-11-06T17:11:47.122Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4822023-11-06T17:11:47.146Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4832023-11-06T17:11:47.170Z+ set +x
4842023-11-06T17:11:47.194Z+ ensure_simulated_links igb0
4852023-11-06T17:11:47.218Z+ local PHYSICAL_LINK=igb0
4862023-11-06T17:11:47.246Z+ INDICES=("0" "1")
4872023-11-06T17:11:47.275Z+ for I in "${INDICES[@]}"
4882023-11-06T17:11:47.300Z++ get_simnet_name_if_exists net0
4892023-11-06T17:11:47.325Z++ dladm show-simnet -p -o LINK net0
4902023-11-06T17:11:47.354Z++ echo ''
4912023-11-06T17:11:47.378Z+ [[ -z '' ]]
4922023-11-06T17:11:47.404Z+ dladm create-simnet -t net0
4932023-11-06T17:11:47.428Z+ dladm create-simnet -t sc0_0
4942023-11-06T17:11:47.457Z+ dladm modify-simnet -t -p net0 sc0_0
4952023-11-06T17:11:48.022Z+ dladm set-linkprop -p mtu=9000 sc0_0
4962023-11-06T17:11:48.046Z+ success 'Simnet net0/sc0_0 exists'
4972023-11-06T17:11:48.070Z+ set +x
4982023-11-06T17:11:48.094Z+ for I in "${INDICES[@]}"
4992023-11-06T17:11:48.118Z++ get_simnet_name_if_exists net1
5002023-11-06T17:11:48.147Z++ dladm show-simnet -p -o LINK net1
5012023-11-06T17:11:48.176Z++ echo ''
5022023-11-06T17:11:48.201Z+ [[ -z '' ]]
5032023-11-06T17:11:48.225Z+ dladm create-simnet -t net1
5042023-11-06T17:11:48.250Z+ dladm create-simnet -t sc1_0
5052023-11-06T17:11:48.278Zdownloading softnpu ...done
5062023-11-06T17:11:48.303Z+ dladm modify-simnet -t -p net1 sc1_0
5072023-11-06T17:11:48.330Z+ dladm set-linkprop -p mtu=9000 sc1_0
5082023-11-06T17:11:48.768Z+ success 'Simnet net1/sc1_0 exists'
5092023-11-06T17:11:48.796Z+ set +x
5102023-11-06T17:11:48.820Z++ get_vnic_name_if_exists sc0_1
5112023-11-06T17:11:48.844Z++ dladm show-vnic -p -o LINK sc0_1
5122023-11-06T17:11:48.868Z++ echo ''
5132023-11-06T17:11:48.892Z+ [[ -z '' ]]
5142023-11-06T17:11:48.917Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5152023-11-06T17:11:48.941Z+ [[ -v PROMISC_FILT_OFF ]]
5162023-11-06T17:11:48.965Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5172023-11-06T17:11:48.989Z+ success 'Vnic sc0_1 exists'
5182023-11-06T17:11:49.014Z+ set +x
5192023-11-06T17:11:49.044Z+ warn_if_no_proxy_arp
5202023-11-06T17:11:49.074Z+ [[ -v PXA_START ]]
5212023-11-06T17:11:49.098Z+ [[ -v PXA_END ]]
5222023-11-06T17:11:49.122Z+ ensure_softnpu_zone
5232023-11-06T17:11:49.147Z+ zoneadm list
5242023-11-06T17:11:49.171Z+ grep -q sidecar_softnpu
5252023-11-06T17:11:49.195Z+ [[ -f out/npuzone/npuzone ]]
5262023-11-06T17:11:49.219Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch main
5272023-11-06T17:11:52.287Zdownloading asic_program.so ...done
5282023-11-06T17:11:52.324Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5292023-11-06T17:11:57.554Zdownloading scadm ...done
5302023-11-06T17:11:57.579Zcreating port tfportrear0_0 for sc0_0
5312023-11-06T17:11:57.603Zcreating port tfportqsfp0_0 for sc0_1
5322023-11-06T17:11:57.627Zsoftnpu zone setup
5332023-11-06T17:11:57.650Zconfigure zone
5342023-11-06T17:11:57.674Zinstall zone
5352023-11-06T17:11:58.654Zboot zone
5362023-11-06T17:12:01.762Z[sidecar_softnpu] svcs milestone/network
5372023-11-06T17:12:01.789ZSTATE STIME FMRI
5382023-11-06T17:12:01.814Zonline 17:11:59 svc:/milestone/network:default
5392023-11-06T17:12:01.838Z[sidecar_softnpu] /softnpu/npu start
5402023-11-06T17:12:01.863Z
5412023-11-06T17:12:01.887Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5422023-11-06T17:12:01.912Z+ GATEWAY_IP=10.151.2.100
5432023-11-06T17:12:01.937Z+ echo 'Using 10.151.2.100 as gateway ip'
5442023-11-06T17:12:01.962ZUsing 10.151.2.100 as gateway ip
5452023-11-06T17:12:01.987Z+ [[ ! -v GATEWAY_MAC ]]
5462023-11-06T17:12:02.013Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5472023-11-06T17:12:02.037Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5482023-11-06T17:12:02.063Z+++ echo 18:c0:4d:0d:a0:2a
5492023-11-06T17:12:02.088Z+ COUNT=1
5502023-11-06T17:12:02.113Z+ [[ 1 -ne 1 ]]
5512023-11-06T17:12:02.137Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5522023-11-06T17:12:02.163ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5532023-11-06T17:12:02.190Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5542023-11-06T17:12:02.215Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5552023-11-06T17:12:02.239Z+ PXA_MAC=a8:e1:de:01:70:1d
5562023-11-06T17:12:02.265Z+ [[ -v PXA_START ]]
5572023-11-06T17:12:02.289Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5582023-11-06T17:12:02.314Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5592023-11-06T17:12:02.340Z+ z_scadm dump-state
5602023-11-06T17:12:02.365Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5612023-11-06T17:12:02.389Zlocal v6:
5622023-11-06T17:12:02.423Zlocal v4:
5632023-11-06T17:12:02.448Zrouter v6:
5642023-11-06T17:12:02.500Zrouter v4_idx:
5652023-11-06T17:12:02.525Zrouter v4_routes:
5662023-11-06T17:12:02.550Zresolver v4:
5672023-11-06T17:12:02.574Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5682023-11-06T17:12:02.599Zresolver v6:
5692023-11-06T17:12:02.623Znat v4:
5702023-11-06T17:12:02.648Znat_v6:
5712023-11-06T17:12:02.708Zport_mac:
5722023-11-06T17:12:02.733Zproxy arp:
5732023-11-06T17:12:02.758Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5742023-11-06T17:12:02.821Z+ success 'softnpu zone exists'
5752023-11-06T17:12:02.853Z+ set +x
5762023-11-06T17:12:02.878Zsoftnpu zone exists
5772023-11-06T17:12:02.902Z
5782023-11-06T17:12:02.973Zreal 25.103438096
5792023-11-06T17:12:02.998Zuser 0.722888123
5802023-11-06T17:12:03.024Zsys 1.592362812
5812023-11-06T17:12:03.049Ztrap 0.000093676
5822023-11-06T17:12:03.076Ztflt 0.000199897
5832023-11-06T17:12:03.103Zdflt 0.001202289
5842023-11-06T17:12:03.128Zkflt 0.000000000
5852023-11-06T17:12:03.155Zlock 19:36.578163151
5862023-11-06T17:12:03.180Zslp 1:14.491510524
5872023-11-06T17:12:03.208Zlat 0.055979240
5882023-11-06T17:12:03.245Zstop 0.370434632
5892023-11-06T17:12:03.288Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5902023-11-06T17:12:03.314Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5912023-11-06T17:12:03.343Z+ SILO_NAME=recovery
5922023-11-06T17:12:03.368Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-06T17:12:03.393Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5942023-11-06T17:12:03.421Z+ sed -i~ '
5952023-11-06T17:12:03.448Z /^external_dns_ips/c\
5962023-11-06T17:12:03.474Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5972023-11-06T17:12:03.500Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5982023-11-06T17:12:03.525Z /^first/c\
5992023-11-06T17:12:03.549Zfirst = "10.151.2.164"
6002023-11-06T17:12:03.574Z /^last/c\
6012023-11-06T17:12:03.599Zlast = "10.151.2.173"
6022023-11-06T17:12:03.624Z }
6032023-11-06T17:12:03.651Z /^\[rack_network_config/,/^$/ {
6042023-11-06T17:12:03.677Z /^infra_ip_first/c\
6052023-11-06T17:12:03.702Zinfra_ip_first = "10.151.2.174"
6062023-11-06T17:12:03.748Z /^infra_ip_last/c\
6072023-11-06T17:12:03.772Zinfra_ip_last = "10.151.2.174"
6082023-11-06T17:12:03.798Z }
6092023-11-06T17:12:03.827Z /^\[\[rack_network_config.ports/,/^$/ {
6102023-11-06T17:12:03.854Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6112023-11-06T17:12:03.884Z /^routes/c\
6122023-11-06T17:12:03.910Z+++ pkg/config-rss.toml Mon Nov 6 17:12:01 2023
6132023-11-06T17:12:03.939Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6142023-11-06T17:12:03.964Z@@ -41,7 +41,7 @@
6152023-11-06T17:12:03.989Z /^addresses/c\
6162023-11-06T17:12:04.017Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6172023-11-06T17:12:04.042Zaddresses = \["10.151.2.174/32"\]
6182023-11-06T17:12:04.067Z # must be contained in one of the "internal services" IP Pool ranges listed
6192023-11-06T17:12:04.092Z }
6202023-11-06T17:12:04.116Z # below.
6212023-11-06T17:12:04.146Z' pkg/config-rss.toml
6222023-11-06T17:12:04.170Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6232023-11-06T17:12:04.195Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6242023-11-06T17:12:04.220Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6252023-11-06T17:12:04.250Z+ true
6262023-11-06T17:12:04.275Z
6272023-11-06T17:12:04.299Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6282023-11-06T17:12:04.324Z # Initial TLS certificates for the external API
6292023-11-06T17:12:04.349Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6302023-11-06T17:12:04.374Z #
6312023-11-06T17:12:04.399Z+ TLS_NAME=recovery.sys.oxide.test
6322023-11-06T17:12:04.424Z@@ -77,8 +77,8 @@
6332023-11-06T17:12:04.448Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6342023-11-06T17:12:04.473Z #
6352023-11-06T17:12:04.512Z # For more on this and what to put here, see docs/how-to-run.adoc.
6362023-11-06T17:12:04.540Z [[internal_services_ip_pool_ranges]]
6372023-11-06T17:12:04.564Z-first = "192.168.1.20"
6382023-11-06T17:12:04.590Z-last = "192.168.1.29"
6392023-11-06T17:12:04.615Z+first = "10.151.2.164"
6402023-11-06T17:12:04.640Z+last = "10.151.2.173"
6412023-11-06T17:12:04.665Z
6422023-11-06T17:12:04.691Z # TODO - this configuration is subject to change going forward. Ultimately these
6432023-11-06T17:12:04.718Z # parameters should be provided to the control plane via wicket, but we need to
6442023-11-06T17:12:04.743Z@@ -92,8 +92,8 @@
6452023-11-06T17:12:04.796Z # A range of IP addresses used by Boundary Services on the external network. In
6462023-11-06T17:12:04.829Z...+...+.+..+....+...+........+...+......+...+......+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.......+...+..+.......+.....+..........+...+...+.....+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+...........+.+........+.....................+.+...............+...............+.....+.......+.....+................+.....+.......+..+.............+...+..+...+......+.........+.+......+............+..+...+...+............+......+.......+..+...+.........+.+...+........................+..+.........+.+...+..+.........+........................+...+....+...+...............+...+...+............+..+.....................+.+......+........+..........+..+.........+......+...+................+...............+..................+.....................+.....+..........+...............+............+........+....+......+......+.....+.............+.....+...+.............+..+.........+....................................+...+.......+..............+..........+........+............+.+........+..................................+...+..+.+......+.....+...+............+...+...+.......+............+......+..............+...+...+....+...+..+..........+.........+...........+....+...........+....+............+..+.......+.........+.....+....+.....+...+....+........+.+.....+.+........+...+...+.+...+...+..+...............+...+.........+.+.....+..........+...+..+.......+.....+................+............+...+..+.........+....+..............+...+....+...........+.......+..+......+..........+.......................+....+...+...+....................+......+.+........+.+.....+.+.........+......+...+............+..+.+..+......................+.....+......+.+...+..................+........+.........+...+...+....+..+....+.....+............+.......+..+...+.+......+......+..+.+..+...+.+.....+.....................+.........+.........+.+.....+.............+....................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6472023-11-06T17:12:04.855Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6482023-11-06T17:12:04.882Z # With softnpu, only one address is used.
6492023-11-06T17:12:04.907Z-infra_ip_first = "192.168.1.30"
6502023-11-06T17:12:04.933Z-infra_ip_last = "192.168.1.30"
6512023-11-06T17:12:04.961Z+infra_ip_first = "10.151.2.174"
6522023-11-06T17:12:04.988Z+infra_ip_last = "10.151.2.174"
6532023-11-06T17:12:05.012Z
6542023-11-06T17:12:05.037Z # Configurations for BGP routers to run on the scrimlets.
6552023-11-06T17:12:05.113Z bgp = []
6562023-11-06T17:12:05.139Z@@ -101,9 +101,9 @@
6572023-11-06T17:12:05.163Z # You can configure multiple uplinks by repeating the following stanza
6582023-11-06T17:12:05.188Z [[rack_network_config.ports]]
6592023-11-06T17:12:05.213Z # Routes associated with this port.
6602023-11-06T17:12:05.240Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6612023-11-06T17:12:05.266Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6622023-11-06T17:12:05.291Z # Addresses associated with this port.
6632023-11-06T17:12:05.316Z-addresses = ["192.168.1.30/32"]
6642023-11-06T17:12:05.340Z+addresses = ["10.151.2.174/32"]
6652023-11-06T17:12:05.365Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6662023-11-06T17:12:05.389Z port = "qsfp0"
6672023-11-06T17:12:05.414Z # The speed of this port.
6682023-11-06T17:12:05.439Z...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.....+...............+.+............+.....+.....................+....+...+...+.....+......+...+......+.+......+..+......+................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...........+......+.+..+.......+...+.....+...+..........+...............+.....+.........+................+........+...............+..............................+....+..+....+.........+..+.+.....+.......+...........+.........+.+.....+................+........+.......+...........+...+......+.............+.....+...............+...............+.+.........+..+....+.....+......+...+.+..+..........+............+........+...............+............+.....................+.......+...+.................+...+............+....+.........+.........+...+.........+..............+................+...+......+...........+..........+........+......+....+......+.....+....+...............+.....+......+..........+..+........................+......+....+......+..+..........+..+.+............+.......................+............+.+......+...+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6692023-11-06T17:12:05.463Za pkg/config-rss.toml 7K
6702023-11-06T17:12:05.487Z-----
6712023-11-06T17:12:05.512Za pkg/initial-tls-cert.pem 2K
6722023-11-06T17:12:05.537Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6732023-11-06T17:12:05.562Za pkg/initial-tls-key.pem 4K
6742023-11-06T17:12:05.589Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6752023-11-06T17:12:05.614Z+ rmdir pkg
6762023-11-06T17:12:05.639Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6772023-11-06T17:12:05.664Z+ pfexec mkdir -p /usr/oxide
6782023-11-06T17:12:05.690Z+ pfexec rm -f /usr/oxide/tcpproxy
6792023-11-06T17:12:05.715Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6802023-11-06T17:12:05.742Z+ pfexec chmod +x /usr/oxide/tcpproxy
6812023-11-06T17:12:05.766Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6822023-11-06T17:12:05.791Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6832023-11-06T17:12:05.847Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6842023-11-06T17:12:05.872Z+ OMICRON_NO_UNINSTALL=1
6852023-11-06T17:12:05.897Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6862023-11-06T17:12:05.922ZNov 06 17:12:03.854 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6872023-11-06T17:12:05.946ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6882023-11-06T17:12:05.971ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6892023-11-06T17:12:05.995ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6902023-11-06T17:12:06.019ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6912023-11-06T17:12:06.046ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6922023-11-06T17:12:06.071ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6932023-11-06T17:12:06.095ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6942023-11-06T17:12:06.120ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6952023-11-06T17:12:06.144ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6962023-11-06T17:12:06.169ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6972023-11-06T17:12:06.194ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6982023-11-06T17:12:06.218ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6992023-11-06T17:12:06.243ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7002023-11-06T17:12:06.268ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7012023-11-06T17:12:06.293ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7022023-11-06T17:12:06.320ZNov 06 17:12:03.857 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7032023-11-06T17:12:06.344ZNov 06 17:12:04.090 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7042023-11-06T17:12:06.369ZNov 06 17:12:04.124 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7052023-11-06T17:12:06.393ZNov 06 17:12:04.188 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7062023-11-06T17:12:06.417Z
7072023-11-06T17:12:06.442Zreal 0.471161838
7082023-11-06T17:12:06.467Zuser 0.098258678
7092023-11-06T17:12:06.493Zsys 1.957494097
7102023-11-06T17:12:06.518Ztrap 0.000006820
7112023-11-06T17:12:06.542Ztflt 0.000020070
7122023-11-06T17:12:06.569Zdflt 0.000341288
7132023-11-06T17:12:06.594Zkflt 0.000000000
7142023-11-06T17:12:06.618Zlock 42.240352158
7152023-11-06T17:12:06.643Zslp 0.675292383
7162023-11-06T17:12:06.669Zlat 0.003077760
7172023-11-06T17:12:06.694Zstop 0.206577159
7182023-11-06T17:12:06.719Z+ retry=0
7192023-11-06T17:12:06.744Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7202023-11-06T17:12:06.769Z+ [[ 0 -gt 30 ]]
7212023-11-06T17:12:06.794Z+ sleep 1
7222023-11-06T17:12:06.818Z+ retry=1
7232023-11-06T17:12:06.843Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7242023-11-06T17:12:06.886Z+ [[ 1 -gt 30 ]]
7252023-11-06T17:12:06.921Z+ sleep 1
7262023-11-06T17:12:06.947Z+ retry=2
7272023-11-06T17:12:06.973Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7282023-11-06T17:12:31.723Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7292023-11-06T17:12:31.748Zlocal v6:
7302023-11-06T17:12:31.772Zfd00:99::1
7312023-11-06T17:12:31.796Zfe80::cc65:6cff:fe74:8e91
7322023-11-06T17:12:31.821Zlocal v4:
7332023-11-06T17:12:31.846Z10.151.2.174
7342023-11-06T17:12:31.871Zrouter v6:
7352023-11-06T17:12:31.895Zfdb0:18c0:4d0d:a02a::/64 -> fe80::3808:e0ff:fedd:a53f (1)
7362023-11-06T17:12:31.920Zfd00:1122:3344:101::/64 -> fe80::3808:e0ff:fedd:a53f (1)
7372023-11-06T17:12:31.943Zrouter v4_idx:
7382023-11-06T17:12:31.967Z0.0.0.0/0 -> 0
7392023-11-06T17:12:32.025Zrouter v4_routes:
7402023-11-06T17:12:32.051Z0 -> 10.151.2.100 (2)
7412023-11-06T17:12:32.076Zresolver v4:
7422023-11-06T17:12:32.100Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7432023-11-06T17:12:32.125Z10.151.2.174 -> de:26:5f:2b:49:41
7442023-11-06T17:12:32.149Zresolver v6:
7452023-11-06T17:12:32.173Zfe80::3808:e0ff:fedd:a53f -> 3a:08:e0:dd:a5:3f
7462023-11-06T17:12:32.198Zfe80::cc65:6cff:fe74:8e91 -> ce:65:6c:74:8e:91
7472023-11-06T17:12:32.223Znat v4:
7482023-11-06T17:12:32.247Znat_v6:
7492023-11-06T17:12:32.271Zport_mac:
7502023-11-06T17:12:32.296Z1: a8:40:25:55:39:f3
7512023-11-06T17:12:32.320Z1000: a8:40:25:55:39:f2
7522023-11-06T17:12:32.345Zproxy arp:
7532023-11-06T17:12:32.391Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7542023-11-06T17:12:32.424Z+ retry=0
7552023-11-06T17:12:32.449Z+++ zoneadm list -n
7562023-11-06T17:12:32.473Z+++ grep oxz_ntp
7572023-11-06T17:12:32.499Z++ pfexec svcs -z -Hostate oxide/ntp
7582023-11-06T17:12:32.551Zsvcs: zone '-Hostate': entity not found
7592023-11-06T17:12:32.575Z++ true
7602023-11-06T17:12:32.600Z+ [[ '' != online ]]
7612023-11-06T17:12:32.624Z+ [[ 0 -gt 60 ]]
7622023-11-06T17:12:32.649Z+ sleep 1
7632023-11-06T17:12:32.773Z+ retry=1
7642023-11-06T17:12:32.797Z+++ zoneadm list -n
7652023-11-06T17:12:32.822Z+++ grep oxz_ntp
7662023-11-06T17:12:32.846Z++ pfexec svcs -z -Hostate oxide/ntp
7672023-11-06T17:12:32.871Zsvcs: zone '-Hostate': entity not found
7682023-11-06T17:12:32.895Z++ true
7692023-11-06T17:12:32.920Z+ [[ '' != online ]]
7702023-11-06T17:12:32.945Z+ [[ 1 -gt 60 ]]
7712023-11-06T17:12:32.970Z+ sleep 1
7722023-11-06T17:12:33.794Z+ retry=2
7732023-11-06T17:12:33.818Z+++ zoneadm list -n
7742023-11-06T17:12:33.843Z+++ grep oxz_ntp
7752023-11-06T17:12:33.868Z++ pfexec svcs -z -Hostate oxide/ntp
7762023-11-06T17:12:33.893Zsvcs: zone '-Hostate': entity not found
7772023-11-06T17:12:33.918Z++ true
7782023-11-06T17:12:33.943Z+ [[ '' != online ]]
7792023-11-06T17:12:33.967Z+ [[ 2 -gt 60 ]]
7802023-11-06T17:12:33.992Z+ sleep 1
7812023-11-06T17:12:34.815Z+ retry=3
7822023-11-06T17:12:34.840Z+++ zoneadm list -n
7832023-11-06T17:12:34.865Z+++ grep oxz_ntp
7842023-11-06T17:12:34.890Z++ pfexec svcs -z -Hostate oxide/ntp
7852023-11-06T17:12:34.914Zsvcs: zone '-Hostate': entity not found
7862023-11-06T17:12:34.942Z++ true
7872023-11-06T17:12:34.969Z+ [[ '' != online ]]
7882023-11-06T17:12:34.997Z+ [[ 3 -gt 60 ]]
7892023-11-06T17:12:35.022Z+ sleep 1
7902023-11-06T17:12:35.839Z+ retry=4
7912023-11-06T17:12:35.866Z+++ zoneadm list -n
7922023-11-06T17:12:35.890Z+++ grep oxz_ntp
7932023-11-06T17:12:35.914Z++ pfexec svcs -z -Hostate oxide/ntp
7942023-11-06T17:12:35.939Zsvcs: zone '-Hostate': entity not found
7952023-11-06T17:12:35.964Z++ true
7962023-11-06T17:12:35.988Z+ [[ '' != online ]]
7972023-11-06T17:12:36.014Z+ [[ 4 -gt 60 ]]
7982023-11-06T17:12:36.039Z+ sleep 1
7992023-11-06T17:12:36.859Z+ retry=5
8002023-11-06T17:12:36.885Z+++ zoneadm list -n
8012023-11-06T17:12:36.911Z+++ grep oxz_ntp
8022023-11-06T17:12:36.936Z++ pfexec svcs -z -Hostate oxide/ntp
8032023-11-06T17:12:36.961Zsvcs: zone '-Hostate': entity not found
8042023-11-06T17:12:36.986Z++ true
8052023-11-06T17:12:37.011Z+ [[ '' != online ]]
8062023-11-06T17:12:37.036Z+ [[ 5 -gt 60 ]]
8072023-11-06T17:12:37.060Z+ sleep 1
8082023-11-06T17:12:37.879Z+ retry=6
8092023-11-06T17:12:37.904Z+++ zoneadm list -n
8102023-11-06T17:12:37.929Z+++ grep oxz_ntp
8112023-11-06T17:12:37.954Z++ pfexec svcs -z -Hostate oxide/ntp
8122023-11-06T17:12:37.979Zsvcs: zone '-Hostate': entity not found
8132023-11-06T17:12:38.004Z++ true
8142023-11-06T17:12:38.028Z+ [[ '' != online ]]
8152023-11-06T17:12:38.056Z+ [[ 6 -gt 60 ]]
8162023-11-06T17:12:38.080Z+ sleep 1
8172023-11-06T17:12:38.899Z+ retry=7
8182023-11-06T17:12:38.924Z+++ zoneadm list -n
8192023-11-06T17:12:38.951Z+++ grep oxz_ntp
8202023-11-06T17:12:38.975Z++ pfexec svcs -z -Hostate oxide/ntp
8212023-11-06T17:12:39.001Zsvcs: zone '-Hostate': entity not found
8222023-11-06T17:12:39.026Z++ true
8232023-11-06T17:12:39.060Z+ [[ '' != online ]]
8242023-11-06T17:12:39.086Z+ [[ 7 -gt 60 ]]
8252023-11-06T17:12:39.111Z+ sleep 1
8262023-11-06T17:12:39.920Z+ retry=8
8272023-11-06T17:12:39.945Z+++ zoneadm list -n
8282023-11-06T17:12:39.970Z+++ grep oxz_ntp
8292023-11-06T17:12:39.994Z++ pfexec svcs -z -Hostate oxide/ntp
8302023-11-06T17:12:40.021Zsvcs: zone '-Hostate': entity not found
8312023-11-06T17:12:40.046Z++ true
8322023-11-06T17:12:40.070Z+ [[ '' != online ]]
8332023-11-06T17:12:40.095Z+ [[ 8 -gt 60 ]]
8342023-11-06T17:12:40.119Z+ sleep 1
8352023-11-06T17:12:40.943Z+ retry=9
8362023-11-06T17:12:40.968Z+++ zoneadm list -n
8372023-11-06T17:12:40.992Z+++ grep oxz_ntp
8382023-11-06T17:12:41.017Z++ pfexec svcs -z -Hostate oxide/ntp
8392023-11-06T17:12:41.044Zsvcs: zone '-Hostate': entity not found
8402023-11-06T17:12:41.069Z++ true
8412023-11-06T17:12:41.094Z+ [[ '' != online ]]
8422023-11-06T17:12:41.119Z+ [[ 9 -gt 60 ]]
8432023-11-06T17:12:41.150Z+ sleep 1
8442023-11-06T17:12:41.971Z+ retry=10
8452023-11-06T17:12:42.005Z+++ zoneadm list -n
8462023-11-06T17:12:42.032Z+++ grep oxz_ntp
8472023-11-06T17:12:42.057Z++ pfexec svcs -z oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64 -Hostate oxide/ntp
8482023-11-06T17:12:42.082Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8492023-11-06T17:12:42.106Z++ true
8502023-11-06T17:12:42.130Z+ [[ '' != online ]]
8512023-11-06T17:12:42.155Z+ [[ 10 -gt 60 ]]
8522023-11-06T17:12:42.179Z+ sleep 1
8532023-11-06T17:12:43.038Z+ retry=11
8542023-11-06T17:12:43.071Z+++ zoneadm list -n
8552023-11-06T17:12:43.096Z+++ grep oxz_ntp
8562023-11-06T17:12:43.121Z++ pfexec svcs -z oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64 -Hostate oxide/ntp
8572023-11-06T17:12:43.145Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8582023-11-06T17:12:43.169Z++ true
8592023-11-06T17:12:43.195Z+ [[ '' != online ]]
8602023-11-06T17:12:43.222Z+ [[ 11 -gt 60 ]]
8612023-11-06T17:12:43.247Z+ sleep 1
8622023-11-06T17:12:44.040Z+ retry=12
8632023-11-06T17:12:44.100Z+++ zoneadm list -n
8642023-11-06T17:12:44.131Z+++ grep oxz_ntp
8652023-11-06T17:12:44.155Z++ pfexec svcs -z oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64 -Hostate oxide/ntp
8662023-11-06T17:12:44.179Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8672023-11-06T17:12:44.203Z++ true
8682023-11-06T17:12:44.227Z+ [[ '' != online ]]
8692023-11-06T17:12:44.252Z+ [[ 12 -gt 60 ]]
8702023-11-06T17:12:44.276Z+ sleep 1
8712023-11-06T17:12:45.070Z+ retry=13
8722023-11-06T17:12:45.094Z+++ zoneadm list -n
8732023-11-06T17:12:45.118Z+++ grep oxz_ntp
8742023-11-06T17:12:45.142Z++ pfexec svcs -z oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64 -Hostate oxide/ntp
8752023-11-06T17:12:45.166Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8762023-11-06T17:12:45.190Z++ true
8772023-11-06T17:12:45.214Z+ [[ '' != online ]]
8782023-11-06T17:12:45.238Z+ [[ 13 -gt 60 ]]
8792023-11-06T17:12:45.262Z+ sleep 1
8802023-11-06T17:12:46.098Z+ retry=14
8812023-11-06T17:12:46.123Z+++ zoneadm list -n
8822023-11-06T17:12:46.147Z+++ grep oxz_ntp
8832023-11-06T17:12:46.171Z++ pfexec svcs -z oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64 -Hostate oxide/ntp
8842023-11-06T17:12:46.195Z+ [[ disabled != online ]]
8852023-11-06T17:12:46.219Z+ [[ 14 -gt 60 ]]
8862023-11-06T17:12:46.243Z+ sleep 1
8872023-11-06T17:12:47.122Z+ retry=15
8882023-11-06T17:12:47.146Z+++ zoneadm list -n
8892023-11-06T17:12:47.171Z+++ grep oxz_ntp
8902023-11-06T17:12:47.195Z++ pfexec svcs -z oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64 -Hostate oxide/ntp
8912023-11-06T17:12:47.220Z+ [[ disabled != online ]]
8922023-11-06T17:12:47.244Z+ [[ 15 -gt 60 ]]
8932023-11-06T17:12:47.268Z+ sleep 1
8942023-11-06T17:12:48.143Z+ retry=16
8952023-11-06T17:12:48.168Z+++ zoneadm list -n
8962023-11-06T17:12:48.192Z+++ grep oxz_ntp
8972023-11-06T17:12:48.215Z++ pfexec svcs -z oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64 -Hostate oxide/ntp
8982023-11-06T17:12:48.240Z+ [[ disabled != online ]]
8992023-11-06T17:12:48.264Z+ [[ 16 -gt 60 ]]
9002023-11-06T17:12:48.288Z+ sleep 1
9012023-11-06T17:12:49.163Z+ retry=17
9022023-11-06T17:12:49.187Z+++ zoneadm list -n
9032023-11-06T17:12:49.211Z+++ grep oxz_ntp
9042023-11-06T17:12:49.236Z++ pfexec svcs -z oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64 -Hostate oxide/ntp
9052023-11-06T17:12:49.260Z+ [[ disabled != online ]]
9062023-11-06T17:12:49.284Z+ [[ 17 -gt 60 ]]
9072023-11-06T17:12:49.308Z+ sleep 1
9082023-11-06T17:12:50.184Z+ retry=18
9092023-11-06T17:12:50.209Z+++ zoneadm list -n
9102023-11-06T17:12:50.233Z+++ grep oxz_ntp
9112023-11-06T17:12:50.257Z++ pfexec svcs -z oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64 -Hostate oxide/ntp
9122023-11-06T17:12:50.281Z+ [[ disabled != online ]]
9132023-11-06T17:12:50.305Z+ [[ 18 -gt 60 ]]
9142023-11-06T17:12:50.329Z+ sleep 1
9152023-11-06T17:12:51.207Z+ retry=19
9162023-11-06T17:12:51.230Z+++ zoneadm list -n
9172023-11-06T17:12:51.254Z+++ grep oxz_ntp
9182023-11-06T17:12:51.278Z++ pfexec svcs -z oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64 -Hostate oxide/ntp
9192023-11-06T17:12:51.302Z+ [[ disabled != online ]]
9202023-11-06T17:12:51.326Z+ [[ 19 -gt 60 ]]
9212023-11-06T17:12:51.350Z+ sleep 1
 
9222023-11-06T17:12:51.763Zsock console login: Nov 6 17:12:51 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9232023-11-06T17:12:52.232Z+ retry=20
9242023-11-06T17:12:52.256Z+++ zoneadm list -n
9252023-11-06T17:12:52.280Z+++ grep oxz_ntp
9262023-11-06T17:12:52.308Z++ pfexec svcs -z oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64 -Hostate oxide/ntp
9272023-11-06T17:12:52.332Z+ [[ online != online ]]
9282023-11-06T17:12:52.356Z+ echo 'Waited for chrony: 20s'
9292023-11-06T17:12:52.380ZWaited for chrony: 20s
9302023-11-06T17:12:52.404Z+ export RUST_BACKTRACE=1
9312023-11-06T17:12:52.428Z+ RUST_BACKTRACE=1
9322023-11-06T17:12:52.452Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9332023-11-06T17:12:52.504Z+ ./tests/bootstrap
9342023-11-06T17:12:52.536ZUsing external DNS server at 10.151.2.164:53
9352023-11-06T17:12:52.563Z2023-11-06 17:12:52.073355704 UTC: attempting to log into API
9362023-11-06T17:13:07.317Z2023-11-06 17:13:07.547500960 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9372023-11-06T17:13:08.316Z2023-11-06 17:13:08.545640212 UTC: attempting to log into API
9382023-11-06T17:13:23.350Z2023-11-06 17:13:23.538253478 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9392023-11-06T17:13:24.758Z2023-11-06 17:13:24.536703325 UTC: attempting to log into API
9402023-11-06T17:13:39.393Z2023-11-06 17:13:39.530255907 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9412023-11-06T17:13:40.386Z2023-11-06 17:13:40.528330168 UTC: attempting to log into API
9422023-11-06T17:13:55.419Z2023-11-06 17:13:55.521059136 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9432023-11-06T17:13:56.423Z2023-11-06 17:13:56.518975234 UTC: attempting to log into API
9442023-11-06T17:14:11.453Z2023-11-06 17:14:11.511532729 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9452023-11-06T17:14:12.454Z2023-11-06 17:14:12.509681852 UTC: attempting to log into API
9462023-11-06T17:14:27.486Z2023-11-06 17:14:27.501200169 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9472023-11-06T17:14:28.487Z2023-11-06 17:14:28.499388452 UTC: attempting to log into API
9482023-11-06T17:14:43.521Z2023-11-06 17:14:43.509756560 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9492023-11-06T17:14:44.525Z2023-11-06 17:14:44.510417269 UTC: attempting to log into API
9502023-11-06T17:14:59.561Z2023-11-06 17:14:59.543506431 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9512023-11-06T17:15:00.555Z2023-11-06 17:15:00.544269092 UTC: attempting to log into API
9522023-11-06T17:15:15.595Z2023-11-06 17:15:15.576668182 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9532023-11-06T17:15:16.936Z2023-11-06 17:15:16.577588814 UTC: attempting to log into API
9542023-11-06T17:15:31.622Z2023-11-06 17:15:31.610820555 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9552023-11-06T17:15:32.622Z2023-11-06 17:15:32.611681517 UTC: attempting to log into API
9562023-11-06T17:15:47.663Z2023-11-06 17:15:47.652463993 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9572023-11-06T17:15:48.666Z2023-11-06 17:15:48.654707339 UTC: attempting to log into API
9582023-11-06T17:16:03.844Z2023-11-06 17:16:03.834159608 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9592023-11-06T17:16:04.846Z2023-11-06 17:16:04.835328075 UTC: attempting to log into API
9602023-11-06T17:16:19.879Z2023-11-06 17:16:19.868996672 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9612023-11-06T17:16:20.880Z2023-11-06 17:16:20.870212361 UTC: attempting to log into API
9622023-11-06T17:16:35.913Z2023-11-06 17:16:35.903806086 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9632023-11-06T17:16:36.917Z2023-11-06 17:16:36.904611797 UTC: attempting to log into API
 
9642023-11-06T17:16:41.750ZNov 6 17:16:41 sok ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9652023-11-06T17:16:49.441ZNov 6 17:16:48 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9662023-11-06T17:16:51.966Z2023-11-06 17:16:51.957075777 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9672023-11-06T17:16:52.970Z2023-11-06 17:16:52.958149543 UTC: attempting to log into API
 
9682023-11-06T17:16:55.462ZNov 6 17:16:55 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9692023-11-06T17:17:04.498ZNov 6 17:17:02 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9702023-11-06T17:17:08.011Z2023-11-06 17:17:07.991639117 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9712023-11-06T17:17:09.013Z2023-11-06 17:17:08.992370896 UTC: attempting to log into API
 
9722023-11-06T17:17:09.793ZNov 6 17:17:09 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9732023-11-06T17:17:24.045Z2023-11-06 17:17:24.026061143 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9742023-11-06T17:17:25.047Z2023-11-06 17:17:25.027180390 UTC: attempting to log into API
9752023-11-06T17:17:33.112Z2023-11-06 17:17:30.820269004 UTC: login succeeded
9762023-11-06T17:17:33.139Zwaiting for nexus to come up...
9772023-11-06T17:17:33.164Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9782023-11-06T17:17:33.188Zensuring datasets are ready...
9792023-11-06T17:17:33.213Zlet's roll.
9802023-11-06T17:17:33.238Z+ rm ./tests/bootstrap
9812023-11-06T17:17:33.262Z+ for test_bin in tests/*
9822023-11-06T17:17:33.334Z+ ./tests/bootstrap-00f20c85209291d2
9832023-11-06T17:17:33.359Z
9842023-11-06T17:17:33.383Zrunning 0 tests
9852023-11-06T17:17:33.408Z
9862023-11-06T17:17:33.433Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9872023-11-06T17:17:33.457Z
9882023-11-06T17:17:33.482Z+ for test_bin in tests/*
9892023-11-06T17:17:33.506Z+ ./tests/end_to_end_tests-2a5726111815475a
9902023-11-06T17:17:33.530Z
9912023-11-06T17:17:33.556Zrunning 2 tests
9922023-11-06T17:17:33.581Ztest no_spoof::no_spoof_here ... ok
9932023-11-06T17:18:33.052Ztest instance_launch::instance_launch has been running for over 60 seconds
9942023-11-06T17:20:25.686Ztest instance_launch::instance_launch ... ok
9952023-11-06T17:20:25.723Z
9962023-11-06T17:20:25.754Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 172.23s
9972023-11-06T17:20:25.779Z
9982023-11-06T17:20:25.804Z+ _exit_trap
9992023-11-06T17:20:25.841Z+ local status=0
10002023-11-06T17:20:25.871Z+ [[ 0 -eq 0 ]]
10012023-11-06T17:20:25.900Z+ exit 0
10022023-11-06T17:20:25.924Zprocess exited: duration 536997 ms, exit code 0
 
10032023-11-06T17:20:26.052Zfound 51 output files
10042023-11-06T17:20:26.235Zuploading: /var/svc/log/oxide-sled-agent:default.log (493710 bytes)
10052023-11-06T17:20:27.733Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10062023-11-06T17:20:27.757Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_03498854-b840-4a02-971f-d0bc06691a2d/root/var/svc/log/oxide-cockroachdb:default.log (5344 bytes)
10072023-11-06T17:20:28.829Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_03498854-b840-4a02-971f-d0bc06691a2d/root/var/svc/log/oxide-cockroachdb:default.log
10082023-11-06T17:20:28.853Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6594f53d-cef4-4ed4-9010-e1e2922fd832/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10092023-11-06T17:20:29.940Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6594f53d-cef4-4ed4-9010-e1e2922fd832/root/var/svc/log/oxide-crucible-agent:default.log
10102023-11-06T17:20:29.971Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6594f53d-cef4-4ed4-9010-e1e2922fd832/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10112023-11-06T17:20:31.070Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6594f53d-cef4-4ed4-9010-e1e2922fd832/root/var/svc/log/oxide-crucible-downstairs:default.log
10122023-11-06T17:20:31.588Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8949b7c4-4fd1-44af-a919-f8b76a599db5/root/var/svc/log/oxide-nexus:default.log (757579 bytes)
10132023-11-06T17:20:31.868Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8949b7c4-4fd1-44af-a919-f8b76a599db5/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 757579 -> 758187
10142023-11-06T17:20:32.952Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8949b7c4-4fd1-44af-a919-f8b76a599db5/root/var/svc/log/oxide-nexus:default.log
10152023-11-06T17:20:32.987Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64/root/var/svc/log/oxide-ntp:default.log (0 bytes)
10162023-11-06T17:20:34.039Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64/root/var/svc/log/oxide-ntp:default.log
10172023-11-06T17:20:34.066Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64/root/var/svc/log/oxide-ntp:default.log.0 (1182 bytes)
10182023-11-06T17:20:35.138Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64/root/var/svc/log/oxide-ntp:default.log.0
10192023-11-06T17:20:35.162Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_129ad49e-f901-40cb-bab6-4c47d0104ef8/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10202023-11-06T17:20:36.234Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_129ad49e-f901-40cb-bab6-4c47d0104ef8/root/var/svc/log/oxide-cockroachdb:default.log
10212023-11-06T17:20:36.258Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_256e2c28-11fd-4baa-a58d-51666929721d/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10222023-11-06T17:20:37.329Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_256e2c28-11fd-4baa-a58d-51666929721d/root/var/svc/log/oxide-crucible-agent:default.log
10232023-11-06T17:20:37.353Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_256e2c28-11fd-4baa-a58d-51666929721d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10242023-11-06T17:20:38.423Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_256e2c28-11fd-4baa-a58d-51666929721d/root/var/svc/log/oxide-crucible-downstairs:default.log
10252023-11-06T17:20:38.447Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dd47e4eb-0e06-4cdb-ab97-bb9d498c98c0/root/var/svc/log/oxide-crucible-agent:default.log (14153 bytes)
10262023-11-06T17:20:39.574Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dd47e4eb-0e06-4cdb-ab97-bb9d498c98c0/root/var/svc/log/oxide-crucible-agent:default.log
10272023-11-06T17:20:39.599Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dd47e4eb-0e06-4cdb-ab97-bb9d498c98c0/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10282023-11-06T17:20:40.681Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dd47e4eb-0e06-4cdb-ab97-bb9d498c98c0/root/var/svc/log/oxide-crucible-downstairs:default.log
10292023-11-06T17:20:40.709Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dd47e4eb-0e06-4cdb-ab97-bb9d498c98c0/root/var/svc/log/oxide-crucible-downstairs:downstairs-75896ef7-318c-4817-b533-f718191ee18f.log (3048 bytes)
10302023-11-06T17:20:41.786Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dd47e4eb-0e06-4cdb-ab97-bb9d498c98c0/root/var/svc/log/oxide-crucible-downstairs:downstairs-75896ef7-318c-4817-b533-f718191ee18f.log
10312023-11-06T17:20:41.811Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_f944efea-d681-42b8-ab61-3b1c2fd161b7/root/var/svc/log/oxide-nexus:default.log (568270 bytes)
10322023-11-06T17:20:42.637Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_f944efea-d681-42b8-ab61-3b1c2fd161b7/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 568270 -> 569486
10332023-11-06T17:20:43.715Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_f944efea-d681-42b8-ab61-3b1c2fd161b7/root/var/svc/log/oxide-nexus:default.log
10342023-11-06T17:20:44.151Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_9d0f6b14-778a-47c5-a1aa-372309a7dbb2/root/var/svc/log/oxide-oximeter:default.log (187267 bytes)
10352023-11-06T17:20:44.212Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_9d0f6b14-778a-47c5-a1aa-372309a7dbb2/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 187267 -> 204582
10362023-11-06T17:20:45.691Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_9d0f6b14-778a-47c5-a1aa-372309a7dbb2/root/var/svc/log/oxide-oximeter:default.log
10372023-11-06T17:20:45.716Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_83162a9b-bb36-4f27-b97a-edc527f04099/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10382023-11-06T17:20:46.812Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_83162a9b-bb36-4f27-b97a-edc527f04099/root/var/svc/log/oxide-crucible-pantry:default.log
10392023-11-06T17:20:46.837Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_1a54e7d7-a6be-4bb6-b582-b0e467ef980b/root/var/svc/log/oxide-clickhouse:default.log (713366 bytes)
10402023-11-06T17:20:48.585Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_1a54e7d7-a6be-4bb6-b582-b0e467ef980b/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 713366 -> 801582
10412023-11-06T17:20:49.656Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_1a54e7d7-a6be-4bb6-b582-b0e467ef980b/root/var/svc/log/oxide-clickhouse:default.log
10422023-11-06T17:20:49.680Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_3d26847a-61c0-467d-98f0-9aa4c1676e3c/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10432023-11-06T17:20:51.160Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_3d26847a-61c0-467d-98f0-9aa4c1676e3c/root/var/svc/log/oxide-cockroachdb:default.log
10442023-11-06T17:20:51.193Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_199a2c52-6c9e-4b9f-ab1f-c6d2f54360e4/root/var/svc/log/oxide-external_dns:default.log (15612 bytes)
10452023-11-06T17:20:51.880Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_199a2c52-6c9e-4b9f-ab1f-c6d2f54360e4/root/var/svc/log/oxide-external_dns:default.log
10462023-11-06T17:20:51.919Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_7a9d68fa-9b7e-4d31-8360-2f54c46e9332/root/var/svc/log/oxide-nexus:default.log (4230200 bytes)
10472023-11-06T17:20:55.509Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_7a9d68fa-9b7e-4d31-8360-2f54c46e9332/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4230200 -> 4232021
10482023-11-06T17:20:56.600Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_7a9d68fa-9b7e-4d31-8360-2f54c46e9332/root/var/svc/log/oxide-nexus:default.log
10492023-11-06T17:20:56.642Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_10820954-0d99-4ff5-9d45-3c804467eb74/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10502023-11-06T17:20:58.176Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_10820954-0d99-4ff5-9d45-3c804467eb74/root/var/svc/log/oxide-crucible-agent:default.log
10512023-11-06T17:20:58.200Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_10820954-0d99-4ff5-9d45-3c804467eb74/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10522023-11-06T17:20:59.680Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_10820954-0d99-4ff5-9d45-3c804467eb74/root/var/svc/log/oxide-crucible-downstairs:default.log
10532023-11-06T17:20:59.708Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4e5897a9-7a14-4576-81ea-9898f3ca19a2/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10542023-11-06T17:21:00.379Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4e5897a9-7a14-4576-81ea-9898f3ca19a2/root/var/svc/log/oxide-crucible-agent:default.log
10552023-11-06T17:21:00.421Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4e5897a9-7a14-4576-81ea-9898f3ca19a2/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10562023-11-06T17:21:01.495Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4e5897a9-7a14-4576-81ea-9898f3ca19a2/root/var/svc/log/oxide-crucible-downstairs:default.log
10572023-11-06T17:21:01.520Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_3192284c-6b02-4bfe-b915-04a84625a5c0/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10582023-11-06T17:21:02.591Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_3192284c-6b02-4bfe-b915-04a84625a5c0/root/var/svc/log/oxide-crucible-pantry:default.log
10592023-11-06T17:21:02.616Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_2b077404-a803-4ad4-870c-1d2af17fddb2/root/var/svc/log/oxide-internal_dns:default.log (93564 bytes)
10602023-11-06T17:21:03.907Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_2b077404-a803-4ad4-870c-1d2af17fddb2/root/var/svc/log/oxide-internal_dns:default.log
10612023-11-06T17:21:03.932Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_2b077404-a803-4ad4-870c-1d2af17fddb2/root/var/svc/log/oxide-internal_dns:default.log.0 (6317 bytes)
10622023-11-06T17:21:10.690Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_2b077404-a803-4ad4-870c-1d2af17fddb2/root/var/svc/log/oxide-internal_dns:default.log.0
10632023-11-06T17:21:10.725Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_72596a7b-9b1d-414f-9c00-2d3040b6fb4e/root/var/svc/log/oxide-crucible-agent:default.log (24397 bytes)
10642023-11-06T17:21:11.905Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_72596a7b-9b1d-414f-9c00-2d3040b6fb4e/root/var/svc/log/oxide-crucible-agent:default.log
10652023-11-06T17:21:11.934Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_72596a7b-9b1d-414f-9c00-2d3040b6fb4e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10662023-11-06T17:21:13.013Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_72596a7b-9b1d-414f-9c00-2d3040b6fb4e/root/var/svc/log/oxide-crucible-downstairs:default.log
10672023-11-06T17:21:13.042Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_72596a7b-9b1d-414f-9c00-2d3040b6fb4e/root/var/svc/log/oxide-crucible-downstairs:downstairs-b6d9522b-9884-472e-a28f-e7fe366fdddb.log (3049 bytes)
10682023-11-06T17:21:14.531Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_72596a7b-9b1d-414f-9c00-2d3040b6fb4e/root/var/svc/log/oxide-crucible-downstairs:downstairs-b6d9522b-9884-472e-a28f-e7fe366fdddb.log
10692023-11-06T17:21:14.556Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_72596a7b-9b1d-414f-9c00-2d3040b6fb4e/root/var/svc/log/oxide-crucible-downstairs:downstairs-f1cc3992-48c2-4bec-90ab-e5cf1ba28055.log (5537 bytes)
10702023-11-06T17:21:15.635Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_72596a7b-9b1d-414f-9c00-2d3040b6fb4e/root/var/svc/log/oxide-crucible-downstairs:downstairs-f1cc3992-48c2-4bec-90ab-e5cf1ba28055.log
10712023-11-06T17:21:15.667Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_85b7193a-9816-4f53-9c4e-fee4fcef45d5/root/var/svc/log/oxide-internal_dns:default.log (93565 bytes)
10722023-11-06T17:21:16.479Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_85b7193a-9816-4f53-9c4e-fee4fcef45d5/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93565 -> 97320
10732023-11-06T17:21:17.550Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_85b7193a-9816-4f53-9c4e-fee4fcef45d5/root/var/svc/log/oxide-internal_dns:default.log
10742023-11-06T17:21:17.596Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_85b7193a-9816-4f53-9c4e-fee4fcef45d5/root/var/svc/log/oxide-internal_dns:default.log.0 (8755 bytes)
10752023-11-06T17:21:18.681Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_85b7193a-9816-4f53-9c4e-fee4fcef45d5/root/var/svc/log/oxide-internal_dns:default.log.0
10762023-11-06T17:21:18.706Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_a4ccd145-a7b7-48a2-90ed-39c30e9dc933/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10772023-11-06T17:21:19.779Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_a4ccd145-a7b7-48a2-90ed-39c30e9dc933/root/var/svc/log/oxide-cockroachdb:default.log
10782023-11-06T17:21:19.803Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_19128728-a876-4a2f-a78c-e4def6b2a7fa/root/var/svc/log/oxide-crucible-agent:default.log (24399 bytes)
10792023-11-06T17:21:21.473Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_19128728-a876-4a2f-a78c-e4def6b2a7fa/root/var/svc/log/oxide-crucible-agent:default.log
10802023-11-06T17:21:21.502Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_19128728-a876-4a2f-a78c-e4def6b2a7fa/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10812023-11-06T17:21:22.582Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_19128728-a876-4a2f-a78c-e4def6b2a7fa/root/var/svc/log/oxide-crucible-downstairs:default.log
10822023-11-06T17:21:22.606Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_19128728-a876-4a2f-a78c-e4def6b2a7fa/root/var/svc/log/oxide-crucible-downstairs:downstairs-b572542e-778d-4ea8-90e7-cc2844873070.log (3049 bytes)
10832023-11-06T17:21:23.681Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_19128728-a876-4a2f-a78c-e4def6b2a7fa/root/var/svc/log/oxide-crucible-downstairs:downstairs-b572542e-778d-4ea8-90e7-cc2844873070.log
10842023-11-06T17:21:23.705Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_19128728-a876-4a2f-a78c-e4def6b2a7fa/root/var/svc/log/oxide-crucible-downstairs:downstairs-c0631edf-f11a-45e9-8b3f-153f812bcd5e.log (5535 bytes)
10852023-11-06T17:21:24.810Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_19128728-a876-4a2f-a78c-e4def6b2a7fa/root/var/svc/log/oxide-crucible-downstairs:downstairs-c0631edf-f11a-45e9-8b3f-153f812bcd5e.log
10862023-11-06T17:21:24.838Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ef632bd8-76fa-4abd-a958-4a62d53c7f1c/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10872023-11-06T17:21:26.320Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ef632bd8-76fa-4abd-a958-4a62d53c7f1c/root/var/svc/log/oxide-crucible-agent:default.log
10882023-11-06T17:21:26.349Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ef632bd8-76fa-4abd-a958-4a62d53c7f1c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10892023-11-06T17:21:27.420Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ef632bd8-76fa-4abd-a958-4a62d53c7f1c/root/var/svc/log/oxide-crucible-downstairs:default.log
10902023-11-06T17:21:27.445Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_2bdef16b-d355-42f9-a19b-9f9b34ab5faa/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10912023-11-06T17:21:28.519Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_2bdef16b-d355-42f9-a19b-9f9b34ab5faa/root/var/svc/log/oxide-crucible-pantry:default.log
10922023-11-06T17:21:28.544Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_8d1ac0ea-ebee-47fe-9985-59dbaf8610b2/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10932023-11-06T17:21:29.625Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_8d1ac0ea-ebee-47fe-9985-59dbaf8610b2/root/var/svc/log/oxide-cockroachdb:default.log
10942023-11-06T17:21:29.650Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a69e4062-dc32-4fda-94aa-00463ca3ecf2/root/var/svc/log/oxide-crucible-agent:default.log (13688 bytes)
10952023-11-06T17:21:30.764Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a69e4062-dc32-4fda-94aa-00463ca3ecf2/root/var/svc/log/oxide-crucible-agent:default.log
10962023-11-06T17:21:30.788Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a69e4062-dc32-4fda-94aa-00463ca3ecf2/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10972023-11-06T17:21:31.857Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a69e4062-dc32-4fda-94aa-00463ca3ecf2/root/var/svc/log/oxide-crucible-downstairs:default.log
10982023-11-06T17:21:31.882Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a69e4062-dc32-4fda-94aa-00463ca3ecf2/root/var/svc/log/oxide-crucible-downstairs:downstairs-1a2ea5ea-08d8-4f54-8528-3b97c7afc842.log (5534 bytes)
10992023-11-06T17:21:32.991Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a69e4062-dc32-4fda-94aa-00463ca3ecf2/root/var/svc/log/oxide-crucible-downstairs:downstairs-1a2ea5ea-08d8-4f54-8528-3b97c7afc842.log
11002023-11-06T17:21:33.016Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_eacc08c5-8386-4fec-82a3-c50bd4a48a91/root/var/svc/log/oxide-external_dns:default.log (15612 bytes)
11012023-11-06T17:21:33.124Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_eacc08c5-8386-4fec-82a3-c50bd4a48a91/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15612 -> 19388
11022023-11-06T17:21:34.210Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_eacc08c5-8386-4fec-82a3-c50bd4a48a91/root/var/svc/log/oxide-external_dns:default.log
11032023-11-06T17:21:34.243Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_65ba4cd4-30ce-4c19-8feb-58a472bed7d7/root/var/svc/log/oxide-internal_dns:default.log (93561 bytes)
11042023-11-06T17:21:34.544Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_65ba4cd4-30ce-4c19-8feb-58a472bed7d7/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93561 -> 97317
11052023-11-06T17:21:35.625Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_65ba4cd4-30ce-4c19-8feb-58a472bed7d7/root/var/svc/log/oxide-internal_dns:default.log
11062023-11-06T17:21:35.649Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_65ba4cd4-30ce-4c19-8feb-58a472bed7d7/root/var/svc/log/oxide-internal_dns:default.log.0 (6437 bytes)
11072023-11-06T17:21:36.759Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_65ba4cd4-30ce-4c19-8feb-58a472bed7d7/root/var/svc/log/oxide-internal_dns:default.log.0
11082023-11-06T17:21:37.203Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64/root/var/log/chrony/measurements.log (15344 bytes)
11092023-11-06T17:21:37.232Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64/root/var/log/chrony/measurements.log" changed size mid upload: 15344 -> 19728
11102023-11-06T17:21:37.964Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64/root/var/log/chrony/measurements.log
11112023-11-06T17:21:37.991Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64/root/var/log/chrony/statistics.log (8190 bytes)
11122023-11-06T17:21:38.060Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64/root/var/log/chrony/statistics.log" changed size mid upload: 8190 -> 10998
11132023-11-06T17:21:39.136Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64/root/var/log/chrony/statistics.log
11142023-11-06T17:21:39.161Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64/root/var/log/chrony/tracking.log (1855 bytes)
11152023-11-06T17:21:39.190Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64/root/var/log/chrony/tracking.log" changed size mid upload: 1855 -> 1987
11162023-11-06T17:21:40.265Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a0799c53-1203-4645-b0ee-45381b9d1a64/root/var/log/chrony/tracking.log