01HEM2VQ8M9W2Q7JKZQCAFJ6GZ: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HEM2XBYGPZJWRQWPX0G54JZA

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-07T06:05:01.646Zjob dependencies complete; ready to run (waiting for 56 m 27 s)
22023-11-07T09:37:09.534Zjob assigned to worker 01HEMJ80MDHGVDM0G476WKPWKT (queued for 3 h 32 m 7 s)
32023-11-07T09:37:09.647Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-11-07T09:37:09.689Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-11-07T09:37:09.723Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-11-07T09:37:09.756Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-07T09:37:09.790Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-11-07T09:37:09.824ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-07T09:37:09.857Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-07T09:37:09.895ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-07T09:37:09.931ZConfiguring devices.
122023-11-07T09:37:09.965Z##################################################
132023-11-07T09:37:09.999Z#### ############# ##
142023-11-07T09:37:10.032Z### ### ############ ##
152023-11-07T09:37:10.065Z## ### # ## ### ## Oxide Computer Company ##
162023-11-07T09:37:10.107Z## ## ## ### # ### ##
172023-11-07T09:37:10.143Z## # ### #### #### This Station Under ##
182023-11-07T09:37:10.177Z### ### #### # ### Computer Control ##
192023-11-07T09:37:10.210Z#### #### ### ## ##
202023-11-07T09:37:10.244Z##################################################
212023-11-07T09:37:10.277ZHostname: buskin
222023-11-07T09:37:10.310ZLoading smf(7) service descriptions:
232023-11-07T09:37:11.121Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-750ee284a2fe677d.gz
242023-11-07T09:37:12.409Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-750ee284a2fe677d.gz
252023-11-07T09:37:12.433Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-07T09:37:29.525Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-07T09:37:29.550Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-13628b0bfef9b756.gz
282023-11-07T09:37:49.014Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-13628b0bfef9b756.gz
292023-11-07T09:37:49.039Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-07T09:37:49.928Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-07T09:37:49.953Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-07T09:37:52.757Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-07T09:37:52.782Zdownloading input: /input/package/work/version.txt
342023-11-07T09:37:52.805Zdownloaded input: /input/package/work/version.txt
352023-11-07T09:37:52.830Zdownloading input: /input/package/work/package.tar.gz
362023-11-07T09:38:07.088Zdownloaded input: /input/package/work/package.tar.gz
372023-11-07T09:38:07.121Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-07T09:38:28.407Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-07T09:38:28.431Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-07T09:38:34.833Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-07T09:38:34.858Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-07T09:39:05.798Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-07T09:39:05.823Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-07T09:39:36.702Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-07T09:39:36.729Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-07T09:39:57.054Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-07T09:39:57.079Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-07T09:40:01.060Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-07T09:40:01.084Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-07T09:40:08.453Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-07T09:40:08.477Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-07T09:40:12.668Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-07T09:40:12.693Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-07T09:40:16.912Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-07T09:40:16.937Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-07T09:40:17.020Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-07T09:40:17.044Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-07T09:40:20.314Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-07T09:40:20.340Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-07T09:40:23.514Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-07T09:40:23.538Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-07T09:40:35.937Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-07T09:40:35.961Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-07T09:40:48.357Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-07T09:40:48.381Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-07T09:40:48.445Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-07T09:40:48.469Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-07T09:40:52.157Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-07T09:40:52.181Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-07T09:41:03.517Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-07T09:41:03.541Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-07T09:41:44.633Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-07T09:41:44.657Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-07T09:42:12.975Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-07T09:42:12.999Zstarting task 0: "setup"
762023-11-07T09:42:13.054Z++ uname -s
772023-11-07T09:42:13.078Z+ kern=SunOS
782023-11-07T09:42:13.102Z+ case "$kern" in
792023-11-07T09:42:13.127Z+ groupadd -g 12345 build
802023-11-07T09:42:13.151Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-07T09:42:15.042Z+ zfs create -o mountpoint=/work rpool/work
822023-11-07T09:42:15.066Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-07T09:42:15.105Z+ home_fs=zfs
842023-11-07T09:42:15.129Z+ [[ zfs == autofs ]]
852023-11-07T09:42:15.154Z+ mkdir -p /home/build
862023-11-07T09:42:15.178Z+ chown build:build /home/build /work
872023-11-07T09:42:17.071Z+ chmod 0700 /home/build /work
882023-11-07T09:42:17.095Zprocess exited: duration 4050 ms, exit code 0
 
892023-11-07T09:42:17.156Zstarting task 1: "authentication"
902023-11-07T09:42:17.224Zprocess exited: duration 43 ms, exit code 0
 
912023-11-07T09:42:17.272Zstarting task 2: "build"
922023-11-07T09:42:17.304Z+ trap _exit_trap EXIT
932023-11-07T09:42:17.328Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-07T09:42:17.505Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-07T09:42:17.529Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-07T09:42:17.553Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-07T09:42:17.577Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-07T09:42:17.601Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-07T09:42:17.625Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-07T09:42:17.649Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-07T09:42:17.673Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-07T09:42:17.697Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-07T09:42:17.721Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-07T09:42:17.745Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-07T09:42:17.769Z+ [[ -d /opt/oxide ]]
1062023-11-07T09:42:17.793Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-07T09:42:17.817Z+ pfexec mkdir /opt/oxide
1082023-11-07T09:42:17.841Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-07T09:42:17.866Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-07T09:42:17.890Z+ cd /opt/oxide-underneath
1112023-11-07T09:42:17.915Z+ pfexec tar ceEp@/f - .
1122023-11-07T09:42:17.939Z+ cd /opt/oxide
1132023-11-07T09:42:17.963Z+ pfexec tar xveEp@/f -
1142023-11-07T09:42:17.989Zx ., 0 bytes, 0 tape blocks
1152023-11-07T09:42:18.013Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-07T09:42:18.037Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-07T09:42:18.061Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-07T09:42:18.085Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-07T09:42:18.109Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-07T09:42:18.133Z+ pfexec mkdir /opt/oxide/work
1212023-11-07T09:42:18.181Z+ pfexec chown build:build /opt/oxide/work
1222023-11-07T09:42:18.206Z+ cd /opt/oxide/work
1232023-11-07T09:42:18.230Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-07T09:42:19.278ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-07T09:42:19.302Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1262023-11-07T09:42:19.326Zx out/omicron-sled-agent.tar, 122153984 bytes, 238582 tape blocks
1272023-11-07T09:42:19.350Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-07T09:42:19.374Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-11-07T09:42:19.398Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-11-07T09:42:19.422Zx smf/sled-agent/non-gimlet/config.toml, 3102 bytes, 7 tape blocks
1312023-11-07T09:42:19.446Zx target/release/omicron-package, 42430032 bytes, 82872 tape blocks
1322023-11-07T09:42:19.470Zx tools/create_virtual_hardware.sh, 3067 bytes, 6 tape blocks
1332023-11-07T09:42:19.494Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-07T09:42:19.518Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-07T09:42:19.542Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-07T09:42:19.567Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-07T09:42:19.591Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-07T09:42:19.615Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-07T09:42:19.639Z
1402023-11-07T09:42:19.663Zreal 1.606552297
1412023-11-07T09:42:19.687Zuser 1.445933832
1422023-11-07T09:42:19.711Zsys 0.283737361
1432023-11-07T09:42:19.735Ztrap 0.000066521
1442023-11-07T09:42:19.759Ztflt 0.000000000
1452023-11-07T09:42:19.783Zdflt 0.000000000
1462023-11-07T09:42:19.807Zkflt 0.000000000
1472023-11-07T09:42:19.831Zlock 0.000000000
1482023-11-07T09:42:19.855Zslp 1.448355991
1492023-11-07T09:42:19.879Zlat 0.032908342
1502023-11-07T09:42:19.904Zstop 0.000127092
1512023-11-07T09:42:19.928Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-07T09:42:20.346Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-07T09:42:20.371Z+ mkdir tests
1542023-11-07T09:42:20.395Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-07T09:42:20.419Z+ ptime -m gunzip
1562023-11-07T09:42:20.443Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-750ee284a2fe677d
1572023-11-07T09:42:20.507Z
1582023-11-07T09:42:20.532Zreal 0.139322033
1592023-11-07T09:42:20.556Zuser 0.127096787
1602023-11-07T09:42:20.580Zsys 0.011971651
1612023-11-07T09:42:20.604Ztrap 0.000010870
1622023-11-07T09:42:20.629Ztflt 0.000000000
1632023-11-07T09:42:20.653Zdflt 0.000000000
1642023-11-07T09:42:20.677Zkflt 0.000000000
1652023-11-07T09:42:20.701Zlock 0.000000000
1662023-11-07T09:42:20.726Zslp 0.000021760
1672023-11-07T09:42:20.750Zlat 0.000021689
1682023-11-07T09:42:20.774Zstop 0.000126922
1692023-11-07T09:42:20.798Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-750ee284a2fe677d
1702023-11-07T09:42:20.823Z+ chmod a+x tests/bootstrap-750ee284a2fe677d
1712023-11-07T09:42:20.847Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-07T09:42:20.871Z+ ptime -m gunzip
1732023-11-07T09:42:20.896Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-07T09:42:22.827Z
1752023-11-07T09:42:22.852Zreal 2.308463017
1762023-11-07T09:42:22.876Zuser 2.144689655
1772023-11-07T09:42:22.901Zsys 0.163388038
1782023-11-07T09:42:22.925Ztrap 0.000171902
1792023-11-07T09:42:22.949Ztflt 0.000000000
1802023-11-07T09:42:22.973Zdflt 0.000000000
1812023-11-07T09:42:22.997Zkflt 0.000000000
1822023-11-07T09:42:23.022Zlock 0.000000000
1832023-11-07T09:42:23.047Zslp 0.000016000
1842023-11-07T09:42:23.072Zlat 0.000017390
1852023-11-07T09:42:23.096Zstop 0.000120542
1862023-11-07T09:42:23.120Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-07T09:42:23.145Z+ chmod a+x tests/bootstrap
1882023-11-07T09:42:23.169Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-07T09:42:23.194Z+ ptime -m gunzip
1902023-11-07T09:42:23.218Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-13628b0bfef9b756
1912023-11-07T09:42:25.434Z
1922023-11-07T09:42:25.458Zreal 2.593520200
1932023-11-07T09:42:25.482Zuser 2.446344132
1942023-11-07T09:42:25.506Zsys 0.146699440
1952023-11-07T09:42:25.531Ztrap 0.000211623
1962023-11-07T09:42:25.555Ztflt 0.000000000
1972023-11-07T09:42:25.579Zdflt 0.000000000
1982023-11-07T09:42:25.603Zkflt 0.000000000
1992023-11-07T09:42:25.627Zlock 0.000000000
2002023-11-07T09:42:25.652Zslp 0.000025550
2012023-11-07T09:42:25.676Zlat 0.000032820
2022023-11-07T09:42:25.700Zstop 0.000131282
2032023-11-07T09:42:25.724Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-13628b0bfef9b756
2042023-11-07T09:42:25.748Z+ chmod a+x tests/end_to_end_tests-13628b0bfef9b756
2052023-11-07T09:42:25.772Z+ bmat address ls
2062023-11-07T09:42:25.797ZNAME CIDR FIRST LAST COUNT
2072023-11-07T09:42:25.821Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-11-07T09:42:25.845Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-07T09:42:25.869Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-11-07T09:42:25.893Z+ EXTRA_IP_START=10.151.1.164
2112023-11-07T09:42:25.917Z+ EXTRA_IP_END=10.151.1.227
2122023-11-07T09:42:25.950Z+ EXTRA_IP_COUNT=64
2132023-11-07T09:42:25.974Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-07T09:42:25.999Z+ EXTRA_IP_BASE=10.151.1
2152023-11-07T09:42:26.024Z+ EXTRA_IP_FOCTET=164
2162023-11-07T09:42:26.048Z+ EXTRA_IP_LOCTET=227
2172023-11-07T09:42:26.072Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-11-07T09:42:26.097Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-11-07T09:42:26.121Z+ DNS_IP1=10.151.1.164
2202023-11-07T09:42:26.145Z+ DNS_IP2=10.151.1.165
2212023-11-07T09:42:26.169Z+ UPLINK_IP=10.151.1.174
2222023-11-07T09:42:26.193Z+ IPPOOL_START=10.151.1.175
2232023-11-07T09:42:26.217Z+ IPPOOL_END=10.151.1.227
2242023-11-07T09:42:26.241Z++ ipadm show-addr -po type,addr
2252023-11-07T09:42:26.265Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-07T09:42:26.315Z+ GATEWAY_IP=10.151.1.100
2272023-11-07T09:42:26.339Z+ [[ -n 10.151.1.100 ]]
2282023-11-07T09:42:26.387Z+ ping -s 10.151.1.100 56 1
2292023-11-07T09:42:26.411ZPING 10.151.1.100: 56 data bytes
2302023-11-07T09:42:26.435Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.010 ms
2312023-11-07T09:42:26.459Z
2322023-11-07T09:42:26.484Z----10.151.1.100 PING Statistics----
2332023-11-07T09:42:26.508Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-07T09:42:26.533Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2352023-11-07T09:42:26.557Z++ arp -an
2362023-11-07T09:42:26.581Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-11-07T09:42:26.605Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-11-07T09:42:26.629Z+ routeadm -e ipv4-forwarding -u
2392023-11-07T09:42:26.654Z+ PXA_START=10.151.1.164
2402023-11-07T09:42:26.677Z+ PXA_END=10.151.1.227
2412023-11-07T09:42:26.701Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-07T09:42:26.726Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-07T09:42:26.750Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-07T09:42:26.774Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-07T09:42:26.798Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-07T09:42:26.822Z++ cd ./tools
2472023-11-07T09:42:26.846Z++ pwd
2482023-11-07T09:42:26.870Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-07T09:42:26.895Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-07T09:42:26.919Z+ SOFTNPU_MODE=zone
2512023-11-07T09:42:26.989Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-07T09:42:27.013Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-07T09:42:27.037Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-07T09:42:27.061Z++ dladm show-phys -p -o LINK
2552023-11-07T09:42:27.086Z++ head -1
2562023-11-07T09:42:27.110Z+ PHYSICAL_LINK=igb0
2572023-11-07T09:42:27.134Z+ echo 'Using igb0 as physical link'
2582023-11-07T09:42:27.159ZUsing igb0 as physical link
2592023-11-07T09:42:27.183Z+ ensure_run_as_root
2602023-11-07T09:42:27.207Z++ id -u
2612023-11-07T09:42:27.232Z+ [[ 0 -ne 0 ]]
2622023-11-07T09:42:27.256Z+ ensure_zpools
2632023-11-07T09:42:27.281Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-07T09:42:27.305Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-11-07T09:42:27.329Z+ readarray -t ZPOOLS
2662023-11-07T09:42:27.357Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-11-07T09:42:27.384Z++ sed 's/[ ",]//g'
2682023-11-07T09:42:27.432Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-11-07T09:42:27.462Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-11-07T09:42:27.498ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-11-07T09:42:27.523Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-11-07T09:42:27.548Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-11-07T09:42:27.572Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-11-07T09:42:27.596Z0+0 records in
2752023-11-07T09:42:27.620ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2762023-11-07T09:42:27.644Z0+0 records out
2772023-11-07T09:42:27.668ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2782023-11-07T09:42:27.692Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2792023-11-07T09:42:27.716ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2802023-11-07T09:42:27.740Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2812023-11-07T09:42:27.764ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2822023-11-07T09:42:27.789Z+ set +x
2832023-11-07T09:42:27.814ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2842023-11-07T09:42:27.838Z++ zpool list -o name
2852023-11-07T09:42:27.869ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2862023-11-07T09:42:27.897Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2872023-11-07T09:42:27.926ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2882023-11-07T09:42:27.959Z+ [[ -z '' ]]
2892023-11-07T09:42:27.984ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2902023-11-07T09:42:28.008Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2912023-11-07T09:42:28.032ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2922023-11-07T09:42:28.056Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2932023-11-07T09:42:28.080ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2942023-11-07T09:42:28.106Z+ set +x
2952023-11-07T09:42:28.130ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2962023-11-07T09:42:28.154Z+ for ZPOOL in "${ZPOOLS[@]}"
2972023-11-07T09:42:28.178ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2982023-11-07T09:42:28.203Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2992023-11-07T09:42:28.233ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3002023-11-07T09:42:28.259Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3012023-11-07T09:42:28.283ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3022023-11-07T09:42:28.307Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3032023-11-07T09:42:28.331ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3042023-11-07T09:42:28.357Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3052023-11-07T09:42:28.388ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3062023-11-07T09:42:28.415Z0+0 records in
3072023-11-07T09:42:28.459ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3082023-11-07T09:42:28.483Z0+0 records out
3092023-11-07T09:42:28.507ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3102023-11-07T09:42:28.531Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3112023-11-07T09:42:28.555ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3122023-11-07T09:42:28.580Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3132023-11-07T09:42:28.604ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3142023-11-07T09:42:28.628Z+ set +x
3152023-11-07T09:42:28.657ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3162023-11-07T09:42:28.684Z++ zpool list -o name
3172023-11-07T09:42:28.707ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3182023-11-07T09:42:28.732Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3192023-11-07T09:42:28.758ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3202023-11-07T09:42:28.786Z+ [[ -z '' ]]
3212023-11-07T09:42:28.815ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3222023-11-07T09:42:28.850Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3232023-11-07T09:42:28.875ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3242023-11-07T09:42:28.899Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3252023-11-07T09:42:28.923ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3262023-11-07T09:42:28.947Z+ set +x
3272023-11-07T09:42:28.971ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3282023-11-07T09:42:28.996Z+ for ZPOOL in "${ZPOOLS[@]}"
3292023-11-07T09:42:29.020ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3302023-11-07T09:42:29.043Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3312023-11-07T09:42:29.071ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3322023-11-07T09:42:29.097Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3332023-11-07T09:42:29.122ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3342023-11-07T09:42:29.146Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3352023-11-07T09:42:29.170ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3362023-11-07T09:42:29.197Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3372023-11-07T09:42:29.227ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3382023-11-07T09:42:29.254Z0+0 records in
3392023-11-07T09:42:29.682ZSimnet net0/sc0_0 exists
3402023-11-07T09:42:29.715Z0+0 records out
3412023-11-07T09:42:30.178ZSimnet net1/sc1_0 exists
3422023-11-07T09:42:30.206Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3432023-11-07T09:42:30.230ZVnic sc0_1 exists
3442023-11-07T09:42:30.254Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3452023-11-07T09:42:30.278Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3462023-11-07T09:42:30.302Z+ set +x
3472023-11-07T09:42:30.326Z++ zpool list -o name
3482023-11-07T09:42:30.350Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3492023-11-07T09:42:30.374Z+ [[ -z '' ]]
3502023-11-07T09:42:30.398Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3512023-11-07T09:42:30.422Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3522023-11-07T09:42:30.446Z+ set +x
3532023-11-07T09:42:30.471Z+ for ZPOOL in "${ZPOOLS[@]}"
3542023-11-07T09:42:30.514Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552023-11-07T09:42:30.542Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562023-11-07T09:42:30.566Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572023-11-07T09:42:30.608Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582023-11-07T09:42:30.639Z0+0 records in
3592023-11-07T09:42:30.663Z0+0 records out
3602023-11-07T09:42:30.687Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3612023-11-07T09:42:30.712Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622023-11-07T09:42:30.736Z+ set +x
3632023-11-07T09:42:30.760Z++ zpool list -o name
3642023-11-07T09:42:30.784Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3652023-11-07T09:42:30.808Z+ [[ -z '' ]]
3662023-11-07T09:42:30.834Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3672023-11-07T09:42:30.858Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3682023-11-07T09:42:30.886Z+ set +x
3692023-11-07T09:42:30.911Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-11-07T09:42:30.939Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3712023-11-07T09:42:30.964Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3722023-11-07T09:42:30.987Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3732023-11-07T09:42:31.012Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3742023-11-07T09:42:31.043Z0+0 records in
3752023-11-07T09:42:31.077Z0+0 records out
3762023-11-07T09:42:31.102Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3772023-11-07T09:42:31.126Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3782023-11-07T09:42:31.150Z+ set +x
3792023-11-07T09:42:31.223Z++ zpool list -o name
3802023-11-07T09:42:31.247Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3812023-11-07T09:42:31.271Z+ [[ -z '' ]]
3822023-11-07T09:42:31.295Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3832023-11-07T09:42:31.319Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3842023-11-07T09:42:31.344Z+ set +x
3852023-11-07T09:42:31.368Z+ for ZPOOL in "${ZPOOLS[@]}"
3862023-11-07T09:42:31.392Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-11-07T09:42:31.416Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-11-07T09:42:31.440Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-11-07T09:42:31.465Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-11-07T09:42:31.492Z0+0 records in
3912023-11-07T09:42:31.540Z0+0 records out
3922023-11-07T09:42:31.565Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3932023-11-07T09:42:31.589Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942023-11-07T09:42:31.613Z+ set +x
3952023-11-07T09:42:31.637Z++ zpool list -o name
3962023-11-07T09:42:31.661Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972023-11-07T09:42:31.685Z+ [[ -z '' ]]
3982023-11-07T09:42:31.708Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992023-11-07T09:42:31.733Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002023-11-07T09:42:31.757Z+ set +x
4012023-11-07T09:42:31.781Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-07T09:42:31.805Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4032023-11-07T09:42:31.829Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-11-07T09:42:31.853Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4052023-11-07T09:42:31.877Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4062023-11-07T09:42:31.901Z0+0 records in
4072023-11-07T09:42:31.925Z0+0 records out
4082023-11-07T09:42:31.949Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4092023-11-07T09:42:31.972Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4102023-11-07T09:42:31.996Z+ set +x
4112023-11-07T09:42:32.020Z++ zpool list -o name
4122023-11-07T09:42:32.045Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4132023-11-07T09:42:32.069Z+ [[ -z '' ]]
4142023-11-07T09:42:32.097Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4152023-11-07T09:42:32.122Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4162023-11-07T09:42:32.146Z+ set +x
4172023-11-07T09:42:32.172Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-11-07T09:42:32.196Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4192023-11-07T09:42:32.220Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4202023-11-07T09:42:32.244Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4212023-11-07T09:42:32.268Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4222023-11-07T09:42:32.292Z0+0 records in
4232023-11-07T09:42:32.319Z0+0 records out
4242023-11-07T09:42:32.342Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4252023-11-07T09:42:32.366Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4262023-11-07T09:42:32.390Z+ set +x
4272023-11-07T09:42:32.414Z++ zpool list -o name
4282023-11-07T09:42:32.438Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4292023-11-07T09:42:32.463Z+ [[ -z '' ]]
4302023-11-07T09:42:32.487Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4312023-11-07T09:42:32.511Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4322023-11-07T09:42:32.535Z+ set +x
4332023-11-07T09:42:32.566Z+ for ZPOOL in "${ZPOOLS[@]}"
4342023-11-07T09:42:32.590Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4352023-11-07T09:42:32.614Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-11-07T09:42:32.638Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4372023-11-07T09:42:32.662Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4382023-11-07T09:42:32.686Z0+0 records in
4392023-11-07T09:42:32.710Z0+0 records out
4402023-11-07T09:42:32.734Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4412023-11-07T09:42:32.758Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4422023-11-07T09:42:32.785Z+ set +x
4432023-11-07T09:42:32.842Z++ zpool list -o name
4442023-11-07T09:42:32.882Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4452023-11-07T09:42:32.907Z+ [[ -z '' ]]
4462023-11-07T09:42:32.931Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4472023-11-07T09:42:32.955Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4482023-11-07T09:42:32.979Z+ set +x
4492023-11-07T09:42:33.003Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4502023-11-07T09:42:33.027Z+ readarray -t ZPOOLS
4512023-11-07T09:42:33.051Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4522023-11-07T09:42:33.075Z++ sed 's/[ ",]//g'
4532023-11-07T09:42:33.099Z+ for ZPOOL in "${ZPOOLS[@]}"
4542023-11-07T09:42:33.123Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4552023-11-07T09:42:33.147Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4562023-11-07T09:42:33.171Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4572023-11-07T09:42:33.195Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4582023-11-07T09:42:33.219Z0+0 records in
4592023-11-07T09:42:33.247Z0+0 records out
4602023-11-07T09:42:33.273Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4612023-11-07T09:42:33.297Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4622023-11-07T09:42:33.321Z+ set +x
4632023-11-07T09:42:33.345Z++ zpool list -o name
4642023-11-07T09:42:33.369Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4652023-11-07T09:42:33.393Z+ [[ -z '' ]]
4662023-11-07T09:42:33.417Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4672023-11-07T09:42:33.442Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4682023-11-07T09:42:33.467Z+ set +x
4692023-11-07T09:42:33.491Z+ for ZPOOL in "${ZPOOLS[@]}"
4702023-11-07T09:42:33.515Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4712023-11-07T09:42:33.539Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4722023-11-07T09:42:33.563Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4732023-11-07T09:42:33.591Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4742023-11-07T09:42:33.616Z0+0 records in
4752023-11-07T09:42:33.640Z0+0 records out
4762023-11-07T09:42:33.664Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4772023-11-07T09:42:33.688Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4782023-11-07T09:42:33.712Z+ set +x
4792023-11-07T09:42:33.736Z++ zpool list -o name
4802023-11-07T09:42:33.761Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4812023-11-07T09:42:33.785Z+ [[ -z '' ]]
4822023-11-07T09:42:33.809Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4832023-11-07T09:42:33.833Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4842023-11-07T09:42:33.857Z+ set +x
4852023-11-07T09:42:33.881Z+ [[ zone == \z\o\n\e ]]
4862023-11-07T09:42:33.905Z+ ensure_simulated_links igb0
4872023-11-07T09:42:33.929Z+ local PHYSICAL_LINK=igb0
4882023-11-07T09:42:33.953Zdownloading softnpu ...done
4892023-11-07T09:42:33.977Z+ INDICES=("0" "1")
4902023-11-07T09:42:34.002Z+ for I in "${INDICES[@]}"
4912023-11-07T09:42:34.026Z++ get_simnet_name_if_exists net0
4922023-11-07T09:42:34.050Z++ dladm show-simnet -p -o LINK net0
4932023-11-07T09:42:34.078Z++ echo ''
4942023-11-07T09:42:34.111Z+ [[ -z '' ]]
4952023-11-07T09:42:34.146Z+ dladm create-simnet -t net0
4962023-11-07T09:42:34.173Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
4972023-11-07T09:42:34.200Z+ dladm create-simnet -t sc0_0
4982023-11-07T09:42:34.228Z+ dladm modify-simnet -t -p net0 sc0_0
4992023-11-07T09:42:34.252Z+ dladm set-linkprop -p mtu=9000 sc0_0
5002023-11-07T09:42:34.277Z+ success 'Simnet net0/sc0_0 exists'
5012023-11-07T09:42:34.301Z+ set +x
5022023-11-07T09:42:34.325Z+ for I in "${INDICES[@]}"
5032023-11-07T09:42:34.349Z++ get_simnet_name_if_exists net1
5042023-11-07T09:42:34.373Z++ dladm show-simnet -p -o LINK net1
5052023-11-07T09:42:34.397Z++ echo ''
5062023-11-07T09:42:34.421Z+ [[ -z '' ]]
5072023-11-07T09:42:34.444Z+ dladm create-simnet -t net1
5082023-11-07T09:42:34.468Z+ dladm create-simnet -t sc1_0
5092023-11-07T09:42:34.494Z+ dladm modify-simnet -t -p net1 sc1_0
5102023-11-07T09:42:34.519Z+ dladm set-linkprop -p mtu=9000 sc1_0
5112023-11-07T09:42:34.554Z+ success 'Simnet net1/sc1_0 exists'
5122023-11-07T09:42:34.983Z+ set +x
5132023-11-07T09:42:35.020Z++ get_vnic_name_if_exists sc0_1
5142023-11-07T09:42:35.051Z++ dladm show-vnic -p -o LINK sc0_1
5152023-11-07T09:42:35.082Z++ echo ''
5162023-11-07T09:42:35.107Z+ [[ -z '' ]]
5172023-11-07T09:42:35.131Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5182023-11-07T09:42:35.155Z+ [[ -v PROMISC_FILT_OFF ]]
5192023-11-07T09:42:35.179Z+ success 'Vnic sc0_1 exists'
5202023-11-07T09:42:35.203Z+ set +x
5212023-11-07T09:42:35.228Z+ warn_if_no_proxy_arp
5222023-11-07T09:42:35.252Z+ [[ -v PXA_START ]]
5232023-11-07T09:42:35.276Z+ [[ -v PXA_END ]]
5242023-11-07T09:42:35.300Z+ ensure_softnpu_zone
5252023-11-07T09:42:35.324Z+ zoneadm list
5262023-11-07T09:42:35.348Z+ grep -q sidecar_softnpu
5272023-11-07T09:42:35.372Z+ [[ -f out/npuzone/npuzone ]]
5282023-11-07T09:42:35.396Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch main
5292023-11-07T09:42:38.004Zdownloading asic_program.so ...done
5302023-11-07T09:42:38.190Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-07T09:42:42.972Zdownloading scadm ...done
5322023-11-07T09:42:42.997Zcreating port tfportrear0_0 for sc0_0
5332023-11-07T09:42:43.021Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-07T09:42:43.045Zsoftnpu zone setup
5352023-11-07T09:42:43.069Zconfigure zone
5362023-11-07T09:42:43.094Zinstall zone
5372023-11-07T09:42:44.087Zboot zone
5382023-11-07T09:42:47.195Z[sidecar_softnpu] svcs milestone/network
5392023-11-07T09:42:47.221ZSTATE STIME FMRI
5402023-11-07T09:42:47.246Zonline 9:42:44 svc:/milestone/network:default
5412023-11-07T09:42:47.270Z[sidecar_softnpu] /softnpu/npu start
5422023-11-07T09:42:47.295Z
5432023-11-07T09:42:47.319Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-07T09:42:47.343Z+ GATEWAY_IP=10.151.1.100
5452023-11-07T09:42:47.368Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-11-07T09:42:47.392ZUsing 10.151.1.100 as gateway ip
5472023-11-07T09:42:47.416Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-07T09:42:47.440Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-11-07T09:42:47.466Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-07T09:42:47.490Z+++ echo 18:c0:4d:0d:9f:b2
5512023-11-07T09:42:47.514Z+ COUNT=1
5522023-11-07T09:42:47.538Z+ [[ 1 -ne 1 ]]
5532023-11-07T09:42:47.563Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-11-07T09:42:47.587ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-11-07T09:42:47.611Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-11-07T09:42:47.635Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-11-07T09:42:47.659Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-07T09:42:47.683Z+ [[ -v PXA_START ]]
5592023-11-07T09:42:47.708Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-11-07T09:42:47.732Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-11-07T09:42:47.757Z+ z_scadm dump-state
5622023-11-07T09:42:47.781Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-07T09:42:47.805Zlocal v6:
5642023-11-07T09:42:47.830Zlocal v4:
5652023-11-07T09:42:47.854Zrouter v6:
5662023-11-07T09:42:47.878Zrouter v4_idx:
5672023-11-07T09:42:47.902Zrouter v4_routes:
5682023-11-07T09:42:47.926Zresolver v4:
5692023-11-07T09:42:47.950Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-11-07T09:42:47.977Zresolver v6:
5712023-11-07T09:42:48.001Znat v4:
5722023-11-07T09:42:48.026Znat_v6:
5732023-11-07T09:42:48.050Zport_mac:
5742023-11-07T09:42:48.074Zproxy arp:
5752023-11-07T09:42:48.098Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-11-07T09:42:48.123Z+ success 'softnpu zone exists'
5772023-11-07T09:42:48.147Z+ set +x
5782023-11-07T09:42:48.171Zsoftnpu zone exists
5792023-11-07T09:42:48.196Z
5802023-11-07T09:42:48.220Zreal 21.593382300
5812023-11-07T09:42:48.245Zuser 0.720842822
5822023-11-07T09:42:48.270Zsys 1.651225547
5832023-11-07T09:42:48.296Ztrap 0.000081307
5842023-11-07T09:42:48.321Ztflt 0.000162735
5852023-11-07T09:42:48.345Zdflt 0.001038605
5862023-11-07T09:42:48.369Zkflt 0.000000000
5872023-11-07T09:42:48.393Zlock 16:47.366478462
5882023-11-07T09:42:48.417Zslp 1:04.026811757
5892023-11-07T09:42:48.441Zlat 0.077461706
5902023-11-07T09:42:48.466Zstop 0.345752012
5912023-11-07T09:42:48.489Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-07T09:42:48.514Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-07T09:42:48.538Z+ SILO_NAME=recovery
5942023-11-07T09:42:48.562Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-07T09:42:48.586Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-07T09:42:48.610Z+ sed -i~ '
5972023-11-07T09:42:48.634Z /^external_dns_ips/c\
5982023-11-07T09:42:48.658Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-11-07T09:42:48.682Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-07T09:42:48.706Z /^first/c\
6012023-11-07T09:42:48.730Zfirst = "10.151.1.164"
6022023-11-07T09:42:48.754Z /^last/c\
6032023-11-07T09:42:48.778Zlast = "10.151.1.173"
6042023-11-07T09:42:48.802Z }
6052023-11-07T09:42:48.828Z /^\[rack_network_config/,/^$/ {
6062023-11-07T09:42:48.852Z /^infra_ip_first/c\
6072023-11-07T09:42:48.876Zinfra_ip_first = "10.151.1.174"
6082023-11-07T09:42:48.900Z /^infra_ip_last/c\
6092023-11-07T09:42:48.924Zinfra_ip_last = "10.151.1.174"
6102023-11-07T09:42:48.948Z }
6112023-11-07T09:42:48.972Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-11-07T09:42:49.004Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6132023-11-07T09:42:49.028Z /^routes/c\
6142023-11-07T09:42:49.053Z+++ pkg/config-rss.toml Tue Nov 7 09:42:46 2023
6152023-11-07T09:42:49.077Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6162023-11-07T09:42:49.101Z@@ -41,7 +41,7 @@
6172023-11-07T09:42:49.125Z /^addresses/c\
6182023-11-07T09:42:49.149Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6192023-11-07T09:42:49.173Zaddresses = \["10.151.1.174/32"\]
6202023-11-07T09:42:49.197Z # must be contained in one of the "internal services" IP Pool ranges listed
6212023-11-07T09:42:49.221Z }
6222023-11-07T09:42:49.245Z # below.
6232023-11-07T09:42:49.269Z' pkg/config-rss.toml
6242023-11-07T09:42:49.293Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6252023-11-07T09:42:49.317Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6262023-11-07T09:42:49.341Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6272023-11-07T09:42:49.365Z+ true
6282023-11-07T09:42:49.390Z
6292023-11-07T09:42:49.417Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6302023-11-07T09:42:49.441Z # Initial TLS certificates for the external API
6312023-11-07T09:42:49.466Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6322023-11-07T09:42:49.490Z #
6332023-11-07T09:42:49.515Z+ TLS_NAME=recovery.sys.oxide.test
6342023-11-07T09:42:49.539Z@@ -77,8 +77,8 @@
6352023-11-07T09:42:49.563Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6362023-11-07T09:42:49.588Z #
6372023-11-07T09:42:49.612Z..........+..+...+....+........+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...+..+.+........+......+......+...+.......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....................+..+...+....+........+...+...+.+...+...................................+...+............+...+.......+...............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6382023-11-07T09:42:49.636Z # For more on this and what to put here, see docs/how-to-run.adoc.
6392023-11-07T09:42:49.660Z.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+...........+.+..+.......+......+.....+....+.........+.........+........+.......+..+......+...+.+...+...+.........+...+...........+...................+...+..+......+.+..............+....+..+......+............+.+.....+............+.........+.+...+...........+....+...........+...+...+....+.........+.........+............+........+..................+.........+..........+.....+.........+......+..........+.........+..+..........+...+...............+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6402023-11-07T09:42:49.684Z [[internal_services_ip_pool_ranges]]
6412023-11-07T09:42:49.709Z-----
6422023-11-07T09:42:49.734Z-first = "192.168.1.20"
6432023-11-07T09:42:49.758Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6442023-11-07T09:42:49.782Z-last = "192.168.1.29"
6452023-11-07T09:42:49.806Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6462023-11-07T09:42:49.830Z+first = "10.151.1.164"
6472023-11-07T09:42:49.854Z+ rmdir pkg
6482023-11-07T09:42:49.878Z+last = "10.151.1.173"
6492023-11-07T09:42:49.902Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6502023-11-07T09:42:49.926Z
6512023-11-07T09:42:49.950Z+ pfexec mkdir -p /usr/oxide
6522023-11-07T09:42:49.976Z # TODO - this configuration is subject to change going forward. Ultimately these
6532023-11-07T09:42:50.000Z+ pfexec rm -f /usr/oxide/tcpproxy
6542023-11-07T09:42:50.032Z # parameters should be provided to the control plane via wicket, but we need to
6552023-11-07T09:42:50.056Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6562023-11-07T09:42:50.081Z@@ -92,8 +92,8 @@
6572023-11-07T09:42:50.105Z+ pfexec chmod +x /usr/oxide/tcpproxy
6582023-11-07T09:42:50.129Z # A range of IP addresses used by Boundary Services on the external network. In
6592023-11-07T09:42:50.153Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6602023-11-07T09:42:50.177Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6612023-11-07T09:42:50.201Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6622023-11-07T09:42:50.225Z # With softnpu, only one address is used.
6632023-11-07T09:42:50.250Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6642023-11-07T09:42:50.274Z-infra_ip_first = "192.168.1.30"
6652023-11-07T09:42:50.299Z+ OMICRON_NO_UNINSTALL=1
6662023-11-07T09:42:50.323Z-infra_ip_last = "192.168.1.30"
6672023-11-07T09:42:50.347Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6682023-11-07T09:42:50.371Z+infra_ip_first = "10.151.1.174"
6692023-11-07T09:42:50.395ZNov 07 09:42:47.915 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6702023-11-07T09:42:50.419Z+infra_ip_last = "10.151.1.174"
6712023-11-07T09:42:50.443ZNov 07 09:42:47.917 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6722023-11-07T09:42:50.466Z
6732023-11-07T09:42:50.490ZNov 07 09:42:47.918 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6742023-11-07T09:42:50.514Z # Configurations for BGP routers to run on the scrimlets.
6752023-11-07T09:42:50.540ZNov 07 09:42:47.918 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6762023-11-07T09:42:50.564Z bgp = []
6772023-11-07T09:42:50.588ZNov 07 09:42:47.918 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6782023-11-07T09:42:50.612Z@@ -101,9 +101,9 @@
6792023-11-07T09:42:50.637ZNov 07 09:42:47.918 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6802023-11-07T09:42:50.661Z # You can configure multiple uplinks by repeating the following stanza
6812023-11-07T09:42:50.685ZNov 07 09:42:47.918 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6822023-11-07T09:42:50.709Z [[rack_network_config.ports]]
6832023-11-07T09:42:50.733ZNov 07 09:42:47.918 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6842023-11-07T09:42:50.757Z # Routes associated with this port.
6852023-11-07T09:42:50.781ZNov 07 09:42:47.918 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6862023-11-07T09:42:50.806Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6872023-11-07T09:42:50.832ZNov 07 09:42:47.918 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6882023-11-07T09:42:50.856Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6892023-11-07T09:42:50.880ZNov 07 09:42:47.918 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6902023-11-07T09:42:50.904Z # Addresses associated with this port.
6912023-11-07T09:42:50.928ZNov 07 09:42:47.918 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6922023-11-07T09:42:50.952Z-addresses = ["192.168.1.30/32"]
6932023-11-07T09:42:50.976ZNov 07 09:42:47.918 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6942023-11-07T09:42:51.000Z+addresses = ["10.151.1.174/32"]
6952023-11-07T09:42:51.024ZNov 07 09:42:47.918 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6962023-11-07T09:42:51.058Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6972023-11-07T09:42:51.084ZNov 07 09:42:47.918 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6982023-11-07T09:42:51.108Z port = "qsfp0"
6992023-11-07T09:42:51.132ZNov 07 09:42:47.918 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7002023-11-07T09:42:51.159Z # The speed of this port.
7012023-11-07T09:42:51.183ZNov 07 09:42:47.918 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7022023-11-07T09:42:51.207Za pkg/config-rss.toml 7K
7032023-11-07T09:42:51.231ZNov 07 09:42:48.142 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7042023-11-07T09:42:51.256Za pkg/initial-tls-cert.pem 2K
7052023-11-07T09:42:51.280ZNov 07 09:42:48.178 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7062023-11-07T09:42:51.304Za pkg/initial-tls-key.pem 4K
7072023-11-07T09:42:51.328ZNov 07 09:42:48.243 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-07T09:42:51.352Z
7092023-11-07T09:42:51.403Zreal 0.465973954
7102023-11-07T09:42:51.427Zuser 0.099142369
7112023-11-07T09:42:51.451Zsys 1.830025305
7122023-11-07T09:42:51.475Ztrap 0.000000280
7132023-11-07T09:42:51.499Ztflt 0.000013479
7142023-11-07T09:42:51.523Zdflt 0.000346116
7152023-11-07T09:42:51.547Zkflt 0.000000000
7162023-11-07T09:42:51.576Zlock 41.876380960
7172023-11-07T09:42:51.604Zslp 0.666079936
7182023-11-07T09:42:51.628Zlat 0.003005460
7192023-11-07T09:42:51.652Zstop 0.202921368
7202023-11-07T09:42:51.677Z+ retry=0
7212023-11-07T09:42:51.702Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-07T09:42:51.726Z+ [[ 0 -gt 30 ]]
7232023-11-07T09:42:51.750Z+ sleep 1
7242023-11-07T09:42:51.774Z+ retry=1
7252023-11-07T09:42:51.798Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-07T09:42:51.823Z+ [[ 1 -gt 30 ]]
7272023-11-07T09:42:51.847Z+ sleep 1
7282023-11-07T09:42:51.872Z+ retry=2
7292023-11-07T09:42:51.896Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-07T09:43:16.151Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7312023-11-07T09:43:16.175Zlocal v6:
7322023-11-07T09:43:16.199Zfe80::8c08:deff:fe21:ec1f
7332023-11-07T09:43:16.223Zfd00:99::1
7342023-11-07T09:43:16.247Zlocal v4:
7352023-11-07T09:43:16.270Z10.151.1.174
7362023-11-07T09:43:16.294Zrouter v6:
7372023-11-07T09:43:16.318Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::4a0:20ff:fef2:a72d (1)
7382023-11-07T09:43:16.342Zfd00:1122:3344:101::/64 -> fe80::4a0:20ff:fef2:a72d (1)
7392023-11-07T09:43:16.366Zrouter v4_idx:
7402023-11-07T09:43:16.390Z0.0.0.0/0 -> 0
7412023-11-07T09:43:16.414Zrouter v4_routes:
7422023-11-07T09:43:16.438Z0 -> 10.151.1.100 (2)
7432023-11-07T09:43:16.462Zresolver v4:
7442023-11-07T09:43:16.486Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7452023-11-07T09:43:16.510Z10.151.1.174 -> 1a:76:2a:45:eb:ca
7462023-11-07T09:43:16.534Zresolver v6:
7472023-11-07T09:43:16.563Zfe80::4a0:20ff:fef2:a72d -> 06:a0:20:f2:a7:2d
7482023-11-07T09:43:16.587Zfe80::8c08:deff:fe21:ec1f -> 8e:08:de:21:ec:1f
7492023-11-07T09:43:16.611Znat v4:
7502023-11-07T09:43:16.637Znat_v6:
7512023-11-07T09:43:16.661Zport_mac:
7522023-11-07T09:43:16.697Z1000: a8:40:25:5e:c0:bf
7532023-11-07T09:43:16.721Z1: a8:40:25:5e:c0:c0
7542023-11-07T09:43:16.745Zproxy arp:
7552023-11-07T09:43:16.769Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7562023-11-07T09:43:16.794Z+ retry=0
7572023-11-07T09:43:16.818Z+++ zoneadm list -n
7582023-11-07T09:43:16.842Z+++ grep oxz_ntp
7592023-11-07T09:43:16.866Z++ pfexec svcs -z -Hostate oxide/ntp
7602023-11-07T09:43:16.891Zsvcs: zone '-Hostate': entity not found
7612023-11-07T09:43:16.916Z++ true
7622023-11-07T09:43:16.940Z+ [[ '' != online ]]
7632023-11-07T09:43:16.965Z+ [[ 0 -gt 60 ]]
7642023-11-07T09:43:16.989Z+ sleep 1
7652023-11-07T09:43:17.189Z+ retry=1
7662023-11-07T09:43:17.214Z+++ zoneadm list -n
7672023-11-07T09:43:17.238Z+++ grep oxz_ntp
7682023-11-07T09:43:17.262Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-11-07T09:43:17.286Zsvcs: zone '-Hostate': entity not found
7702023-11-07T09:43:17.310Z++ true
7712023-11-07T09:43:17.334Z+ [[ '' != online ]]
7722023-11-07T09:43:17.358Z+ [[ 1 -gt 60 ]]
7732023-11-07T09:43:17.382Z+ sleep 1
7742023-11-07T09:43:18.211Z+ retry=2
7752023-11-07T09:43:18.235Z+++ zoneadm list -n
7762023-11-07T09:43:18.259Z+++ grep oxz_ntp
7772023-11-07T09:43:18.283Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-11-07T09:43:18.307Zsvcs: zone '-Hostate': entity not found
7792023-11-07T09:43:18.331Z++ true
7802023-11-07T09:43:18.355Z+ [[ '' != online ]]
7812023-11-07T09:43:18.380Z+ [[ 2 -gt 60 ]]
7822023-11-07T09:43:18.404Z+ sleep 1
7832023-11-07T09:43:19.233Z+ retry=3
7842023-11-07T09:43:19.257Z+++ zoneadm list -n
7852023-11-07T09:43:19.281Z+++ grep oxz_ntp
7862023-11-07T09:43:19.305Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-11-07T09:43:19.329Zsvcs: zone '-Hostate': entity not found
7882023-11-07T09:43:19.352Z++ true
7892023-11-07T09:43:19.377Z+ [[ '' != online ]]
7902023-11-07T09:43:19.401Z+ [[ 3 -gt 60 ]]
7912023-11-07T09:43:19.425Z+ sleep 1
7922023-11-07T09:43:20.253Z+ retry=4
7932023-11-07T09:43:20.277Z+++ zoneadm list -n
7942023-11-07T09:43:20.301Z+++ grep oxz_ntp
7952023-11-07T09:43:20.325Z++ pfexec svcs -z -Hostate oxide/ntp
7962023-11-07T09:43:20.350Zsvcs: zone '-Hostate': entity not found
7972023-11-07T09:43:20.374Z++ true
7982023-11-07T09:43:20.399Z+ [[ '' != online ]]
7992023-11-07T09:43:20.423Z+ [[ 4 -gt 60 ]]
8002023-11-07T09:43:20.447Z+ sleep 1
8012023-11-07T09:43:21.276Z+ retry=5
8022023-11-07T09:43:21.300Z+++ zoneadm list -n
8032023-11-07T09:43:21.324Z+++ grep oxz_ntp
8042023-11-07T09:43:21.348Z++ pfexec svcs -z -Hostate oxide/ntp
8052023-11-07T09:43:21.372Zsvcs: zone '-Hostate': entity not found
8062023-11-07T09:43:21.396Z++ true
8072023-11-07T09:43:21.420Z+ [[ '' != online ]]
8082023-11-07T09:43:21.444Z+ [[ 5 -gt 60 ]]
8092023-11-07T09:43:21.467Z+ sleep 1
8102023-11-07T09:43:22.297Z+ retry=6
8112023-11-07T09:43:22.321Z+++ zoneadm list -n
8122023-11-07T09:43:22.346Z+++ grep oxz_ntp
8132023-11-07T09:43:22.370Z++ pfexec svcs -z -Hostate oxide/ntp
8142023-11-07T09:43:22.394Zsvcs: zone '-Hostate': entity not found
8152023-11-07T09:43:22.419Z++ true
8162023-11-07T09:43:22.443Z+ [[ '' != online ]]
8172023-11-07T09:43:22.467Z+ [[ 6 -gt 60 ]]
8182023-11-07T09:43:22.491Z+ sleep 1
8192023-11-07T09:43:23.317Z+ retry=7
8202023-11-07T09:43:23.341Z+++ zoneadm list -n
8212023-11-07T09:43:23.365Z+++ grep oxz_ntp
8222023-11-07T09:43:23.389Z++ pfexec svcs -z -Hostate oxide/ntp
8232023-11-07T09:43:23.413Zsvcs: zone '-Hostate': entity not found
8242023-11-07T09:43:23.437Z++ true
8252023-11-07T09:43:23.461Z+ [[ '' != online ]]
8262023-11-07T09:43:23.485Z+ [[ 7 -gt 60 ]]
8272023-11-07T09:43:23.509Z+ sleep 1
8282023-11-07T09:43:24.339Z+ retry=8
8292023-11-07T09:43:24.363Z+++ zoneadm list -n
8302023-11-07T09:43:24.388Z+++ grep oxz_ntp
8312023-11-07T09:43:24.412Z++ pfexec svcs -z oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555 -Hostate oxide/ntp
8322023-11-07T09:43:24.436Zsvcs: Could not bind to repository server for zone oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555: repository server unavailable
8332023-11-07T09:43:24.460Z++ true
8342023-11-07T09:43:24.488Z+ [[ '' != online ]]
8352023-11-07T09:43:24.512Z+ [[ 8 -gt 60 ]]
8362023-11-07T09:43:24.536Z+ sleep 1
8372023-11-07T09:43:25.360Z+ retry=9
8382023-11-07T09:43:25.385Z+++ zoneadm list -n
8392023-11-07T09:43:25.410Z+++ grep oxz_ntp
8402023-11-07T09:43:25.434Z++ pfexec svcs -z oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555 -Hostate oxide/ntp
8412023-11-07T09:43:25.459Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8422023-11-07T09:43:25.483Z++ true
8432023-11-07T09:43:25.507Z+ [[ '' != online ]]
8442023-11-07T09:43:25.531Z+ [[ 9 -gt 60 ]]
8452023-11-07T09:43:25.555Z+ sleep 1
8462023-11-07T09:43:26.394Z+ retry=10
8472023-11-07T09:43:26.418Z+++ zoneadm list -n
8482023-11-07T09:43:26.442Z+++ grep oxz_ntp
8492023-11-07T09:43:26.466Z++ pfexec svcs -z oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555 -Hostate oxide/ntp
8502023-11-07T09:43:26.494Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8512023-11-07T09:43:26.519Z++ true
8522023-11-07T09:43:26.547Z+ [[ '' != online ]]
8532023-11-07T09:43:26.571Z+ [[ 10 -gt 60 ]]
8542023-11-07T09:43:26.595Z+ sleep 1
8552023-11-07T09:43:27.423Z+ retry=11
8562023-11-07T09:43:27.447Z+++ zoneadm list -n
8572023-11-07T09:43:27.471Z+++ grep oxz_ntp
8582023-11-07T09:43:27.494Z++ pfexec svcs -z oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555 -Hostate oxide/ntp
8592023-11-07T09:43:27.518Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8602023-11-07T09:43:27.542Z++ true
8612023-11-07T09:43:27.566Z+ [[ '' != online ]]
8622023-11-07T09:43:27.590Z+ [[ 11 -gt 60 ]]
8632023-11-07T09:43:27.614Z+ sleep 1
8642023-11-07T09:43:28.452Z+ retry=12
8652023-11-07T09:43:28.476Z+++ zoneadm list -n
8662023-11-07T09:43:28.500Z+++ grep oxz_ntp
8672023-11-07T09:43:28.524Z++ pfexec svcs -z oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555 -Hostate oxide/ntp
8682023-11-07T09:43:28.548Z+ [[ disabled != online ]]
8692023-11-07T09:43:28.572Z+ [[ 12 -gt 60 ]]
8702023-11-07T09:43:28.597Z+ sleep 1
8712023-11-07T09:43:29.490Z+ retry=13
8722023-11-07T09:43:29.518Z+++ zoneadm list -n
8732023-11-07T09:43:29.548Z+++ grep oxz_ntp
8742023-11-07T09:43:29.580Z++ pfexec svcs -z oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555 -Hostate oxide/ntp
8752023-11-07T09:43:29.604Z+ [[ disabled != online ]]
8762023-11-07T09:43:29.628Z+ [[ 13 -gt 60 ]]
8772023-11-07T09:43:29.657Z+ sleep 1
8782023-11-07T09:43:30.498Z+ retry=14
8792023-11-07T09:43:30.522Z+++ zoneadm list -n
8802023-11-07T09:43:30.546Z+++ grep oxz_ntp
8812023-11-07T09:43:30.570Z++ pfexec svcs -z oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555 -Hostate oxide/ntp
8822023-11-07T09:43:30.594Z+ [[ disabled != online ]]
8832023-11-07T09:43:30.618Z+ [[ 14 -gt 60 ]]
8842023-11-07T09:43:30.642Z+ sleep 1
8852023-11-07T09:43:31.520Z+ retry=15
8862023-11-07T09:43:31.544Z+++ zoneadm list -n
8872023-11-07T09:43:31.568Z+++ grep oxz_ntp
8882023-11-07T09:43:31.592Z++ pfexec svcs -z oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555 -Hostate oxide/ntp
8892023-11-07T09:43:31.616Z+ [[ disabled != online ]]
8902023-11-07T09:43:31.640Z+ [[ 15 -gt 60 ]]
8912023-11-07T09:43:31.664Z+ sleep 1
8922023-11-07T09:43:32.542Z+ retry=16
8932023-11-07T09:43:32.566Z+++ zoneadm list -n
8942023-11-07T09:43:32.590Z+++ grep oxz_ntp
8952023-11-07T09:43:32.614Z++ pfexec svcs -z oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555 -Hostate oxide/ntp
8962023-11-07T09:43:32.638Z+ [[ disabled != online ]]
8972023-11-07T09:43:32.662Z+ [[ 16 -gt 60 ]]
8982023-11-07T09:43:32.686Z+ sleep 1
8992023-11-07T09:43:33.563Z+ retry=17
9002023-11-07T09:43:33.588Z+++ zoneadm list -n
9012023-11-07T09:43:33.612Z+++ grep oxz_ntp
9022023-11-07T09:43:33.636Z++ pfexec svcs -z oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555 -Hostate oxide/ntp
9032023-11-07T09:43:33.660Z+ [[ disabled != online ]]
9042023-11-07T09:43:33.684Z+ [[ 17 -gt 60 ]]
9052023-11-07T09:43:33.708Z+ sleep 1
9062023-11-07T09:43:34.584Z+ retry=18
9072023-11-07T09:43:34.609Z+++ zoneadm list -n
9082023-11-07T09:43:34.633Z+++ grep oxz_ntp
9092023-11-07T09:43:34.657Z++ pfexec svcs -z oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555 -Hostate oxide/ntp
9102023-11-07T09:43:34.681Z+ [[ disabled != online ]]
9112023-11-07T09:43:34.705Z+ [[ 18 -gt 60 ]]
9122023-11-07T09:43:34.729Z+ sleep 1
 
9132023-11-07T09:43:35.565Zbuskin console login: Nov 7 09:43:34 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9142023-11-07T09:43:35.607Z+ retry=19
9152023-11-07T09:43:35.631Z+++ zoneadm list -n
9162023-11-07T09:43:35.655Z+++ grep oxz_ntp
9172023-11-07T09:43:35.679Z++ pfexec svcs -z oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555 -Hostate oxide/ntp
9182023-11-07T09:43:35.703Z+ [[ offline* != online ]]
9192023-11-07T09:43:35.729Z+ [[ 19 -gt 60 ]]
9202023-11-07T09:43:35.753Z+ sleep 1
9212023-11-07T09:43:36.633Z+ retry=20
9222023-11-07T09:43:36.658Z+++ zoneadm list -n
9232023-11-07T09:43:36.682Z+++ grep oxz_ntp
9242023-11-07T09:43:36.709Z++ pfexec svcs -z oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555 -Hostate oxide/ntp
9252023-11-07T09:43:36.733Z+ [[ online != online ]]
9262023-11-07T09:43:36.757Z+ echo 'Waited for chrony: 20s'
9272023-11-07T09:43:36.781ZWaited for chrony: 20s
9282023-11-07T09:43:36.807Z+ export RUST_BACKTRACE=1
9292023-11-07T09:43:36.831Z+ RUST_BACKTRACE=1
9302023-11-07T09:43:36.855Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9312023-11-07T09:43:36.880Z+ ./tests/bootstrap
9322023-11-07T09:43:36.904ZUsing external DNS server at 10.151.1.164:53
9332023-11-07T09:43:36.930Z2023-11-07 09:43:36.115399376 UTC: attempting to log into API
9342023-11-07T09:43:51.721Z2023-11-07 09:43:51.895040318 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9352023-11-07T09:43:52.723Z2023-11-07 09:43:52.895918552 UTC: attempting to log into API
9362023-11-07T09:44:07.755Z2023-11-07 09:44:07.929351462 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9372023-11-07T09:44:08.756Z2023-11-07 09:44:08.930246136 UTC: attempting to log into API
9382023-11-07T09:44:23.790Z2023-11-07 09:44:23.963881519 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9392023-11-07T09:44:24.792Z2023-11-07 09:44:24.965593297 UTC: attempting to log into API
9402023-11-07T09:44:39.824Z2023-11-07 09:44:39.998190133 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9412023-11-07T09:44:40.825Z2023-11-07 09:44:40.998974705 UTC: attempting to log into API
9422023-11-07T09:44:55.858Z2023-11-07 09:44:55.577509347 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9432023-11-07T09:44:56.858Z2023-11-07 09:44:56.578089265 UTC: attempting to log into API
9442023-11-07T09:45:11.893Z2023-11-07 09:45:11.612007434 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9452023-11-07T09:45:12.894Z2023-11-07 09:45:12.613606229 UTC: attempting to log into API
9462023-11-07T09:45:27.927Z2023-11-07 09:45:27.646679453 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9472023-11-07T09:45:28.928Z2023-11-07 09:45:28.647546836 UTC: attempting to log into API
9482023-11-07T09:45:43.961Z2023-11-07 09:45:43.680116002 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9492023-11-07T09:45:44.962Z2023-11-07 09:45:44.681110338 UTC: attempting to log into API
9502023-11-07T09:45:59.995Z2023-11-07 09:45:59.737972654 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9512023-11-07T09:46:00.996Z2023-11-07 09:46:00.741418899 UTC: attempting to log into API
9522023-11-07T09:46:16.030Z2023-11-07 09:46:15.815811095 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9532023-11-07T09:46:17.032Z2023-11-07 09:46:16.820235740 UTC: attempting to log into API
9542023-11-07T09:46:32.065Z2023-11-07 09:46:31.894557331 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9552023-11-07T09:46:33.067Z2023-11-07 09:46:32.898833158 UTC: attempting to log into API
9562023-11-07T09:46:48.101Z2023-11-07 09:46:47.973701368 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9572023-11-07T09:46:49.103Z2023-11-07 09:46:48.978169354 UTC: attempting to log into API
9582023-11-07T09:47:04.137Z2023-11-07 09:47:04.052468548 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9592023-11-07T09:47:05.138Z2023-11-07 09:47:05.056997413 UTC: attempting to log into API
9602023-11-07T09:47:20.177Z2023-11-07 09:47:20.134423090 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9612023-11-07T09:47:21.175Z2023-11-07 09:47:21.138518887 UTC: attempting to log into API
9622023-11-07T09:47:36.211Z2023-11-07 09:47:36.201270044 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9632023-11-07T09:47:37.213Z2023-11-07 09:47:37.203107783 UTC: attempting to log into API
9642023-11-07T09:47:52.247Z2023-11-07 09:47:52.234702821 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9652023-11-07T09:47:53.251Z2023-11-07 09:47:53.236942137 UTC: attempting to log into API
9662023-11-07T09:48:08.311Z2023-11-07 09:48:08.298987562 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9672023-11-07T09:48:09.314Z2023-11-07 09:48:09.299801894 UTC: attempting to log into API
 
9682023-11-07T09:48:11.838ZNov 7 09:48:11 bukin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9692023-11-07T09:48:18.996ZNov 7 09:48:18 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9702023-11-07T09:48:24.344Z2023-11-07 09:48:24.332386650 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9712023-11-07T09:48:25.043ZNov 7 09:48:24 bukin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9722023-11-07T09:48:25.345Z2023-11-07 09:48:25.333148112 UTC: attempting to log into API
 
9732023-11-07T09:48:31.372ZNov 7 09:48:30 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9742023-11-07T09:48:38.527ZNov 7 09:48:37 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9752023-11-07T09:48:40.379Z2023-11-07 09:48:40.366854917 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9762023-11-07T09:48:41.381Z2023-11-07 09:48:41.368392221 UTC: attempting to log into API
9772023-11-07T09:48:56.415Z2023-11-07 09:48:56.403828793 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9782023-11-07T09:48:57.417Z2023-11-07 09:48:57.405919747 UTC: attempting to log into API
9792023-11-07T09:49:08.230Z2023-11-07 09:49:08.218874908 UTC: login succeeded
9802023-11-07T09:49:08.262Zwaiting for nexus to come up...
9812023-11-07T09:49:08.332Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9822023-11-07T09:49:08.381Zensuring datasets are ready...
9832023-11-07T09:49:10.919Zlet's roll.
9842023-11-07T09:49:10.943Z+ rm ./tests/bootstrap
9852023-11-07T09:49:10.985Z+ for test_bin in tests/*
9862023-11-07T09:49:11.010Z+ ./tests/bootstrap-750ee284a2fe677d
9872023-11-07T09:49:11.034Z
9882023-11-07T09:49:11.058Zrunning 0 tests
9892023-11-07T09:49:11.082Z
9902023-11-07T09:49:11.107Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9912023-11-07T09:49:11.157Z
9922023-11-07T09:49:11.181Z+ for test_bin in tests/*
9932023-11-07T09:49:11.205Z+ ./tests/end_to_end_tests-13628b0bfef9b756
9942023-11-07T09:49:11.230Z
9952023-11-07T09:49:11.254Zrunning 2 tests
9962023-11-07T09:49:11.279Ztest no_spoof::no_spoof_here ... ok
9972023-11-07T09:50:11.015Ztest instance_launch::instance_launch has been running for over 60 seconds
9982023-11-07T09:52:05.440Ztest instance_launch::instance_launch ... ok
9992023-11-07T09:52:05.465Z
10002023-11-07T09:52:05.489Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 174.42s
10012023-11-07T09:52:05.513Z
10022023-11-07T09:52:05.537Z+ _exit_trap
10032023-11-07T09:52:05.561Z+ local status=0
10042023-11-07T09:52:05.586Z+ [[ 0 -eq 0 ]]
10052023-11-07T09:52:05.610Z+ exit 0
10062023-11-07T09:52:05.634Zprocess exited: duration 588148 ms, exit code 0
 
10072023-11-07T09:52:05.726Zfound 51 output files
10082023-11-07T09:52:05.777Zuploading: /var/svc/log/oxide-sled-agent:default.log (611179 bytes)
10092023-11-07T09:52:07.016Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10102023-11-07T09:52:07.043Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_7a6bf167-34a4-4694-b712-012526e2e784/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10112023-11-07T09:52:08.114Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_7a6bf167-34a4-4694-b712-012526e2e784/root/var/svc/log/oxide-cockroachdb:default.log
10122023-11-07T09:52:08.138Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_ced945b7-29e3-4f8b-afeb-5e4eabb1d539/root/var/svc/log/oxide-crucible-pantry:default.log (2331 bytes)
10132023-11-07T09:52:09.206Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_ced945b7-29e3-4f8b-afeb-5e4eabb1d539/root/var/svc/log/oxide-crucible-pantry:default.log
10142023-11-07T09:52:09.230Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_34a4cd78-d334-44bf-a360-c22b2e7fcc79/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
10152023-11-07T09:52:10.320Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_34a4cd78-d334-44bf-a360-c22b2e7fcc79/root/var/svc/log/oxide-crucible-agent:default.log
10162023-11-07T09:52:10.344Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_34a4cd78-d334-44bf-a360-c22b2e7fcc79/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10172023-11-07T09:52:11.414Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_34a4cd78-d334-44bf-a360-c22b2e7fcc79/root/var/svc/log/oxide-crucible-downstairs:default.log
10182023-11-07T09:52:11.454Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_34a4cd78-d334-44bf-a360-c22b2e7fcc79/root/var/svc/log/oxide-crucible-downstairs:downstairs-1259d3c6-ce0b-4533-a95b-d10d1f0424b3.log (5535 bytes)
10192023-11-07T09:52:12.525Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_34a4cd78-d334-44bf-a360-c22b2e7fcc79/root/var/svc/log/oxide-crucible-downstairs:downstairs-1259d3c6-ce0b-4533-a95b-d10d1f0424b3.log
10202023-11-07T09:52:12.548Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_6a526853-1300-435d-acbf-a9f1042a7a41/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10212023-11-07T09:52:13.618Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_6a526853-1300-435d-acbf-a9f1042a7a41/root/var/svc/log/oxide-crucible-pantry:default.log
10222023-11-07T09:52:13.642Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555/root/var/svc/log/oxide-ntp:default.log (65 bytes)
10232023-11-07T09:52:14.711Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555/root/var/svc/log/oxide-ntp:default.log
10242023-11-07T09:52:14.743Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555/root/var/svc/log/oxide-ntp:default.log.0 (1052 bytes)
10252023-11-07T09:52:15.805Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555/root/var/svc/log/oxide-ntp:default.log.0
10262023-11-07T09:52:15.831Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_3f5c7019-f2e1-4fbb-9c6b-929429317924/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
10272023-11-07T09:52:16.898Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_3f5c7019-f2e1-4fbb-9c6b-929429317924/root/var/svc/log/oxide-cockroachdb:default.log
10282023-11-07T09:52:16.922Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b1caf545-fc13-4628-a752-6edae488df9b/root/var/svc/log/oxide-crucible-agent:default.log (14149 bytes)
10292023-11-07T09:52:18.013Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b1caf545-fc13-4628-a752-6edae488df9b/root/var/svc/log/oxide-crucible-agent:default.log
10302023-11-07T09:52:18.037Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b1caf545-fc13-4628-a752-6edae488df9b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10312023-11-07T09:52:19.110Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b1caf545-fc13-4628-a752-6edae488df9b/root/var/svc/log/oxide-crucible-downstairs:default.log
10322023-11-07T09:52:19.133Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b1caf545-fc13-4628-a752-6edae488df9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-322ac441-4638-4183-a89c-65a73d9307e6.log (3050 bytes)
10332023-11-07T09:52:20.204Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b1caf545-fc13-4628-a752-6edae488df9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-322ac441-4638-4183-a89c-65a73d9307e6.log
10342023-11-07T09:52:20.228Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f2377967-c7ff-4048-80f1-c5ff07fb8102/root/var/svc/log/oxide-crucible-agent:default.log (13685 bytes)
10352023-11-07T09:52:21.318Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f2377967-c7ff-4048-80f1-c5ff07fb8102/root/var/svc/log/oxide-crucible-agent:default.log
10362023-11-07T09:52:21.341Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f2377967-c7ff-4048-80f1-c5ff07fb8102/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10372023-11-07T09:52:22.410Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f2377967-c7ff-4048-80f1-c5ff07fb8102/root/var/svc/log/oxide-crucible-downstairs:default.log
10382023-11-07T09:52:22.434Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f2377967-c7ff-4048-80f1-c5ff07fb8102/root/var/svc/log/oxide-crucible-downstairs:downstairs-10fc7cde-0770-44da-901a-2ad928282f7a.log (3049 bytes)
10392023-11-07T09:52:23.503Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f2377967-c7ff-4048-80f1-c5ff07fb8102/root/var/svc/log/oxide-crucible-downstairs:downstairs-10fc7cde-0770-44da-901a-2ad928282f7a.log
10402023-11-07T09:52:23.527Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_f0462809-72ad-44fc-9343-3476eb4e4edb/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10412023-11-07T09:52:24.597Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_f0462809-72ad-44fc-9343-3476eb4e4edb/root/var/svc/log/oxide-crucible-pantry:default.log
10422023-11-07T09:52:24.620Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_1431427d-697f-4c08-bf42-0cdaefa31b13/root/var/svc/log/oxide-internal_dns:default.log (93574 bytes)
10432023-11-07T09:52:25.810Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_1431427d-697f-4c08-bf42-0cdaefa31b13/root/var/svc/log/oxide-internal_dns:default.log
10442023-11-07T09:52:25.834Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_1431427d-697f-4c08-bf42-0cdaefa31b13/root/var/svc/log/oxide-internal_dns:default.log.0 (8347 bytes)
10452023-11-07T09:52:26.925Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_1431427d-697f-4c08-bf42-0cdaefa31b13/root/var/svc/log/oxide-internal_dns:default.log.0
10462023-11-07T09:52:26.949Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_884023db-dcf2-48e2-ba08-cd062c22049c/root/var/svc/log/oxide-oximeter:default.log (195349 bytes)
10472023-11-07T09:52:27.126Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_884023db-dcf2-48e2-ba08-cd062c22049c/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 195349 -> 216103
10482023-11-07T09:52:28.195Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_884023db-dcf2-48e2-ba08-cd062c22049c/root/var/svc/log/oxide-oximeter:default.log
10492023-11-07T09:52:28.220Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_ff21b07a-0724-4a07-a28f-ed7de333a312/root/var/svc/log/oxide-external_dns:default.log (15611 bytes)
10502023-11-07T09:52:29.311Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_ff21b07a-0724-4a07-a28f-ed7de333a312/root/var/svc/log/oxide-external_dns:default.log
10512023-11-07T09:52:29.336Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_c0db90d1-bfbc-41f8-b588-75641acd9b1f/root/var/svc/log/oxide-nexus:default.log (500100 bytes)
10522023-11-07T09:52:29.583Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_c0db90d1-bfbc-41f8-b588-75641acd9b1f/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 500100 -> 501924
10532023-11-07T09:52:30.655Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_c0db90d1-bfbc-41f8-b588-75641acd9b1f/root/var/svc/log/oxide-nexus:default.log
10542023-11-07T09:52:30.679Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_161ad915-7994-49a7-b55d-6ff10b67ff0d/root/var/svc/log/oxide-clickhouse:default.log (739599 bytes)
10552023-11-07T09:52:30.941Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_161ad915-7994-49a7-b55d-6ff10b67ff0d/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 739599 -> 827638
10562023-11-07T09:52:32.010Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_161ad915-7994-49a7-b55d-6ff10b67ff0d/root/var/svc/log/oxide-clickhouse:default.log
10572023-11-07T09:52:32.034Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_772133cb-a4b0-4117-bb16-f82be5040d0b/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
10582023-11-07T09:52:33.125Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_772133cb-a4b0-4117-bb16-f82be5040d0b/root/var/svc/log/oxide-crucible-agent:default.log
10592023-11-07T09:52:33.149Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_772133cb-a4b0-4117-bb16-f82be5040d0b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10602023-11-07T09:52:34.218Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_772133cb-a4b0-4117-bb16-f82be5040d0b/root/var/svc/log/oxide-crucible-downstairs:default.log
10612023-11-07T09:52:34.242Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_772133cb-a4b0-4117-bb16-f82be5040d0b/root/var/svc/log/oxide-crucible-downstairs:downstairs-9ccec4e4-f3c8-4687-8514-2fff20095bbe.log (3050 bytes)
10622023-11-07T09:52:35.314Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_772133cb-a4b0-4117-bb16-f82be5040d0b/root/var/svc/log/oxide-crucible-downstairs:downstairs-9ccec4e4-f3c8-4687-8514-2fff20095bbe.log
10632023-11-07T09:52:35.339Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_a64380f5-c245-40fa-9aea-f5e0ba0c67d1/root/var/svc/log/oxide-nexus:default.log (688057 bytes)
10642023-11-07T09:52:36.198Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_a64380f5-c245-40fa-9aea-f5e0ba0c67d1/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 688057 -> 689881
10652023-11-07T09:52:37.267Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_a64380f5-c245-40fa-9aea-f5e0ba0c67d1/root/var/svc/log/oxide-nexus:default.log
10662023-11-07T09:52:37.291Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_8f67ffcb-e183-4301-aa77-aed5890c22b5/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10672023-11-07T09:52:38.361Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_8f67ffcb-e183-4301-aa77-aed5890c22b5/root/var/svc/log/oxide-cockroachdb:default.log
10682023-11-07T09:52:38.385Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_227aa729-a362-4a81-8281-978ebde57364/root/var/svc/log/oxide-internal_dns:default.log (93573 bytes)
10692023-11-07T09:52:39.569Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_227aa729-a362-4a81-8281-978ebde57364/root/var/svc/log/oxide-internal_dns:default.log
10702023-11-07T09:52:39.593Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_227aa729-a362-4a81-8281-978ebde57364/root/var/svc/log/oxide-internal_dns:default.log.0 (6315 bytes)
10712023-11-07T09:52:40.662Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_227aa729-a362-4a81-8281-978ebde57364/root/var/svc/log/oxide-internal_dns:default.log.0
10722023-11-07T09:52:40.686Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_faa84435-a424-477c-8c25-385868fb7e3c/root/var/svc/log/oxide-internal_dns:default.log (93567 bytes)
10732023-11-07T09:52:40.818Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_faa84435-a424-477c-8c25-385868fb7e3c/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93567 -> 97323
10742023-11-07T09:52:41.888Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_faa84435-a424-477c-8c25-385868fb7e3c/root/var/svc/log/oxide-internal_dns:default.log
10752023-11-07T09:52:41.912Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_faa84435-a424-477c-8c25-385868fb7e3c/root/var/svc/log/oxide-internal_dns:default.log.0 (4286 bytes)
10762023-11-07T09:52:42.982Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_faa84435-a424-477c-8c25-385868fb7e3c/root/var/svc/log/oxide-internal_dns:default.log.0
10772023-11-07T09:52:43.006Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_dd24aeb1-69c2-4647-b15b-59d2c2748927/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10782023-11-07T09:52:44.108Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_dd24aeb1-69c2-4647-b15b-59d2c2748927/root/var/svc/log/oxide-cockroachdb:default.log
10792023-11-07T09:52:44.132Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_de9f05db-7d53-4faf-8fab-ca75e8efbd1b/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10802023-11-07T09:52:45.201Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_de9f05db-7d53-4faf-8fab-ca75e8efbd1b/root/var/svc/log/oxide-cockroachdb:default.log
10812023-11-07T09:52:45.239Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_3ea741e1-ad42-4132-b97d-2618c33e4cd1/root/var/svc/log/oxide-nexus:default.log (4351307 bytes)
10822023-11-07T09:52:46.432Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_3ea741e1-ad42-4132-b97d-2618c33e4cd1/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4351307 -> 4411925
10832023-11-07T09:52:47.503Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_3ea741e1-ad42-4132-b97d-2618c33e4cd1/root/var/svc/log/oxide-nexus:default.log
10842023-11-07T09:52:47.527Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ad551723-6aee-46e3-b43b-16577f3490e3/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
10852023-11-07T09:52:48.595Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ad551723-6aee-46e3-b43b-16577f3490e3/root/var/svc/log/oxide-crucible-agent:default.log
10862023-11-07T09:52:48.619Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ad551723-6aee-46e3-b43b-16577f3490e3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10872023-11-07T09:52:49.687Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ad551723-6aee-46e3-b43b-16577f3490e3/root/var/svc/log/oxide-crucible-downstairs:default.log
10882023-11-07T09:52:49.711Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0524638f-9909-4def-abaf-e83c0e9ea968/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10892023-11-07T09:52:50.780Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0524638f-9909-4def-abaf-e83c0e9ea968/root/var/svc/log/oxide-crucible-agent:default.log
10902023-11-07T09:52:50.805Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0524638f-9909-4def-abaf-e83c0e9ea968/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10912023-11-07T09:52:51.873Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0524638f-9909-4def-abaf-e83c0e9ea968/root/var/svc/log/oxide-crucible-downstairs:default.log
10922023-11-07T09:52:51.897Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3b795c1a-55bb-467f-ac94-0095d3033217/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10932023-11-07T09:52:52.966Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3b795c1a-55bb-467f-ac94-0095d3033217/root/var/svc/log/oxide-crucible-agent:default.log
10942023-11-07T09:52:52.991Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3b795c1a-55bb-467f-ac94-0095d3033217/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10952023-11-07T09:52:54.059Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3b795c1a-55bb-467f-ac94-0095d3033217/root/var/svc/log/oxide-crucible-downstairs:default.log
10962023-11-07T09:52:54.083Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5bbbe0a1-834e-4dd1-b766-e1de3139da30/root/var/svc/log/oxide-crucible-agent:default.log (14149 bytes)
10972023-11-07T09:52:55.175Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5bbbe0a1-834e-4dd1-b766-e1de3139da30/root/var/svc/log/oxide-crucible-agent:default.log
10982023-11-07T09:52:55.198Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5bbbe0a1-834e-4dd1-b766-e1de3139da30/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10992023-11-07T09:52:56.267Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5bbbe0a1-834e-4dd1-b766-e1de3139da30/root/var/svc/log/oxide-crucible-downstairs:default.log
11002023-11-07T09:52:56.291Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5bbbe0a1-834e-4dd1-b766-e1de3139da30/root/var/svc/log/oxide-crucible-downstairs:downstairs-2363f236-d585-4d58-8174-2e8f9f8d1048.log (5535 bytes)
11012023-11-07T09:52:57.359Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5bbbe0a1-834e-4dd1-b766-e1de3139da30/root/var/svc/log/oxide-crucible-downstairs:downstairs-2363f236-d585-4d58-8174-2e8f9f8d1048.log
11022023-11-07T09:52:57.383Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9f9016a-abc8-4cad-800c-77ae9bddd55b/root/var/svc/log/oxide-crucible-agent:default.log (13685 bytes)
11032023-11-07T09:52:58.477Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9f9016a-abc8-4cad-800c-77ae9bddd55b/root/var/svc/log/oxide-crucible-agent:default.log
11042023-11-07T09:52:58.501Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9f9016a-abc8-4cad-800c-77ae9bddd55b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11052023-11-07T09:52:59.569Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9f9016a-abc8-4cad-800c-77ae9bddd55b/root/var/svc/log/oxide-crucible-downstairs:default.log
11062023-11-07T09:52:59.593Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9f9016a-abc8-4cad-800c-77ae9bddd55b/root/var/svc/log/oxide-crucible-downstairs:downstairs-f80bfdd1-5484-463b-8111-a18f0b79fc56.log (5536 bytes)
11072023-11-07T09:53:00.663Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9f9016a-abc8-4cad-800c-77ae9bddd55b/root/var/svc/log/oxide-crucible-downstairs:downstairs-f80bfdd1-5484-463b-8111-a18f0b79fc56.log
11082023-11-07T09:53:00.697Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_c058e482-eb07-4548-9a7a-c0e80ba18045/root/var/svc/log/oxide-external_dns:default.log (15603 bytes)
11092023-11-07T09:53:00.753Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_c058e482-eb07-4548-9a7a-c0e80ba18045/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15603 -> 19378
11102023-11-07T09:53:01.825Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_c058e482-eb07-4548-9a7a-c0e80ba18045/root/var/svc/log/oxide-external_dns:default.log
11112023-11-07T09:53:01.849Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555/root/var/log/chrony/measurements.log (14111 bytes)
11122023-11-07T09:53:01.919Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555/root/var/log/chrony/measurements.log" changed size mid upload: 14111 -> 15892
11132023-11-07T09:53:02.991Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555/root/var/log/chrony/measurements.log
11142023-11-07T09:53:03.016Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555/root/var/log/chrony/statistics.log (8190 bytes)
11152023-11-07T09:53:03.060Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555/root/var/log/chrony/statistics.log" changed size mid upload: 8190 -> 9594
11162023-11-07T09:53:04.131Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555/root/var/log/chrony/statistics.log
11172023-11-07T09:53:04.156Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555/root/var/log/chrony/tracking.log (1853 bytes)
11182023-11-07T09:53:05.229Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_36c6408e-3aec-4d74-8b24-2aa2d83b2555/root/var/log/chrony/tracking.log