01HCZX5CWQ01K2JDPA8ZB6P6G3: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.23",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HCZX6F4KH6T85NTXCYY4769X

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-17T23:38:12.867Zjob dependencies complete; ready to run (waiting for 50 m 2 s)
22023-10-17T23:40:17.006Zjob assigned to worker 01HD0027E3WRC9A7Z88VZJ0SWP (queued for 2 m 4 s)
32023-10-17T23:40:17.337Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-17T23:40:17.386Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-10-17T23:40:17.431Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-10-17T23:40:17.474Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-17T23:40:17.520Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-10-17T23:40:17.561ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-17T23:40:17.607Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-17T23:40:17.655ZOxide Helios Version helios-2.0.22216 64-bit
112023-10-17T23:40:17.705ZConfiguring devices.
122023-10-17T23:40:17.753Z#################################################
132023-10-17T23:40:17.807Z#### ############# ##
142023-10-17T23:40:17.844Z### ### ############ ##
152023-10-17T23:40:17.877Z## ### # ## ### ## Oxide Computer Company ##
162023-10-17T23:40:17.911Z## ## ## ### # ### ##
172023-10-17T23:40:17.985Z## # ### #### #### This Station Under ##
182023-10-17T23:40:18.031Z### ### #### # ### Computer Control ##
192023-10-17T23:40:18.066Z#### #### ### ## ##
202023-10-17T23:40:18.100Z##################################################
212023-10-17T23:40:18.545ZHostname: buskin
222023-10-17T23:40:18.578ZLoading smf(7) serice descriptions:
232023-10-17T23:40:19.793Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-323c561dc2d9ecdb.gz
242023-10-17T23:40:22.609Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-323c561dc2d9ecdb.gz
252023-10-17T23:40:22.636Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-17T23:40:53.774Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-17T23:40:53.799Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-183991de1eaf7f15.gz
282023-10-17T23:41:34.668Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-183991de1eaf7f15.gz
292023-10-17T23:41:34.692Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-17T23:41:37.233Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-17T23:41:37.278Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-17T23:41:43.029Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-17T23:41:43.052Zdownloading input: /input/package/work/version.txt
342023-10-17T23:41:43.077Zdownloaded input: /input/package/work/version.txt
352023-10-17T23:41:43.100Zdownloading input: /input/package/work/package.tar.gz
362023-10-17T23:42:15.434Zdownloaded input: /input/package/work/package.tar.gz
372023-10-17T23:42:15.874Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-17T23:43:06.518Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-17T23:43:06.542Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-17T23:43:18.065Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-17T23:43:18.090Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-17T23:43:52.963Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-17T23:43:52.988Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-17T23:44:26.873Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-17T23:44:26.898Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-17T23:44:47.264Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-17T23:44:47.288Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-17T23:44:51.258Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-17T23:44:51.282Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-17T23:44:58.529Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-17T23:44:58.554Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-17T23:45:02.424Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-17T23:45:02.448Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-17T23:45:06.376Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-17T23:45:06.400Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-17T23:45:06.424Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-17T23:45:06.448Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-17T23:45:09.243Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-17T23:45:09.268Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-17T23:45:11.988Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-17T23:45:12.012Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-17T23:45:23.638Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-17T23:45:23.664Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-17T23:45:35.107Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-17T23:45:35.131Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-17T23:45:35.168Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-17T23:45:35.192Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-17T23:45:38.323Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-17T23:45:38.347Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-17T23:45:49.754Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-17T23:45:49.778Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-17T23:46:26.607Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-17T23:46:26.641Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-17T23:46:51.296Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-17T23:46:51.322Zstarting task 0: "setup"
762023-10-17T23:46:51.355Z++ uname -s
772023-10-17T23:46:51.379Z+ kern=SunOS
782023-10-17T23:46:51.409Z+ case "$kern" in
792023-10-17T23:46:51.439Z+ groupadd -g 12345 build
802023-10-17T23:46:51.463Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-17T23:46:53.364Z+ zfs create -o mountpoint=/work rpool/work
822023-10-17T23:46:53.388Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-17T23:46:53.413Z+ home_fs=zfs
842023-10-17T23:46:53.437Z+ [[ zfs == autofs ]]
852023-10-17T23:46:53.462Z+ mkdir -p /home/build
862023-10-17T23:46:53.488Z+ chown build:build /home/build /work
872023-10-17T23:46:55.394Z+ chmod 0700 /home/build /work
882023-10-17T23:46:55.417Zprocess exited: duration 4051 ms, exit code 0
 
892023-10-17T23:46:55.465Zstarting task 1: "authentication"
902023-10-17T23:46:55.535Zprocess exited: duration 45 ms, exit code 0
 
912023-10-17T23:46:55.582Zstarting task 2: "build"
922023-10-17T23:46:55.614Z+ trap _exit_trap EXIT
932023-10-17T23:46:55.637Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-17T23:46:55.807Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-17T23:46:55.854Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-17T23:46:55.879Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-17T23:46:55.903Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-17T23:46:55.928Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-17T23:46:55.951Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-17T23:46:55.976Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-17T23:46:56.001Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-17T23:46:56.025Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-17T23:46:56.050Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-17T23:46:56.079Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-17T23:46:56.103Z+ [[ -d /opt/oxide ]]
1062023-10-17T23:46:56.135Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-17T23:46:56.163Z+ pfexec mkdir /opt/oxide
1082023-10-17T23:46:56.187Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-17T23:46:56.213Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-17T23:46:56.238Z+ cd /opt/oxide-underneath
1112023-10-17T23:46:56.262Z+ pfexec tar ceEp@/f - .
1122023-10-17T23:46:56.286Z+ cd /opt/oxide
1132023-10-17T23:46:56.309Z+ pfexec tar xveEp@/f -
1142023-10-17T23:46:56.333Zx ., 0 bytes, 0 tape blocks
1152023-10-17T23:46:56.357Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-17T23:46:56.381Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-17T23:46:56.405Zx ./opte/bin/opteadm, 50138640 bytes, 97928 tape blocks
1182023-10-17T23:46:56.428Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-17T23:46:56.452Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-17T23:46:56.476Z+ pfexec mkdir /opt/oxide/work
1212023-10-17T23:46:56.503Z+ pfexec chown build:build /opt/oxide/work
1222023-10-17T23:46:56.526Z+ cd /opt/oxide/work
1232023-10-17T23:46:56.550Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-17T23:46:57.583ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-17T23:46:57.608Zx out/maghemite.tar, 74957312 bytes, 146401 tape blocks
1262023-10-17T23:46:57.632Zx out/omicron-sled-agent.tar, 113853952 bytes, 222371 tape blocks
1272023-10-17T23:46:57.661Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-17T23:46:57.686Zx out/npuzone/npuzone, 37379944 bytes, 73008 tape blocks
1292023-10-17T23:46:57.709Zx package-manifest.toml, 21303 bytes, 42 tape blocks
1302023-10-17T23:46:57.733Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-17T23:46:57.758Zx target/release/omicron-package, 37966272 bytes, 74153 tape blocks
1322023-10-17T23:46:57.782Zx tools/create_virtual_hardware.sh, 2818 bytes, 6 tape blocks
1332023-10-17T23:46:57.805Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-17T23:46:57.829Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-17T23:46:57.855Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-17T23:46:57.916Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-17T23:46:57.940Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-17T23:46:57.965Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-17T23:46:57.991Z
1402023-10-17T23:46:58.016Zreal 1.611784311
1412023-10-17T23:46:58.040Zuser 1.443006737
1422023-10-17T23:46:58.065Zsys 0.292083393
1432023-10-17T23:46:58.091Ztrap 0.000066212
1442023-10-17T23:46:58.116Ztflt 0.000000000
1452023-10-17T23:46:58.140Zdflt 0.000000000
1462023-10-17T23:46:58.177Zkflt 0.000000000
1472023-10-17T23:46:58.201Zlock 0.000000000
1482023-10-17T23:46:58.225Zslp 1.450920649
1492023-10-17T23:46:58.250Zlat 0.035420582
1502023-10-17T23:46:58.274Zstop 0.000122191
1512023-10-17T23:46:58.299Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-17T23:46:58.599Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-17T23:46:58.625Z+ mkdir tests
1542023-10-17T23:46:58.650Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-17T23:46:58.676Z+ ptime -m gunzip
1562023-10-17T23:46:58.700Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-323c561dc2d9ecdb
1572023-10-17T23:46:58.749Z
1582023-10-17T23:46:58.773Zreal 0.140869251
1592023-10-17T23:46:58.797Zuser 0.128567392
1602023-10-17T23:46:58.820Zsys 0.012043143
1612023-10-17T23:46:58.844Ztrap 0.000012200
1622023-10-17T23:46:58.868Ztflt 0.000000000
1632023-10-17T23:46:58.892Zdflt 0.000000000
1642023-10-17T23:46:58.939Zkflt 0.000000000
1652023-10-17T23:46:58.967Zlock 0.000000000
1662023-10-17T23:46:58.991Zslp 0.000024810
1672023-10-17T23:46:59.015Zlat 0.000022820
1682023-10-17T23:46:59.039Zstop 0.000131393
1692023-10-17T23:46:59.063Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-323c561dc2d9ecdb
1702023-10-17T23:46:59.086Z+ chmod a+x tests/bootstrap-323c561dc2d9ecdb
1712023-10-17T23:46:59.110Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-17T23:46:59.136Z+ ptime -m gunzip
1732023-10-17T23:46:59.160Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-17T23:47:00.960Z
1752023-10-17T23:47:00.988Zreal 2.198394939
1762023-10-17T23:47:01.013Zuser 2.071071605
1772023-10-17T23:47:01.037Zsys 0.126901435
1782023-10-17T23:47:01.061Ztrap 0.000175815
1792023-10-17T23:47:01.086Ztflt 0.000000000
1802023-10-17T23:47:01.110Zdflt 0.000000000
1812023-10-17T23:47:01.134Zkflt 0.000000000
1822023-10-17T23:47:01.165Zlock 0.000000000
1832023-10-17T23:47:01.195Zslp 0.000021330
1842023-10-17T23:47:01.220Zlat 0.000031410
1852023-10-17T23:47:01.244Zstop 0.000123822
1862023-10-17T23:47:01.268Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-17T23:47:01.294Z+ chmod a+x tests/bootstrap
1882023-10-17T23:47:01.318Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-17T23:47:01.342Z+ ptime -m gunzip
1902023-10-17T23:47:01.366Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-183991de1eaf7f15
1912023-10-17T23:47:03.503Z
1922023-10-17T23:47:03.529Zreal 2.529373240
1932023-10-17T23:47:03.553Zuser 2.386093954
1942023-10-17T23:47:03.578Zsys 0.142854358
1952023-10-17T23:47:03.603Ztrap 0.000195982
1962023-10-17T23:47:03.632Ztflt 0.000000000
1972023-10-17T23:47:03.657Zdflt 0.000000000
1982023-10-17T23:47:03.682Zkflt 0.000000000
1992023-10-17T23:47:03.706Zlock 0.000000000
2002023-10-17T23:47:03.730Zslp 0.000018800
2012023-10-17T23:47:03.755Zlat 0.000019860
2022023-10-17T23:47:03.779Zstop 0.000124612
2032023-10-17T23:47:03.805Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-183991de1eaf7f15
2042023-10-17T23:47:03.830Z+ chmod a+x tests/end_to_end_tests-183991de1eaf7f15
2052023-10-17T23:47:03.856Z+ bmat address ls
2062023-10-17T23:47:03.881ZNAME CIDR FIRST LAST COUNT
2072023-10-17T23:47:03.905Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-10-17T23:47:03.929Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-17T23:47:03.954Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-10-17T23:47:03.978Z+ EXTRA_IP_START=10.151.1.164
2112023-10-17T23:47:04.003Z+ EXTRA_IP_END=10.151.1.227
2122023-10-17T23:47:04.027Z+ EXTRA_IP_COUNT=64
2132023-10-17T23:47:04.066Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-10-17T23:47:04.092Z+ EXTRA_IP_BASE=10.151.1
2152023-10-17T23:47:04.117Z+ EXTRA_IP_FOCTET=164
2162023-10-17T23:47:04.141Z+ EXTRA_IP_LOCTET=227
2172023-10-17T23:47:04.167Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-10-17T23:47:04.191Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-10-17T23:47:04.215Z+ DNS_IP1=10.151.1.164
2202023-10-17T23:47:04.240Z+ DNS_IP2=10.151.1.165
2212023-10-17T23:47:04.264Z+ UPLINK_IP=10.151.1.174
2222023-10-17T23:47:04.289Z+ IPPOOL_START=10.151.1.175
2232023-10-17T23:47:04.313Z+ IPPOOL_END=10.151.1.227
2242023-10-17T23:47:04.338Z++ ipadm show-addr -po type,addr
2252023-10-17T23:47:04.362Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-10-17T23:47:04.386Z+ GATEWAY_IP=10.151.1.100
2272023-10-17T23:47:04.410Z+ [[ -n 10.151.1.100 ]]
2282023-10-17T23:47:04.437Z+ ping -s 10.151.1.100 56 1
2292023-10-17T23:47:04.476ZPING 10.151.1.100: 56 data bytes
2302023-10-17T23:47:04.503Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.012 ms
2312023-10-17T23:47:04.528Z
2322023-10-17T23:47:04.554Z----10.151.1.100 PING Statistics----
2332023-10-17T23:47:04.578Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-10-17T23:47:04.601Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2352023-10-17T23:47:04.625Z++ arp -an
2362023-10-17T23:47:04.649Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-10-17T23:47:04.675Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-10-17T23:47:04.699Z+ routeadm -e ipv4-forwarding -u
2392023-10-17T23:47:04.724Z+ PXA_START=10.151.1.164
2402023-10-17T23:47:04.748Z+ PXA_END=10.151.1.227
2412023-10-17T23:47:04.772Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-10-17T23:47:04.796Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-10-17T23:47:04.848Z+ ZPOOL_VDEV_DIR=/scratch
2442023-10-17T23:47:04.873Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-10-17T23:47:04.898Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-10-17T23:47:04.922Z++ cd ./tools
2472023-10-17T23:47:04.950Z++ pwd
2482023-10-17T23:47:04.975Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-10-17T23:47:05.000Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-10-17T23:47:05.027Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-10-17T23:47:05.051Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-10-17T23:47:05.097Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-10-17T23:47:05.121Z++ dladm show-phys -p -o LINK
2542023-10-17T23:47:05.145Z++ head -1
2552023-10-17T23:47:05.169Z+ PHYSICAL_LINK=igb0
2562023-10-17T23:47:05.193Z+ echo 'Using igb0 as physical link'
2572023-10-17T23:47:05.218ZUsing igb0 as physical link
2582023-10-17T23:47:05.242Z+ ensure_run_as_root
2592023-10-17T23:47:05.266Z++ id -u
2602023-10-17T23:47:05.292Z+ [[ 0 -ne 0 ]]
2612023-10-17T23:47:05.317Z+ ensure_zpools
2622023-10-17T23:47:05.342Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-10-17T23:47:05.366Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-10-17T23:47:05.391Z+ readarray -t ZPOOLS
2652023-10-17T23:47:05.415Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-10-17T23:47:05.439Z++ sed 's/[ ",]//g'
2672023-10-17T23:47:05.471Z+ for ZPOOL in "${ZPOOLS[@]}"
2682023-10-17T23:47:05.498Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2692023-10-17T23:47:05.522ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2702023-10-17T23:47:06.579Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2712023-10-17T23:47:06.628Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2722023-10-17T23:47:06.663Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2732023-10-17T23:47:06.692Z0+0 records in
2742023-10-17T23:47:06.720Z0+0 records out
2752023-10-17T23:47:06.745ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2762023-10-17T23:47:06.774Z0 bytes (0 iB) transferred in 0.000007 secs (0 iB/sec)
2772023-10-17T23:47:06.798ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2782023-10-17T23:47:06.823Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2792023-10-17T23:47:06.859ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2802023-10-17T23:47:06.885Z+ set +x
2812023-10-17T23:47:06.909ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2822023-10-17T23:47:06.933Z++ zpool list -o name
2832023-10-17T23:47:06.957ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2842023-10-17T23:47:06.982Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2852023-10-17T23:47:07.006ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2862023-10-17T23:47:07.031Z+ [[ -z '' ]]
2872023-10-17T23:47:07.463ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2882023-10-17T23:47:07.490Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2892023-10-17T23:47:07.527ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2902023-10-17T23:47:07.569Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2912023-10-17T23:47:07.594ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2922023-10-17T23:47:07.617Z+ set +x
2932023-10-17T23:47:07.642ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2942023-10-17T23:47:07.666Z+ for ZPOOL in "${ZPOOLS[@]}"
2952023-10-17T23:47:07.690ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2962023-10-17T23:47:07.714Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2972023-10-17T23:47:07.738ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2982023-10-17T23:47:07.762Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2992023-10-17T23:47:07.787ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3002023-10-17T23:47:07.811Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3012023-10-17T23:47:07.835ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3022023-10-17T23:47:07.859Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3032023-10-17T23:47:07.883ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3042023-10-17T23:47:07.907Z0+0 records in
3052023-10-17T23:47:07.936ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3062023-10-17T23:47:07.961Z0+0 records out
3072023-10-17T23:47:07.986ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3082023-10-17T23:47:08.017Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3092023-10-17T23:47:08.043ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3102023-10-17T23:47:08.067Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3112023-10-17T23:47:08.096ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3122023-10-17T23:47:08.122Z+ set +x
3132023-10-17T23:47:08.171ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3142023-10-17T23:47:08.196Z++ zpool list -o name
3152023-10-17T23:47:08.633ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3162023-10-17T23:47:08.662Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3172023-10-17T23:47:08.687ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3182023-10-17T23:47:08.711Z+ [[ -z '' ]]
3192023-10-17T23:47:08.736ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3202023-10-17T23:47:08.760Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3212023-10-17T23:47:08.799ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3222023-10-17T23:47:08.834Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-10-17T23:47:08.871ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3242023-10-17T23:47:08.906Z+ set +x
3252023-10-17T23:47:08.937ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3262023-10-17T23:47:08.961Z+ for ZPOOL in "${ZPOOLS[@]}"
3272023-10-17T23:47:08.985ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3282023-10-17T23:47:09.009Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3292023-10-17T23:47:09.033ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3302023-10-17T23:47:09.089Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3312023-10-17T23:47:09.113ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3322023-10-17T23:47:09.137Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3332023-10-17T23:47:09.161ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3342023-10-17T23:47:09.195Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3352023-10-17T23:47:09.219ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3362023-10-17T23:47:09.243Z0+0 records in
3372023-10-17T23:47:09.267ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3382023-10-17T23:47:09.291Z0+0 records out
3392023-10-17T23:47:09.314ZSimnet net0/sc0_0 exists
3402023-10-17T23:47:09.340Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3412023-10-17T23:47:09.364ZSimnet net1/sc1_0 exists
3422023-10-17T23:47:09.388Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3432023-10-17T23:47:09.411ZVnic sc0_1 exists
3442023-10-17T23:47:09.435Z+ set +x
3452023-10-17T23:47:09.459Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3462023-10-17T23:47:09.484Z++ zpool list -o name
3472023-10-17T23:47:09.511Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3482023-10-17T23:47:09.535Z+ [[ -z '' ]]
3492023-10-17T23:47:09.559Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-10-17T23:47:09.584Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-10-17T23:47:09.608Z+ set +x
3522023-10-17T23:47:09.632Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-10-17T23:47:09.656Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-10-17T23:47:09.691Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-10-17T23:47:09.720Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-10-17T23:47:09.743Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-10-17T23:47:09.793Z0+0 records in
3582023-10-17T23:47:09.817Z0+0 records out
3592023-10-17T23:47:09.841Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3602023-10-17T23:47:09.866Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-10-17T23:47:09.889Z+ set +x
3622023-10-17T23:47:09.913Z++ zpool list -o name
3632023-10-17T23:47:09.939Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-10-17T23:47:09.963Z+ [[ -z '' ]]
3652023-10-17T23:47:09.987Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-10-17T23:47:10.011Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-10-17T23:47:10.035Z+ set +x
3682023-10-17T23:47:10.059Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-10-17T23:47:10.083Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-10-17T23:47:10.112Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-10-17T23:47:10.137Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-10-17T23:47:10.161Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-10-17T23:47:10.185Z0+0 records in
3742023-10-17T23:47:10.220Z0+0 records out
3752023-10-17T23:47:10.244Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3762023-10-17T23:47:10.268Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-10-17T23:47:10.292Z+ set +x
3782023-10-17T23:47:10.316Z++ zpool list -o name
3792023-10-17T23:47:10.340Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-10-17T23:47:10.364Z+ [[ -z '' ]]
3812023-10-17T23:47:10.387Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-10-17T23:47:10.411Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-10-17T23:47:10.435Z+ set +x
3842023-10-17T23:47:10.459Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-10-17T23:47:10.486Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862023-10-17T23:47:10.525Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872023-10-17T23:47:10.552Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882023-10-17T23:47:10.576Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892023-10-17T23:47:10.602Z0+0 records in
3902023-10-17T23:47:10.627Z0+0 records out
3912023-10-17T23:47:10.651Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-10-17T23:47:10.676Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932023-10-17T23:47:10.700Z+ set +x
3942023-10-17T23:47:10.724Z++ zpool list -o name
3952023-10-17T23:47:10.748Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3962023-10-17T23:47:10.772Z+ [[ -z '' ]]
3972023-10-17T23:47:10.795Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3982023-10-17T23:47:10.820Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3992023-10-17T23:47:10.844Z+ set +x
4002023-10-17T23:47:10.868Z+ for ZPOOL in "${ZPOOLS[@]}"
4012023-10-17T23:47:10.892Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4022023-10-17T23:47:10.916Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4032023-10-17T23:47:10.940Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4042023-10-17T23:47:10.964Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4052023-10-17T23:47:10.993Z0+0 records in
4062023-10-17T23:47:11.017Z0+0 records out
4072023-10-17T23:47:11.043Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4082023-10-17T23:47:11.472Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4092023-10-17T23:47:11.496Z+ set +x
4102023-10-17T23:47:11.520Z++ zpool list -o name
4112023-10-17T23:47:11.544Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4122023-10-17T23:47:11.568Z+ [[ -z '' ]]
4132023-10-17T23:47:11.592Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4142023-10-17T23:47:11.616Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4152023-10-17T23:47:11.640Z+ set +x
4162023-10-17T23:47:11.664Z+ for ZPOOL in "${ZPOOLS[@]}"
4172023-10-17T23:47:11.688Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4182023-10-17T23:47:11.712Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192023-10-17T23:47:11.736Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4202023-10-17T23:47:11.760Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4212023-10-17T23:47:11.783Z0+0 records in
4222023-10-17T23:47:11.807Z0+0 records out
4232023-10-17T23:47:11.831Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4242023-10-17T23:47:11.855Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4252023-10-17T23:47:11.881Z+ set +x
4262023-10-17T23:47:11.906Z++ zpool list -o name
4272023-10-17T23:47:11.931Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4282023-10-17T23:47:11.964Z+ [[ -z '' ]]
4292023-10-17T23:47:11.988Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4302023-10-17T23:47:12.014Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4312023-10-17T23:47:12.038Z+ set +x
4322023-10-17T23:47:12.063Z+ for ZPOOL in "${ZPOOLS[@]}"
4332023-10-17T23:47:12.088Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4342023-10-17T23:47:12.112Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4352023-10-17T23:47:12.136Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4362023-10-17T23:47:12.160Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4372023-10-17T23:47:12.183Z0+0 records in
4382023-10-17T23:47:12.207Z0+0 records out
4392023-10-17T23:47:12.232Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4402023-10-17T23:47:12.269Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4412023-10-17T23:47:12.293Z+ set +x
4422023-10-17T23:47:12.320Z++ zpool list -o name
4432023-10-17T23:47:12.348Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4442023-10-17T23:47:12.374Z+ [[ -z '' ]]
4452023-10-17T23:47:12.401Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4462023-10-17T23:47:12.427Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4472023-10-17T23:47:12.460Z+ set +x
4482023-10-17T23:47:12.489Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4492023-10-17T23:47:12.514Z+ readarray -t ZPOOLS
4502023-10-17T23:47:12.538Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4512023-10-17T23:47:12.562Z++ sed 's/[ ",]//g'
4522023-10-17T23:47:12.587Z+ for ZPOOL in "${ZPOOLS[@]}"
4532023-10-17T23:47:12.620Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4542023-10-17T23:47:12.644Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4552023-10-17T23:47:12.668Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4562023-10-17T23:47:12.693Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4572023-10-17T23:47:12.717Z0+0 records in
4582023-10-17T23:47:12.741Z0+0 records out
4592023-10-17T23:47:12.765Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4602023-10-17T23:47:12.789Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4612023-10-17T23:47:12.813Z+ set +x
4622023-10-17T23:47:12.837Z++ zpool list -o name
4632023-10-17T23:47:12.861Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4642023-10-17T23:47:12.893Z+ [[ -z '' ]]
4652023-10-17T23:47:12.926Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-10-17T23:47:12.952Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4672023-10-17T23:47:12.977Z+ set +x
4682023-10-17T23:47:13.002Z+ for ZPOOL in "${ZPOOLS[@]}"
4692023-10-17T23:47:13.026Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4702023-10-17T23:47:13.052Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4712023-10-17T23:47:13.078Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4722023-10-17T23:47:13.104Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4732023-10-17T23:47:13.130Z0+0 records in
4742023-10-17T23:47:13.156Z0+0 records out
4752023-10-17T23:47:13.182Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4762023-10-17T23:47:13.209Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4772023-10-17T23:47:13.236Z+ set +x
4782023-10-17T23:47:13.264Z++ zpool list -o name
4792023-10-17T23:47:13.307Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4802023-10-17T23:47:13.335Z+ [[ -z '' ]]
4812023-10-17T23:47:13.365Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4822023-10-17T23:47:13.391Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4832023-10-17T23:47:13.416Z+ set +x
4842023-10-17T23:47:13.442Z+ ensure_simulated_links igb0
4852023-10-17T23:47:13.476Z+ local PHYSICAL_LINK=igb0
4862023-10-17T23:47:13.510Z+ INDICES=("0" "1")
4872023-10-17T23:47:13.537Z+ for I in "${INDICES[@]}"
4882023-10-17T23:47:13.562Z++ get_simnet_name_if_exists net0
4892023-10-17T23:47:13.587Z++ dladm show-simnet -p -o LINK net0
4902023-10-17T23:47:13.610Z++ echo ''
4912023-10-17T23:47:13.634Z+ [[ -z '' ]]
4922023-10-17T23:47:13.658Z+ dladm create-simnet -t net0
4932023-10-17T23:47:13.682Z+ dladm create-simnet -t sc0_0
4942023-10-17T23:47:13.706Z+ dladm modify-simnet -t -p net0 sc0_0
4952023-10-17T23:47:13.731Z+ dladm set-linkprop -p mtu=9000 sc0_0
4962023-10-17T23:47:13.760Z+ success 'Simnet net0/sc0_0 exists'
4972023-10-17T23:47:14.192Z+ set +x
4982023-10-17T23:47:14.223Z+ for I in "${INDICES[@]}"
4992023-10-17T23:47:14.251Z++ get_simnet_name_if_exists net1
5002023-10-17T23:47:14.275Z++ dladm show-simnet -p -o LINK net1
5012023-10-17T23:47:14.299Z++ echo ''
5022023-10-17T23:47:14.332Zdownloading softnpu ...done
5032023-10-17T23:47:14.355Z+ [[ -z '' ]]
5042023-10-17T23:47:14.379Z+ dladm create-simnet -t net1
5052023-10-17T23:47:14.403Z+ dladm create-simnet -t sc1_0
5062023-10-17T23:47:14.427Z+ dladm modify-simnet -t -p net1 sc1_0
5072023-10-17T23:47:14.450Z+ dladm set-linkprop -p mtu=9000 sc1_0
5082023-10-17T23:47:14.474Z+ success 'Simnet net1/sc1_0 exists'
5092023-10-17T23:47:14.498Z+ set +x
5102023-10-17T23:47:14.522Z++ get_vnic_name_if_exists sc0_1
5112023-10-17T23:47:14.546Z++ dladm show-vnic -p -o LINK sc0_1
5122023-10-17T23:47:14.569Z++ echo ''
5132023-10-17T23:47:14.593Z+ [[ -z '' ]]
5142023-10-17T23:47:14.619Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5152023-10-17T23:47:14.652Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/4ba9c18928edece80f83153b9d78b76cf093da19/libsidecar_lite.so.sha256.txt
5162023-10-17T23:47:14.681Z+ success 'Vnic sc0_1 exists'
5172023-10-17T23:47:14.705Z+ set +x
5182023-10-17T23:47:14.729Z+ warn_if_no_proxy_arp
5192023-10-17T23:47:14.778Z+ [[ -v PXA_START ]]
5202023-10-17T23:47:14.801Z+ [[ -v PXA_END ]]
5212023-10-17T23:47:14.825Z+ ensure_softnpu_zone
5222023-10-17T23:47:14.849Z+ zoneadm list
5232023-10-17T23:47:14.873Z+ grep -q sidecar_softnpu
5242023-10-17T23:47:14.896Z+ [[ -f out/npuzone/npuzone ]]
5252023-10-17T23:47:14.921Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0
5262023-10-17T23:47:17.983Zdownloading asic_program.so ...done
5272023-10-17T23:47:18.032Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/4ba9c18928edece80f83153b9d78b76cf093da19/scadm.sha256.txt
5282023-10-17T23:47:24.754Zdownloading scadm ...done
5292023-10-17T23:47:24.783Zcreating port tfportrear0_0 for sc0_0
5302023-10-17T23:47:24.807Zcreating port tfportqsfp0_0 for sc0_1
5312023-10-17T23:47:24.830Zsoftnpu zone setup
5322023-10-17T23:47:24.854Zconfigure zone
5332023-10-17T23:47:24.878Zinstall zone
5342023-10-17T23:47:25.819Zboot zone
5352023-10-17T23:47:28.926Z[sidecar_softnpu] svcs milestone/network
5362023-10-17T23:47:28.954ZSTATE STIME FMRI
5372023-10-17T23:47:28.978Zonline 23:47:26 svc:/milestone/network:default
5382023-10-17T23:47:29.002Z[sidecar_softnpu] /softnpu/npu start
5392023-10-17T23:47:29.026Z
5402023-10-17T23:47:29.051Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5412023-10-17T23:47:29.077Z+ GATEWAY_IP=10.151.1.100
5422023-10-17T23:47:29.103Z+ echo 'Using 10.151.1.100 as gateway ip'
5432023-10-17T23:47:29.132ZUsing 10.151.1.100 as gateway ip
5442023-10-17T23:47:29.161Z+ [[ ! -v GATEWAY_MAC ]]
5452023-10-17T23:47:29.190Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5462023-10-17T23:47:29.216Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5472023-10-17T23:47:29.245Z+++ echo 18:c0:4d:0d:9f:b2
5482023-10-17T23:47:29.269Z+ COUNT=1
5492023-10-17T23:47:29.293Z+ [[ 1 -ne 1 ]]
5502023-10-17T23:47:29.318Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5512023-10-17T23:47:29.343ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5522023-10-17T23:47:29.367Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5532023-10-17T23:47:29.391Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5542023-10-17T23:47:29.415Z+ PXA_MAC=a8:e1:de:01:70:1d
5552023-10-17T23:47:29.443Z+ [[ -v PXA_START ]]
5562023-10-17T23:47:29.471Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5572023-10-17T23:47:29.499Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5582023-10-17T23:47:29.528Z+ z_scadm dump-state
5592023-10-17T23:47:29.552Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5602023-10-17T23:47:29.581Zlocal v6:
5612023-10-17T23:47:29.615Zlocal v4:
5622023-10-17T23:47:29.648Zrouter v6:
5632023-10-17T23:47:29.674Zrouter v4:
5642023-10-17T23:47:29.699Zresolver v4:
5652023-10-17T23:47:29.723Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5662023-10-17T23:47:31.456Zresolver v6:
5672023-10-17T23:47:31.534Znat v4:
5682023-10-17T23:47:31.572Znat_v6:
5692023-10-17T23:47:31.604Zport_mac:
5702023-10-17T23:47:31.631Zproxy arp:
5712023-10-17T23:47:31.659Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5722023-10-17T23:47:31.683Z+ success 'softnpu zone exists'
5732023-10-17T23:47:31.707Z+ set +x
5742023-10-17T23:47:31.734Zsoftnpu zone exists
5752023-10-17T23:47:31.761Z
5762023-10-17T23:47:31.790Zreal 25.240252363
5772023-10-17T23:47:31.890Zuser 0.719720419
5782023-10-17T23:47:31.915Zsys 1.619195115
5792023-10-17T23:47:31.939Ztrap 0.000121510
5802023-10-17T23:47:31.976Ztflt 0.006832229
5812023-10-17T23:47:32.002Zdflt 0.007988337
5822023-10-17T23:47:32.026Zkflt 0.000000000
5832023-10-17T23:47:32.050Zlock 19:38.801156092
5842023-10-17T23:47:32.074Zslp 1:14.874398272
5852023-10-17T23:47:32.098Zlat 0.075628110
5862023-10-17T23:47:32.122Zstop 0.389441859
5872023-10-17T23:47:32.146Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5882023-10-17T23:47:32.170Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5892023-10-17T23:47:32.194Z+ SILO_NAME=recovery
5902023-10-17T23:47:32.218Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5912023-10-17T23:47:32.242Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5922023-10-17T23:47:32.266Z+ sed -i~ '
5932023-10-17T23:47:32.290Z /^external_dns_ips/c\
5942023-10-17T23:47:32.314Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5952023-10-17T23:47:32.338Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5962023-10-17T23:47:32.362Z /^first/c\
5972023-10-17T23:47:32.386Zfirst = "10.151.1.164"
5982023-10-17T23:47:32.410Z /^last/c\
5992023-10-17T23:47:32.434Zlast = "10.151.1.173"
6002023-10-17T23:47:32.458Z }
6012023-10-17T23:47:32.483Z /^\[rack_network_config/,/^$/ {
6022023-10-17T23:47:32.552Z /^infra_ip_first/c\
6032023-10-17T23:47:32.576Zinfra_ip_first = "10.151.1.174"
6042023-10-17T23:47:32.600Z /^infra_ip_last/c\
6052023-10-17T23:47:32.624Zinfra_ip_last = "10.151.1.174"
6062023-10-17T23:47:32.648Z }
6072023-10-17T23:47:32.672Z /^\[\[rack_network_config.uplinks/,/^$/ {
6082023-10-17T23:47:32.696Z /^gateway_ip/c\
6092023-10-17T23:47:32.720Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6102023-10-17T23:47:32.744Zgateway_ip = "10.151.1.100"
6112023-10-17T23:47:32.767Z+++ pkg/config-rss.toml Tue Oct 17 23:47:28 2023
6122023-10-17T23:47:32.791Z /^uplink_cidr/c\
6132023-10-17T23:47:32.815Z@@ -41,7 +41,7 @@
6142023-10-17T23:47:32.839Zuplink_cidr = "10.151.1.174/32"
6152023-10-17T23:47:32.863Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6162023-10-17T23:47:32.889Z }
6172023-10-17T23:47:32.914Z # must be contained in one of the "internal services" IP Pool ranges listed
6182023-10-17T23:47:32.944Z' pkg/config-rss.toml
6192023-10-17T23:47:33.521Z # below.
6202023-10-17T23:47:33.546Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6212023-10-17T23:47:33.621Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6222023-10-17T23:47:33.645Z+ true
6232023-10-17T23:47:33.670Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6242023-10-17T23:47:33.695Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6252023-10-17T23:47:33.719Z
6262023-10-17T23:47:33.747Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6272023-10-17T23:47:33.772Z # Initial TLS certificates for the external API
6282023-10-17T23:47:33.805Z+ TLS_NAME=recovery.sys.oxide.test
6292023-10-17T23:47:33.836Z #
6302023-10-17T23:47:34.898Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6312023-10-17T23:47:34.930Z@@ -77,8 +77,8 @@
6322023-10-17T23:47:34.954Z..........+.+..+....+......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.........+..+....+......+.........+...........+.+........+.......+..+......................+...+..+.........+......+...............+...+...+....+........+...+.............+.....+............+....+.....+...................+........................+.....+.+..............+.+..+...........................+......+.+...............+......................................+.+..+..........+.....+....+...+......+............+...+............+........+......+.+............+.....+...+....+...+.....+.......+...+...+..+............+................+...+.........+..+.+.......................+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6332023-10-17T23:47:35.066Z #
6342023-10-17T23:47:35.096Z.+....+.....+.............+.....+.+......+...+.....+.+.....+.+...+...+...+..+..................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+........+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+.+...............+........+..........................................+.+............+........+....+...+...........+..........+...........+...+.+.....+.......+............+...+........+......+...+...+....+........+.............+........+.+..............+.+.....................+...........................+...+..+..........+..+...+.+..+...+.+.........+.........+..+.........+.+.....+....+.....+....+...+.....+.........................+..............+..........+.....................+...............+..+..........+.....+...+...............+.......+..+.+.........+.........+.....+..........+......+..............+.......+..+......+....+..+.+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6352023-10-17T23:47:35.122Z # For more on this and what to put here, see docs/how-to-run.adoc.
6362023-10-17T23:47:35.150Z-----
6372023-10-17T23:47:35.185Z [[internal_services_ip_pool_ranges]]
6382023-10-17T23:47:35.211Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6392023-10-17T23:47:35.241Z-first = "192.168.1.20"
6402023-10-17T23:47:35.272Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6412023-10-17T23:47:35.297Z-last = "192.168.1.29"
6422023-10-17T23:47:35.322Z+ rmdir pkg
6432023-10-17T23:47:35.346Z+first = "10.151.1.164"
6442023-10-17T23:47:35.370Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6452023-10-17T23:47:35.394Z+last = "10.151.1.173"
6462023-10-17T23:47:35.418Z+ pfexec mkdir -p /usr/oxide
6472023-10-17T23:47:35.442Z
6482023-10-17T23:47:35.466Z+ pfexec rm -f /usr/oxide/tcpproxy
6492023-10-17T23:47:35.490Z # TODO - this configuration is subject to change going forward. Ultimately these
6502023-10-17T23:47:35.514Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6512023-10-17T23:47:35.538Z # parameters should be provided to the control plane via wicket, but we need to
6522023-10-17T23:47:35.562Z+ pfexec chmod +x /usr/oxide/tcpproxy
6532023-10-17T23:47:35.586Z@@ -91,13 +91,13 @@
6542023-10-17T23:47:35.610Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6552023-10-17T23:47:35.634Z # A range of IP addresses used by Boundary Services on the external network. In
6562023-10-17T23:47:35.658Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6572023-10-17T23:47:35.715Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6582023-10-17T23:47:35.757Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6592023-10-17T23:47:35.781Z # With softnpu, only one address is used.
6602023-10-17T23:47:35.828Z+ OMICRON_NO_UNINSTALL=1
6612023-10-17T23:47:35.852Z-infra_ip_first = "192.168.1.30"
6622023-10-17T23:47:35.876Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6632023-10-17T23:47:35.909Z-infra_ip_last = "192.168.1.30"
6642023-10-17T23:47:35.934ZOct 17 23:47:29.678 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6652023-10-17T23:47:35.958Z+infra_ip_first = "10.151.1.174"
6662023-10-17T23:47:35.982ZOct 17 23:47:29.680 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6672023-10-17T23:47:36.006Z+infra_ip_last = "10.151.1.174"
6682023-10-17T23:47:36.029ZOct 17 23:47:29.680 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6692023-10-17T23:47:36.053Z
6702023-10-17T23:47:36.077ZOct 17 23:47:29.680 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6712023-10-17T23:47:36.101Z # You can configure multiple uplinks by repeating the following stanza
6722023-10-17T23:47:36.125ZOct 17 23:47:29.680 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6732023-10-17T23:47:36.149Z [[rack_network_config.uplinks]]
6742023-10-17T23:47:36.173ZOct 17 23:47:29.680 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6752023-10-17T23:47:36.197Z # The gateway IP for the rack's external network
6762023-10-17T23:47:36.221ZOct 17 23:47:29.680 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6772023-10-17T23:47:36.246Z-gateway_ip = "192.168.1.199"
6782023-10-17T23:47:36.270ZOct 17 23:47:29.680 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6792023-10-17T23:47:36.294Z+gateway_ip = "10.151.1.100"
6802023-10-17T23:47:36.321ZOct 17 23:47:29.680 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/maghemite.tar
6812023-10-17T23:47:36.345Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6822023-10-17T23:47:36.369ZOct 17 23:47:29.680 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6832023-10-17T23:47:36.393Z uplink_port = "qsfp0"
6842023-10-17T23:47:36.417ZOct 17 23:47:29.680 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6852023-10-17T23:47:36.441Z uplink_port_speed = "40G"
6862023-10-17T23:47:36.467ZOct 17 23:47:29.680 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6872023-10-17T23:47:36.498Z@@ -104,7 +104,7 @@
6882023-10-17T23:47:36.522ZOct 17 23:47:29.680 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6892023-10-17T23:47:36.547Z uplink_port_fec="none"
6902023-10-17T23:47:36.571ZOct 17 23:47:29.680 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6912023-10-17T23:47:36.595Z # For softnpu, an address within the "infra" block above that will be used for
6922023-10-17T23:47:36.619ZOct 17 23:47:29.681 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6932023-10-17T23:47:36.643Z # the softnpu uplink port. You can just pick the first address in that pool.
6942023-10-17T23:47:36.667ZOct 17 23:47:29.681 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6952023-10-17T23:47:36.691Z-uplink_cidr = "192.168.1.30/32"
6962023-10-17T23:47:36.716ZOct 17 23:47:29.681 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6972023-10-17T23:47:36.740Z+uplink_cidr = "10.151.1.174/32"
6982023-10-17T23:47:36.801ZOct 17 23:47:29.911 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
6992023-10-17T23:47:36.830Z # Switch to use for the uplink. For single-rack deployments this can be
7002023-10-17T23:47:36.854ZOct 17 23:47:29.953 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7012023-10-17T23:47:36.878Z # "switch0" (upper slot) or "switch1" (lower slot). For single-node softnpu
7022023-10-17T23:47:36.906ZOct 17 23:47:30.013 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7032023-10-17T23:47:36.930Z # and dendrite stub environments, use "switch0"
7042023-10-17T23:47:36.953Z
7052023-10-17T23:47:36.978Za pkg/config-rss.toml 7K
7062023-10-17T23:47:37.002Zreal 0.475634362
7072023-10-17T23:47:37.029Za pkg/initial-tls-cert.pem 2K
7082023-10-17T23:47:37.058Zuser 0.095543824
7092023-10-17T23:47:37.082Za pkg/initial-tls-key.pem 4K
7102023-10-17T23:47:37.107Zsys 1.837032556
7112023-10-17T23:47:37.132Ztrap 0.000000660
7122023-10-17T23:47:37.156Ztflt 0.000062159
7132023-10-17T23:47:37.180Zdflt 0.001023226
7142023-10-17T23:47:37.205Zkflt 0.000013640
7152023-10-17T23:47:37.229Zlock 42.828163134
7162023-10-17T23:47:37.254Zslp 0.693771644
7172023-10-17T23:47:37.277Zlat 0.003106723
7182023-10-17T23:47:37.302Zstop 0.215842993
7192023-10-17T23:47:37.328Z+ retry=0
7202023-10-17T23:47:37.355Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7212023-10-17T23:47:37.381Z+ [[ 0 -gt 30 ]]
7222023-10-17T23:47:37.405Z+ sleep 1
7232023-10-17T23:47:37.431Z+ retry=1
7242023-10-17T23:47:37.456Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7252023-10-17T23:47:37.480Z+ [[ 1 -gt 30 ]]
7262023-10-17T23:47:37.504Z+ sleep 1
7272023-10-17T23:47:37.529Z+ retry=2
7282023-10-17T23:47:37.555Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7292023-10-17T23:47:52.544Z+ [[ 2 -gt 30 ]]
7302023-10-17T23:47:52.568Z+ sleep 1
7312023-10-17T23:47:53.545Z+ retry=3
7322023-10-17T23:47:53.569Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7332023-10-17T23:47:53.593Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7342023-10-17T23:47:53.633Zlocal v6:
7352023-10-17T23:47:53.657Zfe80::28fc:81ff:fe88:95b7
7362023-10-17T23:47:53.681Zlocal v4:
7372023-10-17T23:47:53.705Zrouter v6:
7382023-10-17T23:47:53.729Zrouter v4:
7392023-10-17T23:47:53.753Zresolver v4:
7402023-10-17T23:47:53.778Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7412023-10-17T23:47:53.802Zresolver v6:
7422023-10-17T23:47:53.825Zfe80::28fc:81ff:fe88:95b7 -> 2a:fc:81:88:95:b7
7432023-10-17T23:47:53.853Zfe80::d0ca:88ff:fef2:8845 -> d2:ca:88:f2:88:45
7442023-10-17T23:47:53.877Znat v4:
7452023-10-17T23:47:53.906Znat_v6:
7462023-10-17T23:47:53.932Zport_mac:
7472023-10-17T23:47:53.956Z1000: a8:40:25:6b:6a:95
7482023-10-17T23:47:53.980Z1: a8:40:25:6b:6a:96
7492023-10-17T23:47:54.004Zproxy arp:
7502023-10-17T23:47:54.028Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7512023-10-17T23:47:54.052Z+ retry=0
7522023-10-17T23:47:54.075Z+++ zoneadm list -n
7532023-10-17T23:47:54.099Z+++ grep oxz_ntp
7542023-10-17T23:47:54.123Z++ pfexec svcs -z -Hostate oxide/ntp
7552023-10-17T23:47:54.147Zsvcs: zone '-Hostate': entity not found
7562023-10-17T23:47:54.171Z++ true
7572023-10-17T23:47:54.195Z+ [[ '' != online ]]
7582023-10-17T23:47:54.219Z+ [[ 0 -gt 60 ]]
7592023-10-17T23:47:54.243Z+ sleep 1
7602023-10-17T23:47:54.607Z+ retry=1
7612023-10-17T23:47:54.655Z+++ zoneadm list -n
7622023-10-17T23:47:54.680Z+++ grep oxz_ntp
7632023-10-17T23:47:54.711Z++ pfexec svcs -z -Hostate oxide/ntp
7642023-10-17T23:47:54.736Zsvcs: zone '-Hostate': entity not found
7652023-10-17T23:47:54.761Z++ true
7662023-10-17T23:47:54.815Z+ [[ '' != online ]]
7672023-10-17T23:47:54.840Z+ [[ 1 -gt 60 ]]
7682023-10-17T23:47:54.865Z+ sleep 1
7692023-10-17T23:47:55.628Z+ retry=2
7702023-10-17T23:47:55.652Z+++ zoneadm list -n
7712023-10-17T23:47:55.684Z+++ grep oxz_ntp
7722023-10-17T23:47:55.708Z++ pfexec svcs -z -Hostate oxide/ntp
7732023-10-17T23:47:55.732Zsvcs: zone '-Hostate': entity not found
7742023-10-17T23:47:55.756Z++ true
7752023-10-17T23:47:55.789Z+ [[ '' != online ]]
7762023-10-17T23:47:55.813Z+ [[ 2 -gt 60 ]]
7772023-10-17T23:47:55.838Z+ sleep 1
7782023-10-17T23:47:56.649Z+ retry=3
7792023-10-17T23:47:56.673Z+++ zoneadm list -n
7802023-10-17T23:47:56.708Z+++ grep oxz_ntp
7812023-10-17T23:47:56.732Z++ pfexec svcs -z -Hostate oxide/ntp
7822023-10-17T23:47:56.768Zsvcs: zone '-Hostate': entity not found
7832023-10-17T23:47:56.794Z++ true
7842023-10-17T23:47:56.818Z+ [[ '' != online ]]
7852023-10-17T23:47:56.842Z+ [[ 3 -gt 60 ]]
7862023-10-17T23:47:56.866Z+ sleep 1
7872023-10-17T23:47:57.704Z+ retry=4
7882023-10-17T23:47:57.734Z+++ zoneadm list -n
7892023-10-17T23:47:57.758Z+++ grep oxz_ntp
7902023-10-17T23:47:57.782Z++ pfexec svcs -z -Hostate oxide/ntp
7912023-10-17T23:47:57.806Zsvcs: zone '-Hostate': entity not found
7922023-10-17T23:47:57.830Z++ true
7932023-10-17T23:47:57.854Z+ [[ '' != online ]]
7942023-10-17T23:47:57.878Z+ [[ 4 -gt 60 ]]
7952023-10-17T23:47:57.903Z+ sleep 1
7962023-10-17T23:47:58.699Z+ retry=5
7972023-10-17T23:47:58.724Z+++ zoneadm list -n
7982023-10-17T23:47:58.758Z+++ grep oxz_ntp
7992023-10-17T23:47:58.782Z++ pfexec svcs -z -Hostate oxide/ntp
8002023-10-17T23:47:58.806Zsvcs: zone '-Hostate': entity not found
8012023-10-17T23:47:58.831Z++ true
8022023-10-17T23:47:58.854Z+ [[ '' != online ]]
8032023-10-17T23:47:58.878Z+ [[ 5 -gt 60 ]]
8042023-10-17T23:47:58.902Z+ sleep 1
8052023-10-17T23:47:59.720Z+ retry=6
8062023-10-17T23:47:59.744Z+++ zoneadm list -n
8072023-10-17T23:47:59.782Z+++ grep oxz_ntp
8082023-10-17T23:47:59.832Z++ pfexec svcs -z -Hostate oxide/ntp
8092023-10-17T23:47:59.856Zsvcs: zone '-Hostate': entity not found
8102023-10-17T23:47:59.880Z++ true
8112023-10-17T23:47:59.904Z+ [[ '' != online ]]
8122023-10-17T23:47:59.939Z+ [[ 6 -gt 60 ]]
8132023-10-17T23:47:59.964Z+ sleep 1
8142023-10-17T23:48:00.744Z+ retry=7
8152023-10-17T23:48:00.769Z+++ zoneadm list -n
8162023-10-17T23:48:00.807Z+++ grep oxz_ntp
8172023-10-17T23:48:00.831Z++ pfexec svcs -z -Hostate oxide/ntp
8182023-10-17T23:48:00.855Zsvcs: zone '-Hostate': entity not found
8192023-10-17T23:48:00.880Z++ true
8202023-10-17T23:48:00.903Z+ [[ '' != online ]]
8212023-10-17T23:48:00.927Z+ [[ 7 -gt 60 ]]
8222023-10-17T23:48:00.951Z+ sleep 1
8232023-10-17T23:48:01.769Z+ retry=8
8242023-10-17T23:48:01.831Z+++ zoneadm list -n
8252023-10-17T23:48:01.859Z+++ grep oxz_ntp
8262023-10-17T23:48:01.884Z++ pfexec svcs -z -Hostate oxide/ntp
8272023-10-17T23:48:01.908Zsvcs: zone '-Hostate': entity not found
8282023-10-17T23:48:01.934Z++ true
8292023-10-17T23:48:01.971Z+ [[ '' != online ]]
8302023-10-17T23:48:01.998Z+ [[ 8 -gt 60 ]]
8312023-10-17T23:48:02.027Z+ sleep 1
8322023-10-17T23:48:02.795Z+ retry=9
8332023-10-17T23:48:02.820Z+++ zoneadm list -n
8342023-10-17T23:48:02.856Z+++ grep oxz_ntp
8352023-10-17T23:48:02.880Z++ pfexec svcs -z -Hostate oxide/ntp
8362023-10-17T23:48:02.905Zsvcs: zone '-Hostate': entity not found
8372023-10-17T23:48:02.930Z++ true
8382023-10-17T23:48:02.954Z+ [[ '' != online ]]
8392023-10-17T23:48:02.979Z+ [[ 9 -gt 60 ]]
8402023-10-17T23:48:03.004Z+ sleep 1
8412023-10-17T23:48:03.812Z+ retry=10
8422023-10-17T23:48:03.836Z+++ zoneadm list -n
8432023-10-17T23:48:03.880Z+++ grep oxz_ntp
8442023-10-17T23:48:03.906Z++ pfexec svcs -z -Hostate oxide/ntp
8452023-10-17T23:48:03.931Zsvcs: zone '-Hostate': entity not found
8462023-10-17T23:48:03.955Z++ true
8472023-10-17T23:48:03.986Z+ [[ '' != online ]]
8482023-10-17T23:48:04.011Z+ [[ 10 -gt 60 ]]
8492023-10-17T23:48:04.036Z+ sleep 1
8502023-10-17T23:48:04.833Z+ retry=11
8512023-10-17T23:48:04.857Z+++ zoneadm list -n
8522023-10-17T23:48:04.908Z+++ grep oxz_ntp
8532023-10-17T23:48:04.933Z++ pfexec svcs -z -Hostate oxide/ntp
8542023-10-17T23:48:04.971Zsvcs: zone '-Hostate': entity not found
8552023-10-17T23:48:04.998Z++ true
8562023-10-17T23:48:05.023Z+ [[ '' != online ]]
8572023-10-17T23:48:05.047Z+ [[ 11 -gt 60 ]]
8582023-10-17T23:48:05.071Z+ sleep 1
8592023-10-17T23:48:05.855Z+ retry=12
8602023-10-17T23:48:05.879Z+++ zoneadm list -n
8612023-10-17T23:48:05.903Z+++ grep oxz_ntp
8622023-10-17T23:48:05.928Z++ pfexec svcs -z -Hostate oxide/ntp
8632023-10-17T23:48:05.952Zsvcs: zone '-Hostate': entity not found
8642023-10-17T23:48:05.976Z++ true
8652023-10-17T23:48:06.006Z+ [[ '' != online ]]
8662023-10-17T23:48:06.032Z+ [[ 12 -gt 60 ]]
8672023-10-17T23:48:06.056Z+ sleep 1
8682023-10-17T23:48:06.877Z+ retry=13
8692023-10-17T23:48:06.901Z+++ zoneadm list -n
8702023-10-17T23:48:06.926Z+++ grep oxz_ntp
8712023-10-17T23:48:06.954Z++ pfexec svcs -z oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed -Hostate oxide/ntp
8722023-10-17T23:48:06.978Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8732023-10-17T23:48:07.004Z++ true
8742023-10-17T23:48:07.027Z+ [[ '' != online ]]
8752023-10-17T23:48:07.059Z+ [[ 13 -gt 60 ]]
8762023-10-17T23:48:07.093Z+ sleep 1
8772023-10-17T23:48:07.908Z+ retry=14
8782023-10-17T23:48:07.935Z+++ zoneadm list -n
8792023-10-17T23:48:07.981Z+++ grep oxz_ntp
8802023-10-17T23:48:08.007Z++ pfexec svcs -z oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed -Hostate oxide/ntp
8812023-10-17T23:48:08.031Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8822023-10-17T23:48:08.057Z++ true
8832023-10-17T23:48:08.081Z+ [[ '' != online ]]
8842023-10-17T23:48:08.105Z+ [[ 14 -gt 60 ]]
8852023-10-17T23:48:08.131Z+ sleep 1
8862023-10-17T23:48:08.942Z+ retry=15
8872023-10-17T23:48:08.969Z+++ zoneadm list -n
8882023-10-17T23:48:09.010Z+++ grep oxz_ntp
8892023-10-17T23:48:09.035Z++ pfexec svcs -z oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed -Hostate oxide/ntp
8902023-10-17T23:48:09.059Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8912023-10-17T23:48:09.083Z++ true
8922023-10-17T23:48:09.107Z+ [[ '' != online ]]
8932023-10-17T23:48:09.133Z+ [[ 15 -gt 60 ]]
8942023-10-17T23:48:09.157Z+ sleep 1
8952023-10-17T23:48:09.970Z+ retry=16
8962023-10-17T23:48:09.994Z+++ zoneadm list -n
8972023-10-17T23:48:10.035Z+++ grep oxz_ntp
8982023-10-17T23:48:10.064Z++ pfexec svcs -z oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed -Hostate oxide/ntp
8992023-10-17T23:48:10.089Z+ [[ disabled != online ]]
9002023-10-17T23:48:10.113Z+ [[ 16 -gt 60 ]]
9012023-10-17T23:48:10.150Z+ sleep 1
9022023-10-17T23:48:10.992Z+ retry=17
9032023-10-17T23:48:11.016Z+++ zoneadm list -n
9042023-10-17T23:48:11.069Z+++ grep oxz_ntp
9052023-10-17T23:48:11.093Z++ pfexec svcs -z oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed -Hostate oxide/ntp
9062023-10-17T23:48:11.119Z+ [[ disabled != online ]]
9072023-10-17T23:48:11.145Z+ [[ 17 -gt 60 ]]
9082023-10-17T23:48:11.169Z+ sleep 1
9092023-10-17T23:48:12.014Z+ retry=18
9102023-10-17T23:48:12.039Z+++ zoneadm list -n
9112023-10-17T23:48:12.064Z+++ grep oxz_ntp
9122023-10-17T23:48:12.091Z++ pfexec svcs -z oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed -Hostate oxide/ntp
9132023-10-17T23:48:12.116Z+ [[ disabled != online ]]
9142023-10-17T23:48:12.146Z+ [[ 18 -gt 60 ]]
9152023-10-17T23:48:12.175Z+ sleep 1
9162023-10-17T23:48:13.037Z+ retry=19
9172023-10-17T23:48:13.065Z+++ zoneadm list -n
9182023-10-17T23:48:13.543Z+++ grep oxz_ntp
9192023-10-17T23:48:13.572Z++ pfexec svcs -z oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed -Hostate oxide/ntp
9202023-10-17T23:48:13.600Z+ [[ disabled != online ]]
9212023-10-17T23:48:13.625Z+ [[ 19 -gt 60 ]]
9222023-10-17T23:48:13.648Z+ sleep 1
9232023-10-17T23:48:14.059Z+ retry=20
9242023-10-17T23:48:14.084Z+++ zoneadm list -n
9252023-10-17T23:48:14.109Z+++ grep oxz_ntp
9262023-10-17T23:48:14.146Z++ pfexec svcs -z oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed -Hostate oxide/ntp
9272023-10-17T23:48:14.174Z+ [[ disabled != online ]]
9282023-10-17T23:48:14.198Z+ [[ 20 -gt 60 ]]
9292023-10-17T23:48:14.222Z+ sleep 1
9302023-10-17T23:48:15.081Z+ retry=21
9312023-10-17T23:48:15.105Z+++ zoneadm list -n
9322023-10-17T23:48:15.130Z+++ grep oxz_ntp
9332023-10-17T23:48:15.170Z++ pfexec svcs -z oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed -Hostate oxide/ntp
9342023-10-17T23:48:15.194Z+ [[ disabled != online ]]
9352023-10-17T23:48:15.218Z+ [[ 21 -gt 60 ]]
9362023-10-17T23:48:15.242Z+ sleep 1
9372023-10-17T23:48:16.104Z+ retry=22
9382023-10-17T23:48:16.128Z+++ zoneadm list -n
9392023-10-17T23:48:16.152Z+++ grep oxz_ntp
9402023-10-17T23:48:16.194Z++ pfexec svcs -z oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed -Hostate oxide/ntp
9412023-10-17T23:48:16.219Z+ [[ disabled != online ]]
9422023-10-17T23:48:16.242Z+ [[ 22 -gt 60 ]]
9432023-10-17T23:48:16.266Z+ sleep 1
9442023-10-17T23:48:17.125Z+ retry=23
9452023-10-17T23:48:17.149Z+++ zoneadm list -n
9462023-10-17T23:48:17.173Z+++ grep oxz_ntp
9472023-10-17T23:48:17.221Z++ pfexec svcs -z oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed -Hostate oxide/ntp
9482023-10-17T23:48:17.246Z+ [[ disabled != online ]]
9492023-10-17T23:48:17.270Z+ [[ 23 -gt 60 ]]
9502023-10-17T23:48:17.294Z+ sleep 1
9512023-10-17T23:48:18.147Z+ retry=24
9522023-10-17T23:48:18.171Z+++ zoneadm list -n
9532023-10-17T23:48:18.195Z+++ grep oxz_ntp
9542023-10-17T23:48:18.245Z++ pfexec svcs -z oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed -Hostate oxide/ntp
9552023-10-17T23:48:18.269Z+ [[ disabled != online ]]
9562023-10-17T23:48:18.293Z+ [[ 24 -gt 60 ]]
9572023-10-17T23:48:18.317Z+ sleep 1
 
9582023-10-17T23:48:18.554Zbuskin console login: Oct 17 23:48:16 bukin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9592023-10-17T23:48:19.169Z+ retry=25
9602023-10-17T23:48:19.193Z+++ zoneadm list -n
9612023-10-17T23:48:19.217Z+++ grep oxz_ntp
9622023-10-17T23:48:19.241Z++ pfexec svcs -z oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed -Hostate oxide/ntp
9632023-10-17T23:48:19.279Z+ [[ online != online ]]
9642023-10-17T23:48:19.303Z+ echo 'Waited for chrony: 25s'
9652023-10-17T23:48:19.327ZWaited for chrony: 25s
9662023-10-17T23:48:19.350Z+ export RUST_BACKTRACE=1
9672023-10-17T23:48:19.374Z+ RUST_BACKTRACE=1
9682023-10-17T23:48:19.397Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9692023-10-17T23:48:19.421Z+ ./tests/bootstrap
9702023-10-17T23:48:19.447ZUsing external DNS server at 10.151.1.164:53
9712023-10-17T23:48:19.479Z2023-10-17 23:48:18.315156553 UTC: attempting to log into API
9722023-10-17T23:48:34.259Z2023-10-17 23:48:34.668773633 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9732023-10-17T23:48:35.260Z2023-10-17 23:48:35.669672363 UTC: attempting to log into API
9742023-10-17T23:48:50.293Z2023-10-17 23:48:50.702843279 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9752023-10-17T23:48:51.294Z2023-10-17 23:48:51.703996284 UTC: attempting to log into API
9762023-10-17T23:49:06.328Z2023-10-17 23:49:06.737434737 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9772023-10-17T23:49:07.327Z2023-10-17 23:49:07.737550770 UTC: attempting to log into API
9782023-10-17T23:49:22.361Z2023-10-17 23:49:22.771062034 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9792023-10-17T23:49:23.362Z2023-10-17 23:49:23.772184949 UTC: attempting to log into API
9802023-10-17T23:49:38.395Z2023-10-17 23:49:38.497835219 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9812023-10-17T23:49:39.396Z2023-10-17 23:49:39.496254040 UTC: attempting to log into API
9822023-10-17T23:49:54.431Z2023-10-17 23:49:54.489887447 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9832023-10-17T23:49:55.432Z2023-10-17 23:49:55.488104499 UTC: attempting to log into API
9842023-10-17T23:50:10.466Z2023-10-17 23:50:10.481810794 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9852023-10-17T23:50:11.467Z2023-10-17 23:50:11.479922563 UTC: attempting to log into API
9862023-10-17T23:50:34.124Z2023-10-17 23:50:26.492748703 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9872023-10-17T23:50:34.295Z2023-10-17 23:50:27.493883549 UTC: attempting to log into API
9882023-10-17T23:50:42.535Z2023-10-17 23:50:42.527098955 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9892023-10-17T23:50:43.539Z2023-10-17 23:50:43.528064077 UTC: attempting to log into API
9902023-10-17T23:50:58.575Z2023-10-17 23:50:58.561711385 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9912023-10-17T23:50:59.571Z2023-10-17 23:50:59.562553684 UTC: attempting to log into API
9922023-10-17T23:51:14.604Z2023-10-17 23:51:14.596012988 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9932023-10-17T23:51:15.605Z2023-10-17 23:51:15.597145312 UTC: attempting to log into API
9942023-10-17T23:51:30.640Z2023-10-17 23:51:30.631797293 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9952023-10-17T23:51:31.641Z2023-10-17 23:51:31.632890998 UTC: attempting to log into API
9962023-10-17T23:51:46.673Z2023-10-17 23:51:46.665388566 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9972023-10-17T23:51:47.675Z2023-10-17 23:51:47.666495271 UTC: attempting to log into API
 
9982023-10-17T23:51:56.253ZOct 17 23:51:55 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9992023-10-17T23:52:02.261ZOct 17 23:52:02 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
10002023-10-17T23:52:02.714Z2023-10-17 23:52:02.704480199 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10012023-10-17T23:52:03.714Z2023-10-17 23:52:03.705531403 UTC: attempting to log into API
 
10022023-10-17T23:52:08.902ZOct 17 23:52:08 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
10032023-10-17T23:52:15.750ZOct 17 23:52:15 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
10042023-10-17T23:52:18.748Z2023-10-17 23:52:18.739437142 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10052023-10-17T23:52:19.749Z2023-10-17 23:52:19.741518658 UTC: attempting to log into API
 
10062023-10-17T23:52:21.299ZOct 17 23:52:20 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
10072023-10-17T23:52:34.782Z2023-10-17 23:52:34.774114862 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10082023-10-17T23:52:35.785Z2023-10-17 23:52:35.775216837 UTC: attempting to log into API
10092023-10-17T23:52:36.566Z2023-10-17 23:52:36.550904921 UTC: login succeeded
10102023-10-17T23:52:36.605Zwaiting for nexus to come up...
10112023-10-17T23:52:36.660Zcreating IP pool... 10.151.1.175 - 10.151.1.227
10122023-10-17T23:52:36.727Zensuring datasets are ready...
10132023-10-17T23:52:38.587Zlet's roll.
10142023-10-17T23:52:38.611Z+ rm ./tests/bootstrap
10152023-10-17T23:52:38.671Z+ for test_bin in tests/*
10162023-10-17T23:52:38.694Z+ ./tests/bootstrap-323c561dc2d9ecdb
10172023-10-17T23:52:38.718Z
10182023-10-17T23:52:38.741Zrunning 0 tests
10192023-10-17T23:52:38.765Z
10202023-10-17T23:52:38.788Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10212023-10-17T23:52:38.812Z
10222023-10-17T23:52:38.839Z+ for test_bin in tests/*
10232023-10-17T23:52:38.865Z+ ./tests/end_to_end_tests-183991de1eaf7f15
10242023-10-17T23:52:39.303Z
10252023-10-17T23:52:39.332Zrunning 2 tests
10262023-10-17T23:52:39.358Ztest no_spoof::no_spoof_here ... ok
10272023-10-17T23:53:38.702Ztest instance_launch::instance_launch has been running for over 60 seconds
10282023-10-17T23:55:37.831Ztest instance_launch::instance_launch ... ok
10292023-10-17T23:55:37.855Z
10302023-10-17T23:55:37.879Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 179.13s
10312023-10-17T23:55:37.902Z
10322023-10-17T23:55:37.926Z+ _exit_trap
10332023-10-17T23:55:37.950Z+ local status=0
10342023-10-17T23:55:37.975Z+ [[ 0 -eq 0 ]]
10352023-10-17T23:55:38.002Z+ exit 0
10362023-10-17T23:55:38.027Zprocess exited: duration 522231 ms, exit code 0
 
10372023-10-17T23:55:38.105Zfound 47 output files
10382023-10-17T23:55:38.176Zuploading: /var/svc/log/oxide-sled-agent:default.log (463560 bytes)
10392023-10-17T23:55:39.933Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10402023-10-17T23:55:39.961Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_7e13098e-2fd2-49c9-a85d-a605749bf959/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10412023-10-17T23:55:41.055Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_7e13098e-2fd2-49c9-a85d-a605749bf959/root/var/svc/log/oxide-cockroachdb:default.log
10422023-10-17T23:55:41.082Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5d4e23c2-8be1-47e5-a230-02baf24b08b4/root/var/svc/log/oxide-crucible-agent:default.log (13683 bytes)
10432023-10-17T23:55:42.196Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5d4e23c2-8be1-47e5-a230-02baf24b08b4/root/var/svc/log/oxide-crucible-agent:default.log
10442023-10-17T23:55:42.221Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5d4e23c2-8be1-47e5-a230-02baf24b08b4/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10452023-10-17T23:55:43.740Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5d4e23c2-8be1-47e5-a230-02baf24b08b4/root/var/svc/log/oxide-crucible-downstairs:default.log
10462023-10-17T23:55:43.769Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5d4e23c2-8be1-47e5-a230-02baf24b08b4/root/var/svc/log/oxide-crucible-downstairs:downstairs-9f9717c6-eeee-4bd5-abd7-ba748b4923eb.log (3047 bytes)
10472023-10-17T23:55:44.845Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5d4e23c2-8be1-47e5-a230-02baf24b08b4/root/var/svc/log/oxide-crucible-downstairs:downstairs-9f9717c6-eeee-4bd5-abd7-ba748b4923eb.log
10482023-10-17T23:55:44.873Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_6bef25ef-86b3-4096-8217-608cb63dc7bc/root/var/svc/log/oxide-external_dns:default.log (15612 bytes)
10492023-10-17T23:55:46.401Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_6bef25ef-86b3-4096-8217-608cb63dc7bc/root/var/svc/log/oxide-external_dns:default.log
10502023-10-17T23:55:46.443Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f0b32805-2d4d-4d22-bcc3-4c115c1bc6eb/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
10512023-10-17T23:55:47.518Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f0b32805-2d4d-4d22-bcc3-4c115c1bc6eb/root/var/svc/log/oxide-cockroachdb:default.log
10522023-10-17T23:55:47.547Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e94f6afe-8a0a-4051-b902-c0ff71cd22f6/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10532023-10-17T23:55:48.626Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e94f6afe-8a0a-4051-b902-c0ff71cd22f6/root/var/svc/log/oxide-crucible-agent:default.log
10542023-10-17T23:55:49.134Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e94f6afe-8a0a-4051-b902-c0ff71cd22f6/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10552023-10-17T23:55:49.771Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e94f6afe-8a0a-4051-b902-c0ff71cd22f6/root/var/svc/log/oxide-crucible-downstairs:default.log
10562023-10-17T23:55:49.816Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_8fbe7e97-e585-408f-bf33-bc0675c783f7/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10572023-10-17T23:55:51.368Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_8fbe7e97-e585-408f-bf33-bc0675c783f7/root/var/svc/log/oxide-cockroachdb:default.log
10582023-10-17T23:55:51.395Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed/root/var/svc/log/oxide-ntp:default.log (1253 bytes)
10592023-10-17T23:55:52.466Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed/root/var/svc/log/oxide-ntp:default.log
10602023-10-17T23:55:52.490Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_a2258bff-a7a2-4544-a3ec-db025cb07bc8/root/var/svc/log/oxide-oximeter:default.log (140922 bytes)
10612023-10-17T23:55:52.887Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_a2258bff-a7a2-4544-a3ec-db025cb07bc8/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 140922 -> 154515
10622023-10-17T23:55:53.976Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_a2258bff-a7a2-4544-a3ec-db025cb07bc8/root/var/svc/log/oxide-oximeter:default.log
10632023-10-17T23:55:54.010Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_cbb05e1d-e12c-422b-be1a-3a27adeaf745/root/var/svc/log/oxide-crucible-agent:default.log (24393 bytes)
10642023-10-17T23:55:55.176Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_cbb05e1d-e12c-422b-be1a-3a27adeaf745/root/var/svc/log/oxide-crucible-agent:default.log
10652023-10-17T23:55:55.207Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_cbb05e1d-e12c-422b-be1a-3a27adeaf745/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10662023-10-17T23:55:56.290Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_cbb05e1d-e12c-422b-be1a-3a27adeaf745/root/var/svc/log/oxide-crucible-downstairs:default.log
10672023-10-17T23:55:56.314Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_cbb05e1d-e12c-422b-be1a-3a27adeaf745/root/var/svc/log/oxide-crucible-downstairs:downstairs-aa279fd7-3fd7-4964-a5ba-5b157ca8bf94.log (3051 bytes)
10682023-10-17T23:55:57.389Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_cbb05e1d-e12c-422b-be1a-3a27adeaf745/root/var/svc/log/oxide-crucible-downstairs:downstairs-aa279fd7-3fd7-4964-a5ba-5b157ca8bf94.log
10692023-10-17T23:55:57.414Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_cbb05e1d-e12c-422b-be1a-3a27adeaf745/root/var/svc/log/oxide-crucible-downstairs:downstairs-d8a156bf-1a34-46fe-807c-3aa492967b75.log (5535 bytes)
10702023-10-17T23:55:58.527Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_cbb05e1d-e12c-422b-be1a-3a27adeaf745/root/var/svc/log/oxide-crucible-downstairs:downstairs-d8a156bf-1a34-46fe-807c-3aa492967b75.log
10712023-10-17T23:55:58.561Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_dade24a7-67ff-4f0d-ae0f-d787c413cae3/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10722023-10-17T23:55:59.649Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_dade24a7-67ff-4f0d-ae0f-d787c413cae3/root/var/svc/log/oxide-crucible-pantry:default.log
10732023-10-17T23:55:59.675Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_e060ba70-0402-4eeb-b334-86f9c5464264/root/var/svc/log/oxide-external_dns:default.log (15608 bytes)
10742023-10-17T23:56:00.806Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_e060ba70-0402-4eeb-b334-86f9c5464264/root/var/svc/log/oxide-external_dns:default.log
10752023-10-17T23:56:00.851Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_e0dbe227-ade5-4f03-85fb-c1b6d8bcc7a9/root/var/svc/log/oxide-internal_dns:default.log (99877 bytes)
10762023-10-17T23:56:02.207Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_e0dbe227-ade5-4f03-85fb-c1b6d8bcc7a9/root/var/svc/log/oxide-internal_dns:default.log
10772023-10-17T23:56:02.267Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_c1294f38-9354-415b-afd0-b3ca1059f05d/root/var/svc/log/oxide-nexus:default.log (425299 bytes)
10782023-10-17T23:56:04.488Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_c1294f38-9354-415b-afd0-b3ca1059f05d/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 425299 -> 427123
10792023-10-17T23:56:05.605Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_c1294f38-9354-415b-afd0-b3ca1059f05d/root/var/svc/log/oxide-nexus:default.log
10802023-10-17T23:56:05.628Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_30698664-dda4-4074-9873-eb60e4cf7b32/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
10812023-10-17T23:56:06.704Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_30698664-dda4-4074-9873-eb60e4cf7b32/root/var/svc/log/oxide-cockroachdb:default.log
10822023-10-17T23:56:06.728Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1e599609-ad36-4580-b911-66844dfabb88/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10832023-10-17T23:56:07.812Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1e599609-ad36-4580-b911-66844dfabb88/root/var/svc/log/oxide-crucible-agent:default.log
10842023-10-17T23:56:07.845Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1e599609-ad36-4580-b911-66844dfabb88/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10852023-10-17T23:56:08.933Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1e599609-ad36-4580-b911-66844dfabb88/root/var/svc/log/oxide-crucible-downstairs:default.log
10862023-10-17T23:56:08.962Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_d9fa1fa2-fe7e-4935-abcb-28be098ea924/root/var/svc/log/oxide-internal_dns:default.log (97856 bytes)
10872023-10-17T23:56:10.373Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_d9fa1fa2-fe7e-4935-abcb-28be098ea924/root/var/svc/log/oxide-internal_dns:default.log
10882023-10-17T23:56:10.397Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_eacbca86-0770-4923-8e13-9d8d54e77e63/root/var/svc/log/oxide-internal_dns:default.log (102317 bytes)
10892023-10-17T23:56:11.752Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_eacbca86-0770-4923-8e13-9d8d54e77e63/root/var/svc/log/oxide-internal_dns:default.log
10902023-10-17T23:56:11.784Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_59fcd194-c725-42b8-882f-b16180e15adb/root/var/svc/log/oxide-nexus:default.log (4461460 bytes)
10912023-10-17T23:56:15.214Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_59fcd194-c725-42b8-882f-b16180e15adb/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4461460 -> 4463896
10922023-10-17T23:56:16.315Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_59fcd194-c725-42b8-882f-b16180e15adb/root/var/svc/log/oxide-nexus:default.log
10932023-10-17T23:56:16.354Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_179242f0-3994-4ad8-8a28-7fb8d4d98aff/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10942023-10-17T23:56:17.460Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_179242f0-3994-4ad8-8a28-7fb8d4d98aff/root/var/svc/log/oxide-crucible-agent:default.log
10952023-10-17T23:56:17.490Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_179242f0-3994-4ad8-8a28-7fb8d4d98aff/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10962023-10-17T23:56:18.581Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_179242f0-3994-4ad8-8a28-7fb8d4d98aff/root/var/svc/log/oxide-crucible-downstairs:default.log
10972023-10-17T23:56:18.605Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ecda153e-76ec-4ace-bc95-35280e1efbb3/root/var/svc/log/oxide-crucible-agent:default.log (14145 bytes)
10982023-10-17T23:56:20.177Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ecda153e-76ec-4ace-bc95-35280e1efbb3/root/var/svc/log/oxide-crucible-agent:default.log
10992023-10-17T23:56:20.201Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ecda153e-76ec-4ace-bc95-35280e1efbb3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11002023-10-17T23:56:21.291Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ecda153e-76ec-4ace-bc95-35280e1efbb3/root/var/svc/log/oxide-crucible-downstairs:default.log
11012023-10-17T23:56:21.319Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ecda153e-76ec-4ace-bc95-35280e1efbb3/root/var/svc/log/oxide-crucible-downstairs:downstairs-380577cd-b837-496a-a5fa-e87c485e281e.log (5536 bytes)
11022023-10-17T23:56:22.932Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ecda153e-76ec-4ace-bc95-35280e1efbb3/root/var/svc/log/oxide-crucible-downstairs:downstairs-380577cd-b837-496a-a5fa-e87c485e281e.log
11032023-10-17T23:56:22.964Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_54fa9278-a799-4ed9-b97e-5ac341e7102e/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
11042023-10-17T23:56:24.045Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_54fa9278-a799-4ed9-b97e-5ac341e7102e/root/var/svc/log/oxide-crucible-pantry:default.log
11052023-10-17T23:56:24.068Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3c8ede2e-b57e-4e70-a106-32360545b06c/root/var/svc/log/oxide-crucible-agent:default.log (13680 bytes)
11062023-10-17T23:56:25.188Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3c8ede2e-b57e-4e70-a106-32360545b06c/root/var/svc/log/oxide-crucible-agent:default.log
11072023-10-17T23:56:25.212Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3c8ede2e-b57e-4e70-a106-32360545b06c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11082023-10-17T23:56:26.292Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3c8ede2e-b57e-4e70-a106-32360545b06c/root/var/svc/log/oxide-crucible-downstairs:default.log
11092023-10-17T23:56:26.319Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3c8ede2e-b57e-4e70-a106-32360545b06c/root/var/svc/log/oxide-crucible-downstairs:downstairs-4540dd69-1611-4ffa-8ac3-5763ee115f92.log (3050 bytes)
11102023-10-17T23:56:27.392Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3c8ede2e-b57e-4e70-a106-32360545b06c/root/var/svc/log/oxide-crucible-downstairs:downstairs-4540dd69-1611-4ffa-8ac3-5763ee115f92.log
11112023-10-17T23:56:27.416Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_339e33a9-fc57-4808-963d-0242cc2c9876/root/var/svc/log/oxide-clickhouse:default.log (536111 bytes)
11122023-10-17T23:56:28.435Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_339e33a9-fc57-4808-963d-0242cc2c9876/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 536111 -> 696591
11132023-10-17T23:56:29.920Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_339e33a9-fc57-4808-963d-0242cc2c9876/root/var/svc/log/oxide-clickhouse:default.log
11142023-10-17T23:56:29.946Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_efb793e0-c3bf-4a8d-ae6e-6d4f3f0dfe61/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
11152023-10-17T23:56:31.033Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_efb793e0-c3bf-4a8d-ae6e-6d4f3f0dfe61/root/var/svc/log/oxide-cockroachdb:default.log
11162023-10-17T23:56:31.061Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7b795e54-f2cb-462f-b0be-4648a58b044b/root/var/svc/log/oxide-crucible-agent:default.log (14145 bytes)
11172023-10-17T23:56:32.192Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7b795e54-f2cb-462f-b0be-4648a58b044b/root/var/svc/log/oxide-crucible-agent:default.log
11182023-10-17T23:56:32.217Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7b795e54-f2cb-462f-b0be-4648a58b044b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11192023-10-17T23:56:33.310Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7b795e54-f2cb-462f-b0be-4648a58b044b/root/var/svc/log/oxide-crucible-downstairs:default.log
11202023-10-17T23:56:33.355Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7b795e54-f2cb-462f-b0be-4648a58b044b/root/var/svc/log/oxide-crucible-downstairs:downstairs-21aca25f-21e5-452f-a5fb-60572fc7e2b2.log (5537 bytes)
11212023-10-17T23:56:34.429Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7b795e54-f2cb-462f-b0be-4648a58b044b/root/var/svc/log/oxide-crucible-downstairs:downstairs-21aca25f-21e5-452f-a5fb-60572fc7e2b2.log
11222023-10-17T23:56:34.454Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a5b022b0-364a-4e3f-9f70-6f5de8ed68a0/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
11232023-10-17T23:56:35.527Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a5b022b0-364a-4e3f-9f70-6f5de8ed68a0/root/var/svc/log/oxide-crucible-agent:default.log
11242023-10-17T23:56:35.550Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a5b022b0-364a-4e3f-9f70-6f5de8ed68a0/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11252023-10-17T23:56:36.984Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a5b022b0-364a-4e3f-9f70-6f5de8ed68a0/root/var/svc/log/oxide-crucible-downstairs:default.log
11262023-10-17T23:56:37.016Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_37b59863-7823-4814-af5a-d28fdc0654ec/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
11272023-10-17T23:56:38.112Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_37b59863-7823-4814-af5a-d28fdc0654ec/root/var/svc/log/oxide-crucible-pantry:default.log
11282023-10-17T23:56:38.144Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_69f714a3-d9b5-4836-8cde-d674c1b26e09/root/var/svc/log/oxide-nexus:default.log (509684 bytes)
11292023-10-17T23:56:39.011Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_69f714a3-d9b5-4836-8cde-d674c1b26e09/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 509684 -> 571517
11302023-10-17T23:56:40.102Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_69f714a3-d9b5-4836-8cde-d674c1b26e09/root/var/svc/log/oxide-nexus:default.log
11312023-10-17T23:56:40.129Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed/root/var/log/chrony/measurements.log (14385 bytes)
11322023-10-17T23:56:40.205Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed/root/var/log/chrony/measurements.log" changed size mid upload: 14385 -> 18221
11332023-10-17T23:56:42.118Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed/root/var/log/chrony/measurements.log
11342023-10-17T23:56:42.149Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed/root/var/log/chrony/statistics.log (9477 bytes)
11352023-10-17T23:56:42.224Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed/root/var/log/chrony/statistics.log" changed size mid upload: 9477 -> 11583
11362023-10-17T23:56:43.301Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed/root/var/log/chrony/statistics.log
11372023-10-17T23:56:43.325Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed/root/var/log/chrony/tracking.log (1853 bytes)
11382023-10-17T23:56:43.355Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed/root/var/log/chrony/tracking.log" changed size mid upload: 1853 -> 1985
11392023-10-17T23:56:44.848Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_abd01ee0-6d0a-466b-8d19-a562703d18ed/root/var/log/chrony/tracking.log