01HCH1N6FQD6NNPR8YC93FY9QC: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.23",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HCH1P32KSTS8M704G7W35Q1D

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-12T05:05:46.981Zjob dependencies complete; ready to run (waiting for 47 m 47 s)
22023-10-12T05:06:27.558Zjob assigned to worker 01HCH4DKX90AYDNCH767A8W4D8 (queued for 40 s)
32023-10-12T05:06:27.584Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-10-12T05:06:27.637Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-10-12T05:06:27.671Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-10-12T05:06:27.709Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-12T05:06:27.738Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-10-12T05:06:27.771ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-12T05:06:27.805Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-12T05:06:27.838ZOxide Helios Version helios-2.0.22216 64-bit
112023-10-12T05:06:27.883ZConfiguring devices.
122023-10-12T05:06:27.916Z##################################################
132023-10-12T05:06:27.949Z#### ############# ##
142023-10-12T05:06:27.983Z### ### ############ ##
152023-10-12T05:06:28.016Z## ### # ## ### ## Oxide Computer Company ##
162023-10-12T05:06:28.050Z## ## ## ### # ### ##
172023-10-12T05:06:28.084Z## # ### #### #### This Station Under ##
182023-10-12T05:06:28.116Z### ### #### # ### Computer Control ##
192023-10-12T05:06:28.150Z#### #### ### ## ##
202023-10-12T05:06:28.184Z##################################################
212023-10-12T05:06:28.217ZHostname: sock
222023-10-12T05:06:28.251ZLoading smf(7) serice descriptions:
232023-10-12T05:06:28.729Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-80e9eb4937b4fe98.gz
242023-10-12T05:06:30.181Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-80e9eb4937b4fe98.gz
252023-10-12T05:06:30.206Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-12T05:06:52.896Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-12T05:06:52.927Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-9071c6bef4f9a767.gz
282023-10-12T05:07:25.810Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-9071c6bef4f9a767.gz
292023-10-12T05:07:25.834Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-12T05:07:28.203Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-12T05:07:28.228Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-12T05:07:32.635Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-12T05:07:32.665Zdownloading input: /input/package/work/version.txt
342023-10-12T05:07:32.705Zdownloaded input: /input/package/work/version.txt
352023-10-12T05:07:32.734Zdownloading input: /input/package/work/package.tar.gz
362023-10-12T05:08:08.253Zdownloaded input: /input/package/work/package.tar.gz
372023-10-12T05:08:08.279Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-12T05:08:46.119Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-12T05:08:46.151Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-12T05:09:02.306Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-12T05:09:02.343Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-12T05:09:56.787Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-12T05:09:56.812Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-12T05:10:28.642Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-12T05:10:28.695Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-12T05:10:49.453Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-12T05:10:49.478Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-12T05:10:53.486Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-12T05:10:53.513Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-12T05:11:01.083Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-12T05:11:01.109Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-12T05:11:05.250Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-12T05:11:05.275Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-12T05:11:09.266Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-12T05:11:09.293Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-12T05:11:09.317Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-12T05:11:09.342Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-12T05:11:12.166Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-12T05:11:12.191Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-12T05:11:15.035Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-12T05:11:15.060Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-12T05:11:26.792Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-12T05:11:26.817Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-12T05:11:38.874Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-12T05:11:38.899Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-12T05:11:38.939Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-12T05:11:38.964Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-12T05:11:42.170Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-12T05:11:42.194Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-12T05:11:53.912Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-12T05:11:53.936Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-12T05:12:32.420Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-12T05:12:32.445Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-12T05:12:57.219Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-12T05:12:57.244Zstarting task 0: "setup"
762023-10-12T05:12:57.279Z++ uname -s
772023-10-12T05:12:57.303Z+ kern=SunOS
782023-10-12T05:12:57.328Z+ case "$kern" in
792023-10-12T05:12:57.353Z+ groupadd -g 12345 build
802023-10-12T05:12:57.377Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-12T05:12:59.288Z+ zfs create -o mountpoint=/work rpool/work
822023-10-12T05:12:59.313Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-12T05:12:59.338Z+ home_fs=zfs
842023-10-12T05:12:59.362Z+ [[ zfs == autofs ]]
852023-10-12T05:12:59.387Z+ mkdir -p /home/build
862023-10-12T05:12:59.412Z+ chown build:build /home/build /work
872023-10-12T05:13:01.318Z+ chmod 0700 /home/build /work
882023-10-12T05:13:01.343Zprocess exited: duration 4052 ms, exit code 0
 
892023-10-12T05:13:01.393Zstarting task 1: "authentication"
902023-10-12T05:13:01.463Zprocess exited: duration 44 ms, exit code 0
 
912023-10-12T05:13:01.514Zstarting task 2: "build"
922023-10-12T05:13:01.547Z+ trap _exit_trap EXIT
932023-10-12T05:13:01.573Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-12T05:13:01.740Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-12T05:13:01.766Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-12T05:13:01.791Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-12T05:13:01.816Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-12T05:13:01.841Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-12T05:13:01.866Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-12T05:13:01.892Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-12T05:13:01.917Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-12T05:13:01.943Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-12T05:13:01.968Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-12T05:13:01.993Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-12T05:13:02.019Z+ [[ -d /opt/oxide ]]
1062023-10-12T05:13:02.044Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-12T05:13:02.070Z+ pfexec mkdir /opt/oxide
1082023-10-12T05:13:02.095Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-12T05:13:02.120Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-12T05:13:02.146Z+ cd /opt/oxide-underneath
1112023-10-12T05:13:02.171Z+ pfexec tar ceEp@/f - .
1122023-10-12T05:13:02.196Z+ cd /opt/oxide
1132023-10-12T05:13:02.222Z+ pfexec tar xveEp@/f -
1142023-10-12T05:13:02.247Zx ., 0 bytes, 0 tape blocks
1152023-10-12T05:13:02.272Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-12T05:13:02.297Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-12T05:13:02.322Zx ./opte/bin/opteadm, 50138640 bytes, 97928 tape blocks
1182023-10-12T05:13:02.347Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-12T05:13:02.373Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-12T05:13:02.398Z+ pfexec mkdir /opt/oxide/work
1212023-10-12T05:13:02.423Z+ pfexec chown build:build /opt/oxide/work
1222023-10-12T05:13:02.449Z+ cd /opt/oxide/work
1232023-10-12T05:13:02.475Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-12T05:13:03.498ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-12T05:13:03.524Zx out/maghemite.tar, 74957312 bytes, 146401 tape blocks
1262023-10-12T05:13:03.549Zx out/omicron-sled-agent.tar, 113225728 bytes, 221144 tape blocks
1272023-10-12T05:13:03.576Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-12T05:13:03.601Zx out/npuzone/npuzone, 37379944 bytes, 73008 tape blocks
1292023-10-12T05:13:03.627Zx package-manifest.toml, 21303 bytes, 42 tape blocks
1302023-10-12T05:13:03.674Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-12T05:13:03.726Zx target/release/omicron-package, 37917872 bytes, 74059 tape blocks
1322023-10-12T05:13:03.753Zx tools/create_virtual_hardware.sh, 2818 bytes, 6 tape blocks
1332023-10-12T05:13:03.778Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-12T05:13:03.803Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-12T05:13:03.830Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-12T05:13:03.856Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-12T05:13:03.881Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-12T05:13:03.906Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-12T05:13:03.931Z
1402023-10-12T05:13:03.957Zreal 1.598058335
1412023-10-12T05:13:03.982Zuser 1.440723681
1422023-10-12T05:13:04.007Zsys 0.279220082
1432023-10-12T05:13:04.032Ztrap 0.000059180
1442023-10-12T05:13:04.058Ztflt 0.000000000
1452023-10-12T05:13:04.084Zdflt 0.000000000
1462023-10-12T05:13:04.110Zkflt 0.000000000
1472023-10-12T05:13:04.135Zlock 0.000000000
1482023-10-12T05:13:04.160Zslp 1.438032339
1492023-10-12T05:13:04.186Zlat 0.036135533
1502023-10-12T05:13:04.211Zstop 0.000118501
1512023-10-12T05:13:04.237Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-12T05:13:04.492Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-12T05:13:04.518Z+ mkdir tests
1542023-10-12T05:13:04.561Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-12T05:13:04.590Z+ ptime -m gunzip
1562023-10-12T05:13:04.615Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-80e9eb4937b4fe98
1572023-10-12T05:13:04.652Z
1582023-10-12T05:13:04.698Zreal 0.139574452
1592023-10-12T05:13:04.725Zuser 0.127141675
1602023-10-12T05:13:04.750Zsys 0.012184572
1612023-10-12T05:13:04.775Ztrap 0.000016100
1622023-10-12T05:13:04.800Ztflt 0.000000000
1632023-10-12T05:13:04.825Zdflt 0.000000000
1642023-10-12T05:13:04.850Zkflt 0.000000000
1652023-10-12T05:13:04.875Zlock 0.000000000
1662023-10-12T05:13:04.901Zslp 0.000023680
1672023-10-12T05:13:04.926Zlat 0.000020709
1682023-10-12T05:13:04.952Zstop 0.000120822
1692023-10-12T05:13:04.977Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-80e9eb4937b4fe98
1702023-10-12T05:13:05.002Z+ chmod a+x tests/bootstrap-80e9eb4937b4fe98
1712023-10-12T05:13:05.027Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-12T05:13:05.052Z+ ptime -m gunzip
1732023-10-12T05:13:05.076Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-12T05:13:06.861Z
1752023-10-12T05:13:06.886Zreal 2.197625440
1762023-10-12T05:13:06.911Zuser 2.070448357
1772023-10-12T05:13:06.936Zsys 0.126762127
1782023-10-12T05:13:06.960Ztrap 0.000192123
1792023-10-12T05:13:06.985Ztflt 0.000000000
1802023-10-12T05:13:07.010Zdflt 0.000000000
1812023-10-12T05:13:07.035Zkflt 0.000000000
1822023-10-12T05:13:07.059Zlock 0.000000000
1832023-10-12T05:13:07.084Zslp 0.000021380
1842023-10-12T05:13:07.109Zlat 0.000022160
1852023-10-12T05:13:07.134Zstop 0.000123832
1862023-10-12T05:13:07.159Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-12T05:13:07.184Z+ chmod a+x tests/bootstrap
1882023-10-12T05:13:07.209Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-12T05:13:07.267Z+ ptime -m gunzip
1902023-10-12T05:13:07.293Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-9071c6bef4f9a767
1912023-10-12T05:13:09.386Z
1922023-10-12T05:13:09.412Zreal 2.505664621
1932023-10-12T05:13:09.437Zuser 2.356883266
1942023-10-12T05:13:09.462Zsys 0.148339240
1952023-10-12T05:13:09.487Ztrap 0.000205673
1962023-10-12T05:13:09.512Ztflt 0.000000000
1972023-10-12T05:13:09.537Zdflt 0.000000000
1982023-10-12T05:13:09.562Zkflt 0.000000000
1992023-10-12T05:13:09.587Zlock 0.000000000
2002023-10-12T05:13:09.612Zslp 0.000021090
2012023-10-12T05:13:09.637Zlat 0.000021339
2022023-10-12T05:13:09.662Zstop 0.000131602
2032023-10-12T05:13:09.687Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-9071c6bef4f9a767
2042023-10-12T05:13:09.711Z+ chmod a+x tests/end_to_end_tests-9071c6bef4f9a767
2052023-10-12T05:13:09.736Z+ bmat address ls
2062023-10-12T05:13:09.766ZNAME CIDR FIRST LAST COUNT
2072023-10-12T05:13:09.791Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-10-12T05:13:09.822Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-12T05:13:09.847Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-10-12T05:13:09.872Z+ EXTRA_IP_START=10.151.2.164
2112023-10-12T05:13:09.921Z+ EXTRA_IP_END=10.151.2.227
2122023-10-12T05:13:09.946Z+ EXTRA_IP_COUNT=64
2132023-10-12T05:13:09.971Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-10-12T05:13:09.996Z+ EXTRA_IP_BASE=10.151.2
2152023-10-12T05:13:10.020Z+ EXTRA_IP_FOCTET=164
2162023-10-12T05:13:10.045Z+ EXTRA_IP_LOCTET=227
2172023-10-12T05:13:10.069Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-10-12T05:13:10.094Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-10-12T05:13:10.119Z+ DNS_IP1=10.151.2.164
2202023-10-12T05:13:10.144Z+ DNS_IP2=10.151.2.165
2212023-10-12T05:13:10.168Z+ UPLINK_IP=10.151.2.174
2222023-10-12T05:13:10.193Z+ IPPOOL_START=10.151.2.175
2232023-10-12T05:13:10.218Z+ IPPOOL_END=10.151.2.227
2242023-10-12T05:13:10.243Z++ ipadm show-addr -po type,addr
2252023-10-12T05:13:10.268Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-10-12T05:13:10.292Z+ GATEWAY_IP=10.151.2.100
2272023-10-12T05:13:10.317Z+ [[ -n 10.151.2.100 ]]
2282023-10-12T05:13:10.342Z+ ping -s 10.151.2.100 56 1
2292023-10-12T05:13:10.367ZPING 10.151.2.100: 56 data bytes
2302023-10-12T05:13:10.392Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2312023-10-12T05:13:10.417Z
2322023-10-12T05:13:10.442Z----10.151.2.100 PING Statistics----
2332023-10-12T05:13:10.467Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-10-12T05:13:10.492Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-10-12T05:13:10.517Z++ arp -an
2362023-10-12T05:13:10.542Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-10-12T05:13:10.566Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-10-12T05:13:10.592Z+ routeadm -e ipv4-forwarding -u
2392023-10-12T05:13:10.618Z+ PXA_START=10.151.2.164
2402023-10-12T05:13:10.643Z+ PXA_END=10.151.2.227
2412023-10-12T05:13:10.668Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-10-12T05:13:10.693Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-10-12T05:13:10.718Z+ ZPOOL_VDEV_DIR=/scratch
2442023-10-12T05:13:10.742Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-10-12T05:13:10.767Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-10-12T05:13:10.791Z++ cd ./tools
2472023-10-12T05:13:10.816Z++ pwd
2482023-10-12T05:13:10.847Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-10-12T05:13:10.872Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-10-12T05:13:10.897Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-10-12T05:13:10.925Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-10-12T05:13:10.951Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-10-12T05:13:10.976Z++ dladm show-phys -p -o LINK
2542023-10-12T05:13:11.001Z++ head -1
2552023-10-12T05:13:11.026Z+ PHYSICAL_LINK=igb0
2562023-10-12T05:13:11.052Z+ echo 'Using igb0 as physical link'
2572023-10-12T05:13:11.078ZUsing igb0 as physical link
2582023-10-12T05:13:11.105Z+ ensure_run_as_root
2592023-10-12T05:13:11.130Z++ id -u
2602023-10-12T05:13:11.154Z+ [[ 0 -ne 0 ]]
2612023-10-12T05:13:11.179Z+ ensure_zpools
2622023-10-12T05:13:11.205Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-10-12T05:13:11.229Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-10-12T05:13:11.254Z+ readarray -t ZPOOLS
2652023-10-12T05:13:11.279Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-10-12T05:13:11.304Z++ sed 's/[ ",]//g'
2672023-10-12T05:13:11.329Z+ for ZPOOL in "${ZPOOLS[@]}"
2682023-10-12T05:13:11.354Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2692023-10-12T05:13:11.379ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2702023-10-12T05:13:11.404Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2712023-10-12T05:13:11.435Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2722023-10-12T05:13:11.468Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2732023-10-12T05:13:11.498Z0+0 records in
2742023-10-12T05:13:11.524ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2752023-10-12T05:13:11.549Z0+0 records out
2762023-10-12T05:13:11.579ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2772023-10-12T05:13:11.604Z0 bytes (0 iB) transferred in 0.000007 secs (0 iB/sec)
2782023-10-12T05:13:11.632ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2792023-10-12T05:13:11.657Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2802023-10-12T05:13:11.682ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2812023-10-12T05:13:11.707Z+ set +x
2822023-10-12T05:13:11.732ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2832023-10-12T05:13:11.757Z++ zpool list -o name
2842023-10-12T05:13:11.783ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2852023-10-12T05:13:11.807Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2862023-10-12T05:13:11.832ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2872023-10-12T05:13:11.872Z+ [[ -z '' ]]
2882023-10-12T05:13:12.315ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2892023-10-12T05:13:12.347Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2902023-10-12T05:13:12.376ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2912023-10-12T05:13:12.408Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2922023-10-12T05:13:12.438ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2932023-10-12T05:13:12.463Z+ set +x
2942023-10-12T05:13:12.488ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2952023-10-12T05:13:12.512Z+ for ZPOOL in "${ZPOOLS[@]}"
2962023-10-12T05:13:12.537ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2972023-10-12T05:13:12.562Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2982023-10-12T05:13:12.587ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-10-12T05:13:12.612Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3002023-10-12T05:13:12.638ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3012023-10-12T05:13:12.663Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3022023-10-12T05:13:12.688ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3032023-10-12T05:13:12.713Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3042023-10-12T05:13:12.740ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3052023-10-12T05:13:12.765Z0+0 records in
3062023-10-12T05:13:12.792ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3072023-10-12T05:13:12.829Z0+0 records out
3082023-10-12T05:13:12.868ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3092023-10-12T05:13:12.914Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3102023-10-12T05:13:12.952ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3112023-10-12T05:13:12.979Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3122023-10-12T05:13:13.003ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3132023-10-12T05:13:13.028Z+ set +x
3142023-10-12T05:13:13.053ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3152023-10-12T05:13:13.078Z++ zpool list -o name
3162023-10-12T05:13:13.102ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3172023-10-12T05:13:13.127Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3182023-10-12T05:13:13.152ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3192023-10-12T05:13:13.176Z+ [[ -z '' ]]
3202023-10-12T05:13:13.201ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3212023-10-12T05:13:13.225Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-10-12T05:13:13.251ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3232023-10-12T05:13:13.277Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3242023-10-12T05:13:13.302ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3252023-10-12T05:13:13.327Z+ set +x
3262023-10-12T05:13:13.352ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3272023-10-12T05:13:13.377Z+ for ZPOOL in "${ZPOOLS[@]}"
3282023-10-12T05:13:13.401ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3292023-10-12T05:13:13.426Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3302023-10-12T05:13:13.451ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3312023-10-12T05:13:13.476Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3322023-10-12T05:13:13.501ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3332023-10-12T05:13:13.526Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3342023-10-12T05:13:13.551ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3352023-10-12T05:13:13.575Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3362023-10-12T05:13:13.600ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3372023-10-12T05:13:13.625Z0+0 records in
3382023-10-12T05:13:13.656ZSimnet net0/sc0_0 exists
3392023-10-12T05:13:13.685Z0+0 records out
3402023-10-12T05:13:13.712ZSimnet net1/sc1_0 exists
3412023-10-12T05:13:13.768Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3422023-10-12T05:13:13.801ZVnic sc0_1 exists
3432023-10-12T05:13:13.828Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3442023-10-12T05:13:13.855Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3452023-10-12T05:13:13.882Z+ set +x
3462023-10-12T05:13:13.910Z++ zpool list -o name
3472023-10-12T05:13:13.945Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3482023-10-12T05:13:13.973Z+ [[ -z '' ]]
3492023-10-12T05:13:14.004Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-10-12T05:13:14.034Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-10-12T05:13:14.061Z+ set +x
3522023-10-12T05:13:14.093Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-10-12T05:13:14.122Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-10-12T05:13:14.151Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-10-12T05:13:14.181Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-10-12T05:13:14.213Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-10-12T05:13:14.245Z0+0 records in
3582023-10-12T05:13:14.274Z0+0 records out
3592023-10-12T05:13:14.300Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3602023-10-12T05:13:14.324Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-10-12T05:13:14.348Z+ set +x
3622023-10-12T05:13:14.373Z++ zpool list -o name
3632023-10-12T05:13:14.398Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-10-12T05:13:14.423Z+ [[ -z '' ]]
3652023-10-12T05:13:14.447Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-10-12T05:13:14.472Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-10-12T05:13:14.497Z+ set +x
3682023-10-12T05:13:14.522Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-10-12T05:13:14.546Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-10-12T05:13:14.571Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-10-12T05:13:14.595Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-10-12T05:13:14.620Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-10-12T05:13:14.645Z0+0 records in
3742023-10-12T05:13:14.670Z0+0 records out
3752023-10-12T05:13:14.694Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3762023-10-12T05:13:14.719Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-10-12T05:13:14.746Z+ set +x
3782023-10-12T05:13:14.771Z++ zpool list -o name
3792023-10-12T05:13:14.795Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-10-12T05:13:14.820Z+ [[ -z '' ]]
3812023-10-12T05:13:14.845Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-10-12T05:13:14.870Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-10-12T05:13:14.895Z+ set +x
3842023-10-12T05:13:14.920Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-10-12T05:13:14.969Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862023-10-12T05:13:14.994Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872023-10-12T05:13:15.019Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882023-10-12T05:13:15.043Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892023-10-12T05:13:15.069Z0+0 records in
3902023-10-12T05:13:15.094Z0+0 records out
3912023-10-12T05:13:15.119Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-10-12T05:13:15.144Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932023-10-12T05:13:15.169Z+ set +x
3942023-10-12T05:13:15.193Z++ zpool list -o name
3952023-10-12T05:13:15.219Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3962023-10-12T05:13:15.244Z+ [[ -z '' ]]
3972023-10-12T05:13:15.270Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3982023-10-12T05:13:15.294Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3992023-10-12T05:13:15.329Z+ set +x
4002023-10-12T05:13:15.354Z+ for ZPOOL in "${ZPOOLS[@]}"
4012023-10-12T05:13:15.379Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4022023-10-12T05:13:15.403Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4032023-10-12T05:13:15.428Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4042023-10-12T05:13:15.453Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4052023-10-12T05:13:15.478Z0+0 records in
4062023-10-12T05:13:15.503Z0+0 records out
4072023-10-12T05:13:15.527Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4082023-10-12T05:13:15.552Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4092023-10-12T05:13:15.576Z+ set +x
4102023-10-12T05:13:15.601Z++ zpool list -o name
4112023-10-12T05:13:15.626Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4122023-10-12T05:13:15.654Z+ [[ -z '' ]]
4132023-10-12T05:13:15.678Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4142023-10-12T05:13:15.703Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4152023-10-12T05:13:15.728Z+ set +x
4162023-10-12T05:13:15.753Z+ for ZPOOL in "${ZPOOLS[@]}"
4172023-10-12T05:13:15.777Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4182023-10-12T05:13:15.802Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192023-10-12T05:13:15.827Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4202023-10-12T05:13:15.852Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4212023-10-12T05:13:15.877Z0+0 records in
4222023-10-12T05:13:15.902Z0+0 records out
4232023-10-12T05:13:15.927Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4242023-10-12T05:13:15.951Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4252023-10-12T05:13:15.995Z+ set +x
4262023-10-12T05:13:16.021Z++ zpool list -o name
4272023-10-12T05:13:16.046Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4282023-10-12T05:13:16.070Z+ [[ -z '' ]]
4292023-10-12T05:13:16.095Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4302023-10-12T05:13:16.119Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4312023-10-12T05:13:16.144Z+ set +x
4322023-10-12T05:13:16.168Z+ for ZPOOL in "${ZPOOLS[@]}"
4332023-10-12T05:13:16.193Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4342023-10-12T05:13:16.217Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4352023-10-12T05:13:16.242Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4362023-10-12T05:13:16.267Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4372023-10-12T05:13:16.291Z0+0 records in
4382023-10-12T05:13:16.316Z0+0 records out
4392023-10-12T05:13:16.341Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4402023-10-12T05:13:16.367Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4412023-10-12T05:13:16.393Z+ set +x
4422023-10-12T05:13:16.417Z++ zpool list -o name
4432023-10-12T05:13:16.442Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4442023-10-12T05:13:16.466Z+ [[ -z '' ]]
4452023-10-12T05:13:16.491Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4462023-10-12T05:13:16.516Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4472023-10-12T05:13:16.540Z+ set +x
4482023-10-12T05:13:16.565Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4492023-10-12T05:13:16.590Z+ readarray -t ZPOOLS
4502023-10-12T05:13:16.614Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4512023-10-12T05:13:16.639Z++ sed 's/[ ",]//g'
4522023-10-12T05:13:16.663Z+ for ZPOOL in "${ZPOOLS[@]}"
4532023-10-12T05:13:16.688Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4542023-10-12T05:13:16.713Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4552023-10-12T05:13:16.737Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4562023-10-12T05:13:16.762Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4572023-10-12T05:13:16.787Z0+0 records in
4582023-10-12T05:13:16.811Z0+0 records out
4592023-10-12T05:13:16.836Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4602023-10-12T05:13:16.860Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4612023-10-12T05:13:16.885Z+ set +x
4622023-10-12T05:13:16.910Z++ zpool list -o name
4632023-10-12T05:13:16.934Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4642023-10-12T05:13:16.959Z+ [[ -z '' ]]
4652023-10-12T05:13:16.984Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-10-12T05:13:17.021Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4672023-10-12T05:13:17.045Z+ set +x
4682023-10-12T05:13:17.070Z+ for ZPOOL in "${ZPOOLS[@]}"
4692023-10-12T05:13:17.094Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4702023-10-12T05:13:17.119Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4712023-10-12T05:13:17.144Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4722023-10-12T05:13:17.168Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4732023-10-12T05:13:17.193Z0+0 records in
4742023-10-12T05:13:17.218Z0+0 records out
4752023-10-12T05:13:17.242Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4762023-10-12T05:13:17.266Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4772023-10-12T05:13:17.291Z+ set +x
4782023-10-12T05:13:17.316Z++ zpool list -o name
4792023-10-12T05:13:17.341Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4802023-10-12T05:13:17.368Z+ [[ -z '' ]]
4812023-10-12T05:13:17.396Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4822023-10-12T05:13:17.426Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4832023-10-12T05:13:17.452Z+ set +x
4842023-10-12T05:13:17.478Z+ ensure_simulated_links igb0
4852023-10-12T05:13:17.504Z+ local PHYSICAL_LINK=igb0
4862023-10-12T05:13:17.529Z+ INDICES=("0" "1")
4872023-10-12T05:13:17.554Z+ for I in "${INDICES[@]}"
4882023-10-12T05:13:17.580Z++ get_simnet_name_if_exists net0
4892023-10-12T05:13:17.606Z++ dladm show-simnet -p -o LINK net0
4902023-10-12T05:13:17.631Z++ echo ''
4912023-10-12T05:13:17.655Z+ [[ -z '' ]]
4922023-10-12T05:13:17.680Z+ dladm create-simnet -t net0
4932023-10-12T05:13:17.704Z+ dladm create-simnet -t sc0_0
4942023-10-12T05:13:17.729Z+ dladm modify-simnet -t -p net0 sc0_0
4952023-10-12T05:13:17.753Z+ dladm set-linkprop -p mtu=9000 sc0_0
4962023-10-12T05:13:17.778Z+ success 'Simnet net0/sc0_0 exists'
4972023-10-12T05:13:17.803Z+ set +x
4982023-10-12T05:13:18.234Z+ for I in "${INDICES[@]}"
4992023-10-12T05:13:18.259Z++ get_simnet_name_if_exists net1
5002023-10-12T05:13:18.285Z++ dladm show-simnet -p -o LINK net1
5012023-10-12T05:13:18.324Z++ echo ''
5022023-10-12T05:13:18.777Z+ [[ -z '' ]]
5032023-10-12T05:13:18.807Z+ dladm create-simnet -t net1
5042023-10-12T05:13:18.839Z+ dladm create-simnet -t sc1_0
5052023-10-12T05:13:18.869Z+ dladm modify-simnet -t -p net1 sc1_0
5062023-10-12T05:13:18.894Z+ dladm set-linkprop -p mtu=9000 sc1_0
5072023-10-12T05:13:18.950Z+ success 'Simnet net1/sc1_0 exists'
5082023-10-12T05:13:18.988Z+ set +x
5092023-10-12T05:13:19.013Z++ get_vnic_name_if_exists sc0_1
5102023-10-12T05:13:19.038Z++ dladm show-vnic -p -o LINK sc0_1
5112023-10-12T05:13:19.070Z++ echo ''
5122023-10-12T05:13:19.095Z+ [[ -z '' ]]
5132023-10-12T05:13:19.120Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-10-12T05:13:19.144Z+ success 'Vnic sc0_1 exists'
5152023-10-12T05:13:19.169Z+ set +x
5162023-10-12T05:13:19.194Z+ warn_if_no_proxy_arp
5172023-10-12T05:13:19.220Z+ [[ -v PXA_START ]]
5182023-10-12T05:13:19.245Z+ [[ -v PXA_END ]]
5192023-10-12T05:13:19.270Z+ ensure_softnpu_zone
5202023-10-12T05:13:19.295Z+ zoneadm list
5212023-10-12T05:13:19.320Z+ grep -q sidecar_softnpu
5222023-10-12T05:13:19.345Z+ [[ -f out/npuzone/npuzone ]]
5232023-10-12T05:13:19.369Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0
5242023-10-12T05:13:19.396Zdownloading softnpu ...done
5252023-10-12T05:13:19.843Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/libsidecar_lite.so.sha256.txt
5262023-10-12T05:13:22.991Zdownloading asic_program.so ...done
5272023-10-12T05:13:23.044Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/scadm.sha256.txt
5282023-10-12T05:13:29.738Zdownloading scadm ...done
5292023-10-12T05:13:29.763Zcreating port tfportrear0_0 for sc0_0
5302023-10-12T05:13:29.788Zcreating port tfportqsfp0_0 for sc0_1
5312023-10-12T05:13:29.812Zsoftnpu zone setup
5322023-10-12T05:13:29.837Zconfigure zone
5332023-10-12T05:13:29.862Zinstall zone
5342023-10-12T05:13:30.841Zboot zone
5352023-10-12T05:13:33.951Z[sidecar_softnpu] svcs milestone/network
5362023-10-12T05:13:33.978ZSTATE STIME FMRI
5372023-10-12T05:13:34.002Zonline 5:13:32 svc:/milestone/network:default
5382023-10-12T05:13:34.027Z[sidecar_softnpu] /softnpu/npu start
5392023-10-12T05:13:34.052Z
5402023-10-12T05:13:34.077Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5412023-10-12T05:13:34.101Z+ GATEWAY_IP=10.151.2.100
5422023-10-12T05:13:34.126Z+ echo 'Using 10.151.2.100 as gateway ip'
5432023-10-12T05:13:34.151ZUsing 10.151.2.100 as gateway ip
5442023-10-12T05:13:34.175Z+ [[ ! -v GATEWAY_MAC ]]
5452023-10-12T05:13:34.200Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5462023-10-12T05:13:34.225Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5472023-10-12T05:13:34.250Z+++ echo 18:c0:4d:0d:a0:2a
5482023-10-12T05:13:34.275Z+ COUNT=1
5492023-10-12T05:13:34.299Z+ [[ 1 -ne 1 ]]
5502023-10-12T05:13:34.324Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5512023-10-12T05:13:34.349ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5522023-10-12T05:13:34.374Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5532023-10-12T05:13:34.399Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5542023-10-12T05:13:34.424Z+ PXA_MAC=a8:e1:de:01:70:1d
5552023-10-12T05:13:34.464Z+ [[ -v PXA_START ]]
5562023-10-12T05:13:34.491Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5572023-10-12T05:13:34.516Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5582023-10-12T05:13:34.540Z+ z_scadm dump-state
5592023-10-12T05:13:34.565Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5602023-10-12T05:13:34.589Zlocal v6:
5612023-10-12T05:13:34.614Zlocal v4:
5622023-10-12T05:13:34.639Zrouter v6:
5632023-10-12T05:13:34.664Zrouter v4:
5642023-10-12T05:13:34.689Zresolver v4:
5652023-10-12T05:13:34.714Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5662023-10-12T05:13:34.739Zresolver v6:
5672023-10-12T05:13:34.763Znat v4:
5682023-10-12T05:13:34.787Znat_v6:
5692023-10-12T05:13:34.812Zport_mac:
5702023-10-12T05:13:34.836Zproxy arp:
5712023-10-12T05:13:34.861Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5722023-10-12T05:13:34.886Z+ success 'softnpu zone exists'
5732023-10-12T05:13:34.910Z+ set +x
5742023-10-12T05:13:34.935Zsoftnpu zone exists
5752023-10-12T05:13:34.960Z
5762023-10-12T05:13:34.984Zreal 24.377505684
5772023-10-12T05:13:35.009Zuser 0.721937615
5782023-10-12T05:13:35.034Zsys 1.651851843
5792023-10-12T05:13:35.059Ztrap 0.000108020
5802023-10-12T05:13:35.083Ztflt 0.000186893
5812023-10-12T05:13:35.108Zdflt 0.000911943
5822023-10-12T05:13:35.133Zkflt 0.000003538
5832023-10-12T05:13:35.158Zlock 18:57.148641293
5842023-10-12T05:13:35.182Zslp 1:12.321120704
5852023-10-12T05:13:35.207Zlat 0.080510588
5862023-10-12T05:13:35.231Zstop 0.373258358
5872023-10-12T05:13:35.256Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5882023-10-12T05:13:35.282Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5892023-10-12T05:13:35.306Z+ SILO_NAME=recovery
5902023-10-12T05:13:35.331Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5912023-10-12T05:13:35.356Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5922023-10-12T05:13:35.381Z+ sed -i~ '
5932023-10-12T05:13:35.406Z /^external_dns_ips/c\
5942023-10-12T05:13:35.430Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5952023-10-12T05:13:35.455Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5962023-10-12T05:13:35.491Z /^first/c\
5972023-10-12T05:13:35.517Zfirst = "10.151.2.164"
5982023-10-12T05:13:35.543Z /^last/c\
5992023-10-12T05:13:35.568Zlast = "10.151.2.173"
6002023-10-12T05:13:35.594Z }
6012023-10-12T05:13:35.619Z /^\[rack_network_config/,/^$/ {
6022023-10-12T05:13:35.644Z /^infra_ip_first/c\
6032023-10-12T05:13:35.668Zinfra_ip_first = "10.151.2.174"
6042023-10-12T05:13:35.693Z /^infra_ip_last/c\
6052023-10-12T05:13:35.718Zinfra_ip_last = "10.151.2.174"
6062023-10-12T05:13:35.742Z }
6072023-10-12T05:13:35.767Z /^\[\[rack_network_config.uplinks/,/^$/ {
6082023-10-12T05:13:35.792Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6092023-10-12T05:13:35.817Z /^gateway_ip/c\
6102023-10-12T05:13:35.842Z+++ pkg/config-rss.toml Thu Oct 12 05:13:34 2023
6112023-10-12T05:13:35.867Zgateway_ip = "10.151.2.100"
6122023-10-12T05:13:35.891Z@@ -41,7 +41,7 @@
6132023-10-12T05:13:35.916Z /^uplink_cidr/c\
6142023-10-12T05:13:35.940Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6152023-10-12T05:13:35.965Zuplink_cidr = "10.151.2.174/32"
6162023-10-12T05:13:35.991Z # must be contained in one of the "internal services" IP Pool ranges listed
6172023-10-12T05:13:36.016Z }
6182023-10-12T05:13:36.040Z # below.
6192023-10-12T05:13:36.065Z' pkg/config-rss.toml
6202023-10-12T05:13:36.089Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6212023-10-12T05:13:36.114Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6222023-10-12T05:13:36.138Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6232023-10-12T05:13:36.163Z+ true
6242023-10-12T05:13:36.188Z
6252023-10-12T05:13:36.212Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6262023-10-12T05:13:36.237Z # Initial TLS certificates for the external API
6272023-10-12T05:13:36.262Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6282023-10-12T05:13:36.287Z #
6292023-10-12T05:13:36.312Z+ TLS_NAME=recovery.sys.oxide.test
6302023-10-12T05:13:36.337Z@@ -77,8 +77,8 @@
6312023-10-12T05:13:36.363Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6322023-10-12T05:13:36.388Z #
6332023-10-12T05:13:36.413Z # For more on this and what to put here, see docs/how-to-run.adoc.
6342023-10-12T05:13:36.437Z [[internal_services_ip_pool_ranges]]
6352023-10-12T05:13:36.462Z-first = "192.168.1.20"
6362023-10-12T05:13:36.486Z-last = "192.168.1.29"
6372023-10-12T05:13:36.517Z+first = "10.151.2.164"
6382023-10-12T05:13:36.542Z+last = "10.151.2.173"
6392023-10-12T05:13:36.575Z
6402023-10-12T05:13:36.600Z # TODO - this configuration is subject to change going forward. Ultimately these
6412023-10-12T05:13:36.624Z # parameters should be provided to the control plane via wicket, but we need to
6422023-10-12T05:13:36.649Z@@ -91,13 +91,13 @@
6432023-10-12T05:13:36.674Z # A range of IP addresses used by Boundary Services on the external network. In
6442023-10-12T05:13:36.698Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6452023-10-12T05:13:36.723Z..+.......+......+...+...........+...+...+.........+.+........+...................+......+............+..+......+....+..+.+........+.+.....+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+.+..............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+...+...+.+...+...+...+...........+..........+...+........+............+.+.........+..+....+...............+.....+.......+....................+.........+...+...+.............+.....+............+....+...+...+...............+..+.+.........+............+...+..+.........+....+..+...+.+..............+...+...+.........+...+.........+....+......+.....+...+......+......+.+.........+...+....................+.+..............+.+..+.......+......+......+......+...+...........+.........+....+...+.....+....+.....+..................+...............+.+..+..........+..+...+...+....+..................+..+......................+.........+..+..........+.....+.......+.....+.......+......+..+......+.+......+..................+.................+...+......+.+............+..........................+....+.....+.+............+........+......+....+..+..........+..+.............+........+.........+.+..+............+...+...+....+............+...............+..................+...+..+......+....+........+.............+..+....+.....................+.........+..............+.+.........+..+.........+.+.........+...+...............+..+...+.+......+.....+...+..........+............+.........+....................+......+.+........+....+...+...............+.........+.....+......+....+......+..+..........+............+.....+...+...+............+................+............+..................+..+....+..................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6462023-10-12T05:13:36.747Z # With softnpu, only one address is used.
6472023-10-12T05:13:36.772Z-infra_ip_first = "192.168.1.30"
6482023-10-12T05:13:36.797Z-infra_ip_last = "192.168.1.30"
6492023-10-12T05:13:36.822Z+infra_ip_first = "10.151.2.174"
6502023-10-12T05:13:36.846Z+infra_ip_last = "10.151.2.174"
6512023-10-12T05:13:36.872Z
6522023-10-12T05:13:36.896Z # You can configure multiple uplinks by repeating the following stanza
6532023-10-12T05:13:36.921Z.......+...+.............+..+.+.....+....+..+..........+.........+.........+........+....+...+......+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.......+..+....+.....+..........+......+...+...........+..........+.........+.....+............+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6542023-10-12T05:13:36.947Z [[rack_network_config.uplinks]]
6552023-10-12T05:13:36.971Z-----
6562023-10-12T05:13:36.996Z # The gateway IP for the rack's external network
6572023-10-12T05:13:37.021Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6582023-10-12T05:13:37.045Z-gateway_ip = "192.168.1.199"
6592023-10-12T05:13:37.070Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6602023-10-12T05:13:37.095Z+gateway_ip = "10.151.2.100"
6612023-10-12T05:13:37.119Z+ rmdir pkg
6622023-10-12T05:13:37.144Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6632023-10-12T05:13:37.173Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6642023-10-12T05:13:37.198Z uplink_port = "qsfp0"
6652023-10-12T05:13:37.223Z+ pfexec mkdir -p /usr/oxide
6662023-10-12T05:13:37.248Z uplink_port_speed = "40G"
6672023-10-12T05:13:37.272Z+ pfexec rm -f /usr/oxide/tcpproxy
6682023-10-12T05:13:37.297Z@@ -104,7 +104,7 @@
6692023-10-12T05:13:37.363Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6702023-10-12T05:13:37.388Z uplink_port_fec="none"
6712023-10-12T05:13:37.413Z+ pfexec chmod +x /usr/oxide/tcpproxy
6722023-10-12T05:13:37.438Z # For softnpu, an address within the "infra" block above that will be used for
6732023-10-12T05:13:37.462Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6742023-10-12T05:13:37.546Z # the softnpu uplink port. You can just pick the first address in that pool.
6752023-10-12T05:13:37.608Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6762023-10-12T05:13:37.636Z-uplink_cidr = "192.168.1.30/32"
6772023-10-12T05:13:37.687Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6782023-10-12T05:13:37.712Z+uplink_cidr = "10.151.2.174/32"
6792023-10-12T05:13:37.737Z+ OMICRON_NO_UNINSTALL=1
6802023-10-12T05:13:37.805Z # Switch to use for the uplink. For single-rack deployments this can be
6812023-10-12T05:13:37.830Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6822023-10-12T05:13:37.857Z # "switch0" (upper slot) or "switch1" (lower slot). For single-node softnpu
6832023-10-12T05:13:37.883ZOct 12 05:13:35.609 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6842023-10-12T05:13:37.908Z # and dendrite stub environments, use "switch0"
6852023-10-12T05:13:37.932ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6862023-10-12T05:13:37.957Za pkg/config-rss.toml 7K
6872023-10-12T05:13:37.981ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6882023-10-12T05:13:38.006Za pkg/initial-tls-cert.pem 2K
6892023-10-12T05:13:38.031ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6902023-10-12T05:13:38.056Za pkg/initial-tls-key.pem 4K
6912023-10-12T05:13:38.080ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6922023-10-12T05:13:38.105ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6932023-10-12T05:13:38.130ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6942023-10-12T05:13:38.154ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6952023-10-12T05:13:38.179ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6962023-10-12T05:13:38.203ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6972023-10-12T05:13:38.228ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6982023-10-12T05:13:38.253ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6992023-10-12T05:13:38.278ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/maghemite.tar
7002023-10-12T05:13:38.303ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7012023-10-12T05:13:38.328ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7022023-10-12T05:13:38.354ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7032023-10-12T05:13:38.379ZOct 12 05:13:35.611 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7042023-10-12T05:13:38.404ZOct 12 05:13:35.813 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7052023-10-12T05:13:38.429ZOct 12 05:13:35.856 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7062023-10-12T05:13:38.454ZOct 12 05:13:35.921 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7072023-10-12T05:13:38.479Z
7082023-10-12T05:13:38.504Zreal 0.450634332
7092023-10-12T05:13:38.528Zuser 0.095581150
7102023-10-12T05:13:38.573Zsys 1.701199610
7112023-10-12T05:13:38.598Ztrap 0.000000429
7122023-10-12T05:13:38.633Ztflt 0.000053767
7132023-10-12T05:13:38.710Zdflt 0.001172435
7142023-10-12T05:13:38.735Zkflt 0.000000000
7152023-10-12T05:13:38.759Zlock 40.515050370
7162023-10-12T05:13:38.784Zslp 0.661713971
7172023-10-12T05:13:38.808Zlat 0.002777517
7182023-10-12T05:13:38.833Zstop 0.221263888
7192023-10-12T05:13:38.858Z+ retry=0
7202023-10-12T05:13:38.882Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7212023-10-12T05:13:38.907Z+ [[ 0 -gt 30 ]]
7222023-10-12T05:13:38.931Z+ sleep 1
7232023-10-12T05:13:38.956Z+ retry=1
7242023-10-12T05:13:38.981Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7252023-10-12T05:13:39.009Z+ [[ 1 -gt 30 ]]
7262023-10-12T05:13:39.034Z+ sleep 1
7272023-10-12T05:13:39.059Z+ retry=2
7282023-10-12T05:13:39.084Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7292023-10-12T05:13:58.018Z+ [[ 2 -gt 30 ]]
7302023-10-12T05:13:58.043Z+ sleep 1
7312023-10-12T05:13:59.022Z+ retry=3
7322023-10-12T05:13:59.047Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7332023-10-12T05:13:59.072Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7342023-10-12T05:13:59.097Zlocal v6:
7352023-10-12T05:13:59.122Zfe80::1c45:98ff:fef2:8c8d
7362023-10-12T05:13:59.147Zfd00:99::1
7372023-10-12T05:13:59.173Zlocal v4:
7382023-10-12T05:13:59.205Z10.151.2.174
7392023-10-12T05:13:59.231Zrouter v6:
7402023-10-12T05:13:59.256Zrouter v4:
7412023-10-12T05:13:59.280Z0.0.0.0/0 -> 10.151.2.100 (2) vid=0
7422023-10-12T05:13:59.305Zresolver v4:
7432023-10-12T05:13:59.330Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7442023-10-12T05:13:59.355Zresolver v6:
7452023-10-12T05:13:59.379Zfe80::1c45:98ff:fef2:8c8d -> 1e:45:98:f2:8c:8d
7462023-10-12T05:13:59.404Zfe80::a885:5aff:fe87:c098 -> aa:85:5a:87:c0:98
7472023-10-12T05:13:59.429Znat v4:
7482023-10-12T05:13:59.454Znat_v6:
7492023-10-12T05:13:59.479Zport_mac:
7502023-10-12T05:13:59.504Z1000: a8:40:25:16:0f:8a
7512023-10-12T05:13:59.529Z1: a8:40:25:16:0f:8b
7522023-10-12T05:13:59.554Zproxy arp:
7532023-10-12T05:13:59.579Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7542023-10-12T05:13:59.604Z+ retry=0
7552023-10-12T05:13:59.629Z+++ zoneadm list -n
7562023-10-12T05:13:59.654Z+++ grep oxz_ntp
7572023-10-12T05:13:59.679Z++ pfexec svcs -z -Hostate oxide/ntp
7582023-10-12T05:13:59.703Zsvcs: zone '-Hostate': entity not found
7592023-10-12T05:13:59.728Z++ true
7602023-10-12T05:13:59.752Z+ [[ '' != online ]]
7612023-10-12T05:13:59.777Z+ [[ 0 -gt 60 ]]
7622023-10-12T05:13:59.801Z+ sleep 1
7632023-10-12T05:14:00.088Z+ retry=1
7642023-10-12T05:14:00.113Z+++ zoneadm list -n
7652023-10-12T05:14:00.139Z+++ grep oxz_ntp
7662023-10-12T05:14:00.164Z++ pfexec svcs -z -Hostate oxide/ntp
7672023-10-12T05:14:00.189Zsvcs: zone '-Hostate': entity not found
7682023-10-12T05:14:00.230Z++ true
7692023-10-12T05:14:00.262Z+ [[ '' != online ]]
7702023-10-12T05:14:00.287Z+ [[ 1 -gt 60 ]]
7712023-10-12T05:14:00.311Z+ sleep 1
7722023-10-12T05:14:01.111Z+ retry=2
7732023-10-12T05:14:01.136Z+++ zoneadm list -n
7742023-10-12T05:14:01.160Z+++ grep oxz_ntp
7752023-10-12T05:14:01.185Z++ pfexec svcs -z -Hostate oxide/ntp
7762023-10-12T05:14:01.210Zsvcs: zone '-Hostate': entity not found
7772023-10-12T05:14:01.254Z++ true
7782023-10-12T05:14:01.280Z+ [[ '' != online ]]
7792023-10-12T05:14:01.306Z+ [[ 2 -gt 60 ]]
7802023-10-12T05:14:01.332Z+ sleep 1
7812023-10-12T05:14:02.135Z+ retry=3
7822023-10-12T05:14:02.161Z+++ zoneadm list -n
7832023-10-12T05:14:02.186Z+++ grep oxz_ntp
7842023-10-12T05:14:02.211Z++ pfexec svcs -z -Hostate oxide/ntp
7852023-10-12T05:14:02.237Zsvcs: zone '-Hostate': entity not found
7862023-10-12T05:14:02.281Z++ true
7872023-10-12T05:14:02.306Z+ [[ '' != online ]]
7882023-10-12T05:14:02.332Z+ [[ 3 -gt 60 ]]
7892023-10-12T05:14:02.358Z+ sleep 1
7902023-10-12T05:14:03.164Z+ retry=4
7912023-10-12T05:14:03.190Z+++ zoneadm list -n
7922023-10-12T05:14:03.215Z+++ grep oxz_ntp
7932023-10-12T05:14:03.241Z++ pfexec svcs -z -Hostate oxide/ntp
7942023-10-12T05:14:03.266Zsvcs: zone '-Hostate': entity not found
7952023-10-12T05:14:03.310Z++ true
7962023-10-12T05:14:03.336Z+ [[ '' != online ]]
7972023-10-12T05:14:03.362Z+ [[ 4 -gt 60 ]]
7982023-10-12T05:14:03.393Z+ sleep 1
7992023-10-12T05:14:04.189Z+ retry=5
8002023-10-12T05:14:04.215Z+++ zoneadm list -n
8012023-10-12T05:14:04.240Z+++ grep oxz_ntp
8022023-10-12T05:14:04.265Z++ pfexec svcs -z -Hostate oxide/ntp
8032023-10-12T05:14:04.290Zsvcs: zone '-Hostate': entity not found
8042023-10-12T05:14:04.335Z++ true
8052023-10-12T05:14:04.360Z+ [[ '' != online ]]
8062023-10-12T05:14:04.387Z+ [[ 5 -gt 60 ]]
8072023-10-12T05:14:04.411Z+ sleep 1
8082023-10-12T05:14:05.211Z+ retry=6
8092023-10-12T05:14:05.235Z+++ zoneadm list -n
8102023-10-12T05:14:05.260Z+++ grep oxz_ntp
8112023-10-12T05:14:05.285Z++ pfexec svcs -z -Hostate oxide/ntp
8122023-10-12T05:14:05.309Zsvcs: zone '-Hostate': entity not found
8132023-10-12T05:14:05.334Z++ true
8142023-10-12T05:14:05.361Z+ [[ '' != online ]]
8152023-10-12T05:14:05.386Z+ [[ 6 -gt 60 ]]
8162023-10-12T05:14:05.410Z+ sleep 1
8172023-10-12T05:14:06.231Z+ retry=7
8182023-10-12T05:14:06.256Z+++ zoneadm list -n
8192023-10-12T05:14:06.281Z+++ grep oxz_ntp
8202023-10-12T05:14:06.306Z++ pfexec svcs -z -Hostate oxide/ntp
8212023-10-12T05:14:06.331Zsvcs: zone '-Hostate': entity not found
8222023-10-12T05:14:06.357Z++ true
8232023-10-12T05:14:06.385Z+ [[ '' != online ]]
8242023-10-12T05:14:06.410Z+ [[ 7 -gt 60 ]]
8252023-10-12T05:14:06.434Z+ sleep 1
8262023-10-12T05:14:07.255Z+ retry=8
8272023-10-12T05:14:07.280Z+++ zoneadm list -n
8282023-10-12T05:14:07.304Z+++ grep oxz_ntp
8292023-10-12T05:14:07.329Z++ pfexec svcs -z -Hostate oxide/ntp
8302023-10-12T05:14:07.354Zsvcs: zone '-Hostate': entity not found
8312023-10-12T05:14:07.379Z++ true
8322023-10-12T05:14:07.410Z+ [[ '' != online ]]
8332023-10-12T05:14:07.435Z+ [[ 8 -gt 60 ]]
8342023-10-12T05:14:07.459Z+ sleep 1
8352023-10-12T05:14:08.276Z+ retry=9
8362023-10-12T05:14:08.301Z+++ zoneadm list -n
8372023-10-12T05:14:08.327Z+++ grep oxz_ntp
8382023-10-12T05:14:08.352Z++ pfexec svcs -z -Hostate oxide/ntp
8392023-10-12T05:14:08.377Zsvcs: zone '-Hostate': entity not found
8402023-10-12T05:14:08.401Z++ true
8412023-10-12T05:14:08.435Z+ [[ '' != online ]]
8422023-10-12T05:14:08.459Z+ [[ 9 -gt 60 ]]
8432023-10-12T05:14:08.485Z+ sleep 1
8442023-10-12T05:14:09.297Z+ retry=10
8452023-10-12T05:14:09.322Z+++ zoneadm list -n
8462023-10-12T05:14:09.347Z+++ grep oxz_ntp
8472023-10-12T05:14:09.371Z++ pfexec svcs -z -Hostate oxide/ntp
8482023-10-12T05:14:09.396Zsvcs: zone '-Hostate': entity not found
8492023-10-12T05:14:09.421Z++ true
8502023-10-12T05:14:09.460Z+ [[ '' != online ]]
8512023-10-12T05:14:09.486Z+ [[ 10 -gt 60 ]]
8522023-10-12T05:14:09.511Z+ sleep 1
8532023-10-12T05:14:10.317Z+ retry=11
8542023-10-12T05:14:10.372Z+++ zoneadm list -n
8552023-10-12T05:14:10.397Z+++ grep oxz_ntp
8562023-10-12T05:14:10.422Z++ pfexec svcs -z -Hostate oxide/ntp
8572023-10-12T05:14:10.447Zsvcs: zone '-Hostate': entity not found
8582023-10-12T05:14:10.490Z++ true
8592023-10-12T05:14:10.515Z+ [[ '' != online ]]
8602023-10-12T05:14:10.540Z+ [[ 11 -gt 60 ]]
8612023-10-12T05:14:10.576Z+ sleep 1
8622023-10-12T05:14:11.338Z+ retry=12
8632023-10-12T05:14:11.363Z+++ zoneadm list -n
8642023-10-12T05:14:11.389Z+++ grep oxz_ntp
8652023-10-12T05:14:11.413Z++ pfexec svcs -z -Hostate oxide/ntp
8662023-10-12T05:14:11.438Zsvcs: zone '-Hostate': entity not found
8672023-10-12T05:14:11.463Z++ true
8682023-10-12T05:14:11.487Z+ [[ '' != online ]]
8692023-10-12T05:14:11.514Z+ [[ 12 -gt 60 ]]
8702023-10-12T05:14:11.538Z+ sleep 1
8712023-10-12T05:14:12.362Z+ retry=13
8722023-10-12T05:14:12.386Z+++ zoneadm list -n
8732023-10-12T05:14:12.411Z+++ grep oxz_ntp
8742023-10-12T05:14:12.436Z++ pfexec svcs -z -Hostate oxide/ntp
8752023-10-12T05:14:12.460Zsvcs: zone '-Hostate': entity not found
8762023-10-12T05:14:12.486Z++ true
8772023-10-12T05:14:12.510Z+ [[ '' != online ]]
8782023-10-12T05:14:12.539Z+ [[ 13 -gt 60 ]]
8792023-10-12T05:14:12.563Z+ sleep 1
8802023-10-12T05:14:13.386Z+ retry=14
8812023-10-12T05:14:13.412Z+++ zoneadm list -n
8822023-10-12T05:14:13.437Z+++ grep oxz_ntp
8832023-10-12T05:14:13.462Z++ pfexec svcs -z oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06 -Hostate oxide/ntp
8842023-10-12T05:14:13.486Zsvcs: Could not bind to repository server for zone oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06: repository server unavailable
8852023-10-12T05:14:13.513Z++ true
8862023-10-12T05:14:13.537Z+ [[ '' != online ]]
8872023-10-12T05:14:13.563Z+ [[ 14 -gt 60 ]]
8882023-10-12T05:14:13.588Z+ sleep 1
8892023-10-12T05:14:14.419Z+ retry=15
8902023-10-12T05:14:14.443Z+++ zoneadm list -n
8912023-10-12T05:14:14.468Z+++ grep oxz_ntp
8922023-10-12T05:14:14.493Z++ pfexec svcs -z oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06 -Hostate oxide/ntp
8932023-10-12T05:14:14.517Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8942023-10-12T05:14:14.542Z++ true
8952023-10-12T05:14:14.587Z+ [[ '' != online ]]
8962023-10-12T05:14:14.614Z+ [[ 15 -gt 60 ]]
8972023-10-12T05:14:14.638Z+ sleep 1
8982023-10-12T05:14:15.452Z+ retry=16
8992023-10-12T05:14:15.476Z+++ zoneadm list -n
9002023-10-12T05:14:15.501Z+++ grep oxz_ntp
9012023-10-12T05:14:15.526Z++ pfexec svcs -z oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06 -Hostate oxide/ntp
9022023-10-12T05:14:15.550Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9032023-10-12T05:14:15.575Z++ true
9042023-10-12T05:14:15.613Z+ [[ '' != online ]]
9052023-10-12T05:14:15.638Z+ [[ 16 -gt 60 ]]
9062023-10-12T05:14:15.662Z+ sleep 1
9072023-10-12T05:14:16.481Z+ retry=17
9082023-10-12T05:14:16.506Z+++ zoneadm list -n
9092023-10-12T05:14:16.531Z+++ grep oxz_ntp
9102023-10-12T05:14:16.556Z++ pfexec svcs -z oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06 -Hostate oxide/ntp
9112023-10-12T05:14:16.581Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9122023-10-12T05:14:16.605Z++ true
9132023-10-12T05:14:16.640Z+ [[ '' != online ]]
9142023-10-12T05:14:16.665Z+ [[ 17 -gt 60 ]]
9152023-10-12T05:14:16.690Z+ sleep 1
9162023-10-12T05:14:17.510Z+ retry=18
9172023-10-12T05:14:17.534Z+++ zoneadm list -n
9182023-10-12T05:14:17.559Z+++ grep oxz_ntp
9192023-10-12T05:14:17.583Z++ pfexec svcs -z oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06 -Hostate oxide/ntp
9202023-10-12T05:14:17.608Z+ [[ disabled != online ]]
9212023-10-12T05:14:17.633Z+ [[ 18 -gt 60 ]]
9222023-10-12T05:14:17.663Z+ sleep 1
9232023-10-12T05:14:18.533Z+ retry=19
9242023-10-12T05:14:18.559Z+++ zoneadm list -n
9252023-10-12T05:14:18.583Z+++ grep oxz_ntp
9262023-10-12T05:14:18.608Z++ pfexec svcs -z oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06 -Hostate oxide/ntp
9272023-10-12T05:14:18.633Z+ [[ disabled != online ]]
9282023-10-12T05:14:18.659Z+ [[ 19 -gt 60 ]]
9292023-10-12T05:14:18.691Z+ sleep 1
9302023-10-12T05:14:19.555Z+ retry=20
9312023-10-12T05:14:19.580Z+++ zoneadm list -n
9322023-10-12T05:14:19.605Z+++ grep oxz_ntp
9332023-10-12T05:14:19.629Z++ pfexec svcs -z oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06 -Hostate oxide/ntp
9342023-10-12T05:14:19.654Z+ [[ disabled != online ]]
9352023-10-12T05:14:19.678Z+ [[ 20 -gt 60 ]]
9362023-10-12T05:14:19.715Z+ sleep 1
9372023-10-12T05:14:20.577Z+ retry=21
9382023-10-12T05:14:20.602Z+++ zoneadm list -n
9392023-10-12T05:14:20.627Z+++ grep oxz_ntp
9402023-10-12T05:14:20.652Z++ pfexec svcs -z oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06 -Hostate oxide/ntp
9412023-10-12T05:14:20.676Z+ [[ disabled != online ]]
9422023-10-12T05:14:20.701Z+ [[ 21 -gt 60 ]]
9432023-10-12T05:14:20.739Z+ sleep 1
9442023-10-12T05:14:21.599Z+ retry=22
9452023-10-12T05:14:21.624Z+++ zoneadm list -n
9462023-10-12T05:14:21.649Z+++ grep oxz_ntp
9472023-10-12T05:14:21.674Z++ pfexec svcs -z oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06 -Hostate oxide/ntp
9482023-10-12T05:14:21.698Z+ [[ disabled != online ]]
9492023-10-12T05:14:21.723Z+ [[ 22 -gt 60 ]]
9502023-10-12T05:14:21.765Z+ sleep 1
9512023-10-12T05:14:22.621Z+ retry=23
9522023-10-12T05:14:22.646Z+++ zoneadm list -n
9532023-10-12T05:14:22.670Z+++ grep oxz_ntp
9542023-10-12T05:14:22.695Z++ pfexec svcs -z oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06 -Hostate oxide/ntp
9552023-10-12T05:14:22.741Z+ [[ disabled != online ]]
9562023-10-12T05:14:22.789Z+ [[ 23 -gt 60 ]]
9572023-10-12T05:14:22.814Z+ sleep 1
9582023-10-12T05:14:23.643Z+ retry=24
9592023-10-12T05:14:23.694Z+++ zoneadm list -n
9602023-10-12T05:14:23.718Z+++ grep oxz_ntp
9612023-10-12T05:14:23.743Z++ pfexec svcs -z oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06 -Hostate oxide/ntp
9622023-10-12T05:14:23.768Z+ [[ disabled != online ]]
9632023-10-12T05:14:23.815Z+ [[ 24 -gt 60 ]]
9642023-10-12T05:14:23.840Z+ sleep 1
9652023-10-12T05:14:24.665Z+ retry=25
 
9662023-10-12T05:14:24.677Zsock console login: Oct 12 05:14:24 sok ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9672023-10-12T05:14:24.690Z+++ zoneadm list -n
9682023-10-12T05:14:24.715Z+++ grep oxz_ntp
9692023-10-12T05:14:24.740Z++ pfexec svcs -z oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06 -Hostate oxide/ntp
9702023-10-12T05:14:24.765Z+ [[ disabled != online ]]
9712023-10-12T05:14:24.789Z+ [[ 25 -gt 60 ]]
9722023-10-12T05:14:24.839Z+ sleep 1
9732023-10-12T05:14:25.686Z+ retry=26
9742023-10-12T05:14:25.712Z+++ zoneadm list -n
9752023-10-12T05:14:25.737Z+++ grep oxz_ntp
9762023-10-12T05:14:25.762Z++ pfexec svcs -z oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06 -Hostate oxide/ntp
9772023-10-12T05:14:25.923Z+ [[ online != online ]]
9782023-10-12T05:14:25.948Z+ echo 'Waited for chrony: 26s'
9792023-10-12T05:14:25.974ZWaited for chrony: 26s
9802023-10-12T05:14:25.998Z+ export RUST_BACKTRACE=1
9812023-10-12T05:14:26.024Z+ RUST_BACKTRACE=1
9822023-10-12T05:14:26.048Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9832023-10-12T05:14:26.073Z+ ./tests/bootstrap
9842023-10-12T05:14:26.097ZUsing external DNS server at 10.151.2.164:53
9852023-10-12T05:14:26.121Z2023-10-12 05:14:25.594495185 UTC: attempting to log into API
9862023-10-12T05:14:40.771Z2023-10-12 05:14:41.112773306 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9872023-10-12T05:14:41.772Z2023-10-12 05:14:42.111878530 UTC: attempting to log into API
9882023-10-12T05:14:56.805Z2023-10-12 05:14:57.104695896 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9892023-10-12T05:14:57.807Z2023-10-12 05:14:58.103986002 UTC: attempting to log into API
9902023-10-12T05:15:12.841Z2023-10-12 05:15:13.096485083 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9912023-10-12T05:15:13.864Z2023-10-12 05:15:14.094633451 UTC: attempting to log into API
9922023-10-12T05:15:28.904Z2023-10-12 05:15:29.117593207 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9932023-10-12T05:15:29.924Z2023-10-12 05:15:30.116945734 UTC: attempting to log into API
 
9942023-10-12T05:15:31.461ZOct 12 05:15:31 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9952023-10-12T05:15:38.773ZOct 12 05:15:38 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9962023-10-12T05:15:44.781ZOct 12 05:15:44 sok ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9972023-10-12T05:15:44.939Z2023-10-12 05:15:45.109190432 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9982023-10-12T05:15:45.940Z2023-10-12 05:15:46.107564360 UTC: attempting to log into API
 
9992023-10-12T05:15:51.571ZOct 12 05:15:51 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
10002023-10-12T05:15:58.095ZOct 12 05:15:57 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
10012023-10-12T05:16:00.973Z2023-10-12 05:16:01.100074055 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10022023-10-12T05:16:01.974Z2023-10-12 05:16:02.098453196 UTC: attempting to log into API
10032023-10-12T05:16:17.007Z2023-10-12 05:16:17.090981468 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10042023-10-12T05:16:18.007Z2023-10-12 05:16:18.089195926 UTC: attempting to log into API
10052023-10-12T05:16:33.050Z2023-10-12 05:16:33.082560952 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10062023-10-12T05:16:34.042Z2023-10-12 05:16:34.080672509 UTC: attempting to log into API
10072023-10-12T05:16:44.831Z2023-10-12 05:16:44.843339930 UTC: login succeeded
10082023-10-12T05:16:44.864Zwaiting for nexus to come up...
10092023-10-12T05:16:44.929Zcreating IP pool... 10.151.2.175 - 10.151.2.227
10102023-10-12T05:16:44.983Zensuring datasets are ready...
10112023-10-12T05:16:47.446Zlet's roll.
10122023-10-12T05:16:47.471Z+ rm ./tests/bootstrap
10132023-10-12T05:16:47.514Z+ for test_bin in tests/*
10142023-10-12T05:16:47.539Z+ ./tests/bootstrap-80e9eb4937b4fe98
10152023-10-12T05:16:47.564Z
10162023-10-12T05:16:47.588Zrunning 0 tests
10172023-10-12T05:16:47.613Z
10182023-10-12T05:16:47.638Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10192023-10-12T05:16:47.662Z
10202023-10-12T05:16:47.687Z+ for test_bin in tests/*
10212023-10-12T05:16:47.711Z+ ./tests/end_to_end_tests-9071c6bef4f9a767
10222023-10-12T05:16:47.735Z
10232023-10-12T05:16:47.760Zrunning 2 tests
10242023-10-12T05:16:47.785Ztest no_spoof::no_spoof_here ... ok
10252023-10-12T05:17:47.547Ztest instance_launch::instance_launch has been running for over 60 seconds
10262023-10-12T05:19:40.239Ztest instance_launch::instance_launch ... ok
10272023-10-12T05:19:40.264Z
10282023-10-12T05:19:40.289Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 172.70s
10292023-10-12T05:19:40.313Z
10302023-10-12T05:19:40.338Z+ _exit_trap
10312023-10-12T05:19:40.362Z+ local status=0
10322023-10-12T05:19:40.386Z+ [[ 0 -eq 0 ]]
10332023-10-12T05:19:40.411Z+ exit 0
10342023-10-12T05:19:40.436Zprocess exited: duration 398706 ms, exit code 0
 
10352023-10-12T05:19:40.518Zfound 56 output files
10362023-10-12T05:19:40.569Zuploading: /var/svc/log/oxide-sled-agent:default.log (420034 bytes)
10372023-10-12T05:19:41.808Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10382023-10-12T05:19:41.834Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9586546a-e103-4cd1-bc14-284246443667/root/var/svc/log/oxide-cockroachdb:default.log (861 bytes)
10392023-10-12T05:19:42.905Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9586546a-e103-4cd1-bc14-284246443667/root/var/svc/log/oxide-cockroachdb:default.log
10402023-10-12T05:19:42.930Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9586546a-e103-4cd1-bc14-284246443667/root/var/svc/log/oxide-cockroachdb:default.log.0 (4508 bytes)
10412023-10-12T05:19:44.012Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9586546a-e103-4cd1-bc14-284246443667/root/var/svc/log/oxide-cockroachdb:default.log.0
10422023-10-12T05:19:44.092Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e536e21f-abf5-457b-8d8f-94ca0675fb2e/root/var/svc/log/oxide-cockroachdb:default.log (858 bytes)
10432023-10-12T05:19:45.148Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e536e21f-abf5-457b-8d8f-94ca0675fb2e/root/var/svc/log/oxide-cockroachdb:default.log
10442023-10-12T05:19:45.176Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e536e21f-abf5-457b-8d8f-94ca0675fb2e/root/var/svc/log/oxide-cockroachdb:default.log.0 (4508 bytes)
10452023-10-12T05:19:46.243Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e536e21f-abf5-457b-8d8f-94ca0675fb2e/root/var/svc/log/oxide-cockroachdb:default.log.0
10462023-10-12T05:19:46.268Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b5ab335d-ce22-48d9-ad09-82e40369c89d/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10472023-10-12T05:19:47.338Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b5ab335d-ce22-48d9-ad09-82e40369c89d/root/var/svc/log/oxide-crucible-agent:default.log
10482023-10-12T05:19:47.363Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b5ab335d-ce22-48d9-ad09-82e40369c89d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10492023-10-12T05:19:48.433Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b5ab335d-ce22-48d9-ad09-82e40369c89d/root/var/svc/log/oxide-crucible-downstairs:default.log
10502023-10-12T05:19:48.457Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_cfb033e7-aea2-4020-a5f6-cf0617d2eb0f/root/var/svc/log/oxide-cockroachdb:default.log (861 bytes)
10512023-10-12T05:19:49.528Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_cfb033e7-aea2-4020-a5f6-cf0617d2eb0f/root/var/svc/log/oxide-cockroachdb:default.log
10522023-10-12T05:19:49.552Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_cfb033e7-aea2-4020-a5f6-cf0617d2eb0f/root/var/svc/log/oxide-cockroachdb:default.log.0 (4508 bytes)
10532023-10-12T05:19:50.624Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_cfb033e7-aea2-4020-a5f6-cf0617d2eb0f/root/var/svc/log/oxide-cockroachdb:default.log.0
10542023-10-12T05:19:50.648Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5611f974-b1e3-49d6-a5e3-27412f73f4db/root/var/svc/log/oxide-crucible-agent:default.log (14582 bytes)
10552023-10-12T05:19:51.741Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5611f974-b1e3-49d6-a5e3-27412f73f4db/root/var/svc/log/oxide-crucible-agent:default.log
10562023-10-12T05:19:51.765Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5611f974-b1e3-49d6-a5e3-27412f73f4db/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10572023-10-12T05:19:52.837Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5611f974-b1e3-49d6-a5e3-27412f73f4db/root/var/svc/log/oxide-crucible-downstairs:default.log
10582023-10-12T05:19:52.861Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5611f974-b1e3-49d6-a5e3-27412f73f4db/root/var/svc/log/oxide-crucible-downstairs:downstairs-7b9e7e79-ab0c-4c31-9c0c-7e7ccf902a81.log (3494 bytes)
10592023-10-12T05:19:53.932Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5611f974-b1e3-49d6-a5e3-27412f73f4db/root/var/svc/log/oxide-crucible-downstairs:downstairs-7b9e7e79-ab0c-4c31-9c0c-7e7ccf902a81.log
10602023-10-12T05:19:53.961Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b32c5f8b-6d91-413a-b97f-6cf0b050fd44/root/var/svc/log/oxide-crucible-agent:default.log (24824 bytes)
10612023-10-12T05:19:55.072Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b32c5f8b-6d91-413a-b97f-6cf0b050fd44/root/var/svc/log/oxide-crucible-agent:default.log
10622023-10-12T05:19:55.096Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b32c5f8b-6d91-413a-b97f-6cf0b050fd44/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10632023-10-12T05:19:56.169Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b32c5f8b-6d91-413a-b97f-6cf0b050fd44/root/var/svc/log/oxide-crucible-downstairs:default.log
10642023-10-12T05:19:56.193Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b32c5f8b-6d91-413a-b97f-6cf0b050fd44/root/var/svc/log/oxide-crucible-downstairs:downstairs-7a5c18ff-ad5f-40d5-b994-ac4ab9570fe9.log (5963 bytes)
10652023-10-12T05:19:57.263Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b32c5f8b-6d91-413a-b97f-6cf0b050fd44/root/var/svc/log/oxide-crucible-downstairs:downstairs-7a5c18ff-ad5f-40d5-b994-ac4ab9570fe9.log
10662023-10-12T05:19:57.287Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b32c5f8b-6d91-413a-b97f-6cf0b050fd44/root/var/svc/log/oxide-crucible-downstairs:downstairs-97ef73e5-4e19-45ad-a2d0-e35f7596f0f3.log (3494 bytes)
10672023-10-12T05:19:58.357Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b32c5f8b-6d91-413a-b97f-6cf0b050fd44/root/var/svc/log/oxide-crucible-downstairs:downstairs-97ef73e5-4e19-45ad-a2d0-e35f7596f0f3.log
10682023-10-12T05:19:58.381Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_6e928ec8-a244-401b-b4b7-47efaa2de505/root/var/svc/log/oxide-external_dns:default.log (15607 bytes)
10692023-10-12T05:19:59.474Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_6e928ec8-a244-401b-b4b7-47efaa2de505/root/var/svc/log/oxide-external_dns:default.log
10702023-10-12T05:19:59.498Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_0e9ebf20-3d9f-449f-be4e-4c22334da8f2/root/var/svc/log/oxide-internal_dns:default.log (93570 bytes)
10712023-10-12T05:19:59.632Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_0e9ebf20-3d9f-449f-be4e-4c22334da8f2/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93570 -> 97326
10722023-10-12T05:20:00.702Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_0e9ebf20-3d9f-449f-be4e-4c22334da8f2/root/var/svc/log/oxide-internal_dns:default.log
10732023-10-12T05:20:00.726Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_0e9ebf20-3d9f-449f-be4e-4c22334da8f2/root/var/svc/log/oxide-internal_dns:default.log.0 (18920 bytes)
10742023-10-12T05:20:01.847Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_0e9ebf20-3d9f-449f-be4e-4c22334da8f2/root/var/svc/log/oxide-internal_dns:default.log.0
10752023-10-12T05:20:01.872Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06/root/var/svc/log/oxide-ntp:default.log (204 bytes)
10762023-10-12T05:20:02.942Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06/root/var/svc/log/oxide-ntp:default.log
10772023-10-12T05:20:02.966Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06/root/var/svc/log/oxide-ntp:default.log.0 (1176 bytes)
10782023-10-12T05:20:04.038Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06/root/var/svc/log/oxide-ntp:default.log.0
10792023-10-12T05:20:04.063Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_17f79fd9-21e6-4fc0-9721-2e394a946282/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10802023-10-12T05:20:05.135Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_17f79fd9-21e6-4fc0-9721-2e394a946282/root/var/svc/log/oxide-crucible-agent:default.log
10812023-10-12T05:20:05.159Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_17f79fd9-21e6-4fc0-9721-2e394a946282/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10822023-10-12T05:20:06.230Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_17f79fd9-21e6-4fc0-9721-2e394a946282/root/var/svc/log/oxide-crucible-downstairs:default.log
10832023-10-12T05:20:06.255Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_d6db5d21-640d-4baa-a338-d3d671bd9b0d/root/var/svc/log/oxide-crucible-agent:default.log (14149 bytes)
10842023-10-12T05:20:07.356Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_d6db5d21-640d-4baa-a338-d3d671bd9b0d/root/var/svc/log/oxide-crucible-agent:default.log
10852023-10-12T05:20:07.380Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_d6db5d21-640d-4baa-a338-d3d671bd9b0d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10862023-10-12T05:20:08.451Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_d6db5d21-640d-4baa-a338-d3d671bd9b0d/root/var/svc/log/oxide-crucible-downstairs:default.log
10872023-10-12T05:20:08.476Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_d6db5d21-640d-4baa-a338-d3d671bd9b0d/root/var/svc/log/oxide-crucible-downstairs:downstairs-cc98ae71-5c2a-4065-abd9-6b27aea5fccb.log (5965 bytes)
10882023-10-12T05:20:09.546Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_d6db5d21-640d-4baa-a338-d3d671bd9b0d/root/var/svc/log/oxide-crucible-downstairs:downstairs-cc98ae71-5c2a-4065-abd9-6b27aea5fccb.log
10892023-10-12T05:20:09.570Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_39010a2b-d7a4-405b-b777-bb2e031691e2/root/var/svc/log/oxide-internal_dns:default.log (93576 bytes)
10902023-10-12T05:20:09.704Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_39010a2b-d7a4-405b-b777-bb2e031691e2/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93576 -> 97331
10912023-10-12T05:20:10.774Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_39010a2b-d7a4-405b-b777-bb2e031691e2/root/var/svc/log/oxide-internal_dns:default.log
10922023-10-12T05:20:10.799Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_39010a2b-d7a4-405b-b777-bb2e031691e2/root/var/svc/log/oxide-internal_dns:default.log.0 (20136 bytes)
10932023-10-12T05:20:11.914Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_39010a2b-d7a4-405b-b777-bb2e031691e2/root/var/svc/log/oxide-internal_dns:default.log.0
10942023-10-12T05:20:11.948Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_6e27f057-6352-40be-9eed-c5a12c05b555/root/var/svc/log/oxide-nexus:default.log (4486978 bytes)
10952023-10-12T05:20:12.794Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_6e27f057-6352-40be-9eed-c5a12c05b555/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4486978 -> 4546981
10962023-10-12T05:20:13.863Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_6e27f057-6352-40be-9eed-c5a12c05b555/root/var/svc/log/oxide-nexus:default.log
10972023-10-12T05:20:13.887Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_60bafed4-6295-4b72-bd1e-3840a990bba4/root/var/svc/log/oxide-oximeter:default.log (197413 bytes)
10982023-10-12T05:20:14.044Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_60bafed4-6295-4b72-bd1e-3840a990bba4/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 197413 -> 223491
10992023-10-12T05:20:15.114Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_60bafed4-6295-4b72-bd1e-3840a990bba4/root/var/svc/log/oxide-oximeter:default.log
11002023-10-12T05:20:15.138Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4dab3592-1eb9-4868-9088-6dcce3c4b261/root/var/svc/log/oxide-crucible-agent:default.log (14579 bytes)
11012023-10-12T05:20:16.230Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4dab3592-1eb9-4868-9088-6dcce3c4b261/root/var/svc/log/oxide-crucible-agent:default.log
11022023-10-12T05:20:16.254Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4dab3592-1eb9-4868-9088-6dcce3c4b261/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11032023-10-12T05:20:17.323Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4dab3592-1eb9-4868-9088-6dcce3c4b261/root/var/svc/log/oxide-crucible-downstairs:default.log
11042023-10-12T05:20:17.347Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4dab3592-1eb9-4868-9088-6dcce3c4b261/root/var/svc/log/oxide-crucible-downstairs:downstairs-cebbfb60-e79b-4e96-9329-43cbfe1e3d87.log (5960 bytes)
11052023-10-12T05:20:18.417Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4dab3592-1eb9-4868-9088-6dcce3c4b261/root/var/svc/log/oxide-crucible-downstairs:downstairs-cebbfb60-e79b-4e96-9329-43cbfe1e3d87.log
11062023-10-12T05:20:18.441Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_78e43848-99a3-4289-b742-33833a82e058/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
11072023-10-12T05:20:19.511Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_78e43848-99a3-4289-b742-33833a82e058/root/var/svc/log/oxide-crucible-pantry:default.log
11082023-10-12T05:20:19.535Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_187f0e5d-5690-4e36-8595-139ef5f9a815/root/var/svc/log/oxide-crucible-agent:default.log (14578 bytes)
11092023-10-12T05:20:20.631Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_187f0e5d-5690-4e36-8595-139ef5f9a815/root/var/svc/log/oxide-crucible-agent:default.log
11102023-10-12T05:20:20.655Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_187f0e5d-5690-4e36-8595-139ef5f9a815/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11112023-10-12T05:20:21.726Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_187f0e5d-5690-4e36-8595-139ef5f9a815/root/var/svc/log/oxide-crucible-downstairs:default.log
11122023-10-12T05:20:21.749Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_187f0e5d-5690-4e36-8595-139ef5f9a815/root/var/svc/log/oxide-crucible-downstairs:downstairs-8ce35373-81e0-4f3f-9be3-e60fd8e4e00a.log (3495 bytes)
11132023-10-12T05:20:22.819Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_187f0e5d-5690-4e36-8595-139ef5f9a815/root/var/svc/log/oxide-crucible-downstairs:downstairs-8ce35373-81e0-4f3f-9be3-e60fd8e4e00a.log
11142023-10-12T05:20:22.843Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_8f68e1c4-f7aa-4ca1-95d3-8749ed4db382/root/var/svc/log/oxide-crucible-pantry:default.log (2332 bytes)
11152023-10-12T05:20:23.913Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_8f68e1c4-f7aa-4ca1-95d3-8749ed4db382/root/var/svc/log/oxide-crucible-pantry:default.log
11162023-10-12T05:20:23.938Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_13b544d5-b97d-4a29-b521-1fa735e5f048/root/var/svc/log/oxide-internal_dns:default.log (93563 bytes)
11172023-10-12T05:20:24.048Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_13b544d5-b97d-4a29-b521-1fa735e5f048/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93563 -> 97319
11182023-10-12T05:20:25.119Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_13b544d5-b97d-4a29-b521-1fa735e5f048/root/var/svc/log/oxide-internal_dns:default.log
11192023-10-12T05:20:25.143Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_13b544d5-b97d-4a29-b521-1fa735e5f048/root/var/svc/log/oxide-internal_dns:default.log.0 (20314 bytes)
11202023-10-12T05:20:26.255Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_13b544d5-b97d-4a29-b521-1fa735e5f048/root/var/svc/log/oxide-internal_dns:default.log.0
11212023-10-12T05:20:26.279Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_a08c93eb-8871-4002-b3c5-7df5a5219249/root/var/svc/log/oxide-nexus:default.log (540231 bytes)
11222023-10-12T05:20:26.508Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_a08c93eb-8871-4002-b3c5-7df5a5219249/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 540231 -> 600845
11232023-10-12T05:20:27.577Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_a08c93eb-8871-4002-b3c5-7df5a5219249/root/var/svc/log/oxide-nexus:default.log
11242023-10-12T05:20:27.601Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_397882cc-0720-471f-b8a2-389d8007da26/root/var/svc/log/oxide-cockroachdb:default.log (833 bytes)
11252023-10-12T05:20:28.685Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_397882cc-0720-471f-b8a2-389d8007da26/root/var/svc/log/oxide-cockroachdb:default.log
11262023-10-12T05:20:28.709Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_397882cc-0720-471f-b8a2-389d8007da26/root/var/svc/log/oxide-cockroachdb:default.log.0 (4508 bytes)
11272023-10-12T05:20:29.763Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_397882cc-0720-471f-b8a2-389d8007da26/root/var/svc/log/oxide-cockroachdb:default.log.0
11282023-10-12T05:20:29.789Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_e36e1796-1a76-4994-bbd1-a9fee49c74e7/root/var/svc/log/oxide-clickhouse:default.log (711676 bytes)
11292023-10-12T05:20:30.190Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_e36e1796-1a76-4994-bbd1-a9fee49c74e7/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 711676 -> 874985
11302023-10-12T05:20:31.261Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_e36e1796-1a76-4994-bbd1-a9fee49c74e7/root/var/svc/log/oxide-clickhouse:default.log
11312023-10-12T05:20:31.289Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f93cf421-7e99-4e8e-8e01-16a01d3fa14c/root/var/svc/log/oxide-cockroachdb:default.log (858 bytes)
11322023-10-12T05:20:32.357Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f93cf421-7e99-4e8e-8e01-16a01d3fa14c/root/var/svc/log/oxide-cockroachdb:default.log
11332023-10-12T05:20:32.381Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f93cf421-7e99-4e8e-8e01-16a01d3fa14c/root/var/svc/log/oxide-cockroachdb:default.log.0 (4508 bytes)
11342023-10-12T05:20:33.451Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f93cf421-7e99-4e8e-8e01-16a01d3fa14c/root/var/svc/log/oxide-cockroachdb:default.log.0
11352023-10-12T05:20:33.475Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1aa0ec34-4abc-442a-b25d-65ddc29340bb/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
11362023-10-12T05:20:34.545Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1aa0ec34-4abc-442a-b25d-65ddc29340bb/root/var/svc/log/oxide-crucible-agent:default.log
11372023-10-12T05:20:34.571Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1aa0ec34-4abc-442a-b25d-65ddc29340bb/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11382023-10-12T05:20:35.641Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1aa0ec34-4abc-442a-b25d-65ddc29340bb/root/var/svc/log/oxide-crucible-downstairs:default.log
11392023-10-12T05:20:35.666Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c6599f-21fd-4ba9-87c8-925c8d20b7b7/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
11402023-10-12T05:20:36.734Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c6599f-21fd-4ba9-87c8-925c8d20b7b7/root/var/svc/log/oxide-crucible-agent:default.log
11412023-10-12T05:20:36.758Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c6599f-21fd-4ba9-87c8-925c8d20b7b7/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11422023-10-12T05:20:37.827Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c6599f-21fd-4ba9-87c8-925c8d20b7b7/root/var/svc/log/oxide-crucible-downstairs:default.log
11432023-10-12T05:20:37.852Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_55337c30-c6d4-4aa1-8406-b141a4e1f5a3/root/var/svc/log/oxide-crucible-pantry:default.log (2336 bytes)
11442023-10-12T05:20:38.921Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_55337c30-c6d4-4aa1-8406-b141a4e1f5a3/root/var/svc/log/oxide-crucible-pantry:default.log
11452023-10-12T05:20:38.945Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_7172b75c-1a2a-4314-8c8f-ee4dbaa870d7/root/var/svc/log/oxide-external_dns:default.log (15612 bytes)
11462023-10-12T05:20:38.989Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_7172b75c-1a2a-4314-8c8f-ee4dbaa870d7/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15612 -> 19389
11472023-10-12T05:20:40.059Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_7172b75c-1a2a-4314-8c8f-ee4dbaa870d7/root/var/svc/log/oxide-external_dns:default.log
11482023-10-12T05:20:40.102Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_ffc5108d-098c-4320-b17e-d331d7164868/root/var/svc/log/oxide-nexus:default.log (544624 bytes)
11492023-10-12T05:20:40.312Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_ffc5108d-098c-4320-b17e-d331d7164868/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 544624 -> 606445
11502023-10-12T05:20:41.382Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_ffc5108d-098c-4320-b17e-d331d7164868/root/var/svc/log/oxide-nexus:default.log
11512023-10-12T05:20:41.406Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06/root/var/log/chrony/measurements.log (12056 bytes)
11522023-10-12T05:20:41.450Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06/root/var/log/chrony/measurements.log" changed size mid upload: 12056 -> 13426
11532023-10-12T05:20:42.520Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06/root/var/log/chrony/measurements.log
11542023-10-12T05:20:42.544Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06/root/var/log/chrony/statistics.log (5265 bytes)
11552023-10-12T05:20:42.568Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06/root/var/log/chrony/statistics.log" changed size mid upload: 5265 -> 6318
11562023-10-12T05:20:43.636Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06/root/var/log/chrony/statistics.log
11572023-10-12T05:20:43.660Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06/root/var/log/chrony/tracking.log (1981 bytes)
11582023-10-12T05:20:43.710Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06/root/var/log/chrony/tracking.log" changed size mid upload: 1981 -> 2113
11592023-10-12T05:20:44.752Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_9caf1d79-5fd2-4d80-bf25-de5fdfc09d06/root/var/log/chrony/tracking.log