01HF7Z2VRD3G2WZKTVZ8R9WV1B: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HF7Z4CZJ5MVTKN3ADWJ2Y1YQ

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-14T23:19:00.933Zjob dependencies complete; ready to run (waiting for 51 m 42 s)
22023-11-14T23:19:50.837Zjob assigned to worker 01HF8238YFJ2W0KVH8PYDDQ8YK (queued for 49 s)
32023-11-14T23:19:50.938Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-11-14T23:19:51.003Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-11-14T23:19:51.048Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-11-14T23:19:51.093Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-14T23:19:51.138Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-11-14T23:19:51.194ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-14T23:19:51.242Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-14T23:19:51.283ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-14T23:19:51.327ZConfiguring device.
122023-11-14T23:19:51.372Z#################################################
132023-11-14T23:19:51.418Z#### ############# ##
142023-11-14T23:19:51.480Z### ### ############ ##
152023-11-14T23:19:51.539Z## ### # ## ### ## Oxide Computer Company ##
162023-11-14T23:19:51.584Z## ## ## ### # ### ##
172023-11-14T23:19:51.629Z## # ### #### #### This Station Under ##
182023-11-14T23:19:51.680Z### ### #### # ### Computer Control ##
192023-11-14T23:19:52.140Z#### #### ### ## ##
202023-11-14T23:19:52.192Z##################################################
212023-11-14T23:19:52.238ZHostname: buskin
222023-11-14T23:19:52.282ZLoading smf(7) service descriptions:
232023-11-14T23:19:53.420Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-14470ed2888247c6.gz
242023-11-14T23:19:55.177Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-14470ed2888247c6.gz
252023-11-14T23:19:55.210Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-14T23:20:23.689Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-14T23:20:24.125Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-0f2fa1e5548e7ba9.gz
282023-11-14T23:20:56.316Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-0f2fa1e5548e7ba9.gz
292023-11-14T23:20:56.356Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-14T23:20:58.078Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-14T23:20:58.112Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-14T23:21:02.933Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-14T23:21:02.960Zdownloading input: /input/package/work/version.txt
342023-11-14T23:21:02.992Zdownloaded input: /input/package/work/version.txt
352023-11-14T23:21:03.028Zdownloading input: /input/package/work/package.tar.gz
362023-11-14T23:21:38.759Zdownloaded input: /input/package/work/package.tar.gz
372023-11-14T23:21:38.940Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-14T23:22:58.325Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-14T23:22:58.845Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-14T23:23:12.346Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-14T23:23:12.793Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-14T23:24:08.525Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-14T23:24:08.561Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-14T23:24:57.816Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-14T23:24:58.953Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-14T23:25:57.667Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-14T23:25:57.693Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-14T23:26:01.993Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-14T23:26:02.044Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-14T23:26:11.391Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-14T23:26:11.421Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-14T23:26:16.704Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-14T23:26:16.729Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-14T23:26:21.547Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-14T23:26:21.571Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-14T23:26:21.596Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-14T23:26:21.626Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-14T23:26:25.232Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-14T23:26:25.257Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-14T23:26:29.517Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-14T23:26:29.543Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-14T23:26:46.743Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-14T23:26:46.768Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-14T23:27:01.776Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-14T23:27:01.802Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-14T23:27:01.826Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-14T23:27:01.850Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-14T23:27:09.011Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-14T23:27:09.036Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-14T23:27:23.822Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-14T23:27:23.846Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-14T23:28:20.904Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-14T23:28:20.928Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-14T23:29:14.318Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-14T23:29:14.935Zstarting task 0: "setup"
762023-11-14T23:29:15.124Z++ uname -s
772023-11-14T23:29:15.333Z+ kern=SunOS
782023-11-14T23:29:15.477Z+ case "$kern" in
792023-11-14T23:29:15.623Z+ groupadd -g 12345 build
802023-11-14T23:29:15.734Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-14T23:29:17.032Z+ zfs create -o mountpoint=/work rpool/work
822023-11-14T23:29:17.193Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-14T23:29:17.300Z+ home_fs=zfs
842023-11-14T23:29:17.354Z+ [[ zfs == autofs ]]
852023-11-14T23:29:17.406Z+ mkdir -p /home/build
862023-11-14T23:29:17.482Z+ chown build:build /home/build /work
872023-11-14T23:29:19.045Z+ chmod 0700 /home/build /work
882023-11-14T23:29:19.152Zprocess exited: duration 4044 ms, exit code 0
 
892023-11-14T23:29:19.266Zstarting task 1: "authentication"
902023-11-14T23:29:19.383Zprocess exited: duration 77 ms, exit code 0
 
912023-11-14T23:29:19.501Zstarting task 2: "build"
922023-11-14T23:29:19.555Z+ trap _exit_trap EXIT
932023-11-14T23:29:19.591Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-14T23:29:19.747Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-14T23:29:19.775Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-14T23:29:19.810Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-14T23:29:19.842Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-14T23:29:19.875Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-14T23:29:19.901Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-14T23:29:19.926Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-14T23:29:19.951Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-14T23:29:20.007Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-14T23:29:20.046Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-14T23:29:20.085Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-14T23:29:20.138Z+ [[ -d /opt/oxide ]]
1062023-11-14T23:29:20.176Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-14T23:29:20.212Z+ pfexec mkdir /opt/oxide
1082023-11-14T23:29:20.240Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-14T23:29:20.307Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-14T23:29:20.341Z+ cd /opt/oxide-underneath
1112023-11-14T23:29:20.378Z+ pfexec tar ceEp@/f - .
1122023-11-14T23:29:20.407Z+ cd /opt/oxide
1132023-11-14T23:29:20.432Z+ pfexec tar xveEp@/f -
1142023-11-14T23:29:20.456Zx ., 0 bytes, 0 tape blocks
1152023-11-14T23:29:20.482Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-14T23:29:20.506Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-14T23:29:20.536Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-14T23:29:20.587Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-14T23:29:20.612Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-14T23:29:20.637Z+ pfexec mkdir /opt/oxide/work
1212023-11-14T23:29:20.661Z+ pfexec chown build:build /opt/oxide/work
1222023-11-14T23:29:20.692Z+ cd /opt/oxide/work
1232023-11-14T23:29:20.719Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-14T23:29:21.459ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-14T23:29:21.484Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1262023-11-14T23:29:21.509Zx out/omicron-sled-agent.tar, 111582720 bytes, 217935 tape blocks
1272023-11-14T23:29:21.533Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-14T23:29:21.972Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-11-14T23:29:21.999Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-11-14T23:29:22.023Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-11-14T23:29:22.046Zx target/release/omicron-package, 42423568 bytes, 82859 tape blocks
1322023-11-14T23:29:22.075Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-11-14T23:29:22.099Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-14T23:29:22.125Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-14T23:29:22.149Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-14T23:29:22.173Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-14T23:29:22.604Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-14T23:29:22.630Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-14T23:29:22.667Z
1402023-11-14T23:29:22.691Zreal 1.552238824
1412023-11-14T23:29:22.716Zuser 1.397112606
1422023-11-14T23:29:22.740Zsys 0.272169151
1432023-11-14T23:29:22.766Ztrap 0.000055780
1442023-11-14T23:29:22.801Ztflt 0.000000000
1452023-11-14T23:29:22.832Zdflt 0.000000000
1462023-11-14T23:29:22.859Zkflt 0.000000000
1472023-11-14T23:29:22.884Zlock 0.000000000
1482023-11-14T23:29:22.908Zslp 1.398574122
1492023-11-14T23:29:22.932Zlat 0.034714543
1502023-11-14T23:29:22.956Zstop 0.000118341
1512023-11-14T23:29:22.980Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-14T23:29:23.007Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-14T23:29:23.031Z+ mkdir tests
1542023-11-14T23:29:23.056Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-14T23:29:23.082Z+ ptime -m gunzip
1562023-11-14T23:29:23.108Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-14470ed2888247c6
1572023-11-14T23:29:23.132Z
1582023-11-14T23:29:23.156Zreal 0.140354463
1592023-11-14T23:29:23.180Zuser 0.127907223
1602023-11-14T23:29:23.205Zsys 0.012228776
1612023-11-14T23:29:23.235Ztrap 0.000010900
1622023-11-14T23:29:23.266Ztflt 0.000000000
1632023-11-14T23:29:23.291Zdflt 0.000000000
1642023-11-14T23:29:23.315Zkflt 0.000000000
1652023-11-14T23:29:23.339Zlock 0.000000000
1662023-11-14T23:29:23.384Zslp 0.000019350
1672023-11-14T23:29:23.434Zlat 0.000014830
1682023-11-14T23:29:23.458Zstop 0.000116031
1692023-11-14T23:29:23.482Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-14470ed2888247c6
1702023-11-14T23:29:23.506Z+ chmod a+x tests/bootstrap-14470ed2888247c6
1712023-11-14T23:29:23.530Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-14T23:29:23.554Z+ ptime -m gunzip
1732023-11-14T23:29:23.578Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-14T23:29:24.693Z
1752023-11-14T23:29:24.717Zreal 1.974454204
1762023-11-14T23:29:24.741Zuser 1.863536251
1772023-11-14T23:29:24.764Zsys 0.110563380
1782023-11-14T23:29:24.789Ztrap 0.000136491
1792023-11-14T23:29:24.813Ztflt 0.000000000
1802023-11-14T23:29:24.837Zdflt 0.000000000
1812023-11-14T23:29:24.861Zkflt 0.000000000
1822023-11-14T23:29:24.891Zlock 0.000000000
1832023-11-14T23:29:24.919Zslp 0.000016560
1842023-11-14T23:29:24.943Zlat 0.000026800
1852023-11-14T23:29:24.968Zstop 0.000107191
1862023-11-14T23:29:24.992Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-14T23:29:25.018Z+ chmod a+x tests/bootstrap
1882023-11-14T23:29:25.042Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-14T23:29:25.068Z+ ptime -m gunzip
1902023-11-14T23:29:25.094Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-0f2fa1e5548e7ba9
1912023-11-14T23:29:26.965Z
1922023-11-14T23:29:26.989Zreal 2.260494970
1932023-11-14T23:29:27.014Zuser 2.133267633
1942023-11-14T23:29:27.038Zsys 0.126814602
1952023-11-14T23:29:27.062Ztrap 0.000182453
1962023-11-14T23:29:27.092Ztflt 0.000000000
1972023-11-14T23:29:27.125Zdflt 0.000000000
1982023-11-14T23:29:27.152Zkflt 0.000000000
1992023-11-14T23:29:27.178Zlock 0.000000000
2002023-11-14T23:29:27.203Zslp 0.000021390
2012023-11-14T23:29:27.233Zlat 0.000021150
2022023-11-14T23:29:27.261Zstop 0.000123771
2032023-11-14T23:29:27.291Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-0f2fa1e5548e7ba9
2042023-11-14T23:29:27.322Z+ chmod a+x tests/end_to_end_tests-0f2fa1e5548e7ba9
2052023-11-14T23:29:27.352Z+ bmat address ls
2062023-11-14T23:29:27.388ZNAME CIDR FIRST LAST COUNT
2072023-11-14T23:29:27.414Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-11-14T23:29:27.439Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-14T23:29:27.488Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-11-14T23:29:27.513Z+ EXTRA_IP_START=10.151.1.164
2112023-11-14T23:29:27.538Z+ EXTRA_IP_END=10.151.1.227
2122023-11-14T23:29:27.562Z+ EXTRA_IP_COUNT=64
2132023-11-14T23:29:27.586Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-14T23:29:27.610Z+ EXTRA_IP_BASE=10.151.1
2152023-11-14T23:29:27.634Z+ EXTRA_IP_FOCTET=164
2162023-11-14T23:29:27.658Z+ EXTRA_IP_LOCTET=227
2172023-11-14T23:29:27.682Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-11-14T23:29:27.706Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-11-14T23:29:27.730Z+ DNS_IP1=10.151.1.164
2202023-11-14T23:29:27.754Z+ DNS_IP2=10.151.1.165
2212023-11-14T23:29:27.781Z+ UPLINK_IP=10.151.1.174
2222023-11-14T23:29:27.813Z+ IPPOOL_START=10.151.1.175
2232023-11-14T23:29:27.838Z+ IPPOOL_END=10.151.1.227
2242023-11-14T23:29:27.861Z++ ipadm show-addr -po type,addr
2252023-11-14T23:29:27.886Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-14T23:29:27.910Z+ GATEWAY_IP=10.151.1.100
2272023-11-14T23:29:27.934Z+ [[ -n 10.151.1.100 ]]
2282023-11-14T23:29:27.998Z+ ping -s 10.151.1.100 56 1
2292023-11-14T23:29:28.024ZPING 10.151.1.100: 56 data bytes
2302023-11-14T23:29:28.048Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
2312023-11-14T23:29:28.073Z
2322023-11-14T23:29:28.097Z----10.151.1.100 PING Statistics----
2332023-11-14T23:29:28.121Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-14T23:29:28.144Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-11-14T23:29:28.168Z++ arp -an
2362023-11-14T23:29:28.193Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-11-14T23:29:28.217Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-11-14T23:29:28.242Z+ routeadm -e ipv4-forwarding -u
2392023-11-14T23:29:28.267Z+ PXA_START=10.151.1.164
2402023-11-14T23:29:28.295Z+ PXA_END=10.151.1.227
2412023-11-14T23:29:28.323Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-14T23:29:28.348Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-14T23:29:28.379Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-14T23:29:28.405Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-14T23:29:28.432Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-14T23:29:28.464Z++ cd ./tools
2472023-11-14T23:29:28.516Z++ pwd
2482023-11-14T23:29:28.586Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-14T23:29:28.617Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-14T23:29:29.077Z+ SOFTNPU_MODE=zone
2512023-11-14T23:29:29.102Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-14T23:29:29.137Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-14T23:29:29.170Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-14T23:29:29.197Z++ dladm show-phys -p -o LINK
2552023-11-14T23:29:29.226Z++ head -1
2562023-11-14T23:29:29.253Z+ PHYSICAL_LINK=igb0
2572023-11-14T23:29:29.278Z+ echo 'Using igb0 as physical link'
2582023-11-14T23:29:29.302ZUsing igb0 as physical link
2592023-11-14T23:29:29.326Z+ ensure_run_as_root
2602023-11-14T23:29:29.350Z++ id -u
2612023-11-14T23:29:29.374Z+ [[ 0 -ne 0 ]]
2622023-11-14T23:29:29.398Z+ ensure_zpools
2632023-11-14T23:29:29.422Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-14T23:29:29.446Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-11-14T23:29:29.472Z+ readarray -t ZPOOLS
2662023-11-14T23:29:29.496Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-11-14T23:29:29.555Z++ sed 's/[ ",]//g'
2682023-11-14T23:29:29.579Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-11-14T23:29:29.603Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-11-14T23:29:29.632ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-11-14T23:29:29.664Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-11-14T23:29:29.697Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-11-14T23:29:29.727Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-11-14T23:29:29.755ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2752023-11-14T23:29:29.782Z0+0 records in
2762023-11-14T23:29:29.806ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2772023-11-14T23:29:29.832Z0+0 records out
2782023-11-14T23:29:29.857ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2792023-11-14T23:29:29.881Z0 bytes (0 iB) transferred in 0.000007 secs (0 iB/sec)
2802023-11-14T23:29:29.904ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2812023-11-14T23:29:29.928Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2822023-11-14T23:29:29.952ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2832023-11-14T23:29:29.979Z+ set +x
2842023-11-14T23:29:30.006ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2852023-11-14T23:29:30.036Z++ zpool list -o name
2862023-11-14T23:29:30.065ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2872023-11-14T23:29:30.091Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2882023-11-14T23:29:30.123ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2892023-11-14T23:29:30.148Z+ [[ -z '' ]]
2902023-11-14T23:29:30.174ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2912023-11-14T23:29:30.198Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2922023-11-14T23:29:30.222ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2932023-11-14T23:29:30.250Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2942023-11-14T23:29:30.274ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2952023-11-14T23:29:30.298Z+ set +x
2962023-11-14T23:29:30.328ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2972023-11-14T23:29:30.355Z+ for ZPOOL in "${ZPOOLS[@]}"
2982023-11-14T23:29:30.383ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-11-14T23:29:30.413Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3002023-11-14T23:29:30.438ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3012023-11-14T23:29:30.465Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3022023-11-14T23:29:30.493ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3032023-11-14T23:29:30.528Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3042023-11-14T23:29:30.621ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3052023-11-14T23:29:30.647Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3062023-11-14T23:29:30.674ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3072023-11-14T23:29:30.702Z0+0 records in
3082023-11-14T23:29:30.726ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3092023-11-14T23:29:30.750Z0+0 records out
3102023-11-14T23:29:30.775ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3112023-11-14T23:29:30.810Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3122023-11-14T23:29:30.836ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3132023-11-14T23:29:30.864Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3142023-11-14T23:29:30.889ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3152023-11-14T23:29:30.914Z+ set +x
3162023-11-14T23:29:30.939ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3172023-11-14T23:29:30.965Z++ zpool list -o name
3182023-11-14T23:29:30.991ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3192023-11-14T23:29:31.017Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-11-14T23:29:31.051ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3212023-11-14T23:29:31.076Z+ [[ -z '' ]]
3222023-11-14T23:29:31.100ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3232023-11-14T23:29:31.124Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3242023-11-14T23:29:31.148ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3252023-11-14T23:29:31.172Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3262023-11-14T23:29:31.197ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3272023-11-14T23:29:31.221Z+ set +x
3282023-11-14T23:29:31.245ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3292023-11-14T23:29:31.269Z+ for ZPOOL in "${ZPOOLS[@]}"
3302023-11-14T23:29:31.293ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3312023-11-14T23:29:31.319Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3322023-11-14T23:29:31.344ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3332023-11-14T23:29:31.381Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3342023-11-14T23:29:31.415ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3352023-11-14T23:29:31.451Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3362023-11-14T23:29:31.479ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3372023-11-14T23:29:31.512Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3382023-11-14T23:29:31.543ZSimnet net0/sc0_0 exists
3392023-11-14T23:29:31.578Z0+0 records in
3402023-11-14T23:29:31.605ZSimnet net1/sc1_0 exists
3412023-11-14T23:29:31.641Z0+0 records out
3422023-11-14T23:29:31.665ZVnic sc0_1 exists
3432023-11-14T23:29:31.690Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3442023-11-14T23:29:31.714Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
3452023-11-14T23:29:31.738Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3462023-11-14T23:29:31.762Z+ set +x
3472023-11-14T23:29:31.786Z++ zpool list -o name
3482023-11-14T23:29:31.811Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3492023-11-14T23:29:31.835Z+ [[ -z '' ]]
3502023-11-14T23:29:31.866Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3512023-11-14T23:29:31.891Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3522023-11-14T23:29:31.915Z+ set +x
3532023-11-14T23:29:31.943Z+ for ZPOOL in "${ZPOOLS[@]}"
3542023-11-14T23:29:31.980Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552023-11-14T23:29:32.006Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562023-11-14T23:29:32.031Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572023-11-14T23:29:32.055Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582023-11-14T23:29:32.080Z0+0 records in
3592023-11-14T23:29:32.103Z0+0 records out
3602023-11-14T23:29:32.127Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3612023-11-14T23:29:32.152Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622023-11-14T23:29:32.175Z+ set +x
3632023-11-14T23:29:32.202Z++ zpool list -o name
3642023-11-14T23:29:32.665Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3652023-11-14T23:29:32.690Z+ [[ -z '' ]]
3662023-11-14T23:29:32.714Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3672023-11-14T23:29:32.746Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3682023-11-14T23:29:32.772Z+ set +x
3692023-11-14T23:29:32.798Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-11-14T23:29:32.826Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3712023-11-14T23:29:32.853Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3722023-11-14T23:29:32.879Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3732023-11-14T23:29:32.907Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3742023-11-14T23:29:32.931Z0+0 records in
3752023-11-14T23:29:32.958Z0+0 records out
3762023-11-14T23:29:32.989Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3772023-11-14T23:29:33.013Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3782023-11-14T23:29:33.039Z+ set +x
3792023-11-14T23:29:33.064Z++ zpool list -o name
3802023-11-14T23:29:33.088Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3812023-11-14T23:29:33.111Z+ [[ -z '' ]]
3822023-11-14T23:29:33.135Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3832023-11-14T23:29:33.163Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3842023-11-14T23:29:33.605Z+ set +x
3852023-11-14T23:29:33.631Z+ for ZPOOL in "${ZPOOLS[@]}"
3862023-11-14T23:29:33.655Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-11-14T23:29:33.691Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-11-14T23:29:33.785Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-11-14T23:29:33.809Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-11-14T23:29:33.832Z0+0 records in
3912023-11-14T23:29:33.864Z0+0 records out
3922023-11-14T23:29:33.890Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3932023-11-14T23:29:33.918Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942023-11-14T23:29:33.945Z+ set +x
3952023-11-14T23:29:34.373Z++ zpool list -o name
3962023-11-14T23:29:34.400Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972023-11-14T23:29:34.424Z+ [[ -z '' ]]
3982023-11-14T23:29:34.451Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992023-11-14T23:29:34.479Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002023-11-14T23:29:34.505Z+ set +x
4012023-11-14T23:29:34.529Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-14T23:29:34.554Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4032023-11-14T23:29:34.577Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-11-14T23:29:34.601Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4052023-11-14T23:29:34.625Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4062023-11-14T23:29:34.652Z0+0 records in
4072023-11-14T23:29:34.679Z0+0 records out
4082023-11-14T23:29:34.741Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4092023-11-14T23:29:34.789Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4102023-11-14T23:29:34.820Z+ set +x
4112023-11-14T23:29:34.845Z++ zpool list -o name
4122023-11-14T23:29:34.875Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4132023-11-14T23:29:34.906Z+ [[ -z '' ]]
4142023-11-14T23:29:34.933Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4152023-11-14T23:29:35.063Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4162023-11-14T23:29:35.158Z+ set +x
4172023-11-14T23:29:35.230Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-11-14T23:29:35.707Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4192023-11-14T23:29:35.736Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4202023-11-14T23:29:35.783Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4212023-11-14T23:29:35.809Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4222023-11-14T23:29:35.837Z0+0 records in
4232023-11-14T23:29:35.866Z0+0 records out
4242023-11-14T23:29:35.894Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4252023-11-14T23:29:35.922Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4262023-11-14T23:29:35.946Z+ set +x
4272023-11-14T23:29:35.972Z++ zpool list -o name
4282023-11-14T23:29:35.998Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4292023-11-14T23:29:36.025Z+ [[ -z '' ]]
4302023-11-14T23:29:36.051Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4312023-11-14T23:29:36.078Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4322023-11-14T23:29:36.107Z+ set +x
4332023-11-14T23:29:36.141Z+ for ZPOOL in "${ZPOOLS[@]}"
4342023-11-14T23:29:36.174Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4352023-11-14T23:29:36.205Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-11-14T23:29:36.230Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4372023-11-14T23:29:36.257Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4382023-11-14T23:29:36.285Z0+0 records in
4392023-11-14T23:29:36.311Z0+0 records out
4402023-11-14T23:29:36.351Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4412023-11-14T23:29:36.383Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4422023-11-14T23:29:36.414Z+ set +x
4432023-11-14T23:29:36.447Z++ zpool list -o name
4442023-11-14T23:29:36.479Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4452023-11-14T23:29:36.508Z+ [[ -z '' ]]
4462023-11-14T23:29:36.540Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4472023-11-14T23:29:36.982Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4482023-11-14T23:29:37.020Z+ set +x
4492023-11-14T23:29:37.049Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4502023-11-14T23:29:37.480Z+ readarray -t ZPOOLS
4512023-11-14T23:29:37.511Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4522023-11-14T23:29:37.539Z++ sed 's/[ ",]//g'
4532023-11-14T23:29:37.564Z+ for ZPOOL in "${ZPOOLS[@]}"
4542023-11-14T23:29:37.608Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4552023-11-14T23:29:37.632Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4562023-11-14T23:29:37.656Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4572023-11-14T23:29:37.680Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4582023-11-14T23:29:37.704Z0+0 records in
4592023-11-14T23:29:37.727Z0+0 records out
4602023-11-14T23:29:38.162Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4612023-11-14T23:29:38.192Zdownloading softnpu ...done
4622023-11-14T23:29:38.217Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4632023-11-14T23:29:38.246Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
4642023-11-14T23:29:38.270Z+ set +x
4652023-11-14T23:29:38.703Z++ zpool list -o name
4662023-11-14T23:29:38.790Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4672023-11-14T23:29:38.813Z+ [[ -z '' ]]
4682023-11-14T23:29:38.838Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4692023-11-14T23:29:38.871Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4702023-11-14T23:29:38.895Z+ set +x
4712023-11-14T23:29:38.919Z+ for ZPOOL in "${ZPOOLS[@]}"
4722023-11-14T23:29:38.943Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4732023-11-14T23:29:38.967Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4742023-11-14T23:29:38.991Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4752023-11-14T23:29:39.016Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4762023-11-14T23:29:39.040Z0+0 records in
4772023-11-14T23:29:39.064Z0+0 records out
4782023-11-14T23:29:39.088Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4792023-11-14T23:29:39.111Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4802023-11-14T23:29:39.142Z+ set +x
4812023-11-14T23:29:39.167Z++ zpool list -o name
4822023-11-14T23:29:39.192Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4832023-11-14T23:29:39.218Z+ [[ -z '' ]]
4842023-11-14T23:29:39.245Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4852023-11-14T23:29:39.269Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4862023-11-14T23:29:39.293Z+ set +x
4872023-11-14T23:29:39.317Z+ [[ zone == \z\o\n\e ]]
4882023-11-14T23:29:39.341Z+ ensure_simulated_links igb0
4892023-11-14T23:29:39.365Z+ local PHYSICAL_LINK=igb0
4902023-11-14T23:29:39.389Z+ INDICES=("0" "1")
4912023-11-14T23:29:39.413Z+ for I in "${INDICES[@]}"
4922023-11-14T23:29:39.437Z++ get_simnet_name_if_exists net0
4932023-11-14T23:29:39.465Z++ dladm show-simnet -p -o LINK net0
4942023-11-14T23:29:39.498Z++ echo ''
4952023-11-14T23:29:39.525Z+ [[ -z '' ]]
4962023-11-14T23:29:39.551Z+ dladm create-simnet -t net0
4972023-11-14T23:29:39.585Z+ dladm create-simnet -t sc0_0
4982023-11-14T23:29:39.618Z+ dladm modify-simnet -t -p net0 sc0_0
4992023-11-14T23:29:39.655Z+ dladm set-linkprop -p mtu=9000 sc0_0
5002023-11-14T23:29:39.682Z+ success 'Simnet net0/sc0_0 exists'
5012023-11-14T23:29:39.706Z+ set +x
5022023-11-14T23:29:39.732Z+ for I in "${INDICES[@]}"
5032023-11-14T23:29:39.756Z++ get_simnet_name_if_exists net1
5042023-11-14T23:29:39.780Z++ dladm show-simnet -p -o LINK net1
5052023-11-14T23:29:39.804Z++ echo ''
5062023-11-14T23:29:39.828Z+ [[ -z '' ]]
5072023-11-14T23:29:39.852Z+ dladm create-simnet -t net1
5082023-11-14T23:29:39.908Z+ dladm create-simnet -t sc1_0
5092023-11-14T23:29:39.932Z+ dladm modify-simnet -t -p net1 sc1_0
5102023-11-14T23:29:39.956Z+ dladm set-linkprop -p mtu=9000 sc1_0
5112023-11-14T23:29:39.981Z+ success 'Simnet net1/sc1_0 exists'
5122023-11-14T23:29:40.422Z+ set +x
5132023-11-14T23:29:40.447Z++ get_vnic_name_if_exists sc0_1
5142023-11-14T23:29:40.474Z++ dladm show-vnic -p -o LINK sc0_1
5152023-11-14T23:29:40.500Z++ echo ''
5162023-11-14T23:29:40.529Z+ [[ -z '' ]]
5172023-11-14T23:29:40.556Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5182023-11-14T23:29:40.594Z+ [[ -v PROMISC_FILT_OFF ]]
5192023-11-14T23:29:40.621Z+ success 'Vnic sc0_1 exists'
5202023-11-14T23:29:40.645Z+ set +x
5212023-11-14T23:29:40.670Z+ warn_if_no_proxy_arp
5222023-11-14T23:29:40.694Z+ [[ -v PXA_START ]]
5232023-11-14T23:29:40.720Z+ [[ -v PXA_END ]]
5242023-11-14T23:29:40.744Z+ ensure_softnpu_zone
5252023-11-14T23:29:40.767Z+ zoneadm list
5262023-11-14T23:29:40.791Z+ grep -q sidecar_softnpu
5272023-11-14T23:29:40.815Z+ [[ -f out/npuzone/npuzone ]]
5282023-11-14T23:29:40.839Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5292023-11-14T23:29:40.863Zdownloading asic_program.so ...done
5302023-11-14T23:29:40.887Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-14T23:29:44.345Zdownloading scadm ...done
5322023-11-14T23:29:44.369Zcreating port tfportrear0_0 for sc0_0
5332023-11-14T23:29:44.393Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-14T23:29:44.417Zsoftnpu zone setup
5352023-11-14T23:29:44.442Zconfigure zone
5362023-11-14T23:29:44.466Zinstall zone
5372023-11-14T23:29:45.454Zboot zone
5382023-11-14T23:29:48.562Z[sidecar_softnpu] svcs milestone/network
5392023-11-14T23:29:48.588ZSTATE STIME FMRI
5402023-11-14T23:29:48.612Zonline 23:29:46 svc:/milestone/network:default
5412023-11-14T23:29:48.637Z[sidecar_softnpu] /softnpu/npu start
5422023-11-14T23:29:48.663Z
5432023-11-14T23:29:48.687Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-14T23:29:48.711Z+ GATEWAY_IP=10.151.1.100
5452023-11-14T23:29:48.735Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-11-14T23:29:48.762ZUsing 10.151.1.100 as gateway ip
5472023-11-14T23:29:48.787Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-14T23:29:48.811Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-11-14T23:29:48.836Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-14T23:29:48.860Z+++ echo 18:c0:4d:0d:9f:b2
5512023-11-14T23:29:48.885Z+ COUNT=1
5522023-11-14T23:29:48.909Z+ [[ 1 -ne 1 ]]
5532023-11-14T23:29:49.341Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-11-14T23:29:49.371ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-11-14T23:29:49.408Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-11-14T23:29:49.436Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-11-14T23:29:49.467Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-14T23:29:49.499Z+ [[ -v PXA_START ]]
5592023-11-14T23:29:49.523Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-11-14T23:29:49.548Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-11-14T23:29:49.574Z+ z_scadm dump-state
5622023-11-14T23:29:49.598Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-14T23:29:49.622Zlocal v6:
5642023-11-14T23:29:49.646Zlocal v4:
5652023-11-14T23:29:49.670Zrouter v6:
5662023-11-14T23:29:49.695Zrouter v4_idx:
5672023-11-14T23:29:49.724Zrouter v4_routes:
5682023-11-14T23:29:49.749Zresolver v4:
5692023-11-14T23:29:49.774Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-11-14T23:29:49.798Zresolver v6:
5712023-11-14T23:29:49.831Znat v4:
5722023-11-14T23:29:49.860Znat_v6:
5732023-11-14T23:29:49.887Zport_mac:
5742023-11-14T23:29:49.915Zproxy arp:
5752023-11-14T23:29:49.940Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-11-14T23:29:49.965Z+ success 'softnpu zone exists'
5772023-11-14T23:29:49.990Z+ set +x
5782023-11-14T23:29:50.016Zsoftnpu zone exists
5792023-11-14T23:29:50.052Z
5802023-11-14T23:29:50.078Zreal 21.391018500
5812023-11-14T23:29:50.102Zuser 0.706321353
5822023-11-14T23:29:50.128Zsys 1.630389448
5832023-11-14T23:29:50.153Ztrap 0.000111336
5842023-11-14T23:29:50.177Ztflt 0.000214493
5852023-11-14T23:29:50.227Zdflt 0.000978644
5862023-11-14T23:29:50.252Zkflt 0.000000000
5872023-11-14T23:29:50.718Zlock 16:24.243727997
5882023-11-14T23:29:50.748Zslp 1:04.028695370
5892023-11-14T23:29:50.777Zlat 0.076398732
5902023-11-14T23:29:50.806Zstop 0.357459424
5912023-11-14T23:29:50.831Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-14T23:29:50.855Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-14T23:29:50.879Z+ SILO_NAME=recovery
5942023-11-14T23:29:50.903Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-14T23:29:50.927Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-14T23:29:50.951Z+ sed -i~ '
5972023-11-14T23:29:50.977Z /^external_dns_ips/c\
5982023-11-14T23:29:51.006Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-11-14T23:29:51.034Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-14T23:29:51.067Z /^first/c\
6012023-11-14T23:29:51.098Zfirst = "10.151.1.164"
6022023-11-14T23:29:51.131Z /^last/c\
6032023-11-14T23:29:51.157Zlast = "10.151.1.173"
6042023-11-14T23:29:51.184Z }
6052023-11-14T23:29:51.213Z /^\[rack_network_config/,/^$/ {
6062023-11-14T23:29:51.255Z /^infra_ip_first/c\
6072023-11-14T23:29:51.279Zinfra_ip_first = "10.151.1.174"
6082023-11-14T23:29:51.314Z /^infra_ip_last/c\
6092023-11-14T23:29:51.341Zinfra_ip_last = "10.151.1.174"
6102023-11-14T23:29:51.365Z }
6112023-11-14T23:29:51.390Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-11-14T23:29:51.417Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6132023-11-14T23:29:51.442Z /^routes/c\
6142023-11-14T23:29:51.470Z+++ pkg/config-rss.toml Tue Nov 14 23:29:48 2023
6152023-11-14T23:29:51.494Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6162023-11-14T23:29:51.519Z@@ -41,7 +41,7 @@
6172023-11-14T23:29:51.543Z /^addresses/c\
6182023-11-14T23:29:51.573Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6192023-11-14T23:29:51.604Zaddresses = \["10.151.1.174/32"\]
6202023-11-14T23:29:52.041Z # must be contained in one of the "internal services" IP Pool ranges listed
6212023-11-14T23:29:52.066Z }
6222023-11-14T23:29:52.095Z # below.
6232023-11-14T23:29:52.125Z' pkg/config-rss.toml
6242023-11-14T23:29:52.150Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6252023-11-14T23:29:52.174Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6262023-11-14T23:29:52.198Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6272023-11-14T23:29:52.222Z+ true
6282023-11-14T23:29:52.247Z
6292023-11-14T23:29:52.284Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6302023-11-14T23:29:52.308Z # Initial TLS certificates for the external API
6312023-11-14T23:29:52.332Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6322023-11-14T23:29:52.357Z #
6332023-11-14T23:29:52.386Z+ TLS_NAME=recovery.sys.oxide.test
6342023-11-14T23:29:52.410Z@@ -77,8 +77,8 @@
6352023-11-14T23:29:52.434Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6362023-11-14T23:29:52.457Z #
6372023-11-14T23:29:52.481Z..........+..+.+.........+......+.....+.+........+.+..+..........+.....+....+.........+.....+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+....+.........+.....+...............+....+......+.....+..........+.....+....+..+.+..+......+......+.+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...............+.........+.....+......+......+...+......+.......+...+.........+...........+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6382023-11-14T23:29:52.516Z # For more on this and what to put here, see docs/how-to-run.adoc.
6392023-11-14T23:29:52.541Z..+..+....+...+........+...+....+...........+....+......+...........+.....................+....+...+..+.+............+..+.+.....+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+............+.+...+..+.............+.........+..+................+...+...........+.+..+.+......+........+..................+.......+..+....+.........+...+..+.+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+.+..+..........+......+..+.........+......+.........+......+....+....................+...+...............+...............+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6402023-11-14T23:29:52.581Z [[internal_services_ip_pool_ranges]]
6412023-11-14T23:29:52.606Z-----
6422023-11-14T23:29:52.630Z-first = "192.168.1.20"
6432023-11-14T23:29:52.654Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6442023-11-14T23:29:52.678Z-last = "192.168.1.29"
6452023-11-14T23:29:52.702Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6462023-11-14T23:29:52.727Z+first = "10.151.1.164"
6472023-11-14T23:29:52.751Z+ rmdir pkg
6482023-11-14T23:29:52.775Z+last = "10.151.1.173"
6492023-11-14T23:29:53.212Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6502023-11-14T23:29:53.242Z
6512023-11-14T23:29:53.266Z+ pfexec mkdir -p /usr/oxide
6522023-11-14T23:29:53.313Z # TODO - this configuration is subject to change going forward. Ultimately these
6532023-11-14T23:29:53.339Z+ pfexec rm -f /usr/oxide/tcpproxy
6542023-11-14T23:29:53.363Z # parameters should be provided to the control plane via wicket, but we need to
6552023-11-14T23:29:53.799Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6562023-11-14T23:29:53.881Z@@ -92,8 +92,8 @@
6572023-11-14T23:29:53.907Z+ pfexec chmod +x /usr/oxide/tcpproxy
6582023-11-14T23:29:53.933Z # A range of IP addresses used by Boundary Services on the external network. In
6592023-11-14T23:29:53.959Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6602023-11-14T23:29:53.983Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6612023-11-14T23:29:54.009Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6622023-11-14T23:29:54.033Z # With softnpu, only one address is used.
6632023-11-14T23:29:54.057Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6642023-11-14T23:29:54.081Z-infra_ip_first = "192.168.1.30"
6652023-11-14T23:29:54.107Z+ OMICRON_NO_UNINSTALL=1
6662023-11-14T23:29:54.132Z-infra_ip_last = "192.168.1.30"
6672023-11-14T23:29:54.156Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6682023-11-14T23:29:54.182Z+infra_ip_first = "10.151.1.174"
6692023-11-14T23:29:54.206ZNov 14 23:29:49.802 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6702023-11-14T23:29:54.230Z+infra_ip_last = "10.151.1.174"
6712023-11-14T23:29:54.262ZNov 14 23:29:49.804 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6722023-11-14T23:29:54.299Z
6732023-11-14T23:29:54.382ZNov 14 23:29:49.804 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6742023-11-14T23:29:54.408Z # Configurations for BGP routers to run on the scrimlets.
6752023-11-14T23:29:54.434ZNov 14 23:29:49.804 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6762023-11-14T23:29:54.458Z bgp = []
6772023-11-14T23:29:54.482ZNov 14 23:29:49.804 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6782023-11-14T23:29:54.506Z@@ -101,9 +101,9 @@
6792023-11-14T23:29:54.530ZNov 14 23:29:49.804 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6802023-11-14T23:29:54.555Z # You can configure multiple uplinks by repeating the following stanza
6812023-11-14T23:29:54.594ZNov 14 23:29:49.804 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6822023-11-14T23:29:54.619Z [[rack_network_config.ports]]
6832023-11-14T23:29:54.644ZNov 14 23:29:49.804 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6842023-11-14T23:29:54.675Z # Routes associated with this port.
6852023-11-14T23:29:54.717ZNov 14 23:29:49.804 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6862023-11-14T23:29:54.759Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6872023-11-14T23:29:54.786ZNov 14 23:29:49.804 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6882023-11-14T23:29:54.812Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6892023-11-14T23:29:54.837ZNov 14 23:29:49.804 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6902023-11-14T23:29:54.860Z # Addresses associated with this port.
6912023-11-14T23:29:54.884ZNov 14 23:29:49.805 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6922023-11-14T23:29:54.908Z-addresses = ["192.168.1.30/32"]
6932023-11-14T23:29:54.932ZNov 14 23:29:49.805 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6942023-11-14T23:29:54.956Z+addresses = ["10.151.1.174/32"]
6952023-11-14T23:29:54.980ZNov 14 23:29:49.805 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6962023-11-14T23:29:55.005Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6972023-11-14T23:29:55.031ZNov 14 23:29:49.805 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6982023-11-14T23:29:55.055Z port = "qsfp0"
6992023-11-14T23:29:55.080ZNov 14 23:29:49.805 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7002023-11-14T23:29:55.106Z # The speed of this port.
7012023-11-14T23:29:55.565ZNov 14 23:29:49.805 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7022023-11-14T23:29:56.001Za pkg/config-rss.toml 7K
7032023-11-14T23:29:56.032ZNov 14 23:29:50.021 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7042023-11-14T23:29:56.060Za pkg/initial-tls-cert.pem 2K
7052023-11-14T23:29:56.084ZNov 14 23:29:50.056 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7062023-11-14T23:29:56.108Za pkg/initial-tls-key.pem 4K
7072023-11-14T23:29:56.132ZNov 14 23:29:50.118 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-14T23:29:56.156Z
7092023-11-14T23:29:56.180Zreal 0.456113085
7102023-11-14T23:29:56.204Zuser 0.098523869
7112023-11-14T23:29:56.228Zsys 1.808396156
7122023-11-14T23:29:56.252Ztrap 0.000000420
7132023-11-14T23:29:56.276Ztflt 0.000171661
7142023-11-14T23:29:56.299Zdflt 0.001153299
7152023-11-14T23:29:56.329Zkflt 0.000000000
7162023-11-14T23:29:56.357Zlock 40.824229289
7172023-11-14T23:29:56.381Zslp 0.679079272
7182023-11-14T23:29:56.405Zlat 0.003143855
7192023-11-14T23:29:56.434Zstop 0.205840506
7202023-11-14T23:29:56.462Z+ retry=0
7212023-11-14T23:29:56.491Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-14T23:29:56.520Z+ [[ 0 -gt 30 ]]
7232023-11-14T23:29:56.545Z+ sleep 1
7242023-11-14T23:29:56.571Z+ retry=1
7252023-11-14T23:29:56.598Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-14T23:29:56.624Z+ [[ 1 -gt 30 ]]
7272023-11-14T23:29:56.648Z+ sleep 1
7282023-11-14T23:29:56.672Z+ retry=2
7292023-11-14T23:29:56.696Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-14T23:29:56.720Z+ [[ 2 -gt 30 ]]
7312023-11-14T23:29:56.745Z+ sleep 1
7322023-11-14T23:29:56.772Z+ retry=3
7332023-11-14T23:29:56.797Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-11-14T23:29:56.821Z+ [[ 3 -gt 30 ]]
7352023-11-14T23:29:56.844Z+ sleep 1
7362023-11-14T23:29:56.870Z+ retry=4
7372023-11-14T23:29:56.894Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-11-14T23:30:14.084Z+ [[ 4 -gt 30 ]]
7392023-11-14T23:30:14.109Z+ sleep 1
7402023-11-14T23:30:15.087Z+ retry=5
7412023-11-14T23:30:15.112Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7422023-11-14T23:30:15.139Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7432023-11-14T23:30:15.186Zlocal v6:
7442023-11-14T23:30:15.215Zfd00:99::1
7452023-11-14T23:30:15.245Zfe80::608d:a2ff:fe84:4c48
7462023-11-14T23:30:15.281Zlocal v4:
7472023-11-14T23:30:15.326Z10.151.1.174
7482023-11-14T23:30:15.357Zrouter v6:
7492023-11-14T23:30:15.389Zrouter v4_idx:
7502023-11-14T23:30:15.416Z0.0.0.0/0 -> 0
7512023-11-14T23:30:15.441Zrouter v4_routes:
7522023-11-14T23:30:15.465Z0 -> 10.151.1.100 (2)
7532023-11-14T23:30:15.490Zresolver v4:
7542023-11-14T23:30:15.514Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7552023-11-14T23:30:15.538Zresolver v6:
7562023-11-14T23:30:15.562Zfe80::608d:a2ff:fe84:4c48 -> 62:8d:a2:84:4c:48
7572023-11-14T23:30:15.587Zfe80::d0cb:e7ff:fec3:c93a -> d2:cb:e7:c3:c9:3a
7582023-11-14T23:30:15.630Znat v4:
7592023-11-14T23:30:15.655Znat_v6:
7602023-11-14T23:30:15.680Zport_mac:
7612023-11-14T23:30:15.704Z1000: a8:40:25:58:d5:dc
7622023-11-14T23:30:15.729Z1: a8:40:25:58:d5:dd
7632023-11-14T23:30:15.756Zproxy arp:
7642023-11-14T23:30:15.782Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7652023-11-14T23:30:15.806Z+ retry=0
7662023-11-14T23:30:15.836Z+++ zoneadm list -n
7672023-11-14T23:30:15.862Z+++ grep oxz_ntp
7682023-11-14T23:30:15.885Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-11-14T23:30:15.911Zsvcs: zone '-Hostate': entity not found
7702023-11-14T23:30:15.935Z++ true
7712023-11-14T23:30:15.959Z+ [[ '' != online ]]
7722023-11-14T23:30:15.984Z+ [[ 0 -gt 60 ]]
7732023-11-14T23:30:16.035Z+ sleep 1
7742023-11-14T23:30:16.153Z+ retry=1
7752023-11-14T23:30:16.181Z+++ zoneadm list -n
7762023-11-14T23:30:16.206Z+++ grep oxz_ntp
7772023-11-14T23:30:16.230Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-11-14T23:30:16.257Zsvcs: zone '-Hostate': entity not found
7792023-11-14T23:30:16.281Z++ true
7802023-11-14T23:30:16.305Z+ [[ '' != online ]]
7812023-11-14T23:30:16.330Z+ [[ 1 -gt 60 ]]
7822023-11-14T23:30:16.354Z+ sleep 1
7832023-11-14T23:30:17.171Z+ retry=2
7842023-11-14T23:30:17.197Z+++ zoneadm list -n
7852023-11-14T23:30:17.223Z+++ grep oxz_ntp
7862023-11-14T23:30:17.249Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-11-14T23:30:17.273Zsvcs: zone '-Hostate': entity not found
7882023-11-14T23:30:17.297Z++ true
7892023-11-14T23:30:17.322Z+ [[ '' != online ]]
7902023-11-14T23:30:17.346Z+ [[ 2 -gt 60 ]]
7912023-11-14T23:30:17.370Z+ sleep 1
7922023-11-14T23:30:18.195Z+ retry=3
7932023-11-14T23:30:18.219Z+++ zoneadm list -n
7942023-11-14T23:30:18.243Z+++ grep oxz_ntp
7952023-11-14T23:30:18.267Z++ pfexec svcs -z oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0 -Hostate oxide/ntp
7962023-11-14T23:30:18.291Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7972023-11-14T23:30:18.315Z++ true
7982023-11-14T23:30:18.339Z+ [[ '' != online ]]
7992023-11-14T23:30:18.363Z+ [[ 3 -gt 60 ]]
8002023-11-14T23:30:18.415Z+ sleep 1
8012023-11-14T23:30:19.250Z+ retry=4
8022023-11-14T23:30:19.274Z+++ zoneadm list -n
8032023-11-14T23:30:19.298Z+++ grep oxz_ntp
8042023-11-14T23:30:19.323Z++ pfexec svcs -z oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0 -Hostate oxide/ntp
8052023-11-14T23:30:19.348Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8062023-11-14T23:30:19.372Z++ true
8072023-11-14T23:30:19.396Z+ [[ '' != online ]]
8082023-11-14T23:30:19.420Z+ [[ 4 -gt 60 ]]
8092023-11-14T23:30:19.444Z+ sleep 1
8102023-11-14T23:30:20.281Z+ retry=5
8112023-11-14T23:30:20.305Z+++ zoneadm list -n
8122023-11-14T23:30:20.330Z+++ grep oxz_ntp
8132023-11-14T23:30:20.355Z++ pfexec svcs -z oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0 -Hostate oxide/ntp
8142023-11-14T23:30:20.380Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8152023-11-14T23:30:20.404Z++ true
8162023-11-14T23:30:20.429Z+ [[ '' != online ]]
8172023-11-14T23:30:20.453Z+ [[ 5 -gt 60 ]]
8182023-11-14T23:30:20.478Z+ sleep 1
8192023-11-14T23:30:21.309Z+ retry=6
8202023-11-14T23:30:21.337Z+++ zoneadm list -n
8212023-11-14T23:30:21.361Z+++ grep oxz_ntp
8222023-11-14T23:30:21.386Z++ pfexec svcs -z oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0 -Hostate oxide/ntp
8232023-11-14T23:30:21.410Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8242023-11-14T23:30:21.434Z++ true
8252023-11-14T23:30:21.458Z+ [[ '' != online ]]
8262023-11-14T23:30:21.482Z+ [[ 6 -gt 60 ]]
8272023-11-14T23:30:21.505Z+ sleep 1
8282023-11-14T23:30:22.500Z+ retry=7
8292023-11-14T23:30:22.526Z+++ zoneadm list -n
8302023-11-14T23:30:22.551Z+++ grep oxz_ntp
8312023-11-14T23:30:22.575Z++ pfexec svcs -z oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0 -Hostate oxide/ntp
8322023-11-14T23:30:22.599Z+ [[ disabled != online ]]
8332023-11-14T23:30:22.624Z+ [[ 7 -gt 60 ]]
8342023-11-14T23:30:22.649Z+ sleep 1
8352023-11-14T23:30:23.501Z+ retry=8
8362023-11-14T23:30:23.525Z+++ zoneadm list -n
8372023-11-14T23:30:23.557Z+++ grep oxz_ntp
8382023-11-14T23:30:23.586Z++ pfexec svcs -z oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0 -Hostate oxide/ntp
8392023-11-14T23:30:23.611Z+ [[ disabled != online ]]
8402023-11-14T23:30:23.635Z+ [[ 8 -gt 60 ]]
8412023-11-14T23:30:23.659Z+ sleep 1
8422023-11-14T23:30:24.381Z+ retry=9
8432023-11-14T23:30:24.405Z+++ zoneadm list -n
8442023-11-14T23:30:24.429Z+++ grep oxz_ntp
8452023-11-14T23:30:24.454Z++ pfexec svcs -z oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0 -Hostate oxide/ntp
8462023-11-14T23:30:24.477Z+ [[ disabled != online ]]
8472023-11-14T23:30:24.502Z+ [[ 9 -gt 60 ]]
8482023-11-14T23:30:24.526Z+ sleep 1
8492023-11-14T23:30:25.408Z+ retry=10
8502023-11-14T23:30:25.433Z+++ zoneadm list -n
8512023-11-14T23:30:25.458Z+++ grep oxz_ntp
8522023-11-14T23:30:25.482Z++ pfexec svcs -z oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0 -Hostate oxide/ntp
8532023-11-14T23:30:25.505Z+ [[ disabled != online ]]
8542023-11-14T23:30:25.530Z+ [[ 10 -gt 60 ]]
8552023-11-14T23:30:25.558Z+ sleep 1
8562023-11-14T23:30:26.425Z+ retry=11
8572023-11-14T23:30:26.451Z+++ zoneadm list -n
8582023-11-14T23:30:26.475Z+++ grep oxz_ntp
8592023-11-14T23:30:26.500Z++ pfexec svcs -z oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0 -Hostate oxide/ntp
8602023-11-14T23:30:26.525Z+ [[ disabled != online ]]
8612023-11-14T23:30:26.549Z+ [[ 11 -gt 60 ]]
8622023-11-14T23:30:26.573Z+ sleep 1
8632023-11-14T23:30:27.446Z+ retry=12
8642023-11-14T23:30:27.475Z+++ zoneadm list -n
8652023-11-14T23:30:27.500Z+++ grep oxz_ntp
8662023-11-14T23:30:27.524Z++ pfexec svcs -z oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0 -Hostate oxide/ntp
8672023-11-14T23:30:27.550Z+ [[ disabled != online ]]
8682023-11-14T23:30:27.574Z+ [[ 12 -gt 60 ]]
8692023-11-14T23:30:27.598Z+ sleep 1
 
8702023-11-14T23:30:28.164Zbuskin console login: Nov 14 23:30:27 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8712023-11-14T23:30:28.468Z+ retry=13
8722023-11-14T23:30:28.493Z+++ zoneadm list -n
8732023-11-14T23:30:28.517Z+++ grep oxz_ntp
8742023-11-14T23:30:28.540Z++ pfexec svcs -z oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0 -Hostate oxide/ntp
8752023-11-14T23:30:28.566Z+ [[ online != online ]]
8762023-11-14T23:30:28.590Z+ echo 'Waited for chrony: 13s'
8772023-11-14T23:30:28.614Z+ export RUST_BACKTRACE=1
8782023-11-14T23:30:28.643ZWaited for chrony: 13s
8792023-11-14T23:30:28.670Z+ RUST_BACKTRACE=1
8802023-11-14T23:30:28.697Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8812023-11-14T23:30:28.721Z+ ./tests/bootstrap
8822023-11-14T23:30:28.745ZUsing external DNS server at 10.151.1.164:53
8832023-11-14T23:30:28.769Z2023-11-14 23:30:28.661134277 UTC: attempting to log into API
8842023-11-14T23:30:43.558Z2023-11-14 23:30:43.925051065 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8852023-11-14T23:30:44.557Z2023-11-14 23:30:44.923507465 UTC: attempting to log into API
8862023-11-14T23:30:59.589Z2023-11-14 23:30:59.917866314 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8872023-11-14T23:31:00.595Z2023-11-14 23:31:00.916356884 UTC: attempting to log into API
8882023-11-14T23:31:15.715Z2023-11-14 23:31:16.000308031 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8892023-11-14T23:31:16.731Z2023-11-14 23:31:16.999014104 UTC: attempting to log into API
8902023-11-14T23:31:31.767Z2023-11-14 23:31:32.007944159 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8912023-11-14T23:31:32.766Z2023-11-14 23:31:33.006334209 UTC: attempting to log into API
8922023-11-14T23:31:47.819Z2023-11-14 23:31:47.999981580 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8932023-11-14T23:31:48.816Z2023-11-14 23:31:48.998425040 UTC: attempting to log into API
 
8942023-11-14T23:31:49.993ZNov 14 23:31:49 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
8952023-11-14T23:31:57.404ZNov 14 23:31:57 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
8962023-11-14T23:32:03.837Z2023-11-14 23:32:03.992560796 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8972023-11-14T23:32:04.852Z2023-11-14 23:32:04.991784035 UTC: attempting to log into API
 
8982023-11-14T23:32:05.070ZNov 14 23:32:04 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
8992023-11-14T23:32:12.217ZNov 14 23:32:12 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
9002023-11-14T23:32:18.040ZNov 14 23:32:17 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9012023-11-14T23:32:19.872Z2023-11-14 23:32:19.985625539 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9022023-11-14T23:32:20.874Z2023-11-14 23:32:20.984078799 UTC: attempting to log into API
9032023-11-14T23:32:35.907Z2023-11-14 23:32:35.977349148 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9042023-11-14T23:32:36.908Z2023-11-14 23:32:36.975776298 UTC: attempting to log into API
9052023-11-14T23:32:42.698Z2023-11-14 23:32:42.749374619 UTC: login succeeded
9062023-11-14T23:32:42.731Zwaiting for nexus to come up...
9072023-11-14T23:32:42.799Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9082023-11-14T23:32:42.847Zensuring datasets are ready...
9092023-11-14T23:32:44.990Zlet's roll.
9102023-11-14T23:32:45.018Z+ rm ./tests/bootstrap
9112023-11-14T23:32:45.055Z+ for test_bin in tests/*
9122023-11-14T23:32:45.086Z+ ./tests/bootstrap-14470ed2888247c6
9132023-11-14T23:32:45.465Z
9142023-11-14T23:32:45.491Zrunning 0 tests
9152023-11-14T23:32:45.545Z
9162023-11-14T23:32:45.572Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9172023-11-14T23:32:45.597Z
9182023-11-14T23:32:45.621Z+ for test_bin in tests/*
9192023-11-14T23:32:45.659Z+ ./tests/end_to_end_tests-0f2fa1e5548e7ba9
9202023-11-14T23:32:45.683Z
9212023-11-14T23:32:45.719Zrunning 2 tests
9222023-11-14T23:32:45.745Ztest no_spoof::no_spoof_here ... ok
9232023-11-14T23:33:47.633Ztest instance_launch::instance_launch has been running for over 60 seconds
9242023-11-14T23:35:41.424Ztest instance_launch::instance_launch ... ok
9252023-11-14T23:35:41.449Z
9262023-11-14T23:35:41.473Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 176.35s
9272023-11-14T23:35:41.497Z
9282023-11-14T23:35:41.521Z+ _exit_trap
9292023-11-14T23:35:41.546Z+ local status=0
9302023-11-14T23:35:41.570Z+ [[ 0 -eq 0 ]]
9312023-11-14T23:35:41.594Z+ exit 0
9322023-11-14T23:35:41.618Zprocess exited: duration 381886 ms, exit code 0
 
9332023-11-14T23:35:41.720Zfound 47 output files
9342023-11-14T23:35:41.959Zuploading: /var/svc/log/oxide-sled-agent:default.log (726291 bytes)
9352023-11-14T23:35:43.084Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9362023-11-14T23:35:43.108Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_be596ba2-9a0b-4aad-a36e-b2014c3b5f24/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
9372023-11-14T23:35:44.180Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_be596ba2-9a0b-4aad-a36e-b2014c3b5f24/root/var/svc/log/oxide-cockroachdb:default.log
9382023-11-14T23:35:44.204Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eb17585b-6ade-4857-a49d-1f44e51dfae6/root/var/svc/log/oxide-crucible-agent:default.log (14117 bytes)
9392023-11-14T23:35:45.298Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eb17585b-6ade-4857-a49d-1f44e51dfae6/root/var/svc/log/oxide-crucible-agent:default.log
9402023-11-14T23:35:45.329Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eb17585b-6ade-4857-a49d-1f44e51dfae6/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9412023-11-14T23:35:46.409Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eb17585b-6ade-4857-a49d-1f44e51dfae6/root/var/svc/log/oxide-crucible-downstairs:default.log
9422023-11-14T23:35:46.439Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eb17585b-6ade-4857-a49d-1f44e51dfae6/root/var/svc/log/oxide-crucible-downstairs:downstairs-25161666-ed5c-4596-bb5b-a5ff87ace351.log (5537 bytes)
9432023-11-14T23:35:47.520Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eb17585b-6ade-4857-a49d-1f44e51dfae6/root/var/svc/log/oxide-crucible-downstairs:downstairs-25161666-ed5c-4596-bb5b-a5ff87ace351.log
9442023-11-14T23:35:47.544Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_baafb488-7a10-4928-b3bf-d510eeeb100d/root/var/svc/log/oxide-internal_dns:default.log (103122 bytes)
9452023-11-14T23:35:48.727Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_baafb488-7a10-4928-b3bf-d510eeeb100d/root/var/svc/log/oxide-internal_dns:default.log
9462023-11-14T23:35:48.751Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_c53e1084-0450-44ce-b68e-2c5bf1766d4d/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9472023-11-14T23:35:49.821Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_c53e1084-0450-44ce-b68e-2c5bf1766d4d/root/var/svc/log/oxide-crucible-agent:default.log
9482023-11-14T23:35:49.846Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_c53e1084-0450-44ce-b68e-2c5bf1766d4d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9492023-11-14T23:35:50.915Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_c53e1084-0450-44ce-b68e-2c5bf1766d4d/root/var/svc/log/oxide-crucible-downstairs:default.log
9502023-11-14T23:35:50.939Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_b4d99837-d133-47a7-82b7-f6687a7e7775/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
9512023-11-14T23:35:52.010Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_b4d99837-d133-47a7-82b7-f6687a7e7775/root/var/svc/log/oxide-cockroachdb:default.log
9522023-11-14T23:35:52.035Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2eebe89d-2085-4f42-b336-68c63b50002e/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
9532023-11-14T23:35:53.127Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2eebe89d-2085-4f42-b336-68c63b50002e/root/var/svc/log/oxide-crucible-agent:default.log
9542023-11-14T23:35:53.152Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2eebe89d-2085-4f42-b336-68c63b50002e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9552023-11-14T23:35:54.227Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2eebe89d-2085-4f42-b336-68c63b50002e/root/var/svc/log/oxide-crucible-downstairs:default.log
9562023-11-14T23:35:54.254Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2eebe89d-2085-4f42-b336-68c63b50002e/root/var/svc/log/oxide-crucible-downstairs:downstairs-34e3a13c-a6eb-44a9-a12d-55e5c08a10cd.log (5535 bytes)
9572023-11-14T23:35:55.328Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2eebe89d-2085-4f42-b336-68c63b50002e/root/var/svc/log/oxide-crucible-downstairs:downstairs-34e3a13c-a6eb-44a9-a12d-55e5c08a10cd.log
9582023-11-14T23:35:55.354Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ce48495d-1585-4203-92ac-dfca4edc6612/root/var/svc/log/oxide-crucible-agent:default.log (14109 bytes)
9592023-11-14T23:35:56.445Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ce48495d-1585-4203-92ac-dfca4edc6612/root/var/svc/log/oxide-crucible-agent:default.log
9602023-11-14T23:35:56.469Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ce48495d-1585-4203-92ac-dfca4edc6612/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9612023-11-14T23:35:57.542Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ce48495d-1585-4203-92ac-dfca4edc6612/root/var/svc/log/oxide-crucible-downstairs:default.log
9622023-11-14T23:35:57.566Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ce48495d-1585-4203-92ac-dfca4edc6612/root/var/svc/log/oxide-crucible-downstairs:downstairs-caf35cc2-88f7-4154-8442-1b62e318cdd1.log (3050 bytes)
9632023-11-14T23:35:58.636Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ce48495d-1585-4203-92ac-dfca4edc6612/root/var/svc/log/oxide-crucible-downstairs:downstairs-caf35cc2-88f7-4154-8442-1b62e318cdd1.log
9642023-11-14T23:35:58.661Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_de18037f-ce0c-49a9-8770-a4189116140d/root/var/svc/log/oxide-crucible-agent:default.log (14581 bytes)
9652023-11-14T23:35:59.768Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_de18037f-ce0c-49a9-8770-a4189116140d/root/var/svc/log/oxide-crucible-agent:default.log
9662023-11-14T23:35:59.792Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_de18037f-ce0c-49a9-8770-a4189116140d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9672023-11-14T23:36:00.861Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_de18037f-ce0c-49a9-8770-a4189116140d/root/var/svc/log/oxide-crucible-downstairs:default.log
9682023-11-14T23:36:00.891Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_de18037f-ce0c-49a9-8770-a4189116140d/root/var/svc/log/oxide-crucible-downstairs:downstairs-a0bb283a-95ae-4370-be13-5996f8e098f3.log (5531 bytes)
9692023-11-14T23:36:01.956Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_de18037f-ce0c-49a9-8770-a4189116140d/root/var/svc/log/oxide-crucible-downstairs:downstairs-a0bb283a-95ae-4370-be13-5996f8e098f3.log
9702023-11-14T23:36:02.007Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_bdb5a6ea-9c6b-486c-9362-f94ecea4b8dc/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
9712023-11-14T23:36:03.086Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_bdb5a6ea-9c6b-486c-9362-f94ecea4b8dc/root/var/svc/log/oxide-crucible-pantry:default.log
9722023-11-14T23:36:03.113Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_0138dbed-c652-4123-a105-05d895d90dd4/root/var/svc/log/oxide-external_dns:default.log (15609 bytes)
9732023-11-14T23:36:04.223Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_0138dbed-c652-4123-a105-05d895d90dd4/root/var/svc/log/oxide-external_dns:default.log
9742023-11-14T23:36:04.252Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_654d59fd-cff5-4893-a2e3-bf608164aa61/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9752023-11-14T23:36:05.322Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_654d59fd-cff5-4893-a2e3-bf608164aa61/root/var/svc/log/oxide-crucible-agent:default.log
9762023-11-14T23:36:05.347Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_654d59fd-cff5-4893-a2e3-bf608164aa61/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9772023-11-14T23:36:06.418Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_654d59fd-cff5-4893-a2e3-bf608164aa61/root/var/svc/log/oxide-crucible-downstairs:default.log
9782023-11-14T23:36:06.442Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_0fa609c0-cbd3-48e0-a1d8-fb2e3bd92050/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9792023-11-14T23:36:07.512Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_0fa609c0-cbd3-48e0-a1d8-fb2e3bd92050/root/var/svc/log/oxide-crucible-pantry:default.log
9802023-11-14T23:36:07.537Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_3f881278-3f5d-4954-a952-b57de1db44dc/root/var/svc/log/oxide-external_dns:default.log (15614 bytes)
9812023-11-14T23:36:08.632Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_3f881278-3f5d-4954-a952-b57de1db44dc/root/var/svc/log/oxide-external_dns:default.log
9822023-11-14T23:36:08.656Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_237612ed-1dd0-4dab-8dea-f5f95033f7da/root/var/svc/log/oxide-cockroachdb:default.log (5342 bytes)
9832023-11-14T23:36:09.731Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_237612ed-1dd0-4dab-8dea-f5f95033f7da/root/var/svc/log/oxide-cockroachdb:default.log
9842023-11-14T23:36:09.757Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_f7ff1172-7520-4f1e-aee6-421802227c1b/root/var/svc/log/oxide-cockroachdb:default.log (5444 bytes)
9852023-11-14T23:36:11.239Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_f7ff1172-7520-4f1e-aee6-421802227c1b/root/var/svc/log/oxide-cockroachdb:default.log
9862023-11-14T23:36:11.270Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_c8fd7a87-328a-4d09-ad0b-cc5fb592954d/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
9872023-11-14T23:36:12.356Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_c8fd7a87-328a-4d09-ad0b-cc5fb592954d/root/var/svc/log/oxide-crucible-agent:default.log
9882023-11-14T23:36:12.381Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_c8fd7a87-328a-4d09-ad0b-cc5fb592954d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9892023-11-14T23:36:13.451Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_c8fd7a87-328a-4d09-ad0b-cc5fb592954d/root/var/svc/log/oxide-crucible-downstairs:default.log
9902023-11-14T23:36:13.475Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_c8fd7a87-328a-4d09-ad0b-cc5fb592954d/root/var/svc/log/oxide-crucible-downstairs:downstairs-5c5ddb5b-67a4-4c35-9687-95064606a42c.log (3051 bytes)
9912023-11-14T23:36:14.545Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_c8fd7a87-328a-4d09-ad0b-cc5fb592954d/root/var/svc/log/oxide-crucible-downstairs:downstairs-5c5ddb5b-67a4-4c35-9687-95064606a42c.log
9922023-11-14T23:36:14.569Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_a15b6286-9510-45eb-8077-0b406c39ceb2/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9932023-11-14T23:36:15.639Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_a15b6286-9510-45eb-8077-0b406c39ceb2/root/var/svc/log/oxide-crucible-pantry:default.log
9942023-11-14T23:36:15.670Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_dbb3bd35-fdf3-414f-b30c-432a7eb7d4c1/root/var/svc/log/oxide-nexus:default.log (601622 bytes)
9952023-11-14T23:36:16.020Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_dbb3bd35-fdf3-414f-b30c-432a7eb7d4c1/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 601622 -> 603447
9962023-11-14T23:36:17.090Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_dbb3bd35-fdf3-414f-b30c-432a7eb7d4c1/root/var/svc/log/oxide-nexus:default.log
9972023-11-14T23:36:17.115Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_33b83136-a0c1-47f8-9b0f-5ae6384ec096/root/var/svc/log/oxide-internal_dns:default.log (99082 bytes)
9982023-11-14T23:36:18.298Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_33b83136-a0c1-47f8-9b0f-5ae6384ec096/root/var/svc/log/oxide-internal_dns:default.log
9992023-11-14T23:36:18.324Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_42979c7d-bb8e-446e-99de-800104299eda/root/var/svc/log/oxide-internal_dns:default.log (100697 bytes)
10002023-11-14T23:36:19.516Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_42979c7d-bb8e-446e-99de-800104299eda/root/var/svc/log/oxide-internal_dns:default.log
10012023-11-14T23:36:19.541Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_5c55ca96-12fb-4735-8c0b-6823d5d8b3a4/root/var/svc/log/oxide-nexus:default.log (4495206 bytes)
10022023-11-14T23:36:21.225Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_5c55ca96-12fb-4735-8c0b-6823d5d8b3a4/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4495206 -> 4555823
10032023-11-14T23:36:22.298Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_5c55ca96-12fb-4735-8c0b-6823d5d8b3a4/root/var/svc/log/oxide-nexus:default.log
10042023-11-14T23:36:31.239Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_730e5c4a-0485-4292-90ae-ab2d6ecf1cdc/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
10052023-11-14T23:36:32.738Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_730e5c4a-0485-4292-90ae-ab2d6ecf1cdc/root/var/svc/log/oxide-crucible-agent:default.log
10062023-11-14T23:36:32.838Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_730e5c4a-0485-4292-90ae-ab2d6ecf1cdc/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10072023-11-14T23:36:33.849Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_730e5c4a-0485-4292-90ae-ab2d6ecf1cdc/root/var/svc/log/oxide-crucible-downstairs:default.log
10082023-11-14T23:36:33.894Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_730e5c4a-0485-4292-90ae-ab2d6ecf1cdc/root/var/svc/log/oxide-crucible-downstairs:downstairs-e4f03a1e-bac7-48a9-939d-ecf56901e159.log (3048 bytes)
10092023-11-14T23:36:34.963Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_730e5c4a-0485-4292-90ae-ab2d6ecf1cdc/root/var/svc/log/oxide-crucible-downstairs:downstairs-e4f03a1e-bac7-48a9-939d-ecf56901e159.log
10102023-11-14T23:36:34.996Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_74bdba25-e090-41bd-a39e-bbe0fe442f99/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10112023-11-14T23:36:36.057Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_74bdba25-e090-41bd-a39e-bbe0fe442f99/root/var/svc/log/oxide-crucible-agent:default.log
10122023-11-14T23:36:36.082Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_74bdba25-e090-41bd-a39e-bbe0fe442f99/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10132023-11-14T23:36:37.166Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_74bdba25-e090-41bd-a39e-bbe0fe442f99/root/var/svc/log/oxide-crucible-downstairs:default.log
10142023-11-14T23:36:37.193Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_d8416f51-9ba5-4d92-aced-73830f2169cf/root/var/svc/log/oxide-clickhouse:default.log (928889 bytes)
10152023-11-14T23:36:37.473Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_d8416f51-9ba5-4d92-aced-73830f2169cf/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 928889 -> 1143232
10162023-11-14T23:36:38.542Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_d8416f51-9ba5-4d92-aced-73830f2169cf/root/var/svc/log/oxide-clickhouse:default.log
10172023-11-14T23:36:38.566Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_cb80cb45-57c1-439f-b95f-59023e63f8a6/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10182023-11-14T23:36:39.647Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_cb80cb45-57c1-439f-b95f-59023e63f8a6/root/var/svc/log/oxide-cockroachdb:default.log
10192023-11-14T23:36:39.671Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_958ba27a-a2d0-437a-83cc-e70f9df32000/root/var/svc/log/oxide-nexus:default.log (492722 bytes)
10202023-11-14T23:36:39.929Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_958ba27a-a2d0-437a-83cc-e70f9df32000/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 492722 -> 554553
10212023-11-14T23:36:41.420Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_958ba27a-a2d0-437a-83cc-e70f9df32000/root/var/svc/log/oxide-nexus:default.log
10222023-11-14T23:36:41.447Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0/root/var/svc/log/oxide-ntp:default.log (1243 bytes)
10232023-11-14T23:36:42.517Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0/root/var/svc/log/oxide-ntp:default.log
10242023-11-14T23:36:42.542Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3927c1ae-6637-45ec-b5af-fb0619bb471c/root/var/svc/log/oxide-oximeter:default.log (238492 bytes)
10252023-11-14T23:36:42.730Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3927c1ae-6637-45ec-b5af-fb0619bb471c/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 238492 -> 299077
10262023-11-14T23:36:43.807Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3927c1ae-6637-45ec-b5af-fb0619bb471c/root/var/svc/log/oxide-oximeter:default.log
10272023-11-14T23:36:43.831Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0/root/var/log/chrony/measurements.log (8494 bytes)
10282023-11-14T23:36:43.875Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0/root/var/log/chrony/measurements.log" changed size mid upload: 8494 -> 9590
10292023-11-14T23:36:44.968Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0/root/var/log/chrony/measurements.log
10302023-11-14T23:36:44.993Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0/root/var/log/chrony/statistics.log (5382 bytes)
10312023-11-14T23:36:45.019Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0/root/var/log/chrony/statistics.log" changed size mid upload: 5382 -> 6318
10322023-11-14T23:36:46.111Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0/root/var/log/chrony/statistics.log
10332023-11-14T23:36:46.139Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0/root/var/log/chrony/tracking.log (1717 bytes)
10342023-11-14T23:36:46.971Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0/root/var/log/chrony/tracking.log" changed size mid upload: 1717 -> 1849
10352023-11-14T23:36:48.077Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_502402bc-2276-4e08-84d1-36a7f09c4ee0/root/var/log/chrony/tracking.log