01HDQ0ZPVMET8TYNTGW21XSXM7: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HDQ110XNNX3MY61JXCDNY6JY

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-26T23:11:33.586Zjob dependencies complete; ready to run (waiting for 53 m 55 s)
22023-10-26T23:13:15.492Zjob assigned to worker 01HDQ45PBF3QDPP03DGNC8ZA8N (queued for 1 m 41 s)
32023-10-26T23:13:15.566Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-26T23:13:15.618Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-10-26T23:13:15.650Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-10-26T23:13:15.686Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-26T23:13:15.721Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-10-26T23:13:15.752ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-26T23:13:15.785Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-26T23:13:15.818ZOxide Helios Version helios-2.0.22241 64-bit
112023-10-26T23:13:16.259ZConfiguring devices.
122023-10-26T23:13:16.298Z##################################################
132023-10-26T23:13:16.341Z#### ############# ##
142023-10-26T23:13:16.391Z### ### ############ ##
152023-10-26T23:13:16.434Z## ### # ## ### ## Oxide Computer Company ##
162023-10-26T23:13:16.472Z## ## ## ### # ### ##
172023-10-26T23:13:16.519Z## # ### #### #### This Station Under ##
182023-10-26T23:13:16.553Z### ### #### # ### Computer Control ##
192023-10-26T23:13:16.590Z#### #### ### ## ##
202023-10-26T23:13:16.634Z##################################################
212023-10-26T23:13:16.677ZHostname: buskin
222023-10-26T23:13:16.717ZLoading smf(7) service descriptions:
232023-10-26T23:13:17.176Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-ce25c26538ce669c.gz
242023-10-26T23:13:20.121Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-ce25c26538ce669c.gz
252023-10-26T23:13:20.147Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-26T23:14:03.660Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-26T23:14:04.071Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-bcb3df1a66c8d052.gz
282023-10-26T23:14:33.206Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-bcb3df1a66c8d052.gz
292023-10-26T23:14:33.232Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-26T23:14:34.362Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-26T23:14:34.390Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-26T23:14:38.758Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-26T23:14:38.784Zdownloading input: /input/package/work/version.txt
342023-10-26T23:14:38.808Zdownloaded input: /input/package/work/version.txt
352023-10-26T23:14:38.832Zdownloading input: /input/package/work/package.tar.gz
362023-10-26T23:14:59.800Zdownloaded input: /input/package/work/package.tar.gz
372023-10-26T23:14:59.955Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-26T23:16:01.254Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-26T23:16:01.284Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-26T23:16:21.619Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-26T23:16:21.651Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-26T23:17:48.700Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-26T23:17:48.729Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-26T23:19:01.109Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-26T23:19:01.266Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-26T23:20:07.385Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-26T23:20:07.427Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-26T23:20:21.493Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-26T23:20:21.529Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-26T23:20:48.787Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-26T23:20:48.815Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-26T23:20:59.588Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-26T23:20:59.613Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-26T23:21:11.200Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-26T23:21:11.228Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-26T23:21:11.563Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-26T23:21:11.592Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-26T23:21:19.106Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-26T23:21:19.148Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-26T23:21:24.083Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-26T23:21:24.113Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-26T23:21:48.897Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-26T23:21:48.933Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-26T23:22:09.974Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-26T23:22:10.008Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-26T23:22:10.040Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-26T23:22:10.073Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-26T23:22:17.024Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-26T23:22:17.049Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-26T23:22:54.020Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-26T23:22:54.049Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-26T23:24:20.400Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-26T23:24:20.431Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-26T23:25:19.462Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-26T23:25:19.495Zstarting task 0: "setup"
762023-10-26T23:25:19.531Z++ uname -s
772023-10-26T23:25:19.586Z+ kern=SunOS
782023-10-26T23:25:19.618Z+ case "$kern" in
792023-10-26T23:25:20.088Z+ groupadd -g 12345 build
802023-10-26T23:25:20.124Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-26T23:25:21.538Z+ zfs create -o mountpoint=/work rpool/work
822023-10-26T23:25:21.569Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-26T23:25:21.604Z+ home_fs=zfs
842023-10-26T23:25:21.664Z+ [[ zfs == autofs ]]
852023-10-26T23:25:21.697Z+ mkdir -p /home/build
862023-10-26T23:25:21.732Z+ chown build:build /home/build /work
872023-10-26T23:25:23.566Z+ chmod 0700 /home/build /work
882023-10-26T23:25:23.590Zprocess exited: duration 4049 ms, exit code 0
 
892023-10-26T23:25:23.643Zstarting task 1: "authentication"
902023-10-26T23:25:23.713Zprocess exited: duration 43 ms, exit code 0
 
912023-10-26T23:25:23.763Zstarting task 2: "build"
922023-10-26T23:25:23.795Z+ trap _exit_trap EXIT
932023-10-26T23:25:23.820Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-26T23:25:23.995Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-26T23:25:24.022Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-26T23:25:24.048Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-26T23:25:24.073Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-26T23:25:24.098Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-26T23:25:24.122Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-26T23:25:24.148Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-26T23:25:24.187Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-26T23:25:24.216Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-26T23:25:24.242Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-26T23:25:24.267Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-26T23:25:24.296Z+ [[ -d /opt/oxide ]]
1062023-10-26T23:25:24.332Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-26T23:25:24.358Z+ pfexec mkdir /opt/oxide
1082023-10-26T23:25:24.386Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-26T23:25:24.420Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-26T23:25:24.450Z+ cd /opt/oxide-underneath
1112023-10-26T23:25:24.478Z+ pfexec tar ceEp@/f - .
1122023-10-26T23:25:24.503Z+ cd /opt/oxide
1132023-10-26T23:25:24.528Z+ pfexec tar xveEp@/f -
1142023-10-26T23:25:24.553Zx ., 0 bytes, 0 tape blocks
1152023-10-26T23:25:24.578Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-26T23:25:24.603Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-26T23:25:24.629Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-10-26T23:25:24.662Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-26T23:25:24.700Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-26T23:25:24.741Z+ pfexec mkdir /opt/oxide/work
1212023-10-26T23:25:24.772Z+ pfexec chown build:build /opt/oxide/work
1222023-10-26T23:25:24.800Z+ cd /opt/oxide/work
1232023-10-26T23:25:24.825Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-26T23:25:25.698ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-26T23:25:25.725Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1262023-10-26T23:25:25.767Zx out/omicron-sled-agent.tar, 115091968 bytes, 224789 tape blocks
1272023-10-26T23:25:25.792Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-26T23:25:25.816Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-10-26T23:25:25.841Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-10-26T23:25:25.871Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-26T23:25:25.898Zx target/release/omicron-package, 38187008 bytes, 74584 tape blocks
1322023-10-26T23:25:25.922Zx tools/create_virtual_hardware.sh, 2974 bytes, 6 tape blocks
1332023-10-26T23:25:25.947Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-26T23:25:26.381Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-26T23:25:26.406Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-26T23:25:26.430Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-26T23:25:26.455Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-26T23:25:26.479Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-26T23:25:26.503Z
1402023-10-26T23:25:26.528Zreal 1.542540473
1412023-10-26T23:25:26.552Zuser 1.392685554
1422023-10-26T23:25:26.577Zsys 0.266942330
1432023-10-26T23:25:26.601Ztrap 0.000064360
1442023-10-26T23:25:26.627Ztflt 0.000000000
1452023-10-26T23:25:26.651Zdflt 0.000000000
1462023-10-26T23:25:26.675Zkflt 0.000000000
1472023-10-26T23:25:26.701Zlock 0.000000000
1482023-10-26T23:25:27.130Zslp 1.389338115
1492023-10-26T23:25:27.158Zlat 0.033894042
1502023-10-26T23:25:27.228Zstop 0.000122212
1512023-10-26T23:25:27.256Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-26T23:25:27.287Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-26T23:25:27.318Z+ mkdir tests
1542023-10-26T23:25:27.385Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-26T23:25:27.463Z+ ptime -m gunzip
1562023-10-26T23:25:27.487Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-ce25c26538ce669c
1572023-10-26T23:25:27.511Z
1582023-10-26T23:25:27.535Zreal 0.141285860
1592023-10-26T23:25:27.560Zuser 0.129726308
1602023-10-26T23:25:27.585Zsys 0.011335787
1612023-10-26T23:25:27.618Ztrap 0.000008360
1622023-10-26T23:25:27.658Ztflt 0.000000000
1632023-10-26T23:25:27.684Zdflt 0.000000000
1642023-10-26T23:25:27.713Zkflt 0.000000000
1652023-10-26T23:25:27.742Zlock 0.000000000
1662023-10-26T23:25:28.180Zslp 0.000018280
1672023-10-26T23:25:28.208Zlat 0.000015170
1682023-10-26T23:25:28.233Zstop 0.000119782
1692023-10-26T23:25:28.284Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-ce25c26538ce669c
1702023-10-26T23:25:28.309Z+ chmod a+x tests/bootstrap-ce25c26538ce669c
1712023-10-26T23:25:28.333Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-26T23:25:28.357Z+ ptime -m gunzip
1732023-10-26T23:25:28.382Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-26T23:25:29.146Z
1752023-10-26T23:25:29.175Zreal 2.241488097
1762023-10-26T23:25:29.215Zuser 2.104310415
1772023-10-26T23:25:29.250Zsys 0.136795923
1782023-10-26T23:25:29.289Ztrap 0.000161535
1792023-10-26T23:25:29.316Ztflt 0.000000000
1802023-10-26T23:25:29.340Zdflt 0.000000000
1812023-10-26T23:25:29.365Zkflt 0.000000000
1822023-10-26T23:25:29.391Zlock 0.000000000
1832023-10-26T23:25:29.415Zslp 0.000012890
1842023-10-26T23:25:29.439Zlat 0.000022780
1852023-10-26T23:25:29.465Zstop 0.000118752
1862023-10-26T23:25:29.923Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-26T23:25:29.949Z+ chmod a+x tests/bootstrap
1882023-10-26T23:25:29.980Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-26T23:25:30.011Z+ ptime -m gunzip
1902023-10-26T23:25:30.040Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-bcb3df1a66c8d052
1912023-10-26T23:25:31.718Z
1922023-10-26T23:25:31.753Zreal 2.561540718
1932023-10-26T23:25:31.779Zuser 2.394676336
1942023-10-26T23:25:31.808Zsys 0.166425032
1952023-10-26T23:25:31.836Ztrap 0.000209637
1962023-10-26T23:25:31.862Ztflt 0.000000000
1972023-10-26T23:25:31.889Zdflt 0.000000000
1982023-10-26T23:25:31.950Zkflt 0.000000000
1992023-10-26T23:25:31.982Zlock 0.000000000
2002023-10-26T23:25:32.012Zslp 0.000020770
2012023-10-26T23:25:32.096Zlat 0.000029320
2022023-10-26T23:25:32.357Zstop 0.000121172
2032023-10-26T23:25:32.399Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-bcb3df1a66c8d052
2042023-10-26T23:25:32.435Z+ chmod a+x tests/end_to_end_tests-bcb3df1a66c8d052
2052023-10-26T23:25:32.507Z+ bmat address ls
2062023-10-26T23:25:32.538ZNAME CIDR FIRST LAST COUNT
2072023-10-26T23:25:32.563Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-10-26T23:25:32.590Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-26T23:25:32.618Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-10-26T23:25:32.655Z+ EXTRA_IP_START=10.151.1.164
2112023-10-26T23:25:32.687Z+ EXTRA_IP_END=10.151.1.227
2122023-10-26T23:25:32.721Z+ EXTRA_IP_COUNT=64
2132023-10-26T23:25:32.750Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-10-26T23:25:32.784Z+ EXTRA_IP_BASE=10.151.1
2152023-10-26T23:25:32.821Z+ EXTRA_IP_FOCTET=164
2162023-10-26T23:25:32.855Z+ EXTRA_IP_LOCTET=227
2172023-10-26T23:25:33.302Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-10-26T23:25:33.330Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-10-26T23:25:33.356Z+ DNS_IP1=10.151.1.164
2202023-10-26T23:25:33.384Z+ DNS_IP2=10.151.1.165
2212023-10-26T23:25:33.410Z+ UPLINK_IP=10.151.1.174
2222023-10-26T23:25:33.435Z+ IPPOOL_START=10.151.1.175
2232023-10-26T23:25:33.459Z+ IPPOOL_END=10.151.1.227
2242023-10-26T23:25:33.484Z++ ipadm show-addr -po type,addr
2252023-10-26T23:25:33.509Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-10-26T23:25:33.534Z+ GATEWAY_IP=10.151.1.100
2272023-10-26T23:25:33.564Z+ [[ -n 10.151.1.100 ]]
2282023-10-26T23:25:33.594Z+ ping -s 10.151.1.100 56 1
2292023-10-26T23:25:33.620ZPING 10.151.1.100: 56 data bytes
2302023-10-26T23:25:33.644Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.010 ms
2312023-10-26T23:25:33.683Z
2322023-10-26T23:25:34.170Z----10.151.1.100 PING Statistics----
2332023-10-26T23:25:34.203Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-10-26T23:25:34.240Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2352023-10-26T23:25:34.268Z++ arp -an
2362023-10-26T23:25:34.300Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-10-26T23:25:34.325Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-10-26T23:25:34.350Z+ routeadm -e ipv4-forwarding -u
2392023-10-26T23:25:34.377Z+ PXA_START=10.151.1.164
2402023-10-26T23:25:34.410Z+ PXA_END=10.151.1.227
2412023-10-26T23:25:34.438Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-10-26T23:25:34.466Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-10-26T23:25:34.493Z+ ZPOOL_VDEV_DIR=/scratch
2442023-10-26T23:25:34.521Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-10-26T23:25:34.549Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-10-26T23:25:34.575Z++ cd ./tools
2472023-10-26T23:25:34.599Z++ pwd
2482023-10-26T23:25:34.633Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-10-26T23:25:35.143Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-10-26T23:25:35.176Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-10-26T23:25:35.210Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-10-26T23:25:35.239Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-10-26T23:25:35.263Z++ dladm show-phys -p -o LINK
2542023-10-26T23:25:35.287Z++ head -1
2552023-10-26T23:25:35.311Z+ PHYSICAL_LINK=igb0
2562023-10-26T23:25:35.335Z+ echo 'Using igb0 as physical link'
2572023-10-26T23:25:35.360ZUsing igb0 as physical link
2582023-10-26T23:25:35.387Z+ ensure_run_as_root
2592023-10-26T23:25:35.419Z++ id -u
2602023-10-26T23:25:35.449Z+ [[ 0 -ne 0 ]]
2612023-10-26T23:25:35.476Z+ ensure_zpools
2622023-10-26T23:25:35.502Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-10-26T23:25:35.532Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-10-26T23:25:35.566Z+ readarray -t ZPOOLS
2652023-10-26T23:25:35.611Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-10-26T23:25:35.642Z++ sed 's/[ ",]//g'
2672023-10-26T23:25:35.670ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2682023-10-26T23:25:35.701Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-10-26T23:25:35.727ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2702023-10-26T23:25:35.754Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2712023-10-26T23:25:35.784ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2722023-10-26T23:25:35.811Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2732023-10-26T23:25:35.840ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2742023-10-26T23:25:35.866Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2752023-10-26T23:25:35.900ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2762023-10-26T23:25:35.937Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2772023-10-26T23:25:35.965ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2782023-10-26T23:25:35.995Z0+0 records in
2792023-10-26T23:25:36.030ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2802023-10-26T23:25:36.060Z0+0 records out
2812023-10-26T23:25:36.087ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2822023-10-26T23:25:36.162Z0 bytes (0 iB) transferred in 0.000007 secs (0 iB/sec)
2832023-10-26T23:25:36.193ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2842023-10-26T23:25:36.231Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2852023-10-26T23:25:36.257ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2862023-10-26T23:25:36.283Z+ set +x
2872023-10-26T23:25:36.316ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2882023-10-26T23:25:36.346Z++ zpool list -o name
2892023-10-26T23:25:36.371ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2902023-10-26T23:25:36.396Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2912023-10-26T23:25:36.836ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2922023-10-26T23:25:36.875Z+ [[ -z '' ]]
2932023-10-26T23:25:36.913ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2942023-10-26T23:25:36.946Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2952023-10-26T23:25:36.976ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2962023-10-26T23:25:37.006Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2972023-10-26T23:25:37.034ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
2982023-10-26T23:25:37.072Z+ set +x
2992023-10-26T23:25:37.103ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3002023-10-26T23:25:37.136Z+ for ZPOOL in "${ZPOOLS[@]}"
3012023-10-26T23:25:37.585ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3022023-10-26T23:25:37.618Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3032023-10-26T23:25:37.649ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3042023-10-26T23:25:37.678Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3052023-10-26T23:25:37.751ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3062023-10-26T23:25:37.777Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3072023-10-26T23:25:37.805ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3082023-10-26T23:25:37.836Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3092023-10-26T23:25:37.871ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3102023-10-26T23:25:37.927Z0+0 records in
3112023-10-26T23:25:38.369ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3122023-10-26T23:25:38.399Z0+0 records out
3132023-10-26T23:25:38.431ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3142023-10-26T23:25:38.461Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3152023-10-26T23:25:38.488ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3162023-10-26T23:25:38.925Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3172023-10-26T23:25:38.953ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3182023-10-26T23:25:38.988Z+ set +x
3192023-10-26T23:25:39.023ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3202023-10-26T23:25:39.054Z++ zpool list -o name
3212023-10-26T23:25:39.084ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3222023-10-26T23:25:39.529Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3232023-10-26T23:25:39.571ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3242023-10-26T23:25:39.621Z+ [[ -z '' ]]
3252023-10-26T23:25:39.646ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3262023-10-26T23:25:39.698Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3272023-10-26T23:25:39.728ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3282023-10-26T23:25:39.757Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3292023-10-26T23:25:39.793ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3302023-10-26T23:25:39.819Z+ set +x
3312023-10-26T23:25:39.845ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3322023-10-26T23:25:39.871Z+ for ZPOOL in "${ZPOOLS[@]}"
3332023-10-26T23:25:39.897ZSimnet net0/sc0_0 exists
3342023-10-26T23:25:39.926Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3352023-10-26T23:25:39.967ZSimnet net1/sc1_0 exists
3362023-10-26T23:25:39.999Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3372023-10-26T23:25:40.040ZVnic sc0_1 exists
3382023-10-26T23:25:40.077Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3392023-10-26T23:25:40.108Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3402023-10-26T23:25:40.144Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3412023-10-26T23:25:40.183Z0+0 records in
3422023-10-26T23:25:40.221Z0+0 records out
3432023-10-26T23:25:40.261Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3442023-10-26T23:25:40.338Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3452023-10-26T23:25:40.378Z+ set +x
3462023-10-26T23:25:40.410Z++ zpool list -o name
3472023-10-26T23:25:40.438Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3482023-10-26T23:25:40.466Z+ [[ -z '' ]]
3492023-10-26T23:25:40.499Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-10-26T23:25:40.529Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-10-26T23:25:40.566Z+ set +x
3522023-10-26T23:25:40.600Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-10-26T23:25:40.637Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-10-26T23:25:40.674Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-10-26T23:25:40.710Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-10-26T23:25:40.747Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-10-26T23:25:41.235Z0+0 records in
3582023-10-26T23:25:41.273Z0+0 records out
3592023-10-26T23:25:41.299Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3602023-10-26T23:25:41.333Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-10-26T23:25:41.378Z+ set +x
3622023-10-26T23:25:41.405Z++ zpool list -o name
3632023-10-26T23:25:41.432Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-10-26T23:25:41.456Z+ [[ -z '' ]]
3652023-10-26T23:25:41.480Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-10-26T23:25:41.509Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-10-26T23:25:41.535Z+ set +x
3682023-10-26T23:25:41.564Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-10-26T23:25:41.596Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-10-26T23:25:41.624Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-10-26T23:25:41.655Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-10-26T23:25:41.690Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-10-26T23:25:41.717Z0+0 records in
3742023-10-26T23:25:41.751Z0+0 records out
3752023-10-26T23:25:41.786Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3762023-10-26T23:25:41.822Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-10-26T23:25:41.856Z+ set +x
3782023-10-26T23:25:41.890Z++ zpool list -o name
3792023-10-26T23:25:41.920Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-10-26T23:25:41.951Z+ [[ -z '' ]]
3812023-10-26T23:25:41.983Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-10-26T23:25:42.014Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-10-26T23:25:42.053Z+ set +x
3842023-10-26T23:25:42.085Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-10-26T23:25:42.116Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862023-10-26T23:25:42.235Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872023-10-26T23:25:42.271Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882023-10-26T23:25:42.302Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892023-10-26T23:25:42.340Z0+0 records in
3902023-10-26T23:25:42.811Z0+0 records out
3912023-10-26T23:25:42.838Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3922023-10-26T23:25:42.864Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932023-10-26T23:25:42.889Z+ set +x
3942023-10-26T23:25:42.918Z++ zpool list -o name
3952023-10-26T23:25:42.945Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3962023-10-26T23:25:42.971Z+ [[ -z '' ]]
3972023-10-26T23:25:42.998Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3982023-10-26T23:25:43.026Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3992023-10-26T23:25:43.052Z+ set +x
4002023-10-26T23:25:43.493Z+ for ZPOOL in "${ZPOOLS[@]}"
4012023-10-26T23:25:43.519Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4022023-10-26T23:25:43.955Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4032023-10-26T23:25:44.388Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4042023-10-26T23:25:44.420Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4052023-10-26T23:25:44.493Z0+0 records in
4062023-10-26T23:25:44.526Z0+0 records out
4072023-10-26T23:25:44.558Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4082023-10-26T23:25:44.592Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4092023-10-26T23:25:44.631Z+ set +x
4102023-10-26T23:25:44.666Z++ zpool list -o name
4112023-10-26T23:25:44.697Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4122023-10-26T23:25:44.727Z+ [[ -z '' ]]
4132023-10-26T23:25:45.166Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4142023-10-26T23:25:45.203Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4152023-10-26T23:25:45.241Z+ set +x
4162023-10-26T23:25:45.274Z+ for ZPOOL in "${ZPOOLS[@]}"
4172023-10-26T23:25:45.309Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4182023-10-26T23:25:45.343Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192023-10-26T23:25:45.377Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4202023-10-26T23:25:45.413Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4212023-10-26T23:25:45.440Z0+0 records in
4222023-10-26T23:25:45.476Z0+0 records out
4232023-10-26T23:25:45.540Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4242023-10-26T23:25:45.571Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4252023-10-26T23:25:46.034Z+ set +x
4262023-10-26T23:25:46.061Z++ zpool list -o name
4272023-10-26T23:25:46.091Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4282023-10-26T23:25:46.119Z+ [[ -z '' ]]
4292023-10-26T23:25:46.566Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4302023-10-26T23:25:46.609Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4312023-10-26T23:25:46.635Z+ set +x
4322023-10-26T23:25:46.663Z+ for ZPOOL in "${ZPOOLS[@]}"
4332023-10-26T23:25:46.692Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4342023-10-26T23:25:46.726Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4352023-10-26T23:25:46.759Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4362023-10-26T23:25:46.790Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4372023-10-26T23:25:46.822Z0+0 records in
4382023-10-26T23:25:46.860Z0+0 records out
4392023-10-26T23:25:46.901Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4402023-10-26T23:25:46.926Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4412023-10-26T23:25:46.956Z+ set +x
4422023-10-26T23:25:46.983Z++ zpool list -o name
4432023-10-26T23:25:47.019Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4442023-10-26T23:25:47.059Z+ [[ -z '' ]]
4452023-10-26T23:25:47.087Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4462023-10-26T23:25:47.188Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4472023-10-26T23:25:47.214Z+ set +x
4482023-10-26T23:25:47.239Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4492023-10-26T23:25:47.263Z+ readarray -t ZPOOLS
4502023-10-26T23:25:47.294Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4512023-10-26T23:25:47.330Z++ sed 's/[ ",]//g'
4522023-10-26T23:25:47.362Z+ for ZPOOL in "${ZPOOLS[@]}"
4532023-10-26T23:25:47.393Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4542023-10-26T23:25:47.418Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4552023-10-26T23:25:47.445Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4562023-10-26T23:25:47.480Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4572023-10-26T23:25:47.511Z0+0 records in
4582023-10-26T23:25:47.543Z0+0 records out
4592023-10-26T23:25:47.608Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4602023-10-26T23:25:47.638Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4612023-10-26T23:25:47.667Z+ set +x
4622023-10-26T23:25:47.691Z++ zpool list -o name
4632023-10-26T23:25:47.716Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4642023-10-26T23:25:47.744Z+ [[ -z '' ]]
4652023-10-26T23:25:47.776Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-10-26T23:25:47.812Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4672023-10-26T23:25:47.843Z+ set +x
4682023-10-26T23:25:47.872Z+ for ZPOOL in "${ZPOOLS[@]}"
4692023-10-26T23:25:47.901Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4702023-10-26T23:25:47.927Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4712023-10-26T23:25:47.955Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4722023-10-26T23:25:47.984Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4732023-10-26T23:25:48.018Z0+0 records in
4742023-10-26T23:25:48.054Z0+0 records out
4752023-10-26T23:25:48.087Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4762023-10-26T23:25:48.119Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4772023-10-26T23:25:48.149Z+ set +x
4782023-10-26T23:25:48.177Z++ zpool list -o name
4792023-10-26T23:25:48.206Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4802023-10-26T23:25:48.231Z+ [[ -z '' ]]
4812023-10-26T23:25:48.267Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4822023-10-26T23:25:48.292Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4832023-10-26T23:25:48.317Z+ set +x
4842023-10-26T23:25:48.342Z+ ensure_simulated_links igb0
4852023-10-26T23:25:48.367Z+ local PHYSICAL_LINK=igb0
4862023-10-26T23:25:48.392Z+ INDICES=("0" "1")
4872023-10-26T23:25:48.417Z+ for I in "${INDICES[@]}"
4882023-10-26T23:25:48.448Z++ get_simnet_name_if_exists net0
4892023-10-26T23:25:48.474Z++ dladm show-simnet -p -o LINK net0
4902023-10-26T23:25:48.501Z++ echo ''
4912023-10-26T23:25:48.533Z+ [[ -z '' ]]
4922023-10-26T23:25:48.971Z+ dladm create-simnet -t net0
4932023-10-26T23:25:49.414Z+ dladm create-simnet -t sc0_0
4942023-10-26T23:25:49.873Z+ dladm modify-simnet -t -p net0 sc0_0
4952023-10-26T23:25:49.899Z+ dladm set-linkprop -p mtu=9000 sc0_0
4962023-10-26T23:25:49.934Z+ success 'Simnet net0/sc0_0 exists'
4972023-10-26T23:25:49.968Z+ set +x
4982023-10-26T23:25:50.002Z+ for I in "${INDICES[@]}"
4992023-10-26T23:25:50.073Z++ get_simnet_name_if_exists net1
5002023-10-26T23:25:50.114Z++ dladm show-simnet -p -o LINK net1
5012023-10-26T23:25:50.142Z++ echo ''
5022023-10-26T23:25:50.200Z+ [[ -z '' ]]
5032023-10-26T23:25:50.242Z+ dladm create-simnet -t net1
5042023-10-26T23:25:50.285Z+ dladm create-simnet -t sc1_0
5052023-10-26T23:25:50.783Z+ dladm modify-simnet -t -p net1 sc1_0
5062023-10-26T23:25:50.819Z+ dladm set-linkprop -p mtu=9000 sc1_0
5072023-10-26T23:25:51.265Z+ success 'Simnet net1/sc1_0 exists'
5082023-10-26T23:25:51.295Z+ set +x
5092023-10-26T23:25:51.321Z++ get_vnic_name_if_exists sc0_1
5102023-10-26T23:25:51.346Z++ dladm show-vnic -p -o LINK sc0_1
5112023-10-26T23:25:51.379Z++ echo ''
5122023-10-26T23:25:51.415Z+ [[ -z '' ]]
5132023-10-26T23:25:51.445Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-10-26T23:25:51.470Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-10-26T23:25:51.495Z+ success 'Vnic sc0_1 exists'
5162023-10-26T23:25:51.520Z+ set +x
5172023-10-26T23:25:51.544Z+ warn_if_no_proxy_arp
5182023-10-26T23:25:51.568Z+ [[ -v PXA_START ]]
5192023-10-26T23:25:51.593Z+ [[ -v PXA_END ]]
5202023-10-26T23:25:51.618Z+ ensure_softnpu_zone
5212023-10-26T23:25:51.642Z+ zoneadm list
5222023-10-26T23:25:51.667Z+ grep -q sidecar_softnpu
5232023-10-26T23:25:51.702Z+ [[ -f out/npuzone/npuzone ]]
5242023-10-26T23:25:51.729Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch main
5252023-10-26T23:25:51.753Zdownloading softnpu ...done
5262023-10-26T23:25:51.777Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5272023-10-26T23:25:58.352Zdownloading asic_program.so ...done
5282023-10-26T23:25:58.394Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5292023-10-26T23:26:13.780Zdownloading scadm ...done
5302023-10-26T23:26:13.814Zcreating port tfportrear0_0 for sc0_0
5312023-10-26T23:26:14.256Zcreating port tfportqsfp0_0 for sc0_1
5322023-10-26T23:26:14.283Zsoftnpu zone setup
5332023-10-26T23:26:14.311Zconfigure zone
5342023-10-26T23:26:14.341Zinstall zone
5352023-10-26T23:26:15.281Zboot zone
5362023-10-26T23:26:17.972Z[sidecar_softnpu] svcs milestone/network
5372023-10-26T23:26:17.999ZSTATE STIME FMRI
5382023-10-26T23:26:18.029Zonline 23:26:16 svc:/milestone/network:default
5392023-10-26T23:26:18.471Z[sidecar_softnpu] /softnpu/npu start
5402023-10-26T23:26:18.501Z
5412023-10-26T23:26:18.527Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5422023-10-26T23:26:18.552Z+ GATEWAY_IP=10.151.1.100
5432023-10-26T23:26:18.579Z+ echo 'Using 10.151.1.100 as gateway ip'
5442023-10-26T23:26:18.606ZUsing 10.151.1.100 as gateway ip
5452023-10-26T23:26:18.633Z+ [[ ! -v GATEWAY_MAC ]]
5462023-10-26T23:26:18.658Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5472023-10-26T23:26:18.685Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5482023-10-26T23:26:18.716Z+++ echo 18:c0:4d:0d:9f:b2
5492023-10-26T23:26:18.782Z+ COUNT=1
5502023-10-26T23:26:18.812Z+ [[ 1 -ne 1 ]]
5512023-10-26T23:26:18.840Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5522023-10-26T23:26:18.866ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5532023-10-26T23:26:18.899Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5542023-10-26T23:26:18.932Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5552023-10-26T23:26:18.966Z+ PXA_MAC=a8:e1:de:01:70:1d
5562023-10-26T23:26:18.993Z+ [[ -v PXA_START ]]
5572023-10-26T23:26:19.025Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5582023-10-26T23:26:19.051Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5592023-10-26T23:26:19.078Z+ z_scadm dump-state
5602023-10-26T23:26:19.103Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5612023-10-26T23:26:19.128Zlocal v6:
5622023-10-26T23:26:19.160Zlocal v4:
5632023-10-26T23:26:19.187Zrouter v6:
5642023-10-26T23:26:19.215Zrouter v4_idx:
5652023-10-26T23:26:19.252Zrouter v4_routes:
5662023-10-26T23:26:19.285Zresolver v4:
5672023-10-26T23:26:19.315Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5682023-10-26T23:26:19.407Zresolver v6:
5692023-10-26T23:26:19.445Znat v4:
5702023-10-26T23:26:19.483Znat_v6:
5712023-10-26T23:26:19.931Zport_mac:
5722023-10-26T23:26:19.975Zproxy arp:
5732023-10-26T23:26:20.007Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5742023-10-26T23:26:20.036Z+ success 'softnpu zone exists'
5752023-10-26T23:26:20.062Z+ set +x
5762023-10-26T23:26:20.088Zsoftnpu zone exists
5772023-10-26T23:26:20.118Z
5782023-10-26T23:26:20.147Zreal 46.032969224
5792023-10-26T23:26:20.179Zuser 0.706109966
5802023-10-26T23:26:20.211Zsys 1.565804571
5812023-10-26T23:26:20.242Ztrap 0.000092156
5822023-10-26T23:26:20.269Ztflt 0.000222957
5832023-10-26T23:26:20.702Zdflt 0.001230721
5842023-10-26T23:26:20.729Zkflt 0.000000000
5852023-10-26T23:26:21.165Zlock 36:17.809490231
5862023-10-26T23:26:21.200Zslp 2:17.469999251
5872023-10-26T23:26:21.234Zlat 0.059732831
5882023-10-26T23:26:21.266Zstop 0.357942360
5892023-10-26T23:26:21.292Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5902023-10-26T23:26:21.324Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5912023-10-26T23:26:21.354Z+ SILO_NAME=recovery
5922023-10-26T23:26:21.384Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-10-26T23:26:21.414Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5942023-10-26T23:26:21.440Z+ sed -i~ '
5952023-10-26T23:26:21.470Z /^external_dns_ips/c\
5962023-10-26T23:26:21.501Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5972023-10-26T23:26:21.531Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5982023-10-26T23:26:21.557Z /^first/c\
5992023-10-26T23:26:21.581Zfirst = "10.151.1.164"
6002023-10-26T23:26:21.605Z /^last/c\
6012023-10-26T23:26:21.635Zlast = "10.151.1.173"
6022023-10-26T23:26:21.666Z }
6032023-10-26T23:26:21.697Z /^\[rack_network_config/,/^$/ {
6042023-10-26T23:26:21.727Z /^infra_ip_first/c\
6052023-10-26T23:26:21.753Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6062023-10-26T23:26:21.784Zinfra_ip_first = "10.151.1.174"
6072023-10-26T23:26:21.821Z+++ pkg/config-rss.toml Thu Oct 26 23:26:18 2023
6082023-10-26T23:26:21.847Z /^infra_ip_last/c\
6092023-10-26T23:26:21.878Z@@ -41,7 +41,7 @@
6102023-10-26T23:26:21.904Zinfra_ip_last = "10.151.1.174"
6112023-10-26T23:26:21.929Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6122023-10-26T23:26:22.501Z }
6132023-10-26T23:26:22.526Z # must be contained in one of the "internal services" IP Pool ranges listed
6142023-10-26T23:26:22.550Z /^\[\[rack_network_config.ports/,/^$/ {
6152023-10-26T23:26:22.575Z # below.
6162023-10-26T23:26:22.599Z /^routes/c\
6172023-10-26T23:26:22.625Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6182023-10-26T23:26:22.654Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6192023-10-26T23:26:22.690Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6202023-10-26T23:26:22.717Z /^addresses/c\
6212023-10-26T23:26:22.742Z
6222023-10-26T23:26:22.767Zaddresses = \["10.151.1.174/32"\]
6232023-10-26T23:26:22.791Z # Initial TLS certificates for the external API
6242023-10-26T23:26:22.816Z }
6252023-10-26T23:26:22.845Z #
6262023-10-26T23:26:22.870Z' pkg/config-rss.toml
6272023-10-26T23:26:22.899Z@@ -77,8 +77,8 @@
6282023-10-26T23:26:22.924Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6292023-10-26T23:26:22.949Z #
6302023-10-26T23:26:22.973Z+ true
6312023-10-26T23:26:23.021Z # For more on this and what to put here, see docs/how-to-run.adoc.
6322023-10-26T23:26:23.048Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6332023-10-26T23:26:23.087Z [[internal_services_ip_pool_ranges]]
6342023-10-26T23:26:23.127Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6352023-10-26T23:26:23.153Z-first = "192.168.1.20"
6362023-10-26T23:26:23.178Z+ TLS_NAME=recovery.sys.oxide.test
6372023-10-26T23:26:23.216Z-last = "192.168.1.29"
6382023-10-26T23:26:23.244Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6392023-10-26T23:26:23.277Z+first = "10.151.1.164"
6402023-10-26T23:26:23.309Z....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+......+...+.....+......+...+.......+...+......+......+........+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.........+.....+...+...+.....................+.+.........+.....+.+...+......+.....+.+........+....+...........+....+........+....+.....+..........+......+.....+...+......+.+..+.....................+....+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6412023-10-26T23:26:23.338Z+last = "10.151.1.173"
6422023-10-26T23:26:23.368Z.+....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....................+...+......+.+............+..+...+..........+..+....+..+...+.+........+.......+...+............+......+..+...+...+.+........+...+....+.....+.+...+...........+.+..+...+.......+..+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+.+.....+.............+..+.......+........+...+....+...+..+......+....+.................+....+..+...+...............+.+...+...+.....+.........+.+..+............+.....................+.......+..+.+.........+.........+......+..............+...................+.....+...+.........+.............+......+......+.....................+........+.+....................+.............+............+...+........+..........+...+......+.....................+.....+....+...+...+.....+.........................+..+...+......................+.....+............+.+...............+..+......+......+......+..................+....+...+.........+...+.....+......+.+......+...+...............+...........+.+...............+..............+.+.....+.........+....+..+.......+......+..+...+....+.................+....+....................+.......+...............+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6432023-10-26T23:26:23.402Z
6442023-10-26T23:26:23.434Z-----
6452023-10-26T23:26:23.466Z # TODO - this configuration is subject to change going forward. Ultimately these
6462023-10-26T23:26:23.502Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6472023-10-26T23:26:23.539Z # parameters should be provided to the control plane via wicket, but we need to
6482023-10-26T23:26:23.571Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6492023-10-26T23:26:23.607Z@@ -92,8 +92,8 @@
6502023-10-26T23:26:23.635Z+ rmdir pkg
6512023-10-26T23:26:23.661Z # A range of IP addresses used by Boundary Services on the external network. In
6522023-10-26T23:26:23.705Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6532023-10-26T23:26:23.731Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6542023-10-26T23:26:23.758Z+ pfexec mkdir -p /usr/oxide
6552023-10-26T23:26:23.788Z # With softnpu, only one address is used.
6562023-10-26T23:26:23.823Z+ pfexec rm -f /usr/oxide/tcpproxy
6572023-10-26T23:26:23.854Z-infra_ip_first = "192.168.1.30"
6582023-10-26T23:26:23.884Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6592023-10-26T23:26:23.909Z-infra_ip_last = "192.168.1.30"
6602023-10-26T23:26:23.958Z+ pfexec chmod +x /usr/oxide/tcpproxy
6612023-10-26T23:26:23.995Z+infra_ip_first = "10.151.1.174"
6622023-10-26T23:26:24.046Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6632023-10-26T23:26:24.071Z+infra_ip_last = "10.151.1.174"
6642023-10-26T23:26:24.099Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6652023-10-26T23:26:24.124Z
6662023-10-26T23:26:24.150Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6672023-10-26T23:26:24.182Z # Configurations for BGP routers to run on the scrimlets.
6682023-10-26T23:26:24.211Z+ OMICRON_NO_UNINSTALL=1
6692023-10-26T23:26:24.237Z bgp = []
6702023-10-26T23:26:24.269Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6712023-10-26T23:26:24.297Z@@ -101,9 +101,9 @@
6722023-10-26T23:26:24.322ZOct 26 23:26:19.714 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6732023-10-26T23:26:24.347Z # You can configure multiple uplinks by repeating the following stanza
6742023-10-26T23:26:24.373ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6752023-10-26T23:26:24.400Z [[rack_network_config.ports]]
6762023-10-26T23:26:24.431ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6772023-10-26T23:26:24.465Z # Routes associated with this port.
6782023-10-26T23:26:24.503ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6792023-10-26T23:26:24.531Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6802023-10-26T23:26:24.558ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6812023-10-26T23:26:24.586Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6822023-10-26T23:26:25.015ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6832023-10-26T23:26:25.042Z # Addresses associated with this port.
6842023-10-26T23:26:25.071ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6852023-10-26T23:26:25.505Z-addresses = ["192.168.1.30/32"]
6862023-10-26T23:26:25.533ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6872023-10-26T23:26:25.562Z+addresses = ["10.151.1.174/32"]
6882023-10-26T23:26:25.587ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6892023-10-26T23:26:25.615Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6902023-10-26T23:26:25.659ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6912023-10-26T23:26:25.695Z port = "qsfp0"
6922023-10-26T23:26:25.724ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6932023-10-26T23:26:25.760Z # The speed of this port.
6942023-10-26T23:26:25.800ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6952023-10-26T23:26:25.830Za pkg/config-rss.toml 7K
6962023-10-26T23:26:25.866ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6972023-10-26T23:26:25.899Za pkg/initial-tls-cert.pem 2K
6982023-10-26T23:26:25.937ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6992023-10-26T23:26:25.970Za pkg/initial-tls-key.pem 4K
7002023-10-26T23:26:26.000ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7012023-10-26T23:26:26.470ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7022023-10-26T23:26:26.904ZOct 26 23:26:19.716 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7032023-10-26T23:26:26.945ZOct 26 23:26:19.948 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7042023-10-26T23:26:26.979ZOct 26 23:26:19.983 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7052023-10-26T23:26:27.014ZOct 26 23:26:20.048 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7062023-10-26T23:26:27.048Z
7072023-10-26T23:26:27.082Zreal 0.471639339
7082023-10-26T23:26:27.319Zuser 0.097329724
7092023-10-26T23:26:27.351Zsys 1.966578889
7102023-10-26T23:26:27.384Ztrap 0.000001800
7112023-10-26T23:26:27.417Ztflt 0.000014760
7122023-10-26T23:26:27.445Zdflt 0.000793569
7132023-10-26T23:26:27.473Zkflt 0.000005330
7142023-10-26T23:26:27.505Zlock 42.264585488
7152023-10-26T23:26:27.541Zslp 0.674254611
7162023-10-26T23:26:27.574Zlat 0.002818993
7172023-10-26T23:26:27.604Zstop 0.222304200
7182023-10-26T23:26:27.631Z+ retry=0
7192023-10-26T23:26:27.663Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7202023-10-26T23:26:27.788Z+ [[ 0 -gt 30 ]]
7212023-10-26T23:26:27.834Z+ sleep 1
7222023-10-26T23:26:28.300Z+ retry=1
7232023-10-26T23:26:29.199Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7242023-10-26T23:26:29.307Z+ [[ 1 -gt 30 ]]
7252023-10-26T23:26:29.347Z+ sleep 1
7262023-10-26T23:26:29.390Z+ retry=2
7272023-10-26T23:26:29.428Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7282023-10-26T23:26:47.101Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7292023-10-26T23:26:47.153Zlocal v6:
7302023-10-26T23:26:47.177Zfe80::c80:8bff:feb4:2ccb
7312023-10-26T23:26:47.201Zfd00:99::1
7322023-10-26T23:26:47.229Zlocal v4:
7332023-10-26T23:26:47.265Z10.151.1.174
7342023-10-26T23:26:47.294Zrouter v6:
7352023-10-26T23:26:47.319Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::9cca:5aff:fe42:724c (1)
7362023-10-26T23:26:47.348Zfd00:1122:3344:101::/64 -> fe80::9cca:5aff:fe42:724c (1)
7372023-10-26T23:26:47.383Zrouter v4_idx:
7382023-10-26T23:26:47.415Z0.0.0.0/0 -> 0
7392023-10-26T23:26:47.455Zrouter v4_routes:
7402023-10-26T23:26:47.488Z0 -> 10.151.1.100 (2)
7412023-10-26T23:26:47.515Zresolver v4:
7422023-10-26T23:26:47.953Z10.151.1.174 -> b2:5b:8b:c7:87:2f
7432023-10-26T23:26:48.109Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7442023-10-26T23:26:48.152Zresolver v6:
7452023-10-26T23:26:48.209Zfe80::9cca:5aff:fe42:724c -> 9e:ca:5a:42:72:4c
7462023-10-26T23:26:48.242Zfe80::c80:8bff:feb4:2ccb -> 0e:80:8b:b4:2c:cb
7472023-10-26T23:26:48.272Znat v4:
7482023-10-26T23:26:48.297Znat_v6:
7492023-10-26T23:26:48.322Zport_mac:
7502023-10-26T23:26:48.359Z1: a8:40:25:7d:45:40
7512023-10-26T23:26:48.390Z1000: a8:40:25:7d:45:3f
7522023-10-26T23:26:48.419Zproxy arp:
7532023-10-26T23:26:48.444Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7542023-10-26T23:26:48.471Z+ retry=0
7552023-10-26T23:26:48.503Z+++ zoneadm list -n
7562023-10-26T23:26:48.558Z+++ grep oxz_ntp
7572023-10-26T23:26:48.608Z++ pfexec svcs -z -Hostate oxide/ntp
7582023-10-26T23:26:48.636Zsvcs: zone '-Hostate': entity not found
7592023-10-26T23:26:48.666Z++ true
7602023-10-26T23:26:48.698Z+ [[ '' != online ]]
7612023-10-26T23:26:48.730Z+ [[ 0 -gt 60 ]]
7622023-10-26T23:26:48.759Z+ sleep 1
7632023-10-26T23:26:48.802Z+ retry=1
7642023-10-26T23:26:48.851Z+++ zoneadm list -n
7652023-10-26T23:26:48.898Z+++ grep oxz_ntp
7662023-10-26T23:26:48.942Z++ pfexec svcs -z -Hostate oxide/ntp
7672023-10-26T23:26:48.983Zsvcs: zone '-Hostate': entity not found
7682023-10-26T23:26:49.120Z++ true
7692023-10-26T23:26:49.155Z+ [[ '' != online ]]
7702023-10-26T23:26:57.466Z+ [[ 1 -gt 60 ]]
7712023-10-26T23:26:58.041Z+ sleep 1
7722023-10-26T23:26:59.227Z+ retry=2
7732023-10-26T23:26:59.259Z+++ zoneadm list -n
7742023-10-26T23:26:59.292Z+++ grep oxz_ntp
7752023-10-26T23:26:59.318Z++ pfexec svcs -z -Hostate oxide/ntp
7762023-10-26T23:26:59.359Zsvcs: zone '-Hostate': entity not found
7772023-10-26T23:26:59.386Z++ true
7782023-10-26T23:26:59.426Z+ [[ '' != online ]]
7792023-10-26T23:26:59.455Z+ [[ 2 -gt 60 ]]
7802023-10-26T23:26:59.492Z+ sleep 1
7812023-10-26T23:26:59.549Z+ retry=3
7822023-10-26T23:26:59.587Z+++ zoneadm list -n
7832023-10-26T23:26:59.638Z+++ grep oxz_ntp
7842023-10-26T23:26:59.674Z++ pfexec svcs -z -Hostate oxide/ntp
7852023-10-26T23:27:00.120Zsvcs: zone '-Hostate': entity not found
7862023-10-26T23:27:00.157Z++ true
7872023-10-26T23:27:00.190Z+ [[ '' != online ]]
7882023-10-26T23:27:00.218Z+ [[ 3 -gt 60 ]]
7892023-10-26T23:27:00.245Z+ sleep 1
7902023-10-26T23:27:00.274Z+ retry=4
7912023-10-26T23:27:00.302Z+++ zoneadm list -n
7922023-10-26T23:27:00.333Z+++ grep oxz_ntp
7932023-10-26T23:27:00.362Z++ pfexec svcs -z -Hostate oxide/ntp
7942023-10-26T23:27:00.396Zsvcs: zone '-Hostate': entity not found
7952023-10-26T23:27:00.437Z++ true
7962023-10-26T23:27:00.482Z+ [[ '' != online ]]
7972023-10-26T23:27:00.534Z+ [[ 4 -gt 60 ]]
7982023-10-26T23:27:00.575Z+ sleep 1
7992023-10-26T23:27:00.620Z+ retry=5
8002023-10-26T23:27:00.691Z+++ zoneadm list -n
8012023-10-26T23:27:00.717Z+++ grep oxz_ntp
8022023-10-26T23:27:00.756Z++ pfexec svcs -z -Hostate oxide/ntp
8032023-10-26T23:27:00.787Zsvcs: zone '-Hostate': entity not found
8042023-10-26T23:27:00.822Z++ true
8052023-10-26T23:27:00.861Z+ [[ '' != online ]]
8062023-10-26T23:27:00.899Z+ [[ 5 -gt 60 ]]
8072023-10-26T23:27:00.927Z+ sleep 1
8082023-10-26T23:27:00.960Z+ retry=6
8092023-10-26T23:27:01.004Z+++ zoneadm list -n
8102023-10-26T23:27:01.063Z+++ grep oxz_ntp
8112023-10-26T23:27:01.122Z++ pfexec svcs -z -Hostate oxide/ntp
8122023-10-26T23:27:01.176Zsvcs: zone '-Hostate': entity not found
8132023-10-26T23:27:01.209Z++ true
8142023-10-26T23:27:01.240Z+ [[ '' != online ]]
8152023-10-26T23:27:01.281Z+ [[ 6 -gt 60 ]]
8162023-10-26T23:27:01.341Z+ sleep 1
8172023-10-26T23:27:01.378Z+ retry=7
8182023-10-26T23:27:01.410Z+++ zoneadm list -n
8192023-10-26T23:27:01.446Z+++ grep oxz_ntp
8202023-10-26T23:27:01.490Z++ pfexec svcs -z -Hostate oxide/ntp
8212023-10-26T23:27:01.567Zsvcs: zone '-Hostate': entity not found
8222023-10-26T23:27:01.612Z++ true
8232023-10-26T23:27:01.665Z+ [[ '' != online ]]
8242023-10-26T23:27:01.725Z+ [[ 7 -gt 60 ]]
8252023-10-26T23:27:01.753Z+ sleep 1
8262023-10-26T23:27:01.781Z+ retry=8
8272023-10-26T23:27:01.820Z+++ zoneadm list -n
8282023-10-26T23:27:02.449Z+++ grep oxz_ntp
8292023-10-26T23:27:02.479Z++ pfexec svcs -z -Hostate oxide/ntp
8302023-10-26T23:27:02.511Zsvcs: zone '-Hostate': entity not found
8312023-10-26T23:27:02.541Z++ true
8322023-10-26T23:27:02.576Z+ [[ '' != online ]]
8332023-10-26T23:27:02.604Z+ [[ 8 -gt 60 ]]
8342023-10-26T23:27:02.640Z+ sleep 1
8352023-10-26T23:27:02.670Z+ retry=9
8362023-10-26T23:27:02.702Z+++ zoneadm list -n
8372023-10-26T23:27:02.780Z+++ grep oxz_ntp
8382023-10-26T23:27:02.819Z++ pfexec svcs -z oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde -Hostate oxide/ntp
8392023-10-26T23:27:02.847Zsvcs: Could not bind to repository server for zone oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde: repository server unavailable
8402023-10-26T23:27:02.878Z++ true
8412023-10-26T23:27:02.909Z+ [[ '' != online ]]
8422023-10-26T23:27:02.944Z+ [[ 9 -gt 60 ]]
8432023-10-26T23:27:02.974Z+ sleep 1
8442023-10-26T23:27:03.003Z+ retry=10
8452023-10-26T23:27:03.030Z+++ zoneadm list -n
8462023-10-26T23:27:03.470Z+++ grep oxz_ntp
8472023-10-26T23:27:03.497Z++ pfexec svcs -z oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde -Hostate oxide/ntp
8482023-10-26T23:27:03.523Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8492023-10-26T23:27:03.553Z++ true
8502023-10-26T23:27:03.590Z+ [[ '' != online ]]
8512023-10-26T23:27:03.630Z+ [[ 10 -gt 60 ]]
8522023-10-26T23:27:03.660Z+ sleep 1
8532023-10-26T23:27:03.689Z+ retry=11
8542023-10-26T23:27:03.717Z+++ zoneadm list -n
8552023-10-26T23:27:03.743Z+++ grep oxz_ntp
8562023-10-26T23:27:03.775Z++ pfexec svcs -z oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde -Hostate oxide/ntp
8572023-10-26T23:27:03.807Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8582023-10-26T23:27:03.841Z++ true
8592023-10-26T23:27:03.881Z+ [[ '' != online ]]
8602023-10-26T23:27:03.913Z+ [[ 11 -gt 60 ]]
8612023-10-26T23:27:03.949Z+ sleep 1
8622023-10-26T23:27:03.988Z+ retry=12
8632023-10-26T23:27:04.025Z+++ zoneadm list -n
8642023-10-26T23:27:04.055Z+++ grep oxz_ntp
8652023-10-26T23:27:04.088Z++ pfexec svcs -z oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde -Hostate oxide/ntp
8662023-10-26T23:27:04.115Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8672023-10-26T23:27:04.140Z++ true
8682023-10-26T23:27:04.167Z+ [[ '' != online ]]
8692023-10-26T23:27:04.194Z+ [[ 12 -gt 60 ]]
8702023-10-26T23:27:04.221Z+ sleep 1
8712023-10-26T23:27:04.248Z+ retry=13
8722023-10-26T23:27:04.278Z+++ zoneadm list -n
8732023-10-26T23:27:04.306Z+++ grep oxz_ntp
8742023-10-26T23:27:04.343Z++ pfexec svcs -z oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde -Hostate oxide/ntp
8752023-10-26T23:27:04.378Z+ [[ disabled != online ]]
8762023-10-26T23:27:04.412Z+ [[ 13 -gt 60 ]]
8772023-10-26T23:27:04.445Z+ sleep 1
8782023-10-26T23:27:04.483Z+ retry=14
8792023-10-26T23:27:04.514Z+++ zoneadm list -n
8802023-10-26T23:27:04.542Z+++ grep oxz_ntp
8812023-10-26T23:27:04.574Z++ pfexec svcs -z oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde -Hostate oxide/ntp
8822023-10-26T23:27:04.602Z+ [[ disabled != online ]]
8832023-10-26T23:27:04.626Z+ [[ 14 -gt 60 ]]
8842023-10-26T23:27:04.650Z+ sleep 1
8852023-10-26T23:27:04.674Z+ retry=15
8862023-10-26T23:27:04.700Z+++ zoneadm list -n
8872023-10-26T23:27:04.724Z+++ grep oxz_ntp
8882023-10-26T23:27:04.755Z++ pfexec svcs -z oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde -Hostate oxide/ntp
8892023-10-26T23:27:04.780Z+ [[ disabled != online ]]
8902023-10-26T23:27:04.806Z+ [[ 15 -gt 60 ]]
8912023-10-26T23:27:04.832Z+ sleep 1
8922023-10-26T23:27:04.858Z+ retry=16
8932023-10-26T23:27:04.883Z+++ zoneadm list -n
8942023-10-26T23:27:04.910Z+++ grep oxz_ntp
8952023-10-26T23:27:04.945Z++ pfexec svcs -z oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde -Hostate oxide/ntp
8962023-10-26T23:27:04.985Z+ [[ disabled != online ]]
8972023-10-26T23:27:05.012Z+ [[ 16 -gt 60 ]]
8982023-10-26T23:27:05.043Z+ sleep 1
8992023-10-26T23:27:05.072Z+ retry=17
9002023-10-26T23:27:05.102Z+++ zoneadm list -n
9012023-10-26T23:27:05.130Z+++ grep oxz_ntp
9022023-10-26T23:27:05.161Z++ pfexec svcs -z oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde -Hostate oxide/ntp
9032023-10-26T23:27:05.191Z+ [[ disabled != online ]]
9042023-10-26T23:27:05.222Z+ [[ 17 -gt 60 ]]
9052023-10-26T23:27:05.253Z+ sleep 1
9062023-10-26T23:27:05.561Z+ retry=18
9072023-10-26T23:27:05.588Z+++ zoneadm list -n
9082023-10-26T23:27:05.619Z+++ grep oxz_ntp
9092023-10-26T23:27:05.647Z++ pfexec svcs -z oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde -Hostate oxide/ntp
9102023-10-26T23:27:05.676Z+ [[ disabled != online ]]
9112023-10-26T23:27:05.704Z+ [[ 18 -gt 60 ]]
9122023-10-26T23:27:05.733Z+ sleep 1
9132023-10-26T23:27:06.586Z+ retry=19
9142023-10-26T23:27:06.620Z+++ zoneadm list -n
9152023-10-26T23:27:06.651Z+++ grep oxz_ntp
9162023-10-26T23:27:06.685Z++ pfexec svcs -z oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde -Hostate oxide/ntp
9172023-10-26T23:27:06.715Z+ [[ disabled != online ]]
9182023-10-26T23:27:06.751Z+ [[ 19 -gt 60 ]]
9192023-10-26T23:27:06.797Z+ sleep 1
9202023-10-26T23:27:07.606Z+ retry=20
9212023-10-26T23:27:07.637Z+++ zoneadm list -n
9222023-10-26T23:27:07.662Z+++ grep oxz_ntp
9232023-10-26T23:27:07.690Z++ pfexec svcs -z oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde -Hostate oxide/ntp
9242023-10-26T23:27:07.723Z+ [[ disabled != online ]]
9252023-10-26T23:27:07.754Z+ [[ 20 -gt 60 ]]
9262023-10-26T23:27:07.789Z+ sleep 1
 
9272023-10-26T23:27:07.976Zbuskin console login: Oct 26 23:27:07 bukin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9282023-10-26T23:27:08.627Z+ retry=21
9292023-10-26T23:27:08.653Z+++ zoneadm list -n
9302023-10-26T23:27:08.679Z+++ grep oxz_ntp
9312023-10-26T23:27:08.705Z++ pfexec svcs -z oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde -Hostate oxide/ntp
9322023-10-26T23:27:08.731Z+ [[ online != online ]]
9332023-10-26T23:27:08.762Z+ echo 'Waited for chrony: 21s'
9342023-10-26T23:27:08.788ZWaited for chrony: 21s
9352023-10-26T23:27:08.813Z+ export RUST_BACKTRACE=1
9362023-10-26T23:27:08.841Z+ RUST_BACKTRACE=1
9372023-10-26T23:27:08.871Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9382023-10-26T23:27:08.901Z+ ./tests/bootstrap
9392023-10-26T23:27:08.934ZUsing external DNS server at 10.151.1.164:53
9402023-10-26T23:27:08.965Z2023-10-26 23:27:08.954272990 UTC: attempting to log into API
9412023-10-26T23:27:23.715Z2023-10-26 23:27:23.280487924 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9422023-10-26T23:27:25.040Z2023-10-26 23:27:24.281266583 UTC: attempting to log into API
9432023-10-26T23:27:39.749Z2023-10-26 23:27:39.314909040 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9442023-10-26T23:27:40.754Z2023-10-26 23:27:40.315974716 UTC: attempting to log into API
9452023-10-26T23:27:55.792Z2023-10-26 23:27:55.779138888 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9462023-10-26T23:27:56.791Z2023-10-26 23:27:56.781142713 UTC: attempting to log into API
9472023-10-26T23:28:11.836Z2023-10-26 23:28:11.825511124 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9482023-10-26T23:28:12.839Z2023-10-26 23:28:12.826366206 UTC: attempting to log into API
 
9492023-10-26T23:28:24.083ZOct 26 23:28:23 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9502023-10-26T23:28:30.435Z2023-10-26 23:28:27.859219527 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9512023-10-26T23:28:32.078Z2023-10-26 23:28:28.860317244 UTC: attempting to log into API
 
9522023-10-26T23:28:53.027ZOct 26 23:28:32 bukin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9532023-10-26T23:28:54.518ZOct 26 23:28:38 bukin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
9542023-10-26T23:28:54.817ZOct 26 23:28:44 bukin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9552023-10-26T23:28:54.846Z2023-10-26 23:28:43.893286126 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9562023-10-26T23:28:54.938Z2023-10-26 23:28:44.894381041 UTC: attempting to log into API
 
9572023-10-26T23:28:54.995ZOct 26 23:28:50 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9582023-10-26T23:29:00.002Z2023-10-26 23:28:59.927696042 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9592023-10-26T23:29:00.943Z2023-10-26 23:29:00.929734648 UTC: attempting to log into API
9602023-10-26T23:29:15.977Z2023-10-26 23:29:15.961437101 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9612023-10-26T23:29:16.974Z2023-10-26 23:29:16.962453907 UTC: attempting to log into API
9622023-10-26T23:29:17.014Z2023-10-26 23:29:17.001896887 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: tcp connect error: Connection refused (os error 146)
9632023-10-26T23:29:18.016Z2023-10-26 23:29:18.003233839 UTC: attempting to log into API
9642023-10-26T23:29:18.048Z2023-10-26 23:29:18.036138248 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: tcp connect error: Connection refused (os error 146)
9652023-10-26T23:29:19.050Z2023-10-26 23:29:19.037300196 UTC: attempting to log into API
9662023-10-26T23:29:19.823Z2023-10-26 23:29:19.793336794 UTC: login succeeded
9672023-10-26T23:29:19.848Zwaiting for nexus to come up...
9682023-10-26T23:29:19.901Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9692023-10-26T23:29:19.950Zensuring datasets are ready...
9702023-10-26T23:29:22.185Zlet's roll.
9712023-10-26T23:29:22.209Z+ rm ./tests/bootstrap
9722023-10-26T23:29:22.289Z+ for test_bin in tests/*
9732023-10-26T23:29:22.314Z+ ./tests/bootstrap-ce25c26538ce669c
9742023-10-26T23:29:22.341Z
9752023-10-26T23:29:22.372Zrunning 0 tests
9762023-10-26T23:29:22.404Z
9772023-10-26T23:29:22.429Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9782023-10-26T23:29:22.453Z
9792023-10-26T23:29:22.479Z+ for test_bin in tests/*
9802023-10-26T23:29:22.510Z+ ./tests/end_to_end_tests-bcb3df1a66c8d052
9812023-10-26T23:29:22.539Z
9822023-10-26T23:29:22.563Zrunning 2 tests
9832023-10-26T23:29:22.591Ztest no_spoof::no_spoof_here ... ok
9842023-10-26T23:30:22.326Ztest instance_launch::instance_launch has been running for over 60 seconds
9852023-10-26T23:32:18.885Ztest instance_launch::instance_launch ... ok
9862023-10-26T23:32:18.910Z
9872023-10-26T23:32:18.935Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 176.56s
9882023-10-26T23:32:18.959Z
9892023-10-26T23:32:18.983Z+ _exit_trap
9902023-10-26T23:32:19.008Z+ local status=0
9912023-10-26T23:32:19.046Z+ [[ 0 -eq 0 ]]
9922023-10-26T23:32:19.076Z+ exit 0
9932023-10-26T23:32:19.102Zprocess exited: duration 415103 ms, exit code 0
 
9942023-10-26T23:32:19.183Zfound 90 output files
9952023-10-26T23:32:19.240Zuploading: /var/svc/log/oxide-sled-agent:default.log (423558 bytes)
9962023-10-26T23:32:20.529Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9972023-10-26T23:32:20.564Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2c965ce8-8657-43be-b04b-5e4343aa4746/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9982023-10-26T23:32:21.618Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2c965ce8-8657-43be-b04b-5e4343aa4746/root/var/svc/log/oxide-cockroachdb:default.log
9992023-10-26T23:32:21.650Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2c965ce8-8657-43be-b04b-5e4343aa4746/root/var/svc/log/oxide-cockroachdb:default.log.0 (5342 bytes)
10002023-10-26T23:32:22.714Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2c965ce8-8657-43be-b04b-5e4343aa4746/root/var/svc/log/oxide-cockroachdb:default.log.0
10012023-10-26T23:32:22.739Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_efc3e930-d6f2-49ed-8bdf-43fa4f0dde0a/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10022023-10-26T23:32:23.794Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_efc3e930-d6f2-49ed-8bdf-43fa4f0dde0a/root/var/svc/log/oxide-cockroachdb:default.log
10032023-10-26T23:32:23.821Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_efc3e930-d6f2-49ed-8bdf-43fa4f0dde0a/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10042023-10-26T23:32:24.896Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_efc3e930-d6f2-49ed-8bdf-43fa4f0dde0a/root/var/svc/log/oxide-cockroachdb:default.log.0
10052023-10-26T23:32:24.922Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7d15eb46-7f46-419a-9ddd-6c4d9c405b1e/root/var/svc/log/oxide-crucible-agent:default.log (4322 bytes)
10062023-10-26T23:32:25.998Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7d15eb46-7f46-419a-9ddd-6c4d9c405b1e/root/var/svc/log/oxide-crucible-agent:default.log
10072023-10-26T23:32:26.023Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7d15eb46-7f46-419a-9ddd-6c4d9c405b1e/root/var/svc/log/oxide-crucible-agent:default.log.0 (9363 bytes)
10082023-10-26T23:32:27.182Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7d15eb46-7f46-419a-9ddd-6c4d9c405b1e/root/var/svc/log/oxide-crucible-agent:default.log.0
10092023-10-26T23:32:27.215Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7d15eb46-7f46-419a-9ddd-6c4d9c405b1e/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10102023-10-26T23:32:28.269Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7d15eb46-7f46-419a-9ddd-6c4d9c405b1e/root/var/svc/log/oxide-crucible-downstairs:default.log
10112023-10-26T23:32:28.295Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7d15eb46-7f46-419a-9ddd-6c4d9c405b1e/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10122023-10-26T23:32:29.380Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7d15eb46-7f46-419a-9ddd-6c4d9c405b1e/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10132023-10-26T23:32:29.723Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7d15eb46-7f46-419a-9ddd-6c4d9c405b1e/root/var/svc/log/oxide-crucible-downstairs:downstairs-b877c01b-3f02-466f-b533-542c67d7450e.log (1006 bytes)
10142023-10-26T23:32:30.740Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7d15eb46-7f46-419a-9ddd-6c4d9c405b1e/root/var/svc/log/oxide-crucible-downstairs:downstairs-b877c01b-3f02-466f-b533-542c67d7450e.log
10152023-10-26T23:32:31.586Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7d15eb46-7f46-419a-9ddd-6c4d9c405b1e/root/var/svc/log/oxide-crucible-downstairs:downstairs-b877c01b-3f02-466f-b533-542c67d7450e.log.0 (4531 bytes)
10162023-10-26T23:32:34.585Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7d15eb46-7f46-419a-9ddd-6c4d9c405b1e/root/var/svc/log/oxide-crucible-downstairs:downstairs-b877c01b-3f02-466f-b533-542c67d7450e.log.0
10172023-10-26T23:32:34.861Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_2e0a6ce8-51d5-46a9-abe2-c2de636c37a6/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10182023-10-26T23:32:36.877Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_2e0a6ce8-51d5-46a9-abe2-c2de636c37a6/root/var/svc/log/oxide-crucible-pantry:default.log
10192023-10-26T23:32:37.880Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_2e0a6ce8-51d5-46a9-abe2-c2de636c37a6/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10202023-10-26T23:32:59.780Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_2e0a6ce8-51d5-46a9-abe2-c2de636c37a6/root/var/svc/log/oxide-crucible-pantry:default.log.0
10212023-10-26T23:33:00.388Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_d99bac1a-516c-49c9-af2e-02abff12baf6/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10222023-10-26T23:33:01.630Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_d99bac1a-516c-49c9-af2e-02abff12baf6/root/var/svc/log/oxide-crucible-pantry:default.log
10232023-10-26T23:33:01.936Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_d99bac1a-516c-49c9-af2e-02abff12baf6/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
10242023-10-26T23:33:03.327Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_d99bac1a-516c-49c9-af2e-02abff12baf6/root/var/svc/log/oxide-crucible-pantry:default.log.0
10252023-10-26T23:33:03.405Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_e845b55d-8cff-4610-8065-4ee8ea7e7ba4/root/var/svc/log/oxide-internal_dns:default.log (7507 bytes)
10262023-10-26T23:33:03.507Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_e845b55d-8cff-4610-8065-4ee8ea7e7ba4/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 7507 -> 11260
10272023-10-26T23:33:04.654Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_e845b55d-8cff-4610-8065-4ee8ea7e7ba4/root/var/svc/log/oxide-internal_dns:default.log
10282023-10-26T23:33:04.773Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_e845b55d-8cff-4610-8065-4ee8ea7e7ba4/root/var/svc/log/oxide-internal_dns:default.log.0 (107171 bytes)
10292023-10-26T23:33:05.939Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_e845b55d-8cff-4610-8065-4ee8ea7e7ba4/root/var/svc/log/oxide-internal_dns:default.log.0
10302023-10-26T23:33:05.986Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_73a17dc1-26de-4b53-af46-0928dcb5d3cb/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10312023-10-26T23:33:07.832Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_73a17dc1-26de-4b53-af46-0928dcb5d3cb/root/var/svc/log/oxide-cockroachdb:default.log
10322023-10-26T23:33:07.862Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_73a17dc1-26de-4b53-af46-0928dcb5d3cb/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10332023-10-26T23:33:08.945Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_73a17dc1-26de-4b53-af46-0928dcb5d3cb/root/var/svc/log/oxide-cockroachdb:default.log.0
10342023-10-26T23:33:08.970Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_dc194f12-f8ed-4be4-bd1d-0e470804fb78/root/var/svc/log/oxide-external_dns:default.log (7557 bytes)
10352023-10-26T23:33:09.018Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_dc194f12-f8ed-4be4-bd1d-0e470804fb78/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 7557 -> 11334
10362023-10-26T23:33:10.092Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_dc194f12-f8ed-4be4-bd1d-0e470804fb78/root/var/svc/log/oxide-external_dns:default.log
10372023-10-26T23:33:10.117Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_dc194f12-f8ed-4be4-bd1d-0e470804fb78/root/var/svc/log/oxide-external_dns:default.log.0 (8058 bytes)
10382023-10-26T23:33:11.213Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_dc194f12-f8ed-4be4-bd1d-0e470804fb78/root/var/svc/log/oxide-external_dns:default.log.0
10392023-10-26T23:33:11.238Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_cee4e80d-a5d0-46c1-9be8-4bc18c6183bb/root/var/svc/log/oxide-nexus:default.log (125318 bytes)
10402023-10-26T23:33:11.400Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_cee4e80d-a5d0-46c1-9be8-4bc18c6183bb/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 125318 -> 186536
10412023-10-26T23:33:12.566Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_cee4e80d-a5d0-46c1-9be8-4bc18c6183bb/root/var/svc/log/oxide-nexus:default.log
10422023-10-26T23:33:12.591Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_cee4e80d-a5d0-46c1-9be8-4bc18c6183bb/root/var/svc/log/oxide-nexus:default.log.0 (375256 bytes)
10432023-10-26T23:33:13.859Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_cee4e80d-a5d0-46c1-9be8-4bc18c6183bb/root/var/svc/log/oxide-nexus:default.log.0
10442023-10-26T23:33:13.901Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_628c8410-c718-457b-b2cb-43b2aba9c7ef/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10452023-10-26T23:33:14.962Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_628c8410-c718-457b-b2cb-43b2aba9c7ef/root/var/svc/log/oxide-crucible-agent:default.log
10462023-10-26T23:33:14.987Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_628c8410-c718-457b-b2cb-43b2aba9c7ef/root/var/svc/log/oxide-crucible-agent:default.log.0 (14116 bytes)
10472023-10-26T23:33:16.085Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_628c8410-c718-457b-b2cb-43b2aba9c7ef/root/var/svc/log/oxide-crucible-agent:default.log.0
10482023-10-26T23:33:16.136Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_628c8410-c718-457b-b2cb-43b2aba9c7ef/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10492023-10-26T23:33:17.177Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_628c8410-c718-457b-b2cb-43b2aba9c7ef/root/var/svc/log/oxide-crucible-downstairs:default.log
10502023-10-26T23:33:17.207Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_628c8410-c718-457b-b2cb-43b2aba9c7ef/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10512023-10-26T23:33:18.284Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_628c8410-c718-457b-b2cb-43b2aba9c7ef/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10522023-10-26T23:33:18.309Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_628c8410-c718-457b-b2cb-43b2aba9c7ef/root/var/svc/log/oxide-crucible-downstairs:downstairs-0544f6bf-e57d-4cb2-9d6c-ba2c6c52591e.log (0 bytes)
10532023-10-26T23:33:19.356Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_628c8410-c718-457b-b2cb-43b2aba9c7ef/root/var/svc/log/oxide-crucible-downstairs:downstairs-0544f6bf-e57d-4cb2-9d6c-ba2c6c52591e.log
10542023-10-26T23:33:19.382Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_628c8410-c718-457b-b2cb-43b2aba9c7ef/root/var/svc/log/oxide-crucible-downstairs:downstairs-0544f6bf-e57d-4cb2-9d6c-ba2c6c52591e.log.0 (3048 bytes)
10552023-10-26T23:33:20.461Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_628c8410-c718-457b-b2cb-43b2aba9c7ef/root/var/svc/log/oxide-crucible-downstairs:downstairs-0544f6bf-e57d-4cb2-9d6c-ba2c6c52591e.log.0
10562023-10-26T23:33:20.487Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_96996609-bab0-4916-b204-67c1cf4459b8/root/var/svc/log/oxide-nexus:default.log (125318 bytes)
10572023-10-26T23:33:20.674Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_96996609-bab0-4916-b204-67c1cf4459b8/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 125318 -> 187145
10582023-10-26T23:33:21.762Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_96996609-bab0-4916-b204-67c1cf4459b8/root/var/svc/log/oxide-nexus:default.log
10592023-10-26T23:33:21.789Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_96996609-bab0-4916-b204-67c1cf4459b8/root/var/svc/log/oxide-nexus:default.log.0 (369792 bytes)
10602023-10-26T23:33:23.179Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_96996609-bab0-4916-b204-67c1cf4459b8/root/var/svc/log/oxide-nexus:default.log.0
10612023-10-26T23:33:23.237Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_ddcd0fbc-82cb-4e0d-b5ad-03a01e365b40/root/var/svc/log/oxide-oximeter:default.log (131918 bytes)
10622023-10-26T23:33:23.412Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_ddcd0fbc-82cb-4e0d-b5ad-03a01e365b40/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 131918 -> 185445
10632023-10-26T23:33:24.502Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_ddcd0fbc-82cb-4e0d-b5ad-03a01e365b40/root/var/svc/log/oxide-oximeter:default.log
10642023-10-26T23:33:24.526Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_ddcd0fbc-82cb-4e0d-b5ad-03a01e365b40/root/var/svc/log/oxide-oximeter:default.log.0 (43531 bytes)
10652023-10-26T23:33:25.669Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_ddcd0fbc-82cb-4e0d-b5ad-03a01e365b40/root/var/svc/log/oxide-oximeter:default.log.0
10662023-10-26T23:33:25.700Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_de642e12-5497-4399-b7fc-836c1219952b/root/var/svc/log/oxide-crucible-agent:default.log (4324 bytes)
10672023-10-26T23:33:26.779Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_de642e12-5497-4399-b7fc-836c1219952b/root/var/svc/log/oxide-crucible-agent:default.log
10682023-10-26T23:33:26.805Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_de642e12-5497-4399-b7fc-836c1219952b/root/var/svc/log/oxide-crucible-agent:default.log.0 (9792 bytes)
10692023-10-26T23:33:27.901Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_de642e12-5497-4399-b7fc-836c1219952b/root/var/svc/log/oxide-crucible-agent:default.log.0
10702023-10-26T23:33:27.947Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_de642e12-5497-4399-b7fc-836c1219952b/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10712023-10-26T23:33:28.974Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_de642e12-5497-4399-b7fc-836c1219952b/root/var/svc/log/oxide-crucible-downstairs:default.log
10722023-10-26T23:33:29.000Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_de642e12-5497-4399-b7fc-836c1219952b/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10732023-10-26T23:33:30.077Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_de642e12-5497-4399-b7fc-836c1219952b/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10742023-10-26T23:33:30.101Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_de642e12-5497-4399-b7fc-836c1219952b/root/var/svc/log/oxide-crucible-downstairs:downstairs-f9c60134-1cc6-4f48-954e-512e276abdec.log (1006 bytes)
10752023-10-26T23:33:31.180Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_de642e12-5497-4399-b7fc-836c1219952b/root/var/svc/log/oxide-crucible-downstairs:downstairs-f9c60134-1cc6-4f48-954e-512e276abdec.log
10762023-10-26T23:33:31.211Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_de642e12-5497-4399-b7fc-836c1219952b/root/var/svc/log/oxide-crucible-downstairs:downstairs-f9c60134-1cc6-4f48-954e-512e276abdec.log.0 (4531 bytes)
10772023-10-26T23:33:32.287Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_de642e12-5497-4399-b7fc-836c1219952b/root/var/svc/log/oxide-crucible-downstairs:downstairs-f9c60134-1cc6-4f48-954e-512e276abdec.log.0
10782023-10-26T23:33:32.385Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_c37c8bcb-86da-4980-ace6-5e9fc3deaeb7/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10792023-10-26T23:33:33.437Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_c37c8bcb-86da-4980-ace6-5e9fc3deaeb7/root/var/svc/log/oxide-crucible-pantry:default.log
10802023-10-26T23:33:33.462Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_c37c8bcb-86da-4980-ace6-5e9fc3deaeb7/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10812023-10-26T23:33:34.534Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_c37c8bcb-86da-4980-ace6-5e9fc3deaeb7/root/var/svc/log/oxide-crucible-pantry:default.log.0
10822023-10-26T23:33:34.560Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_cc0d8abf-51b6-4ccb-8b0e-c274dded4bad/root/var/svc/log/oxide-clickhouse:default.log (465552 bytes)
10832023-10-26T23:33:34.818Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_cc0d8abf-51b6-4ccb-8b0e-c274dded4bad/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 465552 -> 681816
10842023-10-26T23:33:35.904Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_cc0d8abf-51b6-4ccb-8b0e-c274dded4bad/root/var/svc/log/oxide-clickhouse:default.log
10852023-10-26T23:33:35.930Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_cc0d8abf-51b6-4ccb-8b0e-c274dded4bad/root/var/svc/log/oxide-clickhouse:default.log.0 (171595 bytes)
10862023-10-26T23:33:37.147Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_cc0d8abf-51b6-4ccb-8b0e-c274dded4bad/root/var/svc/log/oxide-clickhouse:default.log.0
10872023-10-26T23:33:37.173Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_acc63c07-0687-415a-8504-19ab24b5d6a1/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10882023-10-26T23:33:38.220Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_acc63c07-0687-415a-8504-19ab24b5d6a1/root/var/svc/log/oxide-cockroachdb:default.log
10892023-10-26T23:33:38.244Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_acc63c07-0687-415a-8504-19ab24b5d6a1/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10902023-10-26T23:33:39.317Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_acc63c07-0687-415a-8504-19ab24b5d6a1/root/var/svc/log/oxide-cockroachdb:default.log.0
10912023-10-26T23:33:39.345Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8dae6aaf-01e5-42a3-b817-48c51d252643/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10922023-10-26T23:33:40.393Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8dae6aaf-01e5-42a3-b817-48c51d252643/root/var/svc/log/oxide-crucible-agent:default.log
10932023-10-26T23:33:40.420Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8dae6aaf-01e5-42a3-b817-48c51d252643/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10942023-10-26T23:33:41.493Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8dae6aaf-01e5-42a3-b817-48c51d252643/root/var/svc/log/oxide-crucible-agent:default.log.0
10952023-10-26T23:33:41.527Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8dae6aaf-01e5-42a3-b817-48c51d252643/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10962023-10-26T23:33:42.976Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8dae6aaf-01e5-42a3-b817-48c51d252643/root/var/svc/log/oxide-crucible-downstairs:default.log
10972023-10-26T23:33:43.002Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8dae6aaf-01e5-42a3-b817-48c51d252643/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10982023-10-26T23:33:44.079Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8dae6aaf-01e5-42a3-b817-48c51d252643/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10992023-10-26T23:33:44.104Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_9d536a02-8a94-4bc8-93cc-e69275a1e994/root/var/svc/log/oxide-internal_dns:default.log (7510 bytes)
11002023-10-26T23:33:44.187Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_9d536a02-8a94-4bc8-93cc-e69275a1e994/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 7510 -> 11266
11012023-10-26T23:33:45.291Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_9d536a02-8a94-4bc8-93cc-e69275a1e994/root/var/svc/log/oxide-internal_dns:default.log
11022023-10-26T23:33:45.316Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_9d536a02-8a94-4bc8-93cc-e69275a1e994/root/var/svc/log/oxide-internal_dns:default.log.0 (109197 bytes)
11032023-10-26T23:33:46.689Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_9d536a02-8a94-4bc8-93cc-e69275a1e994/root/var/svc/log/oxide-internal_dns:default.log.0
11042023-10-26T23:33:46.715Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_f67074af-0363-41c6-bd00-ba73de3dd2ac/root/var/svc/log/oxide-internal_dns:default.log (7504 bytes)
11052023-10-26T23:33:46.776Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_f67074af-0363-41c6-bd00-ba73de3dd2ac/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 7504 -> 11260
11062023-10-26T23:33:47.867Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_f67074af-0363-41c6-bd00-ba73de3dd2ac/root/var/svc/log/oxide-internal_dns:default.log
11072023-10-26T23:33:47.893Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_f67074af-0363-41c6-bd00-ba73de3dd2ac/root/var/svc/log/oxide-internal_dns:default.log.0 (111135 bytes)
11082023-10-26T23:33:49.260Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_f67074af-0363-41c6-bd00-ba73de3dd2ac/root/var/svc/log/oxide-internal_dns:default.log.0
11092023-10-26T23:33:49.291Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_3ff3761f-1737-41b5-a5db-daaefb85ed37/root/var/svc/log/oxide-nexus:default.log (1524022 bytes)
11102023-10-26T23:33:51.225Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_3ff3761f-1737-41b5-a5db-daaefb85ed37/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1524022 -> 1588285
11112023-10-26T23:33:52.299Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_3ff3761f-1737-41b5-a5db-daaefb85ed37/root/var/svc/log/oxide-nexus:default.log
11122023-10-26T23:33:52.338Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_3ff3761f-1737-41b5-a5db-daaefb85ed37/root/var/svc/log/oxide-nexus:default.log.0 (3087800 bytes)
11132023-10-26T23:33:57.178Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_3ff3761f-1737-41b5-a5db-daaefb85ed37/root/var/svc/log/oxide-nexus:default.log.0
11142023-10-26T23:33:57.203Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_cc4f1143-e527-46a9-8b9f-1fd2d7c54afc/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11152023-10-26T23:33:58.208Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_cc4f1143-e527-46a9-8b9f-1fd2d7c54afc/root/var/svc/log/oxide-cockroachdb:default.log
11162023-10-26T23:33:58.234Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_cc4f1143-e527-46a9-8b9f-1fd2d7c54afc/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
11172023-10-26T23:33:59.722Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_cc4f1143-e527-46a9-8b9f-1fd2d7c54afc/root/var/svc/log/oxide-cockroachdb:default.log.0
11182023-10-26T23:33:59.749Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4aadaba2-2039-4998-afbc-a172d2fdaef4/root/var/svc/log/oxide-crucible-agent:default.log (4788 bytes)
11192023-10-26T23:34:00.839Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4aadaba2-2039-4998-afbc-a172d2fdaef4/root/var/svc/log/oxide-crucible-agent:default.log
11202023-10-26T23:34:00.895Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4aadaba2-2039-4998-afbc-a172d2fdaef4/root/var/svc/log/oxide-crucible-agent:default.log.0 (9792 bytes)
11212023-10-26T23:34:01.977Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4aadaba2-2039-4998-afbc-a172d2fdaef4/root/var/svc/log/oxide-crucible-agent:default.log.0
11222023-10-26T23:34:02.007Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4aadaba2-2039-4998-afbc-a172d2fdaef4/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11232023-10-26T23:34:03.055Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4aadaba2-2039-4998-afbc-a172d2fdaef4/root/var/svc/log/oxide-crucible-downstairs:default.log
11242023-10-26T23:34:03.080Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4aadaba2-2039-4998-afbc-a172d2fdaef4/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11252023-10-26T23:34:04.155Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4aadaba2-2039-4998-afbc-a172d2fdaef4/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11262023-10-26T23:34:04.181Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4aadaba2-2039-4998-afbc-a172d2fdaef4/root/var/svc/log/oxide-crucible-downstairs:downstairs-493a78c1-a69c-4999-b15e-84d2f7e7813c.log (1006 bytes)
11272023-10-26T23:34:05.292Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4aadaba2-2039-4998-afbc-a172d2fdaef4/root/var/svc/log/oxide-crucible-downstairs:downstairs-493a78c1-a69c-4999-b15e-84d2f7e7813c.log
11282023-10-26T23:34:05.791Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4aadaba2-2039-4998-afbc-a172d2fdaef4/root/var/svc/log/oxide-crucible-downstairs:downstairs-493a78c1-a69c-4999-b15e-84d2f7e7813c.log.0 (4529 bytes)
11292023-10-26T23:34:06.394Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4aadaba2-2039-4998-afbc-a172d2fdaef4/root/var/svc/log/oxide-crucible-downstairs:downstairs-493a78c1-a69c-4999-b15e-84d2f7e7813c.log.0
11302023-10-26T23:34:06.420Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_892a58f2-9fcc-4a93-b32e-e31a9aedc714/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11312023-10-26T23:34:07.468Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_892a58f2-9fcc-4a93-b32e-e31a9aedc714/root/var/svc/log/oxide-crucible-agent:default.log
11322023-10-26T23:34:07.494Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_892a58f2-9fcc-4a93-b32e-e31a9aedc714/root/var/svc/log/oxide-crucible-agent:default.log.0 (14581 bytes)
11332023-10-26T23:34:08.600Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_892a58f2-9fcc-4a93-b32e-e31a9aedc714/root/var/svc/log/oxide-crucible-agent:default.log.0
11342023-10-26T23:34:08.633Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_892a58f2-9fcc-4a93-b32e-e31a9aedc714/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11352023-10-26T23:34:09.679Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_892a58f2-9fcc-4a93-b32e-e31a9aedc714/root/var/svc/log/oxide-crucible-downstairs:default.log
11362023-10-26T23:34:09.754Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_892a58f2-9fcc-4a93-b32e-e31a9aedc714/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11372023-10-26T23:34:10.841Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_892a58f2-9fcc-4a93-b32e-e31a9aedc714/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11382023-10-26T23:34:10.868Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_892a58f2-9fcc-4a93-b32e-e31a9aedc714/root/var/svc/log/oxide-crucible-downstairs:downstairs-73e13c0d-df82-4a2d-9f73-cba81fe78019.log (0 bytes)
11392023-10-26T23:34:11.915Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_892a58f2-9fcc-4a93-b32e-e31a9aedc714/root/var/svc/log/oxide-crucible-downstairs:downstairs-73e13c0d-df82-4a2d-9f73-cba81fe78019.log
11402023-10-26T23:34:11.940Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_892a58f2-9fcc-4a93-b32e-e31a9aedc714/root/var/svc/log/oxide-crucible-downstairs:downstairs-73e13c0d-df82-4a2d-9f73-cba81fe78019.log.0 (3050 bytes)
11412023-10-26T23:34:13.011Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_892a58f2-9fcc-4a93-b32e-e31a9aedc714/root/var/svc/log/oxide-crucible-downstairs:downstairs-73e13c0d-df82-4a2d-9f73-cba81fe78019.log.0
11422023-10-26T23:34:13.035Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bb057fb0-7b4f-4c67-88bf-6946ffa98f04/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11432023-10-26T23:34:14.085Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bb057fb0-7b4f-4c67-88bf-6946ffa98f04/root/var/svc/log/oxide-crucible-agent:default.log
11442023-10-26T23:34:14.116Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bb057fb0-7b4f-4c67-88bf-6946ffa98f04/root/var/svc/log/oxide-crucible-agent:default.log.0 (3434 bytes)
11452023-10-26T23:34:15.193Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bb057fb0-7b4f-4c67-88bf-6946ffa98f04/root/var/svc/log/oxide-crucible-agent:default.log.0
11462023-10-26T23:34:15.236Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bb057fb0-7b4f-4c67-88bf-6946ffa98f04/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11472023-10-26T23:34:16.280Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bb057fb0-7b4f-4c67-88bf-6946ffa98f04/root/var/svc/log/oxide-crucible-downstairs:default.log
11482023-10-26T23:34:16.356Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bb057fb0-7b4f-4c67-88bf-6946ffa98f04/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11492023-10-26T23:34:19.055Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bb057fb0-7b4f-4c67-88bf-6946ffa98f04/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11502023-10-26T23:34:19.776Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_00701c90-8f29-4e53-a963-aef9ef971fc2/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11512023-10-26T23:34:21.207Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_00701c90-8f29-4e53-a963-aef9ef971fc2/root/var/svc/log/oxide-crucible-agent:default.log
11522023-10-26T23:34:21.661Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_00701c90-8f29-4e53-a963-aef9ef971fc2/root/var/svc/log/oxide-crucible-agent:default.log.0 (13683 bytes)
11532023-10-26T23:34:24.180Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_00701c90-8f29-4e53-a963-aef9ef971fc2/root/var/svc/log/oxide-crucible-agent:default.log.0
11542023-10-26T23:34:24.451Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_00701c90-8f29-4e53-a963-aef9ef971fc2/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11552023-10-26T23:34:26.305Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_00701c90-8f29-4e53-a963-aef9ef971fc2/root/var/svc/log/oxide-crucible-downstairs:default.log
11562023-10-26T23:34:44.558Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_00701c90-8f29-4e53-a963-aef9ef971fc2/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11572023-10-26T23:34:46.058Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_00701c90-8f29-4e53-a963-aef9ef971fc2/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11582023-10-26T23:34:46.083Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_00701c90-8f29-4e53-a963-aef9ef971fc2/root/var/svc/log/oxide-crucible-downstairs:downstairs-5228f2d2-8359-4bbc-821e-94bfd3486428.log (0 bytes)
11592023-10-26T23:34:47.181Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_00701c90-8f29-4e53-a963-aef9ef971fc2/root/var/svc/log/oxide-crucible-downstairs:downstairs-5228f2d2-8359-4bbc-821e-94bfd3486428.log
11602023-10-26T23:34:47.205Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_00701c90-8f29-4e53-a963-aef9ef971fc2/root/var/svc/log/oxide-crucible-downstairs:downstairs-5228f2d2-8359-4bbc-821e-94bfd3486428.log.0 (3049 bytes)
11612023-10-26T23:34:48.232Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_00701c90-8f29-4e53-a963-aef9ef971fc2/root/var/svc/log/oxide-crucible-downstairs:downstairs-5228f2d2-8359-4bbc-821e-94bfd3486428.log.0
11622023-10-26T23:34:48.256Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f2b6dbd9-248e-4034-847d-597cc4b4408b/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11632023-10-26T23:34:49.304Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f2b6dbd9-248e-4034-847d-597cc4b4408b/root/var/svc/log/oxide-crucible-agent:default.log
11642023-10-26T23:34:49.330Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f2b6dbd9-248e-4034-847d-597cc4b4408b/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11652023-10-26T23:34:50.406Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f2b6dbd9-248e-4034-847d-597cc4b4408b/root/var/svc/log/oxide-crucible-agent:default.log.0
11662023-10-26T23:34:50.443Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f2b6dbd9-248e-4034-847d-597cc4b4408b/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11672023-10-26T23:34:51.883Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f2b6dbd9-248e-4034-847d-597cc4b4408b/root/var/svc/log/oxide-crucible-downstairs:default.log
11682023-10-26T23:34:51.909Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f2b6dbd9-248e-4034-847d-597cc4b4408b/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11692023-10-26T23:34:53.403Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f2b6dbd9-248e-4034-847d-597cc4b4408b/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11702023-10-26T23:34:53.843Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_512975aa-38ce-4b05-9689-e6baf08e34b9/root/var/svc/log/oxide-external_dns:default.log (7554 bytes)
11712023-10-26T23:34:54.400Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_512975aa-38ce-4b05-9689-e6baf08e34b9/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 7554 -> 18884
11722023-10-26T23:34:55.275Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_512975aa-38ce-4b05-9689-e6baf08e34b9/root/var/svc/log/oxide-external_dns:default.log
11732023-10-26T23:34:55.300Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_512975aa-38ce-4b05-9689-e6baf08e34b9/root/var/svc/log/oxide-external_dns:default.log.0 (8056 bytes)
11742023-10-26T23:34:56.406Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_512975aa-38ce-4b05-9689-e6baf08e34b9/root/var/svc/log/oxide-external_dns:default.log.0
11752023-10-26T23:34:56.434Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde/root/var/svc/log/oxide-ntp:default.log (0 bytes)
11762023-10-26T23:34:57.490Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde/root/var/svc/log/oxide-ntp:default.log
11772023-10-26T23:34:57.537Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde/root/var/svc/log/oxide-ntp:default.log.0 (1246 bytes)
11782023-10-26T23:34:58.627Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde/root/var/svc/log/oxide-ntp:default.log.0
11792023-10-26T23:34:58.653Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde/root/var/log/chrony/measurements.log (11508 bytes)
11802023-10-26T23:34:58.749Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde/root/var/log/chrony/measurements.log" changed size mid upload: 11508 -> 17536
11812023-10-26T23:34:59.838Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde/root/var/log/chrony/measurements.log
11822023-10-26T23:34:59.893Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde/root/var/log/chrony/statistics.log (5850 bytes)
11832023-10-26T23:34:59.919Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde/root/var/log/chrony/statistics.log" changed size mid upload: 5850 -> 10179
11842023-10-26T23:35:01.002Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde/root/var/log/chrony/statistics.log
11852023-10-26T23:35:01.039Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde/root/var/log/chrony/tracking.log (1722 bytes)
11862023-10-26T23:35:01.067Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde/root/var/log/chrony/tracking.log" changed size mid upload: 1722 -> 1986
11872023-10-26T23:35:02.548Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_33d0a31a-1593-4d06-8fc0-da09a1a74bde/root/var/log/chrony/tracking.log