01HE1V9CS8NNRPJGBGE1RBKJXQ: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HE1VBGKZVPR7TZMF24JGES6S

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-31T04:04:15.192Zjob dependencies complete; ready to run (waiting for 54 m 6 s)
22023-10-31T04:18:09.977Zjob assigned to worker 01HE1Z6XWDSJRTG3Q19FJ6XYX2 (queued for 13 m 54 s)
32023-10-31T04:18:10.012Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-31T04:18:10.062Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-10-31T04:18:10.096Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-10-31T04:18:10.129Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-31T04:18:10.163Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-10-31T04:18:10.196ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-31T04:18:10.230Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-31T04:18:10.265ZOxide Helios Version helios-2.0.22241 64-bit
112023-10-31T04:18:10.297ZConfiguring device.
122023-10-31T04:18:10.330Z#################################################
132023-10-31T04:18:10.364Z#### ############# ##
142023-10-31T04:18:10.397Z### ### ############ ##
152023-10-31T04:18:10.431Z## ### # ## ### ## Oxide Computer Company ##
162023-10-31T04:18:10.464Z## ## ## ### # ### ##
172023-10-31T04:18:10.498Z## # ### #### #### This Station Under ##
182023-10-31T04:18:10.531Z### ### #### # ### Computer Control ##
192023-10-31T04:18:10.565Z#### #### ### ## ##
202023-10-31T04:18:10.598Z##################################################
212023-10-31T04:18:10.634ZHostname: buskin
222023-10-31T04:18:11.074ZLoading smf(7) serice descriptions:
232023-10-31T04:18:12.760Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-13d7796e9c457a2a.gz
242023-10-31T04:18:15.116Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-13d7796e9c457a2a.gz
252023-10-31T04:18:15.144Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-31T04:18:46.885Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-31T04:18:46.920Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cc8f9cfe063d8adb.gz
282023-10-31T04:19:16.943Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cc8f9cfe063d8adb.gz
292023-10-31T04:19:16.968Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-31T04:19:19.806Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-31T04:19:19.839Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-31T04:19:22.884Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-31T04:19:22.910Zdownloading input: /input/package/work/version.txt
342023-10-31T04:19:22.935Zdownloaded input: /input/package/work/version.txt
352023-10-31T04:19:22.961Zdownloading input: /input/package/work/package.tar.gz
362023-10-31T04:19:42.834Zdownloaded input: /input/package/work/package.tar.gz
372023-10-31T04:19:42.862Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-31T04:20:08.873Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-31T04:20:08.901Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-31T04:20:15.941Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-31T04:20:15.965Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-31T04:20:49.158Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-31T04:20:49.181Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-31T04:21:34.295Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-31T04:21:34.320Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-31T04:21:54.806Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-31T04:21:54.830Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-31T04:21:59.216Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-31T04:21:59.240Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-31T04:22:06.970Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-31T04:22:06.995Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-31T04:22:11.680Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-31T04:22:11.705Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-31T04:22:16.540Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-31T04:22:16.565Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-31T04:22:16.662Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-31T04:22:16.687Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-31T04:22:20.241Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-31T04:22:20.274Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-31T04:22:24.375Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-31T04:22:24.399Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-31T04:22:48.728Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-31T04:22:48.756Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-31T04:23:06.425Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-31T04:23:06.450Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-31T04:23:06.526Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-31T04:23:06.551Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-31T04:23:10.024Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-31T04:23:10.049Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-31T04:23:21.607Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-31T04:23:21.631Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-31T04:24:04.276Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-31T04:24:04.302Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-31T04:24:33.587Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-31T04:24:33.611Zstarting task 0: "setup"
762023-10-31T04:24:33.646Z++ uname -s
772023-10-31T04:24:33.670Z+ kern=SunOS
782023-10-31T04:24:33.695Z+ case "$kern" in
792023-10-31T04:24:33.720Z+ groupadd -g 12345 build
802023-10-31T04:24:33.745Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-31T04:24:35.655Z+ zfs create -o mountpoint=/work rpool/work
822023-10-31T04:24:35.680Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-31T04:24:35.704Z+ home_fs=zfs
842023-10-31T04:24:35.729Z+ [[ zfs == autofs ]]
852023-10-31T04:24:35.753Z+ mkdir -p /home/build
862023-10-31T04:24:35.778Z+ chown build:build /home/build /work
872023-10-31T04:24:37.682Z+ chmod 0700 /home/build /work
882023-10-31T04:24:37.734Zprocess exited: duration 4049 ms, exit code 0
 
892023-10-31T04:24:37.783Zstarting task 1: "authentication"
902023-10-31T04:24:37.852Zprocess exited: duration 44 ms, exit code 0
 
912023-10-31T04:24:37.902Zstarting task 2: "build"
922023-10-31T04:24:37.934Z+ trap _exit_trap EXIT
932023-10-31T04:24:37.960Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-31T04:24:38.160Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-31T04:24:38.185Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-31T04:24:38.212Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-31T04:24:38.237Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-31T04:24:38.261Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-31T04:24:38.286Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-31T04:24:38.311Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-31T04:24:38.336Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-31T04:24:38.360Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-31T04:24:38.385Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-31T04:24:38.413Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-31T04:24:38.445Z+ [[ -d /opt/oxide ]]
1062023-10-31T04:24:38.469Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-31T04:24:38.494Z+ pfexec mkdir /opt/oxide
1082023-10-31T04:24:38.519Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-31T04:24:38.543Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-31T04:24:38.568Z+ cd /opt/oxide-underneath
1112023-10-31T04:24:38.593Z+ pfexec tar ceEp@/f - .
1122023-10-31T04:24:38.617Z+ cd /opt/oxide
1132023-10-31T04:24:38.642Z+ pfexec tar xveEp@/f -
1142023-10-31T04:24:38.667Zx ., 0 bytes, 0 tape blocks
1152023-10-31T04:24:38.692Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-31T04:24:38.716Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-31T04:24:38.741Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-10-31T04:24:38.766Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-31T04:24:38.790Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-31T04:24:38.816Z+ pfexec mkdir /opt/oxide/work
1212023-10-31T04:24:38.841Z+ pfexec chown build:build /opt/oxide/work
1222023-10-31T04:24:38.866Z+ cd /opt/oxide/work
1232023-10-31T04:24:38.890Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-31T04:24:39.891ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-31T04:24:39.917Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1262023-10-31T04:24:39.942Zx out/omicron-sled-agent.tar, 122400256 bytes, 239063 tape blocks
1272023-10-31T04:24:39.966Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-31T04:24:39.992Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-10-31T04:24:40.019Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-10-31T04:24:40.043Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-31T04:24:40.068Zx target/release/omicron-package, 41797728 bytes, 81637 tape blocks
1322023-10-31T04:24:40.093Zx tools/create_virtual_hardware.sh, 2974 bytes, 6 tape blocks
1332023-10-31T04:24:40.117Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-31T04:24:40.142Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-31T04:24:40.166Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-31T04:24:40.210Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-31T04:24:40.235Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-31T04:24:40.260Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-31T04:24:40.284Z
1402023-10-31T04:24:40.309Zreal 1.597435674
1412023-10-31T04:24:40.337Zuser 1.441503134
1422023-10-31T04:24:40.361Zsys 0.278626118
1432023-10-31T04:24:40.386Ztrap 0.000063321
1442023-10-31T04:24:40.411Ztflt 0.000000000
1452023-10-31T04:24:40.437Zdflt 0.000000000
1462023-10-31T04:24:40.461Zkflt 0.000000000
1472023-10-31T04:24:40.486Zlock 0.000000000
1482023-10-31T04:24:40.512Zslp 1.437338185
1492023-10-31T04:24:40.536Zlat 0.035279592
1502023-10-31T04:24:40.561Zstop 0.000121991
1512023-10-31T04:24:40.585Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-31T04:24:40.934Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-31T04:24:40.959Z+ mkdir tests
1542023-10-31T04:24:40.984Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-31T04:24:41.011Z+ ptime -m gunzip
1562023-10-31T04:24:41.035Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-13d7796e9c457a2a
1572023-10-31T04:24:41.100Z
1582023-10-31T04:24:41.126Zreal 0.145800074
1592023-10-31T04:24:41.152Zuser 0.131915927
1602023-10-31T04:24:41.177Zsys 0.013639602
1612023-10-31T04:24:41.202Ztrap 0.000013579
1622023-10-31T04:24:41.238Ztflt 0.000000000
1632023-10-31T04:24:41.263Zdflt 0.000000000
1642023-10-31T04:24:41.288Zkflt 0.000000000
1652023-10-31T04:24:41.313Zlock 0.000000000
1662023-10-31T04:24:41.337Zslp 0.000023970
1672023-10-31T04:24:41.362Zlat 0.000023230
1682023-10-31T04:24:41.387Zstop 0.000125532
1692023-10-31T04:24:41.411Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-13d7796e9c457a2a
1702023-10-31T04:24:41.436Z+ chmod a+x tests/bootstrap-13d7796e9c457a2a
1712023-10-31T04:24:41.461Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-31T04:24:41.486Z+ ptime -m gunzip
1732023-10-31T04:24:41.513Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-31T04:24:43.369Z
1752023-10-31T04:24:43.394Zreal 2.257846162
1762023-10-31T04:24:43.419Zuser 2.117521129
1772023-10-31T04:24:43.443Zsys 0.139926757
1782023-10-31T04:24:43.468Ztrap 0.000170191
1792023-10-31T04:24:43.493Ztflt 0.000000000
1802023-10-31T04:24:43.518Zdflt 0.000000000
1812023-10-31T04:24:43.543Zkflt 0.000000000
1822023-10-31T04:24:43.568Zlock 0.000000000
1832023-10-31T04:24:43.593Zslp 0.000020310
1842023-10-31T04:24:43.618Zlat 0.000021680
1852023-10-31T04:24:43.643Zstop 0.000118141
1862023-10-31T04:24:43.667Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-31T04:24:43.692Z+ chmod a+x tests/bootstrap
1882023-10-31T04:24:43.717Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-31T04:24:43.741Z+ ptime -m gunzip
1902023-10-31T04:24:43.766Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cc8f9cfe063d8adb
1912023-10-31T04:24:45.954Z
1922023-10-31T04:24:45.978Zreal 2.572147214
1932023-10-31T04:24:46.003Zuser 2.418172390
1942023-10-31T04:24:46.028Zsys 0.153586069
1952023-10-31T04:24:46.052Ztrap 0.000160180
1962023-10-31T04:24:46.077Ztflt 0.000000000
1972023-10-31T04:24:46.102Zdflt 0.000000000
1982023-10-31T04:24:46.127Zkflt 0.000000000
1992023-10-31T04:24:46.151Zlock 0.000000000
2002023-10-31T04:24:46.176Zslp 0.000020700
2012023-10-31T04:24:46.201Zlat 0.000024900
2022023-10-31T04:24:46.226Zstop 0.000117391
2032023-10-31T04:24:46.250Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cc8f9cfe063d8adb
2042023-10-31T04:24:46.275Z+ chmod a+x tests/end_to_end_tests-cc8f9cfe063d8adb
2052023-10-31T04:24:46.302Z+ bmat address ls
2062023-10-31T04:24:46.327ZNAME CIDR FIRST LAST COUNT
2072023-10-31T04:24:46.366Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-10-31T04:24:46.390Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-31T04:24:46.415Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-10-31T04:24:46.440Z+ EXTRA_IP_START=10.151.1.164
2112023-10-31T04:24:46.465Z+ EXTRA_IP_END=10.151.1.227
2122023-10-31T04:24:46.493Z+ EXTRA_IP_COUNT=64
2132023-10-31T04:24:46.517Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-10-31T04:24:46.543Z+ EXTRA_IP_BASE=10.151.1
2152023-10-31T04:24:46.567Z+ EXTRA_IP_FOCTET=164
2162023-10-31T04:24:46.592Z+ EXTRA_IP_LOCTET=227
2172023-10-31T04:24:46.616Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-10-31T04:24:46.641Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-10-31T04:24:46.666Z+ DNS_IP1=10.151.1.164
2202023-10-31T04:24:46.690Z+ DNS_IP2=10.151.1.165
2212023-10-31T04:24:46.715Z+ UPLINK_IP=10.151.1.174
2222023-10-31T04:24:46.739Z+ IPPOOL_START=10.151.1.175
2232023-10-31T04:24:46.764Z+ IPPOOL_END=10.151.1.227
2242023-10-31T04:24:46.789Z++ ipadm show-addr -po type,addr
2252023-10-31T04:24:46.813Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-10-31T04:24:46.842Z+ GATEWAY_IP=10.151.1.100
2272023-10-31T04:24:46.866Z+ [[ -n 10.151.1.100 ]]
2282023-10-31T04:24:46.891Z+ ping -s 10.151.1.100 56 1
2292023-10-31T04:24:46.915ZPING 10.151.1.100: 56 data bytes
2302023-10-31T04:24:46.941Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.012 ms
2312023-10-31T04:24:46.966Z
2322023-10-31T04:24:46.990Z----10.151.1.100 PING Statistics----
2332023-10-31T04:24:47.015Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-10-31T04:24:47.040Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2352023-10-31T04:24:47.064Z++ arp -an
2362023-10-31T04:24:47.089Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-10-31T04:24:47.114Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-10-31T04:24:47.138Z+ routeadm -e ipv4-forwarding -u
2392023-10-31T04:24:47.163Z+ PXA_START=10.151.1.164
2402023-10-31T04:24:47.188Z+ PXA_END=10.151.1.227
2412023-10-31T04:24:47.212Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-10-31T04:24:47.237Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-10-31T04:24:47.261Z+ ZPOOL_VDEV_DIR=/scratch
2442023-10-31T04:24:47.286Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-10-31T04:24:47.311Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-10-31T04:24:47.335Z++ cd ./tools
2472023-10-31T04:24:47.360Z++ pwd
2482023-10-31T04:24:47.391Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-10-31T04:24:47.416Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-10-31T04:24:47.441Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-10-31T04:24:47.465Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-10-31T04:24:47.490Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-10-31T04:24:47.514Z++ dladm show-phys -p -o LINK
2542023-10-31T04:24:47.539Z++ head -1
2552023-10-31T04:24:47.564Z+ PHYSICAL_LINK=igb0
2562023-10-31T04:24:47.589Z+ echo 'Using igb0 as physical link'
2572023-10-31T04:24:47.613ZUsing igb0 as physical link
2582023-10-31T04:24:47.641Z+ ensure_run_as_root
2592023-10-31T04:24:47.667Z++ id -u
2602023-10-31T04:24:47.692Z+ [[ 0 -ne 0 ]]
2612023-10-31T04:24:47.716Z+ ensure_zpools
2622023-10-31T04:24:47.741Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-10-31T04:24:47.766Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-10-31T04:24:47.791Z+ readarray -t ZPOOLS
2652023-10-31T04:24:47.816Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-10-31T04:24:47.867Z++ sed 's/[ ",]//g'
2672023-10-31T04:24:47.892Z+ for ZPOOL in "${ZPOOLS[@]}"
2682023-10-31T04:24:47.919Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2692023-10-31T04:24:47.946ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2702023-10-31T04:24:47.974Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2712023-10-31T04:24:48.003Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2722023-10-31T04:24:48.031Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2732023-10-31T04:24:48.060Z0+0 records in
2742023-10-31T04:24:48.089ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2752023-10-31T04:24:48.119Z0+0 records out
2762023-10-31T04:24:48.154Z0 bytes (0 iB) transferred in 0.000007 secs (0 iB/sec)
2772023-10-31T04:24:48.181ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2782023-10-31T04:24:48.209Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2792023-10-31T04:24:48.240ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2802023-10-31T04:24:48.269Z+ set +x
2812023-10-31T04:24:48.298ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2822023-10-31T04:24:48.331Z++ zpool list -o name
2832023-10-31T04:24:48.362ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2842023-10-31T04:24:48.455Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2852023-10-31T04:24:48.487ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2862023-10-31T04:24:48.515Z+ [[ -z '' ]]
2872023-10-31T04:24:48.541ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2882023-10-31T04:24:48.569Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2892023-10-31T04:24:48.598ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2902023-10-31T04:24:48.625Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2912023-10-31T04:24:48.651ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2922023-10-31T04:24:48.676Z+ set +x
2932023-10-31T04:24:48.700ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2942023-10-31T04:24:48.725Z+ for ZPOOL in "${ZPOOLS[@]}"
2952023-10-31T04:24:48.750ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2962023-10-31T04:24:48.774Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2972023-10-31T04:24:48.799ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2982023-10-31T04:24:48.827Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2992023-10-31T04:24:48.855ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3002023-10-31T04:24:48.893Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3012023-10-31T04:24:48.923ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3022023-10-31T04:24:48.949Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3032023-10-31T04:24:48.973ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3042023-10-31T04:24:49.000Z0+0 records in
3052023-10-31T04:24:49.025ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3062023-10-31T04:24:49.050Z0+0 records out
3072023-10-31T04:24:49.075ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3082023-10-31T04:24:49.099Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3092023-10-31T04:24:49.124ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3102023-10-31T04:24:49.149Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3112023-10-31T04:24:49.173ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3122023-10-31T04:24:49.198Z+ set +x
3132023-10-31T04:24:49.222ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3142023-10-31T04:24:49.248Z++ zpool list -o name
3152023-10-31T04:24:49.273ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3162023-10-31T04:24:49.297Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3172023-10-31T04:24:49.326ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3182023-10-31T04:24:49.360Z+ [[ -z '' ]]
3192023-10-31T04:24:49.386ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3202023-10-31T04:24:49.412Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3212023-10-31T04:24:49.437ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3222023-10-31T04:24:49.477Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-10-31T04:24:49.502ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3242023-10-31T04:24:49.526Z+ set +x
3252023-10-31T04:24:49.551ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3262023-10-31T04:24:49.576Z+ for ZPOOL in "${ZPOOLS[@]}"
3272023-10-31T04:24:49.601ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3282023-10-31T04:24:49.625Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3292023-10-31T04:24:49.652ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3302023-10-31T04:24:49.679Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3312023-10-31T04:24:49.704ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3322023-10-31T04:24:49.728Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3332023-10-31T04:24:49.756ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3342023-10-31T04:24:49.780Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3352023-10-31T04:24:49.805ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3362023-10-31T04:24:49.830Z0+0 records in
3372023-10-31T04:24:49.854ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3382023-10-31T04:24:49.879Z0+0 records out
3392023-10-31T04:24:49.904ZSimnet net0/sc0_0 exists
3402023-10-31T04:24:49.929Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3412023-10-31T04:24:49.954ZSimnet net1/sc1_0 exists
3422023-10-31T04:24:49.979Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3432023-10-31T04:24:50.004ZVnic sc0_1 exists
3442023-10-31T04:24:50.028Z+ set +x
3452023-10-31T04:24:50.053Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3462023-10-31T04:24:50.078Z++ zpool list -o name
3472023-10-31T04:24:50.102Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3482023-10-31T04:24:50.127Z+ [[ -z '' ]]
3492023-10-31T04:24:50.152Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-10-31T04:24:50.186Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-10-31T04:24:50.211Z+ set +x
3522023-10-31T04:24:50.236Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-10-31T04:24:50.260Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-10-31T04:24:50.285Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-10-31T04:24:50.310Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-10-31T04:24:50.335Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-10-31T04:24:50.360Z0+0 records in
3582023-10-31T04:24:50.385Z0+0 records out
3592023-10-31T04:24:50.410Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3602023-10-31T04:24:50.434Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-10-31T04:24:50.459Z+ set +x
3622023-10-31T04:24:50.512Z++ zpool list -o name
3632023-10-31T04:24:50.538Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-10-31T04:24:50.563Z+ [[ -z '' ]]
3652023-10-31T04:24:50.590Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-10-31T04:24:50.629Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-10-31T04:24:50.661Z+ set +x
3682023-10-31T04:24:50.689Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-10-31T04:24:50.715Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-10-31T04:24:50.740Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-10-31T04:24:50.770Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-10-31T04:24:50.795Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-10-31T04:24:50.820Z0+0 records in
3742023-10-31T04:24:50.845Z0+0 records out
3752023-10-31T04:24:50.870Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3762023-10-31T04:24:50.895Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-10-31T04:24:50.919Z+ set +x
3782023-10-31T04:24:50.944Z++ zpool list -o name
3792023-10-31T04:24:50.969Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-10-31T04:24:50.994Z+ [[ -z '' ]]
3812023-10-31T04:24:51.019Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-10-31T04:24:51.044Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-10-31T04:24:51.069Z+ set +x
3842023-10-31T04:24:51.105Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-10-31T04:24:51.134Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862023-10-31T04:24:51.158Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872023-10-31T04:24:51.183Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882023-10-31T04:24:51.207Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892023-10-31T04:24:51.232Z0+0 records in
3902023-10-31T04:24:51.257Z0+0 records out
3912023-10-31T04:24:51.282Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-10-31T04:24:51.308Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932023-10-31T04:24:51.333Z+ set +x
3942023-10-31T04:24:51.357Z++ zpool list -o name
3952023-10-31T04:24:51.382Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3962023-10-31T04:24:51.407Z+ [[ -z '' ]]
3972023-10-31T04:24:51.432Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3982023-10-31T04:24:51.456Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3992023-10-31T04:24:51.485Z+ set +x
4002023-10-31T04:24:51.557Z+ for ZPOOL in "${ZPOOLS[@]}"
4012023-10-31T04:24:51.583Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4022023-10-31T04:24:51.608Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4032023-10-31T04:24:51.632Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4042023-10-31T04:24:51.658Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4052023-10-31T04:24:51.683Z0+0 records in
4062023-10-31T04:24:51.708Z0+0 records out
4072023-10-31T04:24:51.733Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4082023-10-31T04:24:51.757Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4092023-10-31T04:24:51.782Z+ set +x
4102023-10-31T04:24:51.812Z++ zpool list -o name
4112023-10-31T04:24:51.836Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4122023-10-31T04:24:51.861Z+ [[ -z '' ]]
4132023-10-31T04:24:51.886Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4142023-10-31T04:24:51.911Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4152023-10-31T04:24:51.936Z+ set +x
4162023-10-31T04:24:51.961Z+ for ZPOOL in "${ZPOOLS[@]}"
4172023-10-31T04:24:51.986Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4182023-10-31T04:24:52.011Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192023-10-31T04:24:52.036Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4202023-10-31T04:24:52.060Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4212023-10-31T04:24:52.085Z0+0 records in
4222023-10-31T04:24:52.110Z0+0 records out
4232023-10-31T04:24:52.134Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4242023-10-31T04:24:52.160Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4252023-10-31T04:24:52.184Z+ set +x
4262023-10-31T04:24:52.209Z++ zpool list -o name
4272023-10-31T04:24:52.233Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4282023-10-31T04:24:52.258Z+ [[ -z '' ]]
4292023-10-31T04:24:52.283Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4302023-10-31T04:24:52.308Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4312023-10-31T04:24:52.332Z+ set +x
4322023-10-31T04:24:52.357Z+ for ZPOOL in "${ZPOOLS[@]}"
4332023-10-31T04:24:52.382Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4342023-10-31T04:24:52.407Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4352023-10-31T04:24:52.432Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4362023-10-31T04:24:52.457Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4372023-10-31T04:24:52.483Z0+0 records in
4382023-10-31T04:24:52.508Z0+0 records out
4392023-10-31T04:24:52.533Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4402023-10-31T04:24:52.582Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4412023-10-31T04:24:52.608Z+ set +x
4422023-10-31T04:24:52.632Z++ zpool list -o name
4432023-10-31T04:24:52.657Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4442023-10-31T04:24:52.682Z+ [[ -z '' ]]
4452023-10-31T04:24:52.733Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4462023-10-31T04:24:52.758Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4472023-10-31T04:24:52.784Z+ set +x
4482023-10-31T04:24:52.809Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4492023-10-31T04:24:52.833Z+ readarray -t ZPOOLS
4502023-10-31T04:24:52.860Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4512023-10-31T04:24:52.884Z++ sed 's/[ ",]//g'
4522023-10-31T04:24:52.909Z+ for ZPOOL in "${ZPOOLS[@]}"
4532023-10-31T04:24:52.938Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4542023-10-31T04:24:52.963Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4552023-10-31T04:24:52.988Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4562023-10-31T04:24:53.013Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4572023-10-31T04:24:53.037Z0+0 records in
4582023-10-31T04:24:53.062Z0+0 records out
4592023-10-31T04:24:53.087Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4602023-10-31T04:24:53.140Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4612023-10-31T04:24:53.165Z+ set +x
4622023-10-31T04:24:53.191Z++ zpool list -o name
4632023-10-31T04:24:53.216Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4642023-10-31T04:24:53.247Z+ [[ -z '' ]]
4652023-10-31T04:24:53.272Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-10-31T04:24:53.297Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4672023-10-31T04:24:53.322Z+ set +x
4682023-10-31T04:24:53.347Z+ for ZPOOL in "${ZPOOLS[@]}"
4692023-10-31T04:24:53.372Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4702023-10-31T04:24:53.397Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4712023-10-31T04:24:53.422Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4722023-10-31T04:24:53.447Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4732023-10-31T04:24:53.471Z0+0 records in
4742023-10-31T04:24:53.496Z0+0 records out
4752023-10-31T04:24:53.521Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4762023-10-31T04:24:53.546Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4772023-10-31T04:24:53.571Z+ set +x
4782023-10-31T04:24:53.609Z++ zpool list -o name
4792023-10-31T04:24:53.633Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4802023-10-31T04:24:53.658Z+ [[ -z '' ]]
4812023-10-31T04:24:53.683Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4822023-10-31T04:24:53.709Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4832023-10-31T04:24:53.734Z+ set +x
4842023-10-31T04:24:53.762Z+ ensure_simulated_links igb0
4852023-10-31T04:24:53.787Z+ local PHYSICAL_LINK=igb0
4862023-10-31T04:24:53.812Z+ INDICES=("0" "1")
4872023-10-31T04:24:53.837Z+ for I in "${INDICES[@]}"
4882023-10-31T04:24:53.861Z++ get_simnet_name_if_exists net0
4892023-10-31T04:24:53.887Z++ dladm show-simnet -p -o LINK net0
4902023-10-31T04:24:53.912Z++ echo ''
4912023-10-31T04:24:53.937Z+ [[ -z '' ]]
4922023-10-31T04:24:53.964Z+ dladm create-simnet -t net0
4932023-10-31T04:24:53.989Z+ dladm create-simnet -t sc0_0
4942023-10-31T04:24:54.014Z+ dladm modify-simnet -t -p net0 sc0_0
4952023-10-31T04:24:54.039Z+ dladm set-linkprop -p mtu=9000 sc0_0
4962023-10-31T04:24:54.064Z+ success 'Simnet net0/sc0_0 exists'
4972023-10-31T04:24:54.090Z+ set +x
4982023-10-31T04:24:54.115Z+ for I in "${INDICES[@]}"
4992023-10-31T04:24:54.140Z++ get_simnet_name_if_exists net1
5002023-10-31T04:24:54.165Z++ dladm show-simnet -p -o LINK net1
5012023-10-31T04:24:54.190Z++ echo ''
5022023-10-31T04:24:54.214Z+ [[ -z '' ]]
5032023-10-31T04:24:54.238Z+ dladm create-simnet -t net1
5042023-10-31T04:24:54.263Z+ dladm create-simnet -t sc1_0
5052023-10-31T04:24:54.288Z+ dladm modify-simnet -t -p net1 sc1_0
5062023-10-31T04:24:54.312Z+ dladm set-linkprop -p mtu=9000 sc1_0
5072023-10-31T04:24:54.337Z+ success 'Simnet net1/sc1_0 exists'
5082023-10-31T04:24:54.362Z+ set +x
5092023-10-31T04:24:54.386Z++ get_vnic_name_if_exists sc0_1
5102023-10-31T04:24:54.411Z++ dladm show-vnic -p -o LINK sc0_1
5112023-10-31T04:24:54.435Z++ echo ''
5122023-10-31T04:24:54.460Z+ [[ -z '' ]]
5132023-10-31T04:24:54.485Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-10-31T04:24:54.509Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-10-31T04:24:54.534Z+ success 'Vnic sc0_1 exists'
5162023-10-31T04:24:54.559Z+ set +x
5172023-10-31T04:24:54.587Z+ warn_if_no_proxy_arp
5182023-10-31T04:24:54.636Z+ [[ -v PXA_START ]]
5192023-10-31T04:24:54.661Z+ [[ -v PXA_END ]]
5202023-10-31T04:24:54.686Z+ ensure_softnpu_zone
5212023-10-31T04:24:54.710Z+ zoneadm list
5222023-10-31T04:24:54.734Z+ grep -q sidecar_softnpu
5232023-10-31T04:24:54.759Z+ [[ -f out/npuzone/npuzone ]]
5242023-10-31T04:24:54.784Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch main
5252023-10-31T04:24:56.835Zdownloading softnpu ...done
5262023-10-31T04:24:57.007Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5272023-10-31T04:25:01.344Zdownloading asic_program.so ...done
5282023-10-31T04:25:01.396Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5292023-10-31T04:25:08.134Zdownloading scadm ...done
5302023-10-31T04:25:08.191Zcreating port tfportrear0_0 for sc0_0
5312023-10-31T04:25:08.215Zcreating port tfportqsfp0_0 for sc0_1
5322023-10-31T04:25:08.240Zsoftnpu zone setup
5332023-10-31T04:25:08.265Zconfigure zone
5342023-10-31T04:25:08.291Zinstall zone
5352023-10-31T04:25:09.230Zboot zone
5362023-10-31T04:25:12.339Z[sidecar_softnpu] svcs milestone/network
5372023-10-31T04:25:12.368ZSTATE STIME FMRI
5382023-10-31T04:25:12.393Zonline 4:25:10 svc:/milestone/network:default
5392023-10-31T04:25:12.418Z[sidecar_softnpu] /softnpu/npu start
5402023-10-31T04:25:12.443Z
5412023-10-31T04:25:12.468Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5422023-10-31T04:25:12.497Z+ GATEWAY_IP=10.151.1.100
5432023-10-31T04:25:12.533Z+ echo 'Using 10.151.1.100 as gateway ip'
5442023-10-31T04:25:12.566ZUsing 10.151.1.100 as gateway ip
5452023-10-31T04:25:12.594Z+ [[ ! -v GATEWAY_MAC ]]
5462023-10-31T04:25:12.624Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5472023-10-31T04:25:12.655Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5482023-10-31T04:25:12.683Z+++ echo 18:c0:4d:0d:9f:b2
5492023-10-31T04:25:12.712Z+ COUNT=1
5502023-10-31T04:25:12.744Z+ [[ 1 -ne 1 ]]
5512023-10-31T04:25:12.816Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5522023-10-31T04:25:12.847ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5532023-10-31T04:25:12.875Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5542023-10-31T04:25:12.904Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5552023-10-31T04:25:12.935Z+ PXA_MAC=a8:e1:de:01:70:1d
5562023-10-31T04:25:12.967Z+ [[ -v PXA_START ]]
5572023-10-31T04:25:12.993Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5582023-10-31T04:25:13.023Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5592023-10-31T04:25:13.048Z+ z_scadm dump-state
5602023-10-31T04:25:13.073Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5612023-10-31T04:25:13.098Zlocal v6:
5622023-10-31T04:25:13.122Zlocal v4:
5632023-10-31T04:25:13.148Zrouter v6:
5642023-10-31T04:25:13.186Zrouter v4_idx:
5652023-10-31T04:25:13.212Zrouter v4_routes:
5662023-10-31T04:25:13.240Zresolver v4:
5672023-10-31T04:25:13.266Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5682023-10-31T04:25:13.292Zresolver v6:
5692023-10-31T04:25:13.317Znat v4:
5702023-10-31T04:25:13.343Znat_v6:
5712023-10-31T04:25:13.375Zport_mac:
5722023-10-31T04:25:13.410Zproxy arp:
5732023-10-31T04:25:13.440Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5742023-10-31T04:25:13.465Z+ success 'softnpu zone exists'
5752023-10-31T04:25:13.490Z+ set +x
5762023-10-31T04:25:13.515Zsoftnpu zone exists
5772023-10-31T04:25:13.540Z
5782023-10-31T04:25:13.565Zreal 26.199005290
5792023-10-31T04:25:13.590Zuser 0.724776745
5802023-10-31T04:25:13.616Zsys 1.592219204
5812023-10-31T04:25:13.641Ztrap 0.000090164
5822023-10-31T04:25:13.667Ztflt 0.000123057
5832023-10-31T04:25:13.692Zdflt 0.001378731
5842023-10-31T04:25:13.718Zkflt 0.000004860
5852023-10-31T04:25:13.743Zlock 20:26.255879699
5862023-10-31T04:25:13.768Zslp 1:17.870954747
5872023-10-31T04:25:13.794Zlat 0.071350383
5882023-10-31T04:25:13.826Zstop 0.347473920
5892023-10-31T04:25:13.858Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5902023-10-31T04:25:13.883Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5912023-10-31T04:25:13.908Z+ SILO_NAME=recovery
5922023-10-31T04:25:13.934Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-10-31T04:25:13.961Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5942023-10-31T04:25:13.986Z+ sed -i~ '
5952023-10-31T04:25:14.013Z /^external_dns_ips/c\
5962023-10-31T04:25:14.038Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5972023-10-31T04:25:14.063Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5982023-10-31T04:25:14.090Z /^first/c\
5992023-10-31T04:25:14.115Zfirst = "10.151.1.164"
6002023-10-31T04:25:14.140Z /^last/c\
6012023-10-31T04:25:14.164Zlast = "10.151.1.173"
6022023-10-31T04:25:14.218Z }
6032023-10-31T04:25:14.243Z /^\[rack_network_config/,/^$/ {
6042023-10-31T04:25:14.273Z /^infra_ip_first/c\
6052023-10-31T04:25:14.303Zinfra_ip_first = "10.151.1.174"
6062023-10-31T04:25:14.328Z /^infra_ip_last/c\
6072023-10-31T04:25:14.352Zinfra_ip_last = "10.151.1.174"
6082023-10-31T04:25:14.378Z }
6092023-10-31T04:25:14.404Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6102023-10-31T04:25:14.428Z /^\[\[rack_network_config.ports/,/^$/ {
6112023-10-31T04:25:14.454Z+++ pkg/config-rss.toml Tue Oct 31 04:25:12 2023
6122023-10-31T04:25:14.480Z /^routes/c\
6132023-10-31T04:25:14.505Z@@ -41,7 +41,7 @@
6142023-10-31T04:25:14.530Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6152023-10-31T04:25:14.565Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6162023-10-31T04:25:14.592Z /^addresses/c\
6172023-10-31T04:25:14.617Z # must be contained in one of the "internal services" IP Pool ranges listed
6182023-10-31T04:25:14.642Zaddresses = \["10.151.1.174/32"\]
6192023-10-31T04:25:14.667Z # below.
6202023-10-31T04:25:14.697Z }
6212023-10-31T04:25:14.729Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6222023-10-31T04:25:14.757Z' pkg/config-rss.toml
6232023-10-31T04:25:14.783Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6242023-10-31T04:25:14.809Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6252023-10-31T04:25:14.835Z
6262023-10-31T04:25:14.860Z+ true
6272023-10-31T04:25:14.885Z # Initial TLS certificates for the external API
6282023-10-31T04:25:14.912Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6292023-10-31T04:25:14.937Z #
6302023-10-31T04:25:14.961Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6312023-10-31T04:25:14.988Z@@ -77,8 +77,8 @@
6322023-10-31T04:25:15.013Z+ TLS_NAME=recovery.sys.oxide.test
6332023-10-31T04:25:15.038Z #
6342023-10-31T04:25:15.063Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6352023-10-31T04:25:15.088Z # For more on this and what to put here, see docs/how-to-run.adoc.
6362023-10-31T04:25:15.113Z.......+.......+..+......+......................+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+..+....+...+..+...+......+....+..+.+.................+.+......+.........+......+...........+.........+.+..+.+............+............+...+.........+.....+..................+...+.......+.....+.+......+...+......+..+.........+.........+.+........+......+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+...............+.........+....................+.+.........+...........+...+.+...........+...+.+.....+.......................................+...+...+......+.........+.......+............+...+..+................+.........+...+.....+.......+.....+......+..........+.....................+...........+...............+....+........+......+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6372023-10-31T04:25:15.145Z [[internal_services_ip_pool_ranges]]
6382023-10-31T04:25:15.174Z.........+......+......+.....+.......+...+...+..+...+....+........+.........+...+.......+......+.....+...+....+...+...+.....+......+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+..................+......+..+.........+.............+.....+.+..+.......+...+...+...........+...+.+.....+.......+.....+....+.....+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+..................................+...+.........+...+..+....+.....+.+...+......+..+..................+.........+.......+........+...+...+.......+...+..+.........+.+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6392023-10-31T04:25:15.200Z-first = "192.168.1.20"
6402023-10-31T04:25:15.238Z-----
6412023-10-31T04:25:15.264Z-last = "192.168.1.29"
6422023-10-31T04:25:15.288Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6432023-10-31T04:25:15.313Z+first = "10.151.1.164"
6442023-10-31T04:25:15.337Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6452023-10-31T04:25:15.365Z+last = "10.151.1.173"
6462023-10-31T04:25:15.391Z+ rmdir pkg
6472023-10-31T04:25:15.416Z
6482023-10-31T04:25:15.441Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6492023-10-31T04:25:15.466Z # TODO - this configuration is subject to change going forward. Ultimately these
6502023-10-31T04:25:15.491Z+ pfexec mkdir -p /usr/oxide
6512023-10-31T04:25:15.517Z # parameters should be provided to the control plane via wicket, but we need to
6522023-10-31T04:25:15.548Z+ pfexec rm -f /usr/oxide/tcpproxy
6532023-10-31T04:25:15.575Z@@ -92,8 +92,8 @@
6542023-10-31T04:25:15.604Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6552023-10-31T04:25:15.635Z # A range of IP addresses used by Boundary Services on the external network. In
6562023-10-31T04:25:15.716Z+ pfexec chmod +x /usr/oxide/tcpproxy
6572023-10-31T04:25:15.744Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6582023-10-31T04:25:15.770Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6592023-10-31T04:25:15.796Z # With softnpu, only one address is used.
6602023-10-31T04:25:15.821Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6612023-10-31T04:25:15.847Z-infra_ip_first = "192.168.1.30"
6622023-10-31T04:25:15.873Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6632023-10-31T04:25:15.899Z-infra_ip_last = "192.168.1.30"
6642023-10-31T04:25:15.925Z+ OMICRON_NO_UNINSTALL=1
6652023-10-31T04:25:15.960Z+infra_ip_first = "10.151.1.174"
6662023-10-31T04:25:15.987Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6672023-10-31T04:25:16.013Z+infra_ip_last = "10.151.1.174"
6682023-10-31T04:25:16.039ZOct 31 04:25:13.476 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6692023-10-31T04:25:16.068Z
6702023-10-31T04:25:16.093ZOct 31 04:25:13.478 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6712023-10-31T04:25:16.123Z # Configurations for BGP routers to run on the scrimlets.
6722023-10-31T04:25:16.148ZOct 31 04:25:13.478 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6732023-10-31T04:25:16.176Z bgp = []
6742023-10-31T04:25:16.203ZOct 31 04:25:13.478 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6752023-10-31T04:25:16.230Z@@ -101,9 +101,9 @@
6762023-10-31T04:25:16.281ZOct 31 04:25:13.478 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6772023-10-31T04:25:16.308Z # You can configure multiple uplinks by repeating the following stanza
6782023-10-31T04:25:16.337ZOct 31 04:25:13.478 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6792023-10-31T04:25:16.370Z [[rack_network_config.ports]]
6802023-10-31T04:25:16.398ZOct 31 04:25:13.478 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6812023-10-31T04:25:16.428Z # Routes associated with this port.
6822023-10-31T04:25:16.457ZOct 31 04:25:13.478 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6832023-10-31T04:25:16.487Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6842023-10-31T04:25:16.521ZOct 31 04:25:13.478 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6852023-10-31T04:25:16.550Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6862023-10-31T04:25:16.582ZOct 31 04:25:13.478 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6872023-10-31T04:25:16.613Z # Addresses associated with this port.
6882023-10-31T04:25:16.642ZOct 31 04:25:13.478 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6892023-10-31T04:25:16.672Z-addresses = ["192.168.1.30/32"]
6902023-10-31T04:25:16.696ZOct 31 04:25:13.478 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6912023-10-31T04:25:16.721Z+addresses = ["10.151.1.174/32"]
6922023-10-31T04:25:16.746ZOct 31 04:25:13.478 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6932023-10-31T04:25:16.771Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6942023-10-31T04:25:16.796ZOct 31 04:25:13.478 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6952023-10-31T04:25:16.820Z port = "qsfp0"
6962023-10-31T04:25:16.846ZOct 31 04:25:13.479 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6972023-10-31T04:25:16.872Z # The speed of this port.
6982023-10-31T04:25:16.897ZOct 31 04:25:13.479 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6992023-10-31T04:25:16.925Za pkg/config-rss.toml 7K
7002023-10-31T04:25:16.950ZOct 31 04:25:13.479 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7012023-10-31T04:25:16.974Za pkg/initial-tls-cert.pem 2K
7022023-10-31T04:25:17.008ZOct 31 04:25:13.709 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7032023-10-31T04:25:17.042Za pkg/initial-tls-key.pem 4K
7042023-10-31T04:25:17.071ZOct 31 04:25:13.744 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7052023-10-31T04:25:17.097ZOct 31 04:25:13.806 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7062023-10-31T04:25:17.121Z
7072023-10-31T04:25:17.146Zreal 0.466091910
7082023-10-31T04:25:17.171Zuser 0.098522360
7092023-10-31T04:25:17.195Zsys 1.883058169
7102023-10-31T04:25:17.220Ztrap 0.000008160
7112023-10-31T04:25:17.245Ztflt 0.000021679
7122023-10-31T04:25:17.269Zdflt 0.000651051
7132023-10-31T04:25:17.306Zkflt 0.000000000
7142023-10-31T04:25:17.331Zlock 41.829516705
7152023-10-31T04:25:17.356Zslp 0.663733742
7162023-10-31T04:25:17.380Zlat 0.002726264
7172023-10-31T04:25:17.405Zstop 0.201201803
7182023-10-31T04:25:17.432Z+ retry=0
7192023-10-31T04:25:17.468Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7202023-10-31T04:25:17.503Z+ [[ 0 -gt 30 ]]
7212023-10-31T04:25:17.529Z+ sleep 1
7222023-10-31T04:25:17.557Z+ retry=1
7232023-10-31T04:25:17.584Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7242023-10-31T04:25:17.609Z+ [[ 1 -gt 30 ]]
7252023-10-31T04:25:17.635Z+ sleep 1
7262023-10-31T04:25:17.659Z+ retry=2
7272023-10-31T04:25:17.684Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7282023-10-31T04:25:41.201Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7292023-10-31T04:25:41.228Zlocal v6:
7302023-10-31T04:25:41.253Zfd00:99::1
7312023-10-31T04:25:41.277Zfe80::1019:c1ff:fe1f:6a92
7322023-10-31T04:25:41.302Zlocal v4:
7332023-10-31T04:25:41.327Z10.151.1.174
7342023-10-31T04:25:41.352Zrouter v6:
7352023-10-31T04:25:41.378Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::448f:feff:fe6f:e58 (1)
7362023-10-31T04:25:41.403Zfd00:1122:3344:101::/64 -> fe80::448f:feff:fe6f:e58 (1)
7372023-10-31T04:25:41.428Zrouter v4_idx:
7382023-10-31T04:25:41.453Z0.0.0.0/0 -> 0
7392023-10-31T04:25:41.487Zrouter v4_routes:
7402023-10-31T04:25:41.517Z0 -> 10.151.1.100 (2)
7412023-10-31T04:25:41.558Zresolver v4:
7422023-10-31T04:25:41.584Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7432023-10-31T04:25:41.608Z10.151.1.174 -> de:67:62:23:a4:2c
7442023-10-31T04:25:41.635Zresolver v6:
7452023-10-31T04:25:41.659Zfe80::448f:feff:fe6f:e58 -> 46:8f:fe:6f:0e:58
7462023-10-31T04:25:41.684Zfe80::1019:c1ff:fe1f:6a92 -> 12:19:c1:1f:6a:92
7472023-10-31T04:25:41.710Znat v4:
7482023-10-31T04:25:41.735Znat_v6:
7492023-10-31T04:25:41.761Zport_mac:
7502023-10-31T04:25:41.786Z1000: a8:40:25:0d:3f:6c
7512023-10-31T04:25:41.814Z1: a8:40:25:0d:3f:6d
7522023-10-31T04:25:41.839Zproxy arp:
7532023-10-31T04:25:41.864Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7542023-10-31T04:25:41.892Z+ retry=0
7552023-10-31T04:25:41.919Z+++ zoneadm list -n
7562023-10-31T04:25:42.349Z+++ grep oxz_ntp
7572023-10-31T04:25:42.375Z++ pfexec svcs -z -Hostate oxide/ntp
7582023-10-31T04:25:42.402Zsvcs: zone '-Hostate': entity not found
7592023-10-31T04:25:42.434Z++ true
7602023-10-31T04:25:42.461Z+ [[ '' != online ]]
7612023-10-31T04:25:42.486Z+ [[ 0 -gt 60 ]]
7622023-10-31T04:25:42.512Z+ sleep 1
7632023-10-31T04:25:42.537Z+ retry=1
7642023-10-31T04:25:42.561Z+++ zoneadm list -n
7652023-10-31T04:25:42.586Z+++ grep oxz_ntp
7662023-10-31T04:25:42.611Z++ pfexec svcs -z -Hostate oxide/ntp
7672023-10-31T04:25:42.635Zsvcs: zone '-Hostate': entity not found
7682023-10-31T04:25:42.660Z++ true
7692023-10-31T04:25:42.711Z+ [[ '' != online ]]
7702023-10-31T04:25:42.736Z+ [[ 1 -gt 60 ]]
7712023-10-31T04:25:42.770Z+ sleep 1
7722023-10-31T04:25:43.274Z+ retry=2
7732023-10-31T04:25:43.299Z+++ zoneadm list -n
7742023-10-31T04:25:43.324Z+++ grep oxz_ntp
7752023-10-31T04:25:43.349Z++ pfexec svcs -z -Hostate oxide/ntp
7762023-10-31T04:25:43.373Zsvcs: zone '-Hostate': entity not found
7772023-10-31T04:25:43.398Z++ true
7782023-10-31T04:25:43.423Z+ [[ '' != online ]]
7792023-10-31T04:25:43.448Z+ [[ 2 -gt 60 ]]
7802023-10-31T04:25:43.472Z+ sleep 1
7812023-10-31T04:25:44.297Z+ retry=3
7822023-10-31T04:25:44.323Z+++ zoneadm list -n
7832023-10-31T04:25:44.348Z+++ grep oxz_ntp
7842023-10-31T04:25:44.372Z++ pfexec svcs -z -Hostate oxide/ntp
7852023-10-31T04:25:44.398Zsvcs: zone '-Hostate': entity not found
7862023-10-31T04:25:44.422Z++ true
7872023-10-31T04:25:44.447Z+ [[ '' != online ]]
7882023-10-31T04:25:44.473Z+ [[ 3 -gt 60 ]]
7892023-10-31T04:25:44.507Z+ sleep 1
7902023-10-31T04:25:45.320Z+ retry=4
7912023-10-31T04:25:45.344Z+++ zoneadm list -n
7922023-10-31T04:25:45.376Z+++ grep oxz_ntp
7932023-10-31T04:25:45.413Z++ pfexec svcs -z -Hostate oxide/ntp
7942023-10-31T04:25:45.439Zsvcs: zone '-Hostate': entity not found
7952023-10-31T04:25:45.464Z++ true
7962023-10-31T04:25:45.489Z+ [[ '' != online ]]
7972023-10-31T04:25:45.513Z+ [[ 4 -gt 60 ]]
7982023-10-31T04:25:45.538Z+ sleep 1
7992023-10-31T04:25:46.341Z+ retry=5
8002023-10-31T04:25:46.366Z+++ zoneadm list -n
8012023-10-31T04:25:46.390Z+++ grep oxz_ntp
8022023-10-31T04:25:46.415Z++ pfexec svcs -z -Hostate oxide/ntp
8032023-10-31T04:25:46.440Zsvcs: zone '-Hostate': entity not found
8042023-10-31T04:25:46.465Z++ true
8052023-10-31T04:25:46.490Z+ [[ '' != online ]]
8062023-10-31T04:25:46.515Z+ [[ 5 -gt 60 ]]
8072023-10-31T04:25:46.540Z+ sleep 1
8082023-10-31T04:25:47.362Z+ retry=6
8092023-10-31T04:25:47.387Z+++ zoneadm list -n
8102023-10-31T04:25:47.412Z+++ grep oxz_ntp
8112023-10-31T04:25:47.437Z++ pfexec svcs -z -Hostate oxide/ntp
8122023-10-31T04:25:47.483Zsvcs: zone '-Hostate': entity not found
8132023-10-31T04:25:47.507Z++ true
8142023-10-31T04:25:47.538Z+ [[ '' != online ]]
8152023-10-31T04:25:47.562Z+ [[ 6 -gt 60 ]]
8162023-10-31T04:25:47.594Z+ sleep 1
8172023-10-31T04:25:48.383Z+ retry=7
8182023-10-31T04:25:48.408Z+++ zoneadm list -n
8192023-10-31T04:25:48.432Z+++ grep oxz_ntp
8202023-10-31T04:25:48.457Z++ pfexec svcs -z -Hostate oxide/ntp
8212023-10-31T04:25:48.485Zsvcs: zone '-Hostate': entity not found
8222023-10-31T04:25:48.510Z++ true
8232023-10-31T04:25:48.544Z+ [[ '' != online ]]
8242023-10-31T04:25:48.571Z+ [[ 7 -gt 60 ]]
8252023-10-31T04:25:48.599Z+ sleep 1
8262023-10-31T04:25:49.405Z+ retry=8
8272023-10-31T04:25:49.430Z+++ zoneadm list -n
8282023-10-31T04:25:49.456Z+++ grep oxz_ntp
8292023-10-31T04:25:49.483Z++ pfexec svcs -z -Hostate oxide/ntp
8302023-10-31T04:25:49.508Zsvcs: zone '-Hostate': entity not found
8312023-10-31T04:25:49.532Z++ true
8322023-10-31T04:25:49.557Z+ [[ '' != online ]]
8332023-10-31T04:25:49.583Z+ [[ 8 -gt 60 ]]
8342023-10-31T04:25:49.611Z+ sleep 1
8352023-10-31T04:25:50.547Z+ retry=9
8362023-10-31T04:25:50.582Z+++ zoneadm list -n
8372023-10-31T04:25:50.608Z+++ grep oxz_ntp
8382023-10-31T04:25:50.633Z++ pfexec svcs -z -Hostate oxide/ntp
8392023-10-31T04:25:50.658Zsvcs: zone '-Hostate': entity not found
8402023-10-31T04:25:50.683Z++ true
8412023-10-31T04:25:50.710Z+ [[ '' != online ]]
8422023-10-31T04:25:50.735Z+ [[ 9 -gt 60 ]]
8432023-10-31T04:25:50.760Z+ sleep 1
8442023-10-31T04:25:51.451Z+ retry=10
8452023-10-31T04:25:51.477Z+++ zoneadm list -n
8462023-10-31T04:25:51.504Z+++ grep oxz_ntp
8472023-10-31T04:25:51.532Z++ pfexec svcs -z oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87 -Hostate oxide/ntp
8482023-10-31T04:25:51.557Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8492023-10-31T04:25:51.585Z++ true
8502023-10-31T04:25:51.614Z+ [[ '' != online ]]
8512023-10-31T04:25:51.649Z+ [[ 10 -gt 60 ]]
8522023-10-31T04:25:51.679Z+ sleep 1
8532023-10-31T04:25:52.498Z+ retry=11
8542023-10-31T04:25:52.523Z+++ zoneadm list -n
8552023-10-31T04:25:52.550Z+++ grep oxz_ntp
8562023-10-31T04:25:52.574Z++ pfexec svcs -z oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87 -Hostate oxide/ntp
8572023-10-31T04:25:52.609Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8582023-10-31T04:25:52.645Z++ true
8592023-10-31T04:25:52.672Z+ [[ '' != online ]]
8602023-10-31T04:25:52.697Z+ [[ 11 -gt 60 ]]
8612023-10-31T04:25:52.748Z+ sleep 1
8622023-10-31T04:25:53.535Z+ retry=12
8632023-10-31T04:25:53.563Z+++ zoneadm list -n
8642023-10-31T04:25:53.591Z+++ grep oxz_ntp
8652023-10-31T04:25:53.620Z++ pfexec svcs -z oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87 -Hostate oxide/ntp
8662023-10-31T04:25:54.057Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8672023-10-31T04:25:54.094Z++ true
8682023-10-31T04:25:54.129Z+ [[ '' != online ]]
8692023-10-31T04:25:54.155Z+ [[ 12 -gt 60 ]]
8702023-10-31T04:25:54.183Z+ sleep 1
8712023-10-31T04:25:54.560Z+ retry=13
8722023-10-31T04:25:54.586Z+++ zoneadm list -n
8732023-10-31T04:25:54.611Z+++ grep oxz_ntp
8742023-10-31T04:25:54.636Z++ pfexec svcs -z oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87 -Hostate oxide/ntp
8752023-10-31T04:25:54.667Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8762023-10-31T04:25:54.695Z++ true
8772023-10-31T04:25:54.721Z+ [[ '' != online ]]
8782023-10-31T04:25:54.746Z+ [[ 13 -gt 60 ]]
8792023-10-31T04:25:54.771Z+ sleep 1
8802023-10-31T04:25:55.589Z+ retry=14
8812023-10-31T04:25:55.617Z+++ zoneadm list -n
8822023-10-31T04:25:55.642Z+++ grep oxz_ntp
8832023-10-31T04:25:55.667Z++ pfexec svcs -z oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87 -Hostate oxide/ntp
8842023-10-31T04:25:55.700Z+ [[ disabled != online ]]
8852023-10-31T04:25:56.163Z+ [[ 14 -gt 60 ]]
8862023-10-31T04:25:56.196Z+ sleep 1
8872023-10-31T04:25:56.612Z+ retry=15
8882023-10-31T04:25:56.637Z+++ zoneadm list -n
8892023-10-31T04:25:56.662Z+++ grep oxz_ntp
8902023-10-31T04:25:56.687Z++ pfexec svcs -z oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87 -Hostate oxide/ntp
8912023-10-31T04:25:56.711Z+ [[ disabled != online ]]
8922023-10-31T04:25:56.736Z+ [[ 15 -gt 60 ]]
8932023-10-31T04:25:56.760Z+ sleep 1
8942023-10-31T04:25:57.636Z+ retry=16
8952023-10-31T04:25:57.667Z+++ zoneadm list -n
8962023-10-31T04:25:57.800Z+++ grep oxz_ntp
8972023-10-31T04:25:57.830Z++ pfexec svcs -z oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87 -Hostate oxide/ntp
8982023-10-31T04:25:57.860Z+ [[ disabled != online ]]
8992023-10-31T04:25:57.889Z+ [[ 16 -gt 60 ]]
9002023-10-31T04:25:57.920Z+ sleep 1
9012023-10-31T04:25:58.658Z+ retry=17
9022023-10-31T04:25:58.682Z+++ zoneadm list -n
9032023-10-31T04:25:58.708Z+++ grep oxz_ntp
9042023-10-31T04:25:58.732Z++ pfexec svcs -z oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87 -Hostate oxide/ntp
9052023-10-31T04:25:58.757Z+ [[ disabled != online ]]
9062023-10-31T04:25:58.783Z+ [[ 17 -gt 60 ]]
9072023-10-31T04:25:58.809Z+ sleep 1
9082023-10-31T04:25:59.685Z+ retry=18
9092023-10-31T04:25:59.716Z+++ zoneadm list -n
9102023-10-31T04:25:59.746Z+++ grep oxz_ntp
9112023-10-31T04:25:59.775Z++ pfexec svcs -z oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87 -Hostate oxide/ntp
9122023-10-31T04:25:59.801Z+ [[ disabled != online ]]
9132023-10-31T04:25:59.827Z+ [[ 18 -gt 60 ]]
9142023-10-31T04:25:59.851Z+ sleep 1
9152023-10-31T04:26:00.703Z+ retry=19
9162023-10-31T04:26:00.732Z+++ zoneadm list -n
9172023-10-31T04:26:00.761Z+++ grep oxz_ntp
9182023-10-31T04:26:00.788Z++ pfexec svcs -z oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87 -Hostate oxide/ntp
9192023-10-31T04:26:00.813Z+ [[ disabled != online ]]
9202023-10-31T04:26:00.837Z+ [[ 19 -gt 60 ]]
9212023-10-31T04:26:00.862Z+ sleep 1
9222023-10-31T04:26:01.734Z+ retry=20
9232023-10-31T04:26:01.768Z+++ zoneadm list -n
9242023-10-31T04:26:01.800Z+++ grep oxz_ntp
9252023-10-31T04:26:01.833Z++ pfexec svcs -z oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87 -Hostate oxide/ntp
9262023-10-31T04:26:01.861Z+ [[ disabled != online ]]
9272023-10-31T04:26:01.891Z+ [[ 20 -gt 60 ]]
9282023-10-31T04:26:01.918Z+ sleep 1
 
9292023-10-31T04:26:02.386Zbuskin console login: Oct 31 04:26:01 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9302023-10-31T04:26:03.180Z+ retry=21
9312023-10-31T04:26:03.215Z+++ zoneadm list -n
9322023-10-31T04:26:03.702Z+++ grep oxz_ntp
9332023-10-31T04:26:03.754Z++ pfexec svcs -z oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87 -Hostate oxide/ntp
9342023-10-31T04:26:03.788Z+ [[ online != online ]]
9352023-10-31T04:26:03.819Z+ echo 'Waited for chrony: 21s'
9362023-10-31T04:26:03.846Z+ export RUST_BACKTRACE=1
9372023-10-31T04:26:03.874ZWaited for chrony: 21s
9382023-10-31T04:26:03.902Z+ RUST_BACKTRACE=1
9392023-10-31T04:26:03.941Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9402023-10-31T04:26:03.977Z+ ./tests/bootstrap
9412023-10-31T04:26:04.009ZUsing external DNS server at 10.151.1.164:53
9422023-10-31T04:26:04.043Z2023-10-31 04:26:02.731773579 UTC: attempting to log into API
9432023-10-31T04:26:17.833Z2023-10-31 04:26:17.794714062 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9442023-10-31T04:26:19.284Z2023-10-31 04:26:18.797914505 UTC: attempting to log into API
9452023-10-31T04:26:33.868Z2023-10-31 04:26:33.858299476 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9462023-10-31T04:26:34.868Z2023-10-31 04:26:34.859885584 UTC: attempting to log into API
9472023-10-31T04:26:49.902Z2023-10-31 04:26:49.893368309 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9482023-10-31T04:26:50.904Z2023-10-31 04:26:50.895284552 UTC: attempting to log into API
9492023-10-31T04:27:05.946Z2023-10-31 04:27:05.928273240 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9502023-10-31T04:27:06.939Z2023-10-31 04:27:06.929811637 UTC: attempting to log into API
9512023-10-31T04:27:21.973Z2023-10-31 04:27:21.964251435 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9522023-10-31T04:27:22.974Z2023-10-31 04:27:22.965096941 UTC: attempting to log into API
9532023-10-31T04:27:38.025Z2023-10-31 04:27:38.016614614 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9542023-10-31T04:27:39.027Z2023-10-31 04:27:39.018592557 UTC: attempting to log into API
9552023-10-31T04:27:54.066Z2023-10-31 04:27:54.056729391 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9562023-10-31T04:27:55.135Z2023-10-31 04:27:55.123516171 UTC: attempting to log into API
9572023-10-31T04:28:10.411Z2023-10-31 04:28:10.402146932 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9582023-10-31T04:28:11.413Z2023-10-31 04:28:11.404092055 UTC: attempting to log into API
9592023-10-31T04:28:26.450Z2023-10-31 04:28:26.438486944 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9602023-10-31T04:28:26.662ZOct 31 04:28:26 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9612023-10-31T04:28:27.449Z2023-10-31 04:28:27.439377700 UTC: attempting to log into API
 
9622023-10-31T04:28:34.421ZOct 31 04:28:33 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9632023-10-31T04:28:41.946ZOct 31 04:28:41 bukin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9642023-10-31T04:28:42.482Z2023-10-31 04:28:42.473257073 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9652023-10-31T04:28:43.484Z2023-10-31 04:28:43.474981123 UTC: attempting to log into API
 
9662023-10-31T04:28:51.807ZOct 31 04:28:51 bukin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9672023-10-31T04:28:58.518Z2023-10-31 04:28:58.507913861 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9682023-10-31T04:28:59.315ZOct 31 04:28:58 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9692023-10-31T04:28:59.520Z2023-10-31 04:28:59.509943865 UTC: attempting to log into API
9702023-10-31T04:29:14.553Z2023-10-31 04:29:14.542292914 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9712023-10-31T04:29:15.555Z2023-10-31 04:29:15.543320693 UTC: attempting to log into API
9722023-10-31T04:29:26.357Z2023-10-31 04:29:26.347334022 UTC: login succeeded
9732023-10-31T04:29:26.391Zwaiting for nexus to come up...
9742023-10-31T04:29:26.451Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9752023-10-31T04:29:26.501Zensuring datasets are ready...
9762023-10-31T04:29:29.091Zlet's roll.
9772023-10-31T04:29:29.118Z+ rm ./tests/bootstrap
9782023-10-31T04:29:29.182Z+ for test_bin in tests/*
9792023-10-31T04:29:29.208Z+ ./tests/bootstrap-13d7796e9c457a2a
9802023-10-31T04:29:29.232Z
9812023-10-31T04:29:29.257Zrunning 0 tests
9822023-10-31T04:29:29.281Z
9832023-10-31T04:29:29.306Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9842023-10-31T04:29:29.331Z
9852023-10-31T04:29:29.356Z+ for test_bin in tests/*
9862023-10-31T04:29:29.380Z+ ./tests/end_to_end_tests-cc8f9cfe063d8adb
9872023-10-31T04:29:29.404Z
9882023-10-31T04:29:29.428Zrunning 2 tests
9892023-10-31T04:29:29.454Ztest no_spoof::no_spoof_here ... ok
9902023-10-31T04:30:29.208Ztest instance_launch::instance_launch has been running for over 60 seconds
9912023-10-31T04:32:20.784Ztest instance_launch::instance_launch ... ok
9922023-10-31T04:32:20.810Z
9932023-10-31T04:32:20.840Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 171.57s
9942023-10-31T04:32:20.885Z
9952023-10-31T04:32:20.923Z+ _exit_trap
9962023-10-31T04:32:20.952Z+ local status=0
9972023-10-31T04:32:20.978Z+ [[ 0 -eq 0 ]]
9982023-10-31T04:32:21.004Z+ exit 0
9992023-10-31T04:32:21.029Zprocess exited: duration 462856 ms, exit code 0
 
10002023-10-31T04:32:21.136Zfound 90 output files
10012023-10-31T04:32:21.351Zuploading: /var/svc/log/oxide-sled-agent:default.log (461287 bytes)
10022023-10-31T04:32:22.396Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10032023-10-31T04:32:22.426Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1a212e0f-cc77-4b3d-ad90-63574850d66e/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10042023-10-31T04:32:23.477Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1a212e0f-cc77-4b3d-ad90-63574850d66e/root/var/svc/log/oxide-cockroachdb:default.log
10052023-10-31T04:32:23.508Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1a212e0f-cc77-4b3d-ad90-63574850d66e/root/var/svc/log/oxide-cockroachdb:default.log.0 (5343 bytes)
10062023-10-31T04:32:24.598Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1a212e0f-cc77-4b3d-ad90-63574850d66e/root/var/svc/log/oxide-cockroachdb:default.log.0
10072023-10-31T04:32:24.626Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_94c01d63-ceb7-49c2-b84f-fbe2236568fa/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10082023-10-31T04:32:25.693Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_94c01d63-ceb7-49c2-b84f-fbe2236568fa/root/var/svc/log/oxide-cockroachdb:default.log
10092023-10-31T04:32:25.728Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_94c01d63-ceb7-49c2-b84f-fbe2236568fa/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
10102023-10-31T04:32:26.817Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_94c01d63-ceb7-49c2-b84f-fbe2236568fa/root/var/svc/log/oxide-cockroachdb:default.log.0
10112023-10-31T04:32:26.845Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f8a9437-a579-470d-b4fd-6a58af7313b1/root/var/svc/log/oxide-crucible-agent:default.log (4321 bytes)
10122023-10-31T04:32:27.921Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f8a9437-a579-470d-b4fd-6a58af7313b1/root/var/svc/log/oxide-crucible-agent:default.log
10132023-10-31T04:32:27.946Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f8a9437-a579-470d-b4fd-6a58af7313b1/root/var/svc/log/oxide-crucible-agent:default.log.0 (9789 bytes)
10142023-10-31T04:32:29.038Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f8a9437-a579-470d-b4fd-6a58af7313b1/root/var/svc/log/oxide-crucible-agent:default.log.0
10152023-10-31T04:32:29.063Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f8a9437-a579-470d-b4fd-6a58af7313b1/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10162023-10-31T04:32:30.129Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f8a9437-a579-470d-b4fd-6a58af7313b1/root/var/svc/log/oxide-crucible-downstairs:default.log
10172023-10-31T04:32:30.160Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f8a9437-a579-470d-b4fd-6a58af7313b1/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10182023-10-31T04:32:31.233Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f8a9437-a579-470d-b4fd-6a58af7313b1/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10192023-10-31T04:32:31.266Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f8a9437-a579-470d-b4fd-6a58af7313b1/root/var/svc/log/oxide-crucible-downstairs:downstairs-0473a80e-c344-40d2-a2f4-0ae4ab70bb07.log (1005 bytes)
10202023-10-31T04:32:32.356Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f8a9437-a579-470d-b4fd-6a58af7313b1/root/var/svc/log/oxide-crucible-downstairs:downstairs-0473a80e-c344-40d2-a2f4-0ae4ab70bb07.log
10212023-10-31T04:32:32.385Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f8a9437-a579-470d-b4fd-6a58af7313b1/root/var/svc/log/oxide-crucible-downstairs:downstairs-0473a80e-c344-40d2-a2f4-0ae4ab70bb07.log.0 (4531 bytes)
10222023-10-31T04:32:33.467Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f8a9437-a579-470d-b4fd-6a58af7313b1/root/var/svc/log/oxide-crucible-downstairs:downstairs-0473a80e-c344-40d2-a2f4-0ae4ab70bb07.log.0
10232023-10-31T04:32:33.499Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_43749af3-d9db-4bc9-8e12-6cbad6275afd/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10242023-10-31T04:32:34.545Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_43749af3-d9db-4bc9-8e12-6cbad6275afd/root/var/svc/log/oxide-crucible-pantry:default.log
10252023-10-31T04:32:34.573Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_43749af3-d9db-4bc9-8e12-6cbad6275afd/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10262023-10-31T04:32:35.677Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_43749af3-d9db-4bc9-8e12-6cbad6275afd/root/var/svc/log/oxide-crucible-pantry:default.log.0
10272023-10-31T04:32:35.708Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_f988c0aa-8e3d-4eec-ba69-c41fb9b4d9cb/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10282023-10-31T04:32:36.757Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_f988c0aa-8e3d-4eec-ba69-c41fb9b4d9cb/root/var/svc/log/oxide-crucible-pantry:default.log
10292023-10-31T04:32:36.782Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_f988c0aa-8e3d-4eec-ba69-c41fb9b4d9cb/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
10302023-10-31T04:32:37.868Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_f988c0aa-8e3d-4eec-ba69-c41fb9b4d9cb/root/var/svc/log/oxide-crucible-pantry:default.log.0
10312023-10-31T04:32:37.905Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3369da64-7b6f-4185-8b29-c39003c18e25/root/var/svc/log/oxide-oximeter:default.log (128636 bytes)
10322023-10-31T04:32:38.512Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3369da64-7b6f-4185-8b29-c39003c18e25/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 128636 -> 144591
10332023-10-31T04:32:39.589Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3369da64-7b6f-4185-8b29-c39003c18e25/root/var/svc/log/oxide-oximeter:default.log
10342023-10-31T04:32:39.616Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3369da64-7b6f-4185-8b29-c39003c18e25/root/var/svc/log/oxide-oximeter:default.log.0 (67031 bytes)
10352023-10-31T04:32:40.799Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3369da64-7b6f-4185-8b29-c39003c18e25/root/var/svc/log/oxide-oximeter:default.log.0
10362023-10-31T04:32:40.827Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_2bbd927e-92ae-4ba6-883c-e2bec3c88bba/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10372023-10-31T04:32:41.871Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_2bbd927e-92ae-4ba6-883c-e2bec3c88bba/root/var/svc/log/oxide-cockroachdb:default.log
10382023-10-31T04:32:41.897Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_2bbd927e-92ae-4ba6-883c-e2bec3c88bba/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10392023-10-31T04:32:42.975Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_2bbd927e-92ae-4ba6-883c-e2bec3c88bba/root/var/svc/log/oxide-cockroachdb:default.log.0
10402023-10-31T04:32:43.010Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_a58f410f-1db5-4e87-a28c-25db21257a91/root/var/svc/log/oxide-external_dns:default.log (7551 bytes)
10412023-10-31T04:32:45.006Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_a58f410f-1db5-4e87-a28c-25db21257a91/root/var/svc/log/oxide-external_dns:default.log
10422023-10-31T04:32:45.032Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_a58f410f-1db5-4e87-a28c-25db21257a91/root/var/svc/log/oxide-external_dns:default.log.0 (8056 bytes)
10432023-10-31T04:32:46.141Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_a58f410f-1db5-4e87-a28c-25db21257a91/root/var/svc/log/oxide-external_dns:default.log.0
10442023-10-31T04:32:46.166Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_be649400-193c-4cc5-9b58-d4e43c1d6015/root/var/svc/log/oxide-nexus:default.log (125329 bytes)
10452023-10-31T04:32:46.795Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_be649400-193c-4cc5-9b58-d4e43c1d6015/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 125329 -> 126545
10462023-10-31T04:32:48.406Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_be649400-193c-4cc5-9b58-d4e43c1d6015/root/var/svc/log/oxide-nexus:default.log
10472023-10-31T04:32:48.445Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_be649400-193c-4cc5-9b58-d4e43c1d6015/root/var/svc/log/oxide-nexus:default.log.0 (693246 bytes)
10482023-10-31T04:32:51.818Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_be649400-193c-4cc5-9b58-d4e43c1d6015/root/var/svc/log/oxide-nexus:default.log.0
10492023-10-31T04:32:51.843Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3639055c-cad4-4944-83e6-62f50bfab860/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10502023-10-31T04:32:52.895Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3639055c-cad4-4944-83e6-62f50bfab860/root/var/svc/log/oxide-crucible-agent:default.log
10512023-10-31T04:32:52.921Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3639055c-cad4-4944-83e6-62f50bfab860/root/var/svc/log/oxide-crucible-agent:default.log.0 (14115 bytes)
10522023-10-31T04:32:54.032Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3639055c-cad4-4944-83e6-62f50bfab860/root/var/svc/log/oxide-crucible-agent:default.log.0
10532023-10-31T04:32:54.477Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3639055c-cad4-4944-83e6-62f50bfab860/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10542023-10-31T04:32:55.522Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3639055c-cad4-4944-83e6-62f50bfab860/root/var/svc/log/oxide-crucible-downstairs:default.log
10552023-10-31T04:32:55.548Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3639055c-cad4-4944-83e6-62f50bfab860/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10562023-10-31T04:32:56.636Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3639055c-cad4-4944-83e6-62f50bfab860/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10572023-10-31T04:32:56.662Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3639055c-cad4-4944-83e6-62f50bfab860/root/var/svc/log/oxide-crucible-downstairs:downstairs-21735afa-0631-4e65-bd04-3d1aaaac98b6.log (0 bytes)
10582023-10-31T04:32:57.853Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3639055c-cad4-4944-83e6-62f50bfab860/root/var/svc/log/oxide-crucible-downstairs:downstairs-21735afa-0631-4e65-bd04-3d1aaaac98b6.log
10592023-10-31T04:32:57.879Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3639055c-cad4-4944-83e6-62f50bfab860/root/var/svc/log/oxide-crucible-downstairs:downstairs-21735afa-0631-4e65-bd04-3d1aaaac98b6.log.0 (3050 bytes)
10602023-10-31T04:32:58.814Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3639055c-cad4-4944-83e6-62f50bfab860/root/var/svc/log/oxide-crucible-downstairs:downstairs-21735afa-0631-4e65-bd04-3d1aaaac98b6.log.0
10612023-10-31T04:32:58.847Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7fb3c683-4b65-4163-80ba-55fa5d2ae0a4/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10622023-10-31T04:33:00.307Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7fb3c683-4b65-4163-80ba-55fa5d2ae0a4/root/var/svc/log/oxide-crucible-agent:default.log
10632023-10-31T04:33:00.334Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7fb3c683-4b65-4163-80ba-55fa5d2ae0a4/root/var/svc/log/oxide-crucible-agent:default.log.0 (14108 bytes)
10642023-10-31T04:33:01.435Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7fb3c683-4b65-4163-80ba-55fa5d2ae0a4/root/var/svc/log/oxide-crucible-agent:default.log.0
10652023-10-31T04:33:01.461Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7fb3c683-4b65-4163-80ba-55fa5d2ae0a4/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10662023-10-31T04:33:02.928Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7fb3c683-4b65-4163-80ba-55fa5d2ae0a4/root/var/svc/log/oxide-crucible-downstairs:default.log
10672023-10-31T04:33:02.960Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7fb3c683-4b65-4163-80ba-55fa5d2ae0a4/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10682023-10-31T04:33:04.051Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7fb3c683-4b65-4163-80ba-55fa5d2ae0a4/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10692023-10-31T04:33:04.104Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7fb3c683-4b65-4163-80ba-55fa5d2ae0a4/root/var/svc/log/oxide-crucible-downstairs:downstairs-d5402c9a-dce3-4b9e-a6fe-9e39eeb2ff53.log (0 bytes)
10702023-10-31T04:33:05.165Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7fb3c683-4b65-4163-80ba-55fa5d2ae0a4/root/var/svc/log/oxide-crucible-downstairs:downstairs-d5402c9a-dce3-4b9e-a6fe-9e39eeb2ff53.log
10712023-10-31T04:33:05.189Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7fb3c683-4b65-4163-80ba-55fa5d2ae0a4/root/var/svc/log/oxide-crucible-downstairs:downstairs-d5402c9a-dce3-4b9e-a6fe-9e39eeb2ff53.log.0 (3050 bytes)
10722023-10-31T04:33:06.264Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7fb3c683-4b65-4163-80ba-55fa5d2ae0a4/root/var/svc/log/oxide-crucible-downstairs:downstairs-d5402c9a-dce3-4b9e-a6fe-9e39eeb2ff53.log.0
10732023-10-31T04:33:06.299Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9b47551d-0498-4051-a287-3f20ffb99818/root/var/svc/log/oxide-nexus:default.log (1524661 bytes)
10742023-10-31T04:33:09.459Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9b47551d-0498-4051-a287-3f20ffb99818/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1524661 -> 1585885
10752023-10-31T04:33:10.543Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9b47551d-0498-4051-a287-3f20ffb99818/root/var/svc/log/oxide-nexus:default.log
10762023-10-31T04:33:10.586Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9b47551d-0498-4051-a287-3f20ffb99818/root/var/svc/log/oxide-nexus:default.log.0 (2798120 bytes)
10772023-10-31T04:33:13.655Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9b47551d-0498-4051-a287-3f20ffb99818/root/var/svc/log/oxide-nexus:default.log.0
10782023-10-31T04:33:13.685Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87/root/var/svc/log/oxide-ntp:default.log (0 bytes)
10792023-10-31T04:33:14.757Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87/root/var/svc/log/oxide-ntp:default.log
10802023-10-31T04:33:14.784Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87/root/var/svc/log/oxide-ntp:default.log.0 (783 bytes)
10812023-10-31T04:33:15.869Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87/root/var/svc/log/oxide-ntp:default.log.0
10822023-10-31T04:33:15.895Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_850ef856-c4dd-4758-b70f-4b5d98f61d05/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10832023-10-31T04:33:16.956Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_850ef856-c4dd-4758-b70f-4b5d98f61d05/root/var/svc/log/oxide-cockroachdb:default.log
10842023-10-31T04:33:16.994Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_850ef856-c4dd-4758-b70f-4b5d98f61d05/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10852023-10-31T04:33:18.084Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_850ef856-c4dd-4758-b70f-4b5d98f61d05/root/var/svc/log/oxide-cockroachdb:default.log.0
10862023-10-31T04:33:18.110Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fd3adc06-4168-4b48-ba28-2207c91c8c22/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10872023-10-31T04:33:19.180Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fd3adc06-4168-4b48-ba28-2207c91c8c22/root/var/svc/log/oxide-crucible-agent:default.log
10882023-10-31T04:33:19.217Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fd3adc06-4168-4b48-ba28-2207c91c8c22/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10892023-10-31T04:33:20.297Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fd3adc06-4168-4b48-ba28-2207c91c8c22/root/var/svc/log/oxide-crucible-agent:default.log.0
10902023-10-31T04:33:20.334Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fd3adc06-4168-4b48-ba28-2207c91c8c22/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10912023-10-31T04:33:21.404Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fd3adc06-4168-4b48-ba28-2207c91c8c22/root/var/svc/log/oxide-crucible-downstairs:default.log
10922023-10-31T04:33:21.436Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fd3adc06-4168-4b48-ba28-2207c91c8c22/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10932023-10-31T04:33:22.531Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fd3adc06-4168-4b48-ba28-2207c91c8c22/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10942023-10-31T04:33:22.563Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_0896f66c-755e-4b3c-86c5-bd6f78d7ba80/root/var/svc/log/oxide-external_dns:default.log (7547 bytes)
10952023-10-31T04:33:22.626Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_0896f66c-755e-4b3c-86c5-bd6f78d7ba80/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 7547 -> 11323
10962023-10-31T04:33:23.714Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_0896f66c-755e-4b3c-86c5-bd6f78d7ba80/root/var/svc/log/oxide-external_dns:default.log
10972023-10-31T04:33:23.741Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_0896f66c-755e-4b3c-86c5-bd6f78d7ba80/root/var/svc/log/oxide-external_dns:default.log.0 (8053 bytes)
10982023-10-31T04:33:24.905Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_0896f66c-755e-4b3c-86c5-bd6f78d7ba80/root/var/svc/log/oxide-external_dns:default.log.0
10992023-10-31T04:33:25.386Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_6a668c4f-ac94-47ca-9483-c60dc70d5677/root/var/svc/log/oxide-internal_dns:default.log (17587 bytes)
11002023-10-31T04:33:25.433Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_6a668c4f-ac94-47ca-9483-c60dc70d5677/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 17587 -> 21340
11012023-10-31T04:33:26.522Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_6a668c4f-ac94-47ca-9483-c60dc70d5677/root/var/svc/log/oxide-internal_dns:default.log
11022023-10-31T04:33:26.557Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_6a668c4f-ac94-47ca-9483-c60dc70d5677/root/var/svc/log/oxide-internal_dns:default.log.0 (97395 bytes)
11032023-10-31T04:33:28.223Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_6a668c4f-ac94-47ca-9483-c60dc70d5677/root/var/svc/log/oxide-internal_dns:default.log.0
11042023-10-31T04:33:28.251Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_bd835483-301f-4bf1-8073-d107c1c90912/root/var/svc/log/oxide-internal_dns:default.log (18426 bytes)
11052023-10-31T04:33:28.355Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_bd835483-301f-4bf1-8073-d107c1c90912/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 18426 -> 22181
11062023-10-31T04:33:29.456Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_bd835483-301f-4bf1-8073-d107c1c90912/root/var/svc/log/oxide-internal_dns:default.log
11072023-10-31T04:33:29.481Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_bd835483-301f-4bf1-8073-d107c1c90912/root/var/svc/log/oxide-internal_dns:default.log.0 (104146 bytes)
11082023-10-31T04:33:31.087Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_bd835483-301f-4bf1-8073-d107c1c90912/root/var/svc/log/oxide-internal_dns:default.log.0
11092023-10-31T04:33:31.128Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_52e20159-6098-4a84-9ea0-8e98c43f915c/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11102023-10-31T04:33:32.203Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_52e20159-6098-4a84-9ea0-8e98c43f915c/root/var/svc/log/oxide-crucible-pantry:default.log
11112023-10-31T04:33:32.236Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_52e20159-6098-4a84-9ea0-8e98c43f915c/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
11122023-10-31T04:33:33.326Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_52e20159-6098-4a84-9ea0-8e98c43f915c/root/var/svc/log/oxide-crucible-pantry:default.log.0
11132023-10-31T04:33:33.356Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_08b0be68-e9bb-472c-a333-9f28b01ec4b1/root/var/svc/log/oxide-clickhouse:default.log (484694 bytes)
11142023-10-31T04:33:34.351Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_08b0be68-e9bb-472c-a333-9f28b01ec4b1/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 484694 -> 735944
11152023-10-31T04:33:35.429Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_08b0be68-e9bb-472c-a333-9f28b01ec4b1/root/var/svc/log/oxide-clickhouse:default.log
11162023-10-31T04:33:35.458Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_08b0be68-e9bb-472c-a333-9f28b01ec4b1/root/var/svc/log/oxide-clickhouse:default.log.0 (259303 bytes)
11172023-10-31T04:33:38.022Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_08b0be68-e9bb-472c-a333-9f28b01ec4b1/root/var/svc/log/oxide-clickhouse:default.log.0
11182023-10-31T04:33:38.053Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4a10612c-4f87-444c-8fa1-c9f341a99878/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11192023-10-31T04:33:39.114Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4a10612c-4f87-444c-8fa1-c9f341a99878/root/var/svc/log/oxide-crucible-agent:default.log
11202023-10-31T04:33:39.144Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4a10612c-4f87-444c-8fa1-c9f341a99878/root/var/svc/log/oxide-crucible-agent:default.log.0 (3434 bytes)
11212023-10-31T04:33:40.650Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4a10612c-4f87-444c-8fa1-c9f341a99878/root/var/svc/log/oxide-crucible-agent:default.log.0
11222023-10-31T04:33:40.675Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4a10612c-4f87-444c-8fa1-c9f341a99878/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11232023-10-31T04:33:41.728Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4a10612c-4f87-444c-8fa1-c9f341a99878/root/var/svc/log/oxide-crucible-downstairs:default.log
11242023-10-31T04:33:41.759Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4a10612c-4f87-444c-8fa1-c9f341a99878/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11252023-10-31T04:33:42.847Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4a10612c-4f87-444c-8fa1-c9f341a99878/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11262023-10-31T04:33:42.879Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_d4d5a033-f745-4c62-b5c8-eb84670e0e15/root/var/svc/log/oxide-nexus:default.log (125325 bytes)
11272023-10-31T04:33:43.324Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_d4d5a033-f745-4c62-b5c8-eb84670e0e15/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 125325 -> 188374
11282023-10-31T04:33:44.420Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_d4d5a033-f745-4c62-b5c8-eb84670e0e15/root/var/svc/log/oxide-nexus:default.log
11292023-10-31T04:33:44.453Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_d4d5a033-f745-4c62-b5c8-eb84670e0e15/root/var/svc/log/oxide-nexus:default.log.0 (368154 bytes)
11302023-10-31T04:33:46.242Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_d4d5a033-f745-4c62-b5c8-eb84670e0e15/root/var/svc/log/oxide-nexus:default.log.0
11312023-10-31T04:33:46.273Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_20d16e3a-390d-4775-81f9-6c4ddd4d738f/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11322023-10-31T04:33:47.345Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_20d16e3a-390d-4775-81f9-6c4ddd4d738f/root/var/svc/log/oxide-crucible-agent:default.log
11332023-10-31T04:33:47.380Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_20d16e3a-390d-4775-81f9-6c4ddd4d738f/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11342023-10-31T04:33:48.487Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_20d16e3a-390d-4775-81f9-6c4ddd4d738f/root/var/svc/log/oxide-crucible-agent:default.log.0
11352023-10-31T04:33:48.528Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_20d16e3a-390d-4775-81f9-6c4ddd4d738f/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11362023-10-31T04:33:49.970Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_20d16e3a-390d-4775-81f9-6c4ddd4d738f/root/var/svc/log/oxide-crucible-downstairs:default.log
11372023-10-31T04:33:49.995Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_20d16e3a-390d-4775-81f9-6c4ddd4d738f/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11382023-10-31T04:33:51.076Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_20d16e3a-390d-4775-81f9-6c4ddd4d738f/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11392023-10-31T04:33:51.101Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_5fe14c67-bb91-4a8d-9b9f-09ed2228c5df/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11402023-10-31T04:33:52.153Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_5fe14c67-bb91-4a8d-9b9f-09ed2228c5df/root/var/svc/log/oxide-cockroachdb:default.log
11412023-10-31T04:33:52.182Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_5fe14c67-bb91-4a8d-9b9f-09ed2228c5df/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
11422023-10-31T04:33:53.267Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_5fe14c67-bb91-4a8d-9b9f-09ed2228c5df/root/var/svc/log/oxide-cockroachdb:default.log.0
11432023-10-31T04:33:53.297Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba9f4f06-2c18-4fd2-b730-ef73fffb0e09/root/var/svc/log/oxide-crucible-agent:default.log (4324 bytes)
11442023-10-31T04:33:54.384Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba9f4f06-2c18-4fd2-b730-ef73fffb0e09/root/var/svc/log/oxide-crucible-agent:default.log
11452023-10-31T04:33:54.413Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba9f4f06-2c18-4fd2-b730-ef73fffb0e09/root/var/svc/log/oxide-crucible-agent:default.log.0 (9795 bytes)
11462023-10-31T04:33:55.530Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba9f4f06-2c18-4fd2-b730-ef73fffb0e09/root/var/svc/log/oxide-crucible-agent:default.log.0
11472023-10-31T04:33:55.643Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba9f4f06-2c18-4fd2-b730-ef73fffb0e09/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11482023-10-31T04:33:56.702Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba9f4f06-2c18-4fd2-b730-ef73fffb0e09/root/var/svc/log/oxide-crucible-downstairs:default.log
11492023-10-31T04:33:57.142Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba9f4f06-2c18-4fd2-b730-ef73fffb0e09/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11502023-10-31T04:33:58.245Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba9f4f06-2c18-4fd2-b730-ef73fffb0e09/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11512023-10-31T04:33:58.282Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba9f4f06-2c18-4fd2-b730-ef73fffb0e09/root/var/svc/log/oxide-crucible-downstairs:downstairs-d9ab7adf-8bf7-48e3-989b-c014a3f62b82.log (1005 bytes)
11522023-10-31T04:34:00.166Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba9f4f06-2c18-4fd2-b730-ef73fffb0e09/root/var/svc/log/oxide-crucible-downstairs:downstairs-d9ab7adf-8bf7-48e3-989b-c014a3f62b82.log
11532023-10-31T04:34:00.268Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba9f4f06-2c18-4fd2-b730-ef73fffb0e09/root/var/svc/log/oxide-crucible-downstairs:downstairs-d9ab7adf-8bf7-48e3-989b-c014a3f62b82.log.0 (4531 bytes)
11542023-10-31T04:34:00.890Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba9f4f06-2c18-4fd2-b730-ef73fffb0e09/root/var/svc/log/oxide-crucible-downstairs:downstairs-d9ab7adf-8bf7-48e3-989b-c014a3f62b82.log.0
11552023-10-31T04:34:00.922Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-agent:default.log (4784 bytes)
11562023-10-31T04:34:02.414Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-agent:default.log
11572023-10-31T04:34:02.442Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-agent:default.log.0 (19607 bytes)
11582023-10-31T04:34:03.576Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-agent:default.log.0
11592023-10-31T04:34:03.601Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11602023-10-31T04:34:04.667Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-downstairs:default.log
11612023-10-31T04:34:04.692Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11622023-10-31T04:34:06.179Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11632023-10-31T04:34:06.208Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-downstairs:downstairs-4ae5a2dc-801c-4dd3-9185-656bc8aac892.log (0 bytes)
11642023-10-31T04:34:07.253Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-downstairs:downstairs-4ae5a2dc-801c-4dd3-9185-656bc8aac892.log
11652023-10-31T04:34:07.277Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-downstairs:downstairs-4ae5a2dc-801c-4dd3-9185-656bc8aac892.log.0 (3050 bytes)
11662023-10-31T04:34:08.360Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-downstairs:downstairs-4ae5a2dc-801c-4dd3-9185-656bc8aac892.log.0
11672023-10-31T04:34:08.390Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-downstairs:downstairs-c2f19a2f-b0dc-409e-83f1-20421b229744.log (1005 bytes)
11682023-10-31T04:34:09.869Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-downstairs:downstairs-c2f19a2f-b0dc-409e-83f1-20421b229744.log
11692023-10-31T04:34:09.902Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-downstairs:downstairs-c2f19a2f-b0dc-409e-83f1-20421b229744.log.0 (4530 bytes)
11702023-10-31T04:34:10.989Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c160566f-03e4-4161-8394-9a6f38282e12/root/var/svc/log/oxide-crucible-downstairs:downstairs-c2f19a2f-b0dc-409e-83f1-20421b229744.log.0
11712023-10-31T04:34:11.017Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df272798-2e7c-402d-9623-8109e7b7e6ba/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11722023-10-31T04:34:12.084Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df272798-2e7c-402d-9623-8109e7b7e6ba/root/var/svc/log/oxide-crucible-agent:default.log
11732023-10-31T04:34:12.114Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df272798-2e7c-402d-9623-8109e7b7e6ba/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11742023-10-31T04:34:13.210Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df272798-2e7c-402d-9623-8109e7b7e6ba/root/var/svc/log/oxide-crucible-agent:default.log.0
11752023-10-31T04:34:13.246Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df272798-2e7c-402d-9623-8109e7b7e6ba/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11762023-10-31T04:34:14.304Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df272798-2e7c-402d-9623-8109e7b7e6ba/root/var/svc/log/oxide-crucible-downstairs:default.log
11772023-10-31T04:34:14.335Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df272798-2e7c-402d-9623-8109e7b7e6ba/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11782023-10-31T04:34:15.434Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df272798-2e7c-402d-9623-8109e7b7e6ba/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11792023-10-31T04:34:15.461Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_444e634d-13f0-4b19-854e-d9ee6fd42928/root/var/svc/log/oxide-internal_dns:default.log (17589 bytes)
11802023-10-31T04:34:16.473Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_444e634d-13f0-4b19-854e-d9ee6fd42928/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 17589 -> 25102
11812023-10-31T04:34:17.551Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_444e634d-13f0-4b19-854e-d9ee6fd42928/root/var/svc/log/oxide-internal_dns:default.log
11822023-10-31T04:34:17.578Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_444e634d-13f0-4b19-854e-d9ee6fd42928/root/var/svc/log/oxide-internal_dns:default.log.0 (97529 bytes)
11832023-10-31T04:34:19.188Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_444e634d-13f0-4b19-854e-d9ee6fd42928/root/var/svc/log/oxide-internal_dns:default.log.0
11842023-10-31T04:34:19.217Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87/root/var/log/chrony/measurements.log (9590 bytes)
11852023-10-31T04:34:19.292Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87/root/var/log/chrony/measurements.log" changed size mid upload: 9590 -> 15481
11862023-10-31T04:34:20.804Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87/root/var/log/chrony/measurements.log
11872023-10-31T04:34:20.832Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87/root/var/log/chrony/statistics.log (5382 bytes)
11882023-10-31T04:34:20.893Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87/root/var/log/chrony/statistics.log" changed size mid upload: 5382 -> 7839
11892023-10-31T04:34:21.971Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87/root/var/log/chrony/statistics.log
11902023-10-31T04:34:21.996Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87/root/var/log/chrony/tracking.log (1455 bytes)
11912023-10-31T04:34:22.026Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87/root/var/log/chrony/tracking.log" changed size mid upload: 1455 -> 1719
11922023-10-31T04:34:23.566Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_2f74c702-e7d9-4413-a346-8d150c192e87/root/var/log/chrony/tracking.log