01HGE6CX9EGE1RY4XYKC1FAFR2: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HGE6DQSCDH9RRP8MQKXGDQ4Z

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-29T19:35:27.730Zjob dependencies complete; ready to run (waiting for 49 m 34 s)
22023-11-29T19:37:26.531Zjob assigned to worker 01HGE9ANX6NKZZ5V65YKJCPRJ2 (queued for 1 m 58 s)
32023-11-29T19:37:26.599Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-11-29T19:37:26.658Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-11-29T19:37:26.701Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-11-29T19:37:26.737Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-29T19:37:26.770Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-11-29T19:37:26.809ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-29T19:37:26.852Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-29T19:37:26.892ZOxide Helios Version helios-2.0.22318 64-bit
112023-11-29T19:37:26.929ZConfiguring devices.
122023-11-29T19:37:26.974Z#################################################
132023-11-29T19:37:27.012Z#### ############# ##
142023-11-29T19:37:27.049Z### ### ############ ##
152023-11-29T19:37:27.082Z## ### # ## ### ## Oxide Computer Company ##
162023-11-29T19:37:27.125Z## ## ## ### # ### ##
172023-11-29T19:37:27.162Z## # ### #### #### This Station Under ##
182023-11-29T19:37:27.208Z### ### #### # ### Computer Control ##
192023-11-29T19:37:27.249Z#### #### ### ## ##
202023-11-29T19:37:27.285Z##################################################
212023-11-29T19:37:27.326ZHostname: buskin
222023-11-29T19:37:27.373ZLoading smf(7) service descriptions:
232023-11-29T19:37:27.889Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-fe3b4babdbd1397e.gz
242023-11-29T19:37:29.902Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-fe3b4babdbd1397e.gz
252023-11-29T19:37:29.902Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-29T19:37:58.927Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-29T19:37:58.952Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-29e4ffcd832e7ef8.gz
282023-11-29T19:38:20.904Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-29e4ffcd832e7ef8.gz
292023-11-29T19:38:20.904Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-29T19:38:22.738Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-29T19:38:22.738Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-29T19:38:26.510Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-29T19:38:26.540Zdownloading input: /input/package/work/version.txt
342023-11-29T19:38:26.572Zdownloaded input: /input/package/work/version.txt
352023-11-29T19:38:26.572Zdownloading input: /input/package/work/package.tar.gz
362023-11-29T19:38:44.551Zdownloaded input: /input/package/work/package.tar.gz
372023-11-29T19:38:44.552Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-29T19:39:17.599Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-29T19:39:17.629Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-29T19:39:26.354Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-29T19:39:26.381Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-29T19:40:10.423Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-29T19:40:10.423Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-29T19:40:50.774Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-29T19:40:50.775Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-29T19:41:14.130Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-29T19:41:14.158Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-29T19:41:18.415Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-29T19:41:18.415Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-29T19:41:30.817Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-29T19:41:30.817Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-29T19:41:35.174Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-29T19:41:35.174Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-29T19:41:40.178Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-29T19:41:40.203Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-29T19:41:40.227Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-29T19:41:40.227Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-29T19:41:45.024Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-29T19:41:45.024Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-29T19:41:48.515Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-29T19:41:48.539Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-29T19:42:04.131Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-29T19:42:04.131Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-29T19:42:17.946Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-29T19:42:17.946Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-29T19:42:17.982Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-29T19:42:18.007Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-29T19:42:24.087Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-29T19:42:24.087Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-29T19:42:35.750Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-29T19:42:35.750Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-29T19:43:20.549Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-29T19:43:20.549Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-29T19:43:50.537Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-29T19:43:50.566Zstarting task 0: "setup"
762023-11-29T19:43:50.566Z++ uname -s
772023-11-29T19:43:50.566Z+ kern=SunOS
782023-11-29T19:43:50.566Z+ case "$kern" in
792023-11-29T19:43:50.566Z+ groupadd -g 12345 build
802023-11-29T19:43:50.566Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-29T19:43:52.553Z+ zfs create -o mountpoint=/work rpool/work
822023-11-29T19:43:52.578Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-29T19:43:52.606Z+ home_fs=zfs
842023-11-29T19:43:52.606Z+ [[ zfs == autofs ]]
852023-11-29T19:43:52.606Z+ mkdir -p /home/build
862023-11-29T19:43:52.606Z+ chown build:build /home/build /work
872023-11-29T19:43:54.584Z+ chmod 0700 /home/build /work
882023-11-29T19:43:54.609Zprocess exited: duration 4052 ms, exit code 0
 
892023-11-29T19:43:54.658Zstarting task 1: "authentication"
902023-11-29T19:43:54.701Zprocess exited: duration 42 ms, exit code 0
 
912023-11-29T19:43:54.750Zstarting task 2: "build"
922023-11-29T19:43:54.775Z+ trap _exit_trap EXIT
932023-11-29T19:43:54.775Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-29T19:43:54.955Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-29T19:43:54.955Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-29T19:43:54.955Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-29T19:43:54.955Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-29T19:43:54.955Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-29T19:43:54.955Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-29T19:43:54.955Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-29T19:43:55.008Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-29T19:43:55.008Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-29T19:43:55.012Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-29T19:43:55.012Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-29T19:43:55.012Z+ [[ -d /opt/oxide ]]
1062023-11-29T19:43:55.012Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-29T19:43:55.012Z+ pfexec mkdir /opt/oxide
1082023-11-29T19:43:55.012Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-29T19:43:55.012Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-29T19:43:55.012Z+ cd /opt/oxide-underneath
1112023-11-29T19:43:55.012Z+ pfexec tar ceEp@/f - .
1122023-11-29T19:43:55.012Z+ cd /opt/oxide
1132023-11-29T19:43:55.012Z+ pfexec tar xveEp@/f -
1142023-11-29T19:43:55.081Zx ., 0 bytes, 0 tape blocks
1152023-11-29T19:43:55.081Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-29T19:43:55.081Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-29T19:43:55.081Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-29T19:43:55.105Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-29T19:43:55.105Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-29T19:43:55.128Z+ pfexec mkdir /opt/oxide/work
1212023-11-29T19:43:55.128Z+ pfexec chown build:build /opt/oxide/work
1222023-11-29T19:43:55.128Z+ cd /opt/oxide/work
1232023-11-29T19:43:55.128Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-29T19:43:56.670ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-29T19:43:56.670Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-11-29T19:43:56.670Zx out/omicron-sled-agent.tar, 112199168 bytes, 219139 tape blocks
1272023-11-29T19:43:56.670Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-29T19:43:56.670Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-11-29T19:43:56.670Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-11-29T19:43:56.670Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-11-29T19:43:56.670Zx target/release/omicron-package, 42568976 bytes, 83143 tape blocks
1322023-11-29T19:43:56.670Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-11-29T19:43:56.670Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-29T19:43:56.670Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-29T19:43:56.670Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-29T19:43:56.670Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-29T19:43:56.670Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-29T19:43:56.671Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-29T19:43:56.699Z
1402023-11-29T19:43:56.699Zreal 1.547398971
1412023-11-29T19:43:56.699Zuser 1.389917942
1422023-11-29T19:43:56.699Zsys 0.271646285
1432023-11-29T19:43:56.699Ztrap 0.000071142
1442023-11-29T19:43:56.699Ztflt 0.000000000
1452023-11-29T19:43:56.699Zdflt 0.000000000
1462023-11-29T19:43:56.699Zkflt 0.000000000
1472023-11-29T19:43:56.699Zlock 0.000000000
1482023-11-29T19:43:56.699Zslp 1.392870409
1492023-11-29T19:43:56.699Zlat 0.038195490
1502023-11-29T19:43:56.699Zstop 0.000121461
1512023-11-29T19:43:56.699Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-29T19:43:57.752Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-29T19:43:57.777Z+ mkdir tests
1542023-11-29T19:43:57.777Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-29T19:43:57.777Z+ ptime -m gunzip
1562023-11-29T19:43:57.777Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-fe3b4babdbd1397e
1572023-11-29T19:43:57.915Z
1582023-11-29T19:43:57.940Zreal 0.138873583
1592023-11-29T19:43:57.940Zuser 0.126201025
1602023-11-29T19:43:57.940Zsys 0.012415793
1612023-11-29T19:43:57.940Ztrap 0.000015230
1622023-11-29T19:43:57.940Ztflt 0.000000000
1632023-11-29T19:43:57.940Zdflt 0.000000000
1642023-11-29T19:43:57.940Zkflt 0.000000000
1652023-11-29T19:43:57.940Zlock 0.000000000
1662023-11-29T19:43:57.940Zslp 0.000021490
1672023-11-29T19:43:57.940Zlat 0.000026390
1682023-11-29T19:43:57.940Zstop 0.000122851
1692023-11-29T19:43:57.940Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-fe3b4babdbd1397e
1702023-11-29T19:43:57.940Z+ chmod a+x tests/bootstrap-fe3b4babdbd1397e
1712023-11-29T19:43:57.940Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-29T19:43:57.941Z+ ptime -m gunzip
1732023-11-29T19:43:57.941Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-29T19:44:00.115Z
1752023-11-29T19:44:00.141Zreal 2.188927756
1762023-11-29T19:44:00.141Zuser 2.043466843
1772023-11-29T19:44:00.141Zsys 0.145063727
1782023-11-29T19:44:00.141Ztrap 0.000159942
1792023-11-29T19:44:00.141Ztflt 0.000000000
1802023-11-29T19:44:00.142Zdflt 0.000000000
1812023-11-29T19:44:00.142Zkflt 0.000000000
1822023-11-29T19:44:00.142Zlock 0.000000000
1832023-11-29T19:44:00.142Zslp 0.000011980
1842023-11-29T19:44:00.142Zlat 0.000033960
1852023-11-29T19:44:00.142Zstop 0.000128332
1862023-11-29T19:44:00.142Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-29T19:44:00.142Z+ chmod a+x tests/bootstrap
1882023-11-29T19:44:00.142Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-29T19:44:00.142Z+ ptime -m gunzip
1902023-11-29T19:44:00.142Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-29e4ffcd832e7ef8
1912023-11-29T19:44:02.385Z
1922023-11-29T19:44:02.410Zreal 2.258169731
1932023-11-29T19:44:02.410Zuser 2.110932459
1942023-11-29T19:44:02.410Zsys 0.146696934
1952023-11-29T19:44:02.410Ztrap 0.000163572
1962023-11-29T19:44:02.410Ztflt 0.000000000
1972023-11-29T19:44:02.410Zdflt 0.000000000
1982023-11-29T19:44:02.410Zkflt 0.000000000
1992023-11-29T19:44:02.410Zlock 0.000000000
2002023-11-29T19:44:02.410Zslp 0.000161972
2012023-11-29T19:44:02.410Zlat 0.000026180
2022023-11-29T19:44:02.410Zstop 0.000128401
2032023-11-29T19:44:02.410Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-29e4ffcd832e7ef8
2042023-11-29T19:44:02.410Z+ chmod a+x tests/end_to_end_tests-29e4ffcd832e7ef8
2052023-11-29T19:44:02.410Z+ bmat address ls
2062023-11-29T19:44:02.410ZNAME CIDR FIRST LAST COUNT
2072023-11-29T19:44:02.411Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-11-29T19:44:02.411Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-29T19:44:02.436Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-11-29T19:44:02.436Z+ EXTRA_IP_START=10.151.1.164
2112023-11-29T19:44:02.436Z+ EXTRA_IP_END=10.151.1.227
2122023-11-29T19:44:02.436Z+ EXTRA_IP_COUNT=64
2132023-11-29T19:44:02.436Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-29T19:44:02.436Z+ EXTRA_IP_BASE=10.151.1
2152023-11-29T19:44:02.436Z+ EXTRA_IP_FOCTET=164
2162023-11-29T19:44:02.436Z+ EXTRA_IP_LOCTET=227
2172023-11-29T19:44:02.436Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-11-29T19:44:02.436Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-11-29T19:44:02.436Z+ DNS_IP1=10.151.1.164
2202023-11-29T19:44:02.436Z+ DNS_IP2=10.151.1.165
2212023-11-29T19:44:02.436Z+ UPLINK_IP=10.151.1.174
2222023-11-29T19:44:02.436Z+ IPPOOL_START=10.151.1.175
2232023-11-29T19:44:02.436Z+ IPPOOL_END=10.151.1.227
2242023-11-29T19:44:02.436Z++ ipadm show-addr -po type,addr
2252023-11-29T19:44:02.436Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-29T19:44:02.436Z+ GATEWAY_IP=10.151.1.100
2272023-11-29T19:44:02.436Z+ [[ -n 10.151.1.100 ]]
2282023-11-29T19:44:02.436Z+ ping -s 10.151.1.100 56 1
2292023-11-29T19:44:02.436ZPING 10.151.1.100: 56 data bytes
2302023-11-29T19:44:02.436Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.010 ms
2312023-11-29T19:44:02.436Z
2322023-11-29T19:44:02.436Z----10.151.1.100 PING Statistics----
2332023-11-29T19:44:02.436Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-29T19:44:02.436Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2352023-11-29T19:44:02.437Z++ arp -an
2362023-11-29T19:44:02.437Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-11-29T19:44:02.462Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-11-29T19:44:02.462Z+ routeadm -e ipv4-forwarding -u
2392023-11-29T19:44:02.547Z+ PXA_START=10.151.1.164
2402023-11-29T19:44:02.547Z+ PXA_END=10.151.1.227
2412023-11-29T19:44:02.572Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-29T19:44:02.572Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-29T19:44:02.682Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-29T19:44:02.682Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-29T19:44:02.706Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-29T19:44:02.707Z++ cd ./tools
2472023-11-29T19:44:02.707Z++ pwd
2482023-11-29T19:44:02.707Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-29T19:44:02.707Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-29T19:44:02.707Z+ SOFTNPU_MODE=zone
2512023-11-29T19:44:02.707Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-29T19:44:02.707Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-29T19:44:02.707Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-29T19:44:02.707Z++ dladm show-phys -p -o LINK
2552023-11-29T19:44:02.707Z++ head -1
2562023-11-29T19:44:02.765Z+ PHYSICAL_LINK=igb0
2572023-11-29T19:44:02.789Z+ echo 'Using igb0 as physical link'
2582023-11-29T19:44:02.789ZUsing igb0 as physical link
2592023-11-29T19:44:02.789Z+ ensure_run_as_root
2602023-11-29T19:44:02.790Z++ id -u
2612023-11-29T19:44:02.790Z+ [[ 0 -ne 0 ]]
2622023-11-29T19:44:02.790Z+ ensure_zpools
2632023-11-29T19:44:02.790Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-29T19:44:02.790Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-11-29T19:44:02.790Z+ readarray -t ZPOOLS
2662023-11-29T19:44:02.790Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-11-29T19:44:02.791Z++ sed 's/[ ",]//g'
2682023-11-29T19:44:02.791Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-11-29T19:44:02.791Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-11-29T19:44:02.791ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-11-29T19:44:02.791Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-11-29T19:44:02.791Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-11-29T19:44:02.791Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-11-29T19:44:02.822Z0+0 records in
2752023-11-29T19:44:02.822Z0+0 records out
2762023-11-29T19:44:02.822Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2772023-11-29T19:44:02.822Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-11-29T19:44:02.822Z+ set +x
2792023-11-29T19:44:02.822ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-11-29T19:44:02.822Z++ zpool list -o name
2812023-11-29T19:44:02.822Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-11-29T19:44:02.822Z+ [[ -z '' ]]
2832023-11-29T19:44:02.822Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-11-29T19:44:02.855Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-11-29T19:44:02.880Z+ set +x
2862023-11-29T19:44:02.881ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-11-29T19:44:02.881Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-11-29T19:44:02.881Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-11-29T19:44:02.881ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-11-29T19:44:02.881Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-11-29T19:44:02.881Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-11-29T19:44:02.882Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-11-29T19:44:02.882Z0+0 records in
2942023-11-29T19:44:02.882Z0+0 records out
2952023-11-29T19:44:02.882Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-11-29T19:44:02.882Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-11-29T19:44:02.882Z+ set +x
2982023-11-29T19:44:02.882ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-11-29T19:44:02.882Z++ zpool list -o name
3002023-11-29T19:44:02.882Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-11-29T19:44:02.882Z+ [[ -z '' ]]
3022023-11-29T19:44:02.882Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-11-29T19:44:02.908Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-11-29T19:44:02.908Z+ set +x
3052023-11-29T19:44:02.908ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-11-29T19:44:02.908Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-11-29T19:44:02.908Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-11-29T19:44:02.908ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3092023-11-29T19:44:02.908Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-11-29T19:44:02.908Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-11-29T19:44:02.908Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-11-29T19:44:02.908Z0+0 records in
3132023-11-29T19:44:02.908Z0+0 records out
3142023-11-29T19:44:02.908Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3152023-11-29T19:44:02.908Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-11-29T19:44:02.909Z+ set +x
3172023-11-29T19:44:02.909ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-11-29T19:44:02.909Z++ zpool list -o name
3192023-11-29T19:44:02.909Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-11-29T19:44:02.934Z+ [[ -z '' ]]
3212023-11-29T19:44:02.934Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-11-29T19:44:02.958Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-11-29T19:44:02.958Z+ set +x
3242023-11-29T19:44:02.958ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-11-29T19:44:02.958Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-11-29T19:44:02.958Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-11-29T19:44:02.958ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-11-29T19:44:02.958Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-11-29T19:44:02.958Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-11-29T19:44:02.958Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-11-29T19:44:02.958Z0+0 records in
3322023-11-29T19:44:02.958Z0+0 records out
3332023-11-29T19:44:02.958Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-11-29T19:44:02.958Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-11-29T19:44:02.958Z+ set +x
3362023-11-29T19:44:02.958ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-11-29T19:44:02.958Z++ zpool list -o name
3382023-11-29T19:44:02.958Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-11-29T19:44:02.960Z+ [[ -z '' ]]
3402023-11-29T19:44:02.960Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-11-29T19:44:02.986Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-11-29T19:44:02.986Z+ set +x
3432023-11-29T19:44:02.986ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-11-29T19:44:02.986Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-11-29T19:44:02.986Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-11-29T19:44:02.986ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-11-29T19:44:02.986Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-11-29T19:44:02.986Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-11-29T19:44:02.986Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-11-29T19:44:03.011Z0+0 records in
3512023-11-29T19:44:03.011Z0+0 records out
3522023-11-29T19:44:03.011Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3532023-11-29T19:44:03.011Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-11-29T19:44:03.011Z+ set +x
3552023-11-29T19:44:03.011ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-11-29T19:44:03.011Z++ zpool list -o name
3572023-11-29T19:44:03.011Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-11-29T19:44:03.011Z+ [[ -z '' ]]
3592023-11-29T19:44:03.011Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-11-29T19:44:03.035Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-11-29T19:44:03.035Z+ set +x
3622023-11-29T19:44:03.035ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-11-29T19:44:03.035Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-11-29T19:44:03.035Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-11-29T19:44:03.035ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-11-29T19:44:03.035Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-11-29T19:44:03.035Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-11-29T19:44:03.035Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-11-29T19:44:03.035Z0+0 records in
3702023-11-29T19:44:03.035Z0+0 records out
3712023-11-29T19:44:03.035Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3722023-11-29T19:44:03.035Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-11-29T19:44:03.035Z+ set +x
3742023-11-29T19:44:03.035ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-11-29T19:44:03.035Z++ zpool list -o name
3762023-11-29T19:44:03.035Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-11-29T19:44:03.062Z+ [[ -z '' ]]
3782023-11-29T19:44:03.062Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-11-29T19:44:03.087Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-11-29T19:44:03.087Z+ set +x
3812023-11-29T19:44:03.087ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-11-29T19:44:03.087Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-11-29T19:44:03.087Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-11-29T19:44:03.087ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-11-29T19:44:03.087Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-11-29T19:44:03.087Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-11-29T19:44:03.087Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-11-29T19:44:03.087Z0+0 records in
3892023-11-29T19:44:03.087Z0+0 records out
3902023-11-29T19:44:03.087Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-11-29T19:44:03.087Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-11-29T19:44:03.087Z+ set +x
3932023-11-29T19:44:03.087ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-11-29T19:44:03.087Z++ zpool list -o name
3952023-11-29T19:44:03.087Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-11-29T19:44:03.087Z+ [[ -z '' ]]
3972023-11-29T19:44:03.087Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-11-29T19:44:03.118Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-11-29T19:44:03.118Z+ set +x
4002023-11-29T19:44:03.143ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-11-29T19:44:03.143Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-29T19:44:03.143Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-11-29T19:44:03.143ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-11-29T19:44:03.143Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-11-29T19:44:03.143Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-11-29T19:44:03.143Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-11-29T19:44:03.143Z0+0 records in
4082023-11-29T19:44:03.143Z0+0 records out
4092023-11-29T19:44:03.143Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-11-29T19:44:03.143Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-11-29T19:44:03.143Z+ set +x
4122023-11-29T19:44:03.143ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-11-29T19:44:03.143Z++ zpool list -o name
4142023-11-29T19:44:03.143Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-11-29T19:44:03.143Z+ [[ -z '' ]]
4162023-11-29T19:44:03.143Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-11-29T19:44:03.168Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-11-29T19:44:03.168Z+ set +x
4192023-11-29T19:44:03.168ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-11-29T19:44:03.168Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-11-29T19:44:03.168Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-11-29T19:44:03.168ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-11-29T19:44:03.168Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-11-29T19:44:03.168Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-11-29T19:44:03.168Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-11-29T19:44:03.168Z0+0 records in
4272023-11-29T19:44:03.168Z0+0 records out
4282023-11-29T19:44:03.168Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4292023-11-29T19:44:03.168Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-11-29T19:44:03.168Z+ set +x
4312023-11-29T19:44:03.168ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-11-29T19:44:03.194Z++ zpool list -o name
4332023-11-29T19:44:03.195Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-11-29T19:44:03.195Z+ [[ -z '' ]]
4352023-11-29T19:44:03.195Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-11-29T19:44:03.220Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-11-29T19:44:03.220Z+ set +x
4382023-11-29T19:44:03.220ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-11-29T19:44:03.220Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-11-29T19:44:03.220Z+ readarray -t ZPOOLS
4412023-11-29T19:44:03.220Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-11-29T19:44:03.220Z++ sed 's/[ ",]//g'
4432023-11-29T19:44:03.220Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-11-29T19:44:03.220Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-11-29T19:44:03.220ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-11-29T19:44:03.220Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-11-29T19:44:03.220Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-11-29T19:44:03.220Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-11-29T19:44:03.220Z0+0 records in
4502023-11-29T19:44:03.220Z0+0 records out
4512023-11-29T19:44:03.220Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4522023-11-29T19:44:03.220Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-11-29T19:44:03.220Z+ set +x
4542023-11-29T19:44:03.220ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-11-29T19:44:03.220Z++ zpool list -o name
4562023-11-29T19:44:03.220Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-11-29T19:44:03.245Z+ [[ -z '' ]]
4582023-11-29T19:44:03.246Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-11-29T19:44:03.278Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-11-29T19:44:03.278Z+ set +x
4612023-11-29T19:44:03.278ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-11-29T19:44:03.278Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-11-29T19:44:03.278Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-11-29T19:44:03.278ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-11-29T19:44:03.278Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-11-29T19:44:03.278Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-11-29T19:44:03.278Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-11-29T19:44:03.278Z0+0 records in
4692023-11-29T19:44:03.278Z0+0 records out
4702023-11-29T19:44:03.278Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4712023-11-29T19:44:03.278Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-11-29T19:44:03.278Z+ set +x
4732023-11-29T19:44:03.278ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-11-29T19:44:03.278Z++ zpool list -o name
4752023-11-29T19:44:03.278Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-11-29T19:44:03.304Z+ [[ -z '' ]]
4772023-11-29T19:44:03.304Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-11-29T19:44:03.329Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-11-29T19:44:03.329Z+ set +x
4802023-11-29T19:44:03.329ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-11-29T19:44:03.329Z+ [[ zone == \z\o\n\e ]]
4822023-11-29T19:44:03.329Z+ ensure_simulated_links igb0
4832023-11-29T19:44:03.330Z+ local PHYSICAL_LINK=igb0
4842023-11-29T19:44:03.330Z+ INDICES=("0" "1")
4852023-11-29T19:44:03.330Z+ for I in "${INDICES[@]}"
4862023-11-29T19:44:03.330Z++ get_simnet_name_if_exists net0
4872023-11-29T19:44:03.330Z++ dladm show-simnet -p -o LINK net0
4882023-11-29T19:44:03.330Z++ echo ''
4892023-11-29T19:44:03.330Z+ [[ -z '' ]]
4902023-11-29T19:44:03.330Z+ dladm create-simnet -t net0
4912023-11-29T19:44:03.330Z+ dladm create-simnet -t sc0_0
4922023-11-29T19:44:03.330Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-11-29T19:44:03.330Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-11-29T19:44:03.357Z+ success 'Simnet net0/sc0_0 exists'
4952023-11-29T19:44:03.357Z+ set +x
4962023-11-29T19:44:03.357ZSimnet net0/sc0_0 exists
4972023-11-29T19:44:03.357Z+ for I in "${INDICES[@]}"
4982023-11-29T19:44:03.357Z++ get_simnet_name_if_exists net1
4992023-11-29T19:44:03.357Z++ dladm show-simnet -p -o LINK net1
5002023-11-29T19:44:03.357Z++ echo ''
5012023-11-29T19:44:03.357Z+ [[ -z '' ]]
5022023-11-29T19:44:03.357Z+ dladm create-simnet -t net1
5032023-11-29T19:44:03.357Z+ dladm create-simnet -t sc1_0
5042023-11-29T19:44:03.357Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-11-29T19:44:03.357Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-11-29T19:44:03.357Z+ success 'Simnet net1/sc1_0 exists'
5072023-11-29T19:44:03.357Z+ set +x
5082023-11-29T19:44:03.357ZSimnet net1/sc1_0 exists
5092023-11-29T19:44:03.357Z++ get_vnic_name_if_exists sc0_1
5102023-11-29T19:44:03.357Z++ dladm show-vnic -p -o LINK sc0_1
5112023-11-29T19:44:03.382Z++ echo ''
5122023-11-29T19:44:03.382Z+ [[ -z '' ]]
5132023-11-29T19:44:03.382Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-11-29T19:44:03.382Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-11-29T19:44:03.382Z+ success 'Vnic sc0_1 exists'
5162023-11-29T19:44:03.383Z+ set +x
5172023-11-29T19:44:03.383ZVnic sc0_1 exists
5182023-11-29T19:44:03.383Z+ warn_if_no_proxy_arp
5192023-11-29T19:44:03.383Z+ [[ -v PXA_START ]]
5202023-11-29T19:44:03.383Z+ [[ -v PXA_END ]]
5212023-11-29T19:44:03.383Z+ ensure_softnpu_zone
5222023-11-29T19:44:03.383Z+ zoneadm list
5232023-11-29T19:44:03.383Z+ grep -q sidecar_softnpu
5242023-11-29T19:44:03.383Z+ [[ -f out/npuzone/npuzone ]]
5252023-11-29T19:44:03.384Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5262023-11-29T19:44:03.409Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
5272023-11-29T19:44:08.550Zdownloading softnpu ...done
5282023-11-29T19:44:08.575Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5292023-11-29T19:44:14.475Zdownloading asic_program.so ...done
5302023-11-29T19:44:14.499Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-29T19:44:21.063Zdownloading scadm ...done
5322023-11-29T19:44:21.092Zcreating port tfportrear0_0 for sc0_0
5332023-11-29T19:44:21.092Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-29T19:44:21.092Zsoftnpu zone setup
5352023-11-29T19:44:21.092Zconfigure zone
5362023-11-29T19:44:21.116Zinstall zone
5372023-11-29T19:44:22.137Zboot zone
5382023-11-29T19:44:25.252Z[sidecar_softnpu] svcs milestone/network
5392023-11-29T19:44:25.279ZSTATE STIME FMRI
5402023-11-29T19:44:25.279Zonline 19:44:23 svc:/milestone/network:default
5412023-11-29T19:44:25.279Z[sidecar_softnpu] /softnpu/npu start
5422023-11-29T19:44:25.304Z
5432023-11-29T19:44:25.333Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-29T19:44:25.333Z+ GATEWAY_IP=10.151.1.100
5452023-11-29T19:44:25.333Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-11-29T19:44:25.333ZUsing 10.151.1.100 as gateway ip
5472023-11-29T19:44:25.333Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-29T19:44:25.333Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-11-29T19:44:25.333Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-29T19:44:25.335Z+++ echo 18:c0:4d:0d:9f:b2
5512023-11-29T19:44:25.335Z+ COUNT=1
5522023-11-29T19:44:25.335Z+ [[ 1 -ne 1 ]]
5532023-11-29T19:44:25.335Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-11-29T19:44:25.335ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-11-29T19:44:25.335Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-11-29T19:44:25.335Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-11-29T19:44:25.360Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-29T19:44:25.360Z+ [[ -v PXA_START ]]
5592023-11-29T19:44:25.360Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-11-29T19:44:25.360Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-11-29T19:44:25.384Z+ z_scadm dump-state
5622023-11-29T19:44:25.384Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-29T19:44:25.408Zlocal v6:
5642023-11-29T19:44:25.408Zlocal v4:
5652023-11-29T19:44:25.408Zrouter v6:
5662023-11-29T19:44:25.408Zrouter v4_idx:
5672023-11-29T19:44:25.416Zrouter v4_routes:
5682023-11-29T19:44:25.416Zresolver v4:
5692023-11-29T19:44:25.416Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-11-29T19:44:25.416Zresolver v6:
5712023-11-29T19:44:25.416Znat v4:
5722023-11-29T19:44:25.416Znat_v6:
5732023-11-29T19:44:25.416Zport_mac:
5742023-11-29T19:44:25.416Zproxy arp:
5752023-11-29T19:44:25.416Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-11-29T19:44:25.416Z+ success 'softnpu zone exists'
5772023-11-29T19:44:25.416Z+ set +x
5782023-11-29T19:44:25.416Zsoftnpu zone exists
5792023-11-29T19:44:25.416Z
5802023-11-29T19:44:25.416Zreal 22.722237230
5812023-11-29T19:44:25.416Zuser 0.712692694
5822023-11-29T19:44:25.416Zsys 1.620505765
5832023-11-29T19:44:25.416Ztrap 0.000072223
5842023-11-29T19:44:25.416Ztflt 0.000142684
5852023-11-29T19:44:25.416Zdflt 0.001073468
5862023-11-29T19:44:25.416Zkflt 0.000000000
5872023-11-29T19:44:25.416Zlock 17:31.064803116
5882023-11-29T19:44:25.416Zslp 1:07.939593388
5892023-11-29T19:44:25.416Zlat 0.088150445
5902023-11-29T19:44:25.416Zstop 0.339887249
5912023-11-29T19:44:25.440Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-29T19:44:25.465Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-29T19:44:25.473Z+ SILO_NAME=recovery
5942023-11-29T19:44:25.473Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-29T19:44:25.473Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-29T19:44:25.473Z+ sed -i~ '
5972023-11-29T19:44:25.473Z /^external_dns_ips/c\
5982023-11-29T19:44:25.473Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-11-29T19:44:25.473Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-29T19:44:25.473Z /^first/c\
6012023-11-29T19:44:25.473Zfirst = "10.151.1.164"
6022023-11-29T19:44:25.473Z /^last/c\
6032023-11-29T19:44:25.473Zlast = "10.151.1.173"
6042023-11-29T19:44:25.473Z }
6052023-11-29T19:44:25.473Z /^\[rack_network_config/,/^$/ {
6062023-11-29T19:44:25.473Z /^infra_ip_first/c\
6072023-11-29T19:44:25.473Zinfra_ip_first = "10.151.1.174"
6082023-11-29T19:44:25.473Z /^infra_ip_last/c\
6092023-11-29T19:44:25.473Zinfra_ip_last = "10.151.1.174"
6102023-11-29T19:44:25.473Z }
6112023-11-29T19:44:25.473Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-11-29T19:44:25.473Z /^routes/c\
6132023-11-29T19:44:25.473Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6142023-11-29T19:44:25.473Z /^addresses/c\
6152023-11-29T19:44:25.473Zaddresses = \["10.151.1.174/32"\]
6162023-11-29T19:44:25.473Z }
6172023-11-29T19:44:25.473Z' pkg/config-rss.toml
6182023-11-29T19:44:25.474Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-11-29T19:44:25.474Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-11-29T19:44:25.474Z+++ pkg/config-rss.toml Wed Nov 29 19:44:25 2023
6212023-11-29T19:44:25.475Z@@ -41,7 +41,7 @@
6222023-11-29T19:44:25.475Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-11-29T19:44:25.475Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-11-29T19:44:25.475Z # below.
6252023-11-29T19:44:25.475Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-11-29T19:44:25.475Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6272023-11-29T19:44:25.475Z
6282023-11-29T19:44:25.475Z # Initial TLS certificates for the external API
6292023-11-29T19:44:25.475Z #
6302023-11-29T19:44:25.475Z@@ -77,8 +77,8 @@
6312023-11-29T19:44:25.475Z #
6322023-11-29T19:44:25.475Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-11-29T19:44:25.475Z [[internal_services_ip_pool_ranges]]
6342023-11-29T19:44:25.475Z-first = "192.168.1.20"
6352023-11-29T19:44:25.475Z-last = "192.168.1.29"
6362023-11-29T19:44:25.475Z+first = "10.151.1.164"
6372023-11-29T19:44:25.475Z+last = "10.151.1.173"
6382023-11-29T19:44:25.475Z
6392023-11-29T19:44:25.475Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-11-29T19:44:25.475Z # parameters should be provided to the control plane via wicket, but we need to
6412023-11-29T19:44:25.475Z@@ -92,8 +92,8 @@
6422023-11-29T19:44:25.475Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-11-29T19:44:25.475Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-11-29T19:44:25.475Z # With softnpu, only one address is used.
6452023-11-29T19:44:25.475Z-infra_ip_first = "192.168.1.30"
6462023-11-29T19:44:25.475Z-infra_ip_last = "192.168.1.30"
6472023-11-29T19:44:25.475Z+infra_ip_first = "10.151.1.174"
6482023-11-29T19:44:25.477Z+infra_ip_last = "10.151.1.174"
6492023-11-29T19:44:25.477Z
6502023-11-29T19:44:25.477Z # Configurations for BGP routers to run on the scrimlets.
6512023-11-29T19:44:25.477Z bgp = []
6522023-11-29T19:44:25.477Z@@ -101,9 +101,9 @@
6532023-11-29T19:44:25.477Z # You can configure multiple uplinks by repeating the following stanza
6542023-11-29T19:44:25.477Z [[rack_network_config.ports]]
6552023-11-29T19:44:25.477Z # Routes associated with this port.
6562023-11-29T19:44:25.477Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-11-29T19:44:25.477Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6582023-11-29T19:44:25.477Z # Addresses associated with this port.
6592023-11-29T19:44:25.477Z-addresses = ["192.168.1.30/32"]
6602023-11-29T19:44:25.477Z+addresses = ["10.151.1.174/32"]
6612023-11-29T19:44:25.477Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-11-29T19:44:25.477Z port = "qsfp0"
6632023-11-29T19:44:25.477Z # The speed of this port.
6642023-11-29T19:44:25.478Z+ true
6652023-11-29T19:44:25.478Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-11-29T19:44:25.478Za pkg/config-rss.toml 7K
6672023-11-29T19:44:25.478Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-11-29T19:44:25.478Z+ TLS_NAME=recovery.sys.oxide.test
6692023-11-29T19:44:25.478Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-11-29T19:44:25.791Z.........+..+.............+.....+.+.........+.....+.+...+...+..+..........+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+....+............+...............+......+...+...........+.+...+.....+.........+.+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...................+..+...+......+...............+...+......+......+...+...+.......+...+.....+...+....+......+......+.....................+..+.............+........+......+.+..+.+.....+......+......+.......+......+..+.........+....+......+...+.....+...+..........+........+.....................+...............+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-11-29T19:44:26.215Z.....+.+.....+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+......+...........+.+..+.+.....+.+.....+......+...+..........+.................+............+...+...+.+......+..+.......+.....+..........+......+..+.+..+.......+..+.+.........+.........+.....+....+...+..+.+..+.......+..........................+..........+......+...........+...+.+.....+..........+...+...+........+.........+...+............+.+..+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...+...+.......+......+...............+......+...............+......+.....+...+.+...+..+...............+.+..+.+......+.....+.....................+...+....+........+....+..............+.+...........+.......+...+...+......+.....+....+.....+.......+.....+.........+.+.....+...+.........+.+.........+..+.............+.....+...+...................+.....+...+..........+...............+.....................+...............+............+..+................+.....+.+.........+..+....+..................+...........+.......+..+...+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-11-29T19:44:26.246Z-----
6732023-11-29T19:44:26.246Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-11-29T19:44:26.246Za pkg/initial-tls-cert.pem 2K
6752023-11-29T19:44:26.246Za pkg/initial-tls-key.pem 4K
6762023-11-29T19:44:26.246Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-11-29T19:44:26.247Z+ rmdir pkg
6782023-11-29T19:44:26.247Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-11-29T19:44:26.247Z+ pfexec mkdir -p /usr/oxide
6802023-11-29T19:44:26.247Z+ pfexec rm -f /usr/oxide/tcpproxy
6812023-11-29T19:44:26.247Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6822023-11-29T19:44:26.556Z+ pfexec chmod +x /usr/oxide/tcpproxy
6832023-11-29T19:44:26.585Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6842023-11-29T19:44:26.585Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6852023-11-29T19:44:26.585Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6862023-11-29T19:44:26.677Z+ OMICRON_NO_UNINSTALL=1
6872023-11-29T19:44:26.678Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6882023-11-29T19:44:26.702ZNov 29 19:44:26.712 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6892023-11-29T19:44:26.702ZNov 29 19:44:26.714 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6902023-11-29T19:44:26.702ZNov 29 19:44:26.714 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6912023-11-29T19:44:26.702ZNov 29 19:44:26.714 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6922023-11-29T19:44:26.702ZNov 29 19:44:26.714 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6932023-11-29T19:44:26.702ZNov 29 19:44:26.714 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6942023-11-29T19:44:26.702ZNov 29 19:44:26.714 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6952023-11-29T19:44:26.702ZNov 29 19:44:26.714 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6962023-11-29T19:44:26.702ZNov 29 19:44:26.714 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6972023-11-29T19:44:26.702ZNov 29 19:44:26.714 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6982023-11-29T19:44:26.702ZNov 29 19:44:26.714 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6992023-11-29T19:44:26.702ZNov 29 19:44:26.714 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7002023-11-29T19:44:26.706ZNov 29 19:44:26.714 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7012023-11-29T19:44:26.706ZNov 29 19:44:26.715 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7022023-11-29T19:44:26.706ZNov 29 19:44:26.715 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7032023-11-29T19:44:26.706ZNov 29 19:44:26.715 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7042023-11-29T19:44:26.706ZNov 29 19:44:26.715 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7052023-11-29T19:44:26.935ZNov 29 19:44:26.955 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-11-29T19:44:26.970ZNov 29 19:44:26.991 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-29T19:44:27.040ZNov 29 19:44:27.049 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-29T19:44:27.160Z
7092023-11-29T19:44:27.192Zreal 0.471421754
7102023-11-29T19:44:27.192Zuser 0.097732554
7112023-11-29T19:44:27.193Zsys 1.899511636
7122023-11-29T19:44:27.193Ztrap 0.000004738
7132023-11-29T19:44:27.193Ztflt 0.000038409
7142023-11-29T19:44:27.193Zdflt 0.000837923
7152023-11-29T19:44:27.193Zkflt 0.000015860
7162023-11-29T19:44:27.193Zlock 42.366503099
7172023-11-29T19:44:27.193Zslp 0.670618717
7182023-11-29T19:44:27.193Zlat 0.002654799
7192023-11-29T19:44:27.195Zstop 0.207465179
7202023-11-29T19:44:27.195Z+ retry=0
7212023-11-29T19:44:27.195Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-29T19:44:27.195Z+ [[ 0 -gt 30 ]]
7232023-11-29T19:44:27.195Z+ sleep 1
7242023-11-29T19:44:28.168Z+ retry=1
7252023-11-29T19:44:28.168Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-29T19:44:28.193Z+ [[ 1 -gt 30 ]]
7272023-11-29T19:44:28.193Z+ sleep 1
7282023-11-29T19:44:29.182Z+ retry=2
7292023-11-29T19:44:29.183Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-29T19:44:29.207Z+ [[ 2 -gt 30 ]]
7312023-11-29T19:44:29.207Z+ sleep 1
7322023-11-29T19:44:30.226Z+ retry=3
7332023-11-29T19:44:30.226Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-11-29T19:44:30.257Z+ [[ 3 -gt 30 ]]
7352023-11-29T19:44:30.257Z+ sleep 1
7362023-11-29T19:44:31.211Z+ retry=4
7372023-11-29T19:44:31.211Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-11-29T19:44:56.399Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7392023-11-29T19:44:56.429Zlocal v6:
7402023-11-29T19:44:56.429Zfe80::9ccf:7fff:febc:12d9
7412023-11-29T19:44:56.429Zfd00:99::1
7422023-11-29T19:44:56.429Zlocal v4:
7432023-11-29T19:44:56.429Z10.151.1.174
7442023-11-29T19:44:56.429Zrouter v6:
7452023-11-29T19:44:56.430Zfd00:1122:3344:101::/64 -> fe80::c87f:b3ff:fe2a:31d (1)
7462023-11-29T19:44:56.430Zfd00:1122:3344:3::/64 -> fe80::c87f:b3ff:fe2a:31d (1)
7472023-11-29T19:44:56.430Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::c87f:b3ff:fe2a:31d (1)
7482023-11-29T19:44:56.430Zfd00:1122:3344:2::/64 -> fe80::c87f:b3ff:fe2a:31d (1)
7492023-11-29T19:44:56.430Zfd00:1122:3344:1::/64 -> fe80::c87f:b3ff:fe2a:31d (1)
7502023-11-29T19:44:56.430Zrouter v4_idx:
7512023-11-29T19:44:56.430Z0.0.0.0/0 -> 0
7522023-11-29T19:44:56.430Zrouter v4_routes:
7532023-11-29T19:44:56.458Z0 -> 10.151.1.100 (2)
7542023-11-29T19:44:56.458Zresolver v4:
7552023-11-29T19:44:56.458Z10.151.1.174 -> d2:1c:0d:91:92:f4
7562023-11-29T19:44:56.458Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7572023-11-29T19:44:56.458Zresolver v6:
7582023-11-29T19:44:56.458Zfe80::9ccf:7fff:febc:12d9 -> 9e:cf:7f:bc:12:d9
7592023-11-29T19:44:56.458Zfe80::c87f:b3ff:fe2a:31d -> ca:7f:b3:2a:03:1d
7602023-11-29T19:44:56.458Znat v4:
7612023-11-29T19:44:56.458Z10.151.1.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:c4:90
7622023-11-29T19:44:56.458Znat_v6:
7632023-11-29T19:44:56.458Zport_mac:
7642023-11-29T19:44:56.458Z1000: a8:40:25:44:61:2b
7652023-11-29T19:44:56.458Z1: a8:40:25:44:61:2c
7662023-11-29T19:44:56.458Zproxy arp:
7672023-11-29T19:44:56.459Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7682023-11-29T19:44:56.459Z+ retry=0
7692023-11-29T19:44:56.459Z+++ zoneadm list -n
7702023-11-29T19:44:56.459Z+++ grep oxz_ntp
7712023-11-29T19:44:56.459Z++ pfexec svcs -z oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7 -Hostate oxide/ntp
7722023-11-29T19:44:56.490Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7732023-11-29T19:44:56.490Z++ true
7742023-11-29T19:44:56.490Z+ [[ '' != online ]]
7752023-11-29T19:44:56.490Z+ [[ 0 -gt 60 ]]
7762023-11-29T19:44:56.490Z+ sleep 1
7772023-11-29T19:44:57.464Z+ retry=1
7782023-11-29T19:44:57.488Z+++ zoneadm list -n
7792023-11-29T19:44:57.488Z+++ grep oxz_ntp
7802023-11-29T19:44:57.488Z++ pfexec svcs -z oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7 -Hostate oxide/ntp
7812023-11-29T19:44:57.513Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7822023-11-29T19:44:57.516Z++ true
7832023-11-29T19:44:57.516Z+ [[ '' != online ]]
7842023-11-29T19:44:57.516Z+ [[ 1 -gt 60 ]]
7852023-11-29T19:44:57.516Z+ sleep 1
7862023-11-29T19:44:58.495Z+ retry=2
7872023-11-29T19:44:58.519Z+++ zoneadm list -n
7882023-11-29T19:44:58.519Z+++ grep oxz_ntp
7892023-11-29T19:44:58.519Z++ pfexec svcs -z oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7 -Hostate oxide/ntp
7902023-11-29T19:44:58.544Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7912023-11-29T19:44:58.544Z++ true
7922023-11-29T19:44:58.544Z+ [[ '' != online ]]
7932023-11-29T19:44:58.544Z+ [[ 2 -gt 60 ]]
7942023-11-29T19:44:58.544Z+ sleep 1
7952023-11-29T19:44:59.526Z+ retry=3
7962023-11-29T19:44:59.552Z+++ zoneadm list -n
7972023-11-29T19:44:59.552Z+++ grep oxz_ntp
7982023-11-29T19:44:59.552Z++ pfexec svcs -z oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7 -Hostate oxide/ntp
7992023-11-29T19:44:59.552Z+ [[ disabled != online ]]
8002023-11-29T19:44:59.554Z+ [[ 3 -gt 60 ]]
8012023-11-29T19:44:59.555Z+ sleep 1
8022023-11-29T19:45:00.552Z+ retry=4
8032023-11-29T19:45:00.581Z+++ zoneadm list -n
8042023-11-29T19:45:00.583Z+++ grep oxz_ntp
8052023-11-29T19:45:00.583Z++ pfexec svcs -z oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7 -Hostate oxide/ntp
8062023-11-29T19:45:00.583Z+ [[ disabled != online ]]
8072023-11-29T19:45:00.584Z+ [[ 4 -gt 60 ]]
8082023-11-29T19:45:00.584Z+ sleep 1
8092023-11-29T19:45:01.573Z+ retry=5
8102023-11-29T19:45:01.598Z+++ zoneadm list -n
8112023-11-29T19:45:01.598Z+++ grep oxz_ntp
8122023-11-29T19:45:01.598Z++ pfexec svcs -z oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7 -Hostate oxide/ntp
8132023-11-29T19:45:01.598Z+ [[ disabled != online ]]
8142023-11-29T19:45:01.598Z+ [[ 5 -gt 60 ]]
8152023-11-29T19:45:01.598Z+ sleep 1
8162023-11-29T19:45:02.597Z+ retry=6
8172023-11-29T19:45:03.030Z+++ zoneadm list -n
8182023-11-29T19:45:03.030Z+++ grep oxz_ntp
8192023-11-29T19:45:03.030Z++ pfexec svcs -z oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7 -Hostate oxide/ntp
8202023-11-29T19:45:03.030Z+ [[ disabled != online ]]
8212023-11-29T19:45:03.030Z+ [[ 6 -gt 60 ]]
8222023-11-29T19:45:03.030Z+ sleep 1
8232023-11-29T19:45:03.617Z+ retry=7
8242023-11-29T19:45:03.643Z+++ zoneadm list -n
8252023-11-29T19:45:03.643Z+++ grep oxz_ntp
8262023-11-29T19:45:03.643Z++ pfexec svcs -z oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7 -Hostate oxide/ntp
8272023-11-29T19:45:03.643Z+ [[ disabled != online ]]
8282023-11-29T19:45:03.643Z+ [[ 7 -gt 60 ]]
8292023-11-29T19:45:03.643Z+ sleep 1
8302023-11-29T19:45:04.646Z+ retry=8
8312023-11-29T19:45:04.670Z+++ zoneadm list -n
8322023-11-29T19:45:04.671Z+++ grep oxz_ntp
8332023-11-29T19:45:04.671Z++ pfexec svcs -z oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7 -Hostate oxide/ntp
8342023-11-29T19:45:04.671Z+ [[ disabled != online ]]
8352023-11-29T19:45:04.671Z+ [[ 8 -gt 60 ]]
8362023-11-29T19:45:04.671Z+ sleep 1
 
8372023-11-29T19:45:05.495Zbuskin console login: Nov 29 19:45:04 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8382023-11-29T19:45:05.670Z+ retry=9
8392023-11-29T19:45:05.700Z+++ zoneadm list -n
8402023-11-29T19:45:05.700Z+++ grep oxz_ntp
8412023-11-29T19:45:05.700Z++ pfexec svcs -z oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7 -Hostate oxide/ntp
8422023-11-29T19:45:05.700Z+ [[ online != online ]]
8432023-11-29T19:45:05.700Z+ echo 'Waited for chrony: 9s'
8442023-11-29T19:45:05.700Z+ export RUST_BACKTRACE=1
8452023-11-29T19:45:05.700ZWaited for chrony: 9s
8462023-11-29T19:45:05.700Z+ RUST_BACKTRACE=1
8472023-11-29T19:45:05.700Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8482023-11-29T19:45:05.700Z+ ./tests/bootstrap
8492023-11-29T19:45:05.728ZUsing external DNS server at 10.151.1.164:53
8502023-11-29T19:45:05.732Z2023-11-29 19:45:05.731342046 UTC: attempting to log into API
8512023-11-29T19:45:20.768Z2023-11-29 19:45:20.741679137 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8522023-11-29T19:45:21.752Z2023-11-29 19:45:21.739714772 UTC: attempting to log into API
8532023-11-29T19:45:36.782Z2023-11-29 19:45:36.768743208 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8542023-11-29T19:45:37.782Z2023-11-29 19:45:37.769674727 UTC: attempting to log into API
8552023-11-29T19:45:52.826Z2023-11-29 19:45:52.801897419 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8562023-11-29T19:45:53.817Z2023-11-29 19:45:53.803730461 UTC: attempting to log into API
8572023-11-29T19:46:08.851Z2023-11-29 19:46:08.837218823 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8582023-11-29T19:46:09.851Z2023-11-29 19:46:09.838761830 UTC: attempting to log into API
8592023-11-29T19:46:24.885Z2023-11-29 19:46:24.872744443 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8602023-11-29T19:46:25.913Z2023-11-29 19:46:25.873313366 UTC: attempting to log into API
8612023-11-29T19:46:40.918Z2023-11-29 19:46:40.906204337 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8622023-11-29T19:46:41.920Z2023-11-29 19:46:41.907252517 UTC: attempting to log into API
8632023-11-29T19:46:56.957Z2023-11-29 19:46:56.940275381 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8642023-11-29T19:46:57.996Z2023-11-29 19:46:57.941183568 UTC: attempting to log into API
8652023-11-29T19:47:12.986Z2023-11-29 19:47:12.974856904 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8662023-11-29T19:47:13.987Z2023-11-29 19:47:13.975870673 UTC: attempting to log into API
8672023-11-29T19:47:29.027Z2023-11-29 19:47:29.015580696 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8682023-11-29T19:47:30.053Z2023-11-29 19:47:30.016256090 UTC: attempting to log into API
 
8692023-11-29T19:47:33.776ZNov 29 19:47:33 bukin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
8702023-11-29T19:47:41.874ZNov 29 19:47:41 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8712023-11-29T19:47:45.070Z2023-11-29 19:47:45.059282566 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8722023-11-29T19:47:46.071Z2023-11-29 19:47:46.060409017 UTC: attempting to log into API
 
8732023-11-29T19:47:48.724ZNov 29 19:47:48 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
8742023-11-29T19:47:55.731ZNov 29 19:47:55 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
8752023-11-29T19:48:01.112Z2023-11-29 19:48:01.094315757 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8762023-11-29T19:48:02.107Z2023-11-29 19:48:02.095114023 UTC: attempting to log into API
 
8772023-11-29T19:48:02.298ZNov 29 19:48:02 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
8782023-11-29T19:48:17.139Z2023-11-29 19:48:17.128359293 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8792023-11-29T19:48:18.139Z2023-11-29 19:48:18.129078659 UTC: attempting to log into API
8802023-11-29T19:48:23.922Z2023-11-29 19:48:23.911868704 UTC: login succeeded
8812023-11-29T19:48:23.954Zwaiting for nexus to come up...
8822023-11-29T19:48:24.019Zcreating IP pool... 10.151.1.175 - 10.151.1.227
8832023-11-29T19:48:24.095Zensuring datasets are ready...
8842023-11-29T19:48:27.219Zlet's roll.
8852023-11-29T19:48:27.243Z+ rm ./tests/bootstrap
8862023-11-29T19:48:27.289Z+ for test_bin in tests/*
8872023-11-29T19:48:27.289Z+ ./tests/bootstrap-fe3b4babdbd1397e
8882023-11-29T19:48:27.316Z
8892023-11-29T19:48:27.317Zrunning 0 tests
8902023-11-29T19:48:27.317Z
8912023-11-29T19:48:27.317Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
8922023-11-29T19:48:27.317Z
8932023-11-29T19:48:27.319Z+ for test_bin in tests/*
8942023-11-29T19:48:27.319Z+ ./tests/end_to_end_tests-29e4ffcd832e7ef8
8952023-11-29T19:48:27.319Z
8962023-11-29T19:48:27.319Zrunning 2 tests
8972023-11-29T19:48:27.530Ztest no_spoof::no_spoof_here ... ok
8982023-11-29T19:49:27.311Ztest instance_launch::instance_launch has been running for over 60 seconds
8992023-11-29T19:51:26.647Ztest instance_launch::instance_launch ... ok
9002023-11-29T19:51:26.647Z
9012023-11-29T19:51:26.647Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 179.33s
9022023-11-29T19:51:26.647Z
9032023-11-29T19:51:26.672Z+ _exit_trap
9042023-11-29T19:51:26.672Z+ local status=0
9052023-11-29T19:51:26.672Z+ [[ 0 -eq 0 ]]
9062023-11-29T19:51:26.672Z+ exit 0
9072023-11-29T19:51:26.672Zprocess exited: duration 451898 ms, exit code 0
 
9082023-11-29T19:51:26.782Zfound 51 output files
9092023-11-29T19:51:26.782Zuploading: /var/svc/log/oxide-sled-agent:default.log (683682 bytes)
9102023-11-29T19:51:28.068Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9112023-11-29T19:51:28.068Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_e4bb57d6-d66a-4f50-b763-f501d1c311d1/root/var/svc/log/oxide-clickhouse:default.log (675752 bytes)
9122023-11-29T19:51:29.060Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_e4bb57d6-d66a-4f50-b763-f501d1c311d1/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 675752 -> 685138
9132023-11-29T19:51:30.138Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_e4bb57d6-d66a-4f50-b763-f501d1c311d1/root/var/svc/log/oxide-clickhouse:default.log
9142023-11-29T19:51:30.138Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a0bdfaeb-92f3-4110-8e87-5912c9e7f4fe/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
9152023-11-29T19:51:31.234Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a0bdfaeb-92f3-4110-8e87-5912c9e7f4fe/root/var/svc/log/oxide-cockroachdb:default.log
9162023-11-29T19:51:31.234Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46caff01-b65b-4707-ac9b-41297fea38a8/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
9172023-11-29T19:51:32.368Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46caff01-b65b-4707-ac9b-41297fea38a8/root/var/svc/log/oxide-crucible-agent:default.log
9182023-11-29T19:51:32.369Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46caff01-b65b-4707-ac9b-41297fea38a8/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9192023-11-29T19:51:33.467Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46caff01-b65b-4707-ac9b-41297fea38a8/root/var/svc/log/oxide-crucible-downstairs:default.log
9202023-11-29T19:51:33.467Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46caff01-b65b-4707-ac9b-41297fea38a8/root/var/svc/log/oxide-crucible-downstairs:downstairs-293f1a83-dca8-465f-a8d6-dea7d2a52da2.log (5535 bytes)
9212023-11-29T19:51:34.564Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46caff01-b65b-4707-ac9b-41297fea38a8/root/var/svc/log/oxide-crucible-downstairs:downstairs-293f1a83-dca8-465f-a8d6-dea7d2a52da2.log
9222023-11-29T19:51:34.564Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7/root/var/svc/log/oxide-ntp:default.log (863 bytes)
9232023-11-29T19:51:35.659Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7/root/var/svc/log/oxide-ntp:default.log
9242023-11-29T19:51:35.659Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7/root/var/svc/log/oxide-ntp:default.log.0 (120 bytes)
9252023-11-29T19:51:36.764Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7/root/var/svc/log/oxide-ntp:default.log.0
9262023-11-29T19:51:36.764Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_660d6940-74e8-4f34-9f4b-3f17801fa1b0/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
9272023-11-29T19:51:37.868Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_660d6940-74e8-4f34-9f4b-3f17801fa1b0/root/var/svc/log/oxide-crucible-agent:default.log
9282023-11-29T19:51:37.868Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_660d6940-74e8-4f34-9f4b-3f17801fa1b0/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9292023-11-29T19:51:38.973Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_660d6940-74e8-4f34-9f4b-3f17801fa1b0/root/var/svc/log/oxide-crucible-downstairs:default.log
9302023-11-29T19:51:38.973Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e93e3f28-28f9-4f9f-b218-ebe5f55af032/root/var/svc/log/oxide-crucible-agent:default.log (13683 bytes)
9312023-11-29T19:51:40.092Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e93e3f28-28f9-4f9f-b218-ebe5f55af032/root/var/svc/log/oxide-crucible-agent:default.log
9322023-11-29T19:51:40.093Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e93e3f28-28f9-4f9f-b218-ebe5f55af032/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9332023-11-29T19:51:41.639Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e93e3f28-28f9-4f9f-b218-ebe5f55af032/root/var/svc/log/oxide-crucible-downstairs:default.log
9342023-11-29T19:51:41.639Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e93e3f28-28f9-4f9f-b218-ebe5f55af032/root/var/svc/log/oxide-crucible-downstairs:downstairs-3f0da2b3-22a5-4a87-8067-bc9fd3bfd7d1.log (5536 bytes)
9352023-11-29T19:51:42.745Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e93e3f28-28f9-4f9f-b218-ebe5f55af032/root/var/svc/log/oxide-crucible-downstairs:downstairs-3f0da2b3-22a5-4a87-8067-bc9fd3bfd7d1.log
9362023-11-29T19:51:42.745Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_2edebb4c-c95c-487f-bbdf-0458144b30de/root/var/svc/log/oxide-internal_dns:default.log (93564 bytes)
9372023-11-29T19:51:43.963Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_2edebb4c-c95c-487f-bbdf-0458144b30de/root/var/svc/log/oxide-internal_dns:default.log
9382023-11-29T19:51:43.963Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_2edebb4c-c95c-487f-bbdf-0458144b30de/root/var/svc/log/oxide-internal_dns:default.log.0 (5501 bytes)
9392023-11-29T19:51:45.479Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_2edebb4c-c95c-487f-bbdf-0458144b30de/root/var/svc/log/oxide-internal_dns:default.log.0
9402023-11-29T19:51:45.479Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_176d266b-ca56-4bfd-acdd-7da366b8e86d/root/var/svc/log/oxide-crucible-agent:default.log (13689 bytes)
9412023-11-29T19:51:46.609Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_176d266b-ca56-4bfd-acdd-7da366b8e86d/root/var/svc/log/oxide-crucible-agent:default.log
9422023-11-29T19:51:46.609Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_176d266b-ca56-4bfd-acdd-7da366b8e86d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9432023-11-29T19:51:47.704Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_176d266b-ca56-4bfd-acdd-7da366b8e86d/root/var/svc/log/oxide-crucible-downstairs:default.log
9442023-11-29T19:51:47.704Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_176d266b-ca56-4bfd-acdd-7da366b8e86d/root/var/svc/log/oxide-crucible-downstairs:downstairs-8c63cae5-f9e2-4c91-b217-d3fa7bb652e9.log (3049 bytes)
9452023-11-29T19:51:48.799Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_176d266b-ca56-4bfd-acdd-7da366b8e86d/root/var/svc/log/oxide-crucible-downstairs:downstairs-8c63cae5-f9e2-4c91-b217-d3fa7bb652e9.log
9462023-11-29T19:51:48.799Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_42311d62-8b79-4432-9c9b-b9c1b0019fad/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9472023-11-29T19:51:51.127Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_42311d62-8b79-4432-9c9b-b9c1b0019fad/root/var/svc/log/oxide-crucible-agent:default.log
9482023-11-29T19:51:51.154Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_42311d62-8b79-4432-9c9b-b9c1b0019fad/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9492023-11-29T19:51:52.228Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_42311d62-8b79-4432-9c9b-b9c1b0019fad/root/var/svc/log/oxide-crucible-downstairs:default.log
9502023-11-29T19:51:52.252Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7080bd69-b7db-46e5-9f98-8411dcaa80da/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
9512023-11-29T19:51:53.356Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7080bd69-b7db-46e5-9f98-8411dcaa80da/root/var/svc/log/oxide-crucible-agent:default.log
9522023-11-29T19:51:53.356Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7080bd69-b7db-46e5-9f98-8411dcaa80da/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9532023-11-29T19:51:54.458Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7080bd69-b7db-46e5-9f98-8411dcaa80da/root/var/svc/log/oxide-crucible-downstairs:default.log
9542023-11-29T19:51:54.458Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7080bd69-b7db-46e5-9f98-8411dcaa80da/root/var/svc/log/oxide-crucible-downstairs:downstairs-c03527d4-d9c2-44aa-b603-778347aedc39.log (5536 bytes)
9552023-11-29T19:51:55.553Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7080bd69-b7db-46e5-9f98-8411dcaa80da/root/var/svc/log/oxide-crucible-downstairs:downstairs-c03527d4-d9c2-44aa-b603-778347aedc39.log
9562023-11-29T19:51:55.554Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_858614ea-636d-48a1-818d-6025cfe1a887/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
9572023-11-29T19:51:56.674Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_858614ea-636d-48a1-818d-6025cfe1a887/root/var/svc/log/oxide-crucible-agent:default.log
9582023-11-29T19:51:56.674Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_858614ea-636d-48a1-818d-6025cfe1a887/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9592023-11-29T19:51:57.771Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_858614ea-636d-48a1-818d-6025cfe1a887/root/var/svc/log/oxide-crucible-downstairs:default.log
9602023-11-29T19:51:57.771Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_858614ea-636d-48a1-818d-6025cfe1a887/root/var/svc/log/oxide-crucible-downstairs:downstairs-f3700382-42a3-400d-86d0-ba8bc5211958.log (3051 bytes)
9612023-11-29T19:51:58.925Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_858614ea-636d-48a1-818d-6025cfe1a887/root/var/svc/log/oxide-crucible-downstairs:downstairs-f3700382-42a3-400d-86d0-ba8bc5211958.log
9622023-11-29T19:51:58.925Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_d9e3c343-65de-4b9a-9975-b4a079d74bce/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9632023-11-29T19:52:00.023Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_d9e3c343-65de-4b9a-9975-b4a079d74bce/root/var/svc/log/oxide-crucible-pantry:default.log
9642023-11-29T19:52:00.023Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_e91c3f04-7027-41d5-8b4c-1e903a277b16/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
9652023-11-29T19:52:01.121Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_e91c3f04-7027-41d5-8b4c-1e903a277b16/root/var/svc/log/oxide-cockroachdb:default.log
9662023-11-29T19:52:01.121Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_f090fb66-4445-41a5-a250-9307e78d6dba/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9672023-11-29T19:52:02.238Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_f090fb66-4445-41a5-a250-9307e78d6dba/root/var/svc/log/oxide-crucible-pantry:default.log
9682023-11-29T19:52:02.239Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_55cdd891-867c-4ebb-885a-140b00b66dfd/root/var/svc/log/oxide-cockroachdb:default.log (5344 bytes)
9692023-11-29T19:52:03.337Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_55cdd891-867c-4ebb-885a-140b00b66dfd/root/var/svc/log/oxide-cockroachdb:default.log
9702023-11-29T19:52:03.337Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_12aba0c1-e6ce-418a-b8a5-5719f9f32ff4/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
9712023-11-29T19:52:04.565Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_12aba0c1-e6ce-418a-b8a5-5719f9f32ff4/root/var/svc/log/oxide-crucible-agent:default.log
9722023-11-29T19:52:04.565Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_12aba0c1-e6ce-418a-b8a5-5719f9f32ff4/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9732023-11-29T19:52:05.661Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_12aba0c1-e6ce-418a-b8a5-5719f9f32ff4/root/var/svc/log/oxide-crucible-downstairs:default.log
9742023-11-29T19:52:05.661Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_12aba0c1-e6ce-418a-b8a5-5719f9f32ff4/root/var/svc/log/oxide-crucible-downstairs:downstairs-ce25e2e0-1d2c-4632-bd05-04f897315b90.log (3051 bytes)
9752023-11-29T19:52:06.772Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_12aba0c1-e6ce-418a-b8a5-5719f9f32ff4/root/var/svc/log/oxide-crucible-downstairs:downstairs-ce25e2e0-1d2c-4632-bd05-04f897315b90.log
9762023-11-29T19:52:06.772Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_a7f3ab04-1a93-41b8-8aa3-9e8338cd591c/root/var/svc/log/oxide-external_dns:default.log (15614 bytes)
9772023-11-29T19:52:07.272Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_a7f3ab04-1a93-41b8-8aa3-9e8338cd591c/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15614 -> 19392
9782023-11-29T19:52:08.344Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_a7f3ab04-1a93-41b8-8aa3-9e8338cd591c/root/var/svc/log/oxide-external_dns:default.log
9792023-11-29T19:52:08.344Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_2c51021d-fc05-4c78-adc5-d231c4f2f4a0/root/var/svc/log/oxide-internal_dns:default.log (93552 bytes)
9802023-11-29T19:52:08.511Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_2c51021d-fc05-4c78-adc5-d231c4f2f4a0/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93552 -> 97308
9812023-11-29T19:52:09.583Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_2c51021d-fc05-4c78-adc5-d231c4f2f4a0/root/var/svc/log/oxide-internal_dns:default.log
9822023-11-29T19:52:09.616Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_2c51021d-fc05-4c78-adc5-d231c4f2f4a0/root/var/svc/log/oxide-internal_dns:default.log.0 (10377 bytes)
9832023-11-29T19:52:10.704Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_2c51021d-fc05-4c78-adc5-d231c4f2f4a0/root/var/svc/log/oxide-internal_dns:default.log.0
9842023-11-29T19:52:10.728Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_9afec5f8-be64-4b6b-b777-93024d4e25cb/root/var/svc/log/oxide-internal_dns:default.log (93562 bytes)
9852023-11-29T19:52:10.865Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_9afec5f8-be64-4b6b-b777-93024d4e25cb/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93562 -> 97316
9862023-11-29T19:52:11.937Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_9afec5f8-be64-4b6b-b777-93024d4e25cb/root/var/svc/log/oxide-internal_dns:default.log
9872023-11-29T19:52:11.937Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_9afec5f8-be64-4b6b-b777-93024d4e25cb/root/var/svc/log/oxide-internal_dns:default.log.0 (7537 bytes)
9882023-11-29T19:52:13.066Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_9afec5f8-be64-4b6b-b777-93024d4e25cb/root/var/svc/log/oxide-internal_dns:default.log.0
9892023-11-29T19:52:13.066Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_339089df-b380-4a40-9ddb-7ea9d4968944/root/var/svc/log/oxide-nexus:default.log (4238430 bytes)
9902023-11-29T19:52:14.846Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_339089df-b380-4a40-9ddb-7ea9d4968944/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4238430 -> 4306741
9912023-11-29T19:52:16.334Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_339089df-b380-4a40-9ddb-7ea9d4968944/root/var/svc/log/oxide-nexus:default.log
9922023-11-29T19:52:16.334Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_e6cea7b3-81d6-4438-91ae-22ee8bf74eaf/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
9932023-11-29T19:52:17.443Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_e6cea7b3-81d6-4438-91ae-22ee8bf74eaf/root/var/svc/log/oxide-cockroachdb:default.log
9942023-11-29T19:52:17.468Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_ee117d5f-7fa5-4177-9824-027d91737959/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
9952023-11-29T19:52:18.540Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_ee117d5f-7fa5-4177-9824-027d91737959/root/var/svc/log/oxide-cockroachdb:default.log
9962023-11-29T19:52:18.540Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_d9b7c8a3-183d-4c2d-97aa-f181eb4c2665/root/var/svc/log/oxide-external_dns:default.log (15614 bytes)
9972023-11-29T19:52:18.635Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_d9b7c8a3-183d-4c2d-97aa-f181eb4c2665/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15614 -> 19391
9982023-11-29T19:52:19.706Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_d9b7c8a3-183d-4c2d-97aa-f181eb4c2665/root/var/svc/log/oxide-external_dns:default.log
9992023-11-29T19:52:19.706Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_3a0b16ab-28b7-4ce7-ad90-34ee1b2230f4/root/var/svc/log/oxide-nexus:default.log (824524 bytes)
10002023-11-29T19:52:20.563Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_3a0b16ab-28b7-4ce7-ad90-34ee1b2230f4/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 824524 -> 894627
10012023-11-29T19:52:21.634Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_3a0b16ab-28b7-4ce7-ad90-34ee1b2230f4/root/var/svc/log/oxide-nexus:default.log
10022023-11-29T19:52:21.634Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_8f0c7874-5567-430c-a9a3-d44119c2b303/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10032023-11-29T19:52:22.733Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_8f0c7874-5567-430c-a9a3-d44119c2b303/root/var/svc/log/oxide-crucible-pantry:default.log
10042023-11-29T19:52:22.757Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a2eb69db-92a6-4ab1-b46a-5fcd802b132d/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10052023-11-29T19:52:23.835Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a2eb69db-92a6-4ab1-b46a-5fcd802b132d/root/var/svc/log/oxide-crucible-agent:default.log
10062023-11-29T19:52:23.835Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a2eb69db-92a6-4ab1-b46a-5fcd802b132d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10072023-11-29T19:52:24.930Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a2eb69db-92a6-4ab1-b46a-5fcd802b132d/root/var/svc/log/oxide-crucible-downstairs:default.log
10082023-11-29T19:52:24.931Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_b69358f3-e97b-45ac-aef0-d39e0682737b/root/var/svc/log/oxide-nexus:default.log (672926 bytes)
10092023-11-29T19:52:25.230Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_b69358f3-e97b-45ac-aef0-d39e0682737b/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 672926 -> 741860
10102023-11-29T19:52:26.304Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_b69358f3-e97b-45ac-aef0-d39e0682737b/root/var/svc/log/oxide-nexus:default.log
10112023-11-29T19:52:26.304Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_219370f4-5a29-4e0f-9de5-ef313fbdf811/root/var/svc/log/oxide-oximeter:default.log (173205 bytes)
10122023-11-29T19:52:26.516Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_219370f4-5a29-4e0f-9de5-ef313fbdf811/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 173205 -> 220312
10132023-11-29T19:52:27.605Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_219370f4-5a29-4e0f-9de5-ef313fbdf811/root/var/svc/log/oxide-oximeter:default.log
10142023-11-29T19:52:27.605Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7/root/var/log/chrony/measurements.log (9042 bytes)
10152023-11-29T19:52:27.676Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7/root/var/log/chrony/measurements.log" changed size mid upload: 9042 -> 12193
10162023-11-29T19:52:28.759Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7/root/var/log/chrony/measurements.log
10172023-11-29T19:52:28.792Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7/root/var/log/chrony/statistics.log (5616 bytes)
10182023-11-29T19:52:28.832Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7/root/var/log/chrony/statistics.log" changed size mid upload: 5616 -> 7137
10192023-11-29T19:52:29.903Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7/root/var/log/chrony/statistics.log
10202023-11-29T19:52:29.903Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7/root/var/log/chrony/tracking.log (1455 bytes)
10212023-11-29T19:52:29.951Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7/root/var/log/chrony/tracking.log" changed size mid upload: 1455 -> 1587
10222023-11-29T19:52:31.023Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e073551a-4275-4511-b64b-5bb5c525bfd7/root/var/log/chrony/tracking.log