01HH5056MAXPC5DQVFS54N7K5Q: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HH505SN3EWQGY7BX9VZ2QV8G

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-08T16:04:15.853Zjob dependencies complete; ready to run (waiting for 45 m 2 s)
22023-12-08T16:05:07.305Zjob assigned to worker 01HH52RC3QEEGE1AG8MBZMV7BA (queued for 51 s)
32023-12-08T16:05:07.424Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-12-08T16:05:07.470Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-12-08T16:05:07.503Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-12-08T16:05:07.537Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-08T16:05:07.570Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-12-08T16:05:07.604ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-08T16:05:07.637Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-08T16:05:07.670ZOxide Helios Version helios-2.0.22318 64-bit
112023-12-08T16:05:07.704ZConfiguring devices.
122023-12-08T16:05:07.737Z##################################################
132023-12-08T16:05:07.771Z#### ############# ##
142023-12-08T16:05:07.804Z### ### ############ ##
152023-12-08T16:05:07.847Z## ### # ## ### ## Oxide Computer Company ##
162023-12-08T16:05:07.882Z## ## ## ### # ### ##
172023-12-08T16:05:07.916Z## # ### #### #### This Station Under ##
182023-12-08T16:05:07.949Z### ### #### # ### Computer Control ##
192023-12-08T16:05:07.982Z#### #### ### ## ##
202023-12-08T16:05:08.017Z##################################################
212023-12-08T16:05:08.049ZHostname: sock
222023-12-08T16:05:08.083ZLoading smf(7) service descriptions:
232023-12-08T16:05:08.457Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-2462efdf6b2e3290.gz
242023-12-08T16:05:09.751Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-2462efdf6b2e3290.gz
252023-12-08T16:05:09.751Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-08T16:05:27.032Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-08T16:05:27.058Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7ac35544d5ae1b92.gz
282023-12-08T16:05:45.239Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7ac35544d5ae1b92.gz
292023-12-08T16:05:45.239Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-08T16:05:47.582Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-08T16:05:47.582Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-08T16:05:50.488Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-08T16:05:50.516Zdownloading input: /input/package/work/version.txt
342023-12-08T16:05:50.546Zdownloaded input: /input/package/work/version.txt
352023-12-08T16:05:50.574Zdownloading input: /input/package/work/package.tar.gz
362023-12-08T16:06:05.660Zdownloaded input: /input/package/work/package.tar.gz
372023-12-08T16:06:05.695Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-08T16:06:28.074Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-08T16:06:28.074Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-08T16:06:36.020Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-08T16:06:36.045Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-08T16:07:13.429Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-08T16:07:13.429Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-08T16:07:47.817Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-08T16:07:47.818Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-08T16:08:08.712Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-08T16:08:08.737Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-08T16:08:12.749Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-08T16:08:12.750Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-08T16:08:20.150Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-08T16:08:20.150Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-08T16:08:24.423Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-08T16:08:24.423Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-08T16:08:28.712Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-08T16:08:28.737Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-08T16:08:28.762Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-08T16:08:28.762Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-08T16:08:32.442Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-08T16:08:32.442Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-08T16:08:35.820Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-08T16:08:35.844Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-08T16:08:48.535Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-08T16:08:48.540Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-08T16:09:01.300Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-08T16:09:01.300Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-08T16:09:01.399Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-08T16:09:01.425Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-08T16:09:07.396Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-08T16:09:07.396Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-08T16:09:18.981Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-08T16:09:18.981Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-08T16:10:01.416Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-08T16:10:01.416Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-08T16:10:30.830Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-08T16:10:30.855Zstarting task 0: "setup"
762023-12-08T16:10:30.855Z++ uname -s
772023-12-08T16:10:30.855Z+ kern=SunOS
782023-12-08T16:10:30.855Z+ case "$kern" in
792023-12-08T16:10:30.855Z+ groupadd -g 12345 build
802023-12-08T16:10:30.855Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-08T16:10:32.848Z+ zfs create -o mountpoint=/work rpool/work
822023-12-08T16:10:32.873Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-08T16:10:32.873Z+ home_fs=zfs
842023-12-08T16:10:32.873Z+ [[ zfs == autofs ]]
852023-12-08T16:10:32.873Z+ mkdir -p /home/build
862023-12-08T16:10:32.873Z+ chown build:build /home/build /work
872023-12-08T16:10:34.873Z+ chmod 0700 /home/build /work
882023-12-08T16:10:34.898Zprocess exited: duration 4046 ms, exit code 0
 
892023-12-08T16:10:34.947Zstarting task 1: "authentication"
902023-12-08T16:10:34.992Zprocess exited: duration 44 ms, exit code 0
 
912023-12-08T16:10:35.042Zstarting task 2: "build"
922023-12-08T16:10:35.066Z+ trap _exit_trap EXIT
932023-12-08T16:10:35.066Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-08T16:10:35.266Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-08T16:10:35.266Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-08T16:10:35.266Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-08T16:10:35.266Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-08T16:10:35.266Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-08T16:10:35.271Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-08T16:10:35.271Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-08T16:10:35.271Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-08T16:10:35.271Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-08T16:10:35.271Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-08T16:10:35.271Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-08T16:10:35.296Z+ [[ -d /opt/oxide ]]
1062023-12-08T16:10:35.296Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-08T16:10:35.296Z+ pfexec mkdir /opt/oxide
1082023-12-08T16:10:35.296Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-08T16:10:35.297Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-08T16:10:35.297Z+ cd /opt/oxide-underneath
1112023-12-08T16:10:35.297Z+ pfexec tar ceEp@/f - .
1122023-12-08T16:10:35.297Z+ cd /opt/oxide
1132023-12-08T16:10:35.297Z+ pfexec tar xveEp@/f -
1142023-12-08T16:10:35.369Zx ., 0 bytes, 0 tape blocks
1152023-12-08T16:10:35.369Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-08T16:10:35.369Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-08T16:10:35.369Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-12-08T16:10:35.394Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-08T16:10:35.394Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-08T16:10:35.418Z+ pfexec mkdir /opt/oxide/work
1212023-12-08T16:10:35.418Z+ pfexec chown build:build /opt/oxide/work
1222023-12-08T16:10:35.418Z+ cd /opt/oxide/work
1232023-12-08T16:10:35.418Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-08T16:10:35.418Ztar: blocksize = 10
1252023-12-08T16:10:36.966ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1262023-12-08T16:10:36.967Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1272023-12-08T16:10:36.967Zx out/omicron-sled-agent.tar, 111842816 bytes, 218443 tape blocks
1282023-12-08T16:10:36.967Zx out/target/test, 75 bytes, 1 tape blocks
1292023-12-08T16:10:36.967Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1302023-12-08T16:10:36.967Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1312023-12-08T16:10:36.967Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1322023-12-08T16:10:36.967Zx target/release/omicron-package, 42582768 bytes, 83170 tape blocks
1332023-12-08T16:10:36.967Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1342023-12-08T16:10:36.967Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1352023-12-08T16:10:36.967Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1362023-12-08T16:10:36.967Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1372023-12-08T16:10:36.967Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1382023-12-08T16:10:36.967Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1392023-12-08T16:10:36.967Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1402023-12-08T16:10:36.993Z
1412023-12-08T16:10:36.993Zreal 1.558332833
1422023-12-08T16:10:36.993Zuser 1.396998520
1432023-12-08T16:10:36.993Zsys 0.303397955
1442023-12-08T16:10:36.993Ztrap 0.000061781
1452023-12-08T16:10:36.993Ztflt 0.000000000
1462023-12-08T16:10:36.993Zdflt 0.000000000
1472023-12-08T16:10:36.993Zkflt 0.000000000
1482023-12-08T16:10:36.993Zlock 0.000000000
1492023-12-08T16:10:36.993Zslp 1.373248285
1502023-12-08T16:10:36.993Zlat 0.040990194
1512023-12-08T16:10:36.993Zstop 0.000117943
1522023-12-08T16:10:36.994Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1532023-12-08T16:10:38.041Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1542023-12-08T16:10:38.066Z+ mkdir tests
1552023-12-08T16:10:38.066Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1562023-12-08T16:10:38.066Z+ ptime -m gunzip
1572023-12-08T16:10:38.066Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-2462efdf6b2e3290
1582023-12-08T16:10:38.202Z
1592023-12-08T16:10:38.227Zreal 0.137515584
1602023-12-08T16:10:38.227Zuser 0.126830076
1612023-12-08T16:10:38.227Zsys 0.010444810
1622023-12-08T16:10:38.227Ztrap 0.000014210
1632023-12-08T16:10:38.227Ztflt 0.000000000
1642023-12-08T16:10:38.227Zdflt 0.000000000
1652023-12-08T16:10:38.227Zkflt 0.000000000
1662023-12-08T16:10:38.228Zlock 0.000000000
1672023-12-08T16:10:38.228Zslp 0.000020490
1682023-12-08T16:10:38.228Zlat 0.000017700
1692023-12-08T16:10:38.228Zstop 0.000121723
1702023-12-08T16:10:38.229Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-2462efdf6b2e3290
1712023-12-08T16:10:38.229Z+ chmod a+x tests/bootstrap-2462efdf6b2e3290
1722023-12-08T16:10:38.230Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1732023-12-08T16:10:38.230Z+ ptime -m gunzip
1742023-12-08T16:10:38.230Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1752023-12-08T16:10:40.487Z
1762023-12-08T16:10:40.513Zreal 2.274083000
1772023-12-08T16:10:40.513Zuser 2.118334025
1782023-12-08T16:10:40.513Zsys 0.155299723
1792023-12-08T16:10:40.513Ztrap 0.000189176
1802023-12-08T16:10:40.513Ztflt 0.000000000
1812023-12-08T16:10:40.513Zdflt 0.000000000
1822023-12-08T16:10:40.513Zkflt 0.000000000
1832023-12-08T16:10:40.513Zlock 0.000000000
1842023-12-08T16:10:40.513Zslp 0.000026360
1852023-12-08T16:10:40.513Zlat 0.000051111
1862023-12-08T16:10:40.513Zstop 0.000127173
1872023-12-08T16:10:40.513Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1882023-12-08T16:10:40.513Z+ chmod a+x tests/bootstrap
1892023-12-08T16:10:40.513Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1902023-12-08T16:10:40.514Z+ ptime -m gunzip
1912023-12-08T16:10:40.514Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7ac35544d5ae1b92
1922023-12-08T16:10:42.809Z
1932023-12-08T16:10:42.834Zreal 2.309024105
1942023-12-08T16:10:42.834Zuser 2.179189013
1952023-12-08T16:10:42.834Zsys 0.129435901
1962023-12-08T16:10:42.834Ztrap 0.000166864
1972023-12-08T16:10:42.835Ztflt 0.000000000
1982023-12-08T16:10:42.835Zdflt 0.000000000
1992023-12-08T16:10:42.835Zkflt 0.000000000
2002023-12-08T16:10:42.835Zlock 0.000000000
2012023-12-08T16:10:42.835Zslp 0.000024240
2022023-12-08T16:10:42.835Zlat 0.000022490
2032023-12-08T16:10:42.835Zstop 0.000123563
2042023-12-08T16:10:42.835Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7ac35544d5ae1b92
2052023-12-08T16:10:42.835Z+ chmod a+x tests/end_to_end_tests-7ac35544d5ae1b92
2062023-12-08T16:10:42.835Z+ bmat address ls
2072023-12-08T16:10:42.835ZNAME CIDR FIRST LAST COUNT
2082023-12-08T16:10:42.835Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2092023-12-08T16:10:42.835Z++ bmat address ls -f extra -Ho first,last,count
2102023-12-08T16:10:42.861Z+ set -- 10.151.2.164 10.151.2.227 64
2112023-12-08T16:10:42.861Z+ EXTRA_IP_START=10.151.2.164
2122023-12-08T16:10:42.861Z+ EXTRA_IP_END=10.151.2.227
2132023-12-08T16:10:42.861Z+ EXTRA_IP_COUNT=64
2142023-12-08T16:10:42.861Z+ (( EXTRA_IP_COUNT >= 32 ))
2152023-12-08T16:10:42.861Z+ EXTRA_IP_BASE=10.151.2
2162023-12-08T16:10:42.861Z+ EXTRA_IP_FOCTET=164
2172023-12-08T16:10:42.861Z+ EXTRA_IP_LOCTET=227
2182023-12-08T16:10:42.861Z+ SERVICE_IP_POOL_START=10.151.2.164
2192023-12-08T16:10:42.861Z+ SERVICE_IP_POOL_END=10.151.2.173
2202023-12-08T16:10:42.861Z+ DNS_IP1=10.151.2.164
2212023-12-08T16:10:42.861Z+ DNS_IP2=10.151.2.165
2222023-12-08T16:10:42.861Z+ UPLINK_IP=10.151.2.174
2232023-12-08T16:10:42.861Z+ IPPOOL_START=10.151.2.175
2242023-12-08T16:10:42.861Z+ IPPOOL_END=10.151.2.227
2252023-12-08T16:10:42.862Z++ ipadm show-addr -po type,addr
2262023-12-08T16:10:42.862Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2272023-12-08T16:10:42.862Z+ GATEWAY_IP=10.151.2.100
2282023-12-08T16:10:42.863Z+ [[ -n 10.151.2.100 ]]
2292023-12-08T16:10:42.863Z+ ping -s 10.151.2.100 56 1
2302023-12-08T16:10:42.863ZPING 10.151.2.100: 56 data bytes
2312023-12-08T16:10:42.863Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.010 ms
2322023-12-08T16:10:42.863Z
2332023-12-08T16:10:42.863Z----10.151.2.100 PING Statistics----
2342023-12-08T16:10:42.863Z1 packets transmitted, 1 packets received, 0% packet loss
2352023-12-08T16:10:42.863Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2362023-12-08T16:10:42.864Z++ arp -an
2372023-12-08T16:10:42.864Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2382023-12-08T16:10:42.888Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2392023-12-08T16:10:42.888Z+ routeadm -e ipv4-forwarding -u
2402023-12-08T16:10:42.975Z+ PXA_START=10.151.2.164
2412023-12-08T16:10:42.975Z+ PXA_END=10.151.2.227
2422023-12-08T16:10:43.000Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2432023-12-08T16:10:43.000Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2442023-12-08T16:10:43.106Z+ ZPOOL_VDEV_DIR=/scratch
2452023-12-08T16:10:43.106Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2462023-12-08T16:10:43.131Z+++ dirname ./tools/create_virtual_hardware.sh
2472023-12-08T16:10:43.131Z++ cd ./tools
2482023-12-08T16:10:43.131Z++ pwd
2492023-12-08T16:10:43.131Z+ SOURCE_DIR=/opt/oxide/work/tools
2502023-12-08T16:10:43.131Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2512023-12-08T16:10:43.131Z+ SOFTNPU_MODE=zone
2522023-12-08T16:10:43.131Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2532023-12-08T16:10:43.131Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2542023-12-08T16:10:43.131Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2552023-12-08T16:10:43.131Z++ dladm show-phys -p -o LINK
2562023-12-08T16:10:43.132Z++ head -1
2572023-12-08T16:10:43.190Z+ PHYSICAL_LINK=igb0
2582023-12-08T16:10:43.190Z+ echo 'Using igb0 as physical link'
2592023-12-08T16:10:43.214ZUsing igb0 as physical link
2602023-12-08T16:10:43.215Z+ ensure_run_as_root
2612023-12-08T16:10:43.215Z++ id -u
2622023-12-08T16:10:43.215Z+ [[ 0 -ne 0 ]]
2632023-12-08T16:10:43.215Z+ ensure_zpools
2642023-12-08T16:10:43.216Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2652023-12-08T16:10:43.216Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2662023-12-08T16:10:43.216Z+ readarray -t ZPOOLS
2672023-12-08T16:10:43.216Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2682023-12-08T16:10:43.216Z++ sed 's/[ ",]//g'
2692023-12-08T16:10:43.216Z+ for ZPOOL in "${ZPOOLS[@]}"
2702023-12-08T16:10:43.216Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2712023-12-08T16:10:43.216ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2722023-12-08T16:10:43.216Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2732023-12-08T16:10:43.216Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2742023-12-08T16:10:43.216Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2752023-12-08T16:10:43.241Z0+0 records in
2762023-12-08T16:10:43.241Z0+0 records out
2772023-12-08T16:10:43.241Z0 bytes (0 iB) transferred in 0.000007 secs (0 iB/sec)
2782023-12-08T16:10:43.241Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2792023-12-08T16:10:43.241Z+ set +x
2802023-12-08T16:10:43.241ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2812023-12-08T16:10:43.241Z++ zpool list -o name
2822023-12-08T16:10:43.241Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2832023-12-08T16:10:43.242Z+ [[ -z '' ]]
2842023-12-08T16:10:43.242Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2852023-12-08T16:10:43.280Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2862023-12-08T16:10:43.305Z+ set +x
2872023-12-08T16:10:43.305ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2882023-12-08T16:10:43.305Z+ for ZPOOL in "${ZPOOLS[@]}"
2892023-12-08T16:10:43.305Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2902023-12-08T16:10:43.305ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2912023-12-08T16:10:43.305Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2922023-12-08T16:10:43.305Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2932023-12-08T16:10:43.305Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2942023-12-08T16:10:43.305Z0+0 records in
2952023-12-08T16:10:43.305Z0+0 records out
2962023-12-08T16:10:43.306Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2972023-12-08T16:10:43.306Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2982023-12-08T16:10:43.306Z+ set +x
2992023-12-08T16:10:43.306ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3002023-12-08T16:10:43.306Z++ zpool list -o name
3012023-12-08T16:10:43.306Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3022023-12-08T16:10:43.306Z+ [[ -z '' ]]
3032023-12-08T16:10:43.306Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3042023-12-08T16:10:43.331Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3052023-12-08T16:10:43.331Z+ set +x
3062023-12-08T16:10:43.331ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3072023-12-08T16:10:43.331Z+ for ZPOOL in "${ZPOOLS[@]}"
3082023-12-08T16:10:43.332Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3092023-12-08T16:10:43.332ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3102023-12-08T16:10:43.332Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3112023-12-08T16:10:43.332Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3122023-12-08T16:10:43.332Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3132023-12-08T16:10:43.332Z0+0 records in
3142023-12-08T16:10:43.332Z0+0 records out
3152023-12-08T16:10:43.332Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3162023-12-08T16:10:43.332Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3172023-12-08T16:10:43.332Z+ set +x
3182023-12-08T16:10:43.332ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3192023-12-08T16:10:43.332Z++ zpool list -o name
3202023-12-08T16:10:43.332Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3212023-12-08T16:10:43.357Z+ [[ -z '' ]]
3222023-12-08T16:10:43.357Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3232023-12-08T16:10:43.382Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3242023-12-08T16:10:43.382Z+ set +x
3252023-12-08T16:10:43.382ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3262023-12-08T16:10:43.382Z+ for ZPOOL in "${ZPOOLS[@]}"
3272023-12-08T16:10:43.382Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3282023-12-08T16:10:43.382ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3292023-12-08T16:10:43.382Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3302023-12-08T16:10:43.382Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3312023-12-08T16:10:43.382Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3322023-12-08T16:10:43.383Z0+0 records in
3332023-12-08T16:10:43.383Z0+0 records out
3342023-12-08T16:10:43.383Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3352023-12-08T16:10:43.384Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3362023-12-08T16:10:43.384Z+ set +x
3372023-12-08T16:10:43.384ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3382023-12-08T16:10:43.384Z++ zpool list -o name
3392023-12-08T16:10:43.384Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3402023-12-08T16:10:43.384Z+ [[ -z '' ]]
3412023-12-08T16:10:43.384Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3422023-12-08T16:10:43.413Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3432023-12-08T16:10:43.413Z+ set +x
3442023-12-08T16:10:43.413ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3452023-12-08T16:10:43.413Z+ for ZPOOL in "${ZPOOLS[@]}"
3462023-12-08T16:10:43.413Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3472023-12-08T16:10:43.413ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3482023-12-08T16:10:43.413Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3492023-12-08T16:10:43.413Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3502023-12-08T16:10:43.413Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3512023-12-08T16:10:43.413Z0+0 records in
3522023-12-08T16:10:43.413Z0+0 records out
3532023-12-08T16:10:43.413Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3542023-12-08T16:10:43.413Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3552023-12-08T16:10:43.414Z+ set +x
3562023-12-08T16:10:43.414ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3572023-12-08T16:10:43.439Z++ zpool list -o name
3582023-12-08T16:10:43.439Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3592023-12-08T16:10:43.439Z+ [[ -z '' ]]
3602023-12-08T16:10:43.439Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3612023-12-08T16:10:43.464Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3622023-12-08T16:10:43.464Z+ set +x
3632023-12-08T16:10:43.464ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3642023-12-08T16:10:43.464Z+ for ZPOOL in "${ZPOOLS[@]}"
3652023-12-08T16:10:43.464Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3662023-12-08T16:10:43.464ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3672023-12-08T16:10:43.464Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3682023-12-08T16:10:43.464Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3692023-12-08T16:10:43.464Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3702023-12-08T16:10:43.464Z0+0 records in
3712023-12-08T16:10:43.464Z0+0 records out
3722023-12-08T16:10:43.464Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3732023-12-08T16:10:43.464Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3742023-12-08T16:10:43.465Z+ set +x
3752023-12-08T16:10:43.465ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3762023-12-08T16:10:43.465Z++ zpool list -o name
3772023-12-08T16:10:43.465Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3782023-12-08T16:10:43.465Z+ [[ -z '' ]]
3792023-12-08T16:10:43.503Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3802023-12-08T16:10:43.528Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3812023-12-08T16:10:43.529Z+ set +x
3822023-12-08T16:10:43.529ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3832023-12-08T16:10:43.529Z+ for ZPOOL in "${ZPOOLS[@]}"
3842023-12-08T16:10:43.530Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3852023-12-08T16:10:43.530ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3862023-12-08T16:10:43.530Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3872023-12-08T16:10:43.530Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3882023-12-08T16:10:43.530Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3892023-12-08T16:10:43.530Z0+0 records in
3902023-12-08T16:10:43.530Z0+0 records out
3912023-12-08T16:10:43.530Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-12-08T16:10:43.530Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3932023-12-08T16:10:43.530Z+ set +x
3942023-12-08T16:10:43.530ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3952023-12-08T16:10:43.530Z++ zpool list -o name
3962023-12-08T16:10:43.530Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3972023-12-08T16:10:43.530Z+ [[ -z '' ]]
3982023-12-08T16:10:43.530Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3992023-12-08T16:10:43.555Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4002023-12-08T16:10:43.555Z+ set +x
4012023-12-08T16:10:43.555ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4022023-12-08T16:10:43.556Z+ for ZPOOL in "${ZPOOLS[@]}"
4032023-12-08T16:10:43.556Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4042023-12-08T16:10:43.556ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4052023-12-08T16:10:43.556Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4062023-12-08T16:10:43.556Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4072023-12-08T16:10:43.557Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4082023-12-08T16:10:43.557Z0+0 records in
4092023-12-08T16:10:43.557Z0+0 records out
4102023-12-08T16:10:43.557Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4112023-12-08T16:10:43.557Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4122023-12-08T16:10:43.557Z+ set +x
4132023-12-08T16:10:43.557ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4142023-12-08T16:10:43.557Z++ zpool list -o name
4152023-12-08T16:10:43.557Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4162023-12-08T16:10:43.557Z+ [[ -z '' ]]
4172023-12-08T16:10:43.557Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4182023-12-08T16:10:43.582Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4192023-12-08T16:10:43.582Z+ set +x
4202023-12-08T16:10:43.582ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4212023-12-08T16:10:43.582Z+ for ZPOOL in "${ZPOOLS[@]}"
4222023-12-08T16:10:43.583Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4232023-12-08T16:10:43.583ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4242023-12-08T16:10:43.583Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4252023-12-08T16:10:43.583Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4262023-12-08T16:10:43.583Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4272023-12-08T16:10:43.608Z0+0 records in
4282023-12-08T16:10:43.608Z0+0 records out
4292023-12-08T16:10:43.608Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4302023-12-08T16:10:43.608Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4312023-12-08T16:10:43.609Z+ set +x
4322023-12-08T16:10:43.609ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4332023-12-08T16:10:43.609Z++ zpool list -o name
4342023-12-08T16:10:43.609Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4352023-12-08T16:10:43.609Z+ [[ -z '' ]]
4362023-12-08T16:10:43.609Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4372023-12-08T16:10:43.634Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4382023-12-08T16:10:43.634Z+ set +x
4392023-12-08T16:10:43.634ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4402023-12-08T16:10:43.634Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4412023-12-08T16:10:43.634Z+ readarray -t ZPOOLS
4422023-12-08T16:10:43.635Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4432023-12-08T16:10:43.635Z++ sed 's/[ ",]//g'
4442023-12-08T16:10:43.635Z+ for ZPOOL in "${ZPOOLS[@]}"
4452023-12-08T16:10:43.635Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4462023-12-08T16:10:43.635ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4472023-12-08T16:10:43.635Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4482023-12-08T16:10:43.635Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4492023-12-08T16:10:43.635Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4502023-12-08T16:10:43.636Z0+0 records in
4512023-12-08T16:10:43.636Z0+0 records out
4522023-12-08T16:10:43.636Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4532023-12-08T16:10:43.636Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4542023-12-08T16:10:43.636Z+ set +x
4552023-12-08T16:10:43.636ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4562023-12-08T16:10:43.661Z++ zpool list -o name
4572023-12-08T16:10:43.661Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4582023-12-08T16:10:43.661Z+ [[ -z '' ]]
4592023-12-08T16:10:43.661Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4602023-12-08T16:10:43.686Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4612023-12-08T16:10:43.686Z+ set +x
4622023-12-08T16:10:43.686ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4632023-12-08T16:10:43.686Z+ for ZPOOL in "${ZPOOLS[@]}"
4642023-12-08T16:10:43.687Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4652023-12-08T16:10:43.687ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4662023-12-08T16:10:43.687Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4672023-12-08T16:10:43.687Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4682023-12-08T16:10:43.687Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4692023-12-08T16:10:43.687Z0+0 records in
4702023-12-08T16:10:43.687Z0+0 records out
4712023-12-08T16:10:43.687Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4722023-12-08T16:10:43.687Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4732023-12-08T16:10:43.687Z+ set +x
4742023-12-08T16:10:43.687ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4752023-12-08T16:10:43.687Z++ zpool list -o name
4762023-12-08T16:10:43.687Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4772023-12-08T16:10:43.712Z+ [[ -z '' ]]
4782023-12-08T16:10:43.712Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4792023-12-08T16:10:43.737Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4802023-12-08T16:10:43.738Z+ set +x
4812023-12-08T16:10:43.738ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4822023-12-08T16:10:43.738Z+ [[ zone == \z\o\n\e ]]
4832023-12-08T16:10:43.738Z+ ensure_simulated_links igb0
4842023-12-08T16:10:43.738Z+ local PHYSICAL_LINK=igb0
4852023-12-08T16:10:43.739Z+ INDICES=("0" "1")
4862023-12-08T16:10:43.739Z+ for I in "${INDICES[@]}"
4872023-12-08T16:10:43.739Z++ get_simnet_name_if_exists net0
4882023-12-08T16:10:43.739Z++ dladm show-simnet -p -o LINK net0
4892023-12-08T16:10:43.739Z++ echo ''
4902023-12-08T16:10:43.739Z+ [[ -z '' ]]
4912023-12-08T16:10:43.739Z+ dladm create-simnet -t net0
4922023-12-08T16:10:43.739Z+ dladm create-simnet -t sc0_0
4932023-12-08T16:10:43.739Z+ dladm modify-simnet -t -p net0 sc0_0
4942023-12-08T16:10:43.739Z+ dladm set-linkprop -p mtu=9000 sc0_0
4952023-12-08T16:10:43.764Z+ success 'Simnet net0/sc0_0 exists'
4962023-12-08T16:10:43.764Z+ set +x
4972023-12-08T16:10:43.765ZSimnet net0/sc0_0 exists
4982023-12-08T16:10:43.765Z+ for I in "${INDICES[@]}"
4992023-12-08T16:10:43.765Z++ get_simnet_name_if_exists net1
5002023-12-08T16:10:43.765Z++ dladm show-simnet -p -o LINK net1
5012023-12-08T16:10:43.765Z++ echo ''
5022023-12-08T16:10:43.765Z+ [[ -z '' ]]
5032023-12-08T16:10:43.765Z+ dladm create-simnet -t net1
5042023-12-08T16:10:43.765Z+ dladm create-simnet -t sc1_0
5052023-12-08T16:10:43.765Z+ dladm modify-simnet -t -p net1 sc1_0
5062023-12-08T16:10:43.765Z+ dladm set-linkprop -p mtu=9000 sc1_0
5072023-12-08T16:10:43.765Z+ success 'Simnet net1/sc1_0 exists'
5082023-12-08T16:10:43.768Z+ set +x
5092023-12-08T16:10:43.768ZSimnet net1/sc1_0 exists
5102023-12-08T16:10:43.768Z++ get_vnic_name_if_exists sc0_1
5112023-12-08T16:10:43.768Z++ dladm show-vnic -p -o LINK sc0_1
5122023-12-08T16:10:43.793Z++ echo ''
5132023-12-08T16:10:43.793Z+ [[ -z '' ]]
5142023-12-08T16:10:43.793Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5152023-12-08T16:10:43.793Z+ [[ -v PROMISC_FILT_OFF ]]
5162023-12-08T16:10:43.793Z+ success 'Vnic sc0_1 exists'
5172023-12-08T16:10:43.793Z+ set +x
5182023-12-08T16:10:43.794ZVnic sc0_1 exists
5192023-12-08T16:10:43.794Z+ warn_if_no_proxy_arp
5202023-12-08T16:10:43.794Z+ [[ -v PXA_START ]]
5212023-12-08T16:10:43.794Z+ [[ -v PXA_END ]]
5222023-12-08T16:10:43.794Z+ ensure_softnpu_zone
5232023-12-08T16:10:43.794Z+ zoneadm list
5242023-12-08T16:10:43.794Z+ grep -q sidecar_softnpu
5252023-12-08T16:10:43.794Z+ [[ -f out/npuzone/npuzone ]]
5262023-12-08T16:10:43.794Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5272023-12-08T16:10:43.819Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
5282023-12-08T16:10:48.794Zdownloading softnpu ...done
5292023-12-08T16:10:48.820Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5302023-12-08T16:10:57.260Zdownloading asic_program.so ...done
5312023-12-08T16:10:57.285Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5322023-12-08T16:11:01.835Zdownloading scadm ...done
5332023-12-08T16:11:01.860Zcreating port tfportrear0_0 for sc0_0
5342023-12-08T16:11:01.861Zcreating port tfportqsfp0_0 for sc0_1
5352023-12-08T16:11:01.861Zsoftnpu zone setup
5362023-12-08T16:11:01.861Zconfigure zone
5372023-12-08T16:11:01.887Zinstall zone
5382023-12-08T16:11:02.925Zboot zone
5392023-12-08T16:11:06.041Z[sidecar_softnpu] svcs milestone/network
5402023-12-08T16:11:06.068ZSTATE STIME FMRI
5412023-12-08T16:11:06.068Zonline 16:11:03 svc:/milestone/network:default
5422023-12-08T16:11:06.070Z[sidecar_softnpu] /softnpu/npu start
5432023-12-08T16:11:06.096Z
5442023-12-08T16:11:06.121Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5452023-12-08T16:11:06.121Z+ GATEWAY_IP=10.151.2.100
5462023-12-08T16:11:06.121Z+ echo 'Using 10.151.2.100 as gateway ip'
5472023-12-08T16:11:06.121ZUsing 10.151.2.100 as gateway ip
5482023-12-08T16:11:06.122Z+ [[ ! -v GATEWAY_MAC ]]
5492023-12-08T16:11:06.122Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5502023-12-08T16:11:06.122Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5512023-12-08T16:11:06.122Z+++ echo 18:c0:4d:0d:a0:2a
5522023-12-08T16:11:06.122Z+ COUNT=1
5532023-12-08T16:11:06.122Z+ [[ 1 -ne 1 ]]
5542023-12-08T16:11:06.122Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5552023-12-08T16:11:06.122ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5562023-12-08T16:11:06.122Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-12-08T16:11:06.123Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5582023-12-08T16:11:06.148Z+ PXA_MAC=a8:e1:de:01:70:1d
5592023-12-08T16:11:06.149Z+ [[ -v PXA_START ]]
5602023-12-08T16:11:06.149Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-12-08T16:11:06.149Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5622023-12-08T16:11:06.174Z+ z_scadm dump-state
5632023-12-08T16:11:06.174Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5642023-12-08T16:11:06.199Zlocal v6:
5652023-12-08T16:11:06.200Zlocal v4:
5662023-12-08T16:11:06.200Zrouter v6:
5672023-12-08T16:11:06.200Zrouter v4_idx:
5682023-12-08T16:11:06.200Zrouter v4_routes:
5692023-12-08T16:11:06.200Zresolver v4:
5702023-12-08T16:11:06.200Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5712023-12-08T16:11:06.200Zresolver v6:
5722023-12-08T16:11:06.200Znat v4:
5732023-12-08T16:11:06.200Znat_v6:
5742023-12-08T16:11:06.200Zport_mac:
5752023-12-08T16:11:06.200Zproxy arp:
5762023-12-08T16:11:06.200Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5772023-12-08T16:11:06.200Z+ success 'softnpu zone exists'
5782023-12-08T16:11:06.200Z+ set +x
5792023-12-08T16:11:06.200Zsoftnpu zone exists
5802023-12-08T16:11:06.200Z
5812023-12-08T16:11:06.200Zreal 23.083699591
5822023-12-08T16:11:06.201Zuser 0.724263220
5832023-12-08T16:11:06.201Zsys 1.593199906
5842023-12-08T16:11:06.201Ztrap 0.000078238
5852023-12-08T16:11:06.201Ztflt 0.000100946
5862023-12-08T16:11:06.201Zdflt 0.000767501
5872023-12-08T16:11:06.201Zkflt 0.000000000
5882023-12-08T16:11:06.201Zlock 17:47.875102946
5892023-12-08T16:11:06.201Zslp 1:09.211985235
5902023-12-08T16:11:06.201Zlat 0.069790334
5912023-12-08T16:11:06.201Zstop 0.342119414
5922023-12-08T16:11:06.201Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5932023-12-08T16:11:06.227Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5942023-12-08T16:11:06.253Z+ SILO_NAME=recovery
5952023-12-08T16:11:06.253Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5962023-12-08T16:11:06.253Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5972023-12-08T16:11:06.253Z+ sed -i~ '
5982023-12-08T16:11:06.253Z /^external_dns_ips/c\
5992023-12-08T16:11:06.253Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6002023-12-08T16:11:06.253Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6012023-12-08T16:11:06.253Z /^first/c\
6022023-12-08T16:11:06.253Zfirst = "10.151.2.164"
6032023-12-08T16:11:06.253Z /^last/c\
6042023-12-08T16:11:06.253Zlast = "10.151.2.173"
6052023-12-08T16:11:06.253Z }
6062023-12-08T16:11:06.253Z /^\[rack_network_config/,/^$/ {
6072023-12-08T16:11:06.253Z /^infra_ip_first/c\
6082023-12-08T16:11:06.254Zinfra_ip_first = "10.151.2.174"
6092023-12-08T16:11:06.254Z /^infra_ip_last/c\
6102023-12-08T16:11:06.254Zinfra_ip_last = "10.151.2.174"
6112023-12-08T16:11:06.254Z }
6122023-12-08T16:11:06.254Z /^\[\[rack_network_config.ports/,/^$/ {
6132023-12-08T16:11:06.254Z /^routes/c\
6142023-12-08T16:11:06.254Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6152023-12-08T16:11:06.254Z /^addresses/c\
6162023-12-08T16:11:06.254Zaddresses = \["10.151.2.174/32"\]
6172023-12-08T16:11:06.254Z }
6182023-12-08T16:11:06.254Z' pkg/config-rss.toml
6192023-12-08T16:11:06.254Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6202023-12-08T16:11:06.254Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6212023-12-08T16:11:06.254Z+++ pkg/config-rss.toml Fri Dec 8 16:11:05 2023
6222023-12-08T16:11:06.254Z@@ -41,7 +41,7 @@
6232023-12-08T16:11:06.254Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6242023-12-08T16:11:06.255Z # must be contained in one of the "internal services" IP Pool ranges listed
6252023-12-08T16:11:06.255Z # below.
6262023-12-08T16:11:06.255Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6272023-12-08T16:11:06.255Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6282023-12-08T16:11:06.255Z
6292023-12-08T16:11:06.255Z # Initial TLS certificates for the external API
6302023-12-08T16:11:06.255Z #
6312023-12-08T16:11:06.255Z@@ -77,8 +77,8 @@
6322023-12-08T16:11:06.255Z #
6332023-12-08T16:11:06.255Z # For more on this and what to put here, see docs/how-to-run.adoc.
6342023-12-08T16:11:06.255Z [[internal_services_ip_pool_ranges]]
6352023-12-08T16:11:06.255Z-first = "192.168.1.20"
6362023-12-08T16:11:06.255Z-last = "192.168.1.29"
6372023-12-08T16:11:06.255Z+first = "10.151.2.164"
6382023-12-08T16:11:06.255Z+last = "10.151.2.173"
6392023-12-08T16:11:06.255Z
6402023-12-08T16:11:06.256Z # TODO - this configuration is subject to change going forward. Ultimately these
6412023-12-08T16:11:06.256Z # parameters should be provided to the control plane via wicket, but we need to
6422023-12-08T16:11:06.256Z@@ -92,8 +92,8 @@
6432023-12-08T16:11:06.256Z # A range of IP addresses used by Boundary Services on the external network. In
6442023-12-08T16:11:06.256Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6452023-12-08T16:11:06.256Z # With softnpu, only one address is used.
6462023-12-08T16:11:06.256Z-infra_ip_first = "192.168.1.30"
6472023-12-08T16:11:06.256Z-infra_ip_last = "192.168.1.30"
6482023-12-08T16:11:06.257Z+infra_ip_first = "10.151.2.174"
6492023-12-08T16:11:06.257Z+infra_ip_last = "10.151.2.174"
6502023-12-08T16:11:06.257Z
6512023-12-08T16:11:06.257Z # Configurations for BGP routers to run on the scrimlets.
6522023-12-08T16:11:06.257Z bgp = []
6532023-12-08T16:11:06.258Z@@ -101,9 +101,9 @@
6542023-12-08T16:11:06.258Z # You can configure multiple uplinks by repeating the following stanza
6552023-12-08T16:11:06.258Z [[rack_network_config.ports]]
6562023-12-08T16:11:06.258Z # Routes associated with this port.
6572023-12-08T16:11:06.258Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6582023-12-08T16:11:06.258Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6592023-12-08T16:11:06.258Z # Addresses associated with this port.
6602023-12-08T16:11:06.258Z-addresses = ["192.168.1.30/32"]
6612023-12-08T16:11:06.258Z+addresses = ["10.151.2.174/32"]
6622023-12-08T16:11:06.258Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6632023-12-08T16:11:06.258Z port = "qsfp0"
6642023-12-08T16:11:06.258Z # The speed of this port.
6652023-12-08T16:11:06.258Z+ true
6662023-12-08T16:11:06.258Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6672023-12-08T16:11:06.258Za pkg/config-rss.toml 7K
6682023-12-08T16:11:06.258Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6692023-12-08T16:11:06.258Z+ TLS_NAME=recovery.sys.oxide.test
6702023-12-08T16:11:06.258Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6712023-12-08T16:11:06.958Z...+.........+......+.+...+..+....+...........+.+.....+...+......+.+.........+......+........+......+.+............+..+....+..+.......+......+.....+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.........+...+..+...+..........+...+...+......+.....+.........+.......+...+......+...........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+......+.............+.........+...+.................+...+..........+.....+....+.....+....+............+..+.+............+...............+...+..+...+......+.+...+..............+......+....+........+...+................+......+.........+..................+......+.....+.......+...+..+.......+......+..+...+....+......+.........+..+..........+.........+..+.......+..+.+.....+...+.....................+.............+..................+...+.....+......+..........+...............+..................+...+............+...+..............+...+.+......+...+...+..+......+.+....................+..........+......+.........+......+.....+...+.........+..................+....+......+...+..+.......+.....+...............+...+............+............+...............+...+.....................+.+...+...........+.......+........+.+.....+....+.....+......+...............+......+...+.......+.........+........+......+.+.........+...........+...+............+..........+...+..+......+.......+........+...+.........+...+..................+.......+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-08T16:11:07.229Z...+...+...........+.+...........+...+....+..+...+.......+...+...+...............+..+.+......+...+............+...+.....+.+..+....+.....+...+.......+.....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+.+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+......+...........+....+...+..+...+.+............+...+.................+........................+.+...+...........+...+................+..+...+....+..+........................+.+........+..........+..+.............+..+.+..+.+..............+.............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6732023-12-08T16:11:07.254Z-----
6742023-12-08T16:11:07.254Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6752023-12-08T16:11:07.279Za pkg/initial-tls-cert.pem 2K
6762023-12-08T16:11:07.279Za pkg/initial-tls-key.pem 4K
6772023-12-08T16:11:07.280Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6782023-12-08T16:11:07.280Z+ rmdir pkg
6792023-12-08T16:11:07.280Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6802023-12-08T16:11:07.280Z+ pfexec mkdir -p /usr/oxide
6812023-12-08T16:11:07.280Z+ pfexec rm -f /usr/oxide/tcpproxy
6822023-12-08T16:11:07.280Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6832023-12-08T16:11:07.685Z+ pfexec chmod +x /usr/oxide/tcpproxy
6842023-12-08T16:11:07.710Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6852023-12-08T16:11:07.711Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6862023-12-08T16:11:07.739Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6872023-12-08T16:11:07.827Z+ OMICRON_NO_UNINSTALL=1
6882023-12-08T16:11:07.827Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6892023-12-08T16:11:07.853ZDec 08 16:11:07.396 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6902023-12-08T16:11:07.853ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6912023-12-08T16:11:07.853ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6922023-12-08T16:11:07.853ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6932023-12-08T16:11:07.853ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6942023-12-08T16:11:07.853ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6952023-12-08T16:11:07.853ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6962023-12-08T16:11:07.853ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6972023-12-08T16:11:07.853ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6982023-12-08T16:11:07.853ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6992023-12-08T16:11:07.853ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7002023-12-08T16:11:07.854ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7012023-12-08T16:11:07.854ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7022023-12-08T16:11:07.854ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7032023-12-08T16:11:07.854ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7042023-12-08T16:11:07.854ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7052023-12-08T16:11:07.854ZDec 08 16:11:07.398 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7062023-12-08T16:11:08.068ZDec 08 16:11:07.622 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7072023-12-08T16:11:08.101ZDec 08 16:11:07.656 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7082023-12-08T16:11:08.164ZDec 08 16:11:07.718 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7092023-12-08T16:11:08.291Z
7102023-12-08T16:11:08.316Zreal 0.460205842
7112023-12-08T16:11:08.317Zuser 0.099744576
7122023-12-08T16:11:08.317Zsys 1.832881172
7132023-12-08T16:11:08.317Ztrap 0.000009589
7142023-12-08T16:11:08.317Ztflt 0.000781567
7152023-12-08T16:11:08.317Zdflt 0.000384370
7162023-12-08T16:11:08.317Zkflt 0.000000000
7172023-12-08T16:11:08.317Zlock 41.274504541
7182023-12-08T16:11:08.317Zslp 0.671507420
7192023-12-08T16:11:08.317Zlat 0.003008735
7202023-12-08T16:11:08.317Zstop 0.201041023
7212023-12-08T16:11:08.317Z+ retry=0
7222023-12-08T16:11:08.317Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232023-12-08T16:11:08.317Z+ [[ 0 -gt 30 ]]
7242023-12-08T16:11:08.317Z+ sleep 1
7252023-12-08T16:11:09.307Z+ retry=1
7262023-12-08T16:11:09.307Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272023-12-08T16:11:09.332Z+ [[ 1 -gt 30 ]]
7282023-12-08T16:11:09.345Z+ sleep 1
7292023-12-08T16:11:10.321Z+ retry=2
7302023-12-08T16:11:10.322Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312023-12-08T16:11:10.347Z+ [[ 2 -gt 30 ]]
7322023-12-08T16:11:10.347Z+ sleep 1
7332023-12-08T16:11:11.336Z+ retry=3
7342023-12-08T16:11:11.336Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352023-12-08T16:11:11.361Z+ [[ 3 -gt 30 ]]
7362023-12-08T16:11:11.361Z+ sleep 1
7372023-12-08T16:11:12.355Z+ retry=4
7382023-12-08T16:11:12.355Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7392023-12-08T16:11:32.218Z+ [[ 4 -gt 30 ]]
7402023-12-08T16:11:32.218Z+ sleep 1
7412023-12-08T16:11:33.221Z+ retry=5
7422023-12-08T16:11:33.221Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7432023-12-08T16:11:33.247Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7442023-12-08T16:11:33.272Zlocal v6:
7452023-12-08T16:11:33.272Zfe80::f801:1bff:fe4b:72fe
7462023-12-08T16:11:33.272Zlocal v4:
7472023-12-08T16:11:33.272Zrouter v6:
7482023-12-08T16:11:33.272Zrouter v4_idx:
7492023-12-08T16:11:33.272Zrouter v4_routes:
7502023-12-08T16:11:33.272Zresolver v4:
7512023-12-08T16:11:33.272Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7522023-12-08T16:11:33.272Zresolver v6:
7532023-12-08T16:11:33.273Zfe80::707d:71ff:feee:f99 -> 72:7d:71:ee:0f:99
7542023-12-08T16:11:33.273Zfe80::f801:1bff:fe4b:72fe -> fa:01:1b:4b:72:fe
7552023-12-08T16:11:33.273Znat v4:
7562023-12-08T16:11:33.273Znat_v6:
7572023-12-08T16:11:33.273Zport_mac:
7582023-12-08T16:11:33.273Z1000: a8:40:25:1c:1f:ac
7592023-12-08T16:11:33.273Z1: a8:40:25:1c:1f:ad
7602023-12-08T16:11:33.273Zproxy arp:
7612023-12-08T16:11:33.273Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7622023-12-08T16:11:33.273Z+ retry=0
7632023-12-08T16:11:33.273Z+++ zoneadm list -n
7642023-12-08T16:11:33.298Z+++ grep oxz_ntp
7652023-12-08T16:11:33.298Z++ pfexec svcs -z -Hostate oxide/ntp
7662023-12-08T16:11:33.298Zsvcs: zone '-Hostate': entity not found
7672023-12-08T16:11:33.298Z++ true
7682023-12-08T16:11:33.298Z+ [[ '' != online ]]
7692023-12-08T16:11:33.300Z+ [[ 0 -gt 60 ]]
7702023-12-08T16:11:33.300Z+ sleep 1
7712023-12-08T16:11:34.287Z+ retry=1
7722023-12-08T16:11:34.312Z+++ zoneadm list -n
7732023-12-08T16:11:34.313Z+++ grep oxz_ntp
7742023-12-08T16:11:34.313Z++ pfexec svcs -z -Hostate oxide/ntp
7752023-12-08T16:11:34.313Zsvcs: zone '-Hostate': entity not found
7762023-12-08T16:11:34.313Z++ true
7772023-12-08T16:11:34.313Z+ [[ '' != online ]]
7782023-12-08T16:11:34.313Z+ [[ 1 -gt 60 ]]
7792023-12-08T16:11:34.313Z+ sleep 1
7802023-12-08T16:11:35.310Z+ retry=2
7812023-12-08T16:11:35.335Z+++ zoneadm list -n
7822023-12-08T16:11:35.335Z+++ grep oxz_ntp
7832023-12-08T16:11:35.335Z++ pfexec svcs -z oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe -Hostate oxide/ntp
7842023-12-08T16:11:35.369Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7852023-12-08T16:11:35.394Z++ true
7862023-12-08T16:11:35.395Z+ [[ '' != online ]]
7872023-12-08T16:11:35.395Z+ [[ 2 -gt 60 ]]
7882023-12-08T16:11:35.395Z+ sleep 1
7892023-12-08T16:11:36.375Z+ retry=3
7902023-12-08T16:11:36.400Z+++ zoneadm list -n
7912023-12-08T16:11:36.400Z+++ grep oxz_ntp
7922023-12-08T16:11:36.400Z++ pfexec svcs -z oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe -Hostate oxide/ntp
7932023-12-08T16:11:36.425Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7942023-12-08T16:11:36.425Z++ true
7952023-12-08T16:11:36.425Z+ [[ '' != online ]]
7962023-12-08T16:11:36.425Z+ [[ 3 -gt 60 ]]
7972023-12-08T16:11:36.426Z+ sleep 1
7982023-12-08T16:11:37.407Z+ retry=4
7992023-12-08T16:11:37.432Z+++ zoneadm list -n
8002023-12-08T16:11:37.432Z+++ grep oxz_ntp
8012023-12-08T16:11:37.432Z++ pfexec svcs -z oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe -Hostate oxide/ntp
8022023-12-08T16:11:37.432Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8032023-12-08T16:11:37.458Z++ true
8042023-12-08T16:11:37.458Z+ [[ '' != online ]]
8052023-12-08T16:11:37.458Z+ [[ 4 -gt 60 ]]
8062023-12-08T16:11:37.458Z+ sleep 1
8072023-12-08T16:11:38.436Z+ retry=5
8082023-12-08T16:11:38.461Z+++ zoneadm list -n
8092023-12-08T16:11:38.461Z+++ grep oxz_ntp
8102023-12-08T16:11:38.462Z++ pfexec svcs -z oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe -Hostate oxide/ntp
8112023-12-08T16:11:38.486Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8122023-12-08T16:11:38.486Z++ true
8132023-12-08T16:11:38.486Z+ [[ '' != online ]]
8142023-12-08T16:11:38.486Z+ [[ 5 -gt 60 ]]
8152023-12-08T16:11:38.487Z+ sleep 1
8162023-12-08T16:11:39.469Z+ retry=6
8172023-12-08T16:11:39.495Z+++ zoneadm list -n
8182023-12-08T16:11:39.495Z+++ grep oxz_ntp
8192023-12-08T16:11:39.495Z++ pfexec svcs -z oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe -Hostate oxide/ntp
8202023-12-08T16:11:39.495Z+ [[ disabled != online ]]
8212023-12-08T16:11:39.495Z+ [[ 6 -gt 60 ]]
8222023-12-08T16:11:39.495Z+ sleep 1
8232023-12-08T16:11:40.493Z+ retry=7
8242023-12-08T16:11:40.520Z+++ zoneadm list -n
8252023-12-08T16:11:40.520Z+++ grep oxz_ntp
8262023-12-08T16:11:40.520Z++ pfexec svcs -z oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe -Hostate oxide/ntp
8272023-12-08T16:11:40.520Z+ [[ disabled != online ]]
8282023-12-08T16:11:40.520Z+ [[ 7 -gt 60 ]]
8292023-12-08T16:11:40.520Z+ sleep 1
8302023-12-08T16:11:41.515Z+ retry=8
8312023-12-08T16:11:41.540Z+++ zoneadm list -n
8322023-12-08T16:11:41.540Z+++ grep oxz_ntp
8332023-12-08T16:11:41.540Z++ pfexec svcs -z oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe -Hostate oxide/ntp
8342023-12-08T16:11:41.540Z+ [[ disabled != online ]]
8352023-12-08T16:11:41.540Z+ [[ 8 -gt 60 ]]
8362023-12-08T16:11:41.540Z+ sleep 1
8372023-12-08T16:11:42.535Z+ retry=9
8382023-12-08T16:11:42.560Z+++ zoneadm list -n
8392023-12-08T16:11:42.560Z+++ grep oxz_ntp
8402023-12-08T16:11:42.560Z++ pfexec svcs -z oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe -Hostate oxide/ntp
8412023-12-08T16:11:42.561Z+ [[ disabled != online ]]
8422023-12-08T16:11:42.561Z+ [[ 9 -gt 60 ]]
8432023-12-08T16:11:42.561Z+ sleep 1
8442023-12-08T16:11:43.558Z+ retry=10
8452023-12-08T16:11:43.583Z+++ zoneadm list -n
8462023-12-08T16:11:43.583Z+++ grep oxz_ntp
8472023-12-08T16:11:43.583Z++ pfexec svcs -z oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe -Hostate oxide/ntp
8482023-12-08T16:11:43.583Z+ [[ disabled != online ]]
8492023-12-08T16:11:43.583Z+ [[ 10 -gt 60 ]]
8502023-12-08T16:11:43.583Z+ sleep 1
8512023-12-08T16:11:44.580Z+ retry=11
8522023-12-08T16:11:44.605Z+++ zoneadm list -n
8532023-12-08T16:11:44.605Z+++ grep oxz_ntp
8542023-12-08T16:11:44.605Z++ pfexec svcs -z oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe -Hostate oxide/ntp
8552023-12-08T16:11:44.605Z+ [[ disabled != online ]]
8562023-12-08T16:11:44.605Z+ [[ 11 -gt 60 ]]
8572023-12-08T16:11:44.605Z+ sleep 1
8582023-12-08T16:11:45.602Z+ retry=12
8592023-12-08T16:11:45.627Z+++ zoneadm list -n
8602023-12-08T16:11:45.627Z+++ grep oxz_ntp
8612023-12-08T16:11:45.627Z++ pfexec svcs -z oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe -Hostate oxide/ntp
8622023-12-08T16:11:45.627Z+ [[ disabled != online ]]
8632023-12-08T16:11:45.627Z+ [[ 12 -gt 60 ]]
8642023-12-08T16:11:45.628Z+ sleep 1
8652023-12-08T16:11:46.624Z+ retry=13
8662023-12-08T16:11:46.649Z+++ zoneadm list -n
8672023-12-08T16:11:46.649Z+++ grep oxz_ntp
8682023-12-08T16:11:46.649Z++ pfexec svcs -z oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe -Hostate oxide/ntp
8692023-12-08T16:11:46.649Z+ [[ disabled != online ]]
8702023-12-08T16:11:46.649Z+ [[ 13 -gt 60 ]]
8712023-12-08T16:11:46.649Z+ sleep 1
 
8722023-12-08T16:11:46.814Zsock console login: Dec 8 16:11:45 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8732023-12-08T16:11:47.647Z+ retry=14
8742023-12-08T16:11:47.674Z+++ zoneadm list -n
8752023-12-08T16:11:47.674Z+++ grep oxz_ntp
8762023-12-08T16:11:47.674Z++ pfexec svcs -z oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe -Hostate oxide/ntp
8772023-12-08T16:11:47.674Z+ [[ online != online ]]
8782023-12-08T16:11:47.675Z+ echo 'Waited for chrony: 14s'
8792023-12-08T16:11:47.676ZWaited for chrony: 14s
8802023-12-08T16:11:47.676Z+ export RUST_BACKTRACE=1
8812023-12-08T16:11:47.676Z+ RUST_BACKTRACE=1
8822023-12-08T16:11:47.676Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8832023-12-08T16:11:47.676Z+ ./tests/bootstrap
8842023-12-08T16:11:47.701ZUsing external DNS server at 10.151.2.164:53
8852023-12-08T16:11:47.701Z2023-12-08 16:11:47.252146756 UTC: attempting to log into API
8862023-12-08T16:12:02.735Z2023-12-08 16:12:03.166307644 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8872023-12-08T16:12:03.736Z2023-12-08 16:12:04.167335583 UTC: attempting to log into API
8882023-12-08T16:12:18.770Z2023-12-08 16:12:19.201679458 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8892023-12-08T16:12:19.771Z2023-12-08 16:12:19.448215615 UTC: attempting to log into API
8902023-12-08T16:12:34.814Z2023-12-08 16:12:34.528130411 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8912023-12-08T16:12:35.816Z2023-12-08 16:12:35.532632796 UTC: attempting to log into API
8922023-12-08T16:12:50.849Z2023-12-08 16:12:50.606073808 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8932023-12-08T16:12:51.850Z2023-12-08 16:12:51.609699748 UTC: attempting to log into API
 
8942023-12-08T16:12:55.530ZDec 8 16:12:54 sok ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
8952023-12-08T16:13:01.759ZDec 8 16:13:01 sok ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8962023-12-08T16:13:06.915Z2023-12-08 16:13:06.688207098 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
8972023-12-08T16:13:07.386ZDec 8 16:13:06 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
8982023-12-08T16:13:07.889Z2023-12-08 16:13:07.692839478 UTC: attempting to log into API
 
8992023-12-08T16:13:13.008ZDec 8 16:13:12 sok ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9002023-12-08T16:13:20.784ZDec 8 16:13:19 sok ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9012023-12-08T16:13:22.921Z2023-12-08 16:13:22.765887446 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9022023-12-08T16:13:23.922Z2023-12-08 16:13:23.769640348 UTC: attempting to log into API
9032023-12-08T16:13:38.955Z2023-12-08 16:13:38.843028791 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9042023-12-08T16:13:39.955Z2023-12-08 16:13:39.846658015 UTC: attempting to log into API
9052023-12-08T16:13:40.752Z2023-12-08 16:13:40.644974687 UTC: login succeeded
9062023-12-08T16:13:40.784Zwaiting for nexus to come up...
9072023-12-08T16:13:40.846Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9082023-12-08T16:13:40.904Zensuring datasets are ready...
9092023-12-08T16:13:43.411Zlet's roll.
9102023-12-08T16:13:43.436Z+ rm ./tests/bootstrap
9112023-12-08T16:13:43.494Z+ for test_bin in tests/*
9122023-12-08T16:13:43.494Z+ ./tests/bootstrap-2462efdf6b2e3290
9132023-12-08T16:13:43.519Z
9142023-12-08T16:13:43.519Zrunning 0 tests
9152023-12-08T16:13:43.519Z
9162023-12-08T16:13:43.519Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9172023-12-08T16:13:43.519Z
9182023-12-08T16:13:43.519Z+ for test_bin in tests/*
9192023-12-08T16:13:43.519Z+ ./tests/end_to_end_tests-7ac35544d5ae1b92
9202023-12-08T16:13:43.519Z
9212023-12-08T16:13:43.520Zrunning 2 tests
9222023-12-08T16:13:43.738Ztest no_spoof::no_spoof_here ... ok
9232023-12-08T16:14:43.517Ztest instance_launch::instance_launch has been running for over 60 seconds
9242023-12-08T16:16:41.054Ztest instance_launch::instance_launch ... ok
9252023-12-08T16:16:41.054Z
9262023-12-08T16:16:41.054Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 177.54s
9272023-12-08T16:16:41.056Z
9282023-12-08T16:16:41.082Z+ _exit_trap
9292023-12-08T16:16:41.082Z+ local status=0
9302023-12-08T16:16:41.082Z+ [[ 0 -eq 0 ]]
9312023-12-08T16:16:41.082Z+ exit 0
9322023-12-08T16:16:41.082Zprocess exited: duration 366022 ms, exit code 0
 
9332023-12-08T16:16:41.183Zfound 90 output files
9342023-12-08T16:16:41.183Zuploading: /var/svc/log/oxide-sled-agent:default.log (704894 bytes)
9352023-12-08T16:16:42.874Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9362023-12-08T16:16:42.898Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ca35c857-5b3d-422e-85e5-885b3b473206/root/var/svc/log/oxide-clickhouse:default.log (169917 bytes)
9372023-12-08T16:16:44.105Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ca35c857-5b3d-422e-85e5-885b3b473206/root/var/svc/log/oxide-clickhouse:default.log
9382023-12-08T16:16:44.105Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ca35c857-5b3d-422e-85e5-885b3b473206/root/var/svc/log/oxide-clickhouse:default.log.0 (340718 bytes)
9392023-12-08T16:16:45.434Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ca35c857-5b3d-422e-85e5-885b3b473206/root/var/svc/log/oxide-clickhouse:default.log.0
9402023-12-08T16:16:45.434Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c3bb57e0-8244-465b-aa23-d079985c713d/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9412023-12-08T16:16:46.505Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c3bb57e0-8244-465b-aa23-d079985c713d/root/var/svc/log/oxide-cockroachdb:default.log
9422023-12-08T16:16:46.529Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c3bb57e0-8244-465b-aa23-d079985c713d/root/var/svc/log/oxide-cockroachdb:default.log.0 (5367 bytes)
9432023-12-08T16:16:47.602Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c3bb57e0-8244-465b-aa23-d079985c713d/root/var/svc/log/oxide-cockroachdb:default.log.0
9442023-12-08T16:16:47.602Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1557f880-f55d-44ec-b2b4-1334cd994e7a/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9452023-12-08T16:16:48.673Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1557f880-f55d-44ec-b2b4-1334cd994e7a/root/var/svc/log/oxide-crucible-agent:default.log
9462023-12-08T16:16:48.673Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1557f880-f55d-44ec-b2b4-1334cd994e7a/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
9472023-12-08T16:16:49.770Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1557f880-f55d-44ec-b2b4-1334cd994e7a/root/var/svc/log/oxide-crucible-agent:default.log.0
9482023-12-08T16:16:49.796Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1557f880-f55d-44ec-b2b4-1334cd994e7a/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9492023-12-08T16:16:50.841Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1557f880-f55d-44ec-b2b4-1334cd994e7a/root/var/svc/log/oxide-crucible-downstairs:default.log
9502023-12-08T16:16:50.841Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1557f880-f55d-44ec-b2b4-1334cd994e7a/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9512023-12-08T16:16:51.935Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1557f880-f55d-44ec-b2b4-1334cd994e7a/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9522023-12-08T16:16:51.935Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_71e3cf5e-0933-4808-afe9-249a2432d042/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
9532023-12-08T16:16:53.005Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_71e3cf5e-0933-4808-afe9-249a2432d042/root/var/svc/log/oxide-crucible-pantry:default.log
9542023-12-08T16:16:53.030Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_71e3cf5e-0933-4808-afe9-249a2432d042/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
9552023-12-08T16:16:54.102Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_71e3cf5e-0933-4808-afe9-249a2432d042/root/var/svc/log/oxide-crucible-pantry:default.log.0
9562023-12-08T16:16:54.102Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ad8e05a8-d0d1-43fd-a6f1-0bc98b3ce2ce/root/var/svc/log/oxide-internal_dns:default.log (15055 bytes)
9572023-12-08T16:16:55.220Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ad8e05a8-d0d1-43fd-a6f1-0bc98b3ce2ce/root/var/svc/log/oxide-internal_dns:default.log
9582023-12-08T16:16:55.245Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ad8e05a8-d0d1-43fd-a6f1-0bc98b3ce2ce/root/var/svc/log/oxide-internal_dns:default.log.0 (100261 bytes)
9592023-12-08T16:16:56.427Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ad8e05a8-d0d1-43fd-a6f1-0bc98b3ce2ce/root/var/svc/log/oxide-internal_dns:default.log.0
9602023-12-08T16:16:56.427Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_94659037-20e4-4949-bf94-607bd7066776/root/var/svc/log/oxide-nexus:default.log (137619 bytes)
9612023-12-08T16:16:56.604Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_94659037-20e4-4949-bf94-607bd7066776/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 137619 -> 143892
9622023-12-08T16:16:57.674Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_94659037-20e4-4949-bf94-607bd7066776/root/var/svc/log/oxide-nexus:default.log
9632023-12-08T16:16:57.674Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_94659037-20e4-4949-bf94-607bd7066776/root/var/svc/log/oxide-nexus:default.log.0 (380510 bytes)
9642023-12-08T16:16:58.955Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_94659037-20e4-4949-bf94-607bd7066776/root/var/svc/log/oxide-nexus:default.log.0
9652023-12-08T16:16:58.955Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6d65adcf-4d6c-4ea6-b2df-6cb48cc2abc6/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9662023-12-08T16:17:00.023Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6d65adcf-4d6c-4ea6-b2df-6cb48cc2abc6/root/var/svc/log/oxide-cockroachdb:default.log
9672023-12-08T16:17:00.023Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6d65adcf-4d6c-4ea6-b2df-6cb48cc2abc6/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
9682023-12-08T16:17:01.117Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6d65adcf-4d6c-4ea6-b2df-6cb48cc2abc6/root/var/svc/log/oxide-cockroachdb:default.log.0
9692023-12-08T16:17:01.117Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45c37974-2f06-4868-82dc-eb5354d6b413/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9702023-12-08T16:17:02.192Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45c37974-2f06-4868-82dc-eb5354d6b413/root/var/svc/log/oxide-crucible-agent:default.log
9712023-12-08T16:17:02.193Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45c37974-2f06-4868-82dc-eb5354d6b413/root/var/svc/log/oxide-crucible-agent:default.log.0 (13689 bytes)
9722023-12-08T16:17:03.310Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45c37974-2f06-4868-82dc-eb5354d6b413/root/var/svc/log/oxide-crucible-agent:default.log.0
9732023-12-08T16:17:03.337Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45c37974-2f06-4868-82dc-eb5354d6b413/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9742023-12-08T16:17:04.382Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45c37974-2f06-4868-82dc-eb5354d6b413/root/var/svc/log/oxide-crucible-downstairs:default.log
9752023-12-08T16:17:04.382Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45c37974-2f06-4868-82dc-eb5354d6b413/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9762023-12-08T16:17:05.481Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45c37974-2f06-4868-82dc-eb5354d6b413/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9772023-12-08T16:17:05.481Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45c37974-2f06-4868-82dc-eb5354d6b413/root/var/svc/log/oxide-crucible-downstairs:downstairs-030cf053-7ccf-4089-abd2-f4bac87312a2.log (0 bytes)
9782023-12-08T16:17:06.557Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45c37974-2f06-4868-82dc-eb5354d6b413/root/var/svc/log/oxide-crucible-downstairs:downstairs-030cf053-7ccf-4089-abd2-f4bac87312a2.log
9792023-12-08T16:17:06.557Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45c37974-2f06-4868-82dc-eb5354d6b413/root/var/svc/log/oxide-crucible-downstairs:downstairs-030cf053-7ccf-4089-abd2-f4bac87312a2.log.0 (3050 bytes)
9802023-12-08T16:17:07.650Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45c37974-2f06-4868-82dc-eb5354d6b413/root/var/svc/log/oxide-crucible-downstairs:downstairs-030cf053-7ccf-4089-abd2-f4bac87312a2.log.0
9812023-12-08T16:17:07.677Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_db6bb4b2-1771-47ee-b36d-ba48d35c8612/root/var/svc/log/oxide-nexus:default.log (137609 bytes)
9822023-12-08T16:17:07.833Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_db6bb4b2-1771-47ee-b36d-ba48d35c8612/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 137609 -> 144488
9832023-12-08T16:17:08.905Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_db6bb4b2-1771-47ee-b36d-ba48d35c8612/root/var/svc/log/oxide-nexus:default.log
9842023-12-08T16:17:08.905Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_db6bb4b2-1771-47ee-b36d-ba48d35c8612/root/var/svc/log/oxide-nexus:default.log.0 (506960 bytes)
9852023-12-08T16:17:10.206Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_db6bb4b2-1771-47ee-b36d-ba48d35c8612/root/var/svc/log/oxide-nexus:default.log.0
9862023-12-08T16:17:10.206Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cf5a346c-31a3-4ace-9286-7359f3dec246/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9872023-12-08T16:17:11.278Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cf5a346c-31a3-4ace-9286-7359f3dec246/root/var/svc/log/oxide-crucible-agent:default.log
9882023-12-08T16:17:11.278Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cf5a346c-31a3-4ace-9286-7359f3dec246/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
9892023-12-08T16:17:12.377Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cf5a346c-31a3-4ace-9286-7359f3dec246/root/var/svc/log/oxide-crucible-agent:default.log.0
9902023-12-08T16:17:12.377Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cf5a346c-31a3-4ace-9286-7359f3dec246/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9912023-12-08T16:17:13.449Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cf5a346c-31a3-4ace-9286-7359f3dec246/root/var/svc/log/oxide-crucible-downstairs:default.log
9922023-12-08T16:17:13.449Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cf5a346c-31a3-4ace-9286-7359f3dec246/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9932023-12-08T16:17:14.544Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cf5a346c-31a3-4ace-9286-7359f3dec246/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9942023-12-08T16:17:14.545Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e411c7fa-7dbd-4031-9ab8-6871bb53980e/root/var/svc/log/oxide-crucible-agent:default.log (4324 bytes)
9952023-12-08T16:17:15.645Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e411c7fa-7dbd-4031-9ab8-6871bb53980e/root/var/svc/log/oxide-crucible-agent:default.log
9962023-12-08T16:17:15.645Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e411c7fa-7dbd-4031-9ab8-6871bb53980e/root/var/svc/log/oxide-crucible-agent:default.log.0 (9366 bytes)
9972023-12-08T16:17:16.766Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e411c7fa-7dbd-4031-9ab8-6871bb53980e/root/var/svc/log/oxide-crucible-agent:default.log.0
9982023-12-08T16:17:16.766Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e411c7fa-7dbd-4031-9ab8-6871bb53980e/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9992023-12-08T16:17:17.839Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e411c7fa-7dbd-4031-9ab8-6871bb53980e/root/var/svc/log/oxide-crucible-downstairs:default.log
10002023-12-08T16:17:17.840Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e411c7fa-7dbd-4031-9ab8-6871bb53980e/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10012023-12-08T16:17:18.938Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e411c7fa-7dbd-4031-9ab8-6871bb53980e/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10022023-12-08T16:17:18.939Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e411c7fa-7dbd-4031-9ab8-6871bb53980e/root/var/svc/log/oxide-crucible-downstairs:downstairs-97f2f602-e3b1-42d7-b794-0095af9d0e26.log (1006 bytes)
10032023-12-08T16:17:20.034Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e411c7fa-7dbd-4031-9ab8-6871bb53980e/root/var/svc/log/oxide-crucible-downstairs:downstairs-97f2f602-e3b1-42d7-b794-0095af9d0e26.log
10042023-12-08T16:17:20.034Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e411c7fa-7dbd-4031-9ab8-6871bb53980e/root/var/svc/log/oxide-crucible-downstairs:downstairs-97f2f602-e3b1-42d7-b794-0095af9d0e26.log.0 (4523 bytes)
10052023-12-08T16:17:21.130Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e411c7fa-7dbd-4031-9ab8-6871bb53980e/root/var/svc/log/oxide-crucible-downstairs:downstairs-97f2f602-e3b1-42d7-b794-0095af9d0e26.log.0
10062023-12-08T16:17:21.130Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_7c4351e4-c72c-4739-8fb2-c00f44b70f70/root/var/svc/log/oxide-internal_dns:default.log (10023 bytes)
10072023-12-08T16:17:22.248Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_7c4351e4-c72c-4739-8fb2-c00f44b70f70/root/var/svc/log/oxide-internal_dns:default.log
10082023-12-08T16:17:22.272Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_7c4351e4-c72c-4739-8fb2-c00f44b70f70/root/var/svc/log/oxide-internal_dns:default.log.0 (100430 bytes)
10092023-12-08T16:17:23.457Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_7c4351e4-c72c-4739-8fb2-c00f44b70f70/root/var/svc/log/oxide-internal_dns:default.log.0
10102023-12-08T16:17:23.457Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_7a3059be-bf36-4e26-868f-6aae95eb217d/root/var/svc/log/oxide-nexus:default.log (1254488 bytes)
10112023-12-08T16:17:23.745Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_7a3059be-bf36-4e26-868f-6aae95eb217d/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1254488 -> 1327016
10122023-12-08T16:17:24.817Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_7a3059be-bf36-4e26-868f-6aae95eb217d/root/var/svc/log/oxide-nexus:default.log
10132023-12-08T16:17:24.817Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_7a3059be-bf36-4e26-868f-6aae95eb217d/root/var/svc/log/oxide-nexus:default.log.0 (3503297 bytes)
10142023-12-08T16:17:26.685Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_7a3059be-bf36-4e26-868f-6aae95eb217d/root/var/svc/log/oxide-nexus:default.log.0
10152023-12-08T16:17:26.712Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_7ed74be8-9cc1-4c26-becb-a474e927d3f1/root/var/svc/log/oxide-external_dns:default.log (7547 bytes)
10162023-12-08T16:17:26.755Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_7ed74be8-9cc1-4c26-becb-a474e927d3f1/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 7547 -> 11322
10172023-12-08T16:17:27.826Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_7ed74be8-9cc1-4c26-becb-a474e927d3f1/root/var/svc/log/oxide-external_dns:default.log
10182023-12-08T16:17:27.850Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_7ed74be8-9cc1-4c26-becb-a474e927d3f1/root/var/svc/log/oxide-external_dns:default.log.0 (8059 bytes)
10192023-12-08T16:17:28.945Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_7ed74be8-9cc1-4c26-becb-a474e927d3f1/root/var/svc/log/oxide-external_dns:default.log.0
10202023-12-08T16:17:28.945Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_078f0fff-fad2-4879-8099-f6bff21e0ed3/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10212023-12-08T16:17:30.017Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_078f0fff-fad2-4879-8099-f6bff21e0ed3/root/var/svc/log/oxide-cockroachdb:default.log
10222023-12-08T16:17:30.017Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_078f0fff-fad2-4879-8099-f6bff21e0ed3/root/var/svc/log/oxide-cockroachdb:default.log.0 (5342 bytes)
10232023-12-08T16:17:31.118Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_078f0fff-fad2-4879-8099-f6bff21e0ed3/root/var/svc/log/oxide-cockroachdb:default.log.0
10242023-12-08T16:17:31.118Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_ffed18a9-fd67-4e11-b520-5c1f2e90fe27/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10252023-12-08T16:17:32.187Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_ffed18a9-fd67-4e11-b520-5c1f2e90fe27/root/var/svc/log/oxide-cockroachdb:default.log
10262023-12-08T16:17:32.187Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_ffed18a9-fd67-4e11-b520-5c1f2e90fe27/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
10272023-12-08T16:17:33.283Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_ffed18a9-fd67-4e11-b520-5c1f2e90fe27/root/var/svc/log/oxide-cockroachdb:default.log.0
10282023-12-08T16:17:33.283Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-agent:default.log (4322 bytes)
10292023-12-08T16:17:34.390Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-agent:default.log
10302023-12-08T16:17:34.390Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-agent:default.log.0 (19610 bytes)
10312023-12-08T16:17:35.530Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-agent:default.log.0
10322023-12-08T16:17:35.555Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10332023-12-08T16:17:36.602Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-downstairs:default.log
10342023-12-08T16:17:36.602Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10352023-12-08T16:17:37.696Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10362023-12-08T16:17:37.696Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-downstairs:downstairs-020cccc0-ed3f-44fc-9f14-9cc69225caeb.log (0 bytes)
10372023-12-08T16:17:38.767Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-downstairs:downstairs-020cccc0-ed3f-44fc-9f14-9cc69225caeb.log
10382023-12-08T16:17:38.767Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-downstairs:downstairs-020cccc0-ed3f-44fc-9f14-9cc69225caeb.log.0 (3049 bytes)
10392023-12-08T16:17:39.861Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-downstairs:downstairs-020cccc0-ed3f-44fc-9f14-9cc69225caeb.log.0
10402023-12-08T16:17:39.886Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-downstairs:downstairs-292df26a-818b-4a61-8942-c8e2499060e4.log (1006 bytes)
10412023-12-08T16:17:40.955Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-downstairs:downstairs-292df26a-818b-4a61-8942-c8e2499060e4.log
10422023-12-08T16:17:40.980Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-downstairs:downstairs-292df26a-818b-4a61-8942-c8e2499060e4.log.0 (4527 bytes)
10432023-12-08T16:17:42.051Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0aa626e5-1b94-4748-b272-a9d837e72b61/root/var/svc/log/oxide-crucible-downstairs:downstairs-292df26a-818b-4a61-8942-c8e2499060e4.log.0
10442023-12-08T16:17:42.051Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_8f0667e5-23da-4be9-b42a-0c5cba3b4466/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10452023-12-08T16:17:43.154Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_8f0667e5-23da-4be9-b42a-0c5cba3b4466/root/var/svc/log/oxide-cockroachdb:default.log
10462023-12-08T16:17:43.154Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_8f0667e5-23da-4be9-b42a-0c5cba3b4466/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10472023-12-08T16:17:44.248Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_8f0667e5-23da-4be9-b42a-0c5cba3b4466/root/var/svc/log/oxide-cockroachdb:default.log.0
10482023-12-08T16:17:44.272Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0e69ecd9-f37b-423d-9146-1774b81163cc/root/var/svc/log/oxide-crucible-agent:default.log (4787 bytes)
10492023-12-08T16:17:45.342Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0e69ecd9-f37b-423d-9146-1774b81163cc/root/var/svc/log/oxide-crucible-agent:default.log
10502023-12-08T16:17:45.344Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0e69ecd9-f37b-423d-9146-1774b81163cc/root/var/svc/log/oxide-crucible-agent:default.log.0 (9795 bytes)
10512023-12-08T16:17:46.458Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0e69ecd9-f37b-423d-9146-1774b81163cc/root/var/svc/log/oxide-crucible-agent:default.log.0
10522023-12-08T16:17:46.458Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0e69ecd9-f37b-423d-9146-1774b81163cc/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10532023-12-08T16:17:47.531Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0e69ecd9-f37b-423d-9146-1774b81163cc/root/var/svc/log/oxide-crucible-downstairs:default.log
10542023-12-08T16:17:47.531Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0e69ecd9-f37b-423d-9146-1774b81163cc/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10552023-12-08T16:17:48.626Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0e69ecd9-f37b-423d-9146-1774b81163cc/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10562023-12-08T16:17:48.626Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0e69ecd9-f37b-423d-9146-1774b81163cc/root/var/svc/log/oxide-crucible-downstairs:downstairs-a441f68d-e098-4c45-929a-cd15dd63ebd4.log (1005 bytes)
10572023-12-08T16:17:49.726Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0e69ecd9-f37b-423d-9146-1774b81163cc/root/var/svc/log/oxide-crucible-downstairs:downstairs-a441f68d-e098-4c45-929a-cd15dd63ebd4.log
10582023-12-08T16:17:49.750Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0e69ecd9-f37b-423d-9146-1774b81163cc/root/var/svc/log/oxide-crucible-downstairs:downstairs-a441f68d-e098-4c45-929a-cd15dd63ebd4.log.0 (4530 bytes)
10592023-12-08T16:17:50.819Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0e69ecd9-f37b-423d-9146-1774b81163cc/root/var/svc/log/oxide-crucible-downstairs:downstairs-a441f68d-e098-4c45-929a-cd15dd63ebd4.log.0
10602023-12-08T16:17:50.845Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8ac57c92-9071-4312-8380-836936b0203f/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10612023-12-08T16:17:51.889Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8ac57c92-9071-4312-8380-836936b0203f/root/var/svc/log/oxide-crucible-agent:default.log
10622023-12-08T16:17:51.917Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8ac57c92-9071-4312-8380-836936b0203f/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10632023-12-08T16:17:52.987Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8ac57c92-9071-4312-8380-836936b0203f/root/var/svc/log/oxide-crucible-agent:default.log.0
10642023-12-08T16:17:52.988Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8ac57c92-9071-4312-8380-836936b0203f/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10652023-12-08T16:17:54.061Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8ac57c92-9071-4312-8380-836936b0203f/root/var/svc/log/oxide-crucible-downstairs:default.log
10662023-12-08T16:17:54.061Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8ac57c92-9071-4312-8380-836936b0203f/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10672023-12-08T16:17:55.160Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8ac57c92-9071-4312-8380-836936b0203f/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10682023-12-08T16:17:55.160Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_7a10e1fa-4831-4c32-9c10-3fe7e806f467/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10692023-12-08T16:17:56.231Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_7a10e1fa-4831-4c32-9c10-3fe7e806f467/root/var/svc/log/oxide-crucible-pantry:default.log
10702023-12-08T16:17:56.232Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_7a10e1fa-4831-4c32-9c10-3fe7e806f467/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
10712023-12-08T16:17:57.332Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_7a10e1fa-4831-4c32-9c10-3fe7e806f467/root/var/svc/log/oxide-crucible-pantry:default.log.0
10722023-12-08T16:17:57.332Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ee64d6f6-aa26-4f0c-ae8c-5455b5a82eea/root/var/svc/log/oxide-external_dns:default.log (7552 bytes)
10732023-12-08T16:17:57.402Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ee64d6f6-aa26-4f0c-ae8c-5455b5a82eea/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 7552 -> 11330
10742023-12-08T16:17:58.474Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ee64d6f6-aa26-4f0c-ae8c-5455b5a82eea/root/var/svc/log/oxide-external_dns:default.log
10752023-12-08T16:17:58.474Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ee64d6f6-aa26-4f0c-ae8c-5455b5a82eea/root/var/svc/log/oxide-external_dns:default.log.0 (8060 bytes)
10762023-12-08T16:17:59.571Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ee64d6f6-aa26-4f0c-ae8c-5455b5a82eea/root/var/svc/log/oxide-external_dns:default.log.0
10772023-12-08T16:17:59.572Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3c143813-3c2c-409f-acf9-18171a272b72/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10782023-12-08T16:18:00.699Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3c143813-3c2c-409f-acf9-18171a272b72/root/var/svc/log/oxide-crucible-agent:default.log
10792023-12-08T16:18:00.700Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3c143813-3c2c-409f-acf9-18171a272b72/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10802023-12-08T16:18:01.785Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3c143813-3c2c-409f-acf9-18171a272b72/root/var/svc/log/oxide-crucible-agent:default.log.0
10812023-12-08T16:18:01.785Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3c143813-3c2c-409f-acf9-18171a272b72/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10822023-12-08T16:18:02.861Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3c143813-3c2c-409f-acf9-18171a272b72/root/var/svc/log/oxide-crucible-downstairs:default.log
10832023-12-08T16:18:02.861Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3c143813-3c2c-409f-acf9-18171a272b72/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10842023-12-08T16:18:03.961Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3c143813-3c2c-409f-acf9-18171a272b72/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10852023-12-08T16:18:03.961Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c400b559-11eb-4e8d-bb7f-7c022eea28de/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10862023-12-08T16:18:05.034Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c400b559-11eb-4e8d-bb7f-7c022eea28de/root/var/svc/log/oxide-crucible-agent:default.log
10872023-12-08T16:18:05.035Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c400b559-11eb-4e8d-bb7f-7c022eea28de/root/var/svc/log/oxide-crucible-agent:default.log.0 (14123 bytes)
10882023-12-08T16:18:06.152Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c400b559-11eb-4e8d-bb7f-7c022eea28de/root/var/svc/log/oxide-crucible-agent:default.log.0
10892023-12-08T16:18:06.153Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c400b559-11eb-4e8d-bb7f-7c022eea28de/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10902023-12-08T16:18:07.225Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c400b559-11eb-4e8d-bb7f-7c022eea28de/root/var/svc/log/oxide-crucible-downstairs:default.log
10912023-12-08T16:18:07.250Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c400b559-11eb-4e8d-bb7f-7c022eea28de/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10922023-12-08T16:18:08.321Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c400b559-11eb-4e8d-bb7f-7c022eea28de/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10932023-12-08T16:18:08.322Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c400b559-11eb-4e8d-bb7f-7c022eea28de/root/var/svc/log/oxide-crucible-downstairs:downstairs-12419778-eb20-4d26-ad12-a20f7130cb38.log (0 bytes)
10942023-12-08T16:18:09.393Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c400b559-11eb-4e8d-bb7f-7c022eea28de/root/var/svc/log/oxide-crucible-downstairs:downstairs-12419778-eb20-4d26-ad12-a20f7130cb38.log
10952023-12-08T16:18:09.394Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c400b559-11eb-4e8d-bb7f-7c022eea28de/root/var/svc/log/oxide-crucible-downstairs:downstairs-12419778-eb20-4d26-ad12-a20f7130cb38.log.0 (3051 bytes)
10962023-12-08T16:18:10.490Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c400b559-11eb-4e8d-bb7f-7c022eea28de/root/var/svc/log/oxide-crucible-downstairs:downstairs-12419778-eb20-4d26-ad12-a20f7130cb38.log.0
10972023-12-08T16:18:10.490Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_32645314-8508-40d0-82e5-ac4f804da070/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10982023-12-08T16:18:11.562Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_32645314-8508-40d0-82e5-ac4f804da070/root/var/svc/log/oxide-crucible-pantry:default.log
10992023-12-08T16:18:11.562Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_32645314-8508-40d0-82e5-ac4f804da070/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2337 bytes)
11002023-12-08T16:18:12.658Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_32645314-8508-40d0-82e5-ac4f804da070/root/var/svc/log/oxide-crucible-pantry:default.log.0
11012023-12-08T16:18:12.658Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_53f0a92e-add4-40fc-a0c5-d776e1124397/root/var/svc/log/oxide-internal_dns:default.log (7510 bytes)
11022023-12-08T16:18:12.732Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_53f0a92e-add4-40fc-a0c5-d776e1124397/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 7510 -> 11264
11032023-12-08T16:18:13.804Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_53f0a92e-add4-40fc-a0c5-d776e1124397/root/var/svc/log/oxide-internal_dns:default.log
11042023-12-08T16:18:13.804Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_53f0a92e-add4-40fc-a0c5-d776e1124397/root/var/svc/log/oxide-internal_dns:default.log.0 (104973 bytes)
11052023-12-08T16:18:15.013Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_53f0a92e-add4-40fc-a0c5-d776e1124397/root/var/svc/log/oxide-internal_dns:default.log.0
11062023-12-08T16:18:15.013Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe/root/var/svc/log/oxide-ntp:default.log (0 bytes)
11072023-12-08T16:18:16.088Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe/root/var/svc/log/oxide-ntp:default.log
11082023-12-08T16:18:16.127Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe/root/var/svc/log/oxide-ntp:default.log.0 (1390 bytes)
11092023-12-08T16:18:17.202Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe/root/var/svc/log/oxide-ntp:default.log.0
11102023-12-08T16:18:17.202Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_959597aa-effe-480c-b35e-6e516a384a19/root/var/svc/log/oxide-oximeter:default.log (59401 bytes)
11112023-12-08T16:18:17.362Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_959597aa-effe-480c-b35e-6e516a384a19/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 59401 -> 110671
11122023-12-08T16:18:18.434Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_959597aa-effe-480c-b35e-6e516a384a19/root/var/svc/log/oxide-oximeter:default.log
11132023-12-08T16:18:18.434Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_959597aa-effe-480c-b35e-6e516a384a19/root/var/svc/log/oxide-oximeter:default.log.0 (82751 bytes)
11142023-12-08T16:18:19.620Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_959597aa-effe-480c-b35e-6e516a384a19/root/var/svc/log/oxide-oximeter:default.log.0
11152023-12-08T16:18:19.620Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe/root/var/log/chrony/measurements.log (12056 bytes)
11162023-12-08T16:18:19.690Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe/root/var/log/chrony/measurements.log" changed size mid upload: 12056 -> 13563
11172023-12-08T16:18:20.762Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe/root/var/log/chrony/measurements.log
11182023-12-08T16:18:20.763Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe/root/var/log/chrony/statistics.log (7371 bytes)
11192023-12-08T16:18:20.833Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe/root/var/log/chrony/statistics.log" changed size mid upload: 7371 -> 9009
11202023-12-08T16:18:21.906Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe/root/var/log/chrony/statistics.log
11212023-12-08T16:18:21.906Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe/root/var/log/chrony/tracking.log (1851 bytes)
11222023-12-08T16:18:21.953Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe/root/var/log/chrony/tracking.log" changed size mid upload: 1851 -> 2115
11232023-12-08T16:18:23.029Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_2d2bc60e-3147-4dc3-a52a-b18695c6f9fe/root/var/log/chrony/tracking.log