01HFQ4VSDZWWSPJKE5F28E585M: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HFQ4X6AEB3A7CN7VWQ4HE1HR

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-20T20:47:07.641Zjob dependencies complete; ready to run (waiting for 50 m 17 s)
22023-11-20T20:47:07.681Zjob assigned to worker 01HFQ7BZ06JBENE4KGMGCNTM1Y (queued for 0 s)
32023-11-20T20:47:07.969Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-11-20T20:47:08.004Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-11-20T20:47:08.052Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-11-20T20:47:08.094Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-20T20:47:08.133Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-11-20T20:47:08.189ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-20T20:47:08.245Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-20T20:47:08.283ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-20T20:47:08.317ZConfiguring devices.
122023-11-20T20:47:08.361Z#################################################
132023-11-20T20:47:08.406Z#### ############# ##
142023-11-20T20:47:08.439Z### ### ############ ##
152023-11-20T20:47:08.473Z## ### # ## ### ## Oxide Computer Company ##
162023-11-20T20:47:08.506Z## ## ## ### # ### ##
172023-11-20T20:47:08.540Z## # ### #### #### This Station Under ##
182023-11-20T20:47:08.573Z### ### #### # ### Computer Control ##
192023-11-20T20:47:08.606Z#### #### ### ## ##
202023-11-20T20:47:08.673Z##################################################
212023-11-20T20:47:08.718ZHostname: sock
222023-11-20T20:47:08.751ZLoading smf(7) serice descriptions:
232023-11-20T20:47:09.583Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-4172957c237fed9a.gz
242023-11-20T20:47:14.105Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-4172957c237fed9a.gz
252023-11-20T20:47:14.131Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-20T20:47:31.931Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-20T20:47:31.955Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8b2150e0e5683f6a.gz
282023-11-20T20:47:51.257Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8b2150e0e5683f6a.gz
292023-11-20T20:47:51.282Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-20T20:47:52.535Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-20T20:47:52.559Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-20T20:47:55.495Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-20T20:47:55.520Zdownloading input: /input/package/work/version.txt
342023-11-20T20:47:55.548Zdownloaded input: /input/package/work/version.txt
352023-11-20T20:47:55.573Zdownloading input: /input/package/work/package.tar.gz
362023-11-20T20:48:10.722Zdownloaded input: /input/package/work/package.tar.gz
372023-11-20T20:48:10.751Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-20T20:48:34.955Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-20T20:48:34.984Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-20T20:48:43.722Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-20T20:48:43.747Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-20T20:49:30.290Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-20T20:49:30.315Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-20T20:50:08.151Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-20T20:50:08.176Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-20T20:50:36.444Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-20T20:50:36.470Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-20T20:50:42.361Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-20T20:50:42.405Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-20T20:50:54.223Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-20T20:50:54.251Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-20T20:50:58.897Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-20T20:50:58.922Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-20T20:51:03.237Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-20T20:51:03.272Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-20T20:51:03.297Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-20T20:51:03.326Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-20T20:51:07.847Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-20T20:51:07.871Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-20T20:51:11.191Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-20T20:51:11.216Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-20T20:51:26.443Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-20T20:51:26.468Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-20T20:51:41.688Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-20T20:51:41.714Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-20T20:51:41.778Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-20T20:51:41.802Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-20T20:51:48.116Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-20T20:51:48.142Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-20T20:52:00.267Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-20T20:52:00.293Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-20T20:52:50.340Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-20T20:52:50.365Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-20T20:53:26.145Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-20T20:53:26.173Zstarting task 0: "setup"
762023-11-20T20:53:26.207Z++ uname -s
772023-11-20T20:53:26.231Z+ kern=SunOS
782023-11-20T20:53:26.255Z+ case "$kern" in
792023-11-20T20:53:26.279Z+ groupadd -g 12345 build
802023-11-20T20:53:26.303Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-20T20:53:28.216Z+ zfs create -o mountpoint=/work rpool/work
822023-11-20T20:53:28.264Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-20T20:53:28.288Z+ home_fs=zfs
842023-11-20T20:53:28.312Z+ [[ zfs == autofs ]]
852023-11-20T20:53:28.335Z+ mkdir -p /home/build
862023-11-20T20:53:28.359Z+ chown build:build /home/build /work
872023-11-20T20:53:30.271Z+ chmod 0700 /home/build /work
882023-11-20T20:53:30.304Zprocess exited: duration 4076 ms, exit code 0
 
892023-11-20T20:53:30.356Zstarting task 1: "authentication"
902023-11-20T20:53:30.424Zprocess exited: duration 42 ms, exit code 0
 
912023-11-20T20:53:30.472Zstarting task 2: "build"
922023-11-20T20:53:30.502Z+ trap _exit_trap EXIT
932023-11-20T20:53:30.526Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-20T20:53:30.698Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-20T20:53:30.723Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-20T20:53:30.749Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-20T20:53:30.773Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-20T20:53:30.804Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-20T20:53:30.834Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-20T20:53:30.859Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-20T20:53:30.888Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-20T20:53:30.915Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-20T20:53:30.943Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-20T20:53:30.973Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-20T20:53:31.022Z+ [[ -d /opt/oxide ]]
1062023-11-20T20:53:31.046Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-20T20:53:31.071Z+ pfexec mkdir /opt/oxide
1082023-11-20T20:53:31.095Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-20T20:53:31.119Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-20T20:53:31.144Z+ cd /opt/oxide-underneath
1112023-11-20T20:53:31.168Z+ pfexec tar ceEp@/f - .
1122023-11-20T20:53:31.192Z+ cd /opt/oxide
1132023-11-20T20:53:31.216Z+ pfexec tar xveEp@/f -
1142023-11-20T20:53:31.240Zx ., 0 bytes, 0 tape blocks
1152023-11-20T20:53:31.264Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-20T20:53:31.288Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-20T20:53:31.312Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-20T20:53:31.336Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-20T20:53:31.361Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-20T20:53:31.387Z+ pfexec mkdir /opt/oxide/work
1212023-11-20T20:53:31.412Z+ pfexec chown build:build /opt/oxide/work
1222023-11-20T20:53:31.436Z+ cd /opt/oxide/work
1232023-11-20T20:53:31.460Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-20T20:53:32.411ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-20T20:53:32.435Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-11-20T20:53:32.460Zx out/omicron-sled-agent.tar, 111621632 bytes, 218011 tape blocks
1272023-11-20T20:53:32.484Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-20T20:53:32.508Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-11-20T20:53:32.531Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-11-20T20:53:32.555Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-11-20T20:53:32.579Zx target/release/omicron-package, 42522496 bytes, 83052 tape blocks
1322023-11-20T20:53:33.189Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-11-20T20:53:33.214Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-20T20:53:33.238Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-20T20:53:33.263Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-20T20:53:33.287Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-20T20:53:33.311Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-20T20:53:33.335Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-20T20:53:33.360Z
1402023-11-20T20:53:33.384Zreal 1.550093027
1412023-11-20T20:53:33.412Zuser 1.393522831
1422023-11-20T20:53:33.447Zsys 0.274581827
1432023-11-20T20:53:33.472Ztrap 0.000048599
1442023-11-20T20:53:33.496Ztflt 0.000000000
1452023-11-20T20:53:33.521Zdflt 0.000000000
1462023-11-20T20:53:33.545Zkflt 0.000000000
1472023-11-20T20:53:33.570Zlock 0.000000000
1482023-11-20T20:53:33.594Zslp 1.396551932
1492023-11-20T20:53:33.618Zlat 0.033311347
1502023-11-20T20:53:33.642Zstop 0.000124051
1512023-11-20T20:53:33.668Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-20T20:53:33.694Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-20T20:53:33.717Z+ mkdir tests
1542023-11-20T20:53:33.741Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-20T20:53:33.765Z+ ptime -m gunzip
1562023-11-20T20:53:33.789Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-4172957c237fed9a
1572023-11-20T20:53:33.813Z
1582023-11-20T20:53:33.838Zreal 0.137625014
1592023-11-20T20:53:33.862Zuser 0.127107890
1602023-11-20T20:53:33.895Zsys 0.010260298
1612023-11-20T20:53:33.920Ztrap 0.000014660
1622023-11-20T20:53:33.944Ztflt 0.000000000
1632023-11-20T20:53:33.968Zdflt 0.000000000
1642023-11-20T20:53:33.992Zkflt 0.000000000
1652023-11-20T20:53:34.016Zlock 0.000000000
1662023-11-20T20:53:34.041Zslp 0.000018230
1672023-11-20T20:53:34.066Zlat 0.000015920
1682023-11-20T20:53:34.089Zstop 0.000120702
1692023-11-20T20:53:34.114Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-4172957c237fed9a
1702023-11-20T20:53:34.137Z+ chmod a+x tests/bootstrap-4172957c237fed9a
1712023-11-20T20:53:34.161Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-20T20:53:34.185Z+ ptime -m gunzip
1732023-11-20T20:53:34.218Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-20T20:53:35.669Z
1752023-11-20T20:53:35.693Zreal 2.025182926
1762023-11-20T20:53:35.717Zuser 1.902937069
1772023-11-20T20:53:35.740Zsys 0.121855310
1782023-11-20T20:53:35.764Ztrap 0.000162592
1792023-11-20T20:53:35.788Ztflt 0.000000000
1802023-11-20T20:53:35.818Zdflt 0.000000000
1812023-11-20T20:53:35.844Zkflt 0.000000000
1822023-11-20T20:53:35.868Zlock 0.000000000
1832023-11-20T20:53:35.947Zslp 0.000025070
1842023-11-20T20:53:35.972Zlat 0.000026730
1852023-11-20T20:53:35.996Zstop 0.000113512
1862023-11-20T20:53:36.020Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-20T20:53:36.045Z+ chmod a+x tests/bootstrap
1882023-11-20T20:53:36.074Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-20T20:53:36.101Z+ ptime -m gunzip
1902023-11-20T20:53:36.129Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8b2150e0e5683f6a
1912023-11-20T20:53:37.995Z
1922023-11-20T20:53:38.020Zreal 2.313760333
1932023-11-20T20:53:38.045Zuser 2.177290677
1942023-11-20T20:53:38.069Zsys 0.136032659
1952023-11-20T20:53:38.093Ztrap 0.000190112
1962023-11-20T20:53:38.117Ztflt 0.000000000
1972023-11-20T20:53:38.141Zdflt 0.000000000
1982023-11-20T20:53:38.169Zkflt 0.000000000
1992023-11-20T20:53:38.193Zlock 0.000000000
2002023-11-20T20:53:38.217Zslp 0.000025920
2012023-11-20T20:53:38.241Zlat 0.000036160
2022023-11-20T20:53:38.266Zstop 0.000126862
2032023-11-20T20:53:38.290Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8b2150e0e5683f6a
2042023-11-20T20:53:38.314Z+ chmod a+x tests/end_to_end_tests-8b2150e0e5683f6a
2052023-11-20T20:53:38.359Z+ bmat address ls
2062023-11-20T20:53:38.815ZNAME CIDR FIRST LAST COUNT
2072023-11-20T20:53:38.839Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-11-20T20:53:38.863Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-20T20:53:38.888Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-11-20T20:53:38.914Z+ EXTRA_IP_START=10.151.2.164
2112023-11-20T20:53:38.939Z+ EXTRA_IP_END=10.151.2.227
2122023-11-20T20:53:38.968Z+ EXTRA_IP_COUNT=64
2132023-11-20T20:53:38.998Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-20T20:53:39.022Z+ EXTRA_IP_BASE=10.151.2
2152023-11-20T20:53:39.046Z+ EXTRA_IP_FOCTET=164
2162023-11-20T20:53:39.070Z+ EXTRA_IP_LOCTET=227
2172023-11-20T20:53:39.095Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-11-20T20:53:39.119Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-11-20T20:53:39.143Z+ DNS_IP1=10.151.2.164
2202023-11-20T20:53:39.171Z+ DNS_IP2=10.151.2.165
2212023-11-20T20:53:39.195Z+ UPLINK_IP=10.151.2.174
2222023-11-20T20:53:39.219Z+ IPPOOL_START=10.151.2.175
2232023-11-20T20:53:39.243Z+ IPPOOL_END=10.151.2.227
2242023-11-20T20:53:39.267Z++ ipadm show-addr -po type,addr
2252023-11-20T20:53:39.291Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-20T20:53:39.315Z+ GATEWAY_IP=10.151.2.100
2272023-11-20T20:53:39.339Z+ [[ -n 10.151.2.100 ]]
2282023-11-20T20:53:39.389Z+ ping -s 10.151.2.100 56 1
2292023-11-20T20:53:39.426ZPING 10.151.2.100: 56 data bytes
2302023-11-20T20:53:39.450Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2312023-11-20T20:53:39.476Z
2322023-11-20T20:53:39.511Z----10.151.2.100 PING Statistics----
2332023-11-20T20:53:39.541Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-20T20:53:39.566Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-11-20T20:53:39.595Z++ arp -an
2362023-11-20T20:53:39.703Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-11-20T20:53:39.729Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-11-20T20:53:39.753Z+ routeadm -e ipv4-forwarding -u
2392023-11-20T20:53:39.778Z+ PXA_START=10.151.2.164
2402023-11-20T20:53:39.802Z+ PXA_END=10.151.2.227
2412023-11-20T20:53:39.827Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-20T20:53:39.864Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-20T20:53:39.892Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-20T20:53:39.916Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-20T20:53:39.940Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-20T20:53:39.973Z++ cd ./tools
2472023-11-20T20:53:40.010Z++ pwd
2482023-11-20T20:53:40.042Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-20T20:53:40.074Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-20T20:53:40.104Z+ SOFTNPU_MODE=zone
2512023-11-20T20:53:40.130Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-20T20:53:40.157Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-20T20:53:40.187Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-20T20:53:40.212Z++ dladm show-phys -p -o LINK
2552023-11-20T20:53:40.236Z++ head -1
2562023-11-20T20:53:40.259Z+ PHYSICAL_LINK=igb0
2572023-11-20T20:53:40.284Z+ echo 'Using igb0 as physical link'
2582023-11-20T20:53:40.308ZUsing igb0 as physical link
2592023-11-20T20:53:40.335Z+ ensure_run_as_root
2602023-11-20T20:53:40.360Z++ id -u
2612023-11-20T20:53:40.427Z+ [[ 0 -ne 0 ]]
2622023-11-20T20:53:40.451Z+ ensure_zpools
2632023-11-20T20:53:40.477Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-20T20:53:40.506Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-11-20T20:53:40.542Z+ readarray -t ZPOOLS
2662023-11-20T20:53:40.576Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-11-20T20:53:40.605Z++ sed 's/[ ",]//g'
2682023-11-20T20:53:40.633Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-11-20T20:53:40.657Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-11-20T20:53:40.680ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-11-20T20:53:40.705Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-11-20T20:53:40.729ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2732023-11-20T20:53:41.164Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2742023-11-20T20:53:41.204ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2752023-11-20T20:53:41.239Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2762023-11-20T20:53:41.266ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2772023-11-20T20:53:41.298Z0+0 records in
2782023-11-20T20:53:41.356ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2792023-11-20T20:53:41.393Z0+0 records out
2802023-11-20T20:53:41.479ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2812023-11-20T20:53:41.514Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2822023-11-20T20:53:41.538ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2832023-11-20T20:53:41.562Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2842023-11-20T20:53:41.586ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2852023-11-20T20:53:41.611Z+ set +x
2862023-11-20T20:53:41.634ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2872023-11-20T20:53:41.658Z++ zpool list -o name
2882023-11-20T20:53:41.682ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2892023-11-20T20:53:41.707Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2902023-11-20T20:53:41.731ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2912023-11-20T20:53:41.755Z+ [[ -z '' ]]
2922023-11-20T20:53:41.779ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2932023-11-20T20:53:41.804Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2942023-11-20T20:53:41.831ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2952023-11-20T20:53:41.857Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2962023-11-20T20:53:41.888ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2972023-11-20T20:53:41.915Z+ set +x
2982023-11-20T20:53:41.952ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2992023-11-20T20:53:41.979Z+ for ZPOOL in "${ZPOOLS[@]}"
3002023-11-20T20:53:42.003ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3012023-11-20T20:53:42.027Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3022023-11-20T20:53:42.051ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3032023-11-20T20:53:42.077Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3042023-11-20T20:53:42.100ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3052023-11-20T20:53:42.125Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3062023-11-20T20:53:42.150ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3072023-11-20T20:53:42.174Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3082023-11-20T20:53:42.198ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3092023-11-20T20:53:42.222Z0+0 records in
3102023-11-20T20:53:42.246ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3112023-11-20T20:53:42.271Z0+0 records out
3122023-11-20T20:53:42.298ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3132023-11-20T20:53:42.744Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3142023-11-20T20:53:42.771ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3152023-11-20T20:53:42.815Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-11-20T20:53:42.844ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3172023-11-20T20:53:42.875Z+ set +x
3182023-11-20T20:53:42.899ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3192023-11-20T20:53:42.923Z++ zpool list -o name
3202023-11-20T20:53:42.947ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3212023-11-20T20:53:42.971Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3222023-11-20T20:53:42.995ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3232023-11-20T20:53:43.018Z+ [[ -z '' ]]
3242023-11-20T20:53:43.043ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3252023-11-20T20:53:43.068Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3262023-11-20T20:53:43.092ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3272023-11-20T20:53:43.538Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3282023-11-20T20:53:43.563ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3292023-11-20T20:53:43.586Z+ set +x
3302023-11-20T20:53:43.622ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3312023-11-20T20:53:43.648Z+ for ZPOOL in "${ZPOOLS[@]}"
3322023-11-20T20:53:43.672ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3332023-11-20T20:53:43.698Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3342023-11-20T20:53:43.723ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3352023-11-20T20:53:43.747Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3362023-11-20T20:53:43.811ZSimnet net0/sc0_0 exists
3372023-11-20T20:53:43.835Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3382023-11-20T20:53:43.859ZSimnet net1/sc1_0 exists
3392023-11-20T20:53:43.883Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3402023-11-20T20:53:43.907ZVnic sc0_1 exists
3412023-11-20T20:53:43.934Z0+0 records in
3422023-11-20T20:53:43.960Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
3432023-11-20T20:53:43.984Z0+0 records out
3442023-11-20T20:53:44.008Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3452023-11-20T20:53:44.042Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3462023-11-20T20:53:44.074Z+ set +x
3472023-11-20T20:53:44.099Z++ zpool list -o name
3482023-11-20T20:53:44.123Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3492023-11-20T20:53:44.147Z+ [[ -z '' ]]
3502023-11-20T20:53:44.172Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3512023-11-20T20:53:44.196Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3522023-11-20T20:53:44.220Z+ set +x
3532023-11-20T20:53:44.244Z+ for ZPOOL in "${ZPOOLS[@]}"
3542023-11-20T20:53:44.280Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552023-11-20T20:53:44.305Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562023-11-20T20:53:44.329Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572023-11-20T20:53:44.354Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582023-11-20T20:53:44.382Z0+0 records in
3592023-11-20T20:53:44.410Z0+0 records out
3602023-11-20T20:53:44.438Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3612023-11-20T20:53:44.465Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622023-11-20T20:53:44.493Z+ set +x
3632023-11-20T20:53:44.517Z++ zpool list -o name
3642023-11-20T20:53:44.570Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3652023-11-20T20:53:44.594Z+ [[ -z '' ]]
3662023-11-20T20:53:44.643Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3672023-11-20T20:53:44.667Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3682023-11-20T20:53:44.691Z+ set +x
3692023-11-20T20:53:44.715Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-11-20T20:53:44.740Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3712023-11-20T20:53:44.764Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3722023-11-20T20:53:44.790Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3732023-11-20T20:53:44.817Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3742023-11-20T20:53:44.844Z0+0 records in
3752023-11-20T20:53:44.873Z0+0 records out
3762023-11-20T20:53:44.908Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3772023-11-20T20:53:45.364Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3782023-11-20T20:53:45.388Z+ set +x
3792023-11-20T20:53:45.412Z++ zpool list -o name
3802023-11-20T20:53:45.436Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3812023-11-20T20:53:45.460Z+ [[ -z '' ]]
3822023-11-20T20:53:45.484Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3832023-11-20T20:53:45.508Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3842023-11-20T20:53:45.532Z+ set +x
3852023-11-20T20:53:45.556Z+ for ZPOOL in "${ZPOOLS[@]}"
3862023-11-20T20:53:45.599Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-11-20T20:53:45.623Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-11-20T20:53:45.647Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-11-20T20:53:45.671Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-11-20T20:53:45.696Z0+0 records in
3912023-11-20T20:53:45.724Z0+0 records out
3922023-11-20T20:53:45.750Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3932023-11-20T20:53:45.778Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942023-11-20T20:53:45.805Z+ set +x
3952023-11-20T20:53:45.834Z++ zpool list -o name
3962023-11-20T20:53:45.858Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972023-11-20T20:53:45.882Z+ [[ -z '' ]]
3982023-11-20T20:53:45.911Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992023-11-20T20:53:45.935Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002023-11-20T20:53:45.959Z+ set +x
4012023-11-20T20:53:45.984Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-20T20:53:46.007Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4032023-11-20T20:53:46.035Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-11-20T20:53:46.061Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4052023-11-20T20:53:46.087Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4062023-11-20T20:53:46.112Z0+0 records in
4072023-11-20T20:53:46.139Z0+0 records out
4082023-11-20T20:53:46.164Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4092023-11-20T20:53:46.188Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4102023-11-20T20:53:46.212Z+ set +x
4112023-11-20T20:53:46.239Z++ zpool list -o name
4122023-11-20T20:53:46.267Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4132023-11-20T20:53:46.294Z+ [[ -z '' ]]
4142023-11-20T20:53:46.319Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4152023-11-20T20:53:46.343Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4162023-11-20T20:53:46.368Z+ set +x
4172023-11-20T20:53:46.392Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-11-20T20:53:46.416Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4192023-11-20T20:53:46.443Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4202023-11-20T20:53:46.472Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4212023-11-20T20:53:46.498Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4222023-11-20T20:53:46.522Z0+0 records in
4232023-11-20T20:53:46.547Z0+0 records out
4242023-11-20T20:53:46.572Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4252023-11-20T20:53:46.598Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4262023-11-20T20:53:46.636Z+ set +x
4272023-11-20T20:53:47.075Z++ zpool list -o name
4282023-11-20T20:53:47.116Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4292023-11-20T20:53:47.147Z+ [[ -z '' ]]
4302023-11-20T20:53:47.172Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4312023-11-20T20:53:47.195Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4322023-11-20T20:53:47.220Z+ set +x
4332023-11-20T20:53:47.244Z+ for ZPOOL in "${ZPOOLS[@]}"
4342023-11-20T20:53:47.268Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4352023-11-20T20:53:47.292Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-11-20T20:53:47.316Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4372023-11-20T20:53:47.343Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4382023-11-20T20:53:47.804Z0+0 records in
4392023-11-20T20:53:47.827Z0+0 records out
4402023-11-20T20:53:47.852Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4412023-11-20T20:53:47.876Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4422023-11-20T20:53:47.903Z+ set +x
4432023-11-20T20:53:47.929Z++ zpool list -o name
4442023-11-20T20:53:47.956Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4452023-11-20T20:53:47.987Z+ [[ -z '' ]]
4462023-11-20T20:53:48.025Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4472023-11-20T20:53:48.489Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4482023-11-20T20:53:48.513Z+ set +x
4492023-11-20T20:53:48.555Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4502023-11-20T20:53:48.995Z+ readarray -t ZPOOLS
4512023-11-20T20:53:49.024Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4522023-11-20T20:53:49.050Z++ sed 's/[ ",]//g'
4532023-11-20T20:53:49.074Z+ for ZPOOL in "${ZPOOLS[@]}"
4542023-11-20T20:53:49.098Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4552023-11-20T20:53:49.122Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4562023-11-20T20:53:49.146Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4572023-11-20T20:53:49.170Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4582023-11-20T20:53:49.194Z0+0 records in
4592023-11-20T20:53:49.218Z0+0 records out
4602023-11-20T20:53:49.242Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4612023-11-20T20:53:49.267Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4622023-11-20T20:53:49.294Z+ set +x
4632023-11-20T20:53:49.325Z++ zpool list -o name
4642023-11-20T20:53:49.350Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4652023-11-20T20:53:49.374Z+ [[ -z '' ]]
4662023-11-20T20:53:49.399Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4672023-11-20T20:53:49.423Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4682023-11-20T20:53:49.453Z+ set +x
4692023-11-20T20:53:49.483Z+ for ZPOOL in "${ZPOOLS[@]}"
4702023-11-20T20:53:49.509Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4712023-11-20T20:53:49.535Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4722023-11-20T20:53:49.561Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4732023-11-20T20:53:49.612Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4742023-11-20T20:53:49.636Zdownloading softnpu ...done
4752023-11-20T20:53:49.659Z0+0 records in
4762023-11-20T20:53:49.683Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
4772023-11-20T20:53:49.723Z0+0 records out
4782023-11-20T20:53:49.748Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4792023-11-20T20:53:49.772Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4802023-11-20T20:53:49.797Z+ set +x
4812023-11-20T20:53:49.820Z++ zpool list -o name
4822023-11-20T20:53:49.844Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4832023-11-20T20:53:49.868Z+ [[ -z '' ]]
4842023-11-20T20:53:49.892Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4852023-11-20T20:53:49.916Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4862023-11-20T20:53:49.940Z+ set +x
4872023-11-20T20:53:49.964Z+ [[ zone == \z\o\n\e ]]
4882023-11-20T20:53:49.993Z+ ensure_simulated_links igb0
4892023-11-20T20:53:50.021Z+ local PHYSICAL_LINK=igb0
4902023-11-20T20:53:50.046Z+ INDICES=("0" "1")
4912023-11-20T20:53:50.070Z+ for I in "${INDICES[@]}"
4922023-11-20T20:53:50.094Z++ get_simnet_name_if_exists net0
4932023-11-20T20:53:50.118Z++ dladm show-simnet -p -o LINK net0
4942023-11-20T20:53:50.145Z++ echo ''
4952023-11-20T20:53:50.171Z+ [[ -z '' ]]
4962023-11-20T20:53:50.195Z+ dladm create-simnet -t net0
4972023-11-20T20:53:50.219Z+ dladm create-simnet -t sc0_0
4982023-11-20T20:53:50.243Z+ dladm modify-simnet -t -p net0 sc0_0
4992023-11-20T20:53:50.269Z+ dladm set-linkprop -p mtu=9000 sc0_0
5002023-11-20T20:53:50.300Z+ success 'Simnet net0/sc0_0 exists'
5012023-11-20T20:53:50.325Z+ set +x
5022023-11-20T20:53:50.350Z+ for I in "${INDICES[@]}"
5032023-11-20T20:53:50.374Z++ get_simnet_name_if_exists net1
5042023-11-20T20:53:50.398Z++ dladm show-simnet -p -o LINK net1
5052023-11-20T20:53:50.421Z++ echo ''
5062023-11-20T20:53:50.447Z+ [[ -z '' ]]
5072023-11-20T20:53:50.473Z+ dladm create-simnet -t net1
5082023-11-20T20:53:50.500Z+ dladm create-simnet -t sc1_0
5092023-11-20T20:53:50.533Z+ dladm modify-simnet -t -p net1 sc1_0
5102023-11-20T20:53:50.557Z+ dladm set-linkprop -p mtu=9000 sc1_0
5112023-11-20T20:53:50.581Z+ success 'Simnet net1/sc1_0 exists'
5122023-11-20T20:53:50.605Z+ set +x
5132023-11-20T20:53:50.629Z++ get_vnic_name_if_exists sc0_1
5142023-11-20T20:53:50.653Z++ dladm show-vnic -p -o LINK sc0_1
5152023-11-20T20:53:50.677Z++ echo ''
5162023-11-20T20:53:50.701Z+ [[ -z '' ]]
5172023-11-20T20:53:50.770Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5182023-11-20T20:53:50.794Z+ [[ -v PROMISC_FILT_OFF ]]
5192023-11-20T20:53:50.820Z+ success 'Vnic sc0_1 exists'
5202023-11-20T20:53:50.844Z+ set +x
5212023-11-20T20:53:50.868Z+ warn_if_no_proxy_arp
5222023-11-20T20:53:50.892Z+ [[ -v PXA_START ]]
5232023-11-20T20:53:50.916Z+ [[ -v PXA_END ]]
5242023-11-20T20:53:50.940Z+ ensure_softnpu_zone
5252023-11-20T20:53:51.011Z+ zoneadm list
5262023-11-20T20:53:51.035Z+ grep -q sidecar_softnpu
5272023-11-20T20:53:51.059Z+ [[ -f out/npuzone/npuzone ]]
5282023-11-20T20:53:51.083Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5292023-11-20T20:53:53.913Zdownloading asic_program.so ...done
5302023-11-20T20:53:53.938Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-20T20:54:00.868Zdownloading scadm ...done
5322023-11-20T20:54:00.893Zcreating port tfportrear0_0 for sc0_0
5332023-11-20T20:54:00.917Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-20T20:54:00.941Zsoftnpu zone setup
5352023-11-20T20:54:00.966Zconfigure zone
5362023-11-20T20:54:00.990Zinstall zone
5372023-11-20T20:54:01.951Zboot zone
5382023-11-20T20:54:05.059Z[sidecar_softnpu] svcs milestone/network
5392023-11-20T20:54:05.087ZSTATE STIME FMRI
5402023-11-20T20:54:05.117Zonline 20:54:03 svc:/milestone/network:default
5412023-11-20T20:54:05.146Z[sidecar_softnpu] /softnpu/npu start
5422023-11-20T20:54:05.171Z
5432023-11-20T20:54:05.195Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-20T20:54:05.219Z+ GATEWAY_IP=10.151.2.100
5452023-11-20T20:54:05.260Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-11-20T20:54:05.284ZUsing 10.151.2.100 as gateway ip
5472023-11-20T20:54:05.309Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-20T20:54:05.333Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-11-20T20:54:05.358Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-20T20:54:05.382Z+++ echo 18:c0:4d:0d:a0:2a
5512023-11-20T20:54:05.406Z+ COUNT=1
5522023-11-20T20:54:05.431Z+ [[ 1 -ne 1 ]]
5532023-11-20T20:54:05.455Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-11-20T20:54:05.479ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-11-20T20:54:05.503Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-11-20T20:54:05.527Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-11-20T20:54:05.557Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-20T20:54:05.990Z+ [[ -v PXA_START ]]
5592023-11-20T20:54:06.085Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-11-20T20:54:06.115Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-11-20T20:54:06.139Z+ z_scadm dump-state
5622023-11-20T20:54:06.163Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-20T20:54:06.188Zlocal v6:
5642023-11-20T20:54:06.212Zlocal v4:
5652023-11-20T20:54:06.237Zrouter v6:
5662023-11-20T20:54:06.293Zrouter v4_idx:
5672023-11-20T20:54:06.319Zrouter v4_routes:
5682023-11-20T20:54:06.344Zresolver v4:
5692023-11-20T20:54:06.369Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-11-20T20:54:06.405Zresolver v6:
5712023-11-20T20:54:06.429Znat v4:
5722023-11-20T20:54:06.453Znat_v6:
5732023-11-20T20:54:06.477Zport_mac:
5742023-11-20T20:54:06.501Zproxy arp:
5752023-11-20T20:54:06.530Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-11-20T20:54:06.555Z+ success 'softnpu zone exists'
5772023-11-20T20:54:06.578Z+ set +x
5782023-11-20T20:54:06.602Zsoftnpu zone exists
5792023-11-20T20:54:06.626Z
5802023-11-20T20:54:06.650Zreal 26.896194016
5812023-11-20T20:54:06.674Zuser 0.703793275
5822023-11-20T20:54:06.697Zsys 1.616754720
5832023-11-20T20:54:06.721Ztrap 0.000072186
5842023-11-20T20:54:06.745Ztflt 0.000183059
5852023-11-20T20:54:06.769Zdflt 0.001049981
5862023-11-20T20:54:06.793Zkflt 0.000000000
5872023-11-20T20:54:07.230Zlock 20:50.102442775
5882023-11-20T20:54:07.254Zslp 1:20.448086883
5892023-11-20T20:54:07.287Zlat 0.078270093
5902023-11-20T20:54:07.759Zstop 0.361358646
5912023-11-20T20:54:07.783Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-20T20:54:07.808Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-20T20:54:07.832Z+ SILO_NAME=recovery
5942023-11-20T20:54:07.856Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-20T20:54:07.880Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-20T20:54:07.904Z+ sed -i~ '
5972023-11-20T20:54:07.928Z /^external_dns_ips/c\
5982023-11-20T20:54:07.954Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-11-20T20:54:07.978Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-20T20:54:08.002Z /^first/c\
6012023-11-20T20:54:08.027Zfirst = "10.151.2.164"
6022023-11-20T20:54:08.051Z /^last/c\
6032023-11-20T20:54:08.075Zlast = "10.151.2.173"
6042023-11-20T20:54:08.100Z }
6052023-11-20T20:54:08.124Z /^\[rack_network_config/,/^$/ {
6062023-11-20T20:54:08.148Z /^infra_ip_first/c\
6072023-11-20T20:54:08.173Zinfra_ip_first = "10.151.2.174"
6082023-11-20T20:54:08.197Z /^infra_ip_last/c\
6092023-11-20T20:54:08.221Zinfra_ip_last = "10.151.2.174"
6102023-11-20T20:54:08.246Z }
6112023-11-20T20:54:08.270Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6122023-11-20T20:54:08.295Z /^\[\[rack_network_config.ports/,/^$/ {
6132023-11-20T20:54:08.319Z+++ pkg/config-rss.toml Mon Nov 20 20:54:05 2023
6142023-11-20T20:54:08.354Z /^routes/c\
6152023-11-20T20:54:08.379Z@@ -41,7 +41,7 @@
6162023-11-20T20:54:08.403Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6172023-11-20T20:54:08.427Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6182023-11-20T20:54:08.451Z /^addresses/c\
6192023-11-20T20:54:08.474Z # must be contained in one of the "internal services" IP Pool ranges listed
6202023-11-20T20:54:08.498Zaddresses = \["10.151.2.174/32"\]
6212023-11-20T20:54:08.522Z # below.
6222023-11-20T20:54:08.546Z }
6232023-11-20T20:54:08.577Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6242023-11-20T20:54:08.604Z' pkg/config-rss.toml
6252023-11-20T20:54:08.629Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6262023-11-20T20:54:08.653Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6272023-11-20T20:54:08.677Z
6282023-11-20T20:54:08.701Z+ true
6292023-11-20T20:54:08.725Z # Initial TLS certificates for the external API
6302023-11-20T20:54:08.749Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6312023-11-20T20:54:08.774Z #
6322023-11-20T20:54:08.798Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6332023-11-20T20:54:08.823Z@@ -77,8 +77,8 @@
6342023-11-20T20:54:08.847Z+ TLS_NAME=recovery.sys.oxide.test
6352023-11-20T20:54:08.872Z #
6362023-11-20T20:54:08.896Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6372023-11-20T20:54:08.920Z # For more on this and what to put here, see docs/how-to-run.adoc.
6382023-11-20T20:54:08.944Z.....+............+...+............+...+....+...+...+...+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+......+...+.......+.......................................+...........+.......+.....+......+.........+..........+..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6392023-11-20T20:54:08.970Z [[internal_services_ip_pool_ranges]]
6402023-11-20T20:54:08.997Z.......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.+..+.......+.....+....+......+...+...........+......+....+.....+....+...+.....+....+.....+....+......+...+...+...+..+....+...+....................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+...+...........+.+.....+...+.......+..+............+.+......................................+............+.............+.....+......+.+.....+...+.........+......+...........................+....+........+......+.........+.............+........+.+......+...+..+....+........+......+......+.........+....+..+.+..............+...+..........+...+..+.....................+......+.+.....+...+...................+...............+...+..+.+.................+....+.........+.....+.+........+.+......+.........+..+...+....+..+.........+.+......+........+......+.....................+..........+...+..+...+......+.+..+.+.....+.........+......+.+........+...+....+...+..+.+.................+.+......+...............+...............+......+..+.+......+..............+......+.+...........................+......+.........+..+...+...+............+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6412023-11-20T20:54:09.031Z-first = "192.168.1.20"
6422023-11-20T20:54:09.057Z-----
6432023-11-20T20:54:09.081Z-last = "192.168.1.29"
6442023-11-20T20:54:09.105Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6452023-11-20T20:54:09.129Z+first = "10.151.2.164"
6462023-11-20T20:54:09.153Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6472023-11-20T20:54:09.589Z+last = "10.151.2.173"
6482023-11-20T20:54:09.623Z+ rmdir pkg
6492023-11-20T20:54:10.082Z
6502023-11-20T20:54:10.510Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6512023-11-20T20:54:10.543Z # TODO - this configuration is subject to change going forward. Ultimately these
6522023-11-20T20:54:10.569Z+ pfexec mkdir -p /usr/oxide
6532023-11-20T20:54:10.594Z # parameters should be provided to the control plane via wicket, but we need to
6542023-11-20T20:54:10.618Z+ pfexec rm -f /usr/oxide/tcpproxy
6552023-11-20T20:54:10.642Z@@ -92,8 +92,8 @@
6562023-11-20T20:54:10.666Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6572023-11-20T20:54:10.690Z # A range of IP addresses used by Boundary Services on the external network. In
6582023-11-20T20:54:10.714Z+ pfexec chmod +x /usr/oxide/tcpproxy
6592023-11-20T20:54:10.738Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6602023-11-20T20:54:10.854Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6612023-11-20T20:54:10.879Z # With softnpu, only one address is used.
6622023-11-20T20:54:10.993Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6632023-11-20T20:54:11.018Z-infra_ip_first = "192.168.1.30"
6642023-11-20T20:54:11.041Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6652023-11-20T20:54:11.066Z-infra_ip_last = "192.168.1.30"
6662023-11-20T20:54:11.090Z+ OMICRON_NO_UNINSTALL=1
6672023-11-20T20:54:11.117Z+infra_ip_first = "10.151.2.174"
6682023-11-20T20:54:11.142Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6692023-11-20T20:54:11.168Z+infra_ip_last = "10.151.2.174"
6702023-11-20T20:54:11.203ZNov 20 20:54:06.429 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6712023-11-20T20:54:11.233Z
6722023-11-20T20:54:11.257ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6732023-11-20T20:54:11.282Z # Configurations for BGP routers to run on the scrimlets.
6742023-11-20T20:54:11.306ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6752023-11-20T20:54:11.330Z bgp = []
6762023-11-20T20:54:11.355ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6772023-11-20T20:54:11.382Z@@ -101,9 +101,9 @@
6782023-11-20T20:54:11.407ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6792023-11-20T20:54:11.443Z # You can configure multiple uplinks by repeating the following stanza
6802023-11-20T20:54:11.470ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6812023-11-20T20:54:11.498Z [[rack_network_config.ports]]
6822023-11-20T20:54:11.522ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6832023-11-20T20:54:11.546Z # Routes associated with this port.
6842023-11-20T20:54:11.570ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6852023-11-20T20:54:11.598Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6862023-11-20T20:54:11.623ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6872023-11-20T20:54:11.653Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6882023-11-20T20:54:11.677ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6892023-11-20T20:54:11.702Z # Addresses associated with this port.
6902023-11-20T20:54:11.726ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6912023-11-20T20:54:11.751Z-addresses = ["192.168.1.30/32"]
6922023-11-20T20:54:11.993ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6932023-11-20T20:54:12.471Z+addresses = ["10.151.2.174/32"]
6942023-11-20T20:54:12.496ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6952023-11-20T20:54:12.521Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6962023-11-20T20:54:12.552ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6972023-11-20T20:54:12.578Z port = "qsfp0"
6982023-11-20T20:54:12.601ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6992023-11-20T20:54:12.632Z # The speed of this port.
7002023-11-20T20:54:12.657ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7012023-11-20T20:54:12.681Za pkg/config-rss.toml 7K
7022023-11-20T20:54:12.706ZNov 20 20:54:06.431 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7032023-11-20T20:54:12.732Za pkg/initial-tls-cert.pem 2K
7042023-11-20T20:54:12.758ZNov 20 20:54:06.687 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7052023-11-20T20:54:12.782Za pkg/initial-tls-key.pem 4K
7062023-11-20T20:54:12.807ZNov 20 20:54:06.724 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-20T20:54:12.831ZNov 20 20:54:06.786 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-20T20:54:12.855Z
7092023-11-20T20:54:12.879Zreal 0.493407022
7102023-11-20T20:54:12.903Zuser 0.098501820
7112023-11-20T20:54:12.927Zsys 2.089026258
7122023-11-20T20:54:12.951Ztrap 0.000000350
7132023-11-20T20:54:12.975Ztflt 0.000024408
7142023-11-20T20:54:13.000Zdflt 0.000767825
7152023-11-20T20:54:13.024Zkflt 0.000000000
7162023-11-20T20:54:13.048Zlock 44.356851785
7172023-11-20T20:54:13.072Zslp 0.695784808
7182023-11-20T20:54:13.096Zlat 0.003246681
7192023-11-20T20:54:13.120Zstop 0.208464100
7202023-11-20T20:54:13.144Z+ retry=0
7212023-11-20T20:54:13.168Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-20T20:54:13.200Z+ [[ 0 -gt 30 ]]
7232023-11-20T20:54:13.229Z+ sleep 1
7242023-11-20T20:54:13.253Z+ retry=1
7252023-11-20T20:54:13.278Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-20T20:54:13.302Z+ [[ 1 -gt 30 ]]
7272023-11-20T20:54:13.326Z+ sleep 1
7282023-11-20T20:54:13.350Z+ retry=2
7292023-11-20T20:54:13.374Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-20T20:54:13.398Z+ [[ 2 -gt 30 ]]
7312023-11-20T20:54:13.422Z+ sleep 1
7322023-11-20T20:54:13.446Z+ retry=3
7332023-11-20T20:54:13.470Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-11-20T20:54:13.501Z+ [[ 3 -gt 30 ]]
7352023-11-20T20:54:13.525Z+ sleep 1
7362023-11-20T20:54:13.550Z+ retry=4
7372023-11-20T20:54:13.574Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-11-20T20:54:36.256Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7392023-11-20T20:54:36.282Zlocal v6:
7402023-11-20T20:54:36.306Zfe80::8028:4cff:fe23:f12e
7412023-11-20T20:54:36.330Zfd00:99::1
7422023-11-20T20:54:36.354Zlocal v4:
7432023-11-20T20:54:36.378Z10.151.2.174
7442023-11-20T20:54:36.402Zrouter v6:
7452023-11-20T20:54:36.426Zfdb0:18c0:4d0d:a02a::/64 -> fe80::4cc5:1bff:fe7e:fa68 (1)
7462023-11-20T20:54:36.450Zfd00:1122:3344:1::/64 -> fe80::4cc5:1bff:fe7e:fa68 (1)
7472023-11-20T20:54:36.474Zfd00:1122:3344:101::/64 -> fe80::4cc5:1bff:fe7e:fa68 (1)
7482023-11-20T20:54:36.498Zfd00:1122:3344:2::/64 -> fe80::4cc5:1bff:fe7e:fa68 (1)
7492023-11-20T20:54:36.522Zfd00:1122:3344:3::/64 -> fe80::4cc5:1bff:fe7e:fa68 (1)
7502023-11-20T20:54:36.551Zrouter v4_idx:
7512023-11-20T20:54:36.583Z0.0.0.0/0 -> 0
7522023-11-20T20:54:36.611Zrouter v4_routes:
7532023-11-20T20:54:36.635Z0 -> 10.151.2.100 (2)
7542023-11-20T20:54:36.662Zresolver v4:
7552023-11-20T20:54:36.687Z10.151.2.174 -> 3e:c8:7c:02:bd:e0
7562023-11-20T20:54:36.713Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7572023-11-20T20:54:36.736Zresolver v6:
7582023-11-20T20:54:36.760Zfe80::4cc5:1bff:fe7e:fa68 -> 4e:c5:1b:7e:fa:68
7592023-11-20T20:54:36.784Zfe80::8028:4cff:fe23:f12e -> 82:28:4c:23:f1:2e
7602023-11-20T20:54:36.808Znat v4:
7612023-11-20T20:54:36.831Znat_v6:
7622023-11-20T20:54:36.855Zport_mac:
7632023-11-20T20:54:36.881Z1000: a8:40:25:04:0e:d3
7642023-11-20T20:54:36.905Z1: a8:40:25:04:0e:d4
7652023-11-20T20:54:36.947Zproxy arp:
7662023-11-20T20:54:36.971Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7672023-11-20T20:54:36.997Z+ retry=0
7682023-11-20T20:54:37.026Z+++ zoneadm list -n
7692023-11-20T20:54:37.053Z+++ grep oxz_ntp
7702023-11-20T20:54:37.080Z++ pfexec svcs -z -Hostate oxide/ntp
7712023-11-20T20:54:37.104Zsvcs: zone '-Hostate': entity not found
7722023-11-20T20:54:37.128Z++ true
7732023-11-20T20:54:37.152Z+ [[ '' != online ]]
7742023-11-20T20:54:37.176Z+ [[ 0 -gt 60 ]]
7752023-11-20T20:54:37.200Z+ sleep 1
7762023-11-20T20:54:37.306Z+ retry=1
7772023-11-20T20:54:37.334Z+++ zoneadm list -n
7782023-11-20T20:54:37.359Z+++ grep oxz_ntp
7792023-11-20T20:54:37.383Z++ pfexec svcs -z -Hostate oxide/ntp
7802023-11-20T20:54:37.407Zsvcs: zone '-Hostate': entity not found
7812023-11-20T20:54:37.431Z++ true
7822023-11-20T20:54:37.455Z+ [[ '' != online ]]
7832023-11-20T20:54:37.483Z+ [[ 1 -gt 60 ]]
7842023-11-20T20:54:37.515Z+ sleep 1
7852023-11-20T20:54:38.332Z+ retry=2
7862023-11-20T20:54:38.358Z+++ zoneadm list -n
7872023-11-20T20:54:38.382Z+++ grep oxz_ntp
7882023-11-20T20:54:38.406Z++ pfexec svcs -z oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314 -Hostate oxide/ntp
7892023-11-20T20:54:38.429Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7902023-11-20T20:54:38.453Z++ true
7912023-11-20T20:54:38.480Z+ [[ '' != online ]]
7922023-11-20T20:54:38.505Z+ [[ 2 -gt 60 ]]
7932023-11-20T20:54:38.529Z+ sleep 1
7942023-11-20T20:54:39.362Z+ retry=3
7952023-11-20T20:54:39.386Z+++ zoneadm list -n
7962023-11-20T20:54:39.817Z+++ grep oxz_ntp
7972023-11-20T20:54:39.866Z++ pfexec svcs -z oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314 -Hostate oxide/ntp
7982023-11-20T20:54:39.890Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7992023-11-20T20:54:39.914Z++ true
8002023-11-20T20:54:39.938Z+ [[ '' != online ]]
8012023-11-20T20:54:39.962Z+ [[ 3 -gt 60 ]]
8022023-11-20T20:54:39.985Z+ sleep 1
8032023-11-20T20:54:40.392Z+ retry=4
8042023-11-20T20:54:40.416Z+++ zoneadm list -n
8052023-11-20T20:54:40.440Z+++ grep oxz_ntp
8062023-11-20T20:54:40.465Z++ pfexec svcs -z oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314 -Hostate oxide/ntp
8072023-11-20T20:54:40.489Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8082023-11-20T20:54:40.513Z++ true
8092023-11-20T20:54:40.537Z+ [[ '' != online ]]
8102023-11-20T20:54:40.561Z+ [[ 4 -gt 60 ]]
8112023-11-20T20:54:40.585Z+ sleep 1
8122023-11-20T20:54:41.424Z+ retry=5
8132023-11-20T20:54:41.452Z+++ zoneadm list -n
8142023-11-20T20:54:41.476Z+++ grep oxz_ntp
8152023-11-20T20:54:41.500Z++ pfexec svcs -z oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314 -Hostate oxide/ntp
8162023-11-20T20:54:41.524Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8172023-11-20T20:54:41.548Z++ true
8182023-11-20T20:54:41.572Z+ [[ '' != online ]]
8192023-11-20T20:54:41.596Z+ [[ 5 -gt 60 ]]
8202023-11-20T20:54:41.622Z+ sleep 1
8212023-11-20T20:54:42.452Z+ retry=6
8222023-11-20T20:54:42.482Z+++ zoneadm list -n
8232023-11-20T20:54:42.511Z+++ grep oxz_ntp
8242023-11-20T20:54:42.946Z++ pfexec svcs -z oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314 -Hostate oxide/ntp
8252023-11-20T20:54:42.972Z+ [[ disabled != online ]]
8262023-11-20T20:54:42.996Z+ [[ 6 -gt 60 ]]
8272023-11-20T20:54:43.020Z+ sleep 1
8282023-11-20T20:54:43.475Z+ retry=7
8292023-11-20T20:54:43.499Z+++ zoneadm list -n
8302023-11-20T20:54:43.523Z+++ grep oxz_ntp
8312023-11-20T20:54:43.547Z++ pfexec svcs -z oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314 -Hostate oxide/ntp
8322023-11-20T20:54:43.571Z+ [[ disabled != online ]]
8332023-11-20T20:54:43.595Z+ [[ 7 -gt 60 ]]
8342023-11-20T20:54:44.026Z+ sleep 1
8352023-11-20T20:54:44.499Z+ retry=8
8362023-11-20T20:54:44.533Z+++ zoneadm list -n
8372023-11-20T20:54:45.015Z+++ grep oxz_ntp
8382023-11-20T20:54:45.161Z++ pfexec svcs -z oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314 -Hostate oxide/ntp
8392023-11-20T20:54:45.185Z+ [[ disabled != online ]]
8402023-11-20T20:54:45.223Z+ [[ 8 -gt 60 ]]
8412023-11-20T20:54:45.247Z+ sleep 1
8422023-11-20T20:54:45.529Z+ retry=9
8432023-11-20T20:54:45.561Z+++ zoneadm list -n
8442023-11-20T20:54:45.588Z+++ grep oxz_ntp
8452023-11-20T20:54:45.615Z++ pfexec svcs -z oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314 -Hostate oxide/ntp
8462023-11-20T20:54:45.639Z+ [[ disabled != online ]]
8472023-11-20T20:54:45.663Z+ [[ 9 -gt 60 ]]
8482023-11-20T20:54:45.688Z+ sleep 1
8492023-11-20T20:54:46.551Z+ retry=10
8502023-11-20T20:54:46.576Z+++ zoneadm list -n
8512023-11-20T20:54:46.600Z+++ grep oxz_ntp
8522023-11-20T20:54:46.624Z++ pfexec svcs -z oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314 -Hostate oxide/ntp
8532023-11-20T20:54:46.648Z+ [[ disabled != online ]]
8542023-11-20T20:54:46.672Z+ [[ 10 -gt 60 ]]
8552023-11-20T20:54:46.696Z+ sleep 1
8562023-11-20T20:54:47.574Z+ retry=11
8572023-11-20T20:54:47.598Z+++ zoneadm list -n
8582023-11-20T20:54:47.625Z+++ grep oxz_ntp
8592023-11-20T20:54:47.654Z++ pfexec svcs -z oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314 -Hostate oxide/ntp
8602023-11-20T20:54:47.682Z+ [[ disabled != online ]]
8612023-11-20T20:54:47.708Z+ [[ 11 -gt 60 ]]
8622023-11-20T20:54:47.732Z+ sleep 1
8632023-11-20T20:54:48.596Z+ retry=12
8642023-11-20T20:54:48.620Z+++ zoneadm list -n
8652023-11-20T20:54:48.645Z+++ grep oxz_ntp
8662023-11-20T20:54:48.669Z++ pfexec svcs -z oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314 -Hostate oxide/ntp
8672023-11-20T20:54:48.694Z+ [[ disabled != online ]]
8682023-11-20T20:54:48.718Z+ [[ 12 -gt 60 ]]
8692023-11-20T20:54:48.742Z+ sleep 1
 
8702023-11-20T20:54:49.395Zsock console login: Nov 20 20:54:48 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8712023-11-20T20:54:49.626Z+ retry=13
8722023-11-20T20:54:49.671Z+++ zoneadm list -n
8732023-11-20T20:54:49.701Z+++ grep oxz_ntp
8742023-11-20T20:54:49.743Z++ pfexec svcs -z oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314 -Hostate oxide/ntp
8752023-11-20T20:54:49.781Z+ [[ disabled != online ]]
8762023-11-20T20:54:49.854Z+ [[ 13 -gt 60 ]]
8772023-11-20T20:54:49.920Z+ sleep 1
8782023-11-20T20:55:02.784Z+ retry=14
8792023-11-20T20:55:03.053Z+++ zoneadm list -n
8802023-11-20T20:55:03.593Z+++ grep oxz_ntp
8812023-11-20T20:55:03.654Z++ pfexec svcs -z oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314 -Hostate oxide/ntp
8822023-11-20T20:55:03.751Z+ [[ online != online ]]
8832023-11-20T20:55:03.827Z+ echo 'Waited for chrony: 14s'
8842023-11-20T20:55:04.137Z+ export RUST_BACKTRACE=1
8852023-11-20T20:55:04.679Z+ RUST_BACKTRACE=1
8862023-11-20T20:55:04.737ZWaited for chrony: 14s
8872023-11-20T20:55:04.881Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8882023-11-20T20:55:05.002Z+ ./tests/bootstrap
8892023-11-20T20:55:05.049ZUsing external DNS server at 10.151.2.164:53
8902023-11-20T20:55:05.159Z2023-11-20 20:54:50.580089114 UTC: attempting to log into API
8912023-11-20T20:55:05.727Z2023-11-20 20:55:05.646975917 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8922023-11-20T20:55:06.742Z2023-11-20 20:55:06.651638433 UTC: attempting to log into API
8932023-11-20T20:55:21.877Z2023-11-20 20:55:21.727431736 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8942023-11-20T20:55:22.799Z2023-11-20 20:55:22.731093989 UTC: attempting to log into API
8952023-11-20T20:55:37.797Z2023-11-20 20:55:37.792493453 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8962023-11-20T20:55:38.803Z2023-11-20 20:55:38.793720271 UTC: attempting to log into API
8972023-11-20T20:55:53.833Z2023-11-20 20:55:53.830207565 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8982023-11-20T20:55:56.544Z2023-11-20 20:55:54.831247320 UTC: attempting to log into API
8992023-11-20T20:56:09.874Z2023-11-20 20:56:09.865177734 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9002023-11-20T20:56:10.875Z2023-11-20 20:56:10.865627993 UTC: attempting to log into API
 
9012023-11-20T20:56:16.350ZNov 20 20:56:15 sok ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9022023-11-20T20:56:22.668ZNov 20 20:56:21 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9032023-11-20T20:56:25.925Z2023-11-20 20:56:25.914328393 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9042023-11-20T20:56:26.926Z2023-11-20 20:56:26.915275308 UTC: attempting to log into API
 
9052023-11-20T20:56:28.219ZNov 20 20:56:27 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9062023-11-20T20:56:35.356ZNov 20 20:56:34 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
9072023-11-20T20:56:44.630ZNov 20 20:56:41 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9082023-11-20T20:56:44.650Z2023-11-20 20:56:41.948290493 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9092023-11-20T20:56:44.769Z2023-11-20 20:56:42.949435922 UTC: attempting to log into API
9102023-11-20T20:56:58.435Z2023-11-20 20:56:57.990356228 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9112023-11-20T20:56:59.002Z2023-11-20 20:56:58.991455236 UTC: attempting to log into API
9122023-11-20T20:57:04.826Z2023-11-20 20:57:04.813804211 UTC: login succeeded
9132023-11-20T20:57:04.857Zwaiting for nexus to come up...
9142023-11-20T20:57:04.925Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9152023-11-20T20:57:04.986Zensuring datasets are ready...
9162023-11-20T20:57:08.098Zlet's roll.
9172023-11-20T20:57:08.123Z+ rm ./tests/bootstrap
9182023-11-20T20:57:08.162Z+ for test_bin in tests/*
9192023-11-20T20:57:08.186Z+ ./tests/bootstrap-4172957c237fed9a
9202023-11-20T20:57:08.211Z
9212023-11-20T20:57:08.235Zrunning 0 tests
9222023-11-20T20:57:08.260Z
9232023-11-20T20:57:08.284Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9242023-11-20T20:57:08.309Z
9252023-11-20T20:57:08.334Z+ for test_bin in tests/*
9262023-11-20T20:57:08.358Z+ ./tests/end_to_end_tests-8b2150e0e5683f6a
9272023-11-20T20:57:08.382Z
9282023-11-20T20:57:08.407Zrunning 2 tests
9292023-11-20T20:57:08.431Ztest no_spoof::no_spoof_here ... ok
9302023-11-20T20:58:08.187Ztest instance_launch::instance_launch has been running for over 60 seconds
9312023-11-20T21:00:01.546Ztest instance_launch::instance_launch ... ok
9322023-11-20T21:00:01.580Z
9332023-11-20T21:00:01.608Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 173.35s
9342023-11-20T21:00:01.634Z
9352023-11-20T21:00:01.664Z+ _exit_trap
9362023-11-20T21:00:01.692Z+ local status=0
9372023-11-20T21:00:01.727Z+ [[ 0 -eq 0 ]]
9382023-11-20T21:00:01.761Z+ exit 0
9392023-11-20T21:00:01.787Zprocess exited: duration 391054 ms, exit code 0
 
9402023-11-20T21:00:01.870Zfound 75 output files
9412023-11-20T21:00:01.943Zuploading: /var/svc/log/oxide-sled-agent:default.log (724047 bytes)
9422023-11-20T21:00:03.200Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9432023-11-20T21:00:03.254Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c61f6e61-fcaa-4182-8a7f-2863862c6752/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9442023-11-20T21:00:04.280Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c61f6e61-fcaa-4182-8a7f-2863862c6752/root/var/svc/log/oxide-cockroachdb:default.log
9452023-11-20T21:00:04.304Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c61f6e61-fcaa-4182-8a7f-2863862c6752/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
9462023-11-20T21:00:05.388Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c61f6e61-fcaa-4182-8a7f-2863862c6752/root/var/svc/log/oxide-cockroachdb:default.log.0
9472023-11-20T21:00:05.415Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9174ead9-e0b7-4d50-9156-e5aa2b1bd231/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9482023-11-20T21:00:06.476Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9174ead9-e0b7-4d50-9156-e5aa2b1bd231/root/var/svc/log/oxide-cockroachdb:default.log
9492023-11-20T21:00:06.503Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9174ead9-e0b7-4d50-9156-e5aa2b1bd231/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
9502023-11-20T21:00:07.598Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9174ead9-e0b7-4d50-9156-e5aa2b1bd231/root/var/svc/log/oxide-cockroachdb:default.log.0
9512023-11-20T21:00:07.648Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2b32d8b8-9e9a-4439-84e5-d530cb9d5daf/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9522023-11-20T21:00:09.103Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2b32d8b8-9e9a-4439-84e5-d530cb9d5daf/root/var/svc/log/oxide-crucible-agent:default.log
9532023-11-20T21:00:09.140Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2b32d8b8-9e9a-4439-84e5-d530cb9d5daf/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
9542023-11-20T21:00:10.641Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2b32d8b8-9e9a-4439-84e5-d530cb9d5daf/root/var/svc/log/oxide-crucible-agent:default.log.0
9552023-11-20T21:00:10.666Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2b32d8b8-9e9a-4439-84e5-d530cb9d5daf/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9562023-11-20T21:00:11.309Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2b32d8b8-9e9a-4439-84e5-d530cb9d5daf/root/var/svc/log/oxide-crucible-downstairs:default.log
9572023-11-20T21:00:11.335Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2b32d8b8-9e9a-4439-84e5-d530cb9d5daf/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9582023-11-20T21:00:12.408Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2b32d8b8-9e9a-4439-84e5-d530cb9d5daf/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9592023-11-20T21:00:12.437Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87a4991a-1a01-44a9-9bdd-bab748474e4d/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9602023-11-20T21:00:13.486Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87a4991a-1a01-44a9-9bdd-bab748474e4d/root/var/svc/log/oxide-crucible-agent:default.log
9612023-11-20T21:00:13.515Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87a4991a-1a01-44a9-9bdd-bab748474e4d/root/var/svc/log/oxide-crucible-agent:default.log.0 (14148 bytes)
9622023-11-20T21:00:14.619Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87a4991a-1a01-44a9-9bdd-bab748474e4d/root/var/svc/log/oxide-crucible-agent:default.log.0
9632023-11-20T21:00:14.655Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87a4991a-1a01-44a9-9bdd-bab748474e4d/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9642023-11-20T21:00:15.723Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87a4991a-1a01-44a9-9bdd-bab748474e4d/root/var/svc/log/oxide-crucible-downstairs:default.log
9652023-11-20T21:00:15.762Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87a4991a-1a01-44a9-9bdd-bab748474e4d/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9662023-11-20T21:00:16.847Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87a4991a-1a01-44a9-9bdd-bab748474e4d/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9672023-11-20T21:00:16.875Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87a4991a-1a01-44a9-9bdd-bab748474e4d/root/var/svc/log/oxide-crucible-downstairs:downstairs-e2dcbb8a-673e-4877-a609-db86da50c935.log (3049 bytes)
9682023-11-20T21:00:16.902Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87a4991a-1a01-44a9-9bdd-bab748474e4d/root/var/svc/log/oxide-crucible-downstairs:downstairs-e2dcbb8a-673e-4877-a609-db86da50c935.log" changed size mid upload: 3049 -> 0
9692023-11-20T21:00:17.951Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87a4991a-1a01-44a9-9bdd-bab748474e4d/root/var/svc/log/oxide-crucible-downstairs:downstairs-e2dcbb8a-673e-4877-a609-db86da50c935.log
9702023-11-20T21:00:17.989Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_c679ce2c-3a95-4d47-a8b6-8a83b3ecd036/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9712023-11-20T21:00:19.449Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_c679ce2c-3a95-4d47-a8b6-8a83b3ecd036/root/var/svc/log/oxide-crucible-agent:default.log
9722023-11-20T21:00:19.477Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_c679ce2c-3a95-4d47-a8b6-8a83b3ecd036/root/var/svc/log/oxide-crucible-agent:default.log.0 (14583 bytes)
9732023-11-20T21:00:20.576Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_c679ce2c-3a95-4d47-a8b6-8a83b3ecd036/root/var/svc/log/oxide-crucible-agent:default.log.0
9742023-11-20T21:00:20.604Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_c679ce2c-3a95-4d47-a8b6-8a83b3ecd036/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9752023-11-20T21:00:20.629Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_c679ce2c-3a95-4d47-a8b6-8a83b3ecd036/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
9762023-11-20T21:00:21.680Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_c679ce2c-3a95-4d47-a8b6-8a83b3ecd036/root/var/svc/log/oxide-crucible-downstairs:default.log
9772023-11-20T21:00:21.705Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_c679ce2c-3a95-4d47-a8b6-8a83b3ecd036/root/var/svc/log/oxide-crucible-downstairs:downstairs-e39fd2dc-6421-4b0f-8bfb-845e6858fb33.log (106 bytes)
9782023-11-20T21:00:23.215Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_c679ce2c-3a95-4d47-a8b6-8a83b3ecd036/root/var/svc/log/oxide-crucible-downstairs:downstairs-e39fd2dc-6421-4b0f-8bfb-845e6858fb33.log
9792023-11-20T21:00:23.259Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_c679ce2c-3a95-4d47-a8b6-8a83b3ecd036/root/var/svc/log/oxide-crucible-downstairs:downstairs-e39fd2dc-6421-4b0f-8bfb-845e6858fb33.log.0 (5429 bytes)
9802023-11-20T21:00:24.339Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_c679ce2c-3a95-4d47-a8b6-8a83b3ecd036/root/var/svc/log/oxide-crucible-downstairs:downstairs-e39fd2dc-6421-4b0f-8bfb-845e6858fb33.log.0
9812023-11-20T21:00:24.363Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_f447998b-2000-4301-8f0a-6bf340886d01/root/var/svc/log/oxide-external_dns:default.log (0 bytes)
9822023-11-20T21:00:25.433Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_f447998b-2000-4301-8f0a-6bf340886d01/root/var/svc/log/oxide-external_dns:default.log
9832023-11-20T21:00:25.460Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_f447998b-2000-4301-8f0a-6bf340886d01/root/var/svc/log/oxide-external_dns:default.log.0 (15614 bytes)
9842023-11-20T21:00:26.571Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_f447998b-2000-4301-8f0a-6bf340886d01/root/var/svc/log/oxide-external_dns:default.log.0
9852023-11-20T21:00:26.603Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_c669b2db-48d3-4952-86df-0d183f66f1bb/root/var/svc/log/oxide-internal_dns:default.log (119047 bytes)
9862023-11-20T21:00:26.634Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_c669b2db-48d3-4952-86df-0d183f66f1bb/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 119047 -> 0
9872023-11-20T21:00:28.101Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_c669b2db-48d3-4952-86df-0d183f66f1bb/root/var/svc/log/oxide-internal_dns:default.log
9882023-11-20T21:00:28.131Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_3100eff7-ae2b-4736-9033-7f4c73f937e5/root/var/svc/log/oxide-nexus:default.log (125417 bytes)
9892023-11-20T21:00:28.614Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_3100eff7-ae2b-4736-9033-7f4c73f937e5/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 125417 -> 130795
9902023-11-20T21:00:29.406Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_3100eff7-ae2b-4736-9033-7f4c73f937e5/root/var/svc/log/oxide-nexus:default.log
9912023-11-20T21:00:29.446Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_3100eff7-ae2b-4736-9033-7f4c73f937e5/root/var/svc/log/oxide-nexus:default.log.0 (4448176 bytes)
9922023-11-20T21:00:34.485Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_3100eff7-ae2b-4736-9033-7f4c73f937e5/root/var/svc/log/oxide-nexus:default.log.0
9932023-11-20T21:00:34.516Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_8a4226ff-7035-46b2-a2fe-96e44a9821ad/root/var/svc/log/oxide-nexus:default.log (0 bytes)
9942023-11-20T21:00:34.576Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_8a4226ff-7035-46b2-a2fe-96e44a9821ad/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 0 -> 5982
9952023-11-20T21:00:35.702Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_8a4226ff-7035-46b2-a2fe-96e44a9821ad/root/var/svc/log/oxide-nexus:default.log
9962023-11-20T21:00:35.729Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_8a4226ff-7035-46b2-a2fe-96e44a9821ad/root/var/svc/log/oxide-nexus:default.log.0 (521032 bytes)
9972023-11-20T21:00:38.748Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_8a4226ff-7035-46b2-a2fe-96e44a9821ad/root/var/svc/log/oxide-nexus:default.log.0
9982023-11-20T21:00:38.780Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-agent:default.log (4787 bytes)
9992023-11-20T21:00:39.884Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-agent:default.log
10002023-11-20T21:00:39.908Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-agent:default.log.0 (20507 bytes)
10012023-11-20T21:00:41.059Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-agent:default.log.0
10022023-11-20T21:00:41.086Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10032023-11-20T21:00:42.142Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-downstairs:default.log
10042023-11-20T21:00:42.587Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10052023-11-20T21:00:43.275Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10062023-11-20T21:00:43.307Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-downstairs:downstairs-770e2bd1-7686-4c78-9f97-3da685751064.log (0 bytes)
10072023-11-20T21:00:44.369Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-downstairs:downstairs-770e2bd1-7686-4c78-9f97-3da685751064.log
10082023-11-20T21:00:44.808Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-downstairs:downstairs-770e2bd1-7686-4c78-9f97-3da685751064.log.0 (3048 bytes)
10092023-11-20T21:00:46.717Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-downstairs:downstairs-770e2bd1-7686-4c78-9f97-3da685751064.log.0
10102023-11-20T21:00:46.757Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-downstairs:downstairs-bc2a20ed-614b-4656-8321-f85c6dcb8f29.log (106 bytes)
10112023-11-20T21:00:47.422Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-downstairs:downstairs-bc2a20ed-614b-4656-8321-f85c6dcb8f29.log
10122023-11-20T21:00:47.452Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-downstairs:downstairs-bc2a20ed-614b-4656-8321-f85c6dcb8f29.log.0 (5427 bytes)
10132023-11-20T21:00:48.533Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_87e38475-1df0-485a-8add-127ba077608a/root/var/svc/log/oxide-crucible-downstairs:downstairs-bc2a20ed-614b-4656-8321-f85c6dcb8f29.log.0
10142023-11-20T21:00:48.563Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8cf6c575-17e9-4c36-bef6-84a58930fb71/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10152023-11-20T21:00:49.641Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8cf6c575-17e9-4c36-bef6-84a58930fb71/root/var/svc/log/oxide-crucible-agent:default.log
10162023-11-20T21:00:49.672Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8cf6c575-17e9-4c36-bef6-84a58930fb71/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10172023-11-20T21:00:50.737Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8cf6c575-17e9-4c36-bef6-84a58930fb71/root/var/svc/log/oxide-crucible-agent:default.log.0
10182023-11-20T21:00:50.763Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8cf6c575-17e9-4c36-bef6-84a58930fb71/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10192023-11-20T21:00:51.827Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8cf6c575-17e9-4c36-bef6-84a58930fb71/root/var/svc/log/oxide-crucible-downstairs:default.log
10202023-11-20T21:00:51.872Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8cf6c575-17e9-4c36-bef6-84a58930fb71/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10212023-11-20T21:00:52.964Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8cf6c575-17e9-4c36-bef6-84a58930fb71/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10222023-11-20T21:00:52.988Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dc6ae56c-6dfd-436e-9647-8a5afecf94ec/root/var/svc/log/oxide-crucible-agent:default.log (14580 bytes)
10232023-11-20T21:00:53.014Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dc6ae56c-6dfd-436e-9647-8a5afecf94ec/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 14580 -> 0
10242023-11-20T21:00:54.067Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dc6ae56c-6dfd-436e-9647-8a5afecf94ec/root/var/svc/log/oxide-crucible-agent:default.log
10252023-11-20T21:00:54.096Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dc6ae56c-6dfd-436e-9647-8a5afecf94ec/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10262023-11-20T21:00:54.122Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dc6ae56c-6dfd-436e-9647-8a5afecf94ec/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10272023-11-20T21:00:55.167Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dc6ae56c-6dfd-436e-9647-8a5afecf94ec/root/var/svc/log/oxide-crucible-downstairs:default.log
10282023-11-20T21:00:55.197Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dc6ae56c-6dfd-436e-9647-8a5afecf94ec/root/var/svc/log/oxide-crucible-downstairs:downstairs-6e604edd-3942-4935-b373-7a0d51b52894.log (5534 bytes)
10292023-11-20T21:00:55.238Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dc6ae56c-6dfd-436e-9647-8a5afecf94ec/root/var/svc/log/oxide-crucible-downstairs:downstairs-6e604edd-3942-4935-b373-7a0d51b52894.log" changed size mid upload: 5534 -> 0
10302023-11-20T21:00:56.295Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dc6ae56c-6dfd-436e-9647-8a5afecf94ec/root/var/svc/log/oxide-crucible-downstairs:downstairs-6e604edd-3942-4935-b373-7a0d51b52894.log
10312023-11-20T21:00:56.326Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314/root/var/svc/log/oxide-ntp:default.log (0 bytes)
10322023-11-20T21:00:57.374Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314/root/var/svc/log/oxide-ntp:default.log
10332023-11-20T21:00:57.400Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314/root/var/svc/log/oxide-ntp:default.log.0 (914 bytes)
10342023-11-20T21:00:58.478Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314/root/var/svc/log/oxide-ntp:default.log.0
10352023-11-20T21:00:58.507Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_2e3cf8e5-b881-40ce-ba7c-c56bccb4b85f/root/var/svc/log/oxide-clickhouse:default.log (524197 bytes)
10362023-11-20T21:00:58.800Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_2e3cf8e5-b881-40ce-ba7c-c56bccb4b85f/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 524197 -> 106947
10372023-11-20T21:00:59.884Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_2e3cf8e5-b881-40ce-ba7c-c56bccb4b85f/root/var/svc/log/oxide-clickhouse:default.log
10382023-11-20T21:00:59.909Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_cc61c5a8-bec4-4772-83e6-f18f039c884c/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10392023-11-20T21:01:00.973Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_cc61c5a8-bec4-4772-83e6-f18f039c884c/root/var/svc/log/oxide-crucible-pantry:default.log
10402023-11-20T21:01:01.009Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_cc61c5a8-bec4-4772-83e6-f18f039c884c/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2337 bytes)
10412023-11-20T21:01:02.113Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_cc61c5a8-bec4-4772-83e6-f18f039c884c/root/var/svc/log/oxide-crucible-pantry:default.log.0
10422023-11-20T21:01:02.558Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_66d54510-87a9-4f5c-8c9f-38883a39d417/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10432023-11-20T21:01:03.215Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_66d54510-87a9-4f5c-8c9f-38883a39d417/root/var/svc/log/oxide-crucible-agent:default.log
10442023-11-20T21:01:03.254Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_66d54510-87a9-4f5c-8c9f-38883a39d417/root/var/svc/log/oxide-crucible-agent:default.log.0 (14576 bytes)
10452023-11-20T21:01:04.814Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_66d54510-87a9-4f5c-8c9f-38883a39d417/root/var/svc/log/oxide-crucible-agent:default.log.0
10462023-11-20T21:01:04.840Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_66d54510-87a9-4f5c-8c9f-38883a39d417/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10472023-11-20T21:01:05.898Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_66d54510-87a9-4f5c-8c9f-38883a39d417/root/var/svc/log/oxide-crucible-downstairs:default.log
10482023-11-20T21:01:05.926Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_66d54510-87a9-4f5c-8c9f-38883a39d417/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10492023-11-20T21:01:07.042Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_66d54510-87a9-4f5c-8c9f-38883a39d417/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10502023-11-20T21:01:07.066Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_66d54510-87a9-4f5c-8c9f-38883a39d417/root/var/svc/log/oxide-crucible-downstairs:downstairs-9eb4613c-e521-4576-823e-13e51723938a.log (0 bytes)
10512023-11-20T21:01:08.124Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_66d54510-87a9-4f5c-8c9f-38883a39d417/root/var/svc/log/oxide-crucible-downstairs:downstairs-9eb4613c-e521-4576-823e-13e51723938a.log
10522023-11-20T21:01:08.148Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_66d54510-87a9-4f5c-8c9f-38883a39d417/root/var/svc/log/oxide-crucible-downstairs:downstairs-9eb4613c-e521-4576-823e-13e51723938a.log.0 (3048 bytes)
10532023-11-20T21:01:09.224Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_66d54510-87a9-4f5c-8c9f-38883a39d417/root/var/svc/log/oxide-crucible-downstairs:downstairs-9eb4613c-e521-4576-823e-13e51723938a.log.0
10542023-11-20T21:01:09.250Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_88b80f39-95d5-4f02-b3bc-69b5add8fe48/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10552023-11-20T21:01:10.302Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_88b80f39-95d5-4f02-b3bc-69b5add8fe48/root/var/svc/log/oxide-crucible-pantry:default.log
10562023-11-20T21:01:10.340Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_88b80f39-95d5-4f02-b3bc-69b5add8fe48/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10572023-11-20T21:01:11.427Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_88b80f39-95d5-4f02-b3bc-69b5add8fe48/root/var/svc/log/oxide-crucible-pantry:default.log.0
10582023-11-20T21:01:11.451Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_c9863ed0-3ce3-4f94-a75f-8aafdd12a4a5/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10592023-11-20T21:01:12.506Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_c9863ed0-3ce3-4f94-a75f-8aafdd12a4a5/root/var/svc/log/oxide-crucible-pantry:default.log
10602023-11-20T21:01:12.535Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_c9863ed0-3ce3-4f94-a75f-8aafdd12a4a5/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
10612023-11-20T21:01:13.620Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_c9863ed0-3ce3-4f94-a75f-8aafdd12a4a5/root/var/svc/log/oxide-crucible-pantry:default.log.0
10622023-11-20T21:01:13.654Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_70cdd123-34e9-41d6-88b9-04d1a33ff8a9/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10632023-11-20T21:01:14.713Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_70cdd123-34e9-41d6-88b9-04d1a33ff8a9/root/var/svc/log/oxide-cockroachdb:default.log
10642023-11-20T21:01:14.745Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_70cdd123-34e9-41d6-88b9-04d1a33ff8a9/root/var/svc/log/oxide-cockroachdb:default.log.0 (5367 bytes)
10652023-11-20T21:01:15.836Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_70cdd123-34e9-41d6-88b9-04d1a33ff8a9/root/var/svc/log/oxide-cockroachdb:default.log.0
10662023-11-20T21:01:15.869Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_c4d58369-28ea-48a5-8fee-a43680ec7275/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10672023-11-20T21:01:15.901Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_c4d58369-28ea-48a5-8fee-a43680ec7275/root/var/svc/log/oxide-cockroachdb:default.log" changed size mid upload: 5368 -> 0
10682023-11-20T21:01:16.952Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_c4d58369-28ea-48a5-8fee-a43680ec7275/root/var/svc/log/oxide-cockroachdb:default.log
10692023-11-20T21:01:16.976Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7dcc7f65-3ed2-4b42-a050-7f163f6fbce8/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
10702023-11-20T21:01:17.005Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7dcc7f65-3ed2-4b42-a050-7f163f6fbce8/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 3434 -> 0
10712023-11-20T21:01:18.057Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7dcc7f65-3ed2-4b42-a050-7f163f6fbce8/root/var/svc/log/oxide-crucible-agent:default.log
10722023-11-20T21:01:18.085Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7dcc7f65-3ed2-4b42-a050-7f163f6fbce8/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10732023-11-20T21:01:19.640Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7dcc7f65-3ed2-4b42-a050-7f163f6fbce8/root/var/svc/log/oxide-crucible-downstairs:default.log
10742023-11-20T21:01:19.668Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7dcc7f65-3ed2-4b42-a050-7f163f6fbce8/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10752023-11-20T21:01:20.736Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7dcc7f65-3ed2-4b42-a050-7f163f6fbce8/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10762023-11-20T21:01:21.183Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_36ecfb77-6aaf-46ae-b72e-cc0b6bba4f82/root/var/svc/log/oxide-internal_dns:default.log (116205 bytes)
10772023-11-20T21:01:21.215Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_36ecfb77-6aaf-46ae-b72e-cc0b6bba4f82/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 116205 -> 3755
10782023-11-20T21:01:21.931Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_36ecfb77-6aaf-46ae-b72e-cc0b6bba4f82/root/var/svc/log/oxide-internal_dns:default.log
10792023-11-20T21:01:21.956Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_be2967a7-4a15-47f8-ad2e-30bee978d281/root/var/svc/log/oxide-internal_dns:default.log (0 bytes)
10802023-11-20T21:01:21.981Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_be2967a7-4a15-47f8-ad2e-30bee978d281/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 0 -> 3756
10812023-11-20T21:01:23.076Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_be2967a7-4a15-47f8-ad2e-30bee978d281/root/var/svc/log/oxide-internal_dns:default.log
10822023-11-20T21:01:23.523Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_be2967a7-4a15-47f8-ad2e-30bee978d281/root/var/svc/log/oxide-internal_dns:default.log.0 (114688 bytes)
10832023-11-20T21:01:24.948Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_be2967a7-4a15-47f8-ad2e-30bee978d281/root/var/svc/log/oxide-internal_dns:default.log.0
10842023-11-20T21:01:24.981Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_59876eb1-8671-4aed-9ae9-d5d131357486/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10852023-11-20T21:01:26.024Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_59876eb1-8671-4aed-9ae9-d5d131357486/root/var/svc/log/oxide-cockroachdb:default.log
10862023-11-20T21:01:26.049Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_59876eb1-8671-4aed-9ae9-d5d131357486/root/var/svc/log/oxide-cockroachdb:default.log.0 (5341 bytes)
10872023-11-20T21:01:27.958Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_59876eb1-8671-4aed-9ae9-d5d131357486/root/var/svc/log/oxide-cockroachdb:default.log.0
10882023-11-20T21:01:27.997Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba04375d-4bf7-4b3a-a24f-99f782ca9245/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
10892023-11-20T21:01:28.025Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba04375d-4bf7-4b3a-a24f-99f782ca9245/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 3434 -> 0
10902023-11-20T21:01:29.078Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba04375d-4bf7-4b3a-a24f-99f782ca9245/root/var/svc/log/oxide-crucible-agent:default.log
10912023-11-20T21:01:29.103Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba04375d-4bf7-4b3a-a24f-99f782ca9245/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10922023-11-20T21:01:29.128Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba04375d-4bf7-4b3a-a24f-99f782ca9245/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10932023-11-20T21:01:30.175Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ba04375d-4bf7-4b3a-a24f-99f782ca9245/root/var/svc/log/oxide-crucible-downstairs:default.log
10942023-11-20T21:01:30.204Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4a8d88fa-7413-499b-b0bd-7b7cf5fc40c6/root/var/svc/log/oxide-external_dns:default.log (15613 bytes)
10952023-11-20T21:01:30.234Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4a8d88fa-7413-499b-b0bd-7b7cf5fc40c6/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15613 -> 3778
10962023-11-20T21:01:31.323Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4a8d88fa-7413-499b-b0bd-7b7cf5fc40c6/root/var/svc/log/oxide-external_dns:default.log
10972023-11-20T21:01:31.349Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8d52edca-c939-45c3-84a7-be05b3ae6fd3/root/var/svc/log/oxide-nexus:default.log (654751 bytes)
10982023-11-20T21:01:31.551Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8d52edca-c939-45c3-84a7-be05b3ae6fd3/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 654751 -> 74315
10992023-11-20T21:01:32.623Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8d52edca-c939-45c3-84a7-be05b3ae6fd3/root/var/svc/log/oxide-nexus:default.log
11002023-11-20T21:01:32.650Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_feeb3f33-522e-4ca6-88ae-fbee605103de/root/var/svc/log/oxide-oximeter:default.log (148452 bytes)
11012023-11-20T21:01:32.807Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_feeb3f33-522e-4ca6-88ae-fbee605103de/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 148452 -> 49622
11022023-11-20T21:01:33.886Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_feeb3f33-522e-4ca6-88ae-fbee605103de/root/var/svc/log/oxide-oximeter:default.log
11032023-11-20T21:01:33.914Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314/root/var/log/chrony/measurements.log (7946 bytes)
11042023-11-20T21:01:33.961Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314/root/var/log/chrony/measurements.log" changed size mid upload: 7946 -> 10549
11052023-11-20T21:01:35.052Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314/root/var/log/chrony/measurements.log
11062023-11-20T21:01:35.089Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314/root/var/log/chrony/statistics.log (4914 bytes)
11072023-11-20T21:01:35.749Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314/root/var/log/chrony/statistics.log" changed size mid upload: 4914 -> 6786
11082023-11-20T21:01:36.836Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314/root/var/log/chrony/statistics.log
11092023-11-20T21:01:36.862Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314/root/var/log/chrony/tracking.log (1454 bytes)
11102023-11-20T21:01:36.888Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314/root/var/log/chrony/tracking.log" changed size mid upload: 1454 -> 1718
11112023-11-20T21:01:37.962Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_74eee2ff-d098-4d1e-aa42-c99ffed59314/root/var/log/chrony/tracking.log