01HH1HBN82C5KZQ1JENHRP4ENX: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.27",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HH1HCS4RRNPG7TR6N0P3F1FP

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-07T07:51:54.619Zjob dependencies complete; ready to run (waiting for 48 m 44 s)
22023-12-07T08:14:09.508Zjob assigned to worker 01HH1N58XEWC04G6X2Q2CPHF44 (queued for 22 m 14 s)
32023-12-07T08:14:09.687Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-12-07T08:14:09.735Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-12-07T08:14:09.768Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-12-07T08:14:09.824Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-07T08:14:09.865Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-12-07T08:14:09.903ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-07T08:14:09.939Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-07T08:14:09.979ZOxide Helios Version helios-2.0.22333 64-bit
112023-12-07T08:14:10.015ZConfiguring devices.
122023-12-07T08:14:10.050Z##################################################
132023-12-07T08:14:10.095Z#### ############# ##
142023-12-07T08:14:10.146Z### ### ############ ##
152023-12-07T08:14:10.183Z## ### # ## ### ## Oxide Computer Company ##
162023-12-07T08:14:10.223Z## ## ## ### # ### ##
172023-12-07T08:14:10.260Z## # ### #### #### This Station Under ##
182023-12-07T08:14:10.295Z### ### #### # ### Computer Control ##
192023-12-07T08:14:10.335Z#### #### ### ## ##
202023-12-07T08:14:10.372Z##################################################
212023-12-07T08:14:10.405ZHostname: buskin
222023-12-07T08:14:10.487ZLoading smf(7) service descriptions:
232023-12-07T08:14:11.009Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-4fe365f0266bec52.gz
242023-12-07T08:14:13.078Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-4fe365f0266bec52.gz
252023-12-07T08:14:13.078Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-07T08:14:46.693Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-07T08:14:46.693Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-de860e1ecc3158ae.gz
282023-12-07T08:15:07.029Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-de860e1ecc3158ae.gz
292023-12-07T08:15:07.029Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-07T08:15:08.183Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-07T08:15:08.184Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-07T08:15:11.031Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-07T08:15:11.059Zdownloading input: /input/package/work/version.txt
342023-12-07T08:15:11.104Zdownloaded input: /input/package/work/version.txt
352023-12-07T08:15:11.129Zdownloading input: /input/package/work/package.tar.gz
362023-12-07T08:15:26.787Zdownloaded input: /input/package/work/package.tar.gz
372023-12-07T08:15:26.787Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-07T08:15:48.765Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-07T08:15:48.765Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-07T08:15:55.469Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-07T08:15:55.500Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-07T08:16:28.927Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-07T08:16:28.927Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-07T08:17:04.683Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-07T08:17:04.893Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-07T08:17:27.239Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-07T08:17:27.239Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-07T08:17:31.563Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-07T08:17:31.563Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-07T08:17:39.327Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-07T08:17:39.327Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-07T08:17:43.944Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-07T08:17:43.944Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-07T08:17:48.434Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-07T08:17:48.434Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-07T08:17:50.501Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-07T08:17:50.502Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-07T08:17:53.866Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-07T08:17:53.866Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-07T08:17:57.292Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-07T08:17:57.292Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-07T08:18:10.006Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-07T08:18:10.006Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-07T08:18:23.154Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-07T08:18:23.155Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-07T08:18:23.246Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-07T08:18:23.273Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-07T08:18:29.259Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-07T08:18:29.259Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-07T08:18:41.246Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-07T08:18:41.246Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-07T08:19:25.679Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-07T08:19:25.679Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-07T08:19:57.306Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-07T08:19:57.331Zstarting task 0: "setup"
762023-12-07T08:19:57.331Z++ uname -s
772023-12-07T08:19:57.331Z+ kern=SunOS
782023-12-07T08:19:57.332Z+ case "$kern" in
792023-12-07T08:19:57.332Z+ groupadd -g 12345 build
802023-12-07T08:19:57.332Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-07T08:19:59.333Z+ zfs create -o mountpoint=/work rpool/work
822023-12-07T08:19:59.365Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-07T08:19:59.365Z+ home_fs=zfs
842023-12-07T08:19:59.365Z+ [[ zfs == autofs ]]
852023-12-07T08:19:59.366Z+ mkdir -p /home/build
862023-12-07T08:19:59.366Z+ chown build:build /home/build /work
872023-12-07T08:20:01.353Z+ chmod 0700 /home/build /work
882023-12-07T08:20:01.379Zprocess exited: duration 4051 ms, exit code 0
 
892023-12-07T08:20:01.431Zstarting task 1: "authentication"
902023-12-07T08:20:01.481Zprocess exited: duration 47 ms, exit code 0
 
912023-12-07T08:20:01.549Zstarting task 2: "build"
922023-12-07T08:20:01.578Z+ trap _exit_trap EXIT
932023-12-07T08:20:01.579Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-07T08:20:01.745Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-07T08:20:01.746Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-07T08:20:01.746Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-07T08:20:01.746Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-07T08:20:01.746Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-07T08:20:01.746Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-07T08:20:01.746Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-07T08:20:01.772Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-07T08:20:01.772Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-07T08:20:01.772Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-07T08:20:01.772Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-07T08:20:01.772Z+ [[ -d /opt/oxide ]]
1062023-12-07T08:20:01.772Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-07T08:20:01.772Z+ pfexec mkdir /opt/oxide
1082023-12-07T08:20:01.772Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-07T08:20:01.772Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-07T08:20:01.773Z+ cd /opt/oxide-underneath
1112023-12-07T08:20:01.773Z+ pfexec tar ceEp@/f - .
1122023-12-07T08:20:01.773Z+ cd /opt/oxide
1132023-12-07T08:20:01.773Z+ pfexec tar xveEp@/f -
1142023-12-07T08:20:01.875Zx ., 0 bytes, 0 tape blocks
1152023-12-07T08:20:01.875Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-07T08:20:01.875Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-07T08:20:01.875Zx ./opte/bin/opteadm, 52567720 bytes, 102672 tape blocks
1182023-12-07T08:20:01.912Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-07T08:20:01.912Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-07T08:20:01.941Z+ pfexec mkdir /opt/oxide/work
1212023-12-07T08:20:01.941Z+ pfexec chown build:build /opt/oxide/work
1222023-12-07T08:20:01.941Z+ cd /opt/oxide/work
1232023-12-07T08:20:01.941Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-07T08:20:03.451ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-12-07T08:20:03.451Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-12-07T08:20:03.451Zx out/omicron-sled-agent.tar, 111068672 bytes, 216931 tape blocks
1272023-12-07T08:20:03.451Zx out/target/test, 75 bytes, 1 tape blocks
1282023-12-07T08:20:03.451Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1292023-12-07T08:20:03.451Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-12-07T08:20:03.452Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-12-07T08:20:03.452Zx target/release/omicron-package, 42513936 bytes, 83036 tape blocks
1322023-12-07T08:20:03.452Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-12-07T08:20:03.452Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-12-07T08:20:03.453Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-12-07T08:20:03.453Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-12-07T08:20:03.453Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-12-07T08:20:03.453Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-12-07T08:20:03.453Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-12-07T08:20:03.480Z
1402023-12-07T08:20:03.480Zreal 1.534427706
1412023-12-07T08:20:03.480Zuser 1.380841918
1422023-12-07T08:20:03.480Zsys 0.270048315
1432023-12-07T08:20:03.480Ztrap 0.000066031
1442023-12-07T08:20:03.480Ztflt 0.000000000
1452023-12-07T08:20:03.480Zdflt 0.000000000
1462023-12-07T08:20:03.480Zkflt 0.000000000
1472023-12-07T08:20:03.480Zlock 0.000000000
1482023-12-07T08:20:03.481Zslp 1.382366572
1492023-12-07T08:20:03.481Zlat 0.033452942
1502023-12-07T08:20:03.481Zstop 0.000125162
1512023-12-07T08:20:03.481Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-12-07T08:20:04.528Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-12-07T08:20:04.558Z+ mkdir tests
1542023-12-07T08:20:04.558Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-12-07T08:20:04.558Z+ ptime -m gunzip
1562023-12-07T08:20:04.558Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-4fe365f0266bec52
1572023-12-07T08:20:04.686Z
1582023-12-07T08:20:04.719Zreal 0.138199356
1592023-12-07T08:20:04.720Zuser 0.127317112
1602023-12-07T08:20:04.721Zsys 0.010599067
1612023-12-07T08:20:04.721Ztrap 0.000011850
1622023-12-07T08:20:04.721Ztflt 0.000000000
1632023-12-07T08:20:04.721Zdflt 0.000000000
1642023-12-07T08:20:04.722Zkflt 0.000000000
1652023-12-07T08:20:04.722Zlock 0.000000000
1662023-12-07T08:20:04.722Zslp 0.000022680
1672023-12-07T08:20:04.722Zlat 0.000023090
1682023-12-07T08:20:04.722Zstop 0.000132632
1692023-12-07T08:20:04.723Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-4fe365f0266bec52
1702023-12-07T08:20:04.724Z+ chmod a+x tests/bootstrap-4fe365f0266bec52
1712023-12-07T08:20:04.724Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-12-07T08:20:04.724Z+ ptime -m gunzip
1732023-12-07T08:20:04.724Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-12-07T08:20:06.902Z
1752023-12-07T08:20:06.929Zreal 2.188482129
1762023-12-07T08:20:06.929Zuser 2.055657153
1772023-12-07T08:20:06.929Zsys 0.132434439
1782023-12-07T08:20:06.929Ztrap 0.000166432
1792023-12-07T08:20:06.930Ztflt 0.000000000
1802023-12-07T08:20:06.930Zdflt 0.000000000
1812023-12-07T08:20:06.930Zkflt 0.000000000
1822023-12-07T08:20:06.930Zlock 0.000000000
1832023-12-07T08:20:06.930Zslp 0.000021270
1842023-12-07T08:20:06.930Zlat 0.000020959
1852023-12-07T08:20:06.930Zstop 0.000122803
1862023-12-07T08:20:06.930Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-12-07T08:20:06.930Z+ chmod a+x tests/bootstrap
1882023-12-07T08:20:06.930Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-12-07T08:20:06.931Z+ ptime -m gunzip
1902023-12-07T08:20:06.932Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-de860e1ecc3158ae
1912023-12-07T08:20:09.168Z
1922023-12-07T08:20:09.195Zreal 2.277615805
1932023-12-07T08:20:09.195Zuser 2.140435621
1942023-12-07T08:20:09.196Zsys 0.136783656
1952023-12-07T08:20:09.196Ztrap 0.000163993
1962023-12-07T08:20:09.196Ztflt 0.000000000
1972023-12-07T08:20:09.196Zdflt 0.000000000
1982023-12-07T08:20:09.197Zkflt 0.000000000
1992023-12-07T08:20:09.197Zlock 0.000000000
2002023-12-07T08:20:09.197Zslp 0.000019400
2012023-12-07T08:20:09.197Zlat 0.000023500
2022023-12-07T08:20:09.197Zstop 0.000130812
2032023-12-07T08:20:09.198Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-de860e1ecc3158ae
2042023-12-07T08:20:09.198Z+ chmod a+x tests/end_to_end_tests-de860e1ecc3158ae
2052023-12-07T08:20:09.198Z+ bmat address ls
2062023-12-07T08:20:09.198ZNAME CIDR FIRST LAST COUNT
2072023-12-07T08:20:09.198Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-12-07T08:20:09.199Z++ bmat address ls -f extra -Ho first,last,count
2092023-12-07T08:20:09.226Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-12-07T08:20:09.226Z+ EXTRA_IP_START=10.151.1.164
2112023-12-07T08:20:09.226Z+ EXTRA_IP_END=10.151.1.227
2122023-12-07T08:20:09.227Z+ EXTRA_IP_COUNT=64
2132023-12-07T08:20:09.227Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-12-07T08:20:09.227Z+ EXTRA_IP_BASE=10.151.1
2152023-12-07T08:20:09.227Z+ EXTRA_IP_FOCTET=164
2162023-12-07T08:20:09.227Z+ EXTRA_IP_LOCTET=227
2172023-12-07T08:20:09.227Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-12-07T08:20:09.227Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-12-07T08:20:09.227Z+ DNS_IP1=10.151.1.164
2202023-12-07T08:20:09.227Z+ DNS_IP2=10.151.1.165
2212023-12-07T08:20:09.227Z+ UPLINK_IP=10.151.1.174
2222023-12-07T08:20:09.227Z+ IPPOOL_START=10.151.1.175
2232023-12-07T08:20:09.227Z+ IPPOOL_END=10.151.1.227
2242023-12-07T08:20:09.227Z++ ipadm show-addr -po type,addr
2252023-12-07T08:20:09.227Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-12-07T08:20:09.227Z+ GATEWAY_IP=10.151.1.100
2272023-12-07T08:20:09.227Z+ [[ -n 10.151.1.100 ]]
2282023-12-07T08:20:09.227Z+ ping -s 10.151.1.100 56 1
2292023-12-07T08:20:09.227ZPING 10.151.1.100: 56 data bytes
2302023-12-07T08:20:09.228Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.010 ms
2312023-12-07T08:20:09.228Z
2322023-12-07T08:20:09.228Z----10.151.1.100 PING Statistics----
2332023-12-07T08:20:09.228Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-12-07T08:20:09.228Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2352023-12-07T08:20:09.228Z++ arp -an
2362023-12-07T08:20:09.228Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-12-07T08:20:09.229Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-12-07T08:20:09.229Z+ routeadm -e ipv4-forwarding -u
2392023-12-07T08:20:09.329Z+ PXA_START=10.151.1.164
2402023-12-07T08:20:09.329Z+ PXA_END=10.151.1.227
2412023-12-07T08:20:09.329Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-12-07T08:20:09.354Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-12-07T08:20:09.471Z+ ZPOOL_VDEV_DIR=/scratch
2442023-12-07T08:20:09.471Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-12-07T08:20:09.504Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-12-07T08:20:09.504Z++ cd ./tools
2472023-12-07T08:20:09.504Z++ pwd
2482023-12-07T08:20:09.504Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-12-07T08:20:09.504Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-12-07T08:20:09.504Z+ SOFTNPU_MODE=zone
2512023-12-07T08:20:09.504Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-12-07T08:20:09.504Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-12-07T08:20:09.504Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-12-07T08:20:09.504Z++ dladm show-phys -p -o LINK
2552023-12-07T08:20:09.504Z++ head -1
2562023-12-07T08:20:09.552Z+ PHYSICAL_LINK=igb0
2572023-12-07T08:20:09.552Z+ echo 'Using igb0 as physical link'
2582023-12-07T08:20:09.577ZUsing igb0 as physical link
2592023-12-07T08:20:09.581Z+ ensure_run_as_root
2602023-12-07T08:20:09.581Z++ id -u
2612023-12-07T08:20:09.581Z+ [[ 0 -ne 0 ]]
2622023-12-07T08:20:09.581Z+ ensure_zpools
2632023-12-07T08:20:09.582Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-12-07T08:20:09.582Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-12-07T08:20:09.582Z+ readarray -t ZPOOLS
2662023-12-07T08:20:09.582Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-12-07T08:20:09.582Z++ sed 's/[ ",]//g'
2682023-12-07T08:20:09.582Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-12-07T08:20:09.582Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-12-07T08:20:09.582ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-12-07T08:20:09.582Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-12-07T08:20:09.582Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-12-07T08:20:09.582Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-12-07T08:20:09.607Z0+0 records in
2752023-12-07T08:20:09.607Z0+0 records out
2762023-12-07T08:20:09.607Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2772023-12-07T08:20:09.608Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-12-07T08:20:09.608Z+ set +x
2792023-12-07T08:20:09.608ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-12-07T08:20:09.608Z++ zpool list -o name
2812023-12-07T08:20:09.608Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-12-07T08:20:09.613Z+ [[ -z '' ]]
2832023-12-07T08:20:09.613Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-12-07T08:20:09.644Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-12-07T08:20:09.670Z+ set +x
2862023-12-07T08:20:09.670ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-12-07T08:20:09.670Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-12-07T08:20:09.670Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-12-07T08:20:09.671ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-12-07T08:20:09.671Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-12-07T08:20:09.671Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-12-07T08:20:09.671Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-12-07T08:20:09.671Z0+0 records in
2942023-12-07T08:20:09.671Z0+0 records out
2952023-12-07T08:20:09.671Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-12-07T08:20:09.671Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-12-07T08:20:09.671Z+ set +x
2982023-12-07T08:20:09.671ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-12-07T08:20:09.671Z++ zpool list -o name
3002023-12-07T08:20:09.671Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-12-07T08:20:09.671Z+ [[ -z '' ]]
3022023-12-07T08:20:09.671Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-12-07T08:20:09.696Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-12-07T08:20:09.696Z+ set +x
3052023-12-07T08:20:09.696ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-12-07T08:20:09.696Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-12-07T08:20:09.697Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-12-07T08:20:09.697ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3092023-12-07T08:20:09.697Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-12-07T08:20:09.701Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-12-07T08:20:09.701Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-12-07T08:20:09.726Z0+0 records in
3132023-12-07T08:20:09.727Z0+0 records out
3142023-12-07T08:20:09.727Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3152023-12-07T08:20:09.727Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-12-07T08:20:09.727Z+ set +x
3172023-12-07T08:20:09.727ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-12-07T08:20:09.727Z++ zpool list -o name
3192023-12-07T08:20:09.727Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-12-07T08:20:09.727Z+ [[ -z '' ]]
3212023-12-07T08:20:09.727Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-12-07T08:20:09.752Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-12-07T08:20:09.752Z+ set +x
3242023-12-07T08:20:09.752ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-12-07T08:20:09.752Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-12-07T08:20:09.752Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-12-07T08:20:09.752ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-12-07T08:20:09.753Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-12-07T08:20:09.753Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-12-07T08:20:09.753Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-12-07T08:20:09.753Z0+0 records in
3322023-12-07T08:20:09.753Z0+0 records out
3332023-12-07T08:20:09.753Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-12-07T08:20:09.753Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-12-07T08:20:09.754Z+ set +x
3362023-12-07T08:20:09.754ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-12-07T08:20:09.754Z++ zpool list -o name
3382023-12-07T08:20:09.754Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-12-07T08:20:09.779Z+ [[ -z '' ]]
3402023-12-07T08:20:09.779Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-12-07T08:20:09.804Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-12-07T08:20:09.805Z+ set +x
3432023-12-07T08:20:09.805ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-12-07T08:20:09.805Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-12-07T08:20:09.805Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-12-07T08:20:09.805ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-12-07T08:20:09.805Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-12-07T08:20:09.805Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-12-07T08:20:09.805Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-12-07T08:20:09.805Z0+0 records in
3512023-12-07T08:20:09.805Z0+0 records out
3522023-12-07T08:20:09.805Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3532023-12-07T08:20:09.805Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-12-07T08:20:09.805Z+ set +x
3552023-12-07T08:20:09.805ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-12-07T08:20:09.805Z++ zpool list -o name
3572023-12-07T08:20:09.805Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-12-07T08:20:09.806Z+ [[ -z '' ]]
3592023-12-07T08:20:09.806Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-12-07T08:20:09.837Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-12-07T08:20:09.838Z+ set +x
3622023-12-07T08:20:09.869ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-12-07T08:20:09.869Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-12-07T08:20:09.869Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-12-07T08:20:09.869ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-12-07T08:20:09.869Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-12-07T08:20:09.869Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-12-07T08:20:09.869Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-12-07T08:20:09.870Z0+0 records in
3702023-12-07T08:20:09.870Z0+0 records out
3712023-12-07T08:20:09.870Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3722023-12-07T08:20:09.870Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-12-07T08:20:09.870Z+ set +x
3742023-12-07T08:20:09.870ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-12-07T08:20:09.871Z++ zpool list -o name
3762023-12-07T08:20:09.871Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-12-07T08:20:09.871Z+ [[ -z '' ]]
3782023-12-07T08:20:09.871Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-12-07T08:20:09.912Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-12-07T08:20:09.912Z+ set +x
3812023-12-07T08:20:09.912ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-12-07T08:20:09.912Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-12-07T08:20:09.912Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-12-07T08:20:09.912Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3852023-12-07T08:20:09.913ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3862023-12-07T08:20:09.913Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-12-07T08:20:09.913Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-12-07T08:20:09.913Z0+0 records in
3892023-12-07T08:20:09.913Z0+0 records out
3902023-12-07T08:20:09.913Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-12-07T08:20:09.914Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-12-07T08:20:09.914Z+ set +x
3932023-12-07T08:20:09.914ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-12-07T08:20:09.915Z++ zpool list -o name
3952023-12-07T08:20:09.915Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-12-07T08:20:09.943Z+ [[ -z '' ]]
3972023-12-07T08:20:09.943Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-12-07T08:20:09.944Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-12-07T08:20:09.944Z+ set +x
4002023-12-07T08:20:09.944ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-12-07T08:20:09.944Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-12-07T08:20:09.944Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-12-07T08:20:09.945ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-12-07T08:20:09.945Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-12-07T08:20:09.945Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-12-07T08:20:09.945Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-12-07T08:20:09.945Z0+0 records in
4082023-12-07T08:20:09.945Z0+0 records out
4092023-12-07T08:20:09.946Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-12-07T08:20:09.946Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-12-07T08:20:09.946Z+ set +x
4122023-12-07T08:20:09.946ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-12-07T08:20:09.947Z++ zpool list -o name
4142023-12-07T08:20:09.947Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-12-07T08:20:09.979Z+ [[ -z '' ]]
4162023-12-07T08:20:09.979Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-12-07T08:20:10.021Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-12-07T08:20:10.021Z+ set +x
4192023-12-07T08:20:10.021ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-12-07T08:20:10.021Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-12-07T08:20:10.021Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-12-07T08:20:10.021ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-12-07T08:20:10.021Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-12-07T08:20:10.021Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-12-07T08:20:10.021Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-12-07T08:20:10.021Z0+0 records in
4272023-12-07T08:20:10.021Z0+0 records out
4282023-12-07T08:20:10.021Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4292023-12-07T08:20:10.021Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-12-07T08:20:10.021Z+ set +x
4312023-12-07T08:20:10.021ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-12-07T08:20:10.022Z++ zpool list -o name
4332023-12-07T08:20:10.022Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-12-07T08:20:10.022Z+ [[ -z '' ]]
4352023-12-07T08:20:10.022Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-12-07T08:20:10.048Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-12-07T08:20:10.048Z+ set +x
4382023-12-07T08:20:10.048ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-12-07T08:20:10.049Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-12-07T08:20:10.049Z+ readarray -t ZPOOLS
4412023-12-07T08:20:10.049Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-12-07T08:20:10.049Z++ sed 's/[ ",]//g'
4432023-12-07T08:20:10.049Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-12-07T08:20:10.050Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-12-07T08:20:10.050ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-12-07T08:20:10.050Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-12-07T08:20:10.050Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-12-07T08:20:10.050Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-12-07T08:20:10.050Z0+0 records in
4502023-12-07T08:20:10.050Z0+0 records out
4512023-12-07T08:20:10.050Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4522023-12-07T08:20:10.050Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-12-07T08:20:10.050Z+ set +x
4542023-12-07T08:20:10.050ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-12-07T08:20:10.050Z++ zpool list -o name
4562023-12-07T08:20:10.050Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-12-07T08:20:10.076Z+ [[ -z '' ]]
4582023-12-07T08:20:10.076Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-12-07T08:20:10.104Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-12-07T08:20:10.104Z+ set +x
4612023-12-07T08:20:10.104ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-12-07T08:20:10.104Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-12-07T08:20:10.105Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-12-07T08:20:10.105ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-12-07T08:20:10.105Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-12-07T08:20:10.105Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-12-07T08:20:10.105Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-12-07T08:20:10.118Z0+0 records in
4692023-12-07T08:20:10.118Z0+0 records out
4702023-12-07T08:20:10.118Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4712023-12-07T08:20:10.118Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-12-07T08:20:10.118Z+ set +x
4732023-12-07T08:20:10.119ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-12-07T08:20:10.119Z++ zpool list -o name
4752023-12-07T08:20:10.119Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-12-07T08:20:10.151Z+ [[ -z '' ]]
4772023-12-07T08:20:10.151Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-12-07T08:20:10.151Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-12-07T08:20:10.151Z+ set +x
4802023-12-07T08:20:10.152ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-12-07T08:20:10.152Z+ [[ zone == \z\o\n\e ]]
4822023-12-07T08:20:10.152Z+ ensure_simulated_links igb0
4832023-12-07T08:20:10.152Z+ local PHYSICAL_LINK=igb0
4842023-12-07T08:20:10.152Z+ INDICES=("0" "1")
4852023-12-07T08:20:10.152Z+ for I in "${INDICES[@]}"
4862023-12-07T08:20:10.153Z++ get_simnet_name_if_exists net0
4872023-12-07T08:20:10.153Z++ dladm show-simnet -p -o LINK net0
4882023-12-07T08:20:10.179Z++ echo ''
4892023-12-07T08:20:10.179Z+ [[ -z '' ]]
4902023-12-07T08:20:10.179Z+ dladm create-simnet -t net0
4912023-12-07T08:20:10.179Z+ dladm create-simnet -t sc0_0
4922023-12-07T08:20:10.179Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-12-07T08:20:10.179Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-12-07T08:20:10.180Z+ success 'Simnet net0/sc0_0 exists'
4952023-12-07T08:20:10.180Z+ set +x
4962023-12-07T08:20:10.180ZSimnet net0/sc0_0 exists
4972023-12-07T08:20:10.180Z+ for I in "${INDICES[@]}"
4982023-12-07T08:20:10.180Z++ get_simnet_name_if_exists net1
4992023-12-07T08:20:10.180Z++ dladm show-simnet -p -o LINK net1
5002023-12-07T08:20:10.180Z++ echo ''
5012023-12-07T08:20:10.180Z+ [[ -z '' ]]
5022023-12-07T08:20:10.180Z+ dladm create-simnet -t net1
5032023-12-07T08:20:10.206Z+ dladm create-simnet -t sc1_0
5042023-12-07T08:20:10.206Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-12-07T08:20:10.206Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-12-07T08:20:10.206Z+ success 'Simnet net1/sc1_0 exists'
5072023-12-07T08:20:10.206Z+ set +x
5082023-12-07T08:20:10.206ZSimnet net1/sc1_0 exists
5092023-12-07T08:20:10.206Z++ get_vnic_name_if_exists sc0_1
5102023-12-07T08:20:10.207Z++ dladm show-vnic -p -o LINK sc0_1
5112023-12-07T08:20:10.207Z++ echo ''
5122023-12-07T08:20:10.207Z+ [[ -z '' ]]
5132023-12-07T08:20:10.207Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-12-07T08:20:10.207Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-12-07T08:20:10.207Z+ success 'Vnic sc0_1 exists'
5162023-12-07T08:20:10.207Z+ set +x
5172023-12-07T08:20:10.207ZVnic sc0_1 exists
5182023-12-07T08:20:10.207Z+ warn_if_no_proxy_arp
5192023-12-07T08:20:10.207Z+ [[ -v PXA_START ]]
5202023-12-07T08:20:10.207Z+ [[ -v PXA_END ]]
5212023-12-07T08:20:10.207Z+ ensure_softnpu_zone
5222023-12-07T08:20:10.233Z+ zoneadm list
5232023-12-07T08:20:10.233Z+ grep -q sidecar_softnpu
5242023-12-07T08:20:10.233Z+ [[ -f out/npuzone/npuzone ]]
5252023-12-07T08:20:10.233Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit 45ed98fea5824feb4d42f45bbf218e597dc9fc58 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5262023-12-07T08:20:10.234Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5272023-12-07T08:20:17.685Zdownloading softnpu ...done
5282023-12-07T08:20:17.712Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/libsidecar_lite.so.sha256.txt
5292023-12-07T08:20:22.411Zdownloading asic_program.so ...done
5302023-12-07T08:20:22.438Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/scadm.sha256.txt
5312023-12-07T08:20:30.364Zdownloading scadm ...done
5322023-12-07T08:20:30.393Zcreating port tfportrear0_0 for sc0_0
5332023-12-07T08:20:30.393Zcreating port tfportqsfp0_0 for sc0_1
5342023-12-07T08:20:30.393Zsoftnpu zone setup
5352023-12-07T08:20:30.393Zconfigure zone
5362023-12-07T08:20:30.418Zinstall zone
5372023-12-07T08:20:31.461Zboot zone
5382023-12-07T08:20:34.576Z[sidecar_softnpu] svcs milestone/network
5392023-12-07T08:20:34.602ZSTATE STIME FMRI
5402023-12-07T08:20:34.602Zonline 8:20:33 svc:/milestone/network:default
5412023-12-07T08:20:34.602Z[sidecar_softnpu] /softnpu/npu start
5422023-12-07T08:20:34.627Z
5432023-12-07T08:20:34.663Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-12-07T08:20:34.663Z+ GATEWAY_IP=10.151.1.100
5452023-12-07T08:20:34.663Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-12-07T08:20:34.664ZUsing 10.151.1.100 as gateway ip
5472023-12-07T08:20:34.664Z+ [[ ! -v GATEWAY_MAC ]]
5482023-12-07T08:20:34.664Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-12-07T08:20:34.664Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-12-07T08:20:34.664Z+++ echo 18:c0:4d:0d:9f:b2
5512023-12-07T08:20:34.665Z+ COUNT=1
5522023-12-07T08:20:34.665Z+ [[ 1 -ne 1 ]]
5532023-12-07T08:20:34.665Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-12-07T08:20:34.665ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-12-07T08:20:34.666Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-12-07T08:20:34.666Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-12-07T08:20:34.694Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-12-07T08:20:34.694Z+ [[ -v PXA_START ]]
5592023-12-07T08:20:34.694Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-12-07T08:20:34.694Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-12-07T08:20:34.721Z+ z_scadm dump-state
5622023-12-07T08:20:34.721Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-12-07T08:20:34.746Zlocal v6:
5642023-12-07T08:20:34.746Zlocal v4:
5652023-12-07T08:20:34.746Zrouter v6:
5662023-12-07T08:20:34.746Zrouter v4_idx:
5672023-12-07T08:20:34.747Zrouter v4_routes:
5682023-12-07T08:20:34.747Zresolver v4:
5692023-12-07T08:20:34.747Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-12-07T08:20:34.747Zresolver v6:
5712023-12-07T08:20:34.747Znat v4:
5722023-12-07T08:20:34.747Znat_v6:
5732023-12-07T08:20:34.747Zport_mac:
5742023-12-07T08:20:34.747Zproxy arp:
5752023-12-07T08:20:34.747Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-12-07T08:20:34.747Z+ success 'softnpu zone exists'
5772023-12-07T08:20:34.747Z+ set +x
5782023-12-07T08:20:34.747Zsoftnpu zone exists
5792023-12-07T08:20:34.747Z
5802023-12-07T08:20:34.747Zreal 25.258215840
5812023-12-07T08:20:34.747Zuser 0.717631600
5822023-12-07T08:20:34.747Zsys 1.652354913
5832023-12-07T08:20:34.747Ztrap 0.000089116
5842023-12-07T08:20:34.748Ztflt 0.000156577
5852023-12-07T08:20:34.749Zdflt 0.001478786
5862023-12-07T08:20:34.749Zkflt 0.000000000
5872023-12-07T08:20:34.749Zlock 19:29.893297643
5882023-12-07T08:20:34.749Zslp 1:15.500295960
5892023-12-07T08:20:34.749Zlat 0.076862838
5902023-12-07T08:20:34.749Zstop 0.368494852
5912023-12-07T08:20:34.749Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-12-07T08:20:34.777Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-12-07T08:20:34.777Z+ SILO_NAME=recovery
5942023-12-07T08:20:34.777Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-07T08:20:34.777Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-12-07T08:20:34.777Z+ sed -i~ '
5972023-12-07T08:20:34.777Z /^external_dns_ips/c\
5982023-12-07T08:20:34.777Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-12-07T08:20:34.777Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-12-07T08:20:34.778Z /^first/c\
6012023-12-07T08:20:34.778Zfirst = "10.151.1.164"
6022023-12-07T08:20:34.778Z /^last/c\
6032023-12-07T08:20:34.778Zlast = "10.151.1.173"
6042023-12-07T08:20:34.778Z }
6052023-12-07T08:20:34.778Z /^\[rack_network_config/,/^$/ {
6062023-12-07T08:20:34.778Z /^infra_ip_first/c\
6072023-12-07T08:20:34.778Zinfra_ip_first = "10.151.1.174"
6082023-12-07T08:20:34.778Z /^infra_ip_last/c\
6092023-12-07T08:20:34.778Zinfra_ip_last = "10.151.1.174"
6102023-12-07T08:20:34.778Z }
6112023-12-07T08:20:34.778Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-12-07T08:20:34.778Z /^routes/c\
6132023-12-07T08:20:34.778Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6142023-12-07T08:20:34.779Z /^addresses/c\
6152023-12-07T08:20:34.779Zaddresses = \["10.151.1.174/32"\]
6162023-12-07T08:20:34.779Z }
6172023-12-07T08:20:34.779Z' pkg/config-rss.toml
6182023-12-07T08:20:34.779Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-12-07T08:20:34.805Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-12-07T08:20:34.805Z+++ pkg/config-rss.toml Thu Dec 7 08:20:34 2023
6212023-12-07T08:20:34.805Z@@ -41,7 +41,7 @@
6222023-12-07T08:20:34.805Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-12-07T08:20:34.805Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-12-07T08:20:34.806Z # below.
6252023-12-07T08:20:34.806Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-12-07T08:20:34.806Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6272023-12-07T08:20:34.806Z
6282023-12-07T08:20:34.806Z # Initial TLS certificates for the external API
6292023-12-07T08:20:34.806Z #
6302023-12-07T08:20:34.806Z@@ -77,8 +77,8 @@
6312023-12-07T08:20:34.806Z #
6322023-12-07T08:20:34.806Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-12-07T08:20:34.806Z [[internal_services_ip_pool_ranges]]
6342023-12-07T08:20:34.806Z-first = "192.168.1.20"
6352023-12-07T08:20:34.806Z-last = "192.168.1.29"
6362023-12-07T08:20:34.806Z+first = "10.151.1.164"
6372023-12-07T08:20:34.806Z+last = "10.151.1.173"
6382023-12-07T08:20:34.807Z
6392023-12-07T08:20:34.807Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-12-07T08:20:34.807Z # parameters should be provided to the control plane via wicket, but we need to
6412023-12-07T08:20:34.807Z@@ -92,8 +92,8 @@
6422023-12-07T08:20:34.807Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-12-07T08:20:34.807Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-12-07T08:20:34.807Z # With softnpu, only one address is used.
6452023-12-07T08:20:34.807Z-infra_ip_first = "192.168.1.30"
6462023-12-07T08:20:34.807Z-infra_ip_last = "192.168.1.30"
6472023-12-07T08:20:34.807Z+infra_ip_first = "10.151.1.174"
6482023-12-07T08:20:34.807Z+infra_ip_last = "10.151.1.174"
6492023-12-07T08:20:34.807Z
6502023-12-07T08:20:34.807Z # Configurations for BGP routers to run on the scrimlets.
6512023-12-07T08:20:34.807Z bgp = []
6522023-12-07T08:20:34.807Z@@ -101,9 +101,9 @@
6532023-12-07T08:20:34.807Z # You can configure multiple uplinks by repeating the following stanza
6542023-12-07T08:20:34.808Z [[rack_network_config.ports]]
6552023-12-07T08:20:34.809Z # Routes associated with this port.
6562023-12-07T08:20:34.809Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-12-07T08:20:34.809Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6582023-12-07T08:20:34.809Z # Addresses associated with this port.
6592023-12-07T08:20:34.809Z-addresses = ["192.168.1.30/32"]
6602023-12-07T08:20:34.809Z+addresses = ["10.151.1.174/32"]
6612023-12-07T08:20:34.809Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-12-07T08:20:34.809Z port = "qsfp0"
6632023-12-07T08:20:34.809Z # The speed of this port.
6642023-12-07T08:20:34.809Z+ true
6652023-12-07T08:20:34.809Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-12-07T08:20:34.809Za pkg/config-rss.toml 7K
6672023-12-07T08:20:34.809Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-12-07T08:20:34.809Z+ TLS_NAME=recovery.sys.oxide.test
6692023-12-07T08:20:34.809Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-12-07T08:20:34.951Z..+..........+...+.....+.+...+.....+.............+..+......+......+.........+.......+...+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+......+......+...+......+..+....+.....+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..+.............+...+.....+...................+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-12-07T08:20:35.138Z.+........+.......+...+........+................+.........+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*............+.....+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+.+........+.+...+........+...+......+.+...+.........+....................+...+....+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-07T08:20:35.163Z-----
6732023-12-07T08:20:35.164Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-12-07T08:20:35.189Za pkg/initial-tls-cert.pem 2K
6752023-12-07T08:20:35.189Za pkg/initial-tls-key.pem 4K
6762023-12-07T08:20:35.189Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-12-07T08:20:35.190Z+ rmdir pkg
6782023-12-07T08:20:35.190Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-12-07T08:20:35.190Z+ pfexec mkdir -p /usr/oxide
6802023-12-07T08:20:35.190Z+ pfexec rm -f /usr/oxide/tcpproxy
6812023-12-07T08:20:35.190Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6822023-12-07T08:20:35.437Z+ pfexec chmod +x /usr/oxide/tcpproxy
6832023-12-07T08:20:35.463Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6842023-12-07T08:20:35.463Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6852023-12-07T08:20:35.463Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6862023-12-07T08:20:35.577Z+ OMICRON_NO_UNINSTALL=1
6872023-12-07T08:20:35.577Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6882023-12-07T08:20:35.604ZDec 07 08:20:35.749 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6892023-12-07T08:20:35.604ZDec 07 08:20:35.751 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6902023-12-07T08:20:35.605ZDec 07 08:20:35.751 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6912023-12-07T08:20:35.617ZDec 07 08:20:35.751 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6922023-12-07T08:20:35.617ZDec 07 08:20:35.752 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6932023-12-07T08:20:35.617ZDec 07 08:20:35.752 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6942023-12-07T08:20:35.617ZDec 07 08:20:35.752 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6952023-12-07T08:20:35.617ZDec 07 08:20:35.752 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6962023-12-07T08:20:35.617ZDec 07 08:20:35.752 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6972023-12-07T08:20:35.617ZDec 07 08:20:35.752 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6982023-12-07T08:20:35.618ZDec 07 08:20:35.752 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6992023-12-07T08:20:35.618ZDec 07 08:20:35.752 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7002023-12-07T08:20:35.618ZDec 07 08:20:35.752 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7012023-12-07T08:20:35.618ZDec 07 08:20:35.752 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7022023-12-07T08:20:35.618ZDec 07 08:20:35.752 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7032023-12-07T08:20:35.618ZDec 07 08:20:35.752 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7042023-12-07T08:20:35.618ZDec 07 08:20:35.752 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
7052023-12-07T08:20:35.822ZDec 07 08:20:35.979 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-12-07T08:20:35.859ZDec 07 08:20:36.016 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-12-07T08:20:35.922ZDec 07 08:20:36.078 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-12-07T08:20:36.050Z
7092023-12-07T08:20:36.076Zreal 0.468678127
7102023-12-07T08:20:36.076Zuser 0.098552663
7112023-12-07T08:20:36.076Zsys 1.854497076
7122023-12-07T08:20:36.076Ztrap 0.000016780
7132023-12-07T08:20:36.076Ztflt 0.000686883
7142023-12-07T08:20:36.077Zdflt 0.000373228
7152023-12-07T08:20:36.077Zkflt 0.000000000
7162023-12-07T08:20:36.077Zlock 42.024302829
7172023-12-07T08:20:36.077Zslp 0.680871101
7182023-12-07T08:20:36.077Zlat 0.003133674
7192023-12-07T08:20:36.077Zstop 0.216218540
7202023-12-07T08:20:36.077Z+ retry=0
7212023-12-07T08:20:36.077Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-12-07T08:20:36.077Z+ [[ 0 -gt 30 ]]
7232023-12-07T08:20:36.077Z+ sleep 1
7242023-12-07T08:20:37.066Z+ retry=1
7252023-12-07T08:20:37.066Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-12-07T08:20:37.092Z+ [[ 1 -gt 30 ]]
7272023-12-07T08:20:37.092Z+ sleep 1
7282023-12-07T08:20:38.080Z+ retry=2
7292023-12-07T08:20:38.080Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-12-07T08:20:38.116Z+ [[ 2 -gt 30 ]]
7312023-12-07T08:20:38.116Z+ sleep 1
7322023-12-07T08:20:39.097Z+ retry=3
7332023-12-07T08:20:39.098Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-12-07T08:20:39.129Z+ [[ 3 -gt 30 ]]
7352023-12-07T08:20:39.130Z+ sleep 1
7362023-12-07T08:20:40.115Z+ retry=4
7372023-12-07T08:20:40.115Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-12-07T08:21:05.286Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7392023-12-07T08:21:05.315Zlocal v6:
7402023-12-07T08:21:05.315Zfd00:99::1
7412023-12-07T08:21:05.315Zfe80::a812:ecff:fe31:f9d3
7422023-12-07T08:21:05.315Zlocal v4:
7432023-12-07T08:21:05.315Z10.151.1.174
7442023-12-07T08:21:05.315Zrouter v6:
7452023-12-07T08:21:05.315Zfd00:1122:3344:3::/64 -> fe80::f406:bdff:fe74:f181 (1)
7462023-12-07T08:21:05.315Zfd00:1122:3344:1::/64 -> fe80::f406:bdff:fe74:f181 (1)
7472023-12-07T08:21:05.315Zfd00:1122:3344:2::/64 -> fe80::f406:bdff:fe74:f181 (1)
7482023-12-07T08:21:05.316Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::f406:bdff:fe74:f181 (1)
7492023-12-07T08:21:05.316Zfd00:1122:3344:101::/64 -> fe80::f406:bdff:fe74:f181 (1)
7502023-12-07T08:21:05.316Zrouter v4_idx:
7512023-12-07T08:21:05.316Z0.0.0.0/0 -> 0
7522023-12-07T08:21:05.316Zrouter v4_routes:
7532023-12-07T08:21:05.316Z0 -> 10.151.1.100 (2)
7542023-12-07T08:21:05.341Zresolver v4:
7552023-12-07T08:21:05.341Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7562023-12-07T08:21:05.342Z10.151.1.174 -> be:1f:23:79:a0:92
7572023-12-07T08:21:05.342Zresolver v6:
7582023-12-07T08:21:05.342Zfe80::a812:ecff:fe31:f9d3 -> aa:12:ec:31:f9:d3
7592023-12-07T08:21:05.342Zfe80::f406:bdff:fe74:f181 -> f6:06:bd:74:f1:81
7602023-12-07T08:21:05.342Znat v4:
7612023-12-07T08:21:05.342Z10.151.1.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:80:02
7622023-12-07T08:21:05.342Znat_v6:
7632023-12-07T08:21:05.342Zport_mac:
7642023-12-07T08:21:05.342Z1000: a8:40:25:73:26:f7
7652023-12-07T08:21:05.342Z1: a8:40:25:73:26:f8
7662023-12-07T08:21:05.342Zproxy arp:
7672023-12-07T08:21:05.343Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7682023-12-07T08:21:05.343Z+ retry=0
7692023-12-07T08:21:05.343Z+++ zoneadm list -n
7702023-12-07T08:21:05.343Z+++ grep oxz_ntp
7712023-12-07T08:21:05.343Z++ pfexec svcs -z oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86 -Hostate oxide/ntp
7722023-12-07T08:21:05.368Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7732023-12-07T08:21:05.368Z++ true
7742023-12-07T08:21:05.369Z+ [[ '' != online ]]
7752023-12-07T08:21:05.369Z+ [[ 0 -gt 60 ]]
7762023-12-07T08:21:05.369Z+ sleep 1
7772023-12-07T08:21:06.358Z+ retry=1
7782023-12-07T08:21:06.812Z+++ zoneadm list -n
7792023-12-07T08:21:06.813Z+++ grep oxz_ntp
7802023-12-07T08:21:06.813Z++ pfexec svcs -z oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86 -Hostate oxide/ntp
7812023-12-07T08:21:06.813Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7822023-12-07T08:21:06.813Z++ true
7832023-12-07T08:21:06.813Z+ [[ '' != online ]]
7842023-12-07T08:21:06.813Z+ [[ 1 -gt 60 ]]
7852023-12-07T08:21:06.813Z+ sleep 1
7862023-12-07T08:21:07.385Z+ retry=2
7872023-12-07T08:21:07.415Z+++ zoneadm list -n
7882023-12-07T08:21:07.415Z+++ grep oxz_ntp
7892023-12-07T08:21:07.415Z++ pfexec svcs -z oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86 -Hostate oxide/ntp
7902023-12-07T08:21:07.417Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7912023-12-07T08:21:07.417Z++ true
7922023-12-07T08:21:07.417Z+ [[ '' != online ]]
7932023-12-07T08:21:07.418Z+ [[ 2 -gt 60 ]]
7942023-12-07T08:21:07.418Z+ sleep 1
7952023-12-07T08:21:08.410Z+ retry=3
7962023-12-07T08:21:08.436Z+++ zoneadm list -n
7972023-12-07T08:21:08.436Z+++ grep oxz_ntp
7982023-12-07T08:21:08.436Z++ pfexec svcs -z oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86 -Hostate oxide/ntp
7992023-12-07T08:21:08.436Z+ [[ disabled != online ]]
8002023-12-07T08:21:08.436Z+ [[ 3 -gt 60 ]]
8012023-12-07T08:21:08.436Z+ sleep 1
8022023-12-07T08:21:09.433Z+ retry=4
8032023-12-07T08:21:09.460Z+++ zoneadm list -n
8042023-12-07T08:21:09.460Z+++ grep oxz_ntp
8052023-12-07T08:21:09.460Z++ pfexec svcs -z oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86 -Hostate oxide/ntp
8062023-12-07T08:21:09.460Z+ [[ disabled != online ]]
8072023-12-07T08:21:09.460Z+ [[ 4 -gt 60 ]]
8082023-12-07T08:21:09.460Z+ sleep 1
8092023-12-07T08:21:10.458Z+ retry=5
8102023-12-07T08:21:10.485Z+++ zoneadm list -n
8112023-12-07T08:21:10.485Z+++ grep oxz_ntp
8122023-12-07T08:21:10.486Z++ pfexec svcs -z oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86 -Hostate oxide/ntp
8132023-12-07T08:21:10.486Z+ [[ disabled != online ]]
8142023-12-07T08:21:10.486Z+ [[ 5 -gt 60 ]]
8152023-12-07T08:21:10.486Z+ sleep 1
8162023-12-07T08:21:11.477Z+ retry=6
8172023-12-07T08:21:11.503Z+++ zoneadm list -n
8182023-12-07T08:21:11.504Z+++ grep oxz_ntp
8192023-12-07T08:21:11.504Z++ pfexec svcs -z oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86 -Hostate oxide/ntp
8202023-12-07T08:21:11.504Z+ [[ disabled != online ]]
8212023-12-07T08:21:11.504Z+ [[ 6 -gt 60 ]]
8222023-12-07T08:21:11.504Z+ sleep 1
8232023-12-07T08:21:12.500Z+ retry=7
8242023-12-07T08:21:13.857Z+++ zoneadm list -n
8252023-12-07T08:21:13.857Z+++ grep oxz_ntp
8262023-12-07T08:21:13.858Z++ pfexec svcs -z oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86 -Hostate oxide/ntp
8272023-12-07T08:21:13.858Z+ [[ disabled != online ]]
8282023-12-07T08:21:13.858Z+ [[ 7 -gt 60 ]]
8292023-12-07T08:21:13.858Z+ sleep 1
8302023-12-07T08:21:13.889Z+ retry=8
8312023-12-07T08:21:13.889Z+++ zoneadm list -n
8322023-12-07T08:21:13.890Z+++ grep oxz_ntp
8332023-12-07T08:21:13.891Z++ pfexec svcs -z oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86 -Hostate oxide/ntp
8342023-12-07T08:21:13.891Z+ [[ disabled != online ]]
8352023-12-07T08:21:13.891Z+ [[ 8 -gt 60 ]]
8362023-12-07T08:21:13.891Z+ sleep 1
8372023-12-07T08:21:14.545Z+ retry=9
8382023-12-07T08:21:14.570Z+++ zoneadm list -n
8392023-12-07T08:21:14.570Z+++ grep oxz_ntp
8402023-12-07T08:21:14.571Z++ pfexec svcs -z oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86 -Hostate oxide/ntp
8412023-12-07T08:21:14.571Z+ [[ disabled != online ]]
8422023-12-07T08:21:14.571Z+ [[ 9 -gt 60 ]]
8432023-12-07T08:21:14.572Z+ sleep 1
 
8442023-12-07T08:21:15.386Zbuskin console login: Dec 7 08:21:15 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8452023-12-07T08:21:15.571Z+ retry=10
8462023-12-07T08:21:15.599Z+++ zoneadm list -n
8472023-12-07T08:21:15.599Z+++ grep oxz_ntp
8482023-12-07T08:21:15.600Z++ pfexec svcs -z oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86 -Hostate oxide/ntp
8492023-12-07T08:21:15.600Z+ [[ disabled != online ]]
8502023-12-07T08:21:15.600Z+ [[ 10 -gt 60 ]]
8512023-12-07T08:21:15.600Z+ sleep 1
8522023-12-07T08:21:16.591Z+ retry=11
8532023-12-07T08:21:16.616Z+++ zoneadm list -n
8542023-12-07T08:21:16.616Z+++ grep oxz_ntp
8552023-12-07T08:21:16.617Z++ pfexec svcs -z oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86 -Hostate oxide/ntp
8562023-12-07T08:21:16.617Z+ [[ online != online ]]
8572023-12-07T08:21:16.617Z+ echo 'Waited for chrony: 11s'
8582023-12-07T08:21:16.617Z+ export RUST_BACKTRACE=1
8592023-12-07T08:21:16.617ZWaited for chrony: 11s
8602023-12-07T08:21:16.617Z+ RUST_BACKTRACE=1
8612023-12-07T08:21:16.617Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8622023-12-07T08:21:16.620Z+ ./tests/bootstrap
8632023-12-07T08:21:16.645ZUsing external DNS server at 10.151.1.164:53
8642023-12-07T08:21:16.645Z2023-12-07 08:21:16.799986166 UTC: attempting to log into API
8652023-12-07T08:21:31.680Z2023-12-07 08:21:31.735901338 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8662023-12-07T08:21:32.685Z2023-12-07 08:21:32.734348061 UTC: attempting to log into API
8672023-12-07T08:21:47.719Z2023-12-07 08:21:47.726457480 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8682023-12-07T08:21:48.717Z2023-12-07 08:21:48.724353363 UTC: attempting to log into API
8692023-12-07T08:22:03.748Z2023-12-07 08:22:03.733840204 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8702023-12-07T08:22:04.748Z2023-12-07 08:22:04.734729592 UTC: attempting to log into API
8712023-12-07T08:22:19.781Z2023-12-07 08:22:19.767087131 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8722023-12-07T08:22:20.787Z2023-12-07 08:22:20.768017960 UTC: attempting to log into API
8732023-12-07T08:22:35.821Z2023-12-07 08:22:35.800933774 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8742023-12-07T08:22:36.816Z2023-12-07 08:22:36.801853246 UTC: attempting to log into API
 
8752023-12-07T08:22:38.991ZDec 7 08:22:38 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
8762023-12-07T08:22:46.778ZDec 7 08:22:46 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
8772023-12-07T08:22:51.854Z2023-12-07 08:22:51.834597888 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8782023-12-07T08:22:52.852Z2023-12-07 08:22:52.836397435 UTC: attempting to log into API
 
8792023-12-07T08:22:54.386ZDec 7 08:22:53 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
8802023-12-07T08:23:02.490ZDec 7 08:23:01 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
8812023-12-07T08:23:07.888Z2023-12-07 08:23:07.869063396 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
8822023-12-07T08:23:08.809ZDec 7 08:23:08 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
8832023-12-07T08:23:08.887Z2023-12-07 08:23:08.870120727 UTC: attempting to log into API
8842023-12-07T08:23:23.926Z2023-12-07 08:23:23.903543403 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8852023-12-07T08:23:24.921Z2023-12-07 08:23:24.904529683 UTC: attempting to log into API
8862023-12-07T08:23:30.715Z2023-12-07 08:23:30.700090081 UTC: login succeeded
8872023-12-07T08:23:30.748Zwaiting for nexus to come up...
8882023-12-07T08:23:30.816Zcreating IP pool... 10.151.1.175 - 10.151.1.227
8892023-12-07T08:23:30.859Zensuring datasets are ready...
8902023-12-07T08:23:33.375Zlet's roll.
8912023-12-07T08:23:33.406Z+ rm ./tests/bootstrap
8922023-12-07T08:23:33.438Z+ for test_bin in tests/*
8932023-12-07T08:23:33.438Z+ ./tests/bootstrap-4fe365f0266bec52
8942023-12-07T08:23:33.468Z
8952023-12-07T08:23:33.468Zrunning 0 tests
8962023-12-07T08:23:33.469Z
8972023-12-07T08:23:33.469Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
8982023-12-07T08:23:33.469Z
8992023-12-07T08:23:33.469Z+ for test_bin in tests/*
9002023-12-07T08:23:33.469Z+ ./tests/end_to_end_tests-de860e1ecc3158ae
9012023-12-07T08:23:33.469Z
9022023-12-07T08:23:33.469Zrunning 2 tests
9032023-12-07T08:23:33.668Ztest no_spoof::no_spoof_here ... ok
9042023-12-07T08:24:36.365Ztest instance_launch::instance_launch has been running for over 60 seconds
9052023-12-07T08:26:32.777Ztest instance_launch::instance_launch ... ok
9062023-12-07T08:26:32.778Z
9072023-12-07T08:26:32.778Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 179.32s
9082023-12-07T08:26:32.778Z
9092023-12-07T08:26:32.805Z+ _exit_trap
9102023-12-07T08:26:32.805Z+ local status=0
9112023-12-07T08:26:32.805Z+ [[ 0 -eq 0 ]]
9122023-12-07T08:26:32.805Z+ exit 0
9132023-12-07T08:26:32.805Zprocess exited: duration 391241 ms, exit code 0
 
9142023-12-07T08:26:32.922Zfound 47 output files
9152023-12-07T08:26:32.923Zuploading: /var/svc/log/oxide-sled-agent:default.log (664441 bytes)
9162023-12-07T08:26:34.255Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9172023-12-07T08:26:34.255Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_3d016448-4a52-40aa-bbe9-6e72a637e80d/root/var/svc/log/oxide-cockroachdb:default.log (5341 bytes)
9182023-12-07T08:26:35.973Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_3d016448-4a52-40aa-bbe9-6e72a637e80d/root/var/svc/log/oxide-cockroachdb:default.log
9192023-12-07T08:26:36.002Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_20aa6927-054c-4400-9fd0-4a6171ecf1f7/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9202023-12-07T08:26:37.145Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_20aa6927-054c-4400-9fd0-4a6171ecf1f7/root/var/svc/log/oxide-crucible-pantry:default.log
9212023-12-07T08:26:37.145Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86/root/var/svc/log/oxide-ntp:default.log (984 bytes)
9222023-12-07T08:26:38.250Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86/root/var/svc/log/oxide-ntp:default.log
9232023-12-07T08:26:38.251Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_3ff7d0e2-923b-4af1-9efe-7ac20f974a1c/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
9242023-12-07T08:26:39.379Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_3ff7d0e2-923b-4af1-9efe-7ac20f974a1c/root/var/svc/log/oxide-cockroachdb:default.log
9252023-12-07T08:26:39.501Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_91448e92-bba4-4e57-aa3b-36cb93a73218/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
9262023-12-07T08:26:40.591Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_91448e92-bba4-4e57-aa3b-36cb93a73218/root/var/svc/log/oxide-cockroachdb:default.log
9272023-12-07T08:26:40.591Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0341f6ed-d4d6-455b-b5c7-2638af96fd1e/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9282023-12-07T08:26:41.723Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0341f6ed-d4d6-455b-b5c7-2638af96fd1e/root/var/svc/log/oxide-crucible-agent:default.log
9292023-12-07T08:26:41.724Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0341f6ed-d4d6-455b-b5c7-2638af96fd1e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9302023-12-07T08:26:42.828Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0341f6ed-d4d6-455b-b5c7-2638af96fd1e/root/var/svc/log/oxide-crucible-downstairs:default.log
9312023-12-07T08:26:42.828Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7501ce93-1c04-4627-a511-ddbda12d01b7/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
9322023-12-07T08:26:43.978Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7501ce93-1c04-4627-a511-ddbda12d01b7/root/var/svc/log/oxide-crucible-agent:default.log
9332023-12-07T08:26:43.979Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7501ce93-1c04-4627-a511-ddbda12d01b7/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9342023-12-07T08:26:45.108Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7501ce93-1c04-4627-a511-ddbda12d01b7/root/var/svc/log/oxide-crucible-downstairs:default.log
9352023-12-07T08:26:45.133Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7501ce93-1c04-4627-a511-ddbda12d01b7/root/var/svc/log/oxide-crucible-downstairs:downstairs-65e576d4-99c0-452d-b0c4-36aa52e1dd98.log (3049 bytes)
9362023-12-07T08:26:46.210Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7501ce93-1c04-4627-a511-ddbda12d01b7/root/var/svc/log/oxide-crucible-downstairs:downstairs-65e576d4-99c0-452d-b0c4-36aa52e1dd98.log
9372023-12-07T08:26:46.210Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_db631b8b-f432-4df1-b571-fb870ee07c71/root/var/svc/log/oxide-crucible-agent:default.log (24366 bytes)
9382023-12-07T08:26:47.766Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_db631b8b-f432-4df1-b571-fb870ee07c71/root/var/svc/log/oxide-crucible-agent:default.log
9392023-12-07T08:26:47.795Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_db631b8b-f432-4df1-b571-fb870ee07c71/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9402023-12-07T08:26:48.873Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_db631b8b-f432-4df1-b571-fb870ee07c71/root/var/svc/log/oxide-crucible-downstairs:default.log
9412023-12-07T08:26:48.873Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_db631b8b-f432-4df1-b571-fb870ee07c71/root/var/svc/log/oxide-crucible-downstairs:downstairs-1c768fa9-487f-4d63-9ab6-65d6f7d4f87c.log (3049 bytes)
9422023-12-07T08:26:50.829Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_db631b8b-f432-4df1-b571-fb870ee07c71/root/var/svc/log/oxide-crucible-downstairs:downstairs-1c768fa9-487f-4d63-9ab6-65d6f7d4f87c.log
9432023-12-07T08:26:50.829Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_db631b8b-f432-4df1-b571-fb870ee07c71/root/var/svc/log/oxide-crucible-downstairs:downstairs-db188a6d-a62b-4ed2-b569-618bb97563d9.log (5535 bytes)
9442023-12-07T08:26:51.967Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_db631b8b-f432-4df1-b571-fb870ee07c71/root/var/svc/log/oxide-crucible-downstairs:downstairs-db188a6d-a62b-4ed2-b569-618bb97563d9.log
9452023-12-07T08:26:51.968Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2360a2e9-328a-4982-b483-01a82a8f76c8/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9462023-12-07T08:26:53.530Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2360a2e9-328a-4982-b483-01a82a8f76c8/root/var/svc/log/oxide-crucible-agent:default.log
9472023-12-07T08:26:53.531Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2360a2e9-328a-4982-b483-01a82a8f76c8/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9482023-12-07T08:26:54.684Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2360a2e9-328a-4982-b483-01a82a8f76c8/root/var/svc/log/oxide-crucible-downstairs:default.log
9492023-12-07T08:26:54.711Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_41c55645-7d5b-4949-97d7-100c61ea2d64/root/var/svc/log/oxide-crucible-agent:default.log (14120 bytes)
9502023-12-07T08:26:55.846Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_41c55645-7d5b-4949-97d7-100c61ea2d64/root/var/svc/log/oxide-crucible-agent:default.log
9512023-12-07T08:26:55.846Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_41c55645-7d5b-4949-97d7-100c61ea2d64/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9522023-12-07T08:26:56.957Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_41c55645-7d5b-4949-97d7-100c61ea2d64/root/var/svc/log/oxide-crucible-downstairs:default.log
9532023-12-07T08:26:56.957Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_41c55645-7d5b-4949-97d7-100c61ea2d64/root/var/svc/log/oxide-crucible-downstairs:downstairs-0bf3fc6a-bea8-41b3-8973-ceda13ad0fdf.log (3050 bytes)
9542023-12-07T08:26:58.060Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_41c55645-7d5b-4949-97d7-100c61ea2d64/root/var/svc/log/oxide-crucible-downstairs:downstairs-0bf3fc6a-bea8-41b3-8973-ceda13ad0fdf.log
9552023-12-07T08:26:58.088Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_839d4c8b-74fe-404c-bbdd-97b1ec78668f/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9562023-12-07T08:26:59.220Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_839d4c8b-74fe-404c-bbdd-97b1ec78668f/root/var/svc/log/oxide-crucible-agent:default.log
9572023-12-07T08:26:59.256Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_839d4c8b-74fe-404c-bbdd-97b1ec78668f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9582023-12-07T08:27:00.336Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_839d4c8b-74fe-404c-bbdd-97b1ec78668f/root/var/svc/log/oxide-crucible-downstairs:default.log
9592023-12-07T08:27:00.336Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_ae80ddc5-6966-419b-a1d5-abaf52e94974/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9602023-12-07T08:27:01.459Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_ae80ddc5-6966-419b-a1d5-abaf52e94974/root/var/svc/log/oxide-crucible-pantry:default.log
9612023-12-07T08:27:01.460Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_917c168d-7ba3-4d39-a5dc-0668dbd870f0/root/var/svc/log/oxide-external_dns:default.log (15609 bytes)
9622023-12-07T08:27:02.646Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_917c168d-7ba3-4d39-a5dc-0668dbd870f0/root/var/svc/log/oxide-external_dns:default.log
9632023-12-07T08:27:02.688Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_e44fe226-8169-463e-a0e3-3a5a21113023/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
9642023-12-07T08:27:04.190Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_e44fe226-8169-463e-a0e3-3a5a21113023/root/var/svc/log/oxide-cockroachdb:default.log
9652023-12-07T08:27:04.638Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_68ac3b99-9a37-44ed-b371-4bc1eddb6ab9/root/var/svc/log/oxide-nexus:default.log (654541 bytes)
9662023-12-07T08:27:05.356Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_68ac3b99-9a37-44ed-b371-4bc1eddb6ab9/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 654541 -> 661422
9672023-12-07T08:27:06.449Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_68ac3b99-9a37-44ed-b371-4bc1eddb6ab9/root/var/svc/log/oxide-nexus:default.log
9682023-12-07T08:27:06.449Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_82d7256b-3170-418e-bd96-15752f27f2a4/root/var/svc/log/oxide-nexus:default.log (562099 bytes)
9692023-12-07T08:27:07.646Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_82d7256b-3170-418e-bd96-15752f27f2a4/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 562099 -> 568981
9702023-12-07T08:27:08.725Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_82d7256b-3170-418e-bd96-15752f27f2a4/root/var/svc/log/oxide-nexus:default.log
9712023-12-07T08:27:08.754Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_074d0909-f14f-4866-b8a3-5d3ff903585f/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9722023-12-07T08:27:09.858Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_074d0909-f14f-4866-b8a3-5d3ff903585f/root/var/svc/log/oxide-crucible-agent:default.log
9732023-12-07T08:27:09.893Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_074d0909-f14f-4866-b8a3-5d3ff903585f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9742023-12-07T08:27:10.983Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_074d0909-f14f-4866-b8a3-5d3ff903585f/root/var/svc/log/oxide-crucible-downstairs:default.log
9752023-12-07T08:27:10.984Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8d099513-dfdb-416c-9b0f-eccc61968f8c/root/var/svc/log/oxide-crucible-agent:default.log (14120 bytes)
9762023-12-07T08:27:12.584Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8d099513-dfdb-416c-9b0f-eccc61968f8c/root/var/svc/log/oxide-crucible-agent:default.log
9772023-12-07T08:27:12.585Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8d099513-dfdb-416c-9b0f-eccc61968f8c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9782023-12-07T08:27:13.746Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8d099513-dfdb-416c-9b0f-eccc61968f8c/root/var/svc/log/oxide-crucible-downstairs:default.log
9792023-12-07T08:27:13.783Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8d099513-dfdb-416c-9b0f-eccc61968f8c/root/var/svc/log/oxide-crucible-downstairs:downstairs-508fb493-696d-4542-b035-3345ccf4cb1b.log (5537 bytes)
9802023-12-07T08:27:14.900Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8d099513-dfdb-416c-9b0f-eccc61968f8c/root/var/svc/log/oxide-crucible-downstairs:downstairs-508fb493-696d-4542-b035-3345ccf4cb1b.log
9812023-12-07T08:27:14.901Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_d93bb6dd-567b-48a4-94e0-0dd7aeb6f00d/root/var/svc/log/oxide-oximeter:default.log (146946 bytes)
9822023-12-07T08:27:15.637Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_d93bb6dd-567b-48a4-94e0-0dd7aeb6f00d/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 146946 -> 169172
9832023-12-07T08:27:16.735Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_d93bb6dd-567b-48a4-94e0-0dd7aeb6f00d/root/var/svc/log/oxide-oximeter:default.log
9842023-12-07T08:27:16.737Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_fd6b9457-9c94-4762-8a89-9a004326a0be/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
9852023-12-07T08:27:17.875Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_fd6b9457-9c94-4762-8a89-9a004326a0be/root/var/svc/log/oxide-cockroachdb:default.log
9862023-12-07T08:27:17.875Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0811da43-bee6-4156-a4ed-e28b2eb9cfbe/root/var/svc/log/oxide-crucible-agent:default.log (14114 bytes)
9872023-12-07T08:27:19.009Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0811da43-bee6-4156-a4ed-e28b2eb9cfbe/root/var/svc/log/oxide-crucible-agent:default.log
9882023-12-07T08:27:19.009Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0811da43-bee6-4156-a4ed-e28b2eb9cfbe/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9892023-12-07T08:27:20.536Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0811da43-bee6-4156-a4ed-e28b2eb9cfbe/root/var/svc/log/oxide-crucible-downstairs:default.log
9902023-12-07T08:27:20.537Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0811da43-bee6-4156-a4ed-e28b2eb9cfbe/root/var/svc/log/oxide-crucible-downstairs:downstairs-4206964f-4db0-4eb3-823d-b0336a31c20e.log (5537 bytes)
9912023-12-07T08:27:21.663Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0811da43-bee6-4156-a4ed-e28b2eb9cfbe/root/var/svc/log/oxide-crucible-downstairs:downstairs-4206964f-4db0-4eb3-823d-b0336a31c20e.log
9922023-12-07T08:27:21.664Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_65560472-2bf7-4c92-9d46-408d44a11758/root/var/svc/log/oxide-external_dns:default.log (15612 bytes)
9932023-12-07T08:27:21.760Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_65560472-2bf7-4c92-9d46-408d44a11758/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15612 -> 19386
9942023-12-07T08:27:22.840Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_65560472-2bf7-4c92-9d46-408d44a11758/root/var/svc/log/oxide-external_dns:default.log
9952023-12-07T08:27:22.840Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_51d9d39b-4f17-4b4d-b17e-d496693e9054/root/var/svc/log/oxide-internal_dns:default.log (100293 bytes)
9962023-12-07T08:27:23.030Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_51d9d39b-4f17-4b4d-b17e-d496693e9054/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 100293 -> 104048
9972023-12-07T08:27:24.109Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_51d9d39b-4f17-4b4d-b17e-d496693e9054/root/var/svc/log/oxide-internal_dns:default.log
9982023-12-07T08:27:24.136Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a8803964-51ef-4533-9a85-80475b03f80a/root/var/svc/log/oxide-internal_dns:default.log (98271 bytes)
9992023-12-07T08:27:24.927Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a8803964-51ef-4533-9a85-80475b03f80a/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 98271 -> 102026
10002023-12-07T08:27:26.016Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a8803964-51ef-4533-9a85-80475b03f80a/root/var/svc/log/oxide-internal_dns:default.log
10012023-12-07T08:27:26.017Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_a22b592e-5d84-49d6-98b4-e5f1f2ecebad/root/var/svc/log/oxide-clickhouse:default.log (503667 bytes)
10022023-12-07T08:27:26.945Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_a22b592e-5d84-49d6-98b4-e5f1f2ecebad/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 503667 -> 591572
10032023-12-07T08:27:28.435Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_a22b592e-5d84-49d6-98b4-e5f1f2ecebad/root/var/svc/log/oxide-clickhouse:default.log
10042023-12-07T08:27:28.436Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_55c51a1c-4dc5-4e6e-85dc-241e20fea4ac/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10052023-12-07T08:27:29.605Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_55c51a1c-4dc5-4e6e-85dc-241e20fea4ac/root/var/svc/log/oxide-crucible-pantry:default.log
10062023-12-07T08:27:29.605Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_33ee6240-6e91-4118-82a9-0d6379afde38/root/var/svc/log/oxide-internal_dns:default.log (101109 bytes)
10072023-12-07T08:27:29.631Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_33ee6240-6e91-4118-82a9-0d6379afde38/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 101109 -> 104863
10082023-12-07T08:27:30.469Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_33ee6240-6e91-4118-82a9-0d6379afde38/root/var/svc/log/oxide-internal_dns:default.log
10092023-12-07T08:27:30.469Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5e0a299e-1c84-436f-9f7a-d1581994408c/root/var/svc/log/oxide-nexus:default.log (5027000 bytes)
10102023-12-07T08:27:33.660Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5e0a299e-1c84-436f-9f7a-d1581994408c/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 5027000 -> 5102532
10112023-12-07T08:27:34.750Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5e0a299e-1c84-436f-9f7a-d1581994408c/root/var/svc/log/oxide-nexus:default.log
10122023-12-07T08:27:34.751Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86/root/var/log/chrony/measurements.log (15892 bytes)
10132023-12-07T08:27:35.198Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86/root/var/log/chrony/measurements.log" changed size mid upload: 15892 -> 18084
10142023-12-07T08:27:35.926Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86/root/var/log/chrony/measurements.log
10152023-12-07T08:27:35.926Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86/root/var/log/chrony/statistics.log (9477 bytes)
10162023-12-07T08:27:36.013Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86/root/var/log/chrony/statistics.log" changed size mid upload: 9477 -> 11349
10172023-12-07T08:27:37.106Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86/root/var/log/chrony/statistics.log
10182023-12-07T08:27:37.107Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86/root/var/log/chrony/tracking.log (1456 bytes)
10192023-12-07T08:27:37.203Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86/root/var/log/chrony/tracking.log" changed size mid upload: 1456 -> 1588
10202023-12-07T08:27:38.291Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_941c06fd-46b4-44e3-859a-83d71dc2ac86/root/var/log/chrony/tracking.log