01HH0SDWPMCH4N45F4JRN1JM9Q: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HH0SK3MEMW2QX0P7NBAYWZ0A

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-07T00:57:08.320Zjob dependencies complete; ready to run (waiting for 49 m 56 s)
22023-12-07T00:57:08.397Zjob assigned to worker 01HH0VY9EGK91VA8X86DA1FE42 (queued for 0 s)
32023-12-07T00:57:08.482Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-12-07T00:57:08.541Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-12-07T00:57:09.211Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-12-07T00:57:09.257Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-07T00:57:09.335Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-12-07T00:57:09.376ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-07T00:57:09.436Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-07T00:57:09.494ZOxide Helios Version helios-2.0.22318 64-bit
112023-12-07T00:57:09.565ZConfiguring devices.
122023-12-07T00:57:09.621Z#################################################
132023-12-07T00:57:09.660Z#### ############# ##
142023-12-07T00:57:09.704Z### ### ############ ##
152023-12-07T00:57:09.756Z## ### # ## ### ## Oxide Computer Company ##
162023-12-07T00:57:09.809Z## ## ## ### # ### ##
172023-12-07T00:57:09.856Z## # ### #### #### This Station Under ##
182023-12-07T00:57:09.912Z### ### #### # ### Computer Control ##
192023-12-07T00:57:09.956Z#### #### ### ## ##
202023-12-07T00:57:10.003Z##################################################
212023-12-07T00:57:10.067ZHostname: buskin
222023-12-07T00:57:10.117ZLoading smf(7) serice descriptions:
232023-12-07T00:57:11.379Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a.gz
242023-12-07T00:57:14.430Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a.gz
252023-12-07T00:57:14.431Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-07T00:57:43.079Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-07T00:57:43.079Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4.gz
282023-12-07T00:58:16.831Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4.gz
292023-12-07T00:58:16.858Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-07T00:58:18.395Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-07T00:58:18.421Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-07T00:58:22.416Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-07T00:58:22.449Zdownloading input: /input/package/work/version.txt
342023-12-07T00:58:22.449Zdownloaded input: /input/package/work/version.txt
352023-12-07T00:58:22.449Zdownloading input: /input/package/work/package.tar.gz
362023-12-07T00:58:46.547Zdownloaded input: /input/package/work/package.tar.gz
372023-12-07T00:58:46.547Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-07T00:59:39.837Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-07T00:59:39.838Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-07T00:59:55.227Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-07T00:59:55.255Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-07T01:00:41.079Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-07T01:00:41.080Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-07T01:01:36.964Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-07T01:01:36.964Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-07T01:02:19.463Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-07T01:02:19.463Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-07T01:02:28.726Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-07T01:02:28.726Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-07T01:02:37.617Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-07T01:02:37.618Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-07T01:02:42.820Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-07T01:02:42.820Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-07T01:02:48.739Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-07T01:02:48.739Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-07T01:02:48.800Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-07T01:02:48.801Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-07T01:02:52.203Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-07T01:02:52.637Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-07T01:02:56.706Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-07T01:02:56.706Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-07T01:03:18.182Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-07T01:03:18.182Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-07T01:03:52.659Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-07T01:03:52.660Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-07T01:03:53.155Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-07T01:03:53.155Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-07T01:03:59.707Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-07T01:03:59.707Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-07T01:04:33.802Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-07T01:04:33.803Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-07T01:05:32.340Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-07T01:05:32.366Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-07T01:06:11.364Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-07T01:06:11.365Zstarting task 0: "setup"
762023-12-07T01:06:11.396Z++ uname -s
772023-12-07T01:06:11.396Z+ kern=SunOS
782023-12-07T01:06:11.396Z+ case "$kern" in
792023-12-07T01:06:11.396Z+ groupadd -g 12345 build
802023-12-07T01:06:11.396Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-07T01:06:13.382Z+ zfs create -o mountpoint=/work rpool/work
822023-12-07T01:06:13.415Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-07T01:06:13.415Z+ home_fs=zfs
842023-12-07T01:06:13.415Z+ [[ zfs == autofs ]]
852023-12-07T01:06:13.416Z+ mkdir -p /home/build
862023-12-07T01:06:13.417Z+ chown build:build /home/build /work
872023-12-07T01:06:15.407Z+ chmod 0700 /home/build /work
882023-12-07T01:06:15.433Zprocess exited: duration 4047 ms, exit code 0
 
892023-12-07T01:06:15.501Zstarting task 1: "authentication"
902023-12-07T01:06:15.570Zprocess exited: duration 50 ms, exit code 0
 
912023-12-07T01:06:15.625Zstarting task 2: "build"
922023-12-07T01:06:15.654Z+ trap _exit_trap EXIT
932023-12-07T01:06:15.654Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-07T01:06:15.830Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-07T01:06:15.830Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-07T01:06:15.830Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-07T01:06:15.830Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-07T01:06:15.830Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-07T01:06:15.830Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-07T01:06:15.830Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-07T01:06:15.831Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-07T01:06:15.954Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-07T01:06:15.954Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-07T01:06:15.954Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-07T01:06:15.955Z+ [[ -d /opt/oxide ]]
1062023-12-07T01:06:15.955Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-07T01:06:15.955Z+ pfexec mkdir /opt/oxide
1082023-12-07T01:06:15.955Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-07T01:06:15.955Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-07T01:06:15.955Z+ cd /opt/oxide-underneath
1112023-12-07T01:06:15.955Z+ pfexec tar ceEp@/f - .
1122023-12-07T01:06:15.955Z+ cd /opt/oxide
1132023-12-07T01:06:15.955Z+ pfexec tar xveEp@/f -
1142023-12-07T01:06:15.993Zx ., 0 bytes, 0 tape blocks
1152023-12-07T01:06:15.993Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-07T01:06:15.993Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-07T01:06:15.994Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-12-07T01:06:15.994Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-07T01:06:16.012Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-07T01:06:16.048Z+ pfexec mkdir /opt/oxide/work
1212023-12-07T01:06:16.048Z+ pfexec chown build:build /opt/oxide/work
1222023-12-07T01:06:16.048Z+ cd /opt/oxide/work
1232023-12-07T01:06:16.048Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-07T01:06:25.794ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-12-07T01:06:25.794Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-12-07T01:06:25.794Zx out/omicron-sled-agent.tar, 111087104 bytes, 216967 tape blocks
1272023-12-07T01:06:25.794Zx out/target/test, 75 bytes, 1 tape blocks
1282023-12-07T01:06:25.795Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-12-07T01:06:25.795Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-12-07T01:06:25.795Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-12-07T01:06:25.796Zx target/release/omicron-package, 42582768 bytes, 83170 tape blocks
1322023-12-07T01:06:25.796Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-12-07T01:06:25.797Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-12-07T01:06:25.797Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-12-07T01:06:25.797Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-12-07T01:06:25.797Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-12-07T01:06:25.797Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-12-07T01:06:25.797Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-12-07T01:06:25.992Z
1402023-12-07T01:06:25.992Zreal 1.536261275
1412023-12-07T01:06:25.992Zuser 1.383688490
1422023-12-07T01:06:25.992Zsys 0.266893565
1432023-12-07T01:06:25.992Ztrap 0.000058819
1442023-12-07T01:06:25.992Ztflt 0.000000000
1452023-12-07T01:06:25.992Zdflt 0.000000000
1462023-12-07T01:06:25.992Zkflt 0.000000000
1472023-12-07T01:06:25.992Zlock 0.000000000
1482023-12-07T01:06:25.992Zslp 1.384696948
1492023-12-07T01:06:25.992Zlat 0.035169578
1502023-12-07T01:06:25.992Zstop 0.000115289
1512023-12-07T01:06:25.992Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-12-07T01:06:25.992Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-12-07T01:06:25.992Z+ mkdir tests
1542023-12-07T01:06:25.993Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-12-07T01:06:25.993Z+ ptime -m gunzip
1562023-12-07T01:06:25.993Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a
1572023-12-07T01:06:25.993Z
1582023-12-07T01:06:25.993Zreal 0.138183821
1592023-12-07T01:06:25.993Zuser 0.126937512
1602023-12-07T01:06:25.993Zsys 0.010999486
1612023-12-07T01:06:25.994Ztrap 0.000013109
1622023-12-07T01:06:25.994Ztflt 0.000000000
1632023-12-07T01:06:25.994Zdflt 0.000000000
1642023-12-07T01:06:25.994Zkflt 0.000000000
1652023-12-07T01:06:25.994Zlock 0.000000000
1662023-12-07T01:06:25.994Zslp 0.000023729
1672023-12-07T01:06:25.994Zlat 0.000022200
1682023-12-07T01:06:25.994Zstop 0.000124790
1692023-12-07T01:06:25.994Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a
1702023-12-07T01:06:25.994Z+ chmod a+x tests/bootstrap-be9c817b94af4e8a
1712023-12-07T01:06:25.994Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-12-07T01:06:25.994Z+ ptime -m gunzip
1732023-12-07T01:06:25.994Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-12-07T01:06:25.994Z
1752023-12-07T01:06:25.994Zreal 2.193396500
1762023-12-07T01:06:25.994Zuser 2.054523972
1772023-12-07T01:06:25.994Zsys 0.138501657
1782023-12-07T01:06:25.995Ztrap 0.000143229
1792023-12-07T01:06:25.995Ztflt 0.000000000
1802023-12-07T01:06:25.995Zdflt 0.000000000
1812023-12-07T01:06:25.995Zkflt 0.000000000
1822023-12-07T01:06:25.995Zlock 0.000000000
1832023-12-07T01:06:25.995Zslp 0.000023160
1842023-12-07T01:06:25.995Zlat 0.000035270
1852023-12-07T01:06:25.995Zstop 0.000113040
1862023-12-07T01:06:25.995Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-12-07T01:06:25.995Z+ chmod a+x tests/bootstrap
1882023-12-07T01:06:25.995Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-12-07T01:06:25.995Z+ ptime -m gunzip
1902023-12-07T01:06:25.995Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4
1912023-12-07T01:06:25.995Z
1922023-12-07T01:06:25.995Zreal 2.263508055
1932023-12-07T01:06:25.995Zuser 2.117941575
1942023-12-07T01:06:25.995Zsys 0.145177838
1952023-12-07T01:06:25.996Ztrap 0.000149639
1962023-12-07T01:06:25.996Ztflt 0.000000000
1972023-12-07T01:06:25.996Zdflt 0.000000000
1982023-12-07T01:06:25.996Zkflt 0.000000000
1992023-12-07T01:06:25.996Zlock 0.000000000
2002023-12-07T01:06:25.996Zslp 0.000021420
2012023-12-07T01:06:25.996Zlat 0.000029929
2022023-12-07T01:06:25.997Zstop 0.000121301
2032023-12-07T01:06:25.997Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4
2042023-12-07T01:06:25.997Z+ chmod a+x tests/end_to_end_tests-ff923d3f5c6228d4
2052023-12-07T01:06:25.998Z+ bmat address ls
2062023-12-07T01:06:25.998ZNAME CIDR FIRST LAST COUNT
2072023-12-07T01:06:25.998Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-12-07T01:06:25.998Z++ bmat address ls -f extra -Ho first,last,count
2092023-12-07T01:06:25.998Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-12-07T01:06:25.998Z+ EXTRA_IP_START=10.151.1.164
2112023-12-07T01:06:25.998Z+ EXTRA_IP_END=10.151.1.227
2122023-12-07T01:06:25.998Z+ EXTRA_IP_COUNT=64
2132023-12-07T01:06:25.999Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-12-07T01:06:25.999Z+ EXTRA_IP_BASE=10.151.1
2152023-12-07T01:06:25.999Z+ EXTRA_IP_FOCTET=164
2162023-12-07T01:06:25.999Z+ EXTRA_IP_LOCTET=227
2172023-12-07T01:06:25.999Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-12-07T01:06:25.999Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-12-07T01:06:25.999Z+ DNS_IP1=10.151.1.164
2202023-12-07T01:06:25.999Z+ DNS_IP2=10.151.1.165
2212023-12-07T01:06:25.999Z+ UPLINK_IP=10.151.1.174
2222023-12-07T01:06:25.999Z+ IPPOOL_START=10.151.1.175
2232023-12-07T01:06:26.000Z+ IPPOOL_END=10.151.1.227
2242023-12-07T01:06:26.030Z++ ipadm show-addr -po type,addr
2252023-12-07T01:06:26.034Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-12-07T01:06:26.034Z+ GATEWAY_IP=10.151.1.100
2272023-12-07T01:06:26.034Z+ [[ -n 10.151.1.100 ]]
2282023-12-07T01:06:26.035Z+ ping -s 10.151.1.100 56 1
2292023-12-07T01:06:26.035ZPING 10.151.1.100: 56 data bytes
2302023-12-07T01:06:26.035Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
2312023-12-07T01:06:26.035Z
2322023-12-07T01:06:26.035Z----10.151.1.100 PING Statistics----
2332023-12-07T01:06:26.035Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-12-07T01:06:26.035Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-12-07T01:06:26.035Z++ arp -an
2362023-12-07T01:06:26.036Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-12-07T01:06:26.036Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-12-07T01:06:26.036Z+ routeadm -e ipv4-forwarding -u
2392023-12-07T01:06:26.036Z+ PXA_START=10.151.1.164
2402023-12-07T01:06:26.036Z+ PXA_END=10.151.1.227
2412023-12-07T01:06:26.037Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-12-07T01:06:26.037Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-12-07T01:06:26.106Z+ ZPOOL_VDEV_DIR=/scratch
2442023-12-07T01:06:26.106Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-12-07T01:06:26.133Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-12-07T01:06:26.133Z++ cd ./tools
2472023-12-07T01:06:26.133Z++ pwd
2482023-12-07T01:06:26.133Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-12-07T01:06:26.133Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-12-07T01:06:26.133Z+ SOFTNPU_MODE=zone
2512023-12-07T01:06:26.133Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-12-07T01:06:26.133Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-12-07T01:06:26.133Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-12-07T01:06:26.133Z++ dladm show-phys -p -o LINK
2552023-12-07T01:06:26.133Z++ head -1
2562023-12-07T01:06:26.241Z+ PHYSICAL_LINK=igb0
2572023-12-07T01:06:26.241Z+ echo 'Using igb0 as physical link'
2582023-12-07T01:06:26.410ZUsing igb0 as physical link
2592023-12-07T01:06:26.410Z+ ensure_run_as_root
2602023-12-07T01:06:26.410Z++ id -u
2612023-12-07T01:06:26.410Z+ [[ 0 -ne 0 ]]
2622023-12-07T01:06:26.410Z+ ensure_zpools
2632023-12-07T01:06:26.411Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-12-07T01:06:26.411Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-12-07T01:06:26.411Z+ readarray -t ZPOOLS
2662023-12-07T01:06:26.411Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-12-07T01:06:26.411Z++ sed 's/[ ",]//g'
2682023-12-07T01:06:26.411Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-12-07T01:06:26.411Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-12-07T01:06:26.411ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-12-07T01:06:26.411Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-12-07T01:06:26.411Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-12-07T01:06:26.411Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-12-07T01:06:26.411Z0+0 records in
2752023-12-07T01:06:26.411Z0+0 records out
2762023-12-07T01:06:26.411Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2772023-12-07T01:06:26.411Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-12-07T01:06:26.411Z+ set +x
2792023-12-07T01:06:26.411ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-12-07T01:06:26.412Z++ zpool list -o name
2812023-12-07T01:06:26.412Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-12-07T01:06:26.414Z+ [[ -z '' ]]
2832023-12-07T01:06:26.414Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-12-07T01:06:26.504Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-12-07T01:06:26.504Z+ set +x
2862023-12-07T01:06:26.504ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-12-07T01:06:26.505Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-12-07T01:06:26.505Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-12-07T01:06:26.505ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-12-07T01:06:26.505Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-12-07T01:06:26.505Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-12-07T01:06:26.505Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-12-07T01:06:26.505Z0+0 records in
2942023-12-07T01:06:26.506Z0+0 records out
2952023-12-07T01:06:26.506Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-12-07T01:06:26.506Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-12-07T01:06:26.506Z+ set +x
2982023-12-07T01:06:26.506ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-12-07T01:06:26.506Z++ zpool list -o name
3002023-12-07T01:06:26.506Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-12-07T01:06:26.507Z+ [[ -z '' ]]
3022023-12-07T01:06:26.507Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-12-07T01:06:26.507Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-12-07T01:06:26.507Z+ set +x
3052023-12-07T01:06:26.507ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-12-07T01:06:26.507Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-12-07T01:06:26.507Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-12-07T01:06:26.507ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3092023-12-07T01:06:26.509Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-12-07T01:06:26.509Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-12-07T01:06:26.509Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-12-07T01:06:26.509Z0+0 records in
3132023-12-07T01:06:26.509Z0+0 records out
3142023-12-07T01:06:26.509Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3152023-12-07T01:06:26.509Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-12-07T01:06:26.509Z+ set +x
3172023-12-07T01:06:26.509ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-12-07T01:06:26.509Z++ zpool list -o name
3192023-12-07T01:06:26.509Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-12-07T01:06:26.509Z+ [[ -z '' ]]
3212023-12-07T01:06:26.510Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-12-07T01:06:26.510Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-12-07T01:06:26.510Z+ set +x
3242023-12-07T01:06:26.510ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-12-07T01:06:26.510Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-12-07T01:06:26.510Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-12-07T01:06:26.510ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-12-07T01:06:26.510Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-12-07T01:06:26.510Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-12-07T01:06:26.510Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-12-07T01:06:26.510Z0+0 records in
3322023-12-07T01:06:26.510Z0+0 records out
3332023-12-07T01:06:26.510Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-12-07T01:06:26.510Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-12-07T01:06:26.510Z+ set +x
3362023-12-07T01:06:26.510ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-12-07T01:06:26.510Z++ zpool list -o name
3382023-12-07T01:06:26.510Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-12-07T01:06:26.511Z+ [[ -z '' ]]
3402023-12-07T01:06:26.511Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-12-07T01:06:26.601Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-12-07T01:06:26.601Z+ set +x
3432023-12-07T01:06:26.601ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-12-07T01:06:26.601ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3452023-12-07T01:06:26.601Z+ for ZPOOL in "${ZPOOLS[@]}"
3462023-12-07T01:06:26.601Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3472023-12-07T01:06:26.601Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-12-07T01:06:26.601Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-12-07T01:06:26.601Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-12-07T01:06:26.601Z0+0 records in
3512023-12-07T01:06:26.602Z0+0 records out
3522023-12-07T01:06:26.602Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3532023-12-07T01:06:26.602Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-12-07T01:06:26.602Z+ set +x
3552023-12-07T01:06:26.603ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-12-07T01:06:26.603Z++ zpool list -o name
3572023-12-07T01:06:26.603Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-12-07T01:06:26.603Z+ [[ -z '' ]]
3592023-12-07T01:06:26.603Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-12-07T01:06:26.603Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-12-07T01:06:26.603Z+ set +x
3622023-12-07T01:06:26.603ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-12-07T01:06:26.603Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-12-07T01:06:26.603Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-12-07T01:06:26.603ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-12-07T01:06:26.603Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-12-07T01:06:26.603Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-12-07T01:06:26.603Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-12-07T01:06:26.603Z0+0 records in
3702023-12-07T01:06:26.603Z0+0 records out
3712023-12-07T01:06:26.603Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3722023-12-07T01:06:26.603Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-12-07T01:06:26.604Z+ set +x
3742023-12-07T01:06:26.604ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-12-07T01:06:26.604Z++ zpool list -o name
3762023-12-07T01:06:26.604Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-12-07T01:06:26.604Z+ [[ -z '' ]]
3782023-12-07T01:06:26.604Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-12-07T01:06:26.665Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-12-07T01:06:26.665Z+ set +x
3812023-12-07T01:06:26.665ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-12-07T01:06:26.665Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-12-07T01:06:26.665Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-12-07T01:06:26.665ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-12-07T01:06:26.665Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-12-07T01:06:26.665Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-12-07T01:06:26.666Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-12-07T01:06:26.666Z0+0 records in
3892023-12-07T01:06:26.666Z0+0 records out
3902023-12-07T01:06:26.666Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-12-07T01:06:26.666Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-12-07T01:06:26.666Z+ set +x
3932023-12-07T01:06:26.666ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-12-07T01:06:26.666Z++ zpool list -o name
3952023-12-07T01:06:26.667Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-12-07T01:06:26.667Z+ [[ -z '' ]]
3972023-12-07T01:06:26.667Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-12-07T01:06:26.667Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-12-07T01:06:26.668Z+ set +x
4002023-12-07T01:06:26.668ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-12-07T01:06:26.668Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-12-07T01:06:26.669Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-12-07T01:06:26.669ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-12-07T01:06:26.669Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-12-07T01:06:26.669Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-12-07T01:06:26.669Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-12-07T01:06:26.669Z0+0 records in
4082023-12-07T01:06:26.669Z0+0 records out
4092023-12-07T01:06:26.669Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-12-07T01:06:26.670Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-12-07T01:06:26.670Z+ set +x
4122023-12-07T01:06:26.670ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-12-07T01:06:26.670Z++ zpool list -o name
4142023-12-07T01:06:26.670Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-12-07T01:06:26.670Z+ [[ -z '' ]]
4162023-12-07T01:06:26.670Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-12-07T01:06:26.710Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-12-07T01:06:26.710Z+ set +x
4192023-12-07T01:06:26.710ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-12-07T01:06:26.710Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-12-07T01:06:26.710Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-12-07T01:06:26.710ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-12-07T01:06:26.710Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-12-07T01:06:26.710Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-12-07T01:06:26.710Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-12-07T01:06:26.712Z0+0 records in
4272023-12-07T01:06:26.712Z0+0 records out
4282023-12-07T01:06:26.712Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4292023-12-07T01:06:26.712Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-12-07T01:06:26.712Z+ set +x
4312023-12-07T01:06:26.712ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-12-07T01:06:26.712Z++ zpool list -o name
4332023-12-07T01:06:26.712Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-12-07T01:06:26.713Z+ [[ -z '' ]]
4352023-12-07T01:06:26.713Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-12-07T01:06:26.713Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-12-07T01:06:26.713Z+ set +x
4382023-12-07T01:06:26.713ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-12-07T01:06:26.713Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-12-07T01:06:26.713Z+ readarray -t ZPOOLS
4412023-12-07T01:06:26.713Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-12-07T01:06:26.713Z++ sed 's/[ ",]//g'
4432023-12-07T01:06:26.713Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-12-07T01:06:26.713ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4452023-12-07T01:06:26.713Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4462023-12-07T01:06:26.713Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-12-07T01:06:26.713Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-12-07T01:06:26.713Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-12-07T01:06:26.713Z0+0 records in
4502023-12-07T01:06:26.713Z0+0 records out
4512023-12-07T01:06:26.714Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4522023-12-07T01:06:26.714Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-12-07T01:06:26.714Z+ set +x
4542023-12-07T01:06:26.714ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-12-07T01:06:26.714Z++ zpool list -o name
4562023-12-07T01:06:26.715Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-12-07T01:06:26.790Z+ [[ -z '' ]]
4582023-12-07T01:06:26.790Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-12-07T01:06:26.816Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-12-07T01:06:26.816Z+ set +x
4612023-12-07T01:06:26.816ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-12-07T01:06:26.816Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-12-07T01:06:26.816Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-12-07T01:06:26.816ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-12-07T01:06:26.816Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-12-07T01:06:26.816Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-12-07T01:06:26.816Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-12-07T01:06:26.816Z0+0 records in
4692023-12-07T01:06:26.816Z0+0 records out
4702023-12-07T01:06:26.816Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4712023-12-07T01:06:26.817Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-12-07T01:06:26.817Z+ set +x
4732023-12-07T01:06:26.817ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-12-07T01:06:26.817Z++ zpool list -o name
4752023-12-07T01:06:26.817Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-12-07T01:06:26.818Z+ [[ -z '' ]]
4772023-12-07T01:06:26.818Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-12-07T01:06:26.818Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-12-07T01:06:26.818Z+ set +x
4802023-12-07T01:06:26.818ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-12-07T01:06:26.818Z+ [[ zone == \z\o\n\e ]]
4822023-12-07T01:06:26.818Z+ ensure_simulated_links igb0
4832023-12-07T01:06:26.818Z+ local PHYSICAL_LINK=igb0
4842023-12-07T01:06:26.818Z+ INDICES=("0" "1")
4852023-12-07T01:06:26.818Z+ for I in "${INDICES[@]}"
4862023-12-07T01:06:26.818Z++ get_simnet_name_if_exists net0
4872023-12-07T01:06:26.818Z++ dladm show-simnet -p -o LINK net0
4882023-12-07T01:06:26.818Z++ echo ''
4892023-12-07T01:06:26.818Z+ [[ -z '' ]]
4902023-12-07T01:06:26.818Z+ dladm create-simnet -t net0
4912023-12-07T01:06:26.825Z+ dladm create-simnet -t sc0_0
4922023-12-07T01:06:26.951Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-12-07T01:06:26.952Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-12-07T01:06:26.952Z+ success 'Simnet net0/sc0_0 exists'
4952023-12-07T01:06:26.952Z+ set +x
4962023-12-07T01:06:26.952ZSimnet net0/sc0_0 exists
4972023-12-07T01:06:26.952Z+ for I in "${INDICES[@]}"
4982023-12-07T01:06:26.952Z++ get_simnet_name_if_exists net1
4992023-12-07T01:06:26.952Z++ dladm show-simnet -p -o LINK net1
5002023-12-07T01:06:26.952Z++ echo ''
5012023-12-07T01:06:26.952Z+ [[ -z '' ]]
5022023-12-07T01:06:26.952Z+ dladm create-simnet -t net1
5032023-12-07T01:06:26.952Z+ dladm create-simnet -t sc1_0
5042023-12-07T01:06:26.952Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-12-07T01:06:26.952Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-12-07T01:06:26.952Z+ success 'Simnet net1/sc1_0 exists'
5072023-12-07T01:06:26.952Z+ set +x
5082023-12-07T01:06:26.952ZSimnet net1/sc1_0 exists
5092023-12-07T01:06:26.953Z++ get_vnic_name_if_exists sc0_1
5102023-12-07T01:06:26.953Z++ dladm show-vnic -p -o LINK sc0_1
5112023-12-07T01:06:26.953Z++ echo ''
5122023-12-07T01:06:26.953Z+ [[ -z '' ]]
5132023-12-07T01:06:26.953Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-12-07T01:06:26.953Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-12-07T01:06:26.953Z+ success 'Vnic sc0_1 exists'
5162023-12-07T01:06:26.953Z+ set +x
5172023-12-07T01:06:26.953ZVnic sc0_1 exists
5182023-12-07T01:06:26.953Z+ warn_if_no_proxy_arp
5192023-12-07T01:06:26.953Z+ [[ -v PXA_START ]]
5202023-12-07T01:06:26.953Z+ [[ -v PXA_END ]]
5212023-12-07T01:06:26.953Z+ ensure_softnpu_zone
5222023-12-07T01:06:26.953Z+ zoneadm list
5232023-12-07T01:06:26.953Z+ grep -q sidecar_softnpu
5242023-12-07T01:06:26.953Z+ [[ -f out/npuzone/npuzone ]]
5252023-12-07T01:06:26.953Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5262023-12-07T01:06:27.066Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
5272023-12-07T01:06:33.708Zdownloading softnpu ...done
5282023-12-07T01:06:33.737Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5292023-12-07T01:06:50.754Zdownloading asic_program.so ...done
5302023-12-07T01:06:51.038Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-12-07T01:06:58.775Zdownloading scadm ...done
5322023-12-07T01:06:58.802Zcreating port tfportrear0_0 for sc0_0
5332023-12-07T01:06:58.802Zcreating port tfportqsfp0_0 for sc0_1
5342023-12-07T01:06:58.802Zsoftnpu zone setup
5352023-12-07T01:06:58.802Zconfigure zone
5362023-12-07T01:06:58.828Zinstall zone
5372023-12-07T01:06:59.854Zboot zone
5382023-12-07T01:07:02.961Z[sidecar_softnpu] svcs milestone/network
5392023-12-07T01:07:03.406ZSTATE STIME FMRI
5402023-12-07T01:07:03.406Zonline 1:07:00 svc:/milestone/network:default
5412023-12-07T01:07:03.406Z[sidecar_softnpu] /softnpu/npu start
5422023-12-07T01:07:03.477Z
5432023-12-07T01:07:03.477Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-12-07T01:07:03.477Z+ GATEWAY_IP=10.151.1.100
5452023-12-07T01:07:03.478Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-12-07T01:07:03.478ZUsing 10.151.1.100 as gateway ip
5472023-12-07T01:07:03.478Z+ [[ ! -v GATEWAY_MAC ]]
5482023-12-07T01:07:03.478Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-12-07T01:07:03.478Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-12-07T01:07:03.478Z+++ echo 18:c0:4d:0d:9f:b2
5512023-12-07T01:07:03.478Z+ COUNT=1
5522023-12-07T01:07:03.478Z+ [[ 1 -ne 1 ]]
5532023-12-07T01:07:03.478Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-12-07T01:07:03.479ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-12-07T01:07:03.479Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-12-07T01:07:03.479Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-12-07T01:07:03.480Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-12-07T01:07:03.480Z+ [[ -v PXA_START ]]
5592023-12-07T01:07:03.480Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-12-07T01:07:03.480Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-12-07T01:07:03.480Z+ z_scadm dump-state
5622023-12-07T01:07:03.481Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-12-07T01:07:03.481Zlocal v6:
5642023-12-07T01:07:03.481Zlocal v4:
5652023-12-07T01:07:03.481Zrouter v6:
5662023-12-07T01:07:03.481Zrouter v4_idx:
5672023-12-07T01:07:03.481Zrouter v4_routes:
5682023-12-07T01:07:03.482Zresolver v4:
5692023-12-07T01:07:03.482Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-12-07T01:07:03.482Zresolver v6:
5712023-12-07T01:07:03.482Znat v4:
5722023-12-07T01:07:03.482Znat_v6:
5732023-12-07T01:07:03.482Zport_mac:
5742023-12-07T01:07:03.482Zproxy arp:
5752023-12-07T01:07:03.482Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-12-07T01:07:03.483Z+ success 'softnpu zone exists'
5772023-12-07T01:07:03.483Z+ set +x
5782023-12-07T01:07:03.483Zsoftnpu zone exists
5792023-12-07T01:07:03.483Z
5802023-12-07T01:07:03.483Zreal 36.998416036
5812023-12-07T01:07:03.483Zuser 0.708615919
5822023-12-07T01:07:03.483Zsys 1.625095829
5832023-12-07T01:07:03.484Ztrap 0.000095854
5842023-12-07T01:07:03.484Ztflt 0.000141543
5852023-12-07T01:07:03.484Zdflt 0.000803929
5862023-12-07T01:07:03.484Zkflt 0.000000000
5872023-12-07T01:07:03.491Zlock 28:52.103433542
5882023-12-07T01:07:03.491Zslp 1:50.775667942
5892023-12-07T01:07:03.491Zlat 0.075561853
5902023-12-07T01:07:03.491Zstop 0.352807356
5912023-12-07T01:07:03.492Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-12-07T01:07:03.492Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-12-07T01:07:03.492Z+ SILO_NAME=recovery
5942023-12-07T01:07:03.492Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-07T01:07:03.492Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-12-07T01:07:03.493Z+ sed -i~ '
5972023-12-07T01:07:03.493Z /^external_dns_ips/c\
5982023-12-07T01:07:03.493Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-12-07T01:07:03.493Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-12-07T01:07:03.493Z /^first/c\
6012023-12-07T01:07:03.493Zfirst = "10.151.1.164"
6022023-12-07T01:07:03.494Z /^last/c\
6032023-12-07T01:07:03.494Zlast = "10.151.1.173"
6042023-12-07T01:07:03.494Z }
6052023-12-07T01:07:03.494Z /^\[rack_network_config/,/^$/ {
6062023-12-07T01:07:03.494Z /^infra_ip_first/c\
6072023-12-07T01:07:03.495Zinfra_ip_first = "10.151.1.174"
6082023-12-07T01:07:03.495Z /^infra_ip_last/c\
6092023-12-07T01:07:03.495Zinfra_ip_last = "10.151.1.174"
6102023-12-07T01:07:03.495Z }
6112023-12-07T01:07:03.496Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-12-07T01:07:03.496Z /^routes/c\
6132023-12-07T01:07:03.496Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6142023-12-07T01:07:03.497Z /^addresses/c\
6152023-12-07T01:07:03.498Zaddresses = \["10.151.1.174/32"\]
6162023-12-07T01:07:03.498Z }
6172023-12-07T01:07:03.498Z' pkg/config-rss.toml
6182023-12-07T01:07:03.499Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-12-07T01:07:03.499Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-12-07T01:07:03.499Z+++ pkg/config-rss.toml Thu Dec 7 01:07:02 2023
6212023-12-07T01:07:03.499Z@@ -41,7 +41,7 @@
6222023-12-07T01:07:03.499Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-12-07T01:07:03.500Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-12-07T01:07:03.500Z # below.
6252023-12-07T01:07:03.500Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-12-07T01:07:03.500Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6272023-12-07T01:07:03.501Z
6282023-12-07T01:07:03.501Z # Initial TLS certificates for the external API
6292023-12-07T01:07:03.501Z #
6302023-12-07T01:07:03.501Z@@ -77,8 +77,8 @@
6312023-12-07T01:07:03.501Z #
6322023-12-07T01:07:03.501Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-12-07T01:07:03.501Z [[internal_services_ip_pool_ranges]]
6342023-12-07T01:07:03.501Z-first = "192.168.1.20"
6352023-12-07T01:07:03.501Z-last = "192.168.1.29"
6362023-12-07T01:07:03.501Z+first = "10.151.1.164"
6372023-12-07T01:07:03.501Z+last = "10.151.1.173"
6382023-12-07T01:07:03.502Z
6392023-12-07T01:07:03.502Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-12-07T01:07:03.502Z # parameters should be provided to the control plane via wicket, but we need to
6412023-12-07T01:07:03.505Z@@ -92,8 +92,8 @@
6422023-12-07T01:07:03.532Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-12-07T01:07:03.533Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-12-07T01:07:03.533Z # With softnpu, only one address is used.
6452023-12-07T01:07:03.533Z-infra_ip_first = "192.168.1.30"
6462023-12-07T01:07:03.533Z-infra_ip_last = "192.168.1.30"
6472023-12-07T01:07:03.533Z+infra_ip_first = "10.151.1.174"
6482023-12-07T01:07:03.533Z+infra_ip_last = "10.151.1.174"
6492023-12-07T01:07:03.533Z
6502023-12-07T01:07:03.534Z # Configurations for BGP routers to run on the scrimlets.
6512023-12-07T01:07:03.534Z bgp = []
6522023-12-07T01:07:03.534Z@@ -101,9 +101,9 @@
6532023-12-07T01:07:03.534Z # You can configure multiple uplinks by repeating the following stanza
6542023-12-07T01:07:03.534Z [[rack_network_config.ports]]
6552023-12-07T01:07:03.534Z # Routes associated with this port.
6562023-12-07T01:07:03.535Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-12-07T01:07:03.535Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6582023-12-07T01:07:03.535Z # Addresses associated with this port.
6592023-12-07T01:07:03.535Z-addresses = ["192.168.1.30/32"]
6602023-12-07T01:07:03.535Z+addresses = ["10.151.1.174/32"]
6612023-12-07T01:07:03.535Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-12-07T01:07:03.535Z port = "qsfp0"
6632023-12-07T01:07:03.535Z # The speed of this port.
6642023-12-07T01:07:03.536Z+ true
6652023-12-07T01:07:03.536Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-12-07T01:07:03.536Za pkg/config-rss.toml 7K
6672023-12-07T01:07:03.536Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-12-07T01:07:03.536Z+ TLS_NAME=recovery.sys.oxide.test
6692023-12-07T01:07:03.536Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-12-07T01:07:03.987Z.....+..+...............+..........+........+...+..........+.....+......+.+.....+............+...+...+.......+........+.+...+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.......+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.........+......+............+....+...+..+....+............+................................+...........................+...+...+.........+......+...+.+...+..+.+...........................+........+..........+...+..+..................+..........+......+.....+.........+.......+.....+...................+........+..........+......+......+...+..+...+......+.........+.+............+.........+..............+....+.....+.+...............+......+......+...+.....+................+......+.........+..............+.+..............+.........+.......+...+........+....+...+...+.....+.........+.+...............+...+.....+....+..+.+..............+.+.....+...................+.....+...+.+..+...+.+...+...........................+.....+........................+.+.....................+...+..+..................+....+...+..+............+................+..............+...+.......+.........+.....+..........+...........+..........+...+.....+............+.+...+...............+.....+.+.....+...+...+.........+...............+.+..................+..............+.......+.....+................+............+.....+...+...+.....................+.+...+..+....+..............+.......+............+...+........+.........+...+............+.......+.....+.........+....+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-12-07T01:07:04.155Z.+.........+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..........+.....+......+..........+............+...........+....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-07T01:07:04.186Z-----
6732023-12-07T01:07:04.186Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-12-07T01:07:04.186Za pkg/initial-tls-cert.pem 2K
6752023-12-07T01:07:04.186Za pkg/initial-tls-key.pem 4K
6762023-12-07T01:07:04.186Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-12-07T01:07:04.186Z+ rmdir pkg
6782023-12-07T01:07:04.186Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-12-07T01:07:04.190Z+ pfexec mkdir -p /usr/oxide
6802023-12-07T01:07:04.190Z+ pfexec rm -f /usr/oxide/tcpproxy
6812023-12-07T01:07:04.190Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6822023-12-07T01:07:04.863Z+ pfexec chmod +x /usr/oxide/tcpproxy
6832023-12-07T01:07:05.424Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6842023-12-07T01:07:05.424Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6852023-12-07T01:07:05.424Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6862023-12-07T01:07:05.424Z+ OMICRON_NO_UNINSTALL=1
6872023-12-07T01:07:05.424Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6882023-12-07T01:07:05.424ZDec 07 01:07:04.086 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6892023-12-07T01:07:05.425ZDec 07 01:07:04.087 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6902023-12-07T01:07:05.425ZDec 07 01:07:04.088 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6912023-12-07T01:07:05.425ZDec 07 01:07:04.088 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6922023-12-07T01:07:05.425ZDec 07 01:07:04.088 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6932023-12-07T01:07:05.425ZDec 07 01:07:04.088 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6942023-12-07T01:07:05.425ZDec 07 01:07:04.088 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6952023-12-07T01:07:05.425ZDec 07 01:07:04.088 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6962023-12-07T01:07:05.426ZDec 07 01:07:04.088 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6972023-12-07T01:07:05.432ZDec 07 01:07:04.088 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6982023-12-07T01:07:05.432ZDec 07 01:07:04.088 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6992023-12-07T01:07:05.432ZDec 07 01:07:04.088 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7002023-12-07T01:07:05.432ZDec 07 01:07:04.088 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7012023-12-07T01:07:05.432ZDec 07 01:07:04.088 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7022023-12-07T01:07:05.433ZDec 07 01:07:04.088 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7032023-12-07T01:07:05.433ZDec 07 01:07:04.088 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7042023-12-07T01:07:05.433ZDec 07 01:07:04.088 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
7052023-12-07T01:07:05.433ZDec 07 01:07:04.359 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-12-07T01:07:05.463ZDec 07 01:07:04.393 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-12-07T01:07:05.463ZDec 07 01:07:04.449 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-12-07T01:07:05.463Z
7092023-12-07T01:07:05.463Zreal 0.501515089
7102023-12-07T01:07:05.463Zuser 0.098563048
7112023-12-07T01:07:05.463Zsys 2.064346531
7122023-12-07T01:07:05.463Ztrap 0.000019069
7132023-12-07T01:07:05.464Ztflt 0.000448779
7142023-12-07T01:07:05.464Zdflt 0.000333556
7152023-12-07T01:07:05.464Zkflt 0.000002549
7162023-12-07T01:07:05.464Zlock 45.129749176
7172023-12-07T01:07:05.464Zslp 0.704957406
7182023-12-07T01:07:05.464Zlat 0.002913199
7192023-12-07T01:07:05.466Zstop 0.199323476
7202023-12-07T01:07:05.466Z+ retry=0
7212023-12-07T01:07:05.466Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-12-07T01:07:05.466Z+ [[ 0 -gt 30 ]]
7232023-12-07T01:07:05.466Z+ sleep 1
7242023-12-07T01:07:06.122Z+ retry=1
7252023-12-07T01:07:06.122Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-12-07T01:07:06.122Z+ [[ 1 -gt 30 ]]
7272023-12-07T01:07:06.122Z+ sleep 1
7282023-12-07T01:07:07.115Z+ retry=2
7292023-12-07T01:07:07.116Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-12-07T01:07:07.143Z+ [[ 2 -gt 30 ]]
7312023-12-07T01:07:07.143Z+ sleep 1
7322023-12-07T01:07:08.137Z+ retry=3
7332023-12-07T01:07:08.138Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-12-07T01:07:08.170Z+ [[ 3 -gt 30 ]]
7352023-12-07T01:07:08.170Z+ sleep 1
7362023-12-07T01:07:09.144Z+ retry=4
7372023-12-07T01:07:09.144Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-12-07T01:07:34.322Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7392023-12-07T01:07:34.359Zlocal v6:
7402023-12-07T01:07:34.359Zfd00:99::1
7412023-12-07T01:07:34.359Zfe80::8a:9bff:fea0:fe03
7422023-12-07T01:07:34.359Zlocal v4:
7432023-12-07T01:07:34.359Z10.151.1.174
7442023-12-07T01:07:34.359Zrouter v6:
7452023-12-07T01:07:34.359Zfd00:1122:3344:1::/64 -> fe80::857:40ff:feff:21c3 (1)
7462023-12-07T01:07:34.359Zfd00:1122:3344:101::/64 -> fe80::857:40ff:feff:21c3 (1)
7472023-12-07T01:07:34.360Zfd00:1122:3344:2::/64 -> fe80::857:40ff:feff:21c3 (1)
7482023-12-07T01:07:34.360Zfd00:1122:3344:3::/64 -> fe80::857:40ff:feff:21c3 (1)
7492023-12-07T01:07:34.360Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::857:40ff:feff:21c3 (1)
7502023-12-07T01:07:34.360Zrouter v4_idx:
7512023-12-07T01:07:34.360Z0.0.0.0/0 -> 0
7522023-12-07T01:07:34.360Zrouter v4_routes:
7532023-12-07T01:07:34.360Z0 -> 10.151.1.100 (2)
7542023-12-07T01:07:34.361Zresolver v4:
7552023-12-07T01:07:34.361Z10.151.1.174 -> e6:1e:f7:33:df:73
7562023-12-07T01:07:34.361Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7572023-12-07T01:07:34.377Zresolver v6:
7582023-12-07T01:07:34.377Zfe80::8a:9bff:fea0:fe03 -> 02:8a:9b:a0:fe:03
7592023-12-07T01:07:34.377Zfe80::857:40ff:feff:21c3 -> 0a:57:40:ff:21:c3
7602023-12-07T01:07:34.377Znat v4:
7612023-12-07T01:07:34.378Z10.151.1.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:a1:21
7622023-12-07T01:07:34.378Znat_v6:
7632023-12-07T01:07:34.378Zport_mac:
7642023-12-07T01:07:34.378Z1: a8:40:25:60:e0:3f
7652023-12-07T01:07:34.378Z1000: a8:40:25:60:e0:3e
7662023-12-07T01:07:34.378Zproxy arp:
7672023-12-07T01:07:34.378Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7682023-12-07T01:07:34.405Z+ retry=0
7692023-12-07T01:07:34.405Z+++ zoneadm list -n
7702023-12-07T01:07:34.405Z+++ grep oxz_ntp
7712023-12-07T01:07:34.405Z++ pfexec svcs -z oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b -Hostate oxide/ntp
7722023-12-07T01:07:34.405Zsvcs: Could not bind to repository server for zone oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b: repository server unavailable
7732023-12-07T01:07:34.405Z++ true
7742023-12-07T01:07:34.405Z+ [[ '' != online ]]
7752023-12-07T01:07:34.406Z+ [[ 0 -gt 60 ]]
7762023-12-07T01:07:34.406Z+ sleep 1
7772023-12-07T01:07:35.396Z+ retry=1
7782023-12-07T01:07:35.421Z+++ zoneadm list -n
7792023-12-07T01:07:35.426Z+++ grep oxz_ntp
7802023-12-07T01:07:35.426Z++ pfexec svcs -z oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b -Hostate oxide/ntp
7812023-12-07T01:07:35.459Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7822023-12-07T01:07:35.459Z++ true
7832023-12-07T01:07:35.459Z+ [[ '' != online ]]
7842023-12-07T01:07:35.459Z+ [[ 1 -gt 60 ]]
7852023-12-07T01:07:35.459Z+ sleep 1
7862023-12-07T01:07:36.427Z+ retry=2
7872023-12-07T01:07:36.461Z+++ zoneadm list -n
7882023-12-07T01:07:36.461Z+++ grep oxz_ntp
7892023-12-07T01:07:36.462Z++ pfexec svcs -z oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b -Hostate oxide/ntp
7902023-12-07T01:07:36.462Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7912023-12-07T01:07:36.462Z++ true
7922023-12-07T01:07:36.462Z+ [[ '' != online ]]
7932023-12-07T01:07:36.463Z+ [[ 2 -gt 60 ]]
7942023-12-07T01:07:36.463Z+ sleep 1
7952023-12-07T01:07:37.462Z+ retry=3
7962023-12-07T01:07:37.516Z+++ zoneadm list -n
7972023-12-07T01:07:37.516Z+++ grep oxz_ntp
7982023-12-07T01:07:37.516Z++ pfexec svcs -z oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b -Hostate oxide/ntp
7992023-12-07T01:07:37.516Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8002023-12-07T01:07:37.516Z++ true
8012023-12-07T01:07:37.516Z+ [[ '' != online ]]
8022023-12-07T01:07:37.516Z+ [[ 3 -gt 60 ]]
8032023-12-07T01:07:37.516Z+ sleep 1
8042023-12-07T01:07:38.542Z+ retry=4
8052023-12-07T01:07:38.612Z+++ zoneadm list -n
8062023-12-07T01:07:38.612Z+++ grep oxz_ntp
8072023-12-07T01:07:38.613Z++ pfexec svcs -z oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b -Hostate oxide/ntp
8082023-12-07T01:07:38.613Z+ [[ disabled != online ]]
8092023-12-07T01:07:38.613Z+ [[ 4 -gt 60 ]]
8102023-12-07T01:07:38.613Z+ sleep 1
8112023-12-07T01:07:39.508Z+ retry=5
8122023-12-07T01:07:39.535Z+++ zoneadm list -n
8132023-12-07T01:07:39.535Z+++ grep oxz_ntp
8142023-12-07T01:07:39.535Z++ pfexec svcs -z oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b -Hostate oxide/ntp
8152023-12-07T01:07:39.535Z+ [[ disabled != online ]]
8162023-12-07T01:07:39.535Z+ [[ 5 -gt 60 ]]
8172023-12-07T01:07:39.535Z+ sleep 1
8182023-12-07T01:07:40.531Z+ retry=6
8192023-12-07T01:07:40.558Z+++ zoneadm list -n
8202023-12-07T01:07:40.558Z+++ grep oxz_ntp
8212023-12-07T01:07:40.558Z++ pfexec svcs -z oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b -Hostate oxide/ntp
8222023-12-07T01:07:40.558Z+ [[ disabled != online ]]
8232023-12-07T01:07:40.558Z+ [[ 6 -gt 60 ]]
8242023-12-07T01:07:40.558Z+ sleep 1
8252023-12-07T01:07:41.962Z+ retry=7
8262023-12-07T01:07:42.008Z+++ zoneadm list -n
8272023-12-07T01:07:42.009Z+++ grep oxz_ntp
8282023-12-07T01:07:42.009Z++ pfexec svcs -z oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b -Hostate oxide/ntp
8292023-12-07T01:07:42.009Z+ [[ disabled != online ]]
8302023-12-07T01:07:42.009Z+ [[ 7 -gt 60 ]]
8312023-12-07T01:07:42.009Z+ sleep 1
8322023-12-07T01:07:42.576Z+ retry=8
8332023-12-07T01:07:42.603Z+++ zoneadm list -n
8342023-12-07T01:07:42.603Z+++ grep oxz_ntp
8352023-12-07T01:07:42.603Z++ pfexec svcs -z oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b -Hostate oxide/ntp
8362023-12-07T01:07:42.603Z+ [[ disabled != online ]]
8372023-12-07T01:07:42.603Z+ [[ 8 -gt 60 ]]
8382023-12-07T01:07:42.603Z+ sleep 1
8392023-12-07T01:07:43.598Z+ retry=9
8402023-12-07T01:07:43.631Z+++ zoneadm list -n
8412023-12-07T01:07:43.632Z+++ grep oxz_ntp
8422023-12-07T01:07:43.632Z++ pfexec svcs -z oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b -Hostate oxide/ntp
8432023-12-07T01:07:43.632Z+ [[ disabled != online ]]
8442023-12-07T01:07:43.632Z+ [[ 9 -gt 60 ]]
8452023-12-07T01:07:43.632Z+ sleep 1
8462023-12-07T01:07:44.625Z+ retry=10
8472023-12-07T01:07:44.651Z+++ zoneadm list -n
8482023-12-07T01:07:44.651Z+++ grep oxz_ntp
8492023-12-07T01:07:44.651Z++ pfexec svcs -z oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b -Hostate oxide/ntp
8502023-12-07T01:07:44.651Z+ [[ disabled != online ]]
8512023-12-07T01:07:44.652Z+ [[ 10 -gt 60 ]]
8522023-12-07T01:07:44.652Z+ sleep 1
8532023-12-07T01:07:45.643Z+ retry=11
8542023-12-07T01:07:45.681Z+++ zoneadm list -n
8552023-12-07T01:07:45.681Z+++ grep oxz_ntp
8562023-12-07T01:07:45.681Z++ pfexec svcs -z oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b -Hostate oxide/ntp
8572023-12-07T01:07:45.682Z+ [[ disabled != online ]]
8582023-12-07T01:07:45.682Z+ [[ 11 -gt 60 ]]
8592023-12-07T01:07:45.682Z+ sleep 1
 
8602023-12-07T01:07:45.992Zbuskin console login: Dec 7 01:07:44 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8612023-12-07T01:07:46.667Z+ retry=12
8622023-12-07T01:07:46.696Z+++ zoneadm list -n
8632023-12-07T01:07:46.696Z+++ grep oxz_ntp
8642023-12-07T01:07:46.696Z++ pfexec svcs -z oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b -Hostate oxide/ntp
8652023-12-07T01:07:46.696Z+ [[ online != online ]]
8662023-12-07T01:07:46.696Z+ echo 'Waited for chrony: 12s'
8672023-12-07T01:07:46.696Z+ export RUST_BACKTRACE=1
8682023-12-07T01:07:46.696ZWaited for chrony: 12s
8692023-12-07T01:07:46.696Z+ RUST_BACKTRACE=1
8702023-12-07T01:07:46.697Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8712023-12-07T01:07:46.697Z+ ./tests/bootstrap
8722023-12-07T01:07:46.735ZUsing external DNS server at 10.151.1.164:53
8732023-12-07T01:07:46.736Z2023-12-07 01:07:46.209346542 UTC: attempting to log into API
8742023-12-07T01:08:01.754Z2023-12-07 01:08:02.103349376 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8752023-12-07T01:08:02.757Z2023-12-07 01:08:03.102427619 UTC: attempting to log into API
8762023-12-07T01:08:17.788Z2023-12-07 01:08:18.094486482 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8772023-12-07T01:08:18.789Z2023-12-07 01:08:19.092401391 UTC: attempting to log into API
8782023-12-07T01:08:33.853Z2023-12-07 01:08:34.085635059 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8792023-12-07T01:08:34.824Z2023-12-07 01:08:35.083635207 UTC: attempting to log into API
8802023-12-07T01:08:49.857Z2023-12-07 01:08:50.075467995 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8812023-12-07T01:08:50.858Z2023-12-07 01:08:51.073682199 UTC: attempting to log into API
8822023-12-07T01:09:05.890Z2023-12-07 01:09:06.064898259 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8832023-12-07T01:09:06.893Z2023-12-07 01:09:07.063078949 UTC: attempting to log into API
8842023-12-07T01:09:21.924Z2023-12-07 01:09:22.055064902 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8852023-12-07T01:09:22.925Z2023-12-07 01:09:23.054185284 UTC: attempting to log into API
8862023-12-07T01:09:37.981Z2023-12-07 01:09:38.047064031 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8872023-12-07T01:09:38.960Z2023-12-07 01:09:39.045191529 UTC: attempting to log into API
8882023-12-07T01:09:53.994Z2023-12-07 01:09:54.036958552 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8892023-12-07T01:09:54.994Z2023-12-07 01:09:55.034913880 UTC: attempting to log into API
8902023-12-07T01:10:10.027Z2023-12-07 01:10:10.034704772 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8912023-12-07T01:10:11.028Z2023-12-07 01:10:11.034055541 UTC: attempting to log into API
8922023-12-07T01:10:26.085Z2023-12-07 01:10:26.045456851 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8932023-12-07T01:10:27.060Z2023-12-07 01:10:27.046527521 UTC: attempting to log into API
8942023-12-07T01:10:42.094Z2023-12-07 01:10:42.080139777 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8952023-12-07T01:10:43.096Z2023-12-07 01:10:43.081410497 UTC: attempting to log into API
8962023-12-07T01:10:58.272Z2023-12-07 01:10:58.116539839 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8972023-12-07T01:10:59.302Z2023-12-07 01:10:59.117287398 UTC: attempting to log into API
 
8982023-12-07T01:11:23.704ZDec 7 01:11:03 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
8992023-12-07T01:11:23.891ZDec 7 01:11:09 bukin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9002023-12-07T01:11:23.977ZDec 7 01:11:17 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9012023-12-07T01:11:23.989Z2023-12-07 01:11:14.151542466 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9022023-12-07T01:11:23.989Z2023-12-07 01:11:15.152598015 UTC: attempting to log into API
 
9032023-12-07T01:11:24.411ZDec 7 01:11:23 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9042023-12-07T01:11:30.201Z2023-12-07 01:11:30.186188351 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9052023-12-07T01:11:31.202Z2023-12-07 01:11:31.187209071 UTC: attempting to log into API
 
9062023-12-07T01:11:31.583ZDec 7 01:11:30 bukin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9072023-12-07T01:11:46.239Z2023-12-07 01:11:46.223726457 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9082023-12-07T01:11:47.240Z2023-12-07 01:11:47.224769817 UTC: attempting to log into API
9092023-12-07T01:11:48.085Z2023-12-07 01:11:48.066909828 UTC: login succeeded
9102023-12-07T01:11:48.114Zwaiting for nexus to come up...
9112023-12-07T01:11:48.190Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9122023-12-07T01:11:48.268Zensuring datasets are ready...
9132023-12-07T01:11:52.106Zlet's roll.
9142023-12-07T01:11:52.131Z+ rm ./tests/bootstrap
9152023-12-07T01:11:52.180Z+ for test_bin in tests/*
9162023-12-07T01:11:52.180Z+ ./tests/bootstrap-be9c817b94af4e8a
9172023-12-07T01:11:52.212Z
9182023-12-07T01:11:52.212Zrunning 0 tests
9192023-12-07T01:11:52.212Z
9202023-12-07T01:11:52.212Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9212023-12-07T01:11:52.212Z
9222023-12-07T01:11:52.213Z+ for test_bin in tests/*
9232023-12-07T01:11:52.213Z+ ./tests/end_to_end_tests-ff923d3f5c6228d4
9242023-12-07T01:11:52.213Z
9252023-12-07T01:11:52.213Zrunning 2 tests
9262023-12-07T01:11:52.425Ztest no_spoof::no_spoof_here ... ok
9272023-12-07T01:12:52.206Ztest instance_launch::instance_launch has been running for over 60 seconds
9282023-12-07T01:15:00.423Ztest instance_launch::instance_launch ... ok
9292023-12-07T01:15:00.425Z
9302023-12-07T01:15:00.425Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 179.93s
9312023-12-07T01:15:00.425Z
9322023-12-07T01:15:00.471Z+ _exit_trap
9332023-12-07T01:15:00.471Z+ local status=0
9342023-12-07T01:15:00.471Z+ [[ 0 -eq 0 ]]
9352023-12-07T01:15:00.472Z+ exit 0
9362023-12-07T01:15:00.472Zprocess exited: duration 516505 ms, exit code 0
 
9372023-12-07T01:15:00.606Zfound 54 output files
9382023-12-07T01:15:00.606Zuploading: /var/svc/log/oxide-sled-agent:default.log (744138 bytes)
9392023-12-07T01:15:02.329Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9402023-12-07T01:15:02.329Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_fb9d8f79-600a-4004-a642-96d517bb1d55/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9412023-12-07T01:15:03.409Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_fb9d8f79-600a-4004-a642-96d517bb1d55/root/var/svc/log/oxide-cockroachdb:default.log
9422023-12-07T01:15:03.409Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_fb9d8f79-600a-4004-a642-96d517bb1d55/root/var/svc/log/oxide-cockroachdb:default.log.0 (5370 bytes)
9432023-12-07T01:15:04.531Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_fb9d8f79-600a-4004-a642-96d517bb1d55/root/var/svc/log/oxide-cockroachdb:default.log.0
9442023-12-07T01:15:04.531Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_5158fc46-3806-4f31-b05f-36812386b9f1/root/var/svc/log/oxide-external_dns:default.log (15614 bytes)
9452023-12-07T01:15:04.559Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_5158fc46-3806-4f31-b05f-36812386b9f1/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15614 -> 0
9462023-12-07T01:15:05.637Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_5158fc46-3806-4f31-b05f-36812386b9f1/root/var/svc/log/oxide-external_dns:default.log
9472023-12-07T01:15:05.669Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bf33b194-1a6f-4bf9-ad48-29b12df39aea/root/var/svc/log/oxide-nexus:default.log (934061 bytes)
9482023-12-07T01:15:05.669Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bf33b194-1a6f-4bf9-ad48-29b12df39aea/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 934061 -> 0
9492023-12-07T01:15:06.747Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bf33b194-1a6f-4bf9-ad48-29b12df39aea/root/var/svc/log/oxide-nexus:default.log
9502023-12-07T01:15:06.747Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_52490697-55b4-4e94-8113-7bfd03f54768/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9512023-12-07T01:15:06.772Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_52490697-55b4-4e94-8113-7bfd03f54768/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 2338 -> 0
9522023-12-07T01:15:08.253Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_52490697-55b4-4e94-8113-7bfd03f54768/root/var/svc/log/oxide-crucible-pantry:default.log
9532023-12-07T01:15:08.253Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_291c1fe9-9d5f-4e9f-8b65-1a7cb50467c3/root/var/svc/log/oxide-internal_dns:default.log (109628 bytes)
9542023-12-07T01:15:08.282Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_291c1fe9-9d5f-4e9f-8b65-1a7cb50467c3/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 109628 -> 0
9552023-12-07T01:15:08.976Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_291c1fe9-9d5f-4e9f-8b65-1a7cb50467c3/root/var/svc/log/oxide-internal_dns:default.log
9562023-12-07T01:15:08.976Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1e8f4b9e-085f-418a-8016-b02520cfce76/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
9572023-12-07T01:15:09.002Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1e8f4b9e-085f-418a-8016-b02520cfce76/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 3434 -> 0
9582023-12-07T01:15:10.076Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1e8f4b9e-085f-418a-8016-b02520cfce76/root/var/svc/log/oxide-crucible-agent:default.log
9592023-12-07T01:15:10.076Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1e8f4b9e-085f-418a-8016-b02520cfce76/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9602023-12-07T01:15:10.102Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1e8f4b9e-085f-418a-8016-b02520cfce76/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
9612023-12-07T01:15:11.181Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1e8f4b9e-085f-418a-8016-b02520cfce76/root/var/svc/log/oxide-crucible-downstairs:default.log
9622023-12-07T01:15:11.181Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_79f99269-3df9-47b5-aa2a-574ca33f5ace/root/var/svc/log/oxide-crucible-agent:default.log (14572 bytes)
9632023-12-07T01:15:11.209Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_79f99269-3df9-47b5-aa2a-574ca33f5ace/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 14572 -> 0
9642023-12-07T01:15:12.287Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_79f99269-3df9-47b5-aa2a-574ca33f5ace/root/var/svc/log/oxide-crucible-agent:default.log
9652023-12-07T01:15:12.287Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_79f99269-3df9-47b5-aa2a-574ca33f5ace/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9662023-12-07T01:15:12.313Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_79f99269-3df9-47b5-aa2a-574ca33f5ace/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
9672023-12-07T01:15:13.386Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_79f99269-3df9-47b5-aa2a-574ca33f5ace/root/var/svc/log/oxide-crucible-downstairs:default.log
9682023-12-07T01:15:13.386Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_79f99269-3df9-47b5-aa2a-574ca33f5ace/root/var/svc/log/oxide-crucible-downstairs:downstairs-c644c299-ea3e-40f4-8f2f-631d89b2863f.log (3049 bytes)
9692023-12-07T01:15:13.415Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_79f99269-3df9-47b5-aa2a-574ca33f5ace/root/var/svc/log/oxide-crucible-downstairs:downstairs-c644c299-ea3e-40f4-8f2f-631d89b2863f.log" changed size mid upload: 3049 -> 0
9702023-12-07T01:15:14.502Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_79f99269-3df9-47b5-aa2a-574ca33f5ace/root/var/svc/log/oxide-crucible-downstairs:downstairs-c644c299-ea3e-40f4-8f2f-631d89b2863f.log
9712023-12-07T01:15:14.502Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb962991-cb18-48dc-83fe-e6b511a55b93/root/var/svc/log/oxide-crucible-agent:default.log (25253 bytes)
9722023-12-07T01:15:14.527Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb962991-cb18-48dc-83fe-e6b511a55b93/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 25253 -> 0
9732023-12-07T01:15:15.606Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb962991-cb18-48dc-83fe-e6b511a55b93/root/var/svc/log/oxide-crucible-agent:default.log
9742023-12-07T01:15:15.607Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb962991-cb18-48dc-83fe-e6b511a55b93/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9752023-12-07T01:15:15.638Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb962991-cb18-48dc-83fe-e6b511a55b93/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
9762023-12-07T01:15:16.737Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb962991-cb18-48dc-83fe-e6b511a55b93/root/var/svc/log/oxide-crucible-downstairs:default.log
9772023-12-07T01:15:16.738Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb962991-cb18-48dc-83fe-e6b511a55b93/root/var/svc/log/oxide-crucible-downstairs:downstairs-847b4bd0-d422-4b63-a341-5957e3ef12db.log (5537 bytes)
9782023-12-07T01:15:16.830Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb962991-cb18-48dc-83fe-e6b511a55b93/root/var/svc/log/oxide-crucible-downstairs:downstairs-847b4bd0-d422-4b63-a341-5957e3ef12db.log" changed size mid upload: 5537 -> 0
9792023-12-07T01:15:17.911Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb962991-cb18-48dc-83fe-e6b511a55b93/root/var/svc/log/oxide-crucible-downstairs:downstairs-847b4bd0-d422-4b63-a341-5957e3ef12db.log
9802023-12-07T01:15:17.911Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb962991-cb18-48dc-83fe-e6b511a55b93/root/var/svc/log/oxide-crucible-downstairs:downstairs-96a0e501-fe67-45be-818c-1f4b33ba7d55.log (3049 bytes)
9812023-12-07T01:15:17.936Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb962991-cb18-48dc-83fe-e6b511a55b93/root/var/svc/log/oxide-crucible-downstairs:downstairs-96a0e501-fe67-45be-818c-1f4b33ba7d55.log" changed size mid upload: 3049 -> 0
9822023-12-07T01:15:19.011Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fb962991-cb18-48dc-83fe-e6b511a55b93/root/var/svc/log/oxide-crucible-downstairs:downstairs-96a0e501-fe67-45be-818c-1f4b33ba7d55.log
9832023-12-07T01:15:19.039Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_ba6d3ec0-4a66-41b3-bab3-dba67c9e6853/root/var/svc/log/oxide-nexus:default.log (797403 bytes)
9842023-12-07T01:15:19.077Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_ba6d3ec0-4a66-41b3-bab3-dba67c9e6853/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 797403 -> 6270
9852023-12-07T01:15:20.159Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_ba6d3ec0-4a66-41b3-bab3-dba67c9e6853/root/var/svc/log/oxide-nexus:default.log
9862023-12-07T01:15:20.159Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b8581c69-b37d-4850-a043-f0ad5ae76c9b/root/var/svc/log/oxide-crucible-agent:default.log (24801 bytes)
9872023-12-07T01:15:20.184Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b8581c69-b37d-4850-a043-f0ad5ae76c9b/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 24801 -> 0
9882023-12-07T01:15:21.261Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b8581c69-b37d-4850-a043-f0ad5ae76c9b/root/var/svc/log/oxide-crucible-agent:default.log
9892023-12-07T01:15:21.261Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b8581c69-b37d-4850-a043-f0ad5ae76c9b/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9902023-12-07T01:15:22.357Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b8581c69-b37d-4850-a043-f0ad5ae76c9b/root/var/svc/log/oxide-crucible-downstairs:default.log
9912023-12-07T01:15:22.357Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b8581c69-b37d-4850-a043-f0ad5ae76c9b/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9922023-12-07T01:15:23.457Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b8581c69-b37d-4850-a043-f0ad5ae76c9b/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9932023-12-07T01:15:23.457Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b8581c69-b37d-4850-a043-f0ad5ae76c9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-61e6e31e-57cd-4d2f-b640-7a4320d87ec9.log (0 bytes)
9942023-12-07T01:15:24.538Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b8581c69-b37d-4850-a043-f0ad5ae76c9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-61e6e31e-57cd-4d2f-b640-7a4320d87ec9.log
9952023-12-07T01:15:24.565Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b8581c69-b37d-4850-a043-f0ad5ae76c9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-61e6e31e-57cd-4d2f-b640-7a4320d87ec9.log.0 (3051 bytes)
9962023-12-07T01:15:25.650Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b8581c69-b37d-4850-a043-f0ad5ae76c9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-61e6e31e-57cd-4d2f-b640-7a4320d87ec9.log.0
9972023-12-07T01:15:25.650Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b8581c69-b37d-4850-a043-f0ad5ae76c9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-625ec0bc-b97a-4f32-bbb6-bb7f8ce87979.log (0 bytes)
9982023-12-07T01:15:26.726Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b8581c69-b37d-4850-a043-f0ad5ae76c9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-625ec0bc-b97a-4f32-bbb6-bb7f8ce87979.log
9992023-12-07T01:15:26.726Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b8581c69-b37d-4850-a043-f0ad5ae76c9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-625ec0bc-b97a-4f32-bbb6-bb7f8ce87979.log.0 (5537 bytes)
10002023-12-07T01:15:28.238Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b8581c69-b37d-4850-a043-f0ad5ae76c9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-625ec0bc-b97a-4f32-bbb6-bb7f8ce87979.log.0
10012023-12-07T01:15:28.238Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b/root/var/svc/log/oxide-ntp:default.log (1458 bytes)
10022023-12-07T01:15:28.263Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b/root/var/svc/log/oxide-ntp:default.log" changed size mid upload: 1458 -> 0
10032023-12-07T01:15:29.769Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b/root/var/svc/log/oxide-ntp:default.log
10042023-12-07T01:15:29.769Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_464a152f-274d-46e9-a18b-7545a9aed1b5/root/var/svc/log/oxide-crucible-agent:default.log (14583 bytes)
10052023-12-07T01:15:29.798Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_464a152f-274d-46e9-a18b-7545a9aed1b5/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 14583 -> 0
10062023-12-07T01:15:30.871Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_464a152f-274d-46e9-a18b-7545a9aed1b5/root/var/svc/log/oxide-crucible-agent:default.log
10072023-12-07T01:15:30.871Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_464a152f-274d-46e9-a18b-7545a9aed1b5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10082023-12-07T01:15:30.897Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_464a152f-274d-46e9-a18b-7545a9aed1b5/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10092023-12-07T01:15:31.976Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_464a152f-274d-46e9-a18b-7545a9aed1b5/root/var/svc/log/oxide-crucible-downstairs:default.log
10102023-12-07T01:15:31.977Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_464a152f-274d-46e9-a18b-7545a9aed1b5/root/var/svc/log/oxide-crucible-downstairs:downstairs-cfed90e9-f50e-479a-8e24-920ec29b0f97.log (5537 bytes)
10112023-12-07T01:15:32.004Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_464a152f-274d-46e9-a18b-7545a9aed1b5/root/var/svc/log/oxide-crucible-downstairs:downstairs-cfed90e9-f50e-479a-8e24-920ec29b0f97.log" changed size mid upload: 5537 -> 0
10122023-12-07T01:15:33.089Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_464a152f-274d-46e9-a18b-7545a9aed1b5/root/var/svc/log/oxide-crucible-downstairs:downstairs-cfed90e9-f50e-479a-8e24-920ec29b0f97.log
10132023-12-07T01:15:33.116Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_3b2dd3e3-3a3d-4f29-ac8d-59a379554087/root/var/svc/log/oxide-internal_dns:default.log (111154 bytes)
10142023-12-07T01:15:33.144Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_3b2dd3e3-3a3d-4f29-ac8d-59a379554087/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 111154 -> 3755
10152023-12-07T01:15:34.217Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_3b2dd3e3-3a3d-4f29-ac8d-59a379554087/root/var/svc/log/oxide-internal_dns:default.log
10162023-12-07T01:15:34.217Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_bca496c0-e8d0-42e4-9905-69f0b44bcc91/root/var/svc/log/oxide-clickhouse:default.log (721403 bytes)
10172023-12-07T01:15:34.391Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_bca496c0-e8d0-42e4-9905-69f0b44bcc91/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 721403 -> 77518
10182023-12-07T01:15:35.467Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_bca496c0-e8d0-42e4-9905-69f0b44bcc91/root/var/svc/log/oxide-clickhouse:default.log
10192023-12-07T01:15:35.468Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_87b6f0fc-2509-40ac-98a9-5e75d2a5f038/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10202023-12-07T01:15:35.497Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_87b6f0fc-2509-40ac-98a9-5e75d2a5f038/root/var/svc/log/oxide-cockroachdb:default.log" changed size mid upload: 5369 -> 0
10212023-12-07T01:15:36.577Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_87b6f0fc-2509-40ac-98a9-5e75d2a5f038/root/var/svc/log/oxide-cockroachdb:default.log
10222023-12-07T01:15:36.577Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4ec46fa9-1dc2-4f54-9c50-12eaca4fb30a/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10232023-12-07T01:15:37.651Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4ec46fa9-1dc2-4f54-9c50-12eaca4fb30a/root/var/svc/log/oxide-crucible-agent:default.log
10242023-12-07T01:15:37.651Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4ec46fa9-1dc2-4f54-9c50-12eaca4fb30a/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10252023-12-07T01:15:38.763Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4ec46fa9-1dc2-4f54-9c50-12eaca4fb30a/root/var/svc/log/oxide-crucible-agent:default.log.0
10262023-12-07T01:15:38.764Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4ec46fa9-1dc2-4f54-9c50-12eaca4fb30a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10272023-12-07T01:15:38.789Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4ec46fa9-1dc2-4f54-9c50-12eaca4fb30a/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10282023-12-07T01:15:39.868Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4ec46fa9-1dc2-4f54-9c50-12eaca4fb30a/root/var/svc/log/oxide-crucible-downstairs:default.log
10292023-12-07T01:15:39.868Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_bafbdf06-9862-4b56-985e-76a1c0b55167/root/var/svc/log/oxide-internal_dns:default.log (113206 bytes)
10302023-12-07T01:15:39.938Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_bafbdf06-9862-4b56-985e-76a1c0b55167/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 113206 -> 3756
10312023-12-07T01:15:41.015Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_bafbdf06-9862-4b56-985e-76a1c0b55167/root/var/svc/log/oxide-internal_dns:default.log
10322023-12-07T01:15:41.015Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f14f5985-cc3c-4620-804e-71fd6a2efb69/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10332023-12-07T01:15:41.042Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f14f5985-cc3c-4620-804e-71fd6a2efb69/root/var/svc/log/oxide-cockroachdb:default.log" changed size mid upload: 5370 -> 0
10342023-12-07T01:15:42.117Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f14f5985-cc3c-4620-804e-71fd6a2efb69/root/var/svc/log/oxide-cockroachdb:default.log
10352023-12-07T01:15:42.117Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_081cd11b-e3e4-4099-9b6f-8429e715f17b/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
10362023-12-07T01:15:42.142Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_081cd11b-e3e4-4099-9b6f-8429e715f17b/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 3434 -> 0
10372023-12-07T01:15:43.217Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_081cd11b-e3e4-4099-9b6f-8429e715f17b/root/var/svc/log/oxide-crucible-agent:default.log
10382023-12-07T01:15:43.242Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_081cd11b-e3e4-4099-9b6f-8429e715f17b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10392023-12-07T01:15:43.243Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_081cd11b-e3e4-4099-9b6f-8429e715f17b/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10402023-12-07T01:15:44.316Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_081cd11b-e3e4-4099-9b6f-8429e715f17b/root/var/svc/log/oxide-crucible-downstairs:default.log
10412023-12-07T01:15:44.316Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_21e52a1e-d11e-49f9-b959-ffa324b0696f/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10422023-12-07T01:15:44.750Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_21e52a1e-d11e-49f9-b959-ffa324b0696f/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 2338 -> 0
10432023-12-07T01:15:45.506Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_21e52a1e-d11e-49f9-b959-ffa324b0696f/root/var/svc/log/oxide-crucible-pantry:default.log
10442023-12-07T01:15:45.506Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_241bf012-b777-40ab-8049-75d71094940c/root/var/svc/log/oxide-cockroachdb:default.log (5344 bytes)
10452023-12-07T01:15:45.584Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_241bf012-b777-40ab-8049-75d71094940c/root/var/svc/log/oxide-cockroachdb:default.log" changed size mid upload: 5344 -> 0
10462023-12-07T01:15:47.079Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_241bf012-b777-40ab-8049-75d71094940c/root/var/svc/log/oxide-cockroachdb:default.log
10472023-12-07T01:15:47.079Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_49a56d51-62f7-46ee-9771-f5957cbea350/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10482023-12-07T01:15:48.571Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_49a56d51-62f7-46ee-9771-f5957cbea350/root/var/svc/log/oxide-cockroachdb:default.log
10492023-12-07T01:15:48.603Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_49a56d51-62f7-46ee-9771-f5957cbea350/root/var/svc/log/oxide-cockroachdb:default.log.0 (5370 bytes)
10502023-12-07T01:15:49.699Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_49a56d51-62f7-46ee-9771-f5957cbea350/root/var/svc/log/oxide-cockroachdb:default.log.0
10512023-12-07T01:15:49.699Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8120d95e-cfba-472c-b9e6-b1be896490ab/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10522023-12-07T01:15:49.726Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8120d95e-cfba-472c-b9e6-b1be896490ab/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 3436 -> 0
10532023-12-07T01:15:50.814Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8120d95e-cfba-472c-b9e6-b1be896490ab/root/var/svc/log/oxide-crucible-agent:default.log
10542023-12-07T01:15:50.814Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8120d95e-cfba-472c-b9e6-b1be896490ab/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10552023-12-07T01:15:50.848Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8120d95e-cfba-472c-b9e6-b1be896490ab/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10562023-12-07T01:15:51.972Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8120d95e-cfba-472c-b9e6-b1be896490ab/root/var/svc/log/oxide-crucible-downstairs:default.log
10572023-12-07T01:15:51.972Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b3f5baf8-c870-43b7-af4d-ee725dba88c9/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10582023-12-07T01:15:53.508Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b3f5baf8-c870-43b7-af4d-ee725dba88c9/root/var/svc/log/oxide-crucible-agent:default.log
10592023-12-07T01:15:53.508Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b3f5baf8-c870-43b7-af4d-ee725dba88c9/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
10602023-12-07T01:15:54.606Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b3f5baf8-c870-43b7-af4d-ee725dba88c9/root/var/svc/log/oxide-crucible-agent:default.log.0
10612023-12-07T01:15:54.606Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b3f5baf8-c870-43b7-af4d-ee725dba88c9/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10622023-12-07T01:15:54.634Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b3f5baf8-c870-43b7-af4d-ee725dba88c9/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10632023-12-07T01:15:55.709Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b3f5baf8-c870-43b7-af4d-ee725dba88c9/root/var/svc/log/oxide-crucible-downstairs:default.log
10642023-12-07T01:15:55.709Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_7dc010b9-8d5f-4534-924b-9d4a640ea7f9/root/var/svc/log/oxide-crucible-pantry:default.log (2332 bytes)
10652023-12-07T01:15:55.735Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_7dc010b9-8d5f-4534-924b-9d4a640ea7f9/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 2332 -> 0
10662023-12-07T01:15:56.805Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_7dc010b9-8d5f-4534-924b-9d4a640ea7f9/root/var/svc/log/oxide-crucible-pantry:default.log
10672023-12-07T01:15:56.807Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e01321fb-b769-45f7-ab16-8deaa00ea7ad/root/var/svc/log/oxide-external_dns:default.log (15614 bytes)
10682023-12-07T01:15:56.854Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e01321fb-b769-45f7-ab16-8deaa00ea7ad/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15614 -> 3773
10692023-12-07T01:15:57.934Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e01321fb-b769-45f7-ab16-8deaa00ea7ad/root/var/svc/log/oxide-external_dns:default.log
10702023-12-07T01:15:57.934Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_14a486ee-8d2b-4c8d-814d-85da6238da27/root/var/svc/log/oxide-nexus:default.log (4304563 bytes)
10712023-12-07T01:15:58.230Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_14a486ee-8d2b-4c8d-814d-85da6238da27/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4304563 -> 71953
10722023-12-07T01:15:59.309Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_14a486ee-8d2b-4c8d-814d-85da6238da27/root/var/svc/log/oxide-nexus:default.log
10732023-12-07T01:15:59.346Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_576b4b44-91d4-4d23-81b1-9199efb740ae/root/var/svc/log/oxide-oximeter:default.log (183402 bytes)
10742023-12-07T01:15:59.509Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_576b4b44-91d4-4d23-81b1-9199efb740ae/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 183402 -> 44084
10752023-12-07T01:16:00.600Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_576b4b44-91d4-4d23-81b1-9199efb740ae/root/var/svc/log/oxide-oximeter:default.log
10762023-12-07T01:16:00.600Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b/root/var/log/chrony/measurements.log (11234 bytes)
10772023-12-07T01:16:01.137Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b/root/var/log/chrony/measurements.log" changed size mid upload: 11234 -> 13152
10782023-12-07T01:16:02.230Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b/root/var/log/chrony/measurements.log
10792023-12-07T01:16:02.230Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b/root/var/log/chrony/statistics.log (6318 bytes)
10802023-12-07T01:16:02.304Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b/root/var/log/chrony/statistics.log" changed size mid upload: 6318 -> 7254
10812023-12-07T01:16:03.383Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b/root/var/log/chrony/statistics.log
10822023-12-07T01:16:03.383Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b/root/var/log/chrony/tracking.log (2119 bytes)
10832023-12-07T01:16:03.434Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b/root/var/log/chrony/tracking.log" changed size mid upload: 2119 -> 2251
10842023-12-07T01:16:04.519Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_1becbe90-1e60-4bb1-a792-981ab7b1dc7b/root/var/log/chrony/tracking.log