01HBYDWNNWMEGB70YGEX0VWP07: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.23",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HBYE6S7DHQ01TGHDBVSQXNVV

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-05T00:20:59.580Zjob dependencies complete; ready to run (waiting for 1 h 29 m 44 s)
22023-10-05T03:33:36.393Zjob assigned to worker 01HBYY9NN558E2G4M8SD174P22 (queued for 3 h 12 m 36 s)
32023-10-05T03:33:36.536Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-05T03:33:36.800Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-10-05T03:33:37.376Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-10-05T03:33:37.619Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-05T03:33:38.159Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-10-05T03:33:38.325ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-05T03:33:38.885Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-05T03:33:39.195ZOxide Helios Version helios-2.0.22216 64-bit
112023-10-05T03:33:39.433ZConfiguring device.
122023-10-05T03:33:39.570Z#################################################
132023-10-05T03:33:39.705Z#### ############# ##
142023-10-05T03:33:40.385Z### ### ############ ##
152023-10-05T03:33:41.000Z## ### # ## ### ## Oxide Computer Company ##
162023-10-05T03:33:41.112Z## ## ## ### # ### ##
172023-10-05T03:33:41.753Z## # ### #### #### This Station Under ##
182023-10-05T03:33:41.832Z### ### #### # ### Computer Control ##
192023-10-05T03:33:41.883Z#### #### ### ## ##
202023-10-05T03:33:41.940Z##################################################
212023-10-05T03:33:41.994ZHostname: buskin
222023-10-05T03:33:42.058ZLoading smf(7) serice descriptions:
232023-10-05T03:33:44.007Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-6e3d6cace6bb341c.gz
242023-10-05T03:34:09.458Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-6e3d6cace6bb341c.gz
252023-10-05T03:34:09.733Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-05T03:38:51.463Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-05T03:38:51.492Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2d9aa87f7c0ddffa.gz
282023-10-05T03:45:16.378Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2d9aa87f7c0ddffa.gz
292023-10-05T03:45:17.063Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-05T03:45:27.481Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-05T03:45:27.954Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-05T03:46:30.343Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-05T03:46:32.240Zdownloading input: /input/package/work/version.txt
342023-10-05T03:46:32.768Zdownloaded input: /input/package/work/version.txt
352023-10-05T03:46:33.291Zdownloading input: /input/package/work/package.tar.gz
362023-10-05T03:47:54.221Zdownloaded input: /input/package/work/package.tar.gz
372023-10-05T03:47:54.808Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-05T03:52:46.696Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-05T03:52:46.859Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-05T03:53:35.243Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-05T03:53:36.763Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-05T03:57:44.820Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-05T03:57:45.220Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-05T03:59:54.692Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-05T03:59:54.730Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-05T04:08:33.205Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-05T04:08:33.640Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-05T04:08:57.946Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-05T04:08:58.899Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-05T04:11:03.600Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-05T04:11:03.760Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-05T04:11:50.118Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-05T04:11:50.429Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-05T04:12:48.308Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-05T04:12:48.348Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-05T04:12:48.529Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-05T04:12:48.563Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-05T04:13:46.310Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-05T04:13:46.735Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-05T04:14:28.658Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-05T04:14:28.690Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-05T04:15:20.144Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-05T04:15:20.169Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-05T04:16:20.556Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-05T04:16:22.306Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-05T04:16:22.621Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-05T04:16:23.031Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-05T04:16:37.790Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-05T04:16:37.844Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-05T04:17:28.007Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-05T04:17:28.041Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-05T04:20:03.579Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-05T04:20:03.613Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-05T04:21:57.991Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-05T04:21:58.883Zstarting task 0: "setup"
762023-10-05T04:22:00.398Z++ uname -s
772023-10-05T04:22:07.861Z+ kern=SunOS
782023-10-05T04:22:13.941Z+ case "$kern" in
792023-10-05T04:22:15.439Z+ groupadd -g 12345 build
802023-10-05T04:22:17.807Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-05T04:22:19.711Z+ zfs create -o mountpoint=/work rpool/work
822023-10-05T04:22:41.115Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-05T04:22:41.578Z+ home_fs=zfs
842023-10-05T04:22:42.128Z+ [[ zfs == autofs ]]
852023-10-05T04:22:42.515Z+ mkdir -p /home/build
862023-10-05T04:22:42.573Z+ chown build:build /home/build /work
872023-10-05T04:22:42.872Z+ chmod 0700 /home/build /work
882023-10-05T04:22:43.417Zprocess exited: duration 4047 ms, exit code 0
 
892023-10-05T04:22:43.896Zstarting task 1: "authentication"
902023-10-05T04:22:45.063Zprocess exited: duration 1096 ms, exit code 0
 
912023-10-05T04:22:45.139Zstarting task 2: "build"
922023-10-05T04:22:45.207Z+ trap _exit_trap EXIT
932023-10-05T04:22:45.661Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-05T04:22:45.708Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-05T04:22:45.748Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-05T04:22:45.791Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-05T04:22:46.240Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-05T04:22:46.278Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-05T04:22:46.315Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-05T04:22:46.355Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-05T04:22:46.399Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-05T04:22:46.852Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-05T04:22:46.891Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-05T04:22:46.930Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-05T04:22:46.972Z+ [[ -d /opt/oxide ]]
1062023-10-05T04:22:47.416Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-05T04:22:47.453Z+ pfexec mkdir /opt/oxide
1082023-10-05T04:22:47.495Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-05T04:22:47.524Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-05T04:22:47.552Z+ cd /opt/oxide-underneath
1112023-10-05T04:22:47.579Z+ pfexec tar ceEp@/f - .
1122023-10-05T04:22:47.608Z+ cd /opt/oxide
1132023-10-05T04:22:47.655Z+ pfexec tar xveEp@/f -
1142023-10-05T04:22:47.704Zx ., 0 bytes, 0 tape blocks
1152023-10-05T04:22:47.743Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-05T04:22:47.781Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-05T04:22:47.821Zx ./opte/bin/opteadm, 50138640 bytes, 97928 tape blocks
1182023-10-05T04:22:48.277Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-05T04:22:48.376Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-05T04:22:48.417Z+ pfexec mkdir /opt/oxide/work
1212023-10-05T04:22:48.456Z+ pfexec chown build:build /opt/oxide/work
1222023-10-05T04:22:48.498Z+ cd /opt/oxide/work
1232023-10-05T04:22:48.537Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-05T04:22:48.583ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-05T04:22:48.621Zx out/maghemite.tar, 74957312 bytes, 146401 tape blocks
1262023-10-05T04:22:48.662Zx out/omicron-sled-agent.tar, 113287168 bytes, 221264 tape blocks
1272023-10-05T04:22:48.706Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-05T04:22:48.743Zx out/npuzone/npuzone, 36941864 bytes, 72153 tape blocks
1292023-10-05T04:22:48.783Zx package-manifest.toml, 21303 bytes, 42 tape blocks
1302023-10-05T04:22:48.823Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-05T04:22:48.861Zx target/release/omicron-package, 37951536 bytes, 74125 tape blocks
1322023-10-05T04:22:48.897Zx tools/create_virtual_hardware.sh, 2810 bytes, 6 tape blocks
1332023-10-05T04:22:48.934Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-05T04:22:48.977Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-05T04:22:49.449Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-05T04:22:49.497Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-05T04:22:49.541Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-05T04:22:49.575Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-05T04:22:49.700Z
1402023-10-05T04:22:49.747Zreal 1.598050261
1412023-10-05T04:22:50.221Zuser 1.439818503
1422023-10-05T04:22:50.263Zsys 0.282138355
1432023-10-05T04:22:50.312Ztrap 0.000048687
1442023-10-05T04:22:50.347Ztflt 0.000000000
1452023-10-05T04:22:50.389Zdflt 0.000000000
1462023-10-05T04:22:50.491Zkflt 0.000000000
1472023-10-05T04:22:50.522Zlock 0.000000000
1482023-10-05T04:22:50.555Zslp 1.438507923
1492023-10-05T04:22:50.582Zlat 0.033507839
1502023-10-05T04:22:50.624Zstop 0.000122513
1512023-10-05T04:22:50.667Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-05T04:22:50.703Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-05T04:22:50.743Z+ mkdir tests
1542023-10-05T04:22:50.783Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-05T04:22:50.830Z+ ptime -m gunzip
1562023-10-05T04:22:50.867Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-6e3d6cace6bb341c
1572023-10-05T04:22:50.923Z
1582023-10-05T04:22:50.955Zreal 0.137974996
1592023-10-05T04:22:50.992Zuser 0.126937052
1602023-10-05T04:22:51.026Zsys 0.010795076
1612023-10-05T04:22:51.067Ztrap 0.000009880
1622023-10-05T04:22:51.103Ztflt 0.000000000
1632023-10-05T04:22:51.150Zdflt 0.000000000
1642023-10-05T04:22:51.190Zkflt 0.000000000
1652023-10-05T04:22:51.234Zlock 0.000000000
1662023-10-05T04:22:51.273Zslp 0.000019240
1672023-10-05T04:22:51.313Zlat 0.000023470
1682023-10-05T04:22:51.355Zstop 0.000127224
1692023-10-05T04:22:51.389Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-6e3d6cace6bb341c
1702023-10-05T04:22:51.416Z+ chmod a+x tests/bootstrap-6e3d6cace6bb341c
1712023-10-05T04:22:51.450Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-05T04:22:51.479Z+ ptime -m gunzip
1732023-10-05T04:22:51.531Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-05T04:22:51.568Z
1752023-10-05T04:22:51.595Zreal 2.255100641
1762023-10-05T04:22:51.630Zuser 2.109791485
1772023-10-05T04:22:51.660Zsys 0.144913864
1782023-10-05T04:22:51.709Ztrap 0.000162645
1792023-10-05T04:22:51.760Ztflt 0.000000000
1802023-10-05T04:22:51.815Zdflt 0.000000000
1812023-10-05T04:22:51.855Zkflt 0.000000000
1822023-10-05T04:22:51.890Zlock 0.000000000
1832023-10-05T04:22:51.921Zslp 0.000024740
1842023-10-05T04:22:51.953Zlat 0.000022580
1852023-10-05T04:22:51.987Zstop 0.000117712
1862023-10-05T04:22:52.017Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-05T04:22:52.047Z+ chmod a+x tests/bootstrap
1882023-10-05T04:22:52.073Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-05T04:22:52.099Z+ ptime -m gunzip
1902023-10-05T04:22:52.125Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2d9aa87f7c0ddffa
1912023-10-05T04:22:53.078Z
1922023-10-05T04:22:53.107Zreal 2.507877471
1932023-10-05T04:22:53.139Zuser 2.347074122
1942023-10-05T04:22:53.174Zsys 0.160414846
1952023-10-05T04:22:53.206Ztrap 0.000166435
1962023-10-05T04:22:53.249Ztflt 0.000000000
1972023-10-05T04:22:53.290Zdflt 0.000000000
1982023-10-05T04:22:53.329Zkflt 0.000000000
1992023-10-05T04:22:53.373Zlock 0.000000000
2002023-10-05T04:22:53.417Zslp 0.000021990
2012023-10-05T04:22:53.450Zlat 0.000018020
2022023-10-05T04:22:53.487Zstop 0.000118274
2032023-10-05T04:22:53.528Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2d9aa87f7c0ddffa
2042023-10-05T04:22:53.650Z+ chmod a+x tests/end_to_end_tests-2d9aa87f7c0ddffa
2052023-10-05T04:22:53.684Z+ bmat address ls
2062023-10-05T04:22:53.723ZNAME CIDR FIRST LAST COUNT
2072023-10-05T04:22:53.776Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-10-05T04:22:53.813Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-05T04:22:53.854Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-10-05T04:22:53.925Z+ EXTRA_IP_START=10.151.1.164
2112023-10-05T04:22:54.156Z+ EXTRA_IP_END=10.151.1.227
2122023-10-05T04:22:54.207Z+ EXTRA_IP_COUNT=64
2132023-10-05T04:22:54.244Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-10-05T04:22:54.718Z+ EXTRA_IP_BASE=10.151.1
2152023-10-05T04:22:55.164Z+ EXTRA_IP_FOCTET=164
2162023-10-05T04:22:55.197Z+ EXTRA_IP_LOCTET=227
2172023-10-05T04:22:55.239Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-10-05T04:22:55.277Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-10-05T04:22:55.305Z+ DNS_IP1=10.151.1.164
2202023-10-05T04:22:55.342Z+ DNS_IP2=10.151.1.165
2212023-10-05T04:22:55.382Z+ UPLINK_IP=10.151.1.174
2222023-10-05T04:22:55.418Z+ IPPOOL_START=10.151.1.175
2232023-10-05T04:22:55.455Z+ IPPOOL_END=10.151.1.227
2242023-10-05T04:22:55.495Z++ ipadm show-addr -po type,addr
2252023-10-05T04:22:55.533Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-10-05T04:22:55.570Z+ GATEWAY_IP=10.151.1.100
2272023-10-05T04:22:55.617Z+ [[ -n 10.151.1.100 ]]
2282023-10-05T04:22:55.664Z+ ping -s 10.151.1.100 56 1
2292023-10-05T04:22:56.527ZPING 10.151.1.100: 56 data bytes
2302023-10-05T04:22:56.566Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.045 ms
2312023-10-05T04:22:56.605Z
2322023-10-05T04:22:57.069Z----10.151.1.100 PING Statistics----
2332023-10-05T04:22:57.105Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-10-05T04:22:57.141Zround-trip (ms) min/avg/max/stddev = 0.045/0.045/0.045/-nan
2352023-10-05T04:22:57.175Z++ arp -an
2362023-10-05T04:22:57.214Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-10-05T04:22:57.248Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-10-05T04:22:57.296Z+ routeadm -e ipv4-forwarding -u
2392023-10-05T04:22:57.337Z+ PXA_START=10.151.1.164
2402023-10-05T04:22:57.872Z+ PXA_END=10.151.1.227
2412023-10-05T04:22:57.912Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-10-05T04:22:58.369Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-10-05T04:22:58.403Z+ ZPOOL_VDEV_DIR=/scratch
2442023-10-05T04:22:58.441Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-10-05T04:22:58.487Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-10-05T04:22:58.945Z++ cd ./tools
2472023-10-05T04:22:58.983Z++ pwd
2482023-10-05T04:22:59.023Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-10-05T04:22:59.093Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-10-05T04:22:59.132Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-10-05T04:22:59.382Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-10-05T04:22:59.418Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-10-05T04:22:59.456Z++ dladm show-phys -p -o LINK
2542023-10-05T04:22:59.494Z++ head -1
2552023-10-05T04:22:59.533Z+ PHYSICAL_LINK=igb0
2562023-10-05T04:22:59.569Z+ echo 'Using igb0 as physical link'
2572023-10-05T04:22:59.606ZUsing igb0 as physical link
2582023-10-05T04:22:59.645Z+ ensure_run_as_root
2592023-10-05T04:22:59.687Z++ id -u
2602023-10-05T04:22:59.730Z+ [[ 0 -ne 0 ]]
2612023-10-05T04:22:59.770Z+ ensure_zpools
2622023-10-05T04:22:59.807Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-10-05T04:23:00.277Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-10-05T04:23:00.306Z+ readarray -t ZPOOLS
2652023-10-05T04:23:00.337Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-10-05T04:23:00.371Z++ sed 's/[ ",]//g'
2672023-10-05T04:23:00.409Z+ for ZPOOL in "${ZPOOLS[@]}"
2682023-10-05T04:23:00.441ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2692023-10-05T04:23:00.474Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-10-05T04:23:00.520ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2712023-10-05T04:23:00.554Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-10-05T04:23:00.595ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2732023-10-05T04:23:00.666Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2742023-10-05T04:23:00.700ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2752023-10-05T04:23:00.739Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2762023-10-05T04:23:00.777ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2772023-10-05T04:23:01.215Z0+0 records in
2782023-10-05T04:23:01.253ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2792023-10-05T04:23:01.292Z0+0 records out
2802023-10-05T04:23:01.744ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2812023-10-05T04:23:01.781Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2822023-10-05T04:23:01.817ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2832023-10-05T04:23:01.856Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2842023-10-05T04:23:01.900ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2852023-10-05T04:23:01.943Z+ set +x
2862023-10-05T04:23:03.203ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2872023-10-05T04:23:03.696Z++ zpool list -o name
2882023-10-05T04:23:03.728ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2892023-10-05T04:23:04.212Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2902023-10-05T04:23:04.256ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2912023-10-05T04:23:04.491Z+ [[ -z '' ]]
2922023-10-05T04:23:04.528ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2932023-10-05T04:23:04.561Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2942023-10-05T04:23:04.596ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2952023-10-05T04:23:04.628Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2962023-10-05T04:23:04.672ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2972023-10-05T04:23:04.711Z+ set +x
2982023-10-05T04:23:04.745ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
2992023-10-05T04:23:04.778Z+ for ZPOOL in "${ZPOOLS[@]}"
3002023-10-05T04:23:04.812ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3012023-10-05T04:23:04.848Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3022023-10-05T04:23:04.884ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3032023-10-05T04:23:04.913Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3042023-10-05T04:23:04.955ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3052023-10-05T04:23:04.984Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3062023-10-05T04:23:05.017ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3072023-10-05T04:23:05.048Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3082023-10-05T04:23:05.085ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3092023-10-05T04:23:05.132Z0+0 records in
3102023-10-05T04:23:05.600ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3112023-10-05T04:23:05.639Z0+0 records out
3122023-10-05T04:23:06.087ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3132023-10-05T04:23:06.566Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3142023-10-05T04:23:07.473ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3152023-10-05T04:23:07.518Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-10-05T04:23:07.573ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3172023-10-05T04:23:07.635Z+ set +x
3182023-10-05T04:23:07.689ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3192023-10-05T04:23:07.732Z++ zpool list -o name
3202023-10-05T04:23:08.185ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3212023-10-05T04:23:08.222Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3222023-10-05T04:23:08.260ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3232023-10-05T04:23:08.309Z+ [[ -z '' ]]
3242023-10-05T04:23:08.377ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3252023-10-05T04:23:08.415Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3262023-10-05T04:23:08.450ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3272023-10-05T04:23:08.520Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3282023-10-05T04:23:08.552ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3292023-10-05T04:23:08.585Z+ set +x
3302023-10-05T04:23:08.617ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3312023-10-05T04:23:08.651Z+ for ZPOOL in "${ZPOOLS[@]}"
3322023-10-05T04:23:08.689ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3332023-10-05T04:23:09.146Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3342023-10-05T04:23:09.590ZSimnet net0/sc0_0 exists
3352023-10-05T04:23:09.629Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3362023-10-05T04:23:09.667ZSimnet net1/sc1_0 exists
3372023-10-05T04:23:09.773Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3382023-10-05T04:23:09.806ZVnic sc0_1 exists
3392023-10-05T04:23:09.837Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3402023-10-05T04:23:09.870Z0+0 records in
3412023-10-05T04:23:09.905Z0+0 records out
3422023-10-05T04:23:09.943Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3432023-10-05T04:23:09.974Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3442023-10-05T04:23:10.010Z+ set +x
3452023-10-05T04:23:10.043Z++ zpool list -o name
3462023-10-05T04:23:10.081Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3472023-10-05T04:23:10.122Z+ [[ -z '' ]]
3482023-10-05T04:23:10.151Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3492023-10-05T04:23:10.192Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3502023-10-05T04:23:10.221Z+ set +x
3512023-10-05T04:23:10.253Z+ for ZPOOL in "${ZPOOLS[@]}"
3522023-10-05T04:23:10.283Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3532023-10-05T04:23:10.324Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3542023-10-05T04:23:10.352Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3552023-10-05T04:23:10.380Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3562023-10-05T04:23:10.417Z0+0 records in
3572023-10-05T04:23:10.857Z0+0 records out
3582023-10-05T04:23:10.894Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3592023-10-05T04:23:11.362Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3602023-10-05T04:23:11.397Z+ set +x
3612023-10-05T04:23:11.428Z++ zpool list -o name
3622023-10-05T04:23:11.902Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3632023-10-05T04:23:11.938Z+ [[ -z '' ]]
3642023-10-05T04:23:11.974Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3652023-10-05T04:23:12.019Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3662023-10-05T04:23:12.051Z+ set +x
3672023-10-05T04:23:12.077Z+ for ZPOOL in "${ZPOOLS[@]}"
3682023-10-05T04:23:12.106Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3692023-10-05T04:23:12.137Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3702023-10-05T04:23:12.608Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3712023-10-05T04:23:12.641Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3722023-10-05T04:23:12.692Z0+0 records in
3732023-10-05T04:23:12.723Z0+0 records out
3742023-10-05T04:23:12.757Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3752023-10-05T04:23:12.785Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3762023-10-05T04:23:12.811Z+ set +x
3772023-10-05T04:23:12.844Z++ zpool list -o name
3782023-10-05T04:23:12.874Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3792023-10-05T04:23:12.907Z+ [[ -z '' ]]
3802023-10-05T04:23:12.946Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3812023-10-05T04:23:12.990Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3822023-10-05T04:23:13.024Z+ set +x
3832023-10-05T04:23:13.064Z+ for ZPOOL in "${ZPOOLS[@]}"
3842023-10-05T04:23:13.099Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3852023-10-05T04:23:13.135Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3862023-10-05T04:23:13.189Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3872023-10-05T04:23:13.228Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3882023-10-05T04:23:13.275Z0+0 records in
3892023-10-05T04:23:13.313Z0+0 records out
3902023-10-05T04:23:13.348Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-10-05T04:23:13.384Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3922023-10-05T04:23:13.411Z+ set +x
3932023-10-05T04:23:13.439Z++ zpool list -o name
3942023-10-05T04:23:13.470Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3952023-10-05T04:23:13.499Z+ [[ -z '' ]]
3962023-10-05T04:23:13.526Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3972023-10-05T04:23:13.558Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3982023-10-05T04:23:13.592Z+ set +x
3992023-10-05T04:23:13.674Z+ for ZPOOL in "${ZPOOLS[@]}"
4002023-10-05T04:23:13.718Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4012023-10-05T04:23:13.755Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4022023-10-05T04:23:13.785Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4032023-10-05T04:23:14.222Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4042023-10-05T04:23:14.251Z0+0 records in
4052023-10-05T04:23:14.285Z0+0 records out
4062023-10-05T04:23:14.349Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4072023-10-05T04:23:14.382Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4082023-10-05T04:23:14.420Z+ set +x
4092023-10-05T04:23:14.456Z++ zpool list -o name
4102023-10-05T04:23:14.903Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4112023-10-05T04:23:14.945Z+ [[ -z '' ]]
4122023-10-05T04:23:14.995Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4132023-10-05T04:23:15.046Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4142023-10-05T04:23:15.086Z+ set +x
4152023-10-05T04:23:15.135Z+ for ZPOOL in "${ZPOOLS[@]}"
4162023-10-05T04:23:15.176Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4172023-10-05T04:23:16.061Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4182023-10-05T04:23:16.123Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4192023-10-05T04:23:16.159Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4202023-10-05T04:23:16.195Z0+0 records in
4212023-10-05T04:23:16.639Z0+0 records out
4222023-10-05T04:23:16.668Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4232023-10-05T04:23:17.284Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4242023-10-05T04:23:17.374Z+ set +x
4252023-10-05T04:23:17.410Z++ zpool list -o name
4262023-10-05T04:23:17.473Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4272023-10-05T04:23:17.517Z+ [[ -z '' ]]
4282023-10-05T04:23:17.551Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4292023-10-05T04:23:17.598Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4302023-10-05T04:23:17.684Z+ set +x
4312023-10-05T04:23:17.744Z+ for ZPOOL in "${ZPOOLS[@]}"
4322023-10-05T04:23:17.791Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4332023-10-05T04:23:17.852Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4342023-10-05T04:23:17.947Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4352023-10-05T04:23:18.009Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4362023-10-05T04:23:23.160Z0+0 records in
4372023-10-05T04:23:23.393Z0+0 records out
4382023-10-05T04:23:23.600Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4392023-10-05T04:23:23.724Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4402023-10-05T04:23:24.871Z+ set +x
4412023-10-05T04:23:24.910Z++ zpool list -o name
4422023-10-05T04:23:24.958Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4432023-10-05T04:23:24.999Z+ [[ -z '' ]]
4442023-10-05T04:23:25.036Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4452023-10-05T04:23:25.075Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4462023-10-05T04:23:25.154Z+ set +x
4472023-10-05T04:23:25.193Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4482023-10-05T04:23:25.236Z+ readarray -t ZPOOLS
4492023-10-05T04:23:25.279Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4502023-10-05T04:23:25.320Z++ sed 's/[ ",]//g'
4512023-10-05T04:23:25.361Z+ for ZPOOL in "${ZPOOLS[@]}"
4522023-10-05T04:23:25.394Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4532023-10-05T04:23:25.428Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4542023-10-05T04:23:25.462Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4552023-10-05T04:23:25.497Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4562023-10-05T04:23:25.541Z0+0 records in
4572023-10-05T04:23:25.576Z0+0 records out
4582023-10-05T04:23:25.611Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4592023-10-05T04:23:25.644Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4602023-10-05T04:23:25.676Z+ set +x
4612023-10-05T04:23:25.711Z++ zpool list -o name
4622023-10-05T04:23:25.747Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4632023-10-05T04:23:25.784Z+ [[ -z '' ]]
4642023-10-05T04:23:25.824Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4652023-10-05T04:23:25.867Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4662023-10-05T04:23:25.900Z+ set +x
4672023-10-05T04:23:25.930Z+ for ZPOOL in "${ZPOOLS[@]}"
4682023-10-05T04:23:25.962Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4692023-10-05T04:23:26.009Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4702023-10-05T04:23:26.051Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4712023-10-05T04:23:26.084Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4722023-10-05T04:23:26.530Z0+0 records in
4732023-10-05T04:23:26.572Z0+0 records out
4742023-10-05T04:23:26.607Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4752023-10-05T04:23:26.640Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4762023-10-05T04:23:26.678Z+ set +x
4772023-10-05T04:23:26.719Z++ zpool list -o name
4782023-10-05T04:23:26.760Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4792023-10-05T04:23:26.795Z+ [[ -z '' ]]
4802023-10-05T04:23:26.829Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4812023-10-05T04:23:26.867Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4822023-10-05T04:23:26.898Z+ set +x
4832023-10-05T04:23:26.928Z+ ensure_simulated_links igb0
4842023-10-05T04:23:26.958Z+ local PHYSICAL_LINK=igb0
4852023-10-05T04:23:27.441Z+ INDICES=("0" "1")
4862023-10-05T04:23:27.488Z+ for I in "${INDICES[@]}"
4872023-10-05T04:23:27.568Z++ get_simnet_name_if_exists net0
4882023-10-05T04:23:27.730Z++ dladm show-simnet -p -o LINK net0
4892023-10-05T04:23:27.872Z++ echo ''
4902023-10-05T04:23:27.961Z+ [[ -z '' ]]
4912023-10-05T04:23:28.109Z+ dladm create-simnet -t net0
4922023-10-05T04:23:28.221Z+ dladm create-simnet -t sc0_0
4932023-10-05T04:23:28.279Z+ dladm modify-simnet -t -p net0 sc0_0
4942023-10-05T04:23:28.308Z+ dladm set-linkprop -p mtu=1600 sc0_0
4952023-10-05T04:23:28.337Z+ success 'Simnet net0/sc0_0 exists'
4962023-10-05T04:23:28.379Z+ set +x
4972023-10-05T04:23:28.415Z+ for I in "${INDICES[@]}"
4982023-10-05T04:23:28.445Z++ get_simnet_name_if_exists net1
4992023-10-05T04:23:28.475Z++ dladm show-simnet -p -o LINK net1
5002023-10-05T04:23:28.509Z++ echo ''
5012023-10-05T04:23:28.546Z+ [[ -z '' ]]
5022023-10-05T04:23:29.062Z+ dladm create-simnet -t net1
5032023-10-05T04:23:29.101Z+ dladm create-simnet -t sc1_0
5042023-10-05T04:23:29.134Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-10-05T04:23:29.179Z+ dladm set-linkprop -p mtu=1600 sc1_0
5062023-10-05T04:23:29.293Z+ success 'Simnet net1/sc1_0 exists'
5072023-10-05T04:23:29.325Z+ set +x
5082023-10-05T04:23:29.366Z++ get_vnic_name_if_exists sc0_1
5092023-10-05T04:23:29.403Z++ dladm show-vnic -p -o LINK sc0_1
5102023-10-05T04:23:29.850Z++ echo ''
5112023-10-05T04:23:29.887Z+ [[ -z '' ]]
5122023-10-05T04:23:29.923Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5132023-10-05T04:23:29.959Z+ success 'Vnic sc0_1 exists'
5142023-10-05T04:23:30.008Z+ set +x
5152023-10-05T04:23:30.043Z+ warn_if_no_proxy_arp
5162023-10-05T04:23:30.074Z+ [[ -v PXA_START ]]
5172023-10-05T04:23:30.106Z+ [[ -v PXA_END ]]
5182023-10-05T04:23:30.139Z+ ensure_softnpu_zone
5192023-10-05T04:23:30.180Z+ zoneadm list
5202023-10-05T04:23:30.207Z+ grep -q sidecar_softnpu
5212023-10-05T04:23:30.233Z+ [[ -f out/npuzone/npuzone ]]
5222023-10-05T04:23:30.262Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0
5232023-10-05T04:23:30.741Zdownloading softnpu ...done
5242023-10-05T04:23:51.643Zdownloading asic_program.so ...done
5252023-10-05T04:24:10.698Zdownloading scadm ...done
5262023-10-05T04:24:11.135Zcreating port tfportrear0_0 for sc0_0
5272023-10-05T04:24:11.165Zcreating port tfportqsfp0_0 for sc0_1
5282023-10-05T04:24:11.197Zsoftnpu zone setup
5292023-10-05T04:24:11.236Zconfigure zone
5302023-10-05T04:24:11.276Zinstall zone
5312023-10-05T04:24:11.813Zboot zone
5322023-10-05T04:24:15.143Z[sidecar_softnpu] svcs milestone/network
5332023-10-05T04:24:15.291ZSTATE STIME FMRI
5342023-10-05T04:24:15.374Zonline 4:24:13 svc:/milestone/network:default
5352023-10-05T04:24:15.893Z[sidecar_softnpu] /softnpu/npu start
5362023-10-05T04:24:15.924Z
5372023-10-05T04:24:15.951Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5382023-10-05T04:24:15.983Z+ GATEWAY_IP=10.151.1.100
5392023-10-05T04:24:16.015Z+ echo 'Using 10.151.1.100 as gateway ip'
5402023-10-05T04:24:16.046ZUsing 10.151.1.100 as gateway ip
5412023-10-05T04:24:16.078Z+ [[ ! -v GATEWAY_MAC ]]
5422023-10-05T04:24:16.112Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5432023-10-05T04:24:16.143Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5442023-10-05T04:24:16.176Z+++ echo 18:c0:4d:0d:9f:b2
5452023-10-05T04:24:16.207Z+ COUNT=1
5462023-10-05T04:24:16.234Z+ [[ 1 -ne 1 ]]
5472023-10-05T04:24:16.268Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5482023-10-05T04:24:16.307ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5492023-10-05T04:24:16.354Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5502023-10-05T04:24:16.386Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5512023-10-05T04:24:16.414Z+ PXA_MAC=a8:e1:de:01:70:1d
5522023-10-05T04:24:16.448Z+ [[ -v PXA_START ]]
5532023-10-05T04:24:16.482Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5542023-10-05T04:24:16.512Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5552023-10-05T04:24:16.547Z+ z_scadm dump-state
5562023-10-05T04:24:16.593Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5572023-10-05T04:24:16.625Zlocal v6:
5582023-10-05T04:24:16.656Zlocal v4:
5592023-10-05T04:24:16.688Zrouter v6:
5602023-10-05T04:24:17.537Zrouter v4:
5612023-10-05T04:24:17.579Zresolver v4:
5622023-10-05T04:24:17.619Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5632023-10-05T04:24:17.658Zresolver v6:
5642023-10-05T04:24:17.698Znat v4:
5652023-10-05T04:24:17.732Znat_v6:
5662023-10-05T04:24:17.764Zport_mac:
5672023-10-05T04:24:17.842Zproxy arp:
5682023-10-05T04:24:17.872Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5692023-10-05T04:24:18.324Z+ success 'softnpu zone exists'
5702023-10-05T04:24:18.370Z+ set +x
5712023-10-05T04:24:18.419Zsoftnpu zone exists
5722023-10-05T04:24:18.457Z
5732023-10-05T04:24:18.501Zreal 1:21.654787848
5742023-10-05T04:24:18.549Zuser 0.705048803
5752023-10-05T04:24:18.587Zsys 1.595193549
5762023-10-05T04:24:19.083Ztrap 0.000108722
5772023-10-05T04:24:19.117Ztflt 0.000146097
5782023-10-05T04:24:19.152Zdflt 0.000576557
5792023-10-05T04:24:19.185Zkflt 0.000000000
5802023-10-05T04:24:19.222Zlock 1:04:49.572146170
5812023-10-05T04:24:20.410Zslp 4:04.780327211
5822023-10-05T04:24:20.448Zlat 0.047400920
5832023-10-05T04:24:20.496Zstop 0.361119849
5842023-10-05T04:24:20.539Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5852023-10-05T04:24:20.583Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5862023-10-05T04:24:20.616Z+ SILO_NAME=recovery
5872023-10-05T04:24:20.649Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5882023-10-05T04:24:21.129Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5892023-10-05T04:24:21.171Z+ sed -i~ '
5902023-10-05T04:24:21.219Z /^external_dns_ips/c\
5912023-10-05T04:24:21.252Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5922023-10-05T04:24:21.718Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5932023-10-05T04:24:22.159Z /^first/c\
5942023-10-05T04:24:22.196Zfirst = "10.151.1.164"
5952023-10-05T04:24:22.225Z /^last/c\
5962023-10-05T04:24:22.258Zlast = "10.151.1.173"
5972023-10-05T04:24:22.710Z }
5982023-10-05T04:24:22.769Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
5992023-10-05T04:24:22.830Z /^\[rack_network_config/,/^$/ {
6002023-10-05T04:24:23.279Z+++ pkg/config-rss.toml Thu Oct 5 04:24:15 2023
6012023-10-05T04:24:23.318Z /^infra_ip_first/c\
6022023-10-05T04:24:23.357Z@@ -41,7 +41,7 @@
6032023-10-05T04:24:23.396Zinfra_ip_first = "10.151.1.174"
6042023-10-05T04:24:23.429Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6052023-10-05T04:24:23.471Z /^infra_ip_last/c\
6062023-10-05T04:24:23.498Z # must be contained in one of the "internal services" IP Pool ranges listed
6072023-10-05T04:24:23.528Zinfra_ip_last = "10.151.1.174"
6082023-10-05T04:24:23.579Z # below.
6092023-10-05T04:24:23.611Z }
6102023-10-05T04:24:23.643Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6112023-10-05T04:24:23.670Z /^\[\[rack_network_config.uplinks/,/^$/ {
6122023-10-05T04:24:23.700Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6132023-10-05T04:24:24.137Z /^gateway_ip/c\
6142023-10-05T04:24:24.240Z
6152023-10-05T04:24:24.279Zgateway_ip = "10.151.1.100"
6162023-10-05T04:24:24.308Z # Initial TLS certificates for the external API
6172023-10-05T04:24:24.338Z /^uplink_cidr/c\
6182023-10-05T04:24:24.376Z #
6192023-10-05T04:24:24.413Zuplink_cidr = "10.151.1.174/32"
6202023-10-05T04:24:24.453Z@@ -77,8 +77,8 @@
6212023-10-05T04:24:24.494Z }
6222023-10-05T04:24:24.535Z #
6232023-10-05T04:24:24.568Z' pkg/config-rss.toml
6242023-10-05T04:24:24.600Z # For more on this and what to put here, see docs/how-to-run.adoc.
6252023-10-05T04:24:24.626Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6262023-10-05T04:24:24.654Z [[internal_services_ip_pool_ranges]]
6272023-10-05T04:24:24.684Z+ true
6282023-10-05T04:24:24.720Z-first = "192.168.1.20"
6292023-10-05T04:24:24.755Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6302023-10-05T04:24:24.786Z-last = "192.168.1.29"
6312023-10-05T04:24:24.817Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6322023-10-05T04:24:24.861Z+first = "10.151.1.164"
6332023-10-05T04:24:24.888Z+ TLS_NAME=recovery.sys.oxide.test
6342023-10-05T04:24:24.917Z+last = "10.151.1.173"
6352023-10-05T04:24:24.950Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6362023-10-05T04:24:25.822Z
6372023-10-05T04:24:25.852Z....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.......+......+..+....+...+......+..+............+...+....+......+...+.....+.+.....+....+.....+.......+.....+....+...+..+..................+.+....................+.......+..+...+.......+..+...+......+.+........+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+.......+..+.+......+...........+.......+.....+...+.............+..+...+............+.............+..+...+.........+.+...........+....+...........+....+..+.......+..+.+.....+.........+................+.....+...............+...+.........+......+.+......+...............+............+..+.........+..........+.....................+.....+...+..........+...+..+.+..+....+.....+....+............+..+.+.....+....+.....+.......+..................+...+..+.......+...+............+.......................+.+...............+...........+.+..+...+...............+...+..........+.........+..+...+.............+..+......+.......+..+.............+..+......+......+..........+...+...+............+............+...............+......+.........+........+...............+......+.+...............+.....+......+...+..........+..+............+.+..+.......+........+....+........+......+.........+............+...+......+.....................+.+..+...............+..........+...+...+..+...+.+...+..+.....................+...+.+...+.................+.+.....+...+.......+............+..+................+..+............+.+.........+.........+......+...........+..........+...+..+.......+...+..+.......+...+...........+..........+......+...+..+.+...+..............+......+......+...+............+.+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6382023-10-05T04:24:25.881Z # TODO - this configuration is subject to change going forward. Ultimately these
6392023-10-05T04:24:25.911Z...+...+...+.......+..............+.+.....+...+.......+..+.+....................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+...+.........+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+............+.........+.+.....+..................+.........+...+...+.+........+......................+...+...............+...+.....+...+......+...+.+.........+.....+......+.+.........+.....................+.....+.........+.+..............+.......+........+....+...+..+...+.............+.....+....+...+.....+.........+.........+.+........+............+.........+.+.......................+...............................+..+...+......+.+..+.......+..............+................+...+......+........................+....................+.+......+........+.+......+...............+...+..+...................+.....+....+...+.....+.........+.............+...+.........+.....+..........+.....+....+..+....+...+...+..............+.+...............+..+...+............+...+....+...+.....+...+...+....+..+....+...........+...+.+.........+.....+.......+...+...+...+............+..+.+......+.....+...............................+..+.............+..+......+.+........................+.........+.....+.+..............+..........+...........+.........+......+.........+......+.+..............+......+.+...+.....+....+..+....+......+.....+......+...+.+......+.....+..........+..................+..+..........+.....+....+..............+........................................+...........+...+......+.......+........+....+.................+...............+...+................+......+...+........+.........+.........+.........+.......+...+........+.........+....+............+..............+.............+............+......+..+.....................+.+..............+............+.......+..+...+................+..+....+.....................+...+...............+..+..................+....+...+...+..+............+.......+...+...............+.....+..........+..+...+.......+.....+................+..+...+...+......+.............+........+.........+...+......+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6402023-10-05T04:24:25.957Z # parameters should be provided to the control plane via wicket, but we need to
6412023-10-05T04:24:26.004Z-----
6422023-10-05T04:24:26.032Z@@ -91,13 +91,13 @@
6432023-10-05T04:24:26.062Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6442023-10-05T04:24:26.509Z # A range of IP addresses used by Boundary Services on the external network. In
6452023-10-05T04:24:27.811Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6462023-10-05T04:24:27.860Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6472023-10-05T04:24:27.898Z+ rmdir pkg
6482023-10-05T04:24:27.931Z # With softnpu, only one address is used.
6492023-10-05T04:24:27.979Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6502023-10-05T04:24:28.006Z-infra_ip_first = "192.168.1.30"
6512023-10-05T04:24:28.046Z+ pfexec mkdir -p /usr/oxide
6522023-10-05T04:24:28.087Z-infra_ip_last = "192.168.1.30"
6532023-10-05T04:24:28.538Z+ pfexec rm -f /usr/oxide/tcpproxy
6542023-10-05T04:24:28.571Z+infra_ip_first = "10.151.1.174"
6552023-10-05T04:24:28.603Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6562023-10-05T04:24:28.637Z+infra_ip_last = "10.151.1.174"
6572023-10-05T04:24:28.678Z+ pfexec chmod +x /usr/oxide/tcpproxy
6582023-10-05T04:24:29.756Z
6592023-10-05T04:24:29.814Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6602023-10-05T04:24:30.278Z # You can configure multiple uplinks by repeating the following stanza
6612023-10-05T04:24:30.330Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6622023-10-05T04:24:30.367Z [[rack_network_config.uplinks]]
6632023-10-05T04:24:30.409Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6642023-10-05T04:24:30.445Z # The gateway IP for the rack's external network
6652023-10-05T04:24:30.482Z+ OMICRON_NO_UNINSTALL=1
6662023-10-05T04:24:30.512Z-gateway_ip = "192.168.1.199"
6672023-10-05T04:24:30.539Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6682023-10-05T04:24:30.566Z+gateway_ip = "10.151.1.100"
6692023-10-05T04:24:30.605ZOct 05 04:24:18.087 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6702023-10-05T04:24:30.632Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6712023-10-05T04:24:30.658ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6722023-10-05T04:24:30.684Z uplink_port = "qsfp0"
6732023-10-05T04:24:30.711ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6742023-10-05T04:24:30.739Z uplink_port_speed = "40G"
6752023-10-05T04:24:30.772ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6762023-10-05T04:24:30.804Z@@ -104,7 +104,7 @@
6772023-10-05T04:24:30.837ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6782023-10-05T04:24:30.877Z uplink_port_fec="none"
6792023-10-05T04:24:30.923ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6802023-10-05T04:24:30.971Z # For softnpu, an address within the "infra" block above that will be used for
6812023-10-05T04:24:31.016ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6822023-10-05T04:24:31.067Z # the softnpu uplink port. You can just pick the first address in that pool.
6832023-10-05T04:24:31.118ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6842023-10-05T04:24:31.167Z-uplink_cidr = "192.168.1.30/32"
6852023-10-05T04:24:32.052ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/maghemite.tar
6862023-10-05T04:24:32.093Z+uplink_cidr = "10.151.1.174/32"
6872023-10-05T04:24:32.137ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6882023-10-05T04:24:32.620Z # Switch to use for the uplink. For single-rack deployments this can be
6892023-10-05T04:24:32.657ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6902023-10-05T04:24:32.696Z # "switch0" (upper slot) or "switch1" (lower slot). For single-node softnpu
6912023-10-05T04:24:32.743ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6922023-10-05T04:24:32.783Z # and dendrite stub environments, use "switch0"
6932023-10-05T04:24:32.823ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6942023-10-05T04:24:32.866Za pkg/config-rss.toml 7K
6952023-10-05T04:24:32.906ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6962023-10-05T04:24:33.368Za pkg/initial-tls-cert.pem 2K
6972023-10-05T04:24:33.420ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6982023-10-05T04:24:33.476Za pkg/initial-tls-key.pem 4K
6992023-10-05T04:24:33.933ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
7002023-10-05T04:24:33.988ZOct 05 04:24:18.090 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7012023-10-05T04:24:38.681ZOct 05 04:24:18.312 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7022023-10-05T04:24:39.968ZOct 05 04:24:18.355 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7032023-10-05T04:24:40.556ZOct 05 04:24:18.415 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7042023-10-05T04:24:40.611Z
7052023-10-05T04:24:41.058Zreal 0.466913984
7062023-10-05T04:24:41.100Zuser 0.096828336
7072023-10-05T04:24:41.139Zsys 1.767857628
7082023-10-05T04:24:41.179Ztrap 0.000007789
7092023-10-05T04:24:41.223Ztflt 0.000384140
7102023-10-05T04:24:41.260Zdflt 0.000333160
7112023-10-05T04:24:41.294Zkflt 0.000000000
7122023-10-05T04:24:41.327Zlock 41.989852792
7132023-10-05T04:24:41.367Zslp 0.671124770
7142023-10-05T04:24:41.832Zlat 0.003015032
7152023-10-05T04:24:41.867Zstop 0.212155562
7162023-10-05T04:24:41.893Z+ retry=0
7172023-10-05T04:24:41.927Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7182023-10-05T04:24:42.012Z+ [[ 0 -gt 30 ]]
7192023-10-05T04:24:42.064Z+ sleep 1
7202023-10-05T04:24:42.115Z+ retry=1
7212023-10-05T04:24:42.158Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-10-05T04:24:42.238Z+ [[ 1 -gt 30 ]]
7232023-10-05T04:24:42.291Z+ sleep 1
7242023-10-05T04:24:42.348Z+ retry=2
7252023-10-05T04:24:42.378Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-10-05T04:24:42.448Z+ [[ 2 -gt 30 ]]
7272023-10-05T04:24:42.526Z+ sleep 1
7282023-10-05T04:24:42.659Z+ retry=3
7292023-10-05T04:24:46.030Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-10-05T04:24:46.462Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7312023-10-05T04:24:46.625Zlocal v6:
7322023-10-05T04:24:47.254Zfd00:99::1
7332023-10-05T04:24:47.296Zfe80::4c3f:f4ff:fe2c:a290
7342023-10-05T04:24:47.343Zlocal v4:
7352023-10-05T04:24:47.377Z10.151.1.174
7362023-10-05T04:24:47.420Zrouter v6:
7372023-10-05T04:24:47.459Zrouter v4:
7382023-10-05T04:24:47.509Z0.0.0.0/0 -> 10.151.1.100 (2) vid=0
7392023-10-05T04:24:47.579Zresolver v4:
7402023-10-05T04:24:47.622Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7412023-10-05T04:24:47.669Zresolver v6:
7422023-10-05T04:24:48.240Zfe80::b006:c6ff:fec2:1efa -> b2:06:c6:c2:1e:fa
7432023-10-05T04:24:48.334Zfe80::4c3f:f4ff:fe2c:a290 -> 4e:3f:f4:2c:a2:90
7442023-10-05T04:24:48.452Znat v4:
7452023-10-05T04:24:48.521Znat_v6:
7462023-10-05T04:24:48.579Zport_mac:
7472023-10-05T04:24:48.621Z1000: a8:40:25:54:e3:6e
7482023-10-05T04:24:48.661Z1: a8:40:25:54:e3:6f
7492023-10-05T04:24:48.697Zproxy arp:
7502023-10-05T04:24:48.729Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7512023-10-05T04:24:48.766Z+ retry=0
7522023-10-05T04:24:49.227Z+++ zoneadm list -n
7532023-10-05T04:24:49.280Z+++ grep oxz_ntp
7542023-10-05T04:24:49.474Z++ pfexec svcs -z -Hostate oxide/ntp
7552023-10-05T04:24:49.516Zsvcs: zone '-Hostate': entity not found
7562023-10-05T04:24:49.560Z++ true
7572023-10-05T04:24:49.605Z+ [[ '' != online ]]
7582023-10-05T04:24:49.649Z+ [[ 0 -gt 60 ]]
7592023-10-05T04:24:49.735Z+ sleep 1
7602023-10-05T04:24:50.323Z+ retry=1
7612023-10-05T04:24:50.365Z+++ zoneadm list -n
7622023-10-05T04:24:50.406Z+++ grep oxz_ntp
7632023-10-05T04:24:50.444Z++ pfexec svcs -z -Hostate oxide/ntp
7642023-10-05T04:24:50.483Zsvcs: zone '-Hostate': entity not found
7652023-10-05T04:24:50.525Z++ true
7662023-10-05T04:24:50.988Z+ [[ '' != online ]]
7672023-10-05T04:24:51.044Z+ [[ 1 -gt 60 ]]
7682023-10-05T04:24:51.076Z+ sleep 1
7692023-10-05T04:24:51.106Z+ retry=2
7702023-10-05T04:24:51.141Z+++ zoneadm list -n
7712023-10-05T04:24:51.177Z+++ grep oxz_ntp
7722023-10-05T04:24:51.218Z++ pfexec svcs -z -Hostate oxide/ntp
7732023-10-05T04:24:51.249Zsvcs: zone '-Hostate': entity not found
7742023-10-05T04:24:51.288Z++ true
7752023-10-05T04:24:51.337Z+ [[ '' != online ]]
7762023-10-05T04:24:51.372Z+ [[ 2 -gt 60 ]]
7772023-10-05T04:24:51.409Z+ sleep 1
7782023-10-05T04:24:51.451Z+ retry=3
7792023-10-05T04:24:51.504Z+++ zoneadm list -n
7802023-10-05T04:24:51.997Z+++ grep oxz_ntp
7812023-10-05T04:24:52.042Z++ pfexec svcs -z -Hostate oxide/ntp
7822023-10-05T04:24:52.500Zsvcs: zone '-Hostate': entity not found
7832023-10-05T04:24:52.536Z++ true
7842023-10-05T04:24:52.575Z+ [[ '' != online ]]
7852023-10-05T04:24:52.617Z+ [[ 3 -gt 60 ]]
7862023-10-05T04:24:52.657Z+ sleep 1
7872023-10-05T04:24:52.699Z+ retry=4
7882023-10-05T04:24:52.740Z+++ zoneadm list -n
7892023-10-05T04:24:52.786Z+++ grep oxz_ntp
7902023-10-05T04:24:54.084Z++ pfexec svcs -z -Hostate oxide/ntp
7912023-10-05T04:24:54.410Zsvcs: zone '-Hostate': entity not found
7922023-10-05T04:24:54.468Z++ true
7932023-10-05T04:24:54.510Z+ [[ '' != online ]]
7942023-10-05T04:24:54.556Z+ [[ 4 -gt 60 ]]
7952023-10-05T04:24:54.609Z+ sleep 1
7962023-10-05T04:24:54.660Z+ retry=5
7972023-10-05T04:24:55.139Z+++ zoneadm list -n
7982023-10-05T04:24:55.601Z+++ grep oxz_ntp
7992023-10-05T04:24:55.645Z++ pfexec svcs -z -Hostate oxide/ntp
8002023-10-05T04:24:55.692Zsvcs: zone '-Hostate': entity not found
8012023-10-05T04:24:55.736Z++ true
8022023-10-05T04:24:55.779Z+ [[ '' != online ]]
8032023-10-05T04:24:55.823Z+ [[ 5 -gt 60 ]]
8042023-10-05T04:24:55.865Z+ sleep 1
8052023-10-05T04:24:55.909Z+ retry=6
8062023-10-05T04:24:57.196Z+++ zoneadm list -n
8072023-10-05T04:24:57.248Z+++ grep oxz_ntp
8082023-10-05T04:24:57.287Z++ pfexec svcs -z -Hostate oxide/ntp
8092023-10-05T04:24:57.323Zsvcs: zone '-Hostate': entity not found
8102023-10-05T04:24:57.360Z++ true
8112023-10-05T04:24:57.398Z+ [[ '' != online ]]
8122023-10-05T04:24:57.465Z+ [[ 6 -gt 60 ]]
8132023-10-05T04:24:57.515Z+ sleep 1
8142023-10-05T04:24:57.969Z+ retry=7
8152023-10-05T04:24:58.012Z+++ zoneadm list -n
8162023-10-05T04:24:58.051Z+++ grep oxz_ntp
8172023-10-05T04:24:58.099Z++ pfexec svcs -z -Hostate oxide/ntp
8182023-10-05T04:24:58.147Zsvcs: zone '-Hostate': entity not found
8192023-10-05T04:24:58.203Z++ true
8202023-10-05T04:24:58.257Z+ [[ '' != online ]]
8212023-10-05T04:24:58.317Z+ [[ 7 -gt 60 ]]
8222023-10-05T04:24:59.218Z+ sleep 1
8232023-10-05T04:24:59.259Z+ retry=8
8242023-10-05T04:24:59.304Z+++ zoneadm list -n
8252023-10-05T04:24:59.347Z+++ grep oxz_ntp
8262023-10-05T04:24:59.393Z++ pfexec svcs -z -Hostate oxide/ntp
8272023-10-05T04:24:59.439Zsvcs: zone '-Hostate': entity not found
8282023-10-05T04:24:59.609Z++ true
8292023-10-05T04:24:59.693Z+ [[ '' != online ]]
8302023-10-05T04:24:59.749Z+ [[ 8 -gt 60 ]]
8312023-10-05T04:25:00.224Z+ sleep 1
8322023-10-05T04:25:00.271Z+ retry=9
8332023-10-05T04:25:00.312Z+++ zoneadm list -n
8342023-10-05T04:25:00.356Z+++ grep oxz_ntp
8352023-10-05T04:25:00.398Z++ pfexec svcs -z -Hostate oxide/ntp
8362023-10-05T04:25:00.433Zsvcs: zone '-Hostate': entity not found
8372023-10-05T04:25:00.465Z++ true
8382023-10-05T04:25:00.512Z+ [[ '' != online ]]
8392023-10-05T04:25:00.551Z+ [[ 9 -gt 60 ]]
8402023-10-05T04:25:00.690Z+ sleep 1
8412023-10-05T04:25:00.772Z+ retry=10
8422023-10-05T04:25:00.830Z+++ zoneadm list -n
8432023-10-05T04:25:00.873Z+++ grep oxz_ntp
8442023-10-05T04:25:00.911Z++ pfexec svcs -z -Hostate oxide/ntp
8452023-10-05T04:25:00.942Zsvcs: zone '-Hostate': entity not found
8462023-10-05T04:25:00.979Z++ true
8472023-10-05T04:25:01.023Z+ [[ '' != online ]]
8482023-10-05T04:25:01.511Z+ [[ 10 -gt 60 ]]
8492023-10-05T04:25:01.569Z+ sleep 1
8502023-10-05T04:25:01.616Z+ retry=11
8512023-10-05T04:25:02.080Z+++ zoneadm list -n
8522023-10-05T04:25:02.767Z+++ grep oxz_ntp
8532023-10-05T04:25:03.016Z++ pfexec svcs -z -Hostate oxide/ntp
8542023-10-05T04:25:03.093Zsvcs: zone '-Hostate': entity not found
8552023-10-05T04:25:03.142Z++ true
8562023-10-05T04:25:03.200Z+ [[ '' != online ]]
8572023-10-05T04:25:03.250Z+ [[ 11 -gt 60 ]]
8582023-10-05T04:25:03.704Z+ sleep 1
8592023-10-05T04:25:03.746Z+ retry=12
8602023-10-05T04:25:03.801Z+++ zoneadm list -n
8612023-10-05T04:25:03.870Z+++ grep oxz_ntp
8622023-10-05T04:25:04.346Z++ pfexec svcs -z -Hostate oxide/ntp
8632023-10-05T04:25:04.394Zsvcs: zone '-Hostate': entity not found
8642023-10-05T04:25:04.437Z++ true
8652023-10-05T04:25:04.473Z+ [[ '' != online ]]
8662023-10-05T04:25:04.929Z+ [[ 12 -gt 60 ]]
8672023-10-05T04:25:05.227Z+ sleep 1
8682023-10-05T04:25:05.268Z+ retry=13
8692023-10-05T04:25:05.315Z+++ zoneadm list -n
 
8702023-10-05T04:25:05.473Zbuskin console login: Oct 5 04:25:05 bukin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8712023-10-05T04:25:05.779Z+++ grep oxz_ntp
8722023-10-05T04:25:05.828Z++ pfexec svcs -z oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01 -Hostate oxide/ntp
8732023-10-05T04:25:05.876Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8742023-10-05T04:25:06.350Z++ true
8752023-10-05T04:25:06.395Z+ [[ '' != online ]]
8762023-10-05T04:25:06.865Z+ [[ 13 -gt 60 ]]
8772023-10-05T04:25:06.919Z+ sleep 1
8782023-10-05T04:25:06.982Z+ retry=14
8792023-10-05T04:25:07.477Z+++ zoneadm list -n
8802023-10-05T04:25:07.518Z+++ grep oxz_ntp
8812023-10-05T04:25:07.980Z++ pfexec svcs -z oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01 -Hostate oxide/ntp
8822023-10-05T04:25:08.439Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8832023-10-05T04:25:08.493Z++ true
8842023-10-05T04:25:08.954Z+ [[ '' != online ]]
8852023-10-05T04:25:09.008Z+ [[ 14 -gt 60 ]]
8862023-10-05T04:25:09.616Z+ sleep 1
8872023-10-05T04:25:09.659Z+ retry=15
8882023-10-05T04:25:09.702Z+++ zoneadm list -n
8892023-10-05T04:25:09.746Z+++ grep oxz_ntp
8902023-10-05T04:25:10.249Z++ pfexec svcs -z oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01 -Hostate oxide/ntp
8912023-10-05T04:25:10.283Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8922023-10-05T04:25:10.313Z++ true
8932023-10-05T04:25:10.347Z+ [[ '' != online ]]
8942023-10-05T04:25:10.380Z+ [[ 15 -gt 60 ]]
8952023-10-05T04:25:10.418Z+ sleep 1
8962023-10-05T04:25:10.451Z+ retry=16
8972023-10-05T04:25:10.485Z+++ zoneadm list -n
8982023-10-05T04:25:10.523Z+++ grep oxz_ntp
8992023-10-05T04:25:10.570Z++ pfexec svcs -z oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01 -Hostate oxide/ntp
9002023-10-05T04:25:10.621Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9012023-10-05T04:25:10.663Z++ true
9022023-10-05T04:25:10.704Z+ [[ '' != online ]]
9032023-10-05T04:25:10.740Z+ [[ 16 -gt 60 ]]
9042023-10-05T04:25:10.779Z+ sleep 1
9052023-10-05T04:25:10.811Z+ retry=17
9062023-10-05T04:25:10.854Z+++ zoneadm list -n
9072023-10-05T04:25:10.896Z+++ grep oxz_ntp
9082023-10-05T04:25:10.927Z++ pfexec svcs -z oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01 -Hostate oxide/ntp
9092023-10-05T04:25:10.960Z+ [[ disabled != online ]]
9102023-10-05T04:25:10.996Z+ [[ 17 -gt 60 ]]
9112023-10-05T04:25:11.036Z+ sleep 1
9122023-10-05T04:25:11.080Z+ retry=18
9132023-10-05T04:25:11.135Z+++ zoneadm list -n
9142023-10-05T04:25:11.176Z+++ grep oxz_ntp
9152023-10-05T04:25:11.221Z++ pfexec svcs -z oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01 -Hostate oxide/ntp
9162023-10-05T04:25:11.286Z+ [[ disabled != online ]]
9172023-10-05T04:25:11.327Z+ [[ 18 -gt 60 ]]
9182023-10-05T04:25:11.368Z+ sleep 1
9192023-10-05T04:25:11.409Z+ retry=19
9202023-10-05T04:25:11.446Z+++ zoneadm list -n
9212023-10-05T04:25:11.491Z+++ grep oxz_ntp
9222023-10-05T04:25:11.961Z++ pfexec svcs -z oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01 -Hostate oxide/ntp
9232023-10-05T04:25:12.004Z+ [[ disabled != online ]]
9242023-10-05T04:25:12.040Z+ [[ 19 -gt 60 ]]
9252023-10-05T04:25:12.489Z+ sleep 1
9262023-10-05T04:25:12.533Z+ retry=20
9272023-10-05T04:25:12.576Z+++ zoneadm list -n
9282023-10-05T04:25:13.042Z+++ grep oxz_ntp
9292023-10-05T04:25:13.089Z++ pfexec svcs -z oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01 -Hostate oxide/ntp
9302023-10-05T04:25:13.553Z+ [[ disabled != online ]]
9312023-10-05T04:25:13.585Z+ [[ 20 -gt 60 ]]
9322023-10-05T04:25:13.627Z+ sleep 1
9332023-10-05T04:25:13.673Z+ retry=21
9342023-10-05T04:25:13.712Z+++ zoneadm list -n
9352023-10-05T04:25:13.753Z+++ grep oxz_ntp
9362023-10-05T04:25:13.796Z++ pfexec svcs -z oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01 -Hostate oxide/ntp
9372023-10-05T04:25:13.839Z+ [[ disabled != online ]]
9382023-10-05T04:25:13.883Z+ [[ 21 -gt 60 ]]
9392023-10-05T04:25:13.927Z+ sleep 1
9402023-10-05T04:25:15.235ZWaited for chrony: 25s
9412023-10-05T04:25:16.123Z+ retry=22
9422023-10-05T04:25:16.382Z+++ zoneadm list -n
9432023-10-05T04:25:16.429Z+++ grep oxz_ntp
9442023-10-05T04:25:16.466Z++ pfexec svcs -z oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01 -Hostate oxide/ntp
9452023-10-05T04:25:16.509Z+ [[ disabled != online ]]
9462023-10-05T04:25:16.969Z+ [[ 22 -gt 60 ]]
9472023-10-05T04:25:17.013Z+ sleep 1
9482023-10-05T04:25:17.055Z+ retry=23
9492023-10-05T04:25:17.103Z+++ zoneadm list -n
9502023-10-05T04:25:17.998Z+++ grep oxz_ntp
9512023-10-05T04:25:18.041Z++ pfexec svcs -z oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01 -Hostate oxide/ntp
9522023-10-05T04:25:18.079Z+ [[ disabled != online ]]
9532023-10-05T04:25:19.024Z+ [[ 23 -gt 60 ]]
9542023-10-05T04:25:19.064Z+ sleep 1
9552023-10-05T04:25:19.192Z+ retry=24
9562023-10-05T04:25:19.236Z+++ zoneadm list -n
9572023-10-05T04:25:19.699Z+++ grep oxz_ntp
9582023-10-05T04:25:19.740Z++ pfexec svcs -z oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01 -Hostate oxide/ntp
9592023-10-05T04:25:19.779Z+ [[ disabled != online ]]
9602023-10-05T04:25:19.811Z+ [[ 24 -gt 60 ]]
9612023-10-05T04:25:19.847Z+ sleep 1
9622023-10-05T04:25:19.881Z+ retry=25
9632023-10-05T04:25:19.919Z+++ zoneadm list -n
9642023-10-05T04:25:20.372Z+++ grep oxz_ntp
9652023-10-05T04:25:20.409Z++ pfexec svcs -z oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01 -Hostate oxide/ntp
9662023-10-05T04:25:20.447Z+ [[ online != online ]]
9672023-10-05T04:25:20.491Z+ echo 'Waited for chrony: 25s'
9682023-10-05T04:25:20.974Z+ export RUST_BACKTRACE=1
9692023-10-05T04:25:21.011Z+ RUST_BACKTRACE=1
9702023-10-05T04:25:22.283Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9712023-10-05T04:25:22.323Z+ ./tests/bootstrap
9722023-10-05T04:25:22.363ZUsing external DNS server at 10.151.1.164:53
9732023-10-05T04:25:23.255Z2023-10-05 04:25:06.838012147 UTC: attempting to log into API
9742023-10-05T04:25:24.197Z2023-10-05 04:25:21.459370006 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9752023-10-05T04:25:24.256Z2023-10-05 04:25:22.460312010 UTC: attempting to log into API
9762023-10-05T04:25:39.577Z2023-10-05 04:25:37.493196103 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9772023-10-05T04:25:40.330Z2023-10-05 04:25:38.494109667 UTC: attempting to log into API
9782023-10-05T04:26:01.237Z2023-10-05 04:25:53.525934448 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9792023-10-05T04:26:04.858Z2023-10-05 04:25:54.527083449 UTC: attempting to log into API
9802023-10-05T04:26:24.856Z2023-10-05 04:26:09.559337323 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9812023-10-05T04:26:25.069Z2023-10-05 04:26:10.560458422 UTC: attempting to log into API
9822023-10-05T04:26:25.736Z2023-10-05 04:26:25.190946713 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9832023-10-05T04:26:26.524Z2023-10-05 04:26:26.194763900 UTC: attempting to log into API
9842023-10-05T04:26:41.531Z2023-10-05 04:26:41.268886974 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9852023-10-05T04:26:42.536Z2023-10-05 04:26:42.273436608 UTC: attempting to log into API
9862023-10-05T04:26:57.564Z2023-10-05 04:26:57.347789554 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9872023-10-05T04:26:58.562Z2023-10-05 04:26:58.351555651 UTC: attempting to log into API
9882023-10-05T04:27:13.612Z2023-10-05 04:27:13.425303695 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9892023-10-05T04:27:14.597Z2023-10-05 04:27:14.428995075 UTC: attempting to log into API
9902023-10-05T04:27:29.634Z2023-10-05 04:27:29.502977494 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9912023-10-05T04:27:30.638Z2023-10-05 04:27:30.506808671 UTC: attempting to log into API
9922023-10-05T04:27:45.826Z2023-10-05 04:27:45.581209393 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9932023-10-05T04:27:47.263Z2023-10-05 04:27:46.584967116 UTC: attempting to log into API
9942023-10-05T04:28:08.086Z2023-10-05 04:28:01.659048499 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9952023-10-05T04:28:25.097Z2023-10-05 04:28:02.662804918 UTC: attempting to log into API
9962023-10-05T04:28:26.040Z2023-10-05 04:28:17.720813222 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9972023-10-05T04:28:29.261Z2023-10-05 04:28:18.721691474 UTC: attempting to log into API
9982023-10-05T04:28:33.777Z2023-10-05 04:28:33.754159568 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9992023-10-05T04:28:34.784Z2023-10-05 04:28:34.755265467 UTC: attempting to log into API
10002023-10-05T04:28:49.815Z2023-10-05 04:28:49.787624747 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10012023-10-05T04:28:50.804Z2023-10-05 04:28:50.788669905 UTC: attempting to log into API
10022023-10-05T04:29:05.916Z2023-10-05 04:29:05.822004074 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10032023-10-05T04:29:06.844Z2023-10-05 04:29:06.822541076 UTC: attempting to log into API
10042023-10-05T04:29:22.693Z2023-10-05 04:29:22.671993112 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10052023-10-05T04:29:23.713Z2023-10-05 04:29:23.672717090 UTC: attempting to log into API
 
10062023-10-05T04:29:37.287ZOct 5 04:29:36 bukin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
10072023-10-05T04:29:38.997Z2023-10-05 04:29:38.711434601 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10082023-10-05T04:29:46.867Z2023-10-05 04:29:39.712387595 UTC: attempting to log into API
 
10092023-10-05T04:29:47.146ZOct 5 04:29:44 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
10102023-10-05T04:29:51.340ZOct 5 04:29:50 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
10112023-10-05T04:29:54.771Z2023-10-05 04:29:54.744692656 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10122023-10-05T04:29:55.766Z2023-10-05 04:29:55.745453655 UTC: attempting to log into API
 
10132023-10-05T04:29:57.341ZOct 5 04:29:56 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
10142023-10-05T04:30:06.087ZOct 5 04:30:05 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
10152023-10-05T04:30:11.210Z2023-10-05 04:30:10.778703452 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10162023-10-05T04:30:11.800Z2023-10-05 04:30:11.779748880 UTC: attempting to log into API
10172023-10-05T04:30:26.832Z2023-10-05 04:30:26.812137701 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10182023-10-05T04:30:27.837Z2023-10-05 04:30:27.812987132 UTC: attempting to log into API
10192023-10-05T04:30:33.609Z2023-10-05 04:30:33.593110113 UTC: login succeeded
10202023-10-05T04:30:33.647Zwaiting for nexus to come up...
10212023-10-05T04:30:33.711Zcreating IP pool... 10.151.1.175 - 10.151.1.227
10222023-10-05T04:30:33.764Zensuring datasets are ready...
10232023-10-05T04:30:35.592Zlet's roll.
10242023-10-05T04:30:35.626Z+ rm ./tests/bootstrap
10252023-10-05T04:30:35.662Z+ for test_bin in tests/*
10262023-10-05T04:30:35.702Z+ ./tests/bootstrap-6e3d6cace6bb341c
10272023-10-05T04:30:35.741Z
10282023-10-05T04:30:36.233Zrunning 0 tests
10292023-10-05T04:30:36.267Z
10302023-10-05T04:30:36.302Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10312023-10-05T04:30:37.162Z
10322023-10-05T04:30:37.605Z+ for test_bin in tests/*
10332023-10-05T04:30:38.044Z+ ./tests/end_to_end_tests-2d9aa87f7c0ddffa
10342023-10-05T04:30:38.486Z
10352023-10-05T04:30:38.919Zrunning 2 tests
10362023-10-05T04:30:38.951Ztest no_spoof::no_spoof_here ... ok
10372023-10-05T04:31:35.671Ztest instance_launch::instance_launch has been running for over 60 seconds
10382023-10-05T04:33:32.849Ztest instance_launch::instance_launch ... ok
10392023-10-05T04:33:32.889Z
10402023-10-05T04:33:32.930Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 177.17s
10412023-10-05T04:33:32.975Z
10422023-10-05T04:33:33.042Z+ _exit_trap
10432023-10-05T04:33:33.093Z+ local status=0
10442023-10-05T04:33:33.148Z+ [[ 0 -eq 0 ]]
10452023-10-05T04:33:33.613Z+ exit 0
10462023-10-05T04:33:33.660Zprocess exited: duration 647684 ms, exit code 0
 
10472023-10-05T04:33:33.825Zfound 84 output files
10482023-10-05T04:33:34.561Zuploading: /var/svc/log/oxide-sled-agent:default.log (446641 bytes)
10492023-10-05T04:33:36.384Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10502023-10-05T04:33:36.422Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_92ba487b-456e-430a-81dd-ad7be4bbeed5/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10512023-10-05T04:33:37.951Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_92ba487b-456e-430a-81dd-ad7be4bbeed5/root/var/svc/log/oxide-cockroachdb:default.log
10522023-10-05T04:33:37.984Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_92ba487b-456e-430a-81dd-ad7be4bbeed5/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
10532023-10-05T04:33:39.753Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_92ba487b-456e-430a-81dd-ad7be4bbeed5/root/var/svc/log/oxide-cockroachdb:default.log.0
10542023-10-05T04:33:39.796Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1cd6e990-f200-439f-9d4e-5f852125ad65/root/var/svc/log/oxide-crucible-agent:default.log (10434 bytes)
10552023-10-05T04:33:41.427Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1cd6e990-f200-439f-9d4e-5f852125ad65/root/var/svc/log/oxide-crucible-agent:default.log
10562023-10-05T04:33:41.462Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1cd6e990-f200-439f-9d4e-5f852125ad65/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10572023-10-05T04:33:42.624Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1cd6e990-f200-439f-9d4e-5f852125ad65/root/var/svc/log/oxide-crucible-agent:default.log.0
10582023-10-05T04:33:42.660Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1cd6e990-f200-439f-9d4e-5f852125ad65/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10592023-10-05T04:33:45.811Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1cd6e990-f200-439f-9d4e-5f852125ad65/root/var/svc/log/oxide-crucible-downstairs:default.log
10602023-10-05T04:33:46.259Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1cd6e990-f200-439f-9d4e-5f852125ad65/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10612023-10-05T04:33:47.434Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1cd6e990-f200-439f-9d4e-5f852125ad65/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10622023-10-05T04:33:47.474Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1cd6e990-f200-439f-9d4e-5f852125ad65/root/var/svc/log/oxide-crucible-downstairs:downstairs-9f2cb997-1e3c-4007-bdd6-d52037b42800.log (5941 bytes)
10632023-10-05T04:33:49.091Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1cd6e990-f200-439f-9d4e-5f852125ad65/root/var/svc/log/oxide-crucible-downstairs:downstairs-9f2cb997-1e3c-4007-bdd6-d52037b42800.log
10642023-10-05T04:33:49.593Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b062d788-bfd7-447a-be72-b8e0a63844b7/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10652023-10-05T04:33:51.405Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b062d788-bfd7-447a-be72-b8e0a63844b7/root/var/svc/log/oxide-crucible-pantry:default.log
10662023-10-05T04:33:51.444Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b062d788-bfd7-447a-be72-b8e0a63844b7/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10672023-10-05T04:33:54.685Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b062d788-bfd7-447a-be72-b8e0a63844b7/root/var/svc/log/oxide-crucible-pantry:default.log.0
10682023-10-05T04:33:54.725Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_85d1feb9-e59c-48b5-b944-2b6965706d27/root/var/svc/log/oxide-external_dns:default.log (13049 bytes)
10692023-10-05T04:33:57.302Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_85d1feb9-e59c-48b5-b944-2b6965706d27/root/var/svc/log/oxide-external_dns:default.log
10702023-10-05T04:33:57.337Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_85d1feb9-e59c-48b5-b944-2b6965706d27/root/var/svc/log/oxide-external_dns:default.log.0 (2553 bytes)
10712023-10-05T04:33:58.442Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_85d1feb9-e59c-48b5-b944-2b6965706d27/root/var/svc/log/oxide-external_dns:default.log.0
10722023-10-05T04:33:58.469Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_1914e18d-7561-484d-b533-49aa67af4d9f/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10732023-10-05T04:33:59.540Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_1914e18d-7561-484d-b533-49aa67af4d9f/root/var/svc/log/oxide-cockroachdb:default.log
10742023-10-05T04:33:59.605Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_1914e18d-7561-484d-b533-49aa67af4d9f/root/var/svc/log/oxide-cockroachdb:default.log.0 (5344 bytes)
10752023-10-05T04:34:01.560Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_1914e18d-7561-484d-b533-49aa67af4d9f/root/var/svc/log/oxide-cockroachdb:default.log.0
10762023-10-05T04:34:01.635Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_01b47ffb-ac69-4dcb-a6f7-c7f53cb8d494/root/var/svc/log/oxide-crucible-agent:default.log (10893 bytes)
10772023-10-05T04:34:03.189Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_01b47ffb-ac69-4dcb-a6f7-c7f53cb8d494/root/var/svc/log/oxide-crucible-agent:default.log
10782023-10-05T04:34:03.302Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_01b47ffb-ac69-4dcb-a6f7-c7f53cb8d494/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
10792023-10-05T04:34:05.338Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_01b47ffb-ac69-4dcb-a6f7-c7f53cb8d494/root/var/svc/log/oxide-crucible-agent:default.log.0
10802023-10-05T04:34:05.401Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_01b47ffb-ac69-4dcb-a6f7-c7f53cb8d494/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10812023-10-05T04:34:09.864Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_01b47ffb-ac69-4dcb-a6f7-c7f53cb8d494/root/var/svc/log/oxide-crucible-downstairs:default.log
10822023-10-05T04:34:11.103Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_01b47ffb-ac69-4dcb-a6f7-c7f53cb8d494/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10832023-10-05T04:34:11.141Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_01b47ffb-ac69-4dcb-a6f7-c7f53cb8d494/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10842023-10-05T04:34:11.654Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_01b47ffb-ac69-4dcb-a6f7-c7f53cb8d494/root/var/svc/log/oxide-crucible-downstairs:downstairs-17a3ff10-e6ff-4671-a042-f265bacd87da.log (3480 bytes)
10852023-10-05T04:34:11.723Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_01b47ffb-ac69-4dcb-a6f7-c7f53cb8d494/root/var/svc/log/oxide-crucible-downstairs:downstairs-17a3ff10-e6ff-4671-a042-f265bacd87da.log
10862023-10-05T04:34:11.775Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cfd1ee3f-222a-44d3-ac3c-9c8407a6fe24/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10872023-10-05T04:34:12.740Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cfd1ee3f-222a-44d3-ac3c-9c8407a6fe24/root/var/svc/log/oxide-crucible-agent:default.log
10882023-10-05T04:34:12.801Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cfd1ee3f-222a-44d3-ac3c-9c8407a6fe24/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
10892023-10-05T04:34:14.918Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cfd1ee3f-222a-44d3-ac3c-9c8407a6fe24/root/var/svc/log/oxide-crucible-agent:default.log.0
10902023-10-05T04:34:14.970Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cfd1ee3f-222a-44d3-ac3c-9c8407a6fe24/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10912023-10-05T04:34:15.889Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cfd1ee3f-222a-44d3-ac3c-9c8407a6fe24/root/var/svc/log/oxide-crucible-downstairs:default.log
10922023-10-05T04:34:15.923Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cfd1ee3f-222a-44d3-ac3c-9c8407a6fe24/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10932023-10-05T04:34:17.435Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_cfd1ee3f-222a-44d3-ac3c-9c8407a6fe24/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10942023-10-05T04:34:17.462Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_20efcdfa-335c-4cf2-8ee4-b78921507f8f/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10952023-10-05T04:34:19.036Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_20efcdfa-335c-4cf2-8ee4-b78921507f8f/root/var/svc/log/oxide-crucible-pantry:default.log
10962023-10-05T04:34:19.082Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_20efcdfa-335c-4cf2-8ee4-b78921507f8f/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
10972023-10-05T04:34:21.801Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_20efcdfa-335c-4cf2-8ee4-b78921507f8f/root/var/svc/log/oxide-crucible-pantry:default.log.0
10982023-10-05T04:34:21.844Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_da727086-863a-4124-8455-97ed8565ee63/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10992023-10-05T04:34:23.353Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_da727086-863a-4124-8455-97ed8565ee63/root/var/svc/log/oxide-crucible-pantry:default.log
11002023-10-05T04:34:23.815Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_da727086-863a-4124-8455-97ed8565ee63/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
11012023-10-05T04:34:24.911Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_da727086-863a-4124-8455-97ed8565ee63/root/var/svc/log/oxide-crucible-pantry:default.log.0
11022023-10-05T04:34:24.937Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_633366be-b711-4da7-908b-e832e1c64a8f/root/var/svc/log/oxide-external_dns:default.log (15485 bytes)
11032023-10-05T04:34:25.058Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_633366be-b711-4da7-908b-e832e1c64a8f/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15485 -> 19250
11042023-10-05T04:34:26.608Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_633366be-b711-4da7-908b-e832e1c64a8f/root/var/svc/log/oxide-external_dns:default.log
11052023-10-05T04:34:26.687Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_633366be-b711-4da7-908b-e832e1c64a8f/root/var/svc/log/oxide-external_dns:default.log.0 (75 bytes)
11062023-10-05T04:34:28.239Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_633366be-b711-4da7-908b-e832e1c64a8f/root/var/svc/log/oxide-external_dns:default.log.0
11072023-10-05T04:34:28.390Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01/root/var/svc/log/oxide-ntp:default.log (0 bytes)
11082023-10-05T04:34:37.541Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01/root/var/svc/log/oxide-ntp:default.log
11092023-10-05T04:34:38.087Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01/root/var/svc/log/oxide-ntp:default.log.0 (1247 bytes)
11102023-10-05T04:34:40.078Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01/root/var/svc/log/oxide-ntp:default.log.0
11112023-10-05T04:34:40.148Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_1b7845b8-93eb-48d6-9f23-6efb4339357f/root/var/svc/log/oxide-clickhouse:default.log (555370 bytes)
11122023-10-05T04:34:48.314Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_1b7845b8-93eb-48d6-9f23-6efb4339357f/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 555370 -> 804113
11132023-10-05T04:34:49.830Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_1b7845b8-93eb-48d6-9f23-6efb4339357f/root/var/svc/log/oxide-clickhouse:default.log
11142023-10-05T04:34:49.888Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_1b7845b8-93eb-48d6-9f23-6efb4339357f/root/var/svc/log/oxide-clickhouse:default.log.0 (98031 bytes)
11152023-10-05T04:34:51.762Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_1b7845b8-93eb-48d6-9f23-6efb4339357f/root/var/svc/log/oxide-clickhouse:default.log.0
11162023-10-05T04:34:51.803Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_009b1cd8-7063-4228-8247-c5fb181590f9/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11172023-10-05T04:34:52.869Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_009b1cd8-7063-4228-8247-c5fb181590f9/root/var/svc/log/oxide-crucible-agent:default.log
11182023-10-05T04:34:52.907Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_009b1cd8-7063-4228-8247-c5fb181590f9/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11192023-10-05T04:34:54.014Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_009b1cd8-7063-4228-8247-c5fb181590f9/root/var/svc/log/oxide-crucible-agent:default.log.0
11202023-10-05T04:34:54.468Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_009b1cd8-7063-4228-8247-c5fb181590f9/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11212023-10-05T04:34:55.539Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_009b1cd8-7063-4228-8247-c5fb181590f9/root/var/svc/log/oxide-crucible-downstairs:default.log
11222023-10-05T04:34:55.569Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_009b1cd8-7063-4228-8247-c5fb181590f9/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11232023-10-05T04:34:56.327Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_009b1cd8-7063-4228-8247-c5fb181590f9/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11242023-10-05T04:34:56.371Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_a5e35855-7574-4834-ad4f-3c7f177f21cc/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11252023-10-05T04:34:57.879Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_a5e35855-7574-4834-ad4f-3c7f177f21cc/root/var/svc/log/oxide-crucible-agent:default.log
11262023-10-05T04:34:58.377Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_a5e35855-7574-4834-ad4f-3c7f177f21cc/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11272023-10-05T04:34:59.084Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_a5e35855-7574-4834-ad4f-3c7f177f21cc/root/var/svc/log/oxide-crucible-agent:default.log.0
11282023-10-05T04:34:59.122Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_a5e35855-7574-4834-ad4f-3c7f177f21cc/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11292023-10-05T04:35:00.996Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_a5e35855-7574-4834-ad4f-3c7f177f21cc/root/var/svc/log/oxide-crucible-downstairs:default.log
11302023-10-05T04:35:01.040Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_a5e35855-7574-4834-ad4f-3c7f177f21cc/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11312023-10-05T04:35:02.161Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_a5e35855-7574-4834-ad4f-3c7f177f21cc/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11322023-10-05T04:35:02.207Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_a9959bd7-9491-4d12-8697-b845f0233e4f/root/var/svc/log/oxide-nexus:default.log (644545 bytes)
11332023-10-05T04:35:03.741Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_a9959bd7-9491-4d12-8697-b845f0233e4f/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 644545 -> 708079
11342023-10-05T04:35:06.097Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_a9959bd7-9491-4d12-8697-b845f0233e4f/root/var/svc/log/oxide-nexus:default.log
11352023-10-05T04:35:06.127Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_a9959bd7-9491-4d12-8697-b845f0233e4f/root/var/svc/log/oxide-nexus:default.log.0 (120 bytes)
11362023-10-05T04:35:07.239Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_a9959bd7-9491-4d12-8697-b845f0233e4f/root/var/svc/log/oxide-nexus:default.log.0
11372023-10-05T04:35:07.291Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_ee98c163-0a0f-4a91-a41c-01a13dc1e53b/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11382023-10-05T04:35:08.374Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_ee98c163-0a0f-4a91-a41c-01a13dc1e53b/root/var/svc/log/oxide-cockroachdb:default.log
11392023-10-05T04:35:08.431Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_ee98c163-0a0f-4a91-a41c-01a13dc1e53b/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
11402023-10-05T04:35:09.989Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_ee98c163-0a0f-4a91-a41c-01a13dc1e53b/root/var/svc/log/oxide-cockroachdb:default.log.0
11412023-10-05T04:35:10.025Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1ebb93e8-114b-4dee-96d7-0df975eb559d/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11422023-10-05T04:35:16.857Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1ebb93e8-114b-4dee-96d7-0df975eb559d/root/var/svc/log/oxide-crucible-agent:default.log
11432023-10-05T04:35:17.417Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1ebb93e8-114b-4dee-96d7-0df975eb559d/root/var/svc/log/oxide-crucible-agent:default.log.0 (3433 bytes)
11442023-10-05T04:35:19.206Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1ebb93e8-114b-4dee-96d7-0df975eb559d/root/var/svc/log/oxide-crucible-agent:default.log.0
11452023-10-05T04:35:19.240Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1ebb93e8-114b-4dee-96d7-0df975eb559d/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11462023-10-05T04:35:20.736Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1ebb93e8-114b-4dee-96d7-0df975eb559d/root/var/svc/log/oxide-crucible-downstairs:default.log
11472023-10-05T04:35:20.778Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1ebb93e8-114b-4dee-96d7-0df975eb559d/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11482023-10-05T04:35:21.503Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1ebb93e8-114b-4dee-96d7-0df975eb559d/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11492023-10-05T04:35:21.548Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_b8b860a7-61e8-4aca-8b77-c7f46d1bdf36/root/var/svc/log/oxide-internal_dns:default.log (12513 bytes)
11502023-10-05T04:35:21.722Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_b8b860a7-61e8-4aca-8b77-c7f46d1bdf36/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 12513 -> 20024
11512023-10-05T04:35:22.846Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_b8b860a7-61e8-4aca-8b77-c7f46d1bdf36/root/var/svc/log/oxide-internal_dns:default.log
11522023-10-05T04:35:22.872Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_b8b860a7-61e8-4aca-8b77-c7f46d1bdf36/root/var/svc/log/oxide-internal_dns:default.log.0 (102164 bytes)
11532023-10-05T04:35:24.263Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_b8b860a7-61e8-4aca-8b77-c7f46d1bdf36/root/var/svc/log/oxide-internal_dns:default.log.0
11542023-10-05T04:35:24.787Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_ec5e52d2-e12b-4888-b1ec-8fc384e8ade1/root/var/svc/log/oxide-internal_dns:default.log (12516 bytes)
11552023-10-05T04:35:24.819Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_ec5e52d2-e12b-4888-b1ec-8fc384e8ade1/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 12516 -> 20026
11562023-10-05T04:35:25.592Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_ec5e52d2-e12b-4888-b1ec-8fc384e8ade1/root/var/svc/log/oxide-internal_dns:default.log
11572023-10-05T04:35:26.045Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_ec5e52d2-e12b-4888-b1ec-8fc384e8ade1/root/var/svc/log/oxide-internal_dns:default.log.0 (106209 bytes)
11582023-10-05T04:35:29.388Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_ec5e52d2-e12b-4888-b1ec-8fc384e8ade1/root/var/svc/log/oxide-internal_dns:default.log.0
11592023-10-05T04:35:29.420Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20e29a21-5c3b-40a0-9ad8-9e3e070c819f/root/var/svc/log/oxide-crucible-agent:default.log (10433 bytes)
11602023-10-05T04:35:31.535Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20e29a21-5c3b-40a0-9ad8-9e3e070c819f/root/var/svc/log/oxide-crucible-agent:default.log
11612023-10-05T04:35:31.975Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20e29a21-5c3b-40a0-9ad8-9e3e070c819f/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11622023-10-05T04:35:33.068Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20e29a21-5c3b-40a0-9ad8-9e3e070c819f/root/var/svc/log/oxide-crucible-agent:default.log.0
11632023-10-05T04:35:33.093Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20e29a21-5c3b-40a0-9ad8-9e3e070c819f/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11642023-10-05T04:35:35.031Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20e29a21-5c3b-40a0-9ad8-9e3e070c819f/root/var/svc/log/oxide-crucible-downstairs:default.log
11652023-10-05T04:35:35.699Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20e29a21-5c3b-40a0-9ad8-9e3e070c819f/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11662023-10-05T04:35:36.810Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20e29a21-5c3b-40a0-9ad8-9e3e070c819f/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11672023-10-05T04:35:36.854Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20e29a21-5c3b-40a0-9ad8-9e3e070c819f/root/var/svc/log/oxide-crucible-downstairs:downstairs-8df7b625-b049-4aa9-ae53-b1bb3d32354a.log (3479 bytes)
11682023-10-05T04:35:37.943Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20e29a21-5c3b-40a0-9ad8-9e3e070c819f/root/var/svc/log/oxide-crucible-downstairs:downstairs-8df7b625-b049-4aa9-ae53-b1bb3d32354a.log
11692023-10-05T04:35:37.989Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_0beee836-b0b0-4bb2-8bc1-b993ef00f929/root/var/svc/log/oxide-nexus:default.log (4052392 bytes)
11702023-10-05T04:36:01.371Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_0beee836-b0b0-4bb2-8bc1-b993ef00f929/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4052392 -> 4177637
11712023-10-05T04:36:02.701Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_0beee836-b0b0-4bb2-8bc1-b993ef00f929/root/var/svc/log/oxide-nexus:default.log
11722023-10-05T04:36:02.811Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_0beee836-b0b0-4bb2-8bc1-b993ef00f929/root/var/svc/log/oxide-nexus:default.log.0 (120 bytes)
11732023-10-05T04:36:13.127Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_0beee836-b0b0-4bb2-8bc1-b993ef00f929/root/var/svc/log/oxide-nexus:default.log.0
11742023-10-05T04:36:13.596Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4e18be49-7023-4150-9ebd-80c609918955/root/var/svc/log/oxide-crucible-agent:default.log (22307 bytes)
11752023-10-05T04:36:14.860Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4e18be49-7023-4150-9ebd-80c609918955/root/var/svc/log/oxide-crucible-agent:default.log
11762023-10-05T04:36:14.936Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4e18be49-7023-4150-9ebd-80c609918955/root/var/svc/log/oxide-crucible-agent:default.log.0 (3433 bytes)
11772023-10-05T04:36:16.494Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4e18be49-7023-4150-9ebd-80c609918955/root/var/svc/log/oxide-crucible-agent:default.log.0
11782023-10-05T04:36:16.958Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4e18be49-7023-4150-9ebd-80c609918955/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11792023-10-05T04:36:18.475Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4e18be49-7023-4150-9ebd-80c609918955/root/var/svc/log/oxide-crucible-downstairs:default.log
11802023-10-05T04:36:18.942Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4e18be49-7023-4150-9ebd-80c609918955/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11812023-10-05T04:36:20.509Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4e18be49-7023-4150-9ebd-80c609918955/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11822023-10-05T04:36:20.971Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4e18be49-7023-4150-9ebd-80c609918955/root/var/svc/log/oxide-crucible-downstairs:downstairs-df1651db-0753-4776-b361-2c1c8b6888ee.log (3478 bytes)
11832023-10-05T04:36:22.075Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4e18be49-7023-4150-9ebd-80c609918955/root/var/svc/log/oxide-crucible-downstairs:downstairs-df1651db-0753-4776-b361-2c1c8b6888ee.log
11842023-10-05T04:36:22.106Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4e18be49-7023-4150-9ebd-80c609918955/root/var/svc/log/oxide-crucible-downstairs:downstairs-ea60c4f9-1fd0-4d4b-aa59-5ca22069b12e.log (5942 bytes)
11852023-10-05T04:36:23.694Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4e18be49-7023-4150-9ebd-80c609918955/root/var/svc/log/oxide-crucible-downstairs:downstairs-ea60c4f9-1fd0-4d4b-aa59-5ca22069b12e.log
11862023-10-05T04:36:23.735Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a48a1575-a537-4ff2-bfe2-306a920574e8/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11872023-10-05T04:36:25.706Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a48a1575-a537-4ff2-bfe2-306a920574e8/root/var/svc/log/oxide-cockroachdb:default.log
11882023-10-05T04:36:25.748Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a48a1575-a537-4ff2-bfe2-306a920574e8/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
11892023-10-05T04:36:27.294Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a48a1575-a537-4ff2-bfe2-306a920574e8/root/var/svc/log/oxide-cockroachdb:default.log.0
11902023-10-05T04:36:27.744Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a47e28d3-b0d7-4dab-b5cc-b6c21a442c12/root/var/svc/log/oxide-internal_dns:default.log (12510 bytes)
11912023-10-05T04:36:27.779Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a47e28d3-b0d7-4dab-b5cc-b6c21a442c12/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 12510 -> 23772
11922023-10-05T04:36:28.593Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a47e28d3-b0d7-4dab-b5cc-b6c21a442c12/root/var/svc/log/oxide-internal_dns:default.log
11932023-10-05T04:36:28.640Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a47e28d3-b0d7-4dab-b5cc-b6c21a442c12/root/var/svc/log/oxide-internal_dns:default.log.0 (102074 bytes)
11942023-10-05T04:36:31.008Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a47e28d3-b0d7-4dab-b5cc-b6c21a442c12/root/var/svc/log/oxide-internal_dns:default.log.0
11952023-10-05T04:36:31.064Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bf626afb-0aaa-4293-934d-d1c8de8a9ee0/root/var/svc/log/oxide-nexus:default.log (422377 bytes)
11962023-10-05T04:36:33.053Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bf626afb-0aaa-4293-934d-d1c8de8a9ee0/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 422377 -> 607485
11972023-10-05T04:36:34.639Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bf626afb-0aaa-4293-934d-d1c8de8a9ee0/root/var/svc/log/oxide-nexus:default.log
11982023-10-05T04:36:34.672Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bf626afb-0aaa-4293-934d-d1c8de8a9ee0/root/var/svc/log/oxide-nexus:default.log.0 (120 bytes)
11992023-10-05T04:36:35.778Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bf626afb-0aaa-4293-934d-d1c8de8a9ee0/root/var/svc/log/oxide-nexus:default.log.0
12002023-10-05T04:36:35.815Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e4ce7e60-8b9b-463c-a759-ef752c0a7983/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12012023-10-05T04:36:37.306Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e4ce7e60-8b9b-463c-a759-ef752c0a7983/root/var/svc/log/oxide-cockroachdb:default.log
12022023-10-05T04:36:37.339Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e4ce7e60-8b9b-463c-a759-ef752c0a7983/root/var/svc/log/oxide-cockroachdb:default.log.0 (5370 bytes)
12032023-10-05T04:36:38.471Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e4ce7e60-8b9b-463c-a759-ef752c0a7983/root/var/svc/log/oxide-cockroachdb:default.log.0
12042023-10-05T04:36:38.506Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0827e689-2ebb-4004-87ef-2963ca118f58/root/var/svc/log/oxide-crucible-agent:default.log (10860 bytes)
12052023-10-05T04:36:40.287Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0827e689-2ebb-4004-87ef-2963ca118f58/root/var/svc/log/oxide-crucible-agent:default.log
12062023-10-05T04:36:40.320Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0827e689-2ebb-4004-87ef-2963ca118f58/root/var/svc/log/oxide-crucible-agent:default.log.0 (3433 bytes)
12072023-10-05T04:36:41.516Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0827e689-2ebb-4004-87ef-2963ca118f58/root/var/svc/log/oxide-crucible-agent:default.log.0
12082023-10-05T04:36:41.583Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0827e689-2ebb-4004-87ef-2963ca118f58/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12092023-10-05T04:36:43.408Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0827e689-2ebb-4004-87ef-2963ca118f58/root/var/svc/log/oxide-crucible-downstairs:default.log
12102023-10-05T04:36:43.454Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0827e689-2ebb-4004-87ef-2963ca118f58/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12112023-10-05T04:36:45.632Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0827e689-2ebb-4004-87ef-2963ca118f58/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12122023-10-05T04:36:45.671Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0827e689-2ebb-4004-87ef-2963ca118f58/root/var/svc/log/oxide-crucible-downstairs:downstairs-17d4e5ea-362c-4af4-bb7b-977519a79c45.log (5942 bytes)
12132023-10-05T04:36:47.726Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0827e689-2ebb-4004-87ef-2963ca118f58/root/var/svc/log/oxide-crucible-downstairs:downstairs-17d4e5ea-362c-4af4-bb7b-977519a79c45.log
12142023-10-05T04:36:48.189Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_981480a3-40bf-4e2e-b18e-81de70e8ef53/root/var/svc/log/oxide-oximeter:default.log (174137 bytes)
12152023-10-05T04:36:54.799Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_981480a3-40bf-4e2e-b18e-81de70e8ef53/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 174137 -> 338872
12162023-10-05T04:36:55.718Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_981480a3-40bf-4e2e-b18e-81de70e8ef53/root/var/svc/log/oxide-oximeter:default.log
12172023-10-05T04:36:55.873Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_981480a3-40bf-4e2e-b18e-81de70e8ef53/root/var/svc/log/oxide-oximeter:default.log.0 (5627 bytes)
12182023-10-05T04:36:58.067Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_981480a3-40bf-4e2e-b18e-81de70e8ef53/root/var/svc/log/oxide-oximeter:default.log.0
12192023-10-05T04:36:58.257Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01/root/var/log/chrony/measurements.log (13837 bytes)
12202023-10-05T04:36:58.926Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01/root/var/log/chrony/measurements.log" changed size mid upload: 13837 -> 18769
12212023-10-05T04:37:00.781Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01/root/var/log/chrony/measurements.log
12222023-10-05T04:37:01.080Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01/root/var/log/chrony/statistics.log (9009 bytes)
12232023-10-05T04:37:01.924Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01/root/var/log/chrony/statistics.log" changed size mid upload: 9009 -> 13221
12242023-10-05T04:37:16.734Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01/root/var/log/chrony/statistics.log
12252023-10-05T04:37:17.168Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01/root/var/log/chrony/tracking.log (2118 bytes)
12262023-10-05T04:37:17.373Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01/root/var/log/chrony/tracking.log" changed size mid upload: 2118 -> 2646
12272023-10-05T04:37:22.066Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_66b3f9d3-880a-4053-91e6-72ab57428b01/root/var/log/chrony/tracking.log