01HFCCY7WHCEG1040BEP909AQY: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HFCCYVR6TPME5P2139NBWAEN

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-16T16:33:41.535Zjob dependencies complete; ready to run (waiting for 47 m 46 s)
22023-11-16T16:33:41.553Zjob assigned to worker 01HFCE9GGE1ZNP2BAX5JQYEZAB (queued for 0 s)
32023-11-16T16:33:41.741Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-11-16T16:33:41.787Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-11-16T16:33:41.833Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-11-16T16:33:41.880Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-16T16:33:41.913Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-11-16T16:33:41.950ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-16T16:33:42.003Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-16T16:33:42.047ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-16T16:33:42.100ZConfiguring devices.
122023-11-16T16:33:42.136Z##################################################
132023-11-16T16:33:42.180Z#### ############# ##
142023-11-16T16:33:42.223Z### ### ############ ##
152023-11-16T16:33:42.258Z## ### # ## ### ## Oxide Computer Company ##
162023-11-16T16:33:42.291Z## ## ## ### # ### ##
172023-11-16T16:33:42.339Z## # ### #### #### This Station Under ##
182023-11-16T16:33:42.379Z### ### #### # ### Computer Control ##
192023-11-16T16:33:42.416Z#### #### ### ## ##
202023-11-16T16:33:42.666Z##################################################
212023-11-16T16:33:42.726ZHostname: buskin
222023-11-16T16:33:42.759ZLoading smf(7) serice descriptions:
232023-11-16T16:33:43.870Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-b153fb5eda845188.gz
242023-11-16T16:33:45.481Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-b153fb5eda845188.gz
252023-11-16T16:33:45.519Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-16T16:34:06.845Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-16T16:34:06.895Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-6d5617b4bb093364.gz
282023-11-16T16:34:31.993Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-6d5617b4bb093364.gz
292023-11-16T16:34:32.025Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-16T16:34:33.159Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-16T16:34:33.190Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-16T16:34:36.952Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-16T16:34:36.984Zdownloading input: /input/package/work/version.txt
342023-11-16T16:34:37.021Zdownloaded input: /input/package/work/version.txt
352023-11-16T16:34:37.868Zdownloading input: /input/package/work/package.tar.gz
362023-11-16T16:34:55.602Zdownloaded input: /input/package/work/package.tar.gz
372023-11-16T16:34:55.634Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-16T16:35:25.929Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-16T16:35:25.962Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-16T16:35:37.157Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-16T16:35:37.191Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-16T16:36:25.700Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-16T16:36:25.750Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-16T16:37:13.700Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-16T16:37:13.731Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-16T16:37:44.059Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-16T16:37:44.134Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-16T16:37:49.418Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-16T16:37:49.451Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-16T16:38:02.089Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-16T16:38:02.121Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-16T16:38:08.909Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-16T16:38:08.941Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-16T16:38:15.399Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-16T16:38:15.431Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-16T16:38:15.462Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-16T16:38:15.493Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-16T16:38:19.707Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-16T16:38:19.745Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-16T16:38:24.152Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-16T16:38:24.183Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-16T16:38:45.797Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-16T16:38:45.840Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-16T16:39:12.884Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-16T16:39:12.921Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-16T16:39:13.030Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-16T16:39:13.066Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-16T16:39:22.491Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-16T16:39:22.523Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-16T16:39:41.022Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-16T16:39:41.056Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-16T16:40:55.468Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-16T16:40:56.036Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-16T16:41:44.550Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-16T16:41:44.589Zstarting task 0: "setup"
762023-11-16T16:41:44.638Z++ uname -s
772023-11-16T16:41:44.673Z+ kern=SunOS
782023-11-16T16:41:44.715Z+ case "$kern" in
792023-11-16T16:41:44.752Z+ groupadd -g 12345 build
802023-11-16T16:41:44.786Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-16T16:41:46.647Z+ zfs create -o mountpoint=/work rpool/work
822023-11-16T16:41:46.704Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-16T16:41:46.737Z+ home_fs=zfs
842023-11-16T16:41:46.783Z+ [[ zfs == autofs ]]
852023-11-16T16:41:46.819Z+ mkdir -p /home/build
862023-11-16T16:41:46.868Z+ chown build:build /home/build /work
872023-11-16T16:41:48.677Z+ chmod 0700 /home/build /work
882023-11-16T16:41:48.708Zprocess exited: duration 4051 ms, exit code 0
 
892023-11-16T16:41:48.772Zstarting task 1: "authentication"
902023-11-16T16:41:48.878Zprocess exited: duration 74 ms, exit code 0
 
912023-11-16T16:41:48.941Zstarting task 2: "build"
922023-11-16T16:41:48.979Z+ trap _exit_trap EXIT
932023-11-16T16:41:49.013Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-16T16:41:49.179Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-16T16:41:49.211Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-16T16:41:49.242Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-16T16:41:49.274Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-16T16:41:49.306Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-16T16:41:49.338Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-16T16:41:49.369Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-16T16:41:49.401Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-16T16:41:49.433Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-16T16:41:49.537Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-16T16:41:49.569Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-16T16:41:49.601Z+ [[ -d /opt/oxide ]]
1062023-11-16T16:41:49.633Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-16T16:41:49.664Z+ pfexec mkdir /opt/oxide
1082023-11-16T16:41:49.696Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-16T16:41:49.727Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-16T16:41:49.759Z+ cd /opt/oxide-underneath
1112023-11-16T16:41:49.791Z+ pfexec tar ceEp@/f - .
1122023-11-16T16:41:49.822Z+ cd /opt/oxide
1132023-11-16T16:41:49.854Z+ pfexec tar xveEp@/f -
1142023-11-16T16:41:49.885Zx ., 0 bytes, 0 tape blocks
1152023-11-16T16:41:49.916Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-16T16:41:49.948Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-16T16:41:49.980Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-16T16:41:50.012Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-16T16:41:50.043Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-16T16:41:50.075Z+ pfexec mkdir /opt/oxide/work
1212023-11-16T16:41:50.106Z+ pfexec chown build:build /opt/oxide/work
1222023-11-16T16:41:50.138Z+ cd /opt/oxide/work
1232023-11-16T16:41:50.169Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-16T16:41:50.882ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-16T16:41:50.914Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-11-16T16:41:50.946Zx out/omicron-sled-agent.tar, 111557120 bytes, 217885 tape blocks
1272023-11-16T16:41:50.978Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-16T16:41:51.010Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-11-16T16:41:51.042Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-11-16T16:41:51.073Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-11-16T16:41:51.105Zx target/release/omicron-package, 42423568 bytes, 82859 tape blocks
1322023-11-16T16:41:51.136Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-11-16T16:41:51.167Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-16T16:41:51.199Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-16T16:41:51.231Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-16T16:41:51.263Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-16T16:41:51.294Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-16T16:41:51.325Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-16T16:41:51.357Z
1402023-11-16T16:41:51.389Zreal 1.543111226
1412023-11-16T16:41:51.422Zuser 1.387622887
1422023-11-16T16:41:51.454Zsys 0.270703181
1432023-11-16T16:41:51.485Ztrap 0.000062950
1442023-11-16T16:41:51.517Ztflt 0.000000000
1452023-11-16T16:41:51.548Zdflt 0.000000000
1462023-11-16T16:41:51.580Zkflt 0.000000000
1472023-11-16T16:41:51.655Zlock 0.000000000
1482023-11-16T16:41:51.687Zslp 1.389714542
1492023-11-16T16:41:51.719Zlat 0.036272642
1502023-11-16T16:41:51.751Zstop 0.000112320
1512023-11-16T16:41:51.783Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-16T16:41:51.926Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-16T16:41:51.958Z+ mkdir tests
1542023-11-16T16:41:51.990Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-16T16:41:52.021Z+ ptime -m gunzip
1562023-11-16T16:41:52.053Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-b153fb5eda845188
1572023-11-16T16:41:52.085Z
1582023-11-16T16:41:52.117Zreal 0.138570073
1592023-11-16T16:41:52.148Zuser 0.127309275
1602023-11-16T16:41:52.180Zsys 0.011013496
1612023-11-16T16:41:52.211Ztrap 0.000011019
1622023-11-16T16:41:52.242Ztflt 0.000000000
1632023-11-16T16:41:52.274Zdflt 0.000000000
1642023-11-16T16:41:52.311Zkflt 0.000000000
1652023-11-16T16:41:52.342Zlock 0.000000000
1662023-11-16T16:41:52.374Zslp 0.000019289
1672023-11-16T16:41:52.405Zlat 0.000022149
1682023-11-16T16:41:52.437Zstop 0.000125291
1692023-11-16T16:41:52.468Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-b153fb5eda845188
1702023-11-16T16:41:52.500Z+ chmod a+x tests/bootstrap-b153fb5eda845188
1712023-11-16T16:41:52.531Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-16T16:41:52.563Z+ ptime -m gunzip
1732023-11-16T16:41:52.594Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-16T16:41:54.086Z
1752023-11-16T16:41:54.118Zreal 1.989228869
1762023-11-16T16:41:54.150Zuser 1.859492286
1772023-11-16T16:41:54.181Zsys 0.129361460
1782023-11-16T16:41:54.213Ztrap 0.000150451
1792023-11-16T16:41:54.244Ztflt 0.000000000
1802023-11-16T16:41:54.275Zdflt 0.000000000
1812023-11-16T16:41:54.307Zkflt 0.000000000
1822023-11-16T16:41:54.338Zlock 0.000000000
1832023-11-16T16:41:54.369Zslp 0.000013830
1842023-11-16T16:41:54.403Zlat 0.000023590
1852023-11-16T16:41:54.441Zstop 0.000130360
1862023-11-16T16:41:54.474Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-16T16:41:54.510Z+ chmod a+x tests/bootstrap
1882023-11-16T16:41:54.547Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-16T16:41:54.585Z+ ptime -m gunzip
1902023-11-16T16:41:54.620Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-6d5617b4bb093364
1912023-11-16T16:41:56.427Z
1922023-11-16T16:41:56.458Zreal 2.329089084
1932023-11-16T16:41:56.490Zuser 2.174266368
1942023-11-16T16:41:56.521Zsys 0.154427025
1952023-11-16T16:41:56.553Ztrap 0.000161759
1962023-11-16T16:41:56.584Ztflt 0.000000000
1972023-11-16T16:41:56.615Zdflt 0.000000000
1982023-11-16T16:41:56.647Zkflt 0.000000000
1992023-11-16T16:41:56.678Zlock 0.000000000
2002023-11-16T16:41:56.710Zslp 0.000027189
2012023-11-16T16:41:56.741Zlat 0.000023469
2022023-11-16T16:41:56.775Zstop 0.000122320
2032023-11-16T16:41:56.809Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-6d5617b4bb093364
2042023-11-16T16:41:56.841Z+ chmod a+x tests/end_to_end_tests-6d5617b4bb093364
2052023-11-16T16:41:56.873Z+ bmat address ls
2062023-11-16T16:41:56.907ZNAME CIDR FIRST LAST COUNT
2072023-11-16T16:41:56.939Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-11-16T16:41:57.008Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-16T16:41:57.040Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-11-16T16:41:57.071Z+ EXTRA_IP_START=10.151.1.164
2112023-11-16T16:41:57.103Z+ EXTRA_IP_END=10.151.1.227
2122023-11-16T16:41:57.136Z+ EXTRA_IP_COUNT=64
2132023-11-16T16:41:57.168Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-16T16:41:57.201Z+ EXTRA_IP_BASE=10.151.1
2152023-11-16T16:41:57.236Z+ EXTRA_IP_FOCTET=164
2162023-11-16T16:41:57.272Z+ EXTRA_IP_LOCTET=227
2172023-11-16T16:41:57.309Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-11-16T16:41:57.344Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-11-16T16:41:57.380Z+ DNS_IP1=10.151.1.164
2202023-11-16T16:41:57.418Z+ DNS_IP2=10.151.1.165
2212023-11-16T16:41:57.452Z+ UPLINK_IP=10.151.1.174
2222023-11-16T16:41:57.484Z+ IPPOOL_START=10.151.1.175
2232023-11-16T16:41:57.516Z+ IPPOOL_END=10.151.1.227
2242023-11-16T16:41:57.548Z++ ipadm show-addr -po type,addr
2252023-11-16T16:41:57.580Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-16T16:41:57.612Z+ GATEWAY_IP=10.151.1.100
2272023-11-16T16:41:57.644Z+ [[ -n 10.151.1.100 ]]
2282023-11-16T16:41:57.676Z+ ping -s 10.151.1.100 56 1
2292023-11-16T16:41:57.708ZPING 10.151.1.100: 56 data bytes
2302023-11-16T16:41:57.740Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
2312023-11-16T16:41:57.775Z
2322023-11-16T16:41:57.807Z----10.151.1.100 PING Statistics----
2332023-11-16T16:41:57.850Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-16T16:41:57.885Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-11-16T16:41:57.916Z++ arp -an
2362023-11-16T16:41:57.948Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-11-16T16:41:57.980Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-11-16T16:41:58.082Z+ routeadm -e ipv4-forwarding -u
2392023-11-16T16:41:58.114Z+ PXA_START=10.151.1.164
2402023-11-16T16:41:58.145Z+ PXA_END=10.151.1.227
2412023-11-16T16:41:58.177Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-16T16:41:58.210Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-16T16:41:58.241Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-16T16:41:58.272Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-16T16:41:58.305Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-16T16:41:58.338Z++ cd ./tools
2472023-11-16T16:41:58.370Z++ pwd
2482023-11-16T16:41:58.402Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-16T16:41:58.440Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-16T16:41:58.472Z+ SOFTNPU_MODE=zone
2512023-11-16T16:41:58.504Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-16T16:41:58.539Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-16T16:41:58.573Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-16T16:41:58.605Z++ dladm show-phys -p -o LINK
2552023-11-16T16:41:58.637Z++ head -1
2562023-11-16T16:41:58.668Z+ PHYSICAL_LINK=igb0
2572023-11-16T16:41:58.700Z+ echo 'Using igb0 as physical link'
2582023-11-16T16:41:58.731ZUsing igb0 as physical link
2592023-11-16T16:41:58.762Z+ ensure_run_as_root
2602023-11-16T16:41:58.827Z++ id -u
2612023-11-16T16:41:58.858Z+ [[ 0 -ne 0 ]]
2622023-11-16T16:41:58.890Z+ ensure_zpools
2632023-11-16T16:41:58.921Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-16T16:41:58.961Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-11-16T16:41:58.999Z+ readarray -t ZPOOLS
2662023-11-16T16:41:59.042Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-11-16T16:41:59.077Z++ sed 's/[ ",]//g'
2682023-11-16T16:41:59.170Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-11-16T16:41:59.204Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-11-16T16:41:59.237ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-11-16T16:41:59.269Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-11-16T16:41:59.301ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2732023-11-16T16:41:59.334Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2742023-11-16T16:41:59.785ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2752023-11-16T16:41:59.825Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2762023-11-16T16:41:59.858ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2772023-11-16T16:41:59.892Z0+0 records in
2782023-11-16T16:41:59.926ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2792023-11-16T16:41:59.962Z0+0 records out
2802023-11-16T16:42:00.004ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2812023-11-16T16:42:00.040Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2822023-11-16T16:42:00.074ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2832023-11-16T16:42:00.108Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2842023-11-16T16:42:00.140ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2852023-11-16T16:42:00.287Z+ set +x
2862023-11-16T16:42:00.321ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2872023-11-16T16:42:00.357Z++ zpool list -o name
2882023-11-16T16:42:00.398ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2892023-11-16T16:42:00.431Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2902023-11-16T16:42:00.464ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2912023-11-16T16:42:00.496Z+ [[ -z '' ]]
2922023-11-16T16:42:00.528ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2932023-11-16T16:42:00.560Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2942023-11-16T16:42:00.592ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2952023-11-16T16:42:00.624Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2962023-11-16T16:42:00.655ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2972023-11-16T16:42:00.689Z+ set +x
2982023-11-16T16:42:00.724ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2992023-11-16T16:42:00.763Z+ for ZPOOL in "${ZPOOLS[@]}"
3002023-11-16T16:42:00.796ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3012023-11-16T16:42:00.833Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3022023-11-16T16:42:00.876ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3032023-11-16T16:42:00.910Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3042023-11-16T16:42:00.943ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3052023-11-16T16:42:00.977Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3062023-11-16T16:42:01.009ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3072023-11-16T16:42:01.041Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3082023-11-16T16:42:01.073ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3092023-11-16T16:42:01.105Z0+0 records in
3102023-11-16T16:42:01.138ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3112023-11-16T16:42:01.196Z0+0 records out
3122023-11-16T16:42:01.232ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3132023-11-16T16:42:01.277Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3142023-11-16T16:42:01.330ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3152023-11-16T16:42:01.382Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-11-16T16:42:01.417ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3172023-11-16T16:42:01.460Z+ set +x
3182023-11-16T16:42:01.495ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3192023-11-16T16:42:01.529Z++ zpool list -o name
3202023-11-16T16:42:01.561ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3212023-11-16T16:42:01.594Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3222023-11-16T16:42:01.629ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3232023-11-16T16:42:01.669Z+ [[ -z '' ]]
3242023-11-16T16:42:01.708ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3252023-11-16T16:42:01.740Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3262023-11-16T16:42:01.774ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3272023-11-16T16:42:01.808Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3282023-11-16T16:42:01.842ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3292023-11-16T16:42:01.875Z+ set +x
3302023-11-16T16:42:01.907ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3312023-11-16T16:42:01.939Z+ for ZPOOL in "${ZPOOLS[@]}"
3322023-11-16T16:42:01.972ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3332023-11-16T16:42:02.004Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3342023-11-16T16:42:02.037ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3352023-11-16T16:42:02.072Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3362023-11-16T16:42:02.116ZSimnet net0/sc0_0 exists
3372023-11-16T16:42:02.150Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3382023-11-16T16:42:02.183ZSimnet net1/sc1_0 exists
3392023-11-16T16:42:02.218Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3402023-11-16T16:42:02.251ZVnic sc0_1 exists
3412023-11-16T16:42:02.284Z0+0 records in
3422023-11-16T16:42:02.316Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
3432023-11-16T16:42:02.349Z0+0 records out
3442023-11-16T16:42:02.381Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3452023-11-16T16:42:02.458Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3462023-11-16T16:42:02.491Z+ set +x
3472023-11-16T16:42:02.523Z++ zpool list -o name
3482023-11-16T16:42:02.559Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3492023-11-16T16:42:02.596Z+ [[ -z '' ]]
3502023-11-16T16:42:02.630Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3512023-11-16T16:42:02.662Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3522023-11-16T16:42:02.694Z+ set +x
3532023-11-16T16:42:02.726Z+ for ZPOOL in "${ZPOOLS[@]}"
3542023-11-16T16:42:02.759Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552023-11-16T16:42:02.790Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562023-11-16T16:42:02.834Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572023-11-16T16:42:02.866Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582023-11-16T16:42:02.901Z0+0 records in
3592023-11-16T16:42:02.934Z0+0 records out
3602023-11-16T16:42:02.967Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3612023-11-16T16:42:03.002Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622023-11-16T16:42:03.036Z+ set +x
3632023-11-16T16:42:03.071Z++ zpool list -o name
3642023-11-16T16:42:03.103Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3652023-11-16T16:42:03.136Z+ [[ -z '' ]]
3662023-11-16T16:42:03.168Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3672023-11-16T16:42:03.201Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3682023-11-16T16:42:03.235Z+ set +x
3692023-11-16T16:42:03.268Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-11-16T16:42:03.302Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3712023-11-16T16:42:03.334Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3722023-11-16T16:42:03.369Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3732023-11-16T16:42:03.424Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3742023-11-16T16:42:03.482Z0+0 records in
3752023-11-16T16:42:03.535Z0+0 records out
3762023-11-16T16:42:03.567Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3772023-11-16T16:42:03.599Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3782023-11-16T16:42:03.631Z+ set +x
3792023-11-16T16:42:03.663Z++ zpool list -o name
3802023-11-16T16:42:03.696Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3812023-11-16T16:42:03.727Z+ [[ -z '' ]]
3822023-11-16T16:42:03.759Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3832023-11-16T16:42:03.831Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3842023-11-16T16:42:03.865Z+ set +x
3852023-11-16T16:42:03.898Z+ for ZPOOL in "${ZPOOLS[@]}"
3862023-11-16T16:42:03.930Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-11-16T16:42:03.963Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-11-16T16:42:03.995Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-11-16T16:42:04.028Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-11-16T16:42:04.062Z0+0 records in
3912023-11-16T16:42:04.095Z0+0 records out
3922023-11-16T16:42:04.134Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3932023-11-16T16:42:04.167Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942023-11-16T16:42:04.199Z+ set +x
3952023-11-16T16:42:04.231Z++ zpool list -o name
3962023-11-16T16:42:04.265Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972023-11-16T16:42:04.297Z+ [[ -z '' ]]
3982023-11-16T16:42:04.329Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992023-11-16T16:42:04.373Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002023-11-16T16:42:04.407Z+ set +x
4012023-11-16T16:42:04.439Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-16T16:42:04.471Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4032023-11-16T16:42:04.507Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-11-16T16:42:04.576Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4052023-11-16T16:42:04.635Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4062023-11-16T16:42:04.668Z0+0 records in
4072023-11-16T16:42:04.699Z0+0 records out
4082023-11-16T16:42:04.732Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4092023-11-16T16:42:04.764Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4102023-11-16T16:42:04.796Z+ set +x
4112023-11-16T16:42:04.829Z++ zpool list -o name
4122023-11-16T16:42:04.860Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4132023-11-16T16:42:04.892Z+ [[ -z '' ]]
4142023-11-16T16:42:04.925Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4152023-11-16T16:42:04.967Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4162023-11-16T16:42:05.003Z+ set +x
4172023-11-16T16:42:05.037Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-11-16T16:42:05.069Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4192023-11-16T16:42:05.102Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4202023-11-16T16:42:05.146Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4212023-11-16T16:42:05.178Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4222023-11-16T16:42:05.210Z0+0 records in
4232023-11-16T16:42:05.242Z0+0 records out
4242023-11-16T16:42:05.274Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4252023-11-16T16:42:05.306Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4262023-11-16T16:42:05.341Z+ set +x
4272023-11-16T16:42:05.379Z++ zpool list -o name
4282023-11-16T16:42:05.417Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4292023-11-16T16:42:05.455Z+ [[ -z '' ]]
4302023-11-16T16:42:05.493Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4312023-11-16T16:42:05.529Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4322023-11-16T16:42:05.561Z+ set +x
4332023-11-16T16:42:05.593Z+ for ZPOOL in "${ZPOOLS[@]}"
4342023-11-16T16:42:05.626Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4352023-11-16T16:42:05.691Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-11-16T16:42:05.724Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4372023-11-16T16:42:05.756Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4382023-11-16T16:42:05.788Z0+0 records in
4392023-11-16T16:42:05.820Z0+0 records out
4402023-11-16T16:42:05.852Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4412023-11-16T16:42:05.887Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4422023-11-16T16:42:05.919Z+ set +x
4432023-11-16T16:42:05.951Z++ zpool list -o name
4442023-11-16T16:42:05.983Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4452023-11-16T16:42:06.017Z+ [[ -z '' ]]
4462023-11-16T16:42:06.050Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4472023-11-16T16:42:06.082Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4482023-11-16T16:42:06.115Z+ set +x
4492023-11-16T16:42:06.149Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4502023-11-16T16:42:06.180Z+ readarray -t ZPOOLS
4512023-11-16T16:42:06.212Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4522023-11-16T16:42:06.245Z++ sed 's/[ ",]//g'
4532023-11-16T16:42:06.277Z+ for ZPOOL in "${ZPOOLS[@]}"
4542023-11-16T16:42:06.312Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4552023-11-16T16:42:06.354Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4562023-11-16T16:42:06.402Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4572023-11-16T16:42:06.439Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4582023-11-16T16:42:06.475Zdownloading softnpu ...done
4592023-11-16T16:42:06.509Z0+0 records in
4602023-11-16T16:42:06.543Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
4612023-11-16T16:42:06.579Z0+0 records out
4622023-11-16T16:42:06.616Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4632023-11-16T16:42:06.652Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4642023-11-16T16:42:06.691Z+ set +x
4652023-11-16T16:42:06.821Z++ zpool list -o name
4662023-11-16T16:42:06.865Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4672023-11-16T16:42:06.900Z+ [[ -z '' ]]
4682023-11-16T16:42:06.933Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4692023-11-16T16:42:06.966Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4702023-11-16T16:42:06.998Z+ set +x
4712023-11-16T16:42:07.030Z+ for ZPOOL in "${ZPOOLS[@]}"
4722023-11-16T16:42:07.061Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4732023-11-16T16:42:07.093Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4742023-11-16T16:42:07.125Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4752023-11-16T16:42:07.156Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4762023-11-16T16:42:07.188Z0+0 records in
4772023-11-16T16:42:07.219Z0+0 records out
4782023-11-16T16:42:07.251Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4792023-11-16T16:42:07.291Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4802023-11-16T16:42:07.333Z+ set +x
4812023-11-16T16:42:07.367Z++ zpool list -o name
4822023-11-16T16:42:07.398Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4832023-11-16T16:42:07.430Z+ [[ -z '' ]]
4842023-11-16T16:42:07.461Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4852023-11-16T16:42:07.493Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4862023-11-16T16:42:07.525Z+ set +x
4872023-11-16T16:42:07.557Z+ [[ zone == \z\o\n\e ]]
4882023-11-16T16:42:07.589Z+ ensure_simulated_links igb0
4892023-11-16T16:42:07.621Z+ local PHYSICAL_LINK=igb0
4902023-11-16T16:42:07.654Z+ INDICES=("0" "1")
4912023-11-16T16:42:07.690Z+ for I in "${INDICES[@]}"
4922023-11-16T16:42:07.728Z++ get_simnet_name_if_exists net0
4932023-11-16T16:42:07.763Z++ dladm show-simnet -p -o LINK net0
4942023-11-16T16:42:07.805Z++ echo ''
4952023-11-16T16:42:07.892Z+ [[ -z '' ]]
4962023-11-16T16:42:07.924Z+ dladm create-simnet -t net0
4972023-11-16T16:42:07.956Z+ dladm create-simnet -t sc0_0
4982023-11-16T16:42:07.988Z+ dladm modify-simnet -t -p net0 sc0_0
4992023-11-16T16:42:08.020Z+ dladm set-linkprop -p mtu=9000 sc0_0
5002023-11-16T16:42:08.051Z+ success 'Simnet net0/sc0_0 exists'
5012023-11-16T16:42:08.083Z+ set +x
5022023-11-16T16:42:08.121Z+ for I in "${INDICES[@]}"
5032023-11-16T16:42:08.568Z++ get_simnet_name_if_exists net1
5042023-11-16T16:42:08.605Z++ dladm show-simnet -p -o LINK net1
5052023-11-16T16:42:08.636Z++ echo ''
5062023-11-16T16:42:08.667Z+ [[ -z '' ]]
5072023-11-16T16:42:08.700Z+ dladm create-simnet -t net1
5082023-11-16T16:42:08.731Z+ dladm create-simnet -t sc1_0
5092023-11-16T16:42:08.763Z+ dladm modify-simnet -t -p net1 sc1_0
5102023-11-16T16:42:08.794Z+ dladm set-linkprop -p mtu=9000 sc1_0
5112023-11-16T16:42:08.859Z+ success 'Simnet net1/sc1_0 exists'
5122023-11-16T16:42:08.890Z+ set +x
5132023-11-16T16:42:08.954Z++ get_vnic_name_if_exists sc0_1
5142023-11-16T16:42:08.989Z++ dladm show-vnic -p -o LINK sc0_1
5152023-11-16T16:42:09.028Z++ echo ''
5162023-11-16T16:42:09.098Z+ [[ -z '' ]]
5172023-11-16T16:42:09.148Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5182023-11-16T16:42:09.193Z+ [[ -v PROMISC_FILT_OFF ]]
5192023-11-16T16:42:09.226Z+ success 'Vnic sc0_1 exists'
5202023-11-16T16:42:09.258Z+ set +x
5212023-11-16T16:42:09.289Z+ warn_if_no_proxy_arp
5222023-11-16T16:42:09.321Z+ [[ -v PXA_START ]]
5232023-11-16T16:42:09.352Z+ [[ -v PXA_END ]]
5242023-11-16T16:42:09.383Z+ ensure_softnpu_zone
5252023-11-16T16:42:09.415Z+ zoneadm list
5262023-11-16T16:42:09.446Z+ grep -q sidecar_softnpu
5272023-11-16T16:42:09.478Z+ [[ -f out/npuzone/npuzone ]]
5282023-11-16T16:42:09.509Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5292023-11-16T16:42:10.960Zdownloading asic_program.so ...done
5302023-11-16T16:42:10.993Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-16T16:42:18.182Zdownloading scadm ...done
5322023-11-16T16:42:18.217Zcreating port tfportrear0_0 for sc0_0
5332023-11-16T16:42:18.250Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-16T16:42:18.282Zsoftnpu zone setup
5352023-11-16T16:42:18.314Zconfigure zone
5362023-11-16T16:42:18.345Zinstall zone
5372023-11-16T16:42:19.243Zboot zone
5382023-11-16T16:42:22.359Z[sidecar_softnpu] svcs milestone/network
5392023-11-16T16:42:22.391ZSTATE STIME FMRI
5402023-11-16T16:42:22.422Zonline 16:42:20 svc:/milestone/network:default
5412023-11-16T16:42:22.454Z[sidecar_softnpu] /softnpu/npu start
5422023-11-16T16:42:22.485Z
5432023-11-16T16:42:22.517Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-16T16:42:22.549Z+ GATEWAY_IP=10.151.1.100
5452023-11-16T16:42:22.580Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-11-16T16:42:22.612ZUsing 10.151.1.100 as gateway ip
5472023-11-16T16:42:22.651Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-16T16:42:22.683Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-11-16T16:42:22.715Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-16T16:42:22.747Z+++ echo 18:c0:4d:0d:9f:b2
5512023-11-16T16:42:22.786Z+ COUNT=1
5522023-11-16T16:42:22.817Z+ [[ 1 -ne 1 ]]
5532023-11-16T16:42:22.848Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-11-16T16:42:22.880ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-11-16T16:42:22.911Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-11-16T16:42:22.943Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-11-16T16:42:22.976Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-16T16:42:23.007Z+ [[ -v PXA_START ]]
5592023-11-16T16:42:23.039Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-11-16T16:42:23.078Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-11-16T16:42:23.112Z+ z_scadm dump-state
5622023-11-16T16:42:23.149Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-16T16:42:23.237Zlocal v6:
5642023-11-16T16:42:23.269Zlocal v4:
5652023-11-16T16:42:23.300Zrouter v6:
5662023-11-16T16:42:23.331Zrouter v4_idx:
5672023-11-16T16:42:23.363Zrouter v4_routes:
5682023-11-16T16:42:23.394Zresolver v4:
5692023-11-16T16:42:23.426Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-11-16T16:42:23.457Zresolver v6:
5712023-11-16T16:42:23.491Znat v4:
5722023-11-16T16:42:23.525Znat_v6:
5732023-11-16T16:42:23.556Zport_mac:
5742023-11-16T16:42:23.588Zproxy arp:
5752023-11-16T16:42:23.620Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-11-16T16:42:23.652Z+ success 'softnpu zone exists'
5772023-11-16T16:42:23.684Z+ set +x
5782023-11-16T16:42:23.716Zsoftnpu zone exists
5792023-11-16T16:42:23.748Z
5802023-11-16T16:42:23.780Zreal 25.731064309
5812023-11-16T16:42:23.846Zuser 0.723247192
5822023-11-16T16:42:23.889Zsys 1.556273492
5832023-11-16T16:42:23.930Ztrap 0.000110015
5842023-11-16T16:42:23.968Ztflt 0.000113856
5852023-11-16T16:42:24.000Zdflt 0.001275069
5862023-11-16T16:42:24.037Zkflt 0.000002629
5872023-11-16T16:42:24.070Zlock 19:53.522799352
5882023-11-16T16:42:24.102Zslp 1:17.036594276
5892023-11-16T16:42:24.134Zlat 0.074859185
5902023-11-16T16:42:24.165Zstop 0.356566450
5912023-11-16T16:42:24.197Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-16T16:42:24.228Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-16T16:42:24.296Z+ SILO_NAME=recovery
5942023-11-16T16:42:24.328Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-16T16:42:24.360Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-16T16:42:24.393Z+ sed -i~ '
5972023-11-16T16:42:24.426Z /^external_dns_ips/c\
5982023-11-16T16:42:24.458Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-11-16T16:42:24.490Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-16T16:42:24.523Z /^first/c\
6012023-11-16T16:42:24.556Zfirst = "10.151.1.164"
6022023-11-16T16:42:24.588Z /^last/c\
6032023-11-16T16:42:24.619Zlast = "10.151.1.173"
6042023-11-16T16:42:24.651Z }
6052023-11-16T16:42:24.683Z /^\[rack_network_config/,/^$/ {
6062023-11-16T16:42:24.716Z /^infra_ip_first/c\
6072023-11-16T16:42:24.748Zinfra_ip_first = "10.151.1.174"
6082023-11-16T16:42:24.780Z /^infra_ip_last/c\
6092023-11-16T16:42:24.813Zinfra_ip_last = "10.151.1.174"
6102023-11-16T16:42:24.845Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6112023-11-16T16:42:24.878Z }
6122023-11-16T16:42:24.910Z+++ pkg/config-rss.toml Thu Nov 16 16:42:21 2023
6132023-11-16T16:42:24.951Z /^\[\[rack_network_config.ports/,/^$/ {
6142023-11-16T16:42:24.983Z@@ -41,7 +41,7 @@
6152023-11-16T16:42:25.016Z /^routes/c\
6162023-11-16T16:42:25.048Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6172023-11-16T16:42:25.081Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6182023-11-16T16:42:25.113Z # must be contained in one of the "internal services" IP Pool ranges listed
6192023-11-16T16:42:25.144Z /^addresses/c\
6202023-11-16T16:42:25.179Z # below.
6212023-11-16T16:42:25.217Zaddresses = \["10.151.1.174/32"\]
6222023-11-16T16:42:25.269Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6232023-11-16T16:42:25.356Z }
6242023-11-16T16:42:25.388Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6252023-11-16T16:42:25.420Z' pkg/config-rss.toml
6262023-11-16T16:42:25.451Z
6272023-11-16T16:42:25.483Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6282023-11-16T16:42:25.515Z # Initial TLS certificates for the external API
6292023-11-16T16:42:25.546Z+ true
6302023-11-16T16:42:25.577Z #
6312023-11-16T16:42:25.610Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6322023-11-16T16:42:25.649Z@@ -77,8 +77,8 @@
6332023-11-16T16:42:25.681Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6342023-11-16T16:42:25.712Z #
6352023-11-16T16:42:25.743Z+ TLS_NAME=recovery.sys.oxide.test
6362023-11-16T16:42:25.775Z # For more on this and what to put here, see docs/how-to-run.adoc.
6372023-11-16T16:42:25.806Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6382023-11-16T16:42:25.838Z [[internal_services_ip_pool_ranges]]
6392023-11-16T16:42:25.869Z...+.+.....+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+..+...................+.....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6402023-11-16T16:42:25.901Z-first = "192.168.1.20"
6412023-11-16T16:42:25.933Z..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.........+..........+..+..........+...............+........+..........+...........+...+.............+.........+...............+........+.+..+...+......+...............+...............+....+.........+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6422023-11-16T16:42:25.965Z-last = "192.168.1.29"
6432023-11-16T16:42:25.997Z-----
6442023-11-16T16:42:26.029Z+first = "10.151.1.164"
6452023-11-16T16:42:26.064Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6462023-11-16T16:42:26.099Z+last = "10.151.1.173"
6472023-11-16T16:42:26.131Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6482023-11-16T16:42:26.165Z
6492023-11-16T16:42:26.197Z+ rmdir pkg
6502023-11-16T16:42:26.230Z # TODO - this configuration is subject to change going forward. Ultimately these
6512023-11-16T16:42:26.262Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6522023-11-16T16:42:26.295Z # parameters should be provided to the control plane via wicket, but we need to
6532023-11-16T16:42:26.327Z+ pfexec mkdir -p /usr/oxide
6542023-11-16T16:42:26.383Z@@ -92,8 +92,8 @@
6552023-11-16T16:42:26.434Z+ pfexec rm -f /usr/oxide/tcpproxy
6562023-11-16T16:42:26.468Z # A range of IP addresses used by Boundary Services on the external network. In
6572023-11-16T16:42:26.504Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6582023-11-16T16:42:26.539Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6592023-11-16T16:42:26.573Z+ pfexec chmod +x /usr/oxide/tcpproxy
6602023-11-16T16:42:26.604Z # With softnpu, only one address is used.
6612023-11-16T16:42:26.636Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6622023-11-16T16:42:26.668Z-infra_ip_first = "192.168.1.30"
6632023-11-16T16:42:26.700Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6642023-11-16T16:42:26.732Z-infra_ip_last = "192.168.1.30"
6652023-11-16T16:42:26.764Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6662023-11-16T16:42:26.798Z+infra_ip_first = "10.151.1.174"
6672023-11-16T16:42:26.831Z+ OMICRON_NO_UNINSTALL=1
6682023-11-16T16:42:26.865Z+infra_ip_last = "10.151.1.174"
6692023-11-16T16:42:26.901Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6702023-11-16T16:42:26.935Z
6712023-11-16T16:42:26.969ZNov 16 16:42:22.853 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6722023-11-16T16:42:27.004Z # Configurations for BGP routers to run on the scrimlets.
6732023-11-16T16:42:27.036ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6742023-11-16T16:42:27.068Z bgp = []
6752023-11-16T16:42:27.100ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6762023-11-16T16:42:27.132Z@@ -101,9 +101,9 @@
6772023-11-16T16:42:27.164ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6782023-11-16T16:42:27.196Z # You can configure multiple uplinks by repeating the following stanza
6792023-11-16T16:42:27.228ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6802023-11-16T16:42:27.260Z [[rack_network_config.ports]]
6812023-11-16T16:42:27.291ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6822023-11-16T16:42:27.327Z # Routes associated with this port.
6832023-11-16T16:42:27.364ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6842023-11-16T16:42:27.398Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6852023-11-16T16:42:27.430ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6862023-11-16T16:42:27.470Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6872023-11-16T16:42:27.522ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6882023-11-16T16:42:27.553Z # Addresses associated with this port.
6892023-11-16T16:42:27.585ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6902023-11-16T16:42:27.617Z-addresses = ["192.168.1.30/32"]
6912023-11-16T16:42:27.648ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6922023-11-16T16:42:27.680Z+addresses = ["10.151.1.174/32"]
6932023-11-16T16:42:27.720ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6942023-11-16T16:42:27.756Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6952023-11-16T16:42:27.791ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6962023-11-16T16:42:27.823Z port = "qsfp0"
6972023-11-16T16:42:27.854ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6982023-11-16T16:42:27.886Z # The speed of this port.
6992023-11-16T16:42:27.917ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7002023-11-16T16:42:27.950Za pkg/config-rss.toml 7K
7012023-11-16T16:42:27.982ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7022023-11-16T16:42:28.014Za pkg/initial-tls-cert.pem 2K
7032023-11-16T16:42:28.045ZNov 16 16:42:22.855 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7042023-11-16T16:42:28.077Za pkg/initial-tls-key.pem 4K
7052023-11-16T16:42:28.109ZNov 16 16:42:23.101 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-11-16T16:42:28.140ZNov 16 16:42:23.138 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-16T16:42:28.175ZNov 16 16:42:23.198 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-16T16:42:28.209Z
7092023-11-16T16:42:28.242Zreal 0.484362890
7102023-11-16T16:42:28.273Zuser 0.099391769
7112023-11-16T16:42:28.305Zsys 2.013639505
7122023-11-16T16:42:28.336Ztrap 0.000000269
7132023-11-16T16:42:28.368Ztflt 0.000037218
7142023-11-16T16:42:28.399Zdflt 0.000287998
7152023-11-16T16:42:28.430Zkflt 0.000000000
7162023-11-16T16:42:28.462Zlock 43.457392317
7172023-11-16T16:42:28.494Zslp 0.692236862
7182023-11-16T16:42:28.578Zlat 0.003108625
7192023-11-16T16:42:28.612Zstop 0.205672373
7202023-11-16T16:42:28.644Z+ retry=0
7212023-11-16T16:42:28.676Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-16T16:42:28.707Z+ [[ 0 -gt 30 ]]
7232023-11-16T16:42:28.740Z+ sleep 1
7242023-11-16T16:42:28.772Z+ retry=1
7252023-11-16T16:42:28.803Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-16T16:42:28.836Z+ [[ 1 -gt 30 ]]
7272023-11-16T16:42:28.869Z+ sleep 1
7282023-11-16T16:42:28.900Z+ retry=2
7292023-11-16T16:42:28.932Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-16T16:42:28.963Z+ [[ 2 -gt 30 ]]
7312023-11-16T16:42:28.995Z+ sleep 1
7322023-11-16T16:42:29.026Z+ retry=3
7332023-11-16T16:42:29.092Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-11-16T16:42:29.127Z+ [[ 3 -gt 30 ]]
7352023-11-16T16:42:29.166Z+ sleep 1
7362023-11-16T16:42:29.199Z+ retry=4
7372023-11-16T16:42:29.231Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-11-16T16:42:47.854Z+ [[ 4 -gt 30 ]]
7392023-11-16T16:42:47.885Z+ sleep 1
7402023-11-16T16:42:48.873Z+ retry=5
7412023-11-16T16:42:48.905Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7422023-11-16T16:42:48.937Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7432023-11-16T16:42:48.968Zlocal v6:
7442023-11-16T16:42:49.000Zfe80::30ca:afff:fe51:d16d
7452023-11-16T16:42:49.031Zfd00:99::1
7462023-11-16T16:42:49.063Zlocal v4:
7472023-11-16T16:42:49.094Z10.151.1.174
7482023-11-16T16:42:49.127Zrouter v6:
7492023-11-16T16:42:49.158Zrouter v4_idx:
7502023-11-16T16:42:49.189Z0.0.0.0/0 -> 0
7512023-11-16T16:42:49.223Zrouter v4_routes:
7522023-11-16T16:42:49.254Z0 -> 10.151.1.100 (2)
7532023-11-16T16:42:49.693Zresolver v4:
7542023-11-16T16:42:49.725Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7552023-11-16T16:42:49.757Zresolver v6:
7562023-11-16T16:42:49.788Zfe80::30ca:afff:fe51:d16d -> 32:ca:af:51:d1:6d
7572023-11-16T16:42:49.820Zfe80::6cf6:48ff:feaa:6df7 -> 6e:f6:48:aa:6d:f7
7582023-11-16T16:42:49.861Znat v4:
7592023-11-16T16:42:49.892Znat_v6:
7602023-11-16T16:42:49.924Zport_mac:
7612023-11-16T16:42:49.955Z1000: a8:40:25:06:41:46
7622023-11-16T16:42:49.987Z1: a8:40:25:06:41:47
7632023-11-16T16:42:50.018Zproxy arp:
7642023-11-16T16:42:50.050Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7652023-11-16T16:42:50.081Z+ retry=0
7662023-11-16T16:42:50.113Z+++ zoneadm list -n
7672023-11-16T16:42:50.555Z+++ grep oxz_ntp
7682023-11-16T16:42:50.592Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-11-16T16:42:50.627Zsvcs: zone '-Hostate': entity not found
7702023-11-16T16:42:50.663Z++ true
7712023-11-16T16:42:50.701Z+ [[ '' != online ]]
7722023-11-16T16:42:50.737Z+ [[ 0 -gt 60 ]]
7732023-11-16T16:42:50.776Z+ sleep 1
7742023-11-16T16:42:50.811Z+ retry=1
7752023-11-16T16:42:50.850Z+++ zoneadm list -n
7762023-11-16T16:42:50.901Z+++ grep oxz_ntp
7772023-11-16T16:42:50.937Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-11-16T16:42:50.974Zsvcs: zone '-Hostate': entity not found
7792023-11-16T16:42:51.454Z++ true
7802023-11-16T16:42:51.492Z+ [[ '' != online ]]
7812023-11-16T16:42:51.530Z+ [[ 1 -gt 60 ]]
7822023-11-16T16:42:51.567Z+ sleep 1
7832023-11-16T16:42:51.603Z+ retry=2
7842023-11-16T16:42:51.638Z+++ zoneadm list -n
7852023-11-16T16:42:51.674Z+++ grep oxz_ntp
7862023-11-16T16:42:51.710Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-11-16T16:42:51.747Zsvcs: zone '-Hostate': entity not found
7882023-11-16T16:42:51.793Z++ true
7892023-11-16T16:42:51.828Z+ [[ '' != online ]]
7902023-11-16T16:42:51.863Z+ [[ 2 -gt 60 ]]
7912023-11-16T16:42:51.894Z+ sleep 1
7922023-11-16T16:42:51.961Z+ retry=3
7932023-11-16T16:42:51.993Z+++ zoneadm list -n
7942023-11-16T16:42:52.025Z+++ grep oxz_ntp
7952023-11-16T16:42:52.056Z++ pfexec svcs -z oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96 -Hostate oxide/ntp
7962023-11-16T16:42:52.089Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7972023-11-16T16:42:52.121Z++ true
7982023-11-16T16:42:52.162Z+ [[ '' != online ]]
7992023-11-16T16:42:52.194Z+ [[ 3 -gt 60 ]]
8002023-11-16T16:42:52.226Z+ sleep 1
8012023-11-16T16:42:52.993Z+ retry=4
8022023-11-16T16:42:53.025Z+++ zoneadm list -n
8032023-11-16T16:42:53.057Z+++ grep oxz_ntp
8042023-11-16T16:42:53.088Z++ pfexec svcs -z oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96 -Hostate oxide/ntp
8052023-11-16T16:42:53.119Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8062023-11-16T16:42:53.151Z++ true
8072023-11-16T16:42:53.182Z+ [[ '' != online ]]
8082023-11-16T16:42:53.214Z+ [[ 4 -gt 60 ]]
8092023-11-16T16:42:53.245Z+ sleep 1
8102023-11-16T16:42:54.024Z+ retry=5
8112023-11-16T16:42:54.056Z+++ zoneadm list -n
8122023-11-16T16:42:54.088Z+++ grep oxz_ntp
8132023-11-16T16:42:54.120Z++ pfexec svcs -z oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96 -Hostate oxide/ntp
8142023-11-16T16:42:54.152Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8152023-11-16T16:42:54.183Z++ true
8162023-11-16T16:42:54.216Z+ [[ '' != online ]]
8172023-11-16T16:42:54.247Z+ [[ 5 -gt 60 ]]
8182023-11-16T16:42:54.279Z+ sleep 1
8192023-11-16T16:42:55.055Z+ retry=6
8202023-11-16T16:42:55.086Z+++ zoneadm list -n
8212023-11-16T16:42:55.118Z+++ grep oxz_ntp
8222023-11-16T16:42:55.149Z++ pfexec svcs -z oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96 -Hostate oxide/ntp
8232023-11-16T16:42:55.181Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8242023-11-16T16:42:55.212Z++ true
8252023-11-16T16:42:55.243Z+ [[ '' != online ]]
8262023-11-16T16:42:55.275Z+ [[ 6 -gt 60 ]]
8272023-11-16T16:42:55.306Z+ sleep 1
8282023-11-16T16:42:56.089Z+ retry=7
8292023-11-16T16:42:56.121Z+++ zoneadm list -n
8302023-11-16T16:42:56.153Z+++ grep oxz_ntp
8312023-11-16T16:42:56.184Z++ pfexec svcs -z oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96 -Hostate oxide/ntp
8322023-11-16T16:42:56.215Z+ [[ disabled != online ]]
8332023-11-16T16:42:56.247Z+ [[ 7 -gt 60 ]]
8342023-11-16T16:42:56.278Z+ sleep 1
8352023-11-16T16:42:57.111Z+ retry=8
8362023-11-16T16:42:57.143Z+++ zoneadm list -n
8372023-11-16T16:42:57.175Z+++ grep oxz_ntp
8382023-11-16T16:42:57.206Z++ pfexec svcs -z oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96 -Hostate oxide/ntp
8392023-11-16T16:42:57.238Z+ [[ disabled != online ]]
8402023-11-16T16:42:57.270Z+ [[ 8 -gt 60 ]]
8412023-11-16T16:42:57.301Z+ sleep 1
8422023-11-16T16:42:58.133Z+ retry=9
8432023-11-16T16:42:58.164Z+++ zoneadm list -n
8442023-11-16T16:42:58.196Z+++ grep oxz_ntp
8452023-11-16T16:42:58.227Z++ pfexec svcs -z oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96 -Hostate oxide/ntp
8462023-11-16T16:42:58.258Z+ [[ disabled != online ]]
8472023-11-16T16:42:58.290Z+ [[ 9 -gt 60 ]]
8482023-11-16T16:42:58.321Z+ sleep 1
8492023-11-16T16:42:59.155Z+ retry=10
8502023-11-16T16:42:59.187Z+++ zoneadm list -n
8512023-11-16T16:42:59.218Z+++ grep oxz_ntp
8522023-11-16T16:42:59.250Z++ pfexec svcs -z oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96 -Hostate oxide/ntp
8532023-11-16T16:42:59.281Z+ [[ disabled != online ]]
8542023-11-16T16:42:59.313Z+ [[ 10 -gt 60 ]]
8552023-11-16T16:42:59.346Z+ sleep 1
8562023-11-16T16:43:00.177Z+ retry=11
8572023-11-16T16:43:00.209Z+++ zoneadm list -n
8582023-11-16T16:43:00.242Z+++ grep oxz_ntp
8592023-11-16T16:43:00.273Z++ pfexec svcs -z oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96 -Hostate oxide/ntp
8602023-11-16T16:43:00.305Z+ [[ disabled != online ]]
8612023-11-16T16:43:00.336Z+ [[ 11 -gt 60 ]]
8622023-11-16T16:43:00.368Z+ sleep 1
8632023-11-16T16:43:01.199Z+ retry=12
8642023-11-16T16:43:01.231Z+++ zoneadm list -n
8652023-11-16T16:43:01.262Z+++ grep oxz_ntp
8662023-11-16T16:43:01.294Z++ pfexec svcs -z oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96 -Hostate oxide/ntp
8672023-11-16T16:43:01.326Z+ [[ disabled != online ]]
8682023-11-16T16:43:01.359Z+ [[ 12 -gt 60 ]]
8692023-11-16T16:43:01.391Z+ sleep 1
8702023-11-16T16:43:02.222Z+ retry=13
8712023-11-16T16:43:02.254Z+++ zoneadm list -n
8722023-11-16T16:43:02.286Z+++ grep oxz_ntp
8732023-11-16T16:43:02.318Z++ pfexec svcs -z oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96 -Hostate oxide/ntp
8742023-11-16T16:43:02.349Z+ [[ disabled != online ]]
8752023-11-16T16:43:02.383Z+ [[ 13 -gt 60 ]]
8762023-11-16T16:43:02.415Z+ sleep 1
8772023-11-16T16:43:03.245Z+ retry=14
8782023-11-16T16:43:03.277Z+++ zoneadm list -n
8792023-11-16T16:43:03.308Z+++ grep oxz_ntp
8802023-11-16T16:43:03.341Z++ pfexec svcs -z oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96 -Hostate oxide/ntp
8812023-11-16T16:43:03.373Z+ [[ disabled != online ]]
8822023-11-16T16:43:03.407Z+ [[ 14 -gt 60 ]]
8832023-11-16T16:43:03.439Z+ sleep 1
8842023-11-16T16:43:04.264Z+ retry=15
8852023-11-16T16:43:04.296Z+++ zoneadm list -n
8862023-11-16T16:43:04.329Z+++ grep oxz_ntp
 
8872023-11-16T16:43:04.334Zbuskin console login: Nov 16 16:43:03 bukin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8882023-11-16T16:43:04.362Z++ pfexec svcs -z oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96 -Hostate oxide/ntp
8892023-11-16T16:43:04.395Z+ [[ offline* != online ]]
8902023-11-16T16:43:04.427Z+ [[ 15 -gt 60 ]]
8912023-11-16T16:43:04.458Z+ sleep 1
8922023-11-16T16:43:05.290Z+ retry=16
8932023-11-16T16:43:05.361Z+++ zoneadm list -n
8942023-11-16T16:43:05.394Z+++ grep oxz_ntp
8952023-11-16T16:43:05.425Z++ pfexec svcs -z oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96 -Hostate oxide/ntp
8962023-11-16T16:43:05.456Z+ [[ online != online ]]
8972023-11-16T16:43:05.487Z+ echo 'Waited for chrony: 16s'
8982023-11-16T16:43:05.518ZWaited for chrony: 16s
8992023-11-16T16:43:05.550Z+ export RUST_BACKTRACE=1
9002023-11-16T16:43:05.581Z+ RUST_BACKTRACE=1
9012023-11-16T16:43:05.612Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9022023-11-16T16:43:05.644Z+ ./tests/bootstrap
9032023-11-16T16:43:05.675ZUsing external DNS server at 10.151.1.164:53
9042023-11-16T16:43:05.707Z2023-11-16 16:43:04.785636118 UTC: attempting to log into API
9052023-11-16T16:43:20.420Z2023-11-16 16:43:20.099766617 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9062023-11-16T16:43:21.377Z2023-11-16 16:43:21.104476134 UTC: attempting to log into API
9072023-11-16T16:43:36.418Z2023-11-16 16:43:36.182068156 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9082023-11-16T16:43:37.421Z2023-11-16 16:43:37.185780485 UTC: attempting to log into API
9092023-11-16T16:43:52.461Z2023-11-16 16:43:52.272870785 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9102023-11-16T16:43:53.466Z2023-11-16 16:43:53.276645966 UTC: attempting to log into API
9112023-11-16T16:44:08.502Z2023-11-16 16:44:08.350861695 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9122023-11-16T16:44:09.498Z2023-11-16 16:44:09.354549468 UTC: attempting to log into API
 
9132023-11-16T16:44:20.288ZNov 16 16:44:19 bukin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9142023-11-16T16:44:24.532Z2023-11-16 16:44:24.429626110 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9152023-11-16T16:44:25.533Z2023-11-16 16:44:25.433359789 UTC: attempting to log into API
 
9162023-11-16T16:44:28.476ZNov 16 16:44:27 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9172023-11-16T16:44:35.147ZNov 16 16:44:34 bukin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9182023-11-16T16:44:40.566Z2023-11-16 16:44:40.507278251 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9192023-11-16T16:44:41.218ZNov 16 16:44:40 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9202023-11-16T16:44:41.567Z2023-11-16 16:44:41.511067158 UTC: attempting to log into API
 
9212023-11-16T16:44:48.338ZNov 16 16:44:47 bukin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9222023-11-16T16:44:56.599Z2023-11-16 16:44:56.584654054 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9232023-11-16T16:44:57.601Z2023-11-16 16:44:57.586924769 UTC: attempting to log into API
9242023-11-16T16:45:12.634Z2023-11-16 16:45:12.620222060 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9252023-11-16T16:45:13.634Z2023-11-16 16:45:13.621126935 UTC: attempting to log into API
9262023-11-16T16:45:19.438Z2023-11-16 16:45:19.421104368 UTC: login succeeded
9272023-11-16T16:45:19.470Zwaiting for nexus to come up...
9282023-11-16T16:45:19.534Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9292023-11-16T16:45:19.590Zensuring datasets are ready...
9302023-11-16T16:45:21.594Zlet's roll.
9312023-11-16T16:45:21.625Z+ rm ./tests/bootstrap
9322023-11-16T16:45:21.657Z+ for test_bin in tests/*
9332023-11-16T16:45:21.688Z+ ./tests/bootstrap-b153fb5eda845188
9342023-11-16T16:45:21.720Z
9352023-11-16T16:45:21.751Zrunning 0 tests
9362023-11-16T16:45:21.782Z
9372023-11-16T16:45:21.814Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9382023-11-16T16:45:21.846Z
9392023-11-16T16:45:21.877Z+ for test_bin in tests/*
9402023-11-16T16:45:21.909Z+ ./tests/end_to_end_tests-6d5617b4bb093364
9412023-11-16T16:45:21.952Z
9422023-11-16T16:45:21.987Zrunning 2 tests
9432023-11-16T16:45:22.019Ztest no_spoof::no_spoof_here ... ok
9442023-11-16T16:46:21.679Ztest instance_launch::instance_launch has been running for over 60 seconds
9452023-11-16T16:48:18.960Ztest instance_launch::instance_launch ... ok
9462023-11-16T16:48:18.992Z
9472023-11-16T16:48:19.024Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 177.28s
9482023-11-16T16:48:19.056Z
9492023-11-16T16:48:19.088Z+ _exit_trap
9502023-11-16T16:48:19.121Z+ local status=0
9512023-11-16T16:48:19.153Z+ [[ 0 -eq 0 ]]
9522023-11-16T16:48:19.184Z+ exit 0
9532023-11-16T16:48:19.216Zprocess exited: duration 389987 ms, exit code 0
 
9542023-11-16T16:48:19.311Zfound 84 output files
9552023-11-16T16:48:19.379Zuploading: /var/svc/log/oxide-sled-agent:default.log (759899 bytes)
9562023-11-16T16:48:20.692Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9572023-11-16T16:48:20.758Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_574a73f4-7966-4c0e-b6db-c522eb58ccb0/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9582023-11-16T16:48:21.819Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_574a73f4-7966-4c0e-b6db-c522eb58ccb0/root/var/svc/log/oxide-crucible-agent:default.log
9592023-11-16T16:48:21.851Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_574a73f4-7966-4c0e-b6db-c522eb58ccb0/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
9602023-11-16T16:48:22.941Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_574a73f4-7966-4c0e-b6db-c522eb58ccb0/root/var/svc/log/oxide-crucible-agent:default.log.0
9612023-11-16T16:48:22.973Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_574a73f4-7966-4c0e-b6db-c522eb58ccb0/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9622023-11-16T16:48:24.025Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_574a73f4-7966-4c0e-b6db-c522eb58ccb0/root/var/svc/log/oxide-crucible-downstairs:default.log
9632023-11-16T16:48:24.056Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_574a73f4-7966-4c0e-b6db-c522eb58ccb0/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9642023-11-16T16:48:25.149Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_574a73f4-7966-4c0e-b6db-c522eb58ccb0/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9652023-11-16T16:48:25.179Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_277f5174-ada0-4b1c-8ca5-4da17328cc71/root/var/svc/log/oxide-internal_dns:default.log (12520 bytes)
9662023-11-16T16:48:26.302Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_277f5174-ada0-4b1c-8ca5-4da17328cc71/root/var/svc/log/oxide-internal_dns:default.log
9672023-11-16T16:48:26.333Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_277f5174-ada0-4b1c-8ca5-4da17328cc71/root/var/svc/log/oxide-internal_dns:default.log.0 (103293 bytes)
9682023-11-16T16:48:27.575Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_277f5174-ada0-4b1c-8ca5-4da17328cc71/root/var/svc/log/oxide-internal_dns:default.log.0
9692023-11-16T16:48:27.606Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_348a1ef3-a73a-4c76-be1f-4381eae21a4a/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9702023-11-16T16:48:28.674Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_348a1ef3-a73a-4c76-be1f-4381eae21a4a/root/var/svc/log/oxide-cockroachdb:default.log
9712023-11-16T16:48:28.705Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_348a1ef3-a73a-4c76-be1f-4381eae21a4a/root/var/svc/log/oxide-cockroachdb:default.log.0 (5342 bytes)
9722023-11-16T16:48:29.796Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_348a1ef3-a73a-4c76-be1f-4381eae21a4a/root/var/svc/log/oxide-cockroachdb:default.log.0
9732023-11-16T16:48:29.828Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60c545a5-1a2b-4036-81e5-f65193b7b22e/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9742023-11-16T16:48:30.881Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60c545a5-1a2b-4036-81e5-f65193b7b22e/root/var/svc/log/oxide-crucible-agent:default.log
9752023-11-16T16:48:30.912Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60c545a5-1a2b-4036-81e5-f65193b7b22e/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
9762023-11-16T16:48:32.005Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60c545a5-1a2b-4036-81e5-f65193b7b22e/root/var/svc/log/oxide-crucible-agent:default.log.0
9772023-11-16T16:48:32.036Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60c545a5-1a2b-4036-81e5-f65193b7b22e/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9782023-11-16T16:48:33.117Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60c545a5-1a2b-4036-81e5-f65193b7b22e/root/var/svc/log/oxide-crucible-downstairs:default.log
9792023-11-16T16:48:33.150Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60c545a5-1a2b-4036-81e5-f65193b7b22e/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9802023-11-16T16:48:34.245Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60c545a5-1a2b-4036-81e5-f65193b7b22e/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9812023-11-16T16:48:34.277Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_da241801-2d37-4b10-8bde-8d35e6635c2c/root/var/svc/log/oxide-crucible-agent:default.log (10249 bytes)
9822023-11-16T16:48:35.396Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_da241801-2d37-4b10-8bde-8d35e6635c2c/root/var/svc/log/oxide-crucible-agent:default.log
9832023-11-16T16:48:35.427Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_da241801-2d37-4b10-8bde-8d35e6635c2c/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
9842023-11-16T16:48:36.518Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_da241801-2d37-4b10-8bde-8d35e6635c2c/root/var/svc/log/oxide-crucible-agent:default.log.0
9852023-11-16T16:48:36.549Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_da241801-2d37-4b10-8bde-8d35e6635c2c/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9862023-11-16T16:48:37.602Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_da241801-2d37-4b10-8bde-8d35e6635c2c/root/var/svc/log/oxide-crucible-downstairs:default.log
9872023-11-16T16:48:37.633Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_da241801-2d37-4b10-8bde-8d35e6635c2c/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9882023-11-16T16:48:38.726Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_da241801-2d37-4b10-8bde-8d35e6635c2c/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9892023-11-16T16:48:38.757Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_da241801-2d37-4b10-8bde-8d35e6635c2c/root/var/svc/log/oxide-crucible-downstairs:downstairs-903bfede-f14c-4310-bf61-2d592c1cdc4c.log (3036 bytes)
9902023-11-16T16:48:39.855Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_da241801-2d37-4b10-8bde-8d35e6635c2c/root/var/svc/log/oxide-crucible-downstairs:downstairs-903bfede-f14c-4310-bf61-2d592c1cdc4c.log
9912023-11-16T16:48:39.887Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_9dae3121-eed4-40e0-8010-e740eab05c67/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9922023-11-16T16:48:40.940Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_9dae3121-eed4-40e0-8010-e740eab05c67/root/var/svc/log/oxide-cockroachdb:default.log
9932023-11-16T16:48:40.971Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_9dae3121-eed4-40e0-8010-e740eab05c67/root/var/svc/log/oxide-cockroachdb:default.log.0 (5367 bytes)
9942023-11-16T16:48:42.064Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_9dae3121-eed4-40e0-8010-e740eab05c67/root/var/svc/log/oxide-cockroachdb:default.log.0
9952023-11-16T16:48:42.095Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_145ef934-a7ec-459b-8f02-fb6cacd9a98e/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
9962023-11-16T16:48:43.164Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_145ef934-a7ec-459b-8f02-fb6cacd9a98e/root/var/svc/log/oxide-crucible-pantry:default.log
9972023-11-16T16:48:43.195Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_145ef934-a7ec-459b-8f02-fb6cacd9a98e/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
9982023-11-16T16:48:44.285Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_145ef934-a7ec-459b-8f02-fb6cacd9a98e/root/var/svc/log/oxide-crucible-pantry:default.log.0
9992023-11-16T16:48:44.316Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96/root/var/svc/log/oxide-ntp:default.log (67 bytes)
10002023-11-16T16:48:45.409Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96/root/var/svc/log/oxide-ntp:default.log
10012023-11-16T16:48:45.452Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96/root/var/svc/log/oxide-ntp:default.log.0 (1324 bytes)
10022023-11-16T16:48:46.551Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96/root/var/svc/log/oxide-ntp:default.log.0
10032023-11-16T16:48:46.584Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_59fa68c0-5937-49e6-95a1-f9a0bd3508a2/root/var/svc/log/oxide-clickhouse:default.log (816052 bytes)
10042023-11-16T16:48:47.454Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_59fa68c0-5937-49e6-95a1-f9a0bd3508a2/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 816052 -> 930462
10052023-11-16T16:48:48.571Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_59fa68c0-5937-49e6-95a1-f9a0bd3508a2/root/var/svc/log/oxide-clickhouse:default.log
10062023-11-16T16:48:48.602Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_59fa68c0-5937-49e6-95a1-f9a0bd3508a2/root/var/svc/log/oxide-clickhouse:default.log.0 (129111 bytes)
10072023-11-16T16:48:49.868Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_59fa68c0-5937-49e6-95a1-f9a0bd3508a2/root/var/svc/log/oxide-clickhouse:default.log.0
10082023-11-16T16:48:49.899Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d019212f-7c46-4a72-9845-660416b10b18/root/var/svc/log/oxide-crucible-agent:default.log (10714 bytes)
10092023-11-16T16:48:51.019Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d019212f-7c46-4a72-9845-660416b10b18/root/var/svc/log/oxide-crucible-agent:default.log
10102023-11-16T16:48:51.050Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d019212f-7c46-4a72-9845-660416b10b18/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10112023-11-16T16:48:52.140Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d019212f-7c46-4a72-9845-660416b10b18/root/var/svc/log/oxide-crucible-agent:default.log.0
10122023-11-16T16:48:52.172Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d019212f-7c46-4a72-9845-660416b10b18/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10132023-11-16T16:48:53.224Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d019212f-7c46-4a72-9845-660416b10b18/root/var/svc/log/oxide-crucible-downstairs:default.log
10142023-11-16T16:48:53.257Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d019212f-7c46-4a72-9845-660416b10b18/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10152023-11-16T16:48:54.347Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d019212f-7c46-4a72-9845-660416b10b18/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10162023-11-16T16:48:54.384Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d019212f-7c46-4a72-9845-660416b10b18/root/var/svc/log/oxide-crucible-downstairs:downstairs-470a8fe7-da2c-4314-92f5-bdc01f243651.log (3034 bytes)
10172023-11-16T16:48:55.471Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d019212f-7c46-4a72-9845-660416b10b18/root/var/svc/log/oxide-crucible-downstairs:downstairs-470a8fe7-da2c-4314-92f5-bdc01f243651.log
10182023-11-16T16:48:55.502Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_97969733-1d32-4c8c-a081-ec61b97232c9/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10192023-11-16T16:48:56.570Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_97969733-1d32-4c8c-a081-ec61b97232c9/root/var/svc/log/oxide-crucible-pantry:default.log
10202023-11-16T16:48:56.601Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_97969733-1d32-4c8c-a081-ec61b97232c9/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2332 bytes)
10212023-11-16T16:48:57.692Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_97969733-1d32-4c8c-a081-ec61b97232c9/root/var/svc/log/oxide-crucible-pantry:default.log.0
10222023-11-16T16:48:57.723Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_e9a9fce0-a75d-4005-b97a-f68f1ca81cc7/root/var/svc/log/oxide-oximeter:default.log (229923 bytes)
10232023-11-16T16:48:57.967Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_e9a9fce0-a75d-4005-b97a-f68f1ca81cc7/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 229923 -> 267735
10242023-11-16T16:48:59.057Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_e9a9fce0-a75d-4005-b97a-f68f1ca81cc7/root/var/svc/log/oxide-oximeter:default.log
10252023-11-16T16:48:59.093Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_e9a9fce0-a75d-4005-b97a-f68f1ca81cc7/root/var/svc/log/oxide-oximeter:default.log.0 (15369 bytes)
10262023-11-16T16:49:00.214Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_e9a9fce0-a75d-4005-b97a-f68f1ca81cc7/root/var/svc/log/oxide-oximeter:default.log.0
10272023-11-16T16:49:00.247Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_b3575ba2-6ef2-4d19-bb6a-662e8f854d74/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10282023-11-16T16:49:01.299Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_b3575ba2-6ef2-4d19-bb6a-662e8f854d74/root/var/svc/log/oxide-cockroachdb:default.log
10292023-11-16T16:49:01.331Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_b3575ba2-6ef2-4d19-bb6a-662e8f854d74/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10302023-11-16T16:49:02.424Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_b3575ba2-6ef2-4d19-bb6a-662e8f854d74/root/var/svc/log/oxide-cockroachdb:default.log.0
10312023-11-16T16:49:02.457Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_dfe696c3-b56e-4518-8b2f-b8fe03ad2e8b/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10322023-11-16T16:49:03.523Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_dfe696c3-b56e-4518-8b2f-b8fe03ad2e8b/root/var/svc/log/oxide-cockroachdb:default.log
10332023-11-16T16:49:03.555Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_dfe696c3-b56e-4518-8b2f-b8fe03ad2e8b/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10342023-11-16T16:49:04.647Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_dfe696c3-b56e-4518-8b2f-b8fe03ad2e8b/root/var/svc/log/oxide-cockroachdb:default.log.0
10352023-11-16T16:49:04.679Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4a9a788d-540e-49e6-826d-20e5a914eb8b/root/var/svc/log/oxide-crucible-agent:default.log (10251 bytes)
10362023-11-16T16:49:05.799Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4a9a788d-540e-49e6-826d-20e5a914eb8b/root/var/svc/log/oxide-crucible-agent:default.log
10372023-11-16T16:49:05.831Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4a9a788d-540e-49e6-826d-20e5a914eb8b/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10382023-11-16T16:49:06.924Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4a9a788d-540e-49e6-826d-20e5a914eb8b/root/var/svc/log/oxide-crucible-agent:default.log.0
10392023-11-16T16:49:06.956Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4a9a788d-540e-49e6-826d-20e5a914eb8b/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10402023-11-16T16:49:08.007Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4a9a788d-540e-49e6-826d-20e5a914eb8b/root/var/svc/log/oxide-crucible-downstairs:default.log
10412023-11-16T16:49:08.041Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4a9a788d-540e-49e6-826d-20e5a914eb8b/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10422023-11-16T16:49:09.130Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4a9a788d-540e-49e6-826d-20e5a914eb8b/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10432023-11-16T16:49:09.161Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4a9a788d-540e-49e6-826d-20e5a914eb8b/root/var/svc/log/oxide-crucible-downstairs:downstairs-bef34f1c-2a56-4337-a439-095228e53252.log (3039 bytes)
10442023-11-16T16:49:10.254Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4a9a788d-540e-49e6-826d-20e5a914eb8b/root/var/svc/log/oxide-crucible-downstairs:downstairs-bef34f1c-2a56-4337-a439-095228e53252.log
10452023-11-16T16:49:10.284Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_d7013ac7-beb9-42e7-b2e6-81493864c4d8/root/var/svc/log/oxide-external_dns:default.log (13060 bytes)
10462023-11-16T16:49:10.374Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_d7013ac7-beb9-42e7-b2e6-81493864c4d8/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 13060 -> 16840
10472023-11-16T16:49:11.466Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_d7013ac7-beb9-42e7-b2e6-81493864c4d8/root/var/svc/log/oxide-external_dns:default.log
10482023-11-16T16:49:11.497Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_d7013ac7-beb9-42e7-b2e6-81493864c4d8/root/var/svc/log/oxide-external_dns:default.log.0 (2554 bytes)
10492023-11-16T16:49:12.589Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_d7013ac7-beb9-42e7-b2e6-81493864c4d8/root/var/svc/log/oxide-external_dns:default.log.0
10502023-11-16T16:49:12.620Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_dd9fb0be-3ac4-4f56-956e-b3ce6232a72e/root/var/svc/log/oxide-crucible-agent:default.log (11144 bytes)
10512023-11-16T16:49:13.742Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_dd9fb0be-3ac4-4f56-956e-b3ce6232a72e/root/var/svc/log/oxide-crucible-agent:default.log
10522023-11-16T16:49:13.773Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_dd9fb0be-3ac4-4f56-956e-b3ce6232a72e/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10532023-11-16T16:49:14.872Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_dd9fb0be-3ac4-4f56-956e-b3ce6232a72e/root/var/svc/log/oxide-crucible-agent:default.log.0
10542023-11-16T16:49:14.903Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_dd9fb0be-3ac4-4f56-956e-b3ce6232a72e/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10552023-11-16T16:49:15.956Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_dd9fb0be-3ac4-4f56-956e-b3ce6232a72e/root/var/svc/log/oxide-crucible-downstairs:default.log
10562023-11-16T16:49:15.986Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_dd9fb0be-3ac4-4f56-956e-b3ce6232a72e/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10572023-11-16T16:49:17.079Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_dd9fb0be-3ac4-4f56-956e-b3ce6232a72e/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10582023-11-16T16:49:17.110Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_dd9fb0be-3ac4-4f56-956e-b3ce6232a72e/root/var/svc/log/oxide-crucible-downstairs:downstairs-09d331a2-cfe1-4eff-b9f5-486b358fcf16.log (5511 bytes)
10592023-11-16T16:49:18.202Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_dd9fb0be-3ac4-4f56-956e-b3ce6232a72e/root/var/svc/log/oxide-crucible-downstairs:downstairs-09d331a2-cfe1-4eff-b9f5-486b358fcf16.log
10602023-11-16T16:49:18.233Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_68c7cff2-ba28-4923-aeda-607bcc6d2ea3/root/var/svc/log/oxide-internal_dns:default.log (12522 bytes)
10612023-11-16T16:49:18.329Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_68c7cff2-ba28-4923-aeda-607bcc6d2ea3/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 12522 -> 16277
10622023-11-16T16:49:19.421Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_68c7cff2-ba28-4923-aeda-607bcc6d2ea3/root/var/svc/log/oxide-internal_dns:default.log
10632023-11-16T16:49:19.452Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_68c7cff2-ba28-4923-aeda-607bcc6d2ea3/root/var/svc/log/oxide-internal_dns:default.log.0 (102163 bytes)
10642023-11-16T16:49:20.692Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_68c7cff2-ba28-4923-aeda-607bcc6d2ea3/root/var/svc/log/oxide-internal_dns:default.log.0
10652023-11-16T16:49:20.723Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_e51df67e-ebcd-4c7a-9507-c7ad5242a77a/root/var/svc/log/oxide-crucible-agent:default.log (10677 bytes)
10662023-11-16T16:49:21.846Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_e51df67e-ebcd-4c7a-9507-c7ad5242a77a/root/var/svc/log/oxide-crucible-agent:default.log
10672023-11-16T16:49:21.877Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_e51df67e-ebcd-4c7a-9507-c7ad5242a77a/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
10682023-11-16T16:49:22.968Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_e51df67e-ebcd-4c7a-9507-c7ad5242a77a/root/var/svc/log/oxide-crucible-agent:default.log.0
10692023-11-16T16:49:22.999Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_e51df67e-ebcd-4c7a-9507-c7ad5242a77a/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10702023-11-16T16:49:24.067Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_e51df67e-ebcd-4c7a-9507-c7ad5242a77a/root/var/svc/log/oxide-crucible-downstairs:default.log
10712023-11-16T16:49:24.098Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_e51df67e-ebcd-4c7a-9507-c7ad5242a77a/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10722023-11-16T16:49:25.188Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_e51df67e-ebcd-4c7a-9507-c7ad5242a77a/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10732023-11-16T16:49:25.219Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_e51df67e-ebcd-4c7a-9507-c7ad5242a77a/root/var/svc/log/oxide-crucible-downstairs:downstairs-4255e22b-9191-4294-8524-3cc918a9083d.log (5516 bytes)
10742023-11-16T16:49:26.309Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_e51df67e-ebcd-4c7a-9507-c7ad5242a77a/root/var/svc/log/oxide-crucible-downstairs:downstairs-4255e22b-9191-4294-8524-3cc918a9083d.log
10752023-11-16T16:49:26.340Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_c9a8e1d6-f6cf-486b-9309-09c5c61a4ebf/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10762023-11-16T16:49:27.392Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_c9a8e1d6-f6cf-486b-9309-09c5c61a4ebf/root/var/svc/log/oxide-crucible-pantry:default.log
10772023-11-16T16:49:27.423Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_c9a8e1d6-f6cf-486b-9309-09c5c61a4ebf/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10782023-11-16T16:49:28.515Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_c9a8e1d6-f6cf-486b-9309-09c5c61a4ebf/root/var/svc/log/oxide-crucible-pantry:default.log.0
10792023-11-16T16:49:28.565Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_223892e8-0325-4f2d-b630-3f5a1059a0b7/root/var/svc/log/oxide-external_dns:default.log (13065 bytes)
10802023-11-16T16:49:28.634Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_223892e8-0325-4f2d-b630-3f5a1059a0b7/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 13065 -> 16844
10812023-11-16T16:49:29.726Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_223892e8-0325-4f2d-b630-3f5a1059a0b7/root/var/svc/log/oxide-external_dns:default.log
10822023-11-16T16:49:29.757Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_223892e8-0325-4f2d-b630-3f5a1059a0b7/root/var/svc/log/oxide-external_dns:default.log.0 (2554 bytes)
10832023-11-16T16:49:30.848Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_223892e8-0325-4f2d-b630-3f5a1059a0b7/root/var/svc/log/oxide-external_dns:default.log.0
10842023-11-16T16:49:30.879Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_eecbaacb-b909-4bee-a94f-0301de9a34b3/root/var/svc/log/oxide-internal_dns:default.log (12519 bytes)
10852023-11-16T16:49:30.969Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_eecbaacb-b909-4bee-a94f-0301de9a34b3/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 12519 -> 16275
10862023-11-16T16:49:32.061Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_eecbaacb-b909-4bee-a94f-0301de9a34b3/root/var/svc/log/oxide-internal_dns:default.log
10872023-11-16T16:49:32.092Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_eecbaacb-b909-4bee-a94f-0301de9a34b3/root/var/svc/log/oxide-internal_dns:default.log.0 (108658 bytes)
10882023-11-16T16:49:33.334Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_eecbaacb-b909-4bee-a94f-0301de9a34b3/root/var/svc/log/oxide-internal_dns:default.log.0
10892023-11-16T16:49:33.376Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_3d75bc33-9cff-4fb6-8463-1ede0a8653c8/root/var/svc/log/oxide-nexus:default.log (3942718 bytes)
10902023-11-16T16:49:35.474Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_3d75bc33-9cff-4fb6-8463-1ede0a8653c8/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 3942718 -> 4005141
10912023-11-16T16:49:36.567Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_3d75bc33-9cff-4fb6-8463-1ede0a8653c8/root/var/svc/log/oxide-nexus:default.log
10922023-11-16T16:49:36.601Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_3d75bc33-9cff-4fb6-8463-1ede0a8653c8/root/var/svc/log/oxide-nexus:default.log.0 (827627 bytes)
10932023-11-16T16:49:38.653Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_3d75bc33-9cff-4fb6-8463-1ede0a8653c8/root/var/svc/log/oxide-nexus:default.log.0
10942023-11-16T16:49:38.684Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_54ec747e-1485-4343-b58a-a1f26532aae3/root/var/svc/log/oxide-nexus:default.log (345480 bytes)
10952023-11-16T16:49:38.988Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_54ec747e-1485-4343-b58a-a1f26532aae3/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 345480 -> 466725
10962023-11-16T16:49:40.081Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_54ec747e-1485-4343-b58a-a1f26532aae3/root/var/svc/log/oxide-nexus:default.log
10972023-11-16T16:49:40.113Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_54ec747e-1485-4343-b58a-a1f26532aae3/root/var/svc/log/oxide-nexus:default.log.0 (283117 bytes)
10982023-11-16T16:49:41.417Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_54ec747e-1485-4343-b58a-a1f26532aae3/root/var/svc/log/oxide-nexus:default.log.0
10992023-11-16T16:49:41.448Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c0b8c479-e04e-4abe-ac1a-03e6aa7929cc/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11002023-11-16T16:49:42.515Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c0b8c479-e04e-4abe-ac1a-03e6aa7929cc/root/var/svc/log/oxide-cockroachdb:default.log
11012023-11-16T16:49:42.547Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c0b8c479-e04e-4abe-ac1a-03e6aa7929cc/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
11022023-11-16T16:49:43.637Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c0b8c479-e04e-4abe-ac1a-03e6aa7929cc/root/var/svc/log/oxide-cockroachdb:default.log.0
11032023-11-16T16:49:43.669Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f6b3f2c1-d382-4878-9ec9-2808994b9a09/root/var/svc/log/oxide-crucible-agent:default.log (10681 bytes)
11042023-11-16T16:49:44.788Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f6b3f2c1-d382-4878-9ec9-2808994b9a09/root/var/svc/log/oxide-crucible-agent:default.log
11052023-11-16T16:49:44.820Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f6b3f2c1-d382-4878-9ec9-2808994b9a09/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
11062023-11-16T16:49:45.909Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f6b3f2c1-d382-4878-9ec9-2808994b9a09/root/var/svc/log/oxide-crucible-agent:default.log.0
11072023-11-16T16:49:45.942Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f6b3f2c1-d382-4878-9ec9-2808994b9a09/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11082023-11-16T16:49:46.993Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f6b3f2c1-d382-4878-9ec9-2808994b9a09/root/var/svc/log/oxide-crucible-downstairs:default.log
11092023-11-16T16:49:47.041Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f6b3f2c1-d382-4878-9ec9-2808994b9a09/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11102023-11-16T16:49:48.127Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f6b3f2c1-d382-4878-9ec9-2808994b9a09/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11112023-11-16T16:49:48.158Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f6b3f2c1-d382-4878-9ec9-2808994b9a09/root/var/svc/log/oxide-crucible-downstairs:downstairs-9e885fb8-89a4-4008-96a0-bf8f51e19f21.log (5513 bytes)
11122023-11-16T16:49:49.250Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f6b3f2c1-d382-4878-9ec9-2808994b9a09/root/var/svc/log/oxide-crucible-downstairs:downstairs-9e885fb8-89a4-4008-96a0-bf8f51e19f21.log
11132023-11-16T16:49:49.281Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_61c03e11-aff2-45f8-86f7-cda5a253b67f/root/var/svc/log/oxide-nexus:default.log (347780 bytes)
11142023-11-16T16:49:49.551Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_61c03e11-aff2-45f8-86f7-cda5a253b67f/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 347780 -> 469612
11152023-11-16T16:49:50.643Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_61c03e11-aff2-45f8-86f7-cda5a253b67f/root/var/svc/log/oxide-nexus:default.log
11162023-11-16T16:49:50.675Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_61c03e11-aff2-45f8-86f7-cda5a253b67f/root/var/svc/log/oxide-nexus:default.log.0 (301555 bytes)
11172023-11-16T16:49:51.977Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_61c03e11-aff2-45f8-86f7-cda5a253b67f/root/var/svc/log/oxide-nexus:default.log.0
11182023-11-16T16:49:52.008Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45580ae6-c132-4165-a046-7c325ec959e9/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11192023-11-16T16:49:53.077Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45580ae6-c132-4165-a046-7c325ec959e9/root/var/svc/log/oxide-crucible-agent:default.log
11202023-11-16T16:49:53.109Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45580ae6-c132-4165-a046-7c325ec959e9/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11212023-11-16T16:49:54.199Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45580ae6-c132-4165-a046-7c325ec959e9/root/var/svc/log/oxide-crucible-agent:default.log.0
11222023-11-16T16:49:54.232Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45580ae6-c132-4165-a046-7c325ec959e9/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11232023-11-16T16:49:55.299Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45580ae6-c132-4165-a046-7c325ec959e9/root/var/svc/log/oxide-crucible-downstairs:default.log
11242023-11-16T16:49:55.332Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45580ae6-c132-4165-a046-7c325ec959e9/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11252023-11-16T16:49:56.421Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45580ae6-c132-4165-a046-7c325ec959e9/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11262023-11-16T16:49:56.452Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96/root/var/log/chrony/measurements.log (10960 bytes)
11272023-11-16T16:49:56.510Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96/root/var/log/chrony/measurements.log" changed size mid upload: 10960 -> 13563
11282023-11-16T16:49:57.601Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96/root/var/log/chrony/measurements.log
11292023-11-16T16:49:57.633Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96/root/var/log/chrony/statistics.log (6435 bytes)
11302023-11-16T16:49:57.691Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96/root/var/log/chrony/statistics.log" changed size mid upload: 6435 -> 9009
11312023-11-16T16:49:58.781Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96/root/var/log/chrony/statistics.log
11322023-11-16T16:49:58.845Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96/root/var/log/chrony/tracking.log (1722 bytes)
11332023-11-16T16:49:58.877Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96/root/var/log/chrony/tracking.log" changed size mid upload: 1722 -> 1986
11342023-11-16T16:49:59.933Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_badca0ff-3c59-47ee-a1f0-5e5bbf311d96/root/var/log/chrony/tracking.log