01HGYN0E7JNQ6EMMD0H6RXKAW6: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HGYN1EN6ZK0EN9W96HXZ6SNH

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-06T04:54:38.200Zjob dependencies complete; ready to run (waiting for 45 m 28 s)
22023-12-06T04:55:23.935Zjob assigned to worker 01HGYQMX6VQPPKCGFFQBDBPVP3 (queued for 45 s)
32023-12-06T04:55:24.136Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-12-06T04:55:24.183Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-12-06T04:55:24.224Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-12-06T04:55:24.263Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-06T04:55:24.306Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-12-06T04:55:24.347ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-06T04:55:24.385Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-06T04:55:24.425ZOxide Helios Version helios-2.0.22318 64-bit
112023-12-06T04:55:24.465ZConfiguring devices.
122023-12-06T04:55:24.507Z#################################################
132023-12-06T04:55:24.547Z#### ############# ##
142023-12-06T04:55:24.585Z### ### ############ ##
152023-12-06T04:55:24.624Z## ### # ## ### ## Oxide Computer Company ##
162023-12-06T04:55:24.705Z## ## ## ### # ### ##
172023-12-06T04:55:24.746Z## # ### #### #### This Station Under ##
182023-12-06T04:55:24.785Z### ### #### # ### Computer Control ##
192023-12-06T04:55:24.825Z#### #### ### ## ##
202023-12-06T04:55:24.863Z##################################################
212023-12-06T04:55:24.902ZHostname: sock
222023-12-06T04:55:24.945ZLoading smf(7) serice descriptions:
232023-12-06T04:55:25.477Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a.gz
242023-12-06T04:55:26.744Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a.gz
252023-12-06T04:55:26.744Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-06T04:55:47.065Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-06T04:55:47.065Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4.gz
282023-12-06T04:56:09.019Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4.gz
292023-12-06T04:56:09.020Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-06T04:56:10.078Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-06T04:56:10.111Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-06T04:56:14.513Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-06T04:56:14.547Zdownloading input: /input/package/work/version.txt
342023-12-06T04:56:14.578Zdownloaded input: /input/package/work/version.txt
352023-12-06T04:56:14.579Zdownloading input: /input/package/work/package.tar.gz
362023-12-06T04:56:32.576Zdownloaded input: /input/package/work/package.tar.gz
372023-12-06T04:56:32.577Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-06T04:57:05.770Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-06T04:57:05.809Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-06T04:57:18.661Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-06T04:57:18.688Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-06T04:58:12.478Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-06T04:58:12.478Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-06T04:58:54.768Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-06T04:58:54.768Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-06T04:59:41.680Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-06T04:59:41.704Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-06T04:59:48.445Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-06T04:59:48.445Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-06T04:59:59.071Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-06T04:59:59.071Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-06T05:00:04.362Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-06T05:00:04.395Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-06T05:00:12.346Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-06T05:00:12.376Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-06T05:00:12.410Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-06T05:00:12.410Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-06T05:00:19.163Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-06T05:00:19.163Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-06T05:00:24.333Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-06T05:00:24.363Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-06T05:00:49.379Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-06T05:00:49.380Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-06T05:01:05.362Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-06T05:01:05.362Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-06T05:01:05.408Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-06T05:01:05.408Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-06T05:01:17.335Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-06T05:01:17.360Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-06T05:01:32.527Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-06T05:01:32.527Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-06T05:02:46.354Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-06T05:02:46.388Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-06T05:03:16.916Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-06T05:03:16.917Zstarting task 0: "setup"
762023-12-06T05:03:16.959Z++ uname -s
772023-12-06T05:03:16.959Z+ kern=SunOS
782023-12-06T05:03:16.959Z+ case "$kern" in
792023-12-06T05:03:16.959Z+ groupadd -g 12345 build
802023-12-06T05:03:16.959Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-06T05:03:18.935Z+ zfs create -o mountpoint=/work rpool/work
822023-12-06T05:03:18.964Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-06T05:03:18.964Z+ home_fs=zfs
842023-12-06T05:03:18.964Z+ [[ zfs == autofs ]]
852023-12-06T05:03:18.964Z+ mkdir -p /home/build
862023-12-06T05:03:18.964Z+ chown build:build /home/build /work
872023-12-06T05:03:20.961Z+ chmod 0700 /home/build /work
882023-12-06T05:03:20.985Zprocess exited: duration 4048 ms, exit code 0
 
892023-12-06T05:03:21.035Zstarting task 1: "authentication"
902023-12-06T05:03:21.078Zprocess exited: duration 43 ms, exit code 0
 
912023-12-06T05:03:21.128Zstarting task 2: "build"
922023-12-06T05:03:21.152Z+ trap _exit_trap EXIT
932023-12-06T05:03:21.152Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-06T05:03:21.332Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-06T05:03:21.332Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-06T05:03:21.332Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-06T05:03:21.332Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-06T05:03:21.332Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-06T05:03:21.332Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-06T05:03:21.332Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-06T05:03:21.332Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-06T05:03:21.332Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-06T05:03:21.332Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-06T05:03:21.332Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-06T05:03:21.357Z+ [[ -d /opt/oxide ]]
1062023-12-06T05:03:21.357Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-06T05:03:21.357Z+ pfexec mkdir /opt/oxide
1082023-12-06T05:03:21.358Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-06T05:03:21.358Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-06T05:03:21.358Z+ cd /opt/oxide-underneath
1112023-12-06T05:03:21.358Z+ pfexec tar ceEp@/f - .
1122023-12-06T05:03:21.358Z+ cd /opt/oxide
1132023-12-06T05:03:21.358Z+ pfexec tar xveEp@/f -
1142023-12-06T05:03:21.453Zx ., 0 bytes, 0 tape blocks
1152023-12-06T05:03:21.453Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-06T05:03:21.453Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-06T05:03:21.453Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-12-06T05:03:21.477Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-06T05:03:21.477Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-06T05:03:21.502Z+ pfexec mkdir /opt/oxide/work
1212023-12-06T05:03:21.502Z+ pfexec chown build:build /opt/oxide/work
1222023-12-06T05:03:21.502Z+ cd /opt/oxide/work
1232023-12-06T05:03:21.502Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-06T05:03:22.978ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-12-06T05:03:22.978Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-12-06T05:03:22.978Zx out/omicron-sled-agent.tar, 111105536 bytes, 217003 tape blocks
1272023-12-06T05:03:22.979Zx out/target/test, 75 bytes, 1 tape blocks
1282023-12-06T05:03:22.979Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-12-06T05:03:22.979Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-12-06T05:03:22.979Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-12-06T05:03:22.979Zx target/release/omicron-package, 42582768 bytes, 83170 tape blocks
1322023-12-06T05:03:22.979Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-12-06T05:03:22.979Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-12-06T05:03:22.979Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-12-06T05:03:22.979Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-12-06T05:03:22.979Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-12-06T05:03:22.979Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-12-06T05:03:22.979Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-12-06T05:03:23.004Z
1402023-12-06T05:03:23.004Zreal 1.488067096
1412023-12-06T05:03:23.004Zuser 1.389781511
1422023-12-06T05:03:23.004Zsys 0.173894895
1432023-12-06T05:03:23.004Ztrap 0.000059641
1442023-12-06T05:03:23.004Ztflt 0.000000000
1452023-12-06T05:03:23.004Zdflt 0.000000000
1462023-12-06T05:03:23.004Zkflt 0.000000000
1472023-12-06T05:03:23.004Zlock 0.000000000
1482023-12-06T05:03:23.004Zslp 1.385538466
1492023-12-06T05:03:23.004Zlat 0.024902544
1502023-12-06T05:03:23.004Zstop 0.000119301
1512023-12-06T05:03:23.004Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-12-06T05:03:24.054Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-12-06T05:03:24.079Z+ mkdir tests
1542023-12-06T05:03:24.079Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-12-06T05:03:24.079Z+ ptime -m gunzip
1562023-12-06T05:03:24.079Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a
1572023-12-06T05:03:24.214Z
1582023-12-06T05:03:24.238Zreal 0.138817063
1592023-12-06T05:03:24.238Zuser 0.126208574
1602023-12-06T05:03:24.239Zsys 0.012364814
1612023-12-06T05:03:24.239Ztrap 0.000012259
1622023-12-06T05:03:24.239Ztflt 0.000000000
1632023-12-06T05:03:24.239Zdflt 0.000000000
1642023-12-06T05:03:24.239Zkflt 0.000000000
1652023-12-06T05:03:24.239Zlock 0.000000000
1662023-12-06T05:03:24.239Zslp 0.000018720
1672023-12-06T05:03:24.239Zlat 0.000022390
1682023-12-06T05:03:24.239Zstop 0.000125852
1692023-12-06T05:03:24.239Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a
1702023-12-06T05:03:24.239Z+ chmod a+x tests/bootstrap-be9c817b94af4e8a
1712023-12-06T05:03:24.239Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-12-06T05:03:24.239Z+ ptime -m gunzip
1732023-12-06T05:03:24.239Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-12-06T05:03:26.402Z
1752023-12-06T05:03:26.431Zreal 2.175435701
1762023-12-06T05:03:26.431Zuser 2.051313655
1772023-12-06T05:03:26.431Zsys 0.123723209
1782023-12-06T05:03:26.431Ztrap 0.000165133
1792023-12-06T05:03:26.431Ztflt 0.000000000
1802023-12-06T05:03:26.431Zdflt 0.000000000
1812023-12-06T05:03:26.431Zkflt 0.000000000
1822023-12-06T05:03:26.431Zlock 0.000000000
1832023-12-06T05:03:26.431Zslp 0.000021770
1842023-12-06T05:03:26.431Zlat 0.000024190
1852023-12-06T05:03:26.432Zstop 0.000120611
1862023-12-06T05:03:26.432Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-12-06T05:03:26.432Z+ chmod a+x tests/bootstrap
1882023-12-06T05:03:26.432Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-12-06T05:03:26.432Z+ ptime -m gunzip
1902023-12-06T05:03:26.432Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4
1912023-12-06T05:03:28.682Z
1922023-12-06T05:03:28.707Zreal 2.269870901
1932023-12-06T05:03:28.707Zuser 2.117293538
1942023-12-06T05:03:28.707Zsys 0.152158477
1952023-12-06T05:03:28.707Ztrap 0.000187052
1962023-12-06T05:03:28.707Ztflt 0.000000000
1972023-12-06T05:03:28.707Zdflt 0.000000000
1982023-12-06T05:03:28.707Zkflt 0.000000000
1992023-12-06T05:03:28.707Zlock 0.000000000
2002023-12-06T05:03:28.707Zslp 0.000022830
2012023-12-06T05:03:28.707Zlat 0.000023380
2022023-12-06T05:03:28.707Zstop 0.000120411
2032023-12-06T05:03:28.707Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4
2042023-12-06T05:03:28.707Z+ chmod a+x tests/end_to_end_tests-ff923d3f5c6228d4
2052023-12-06T05:03:28.707Z+ bmat address ls
2062023-12-06T05:03:28.707ZNAME CIDR FIRST LAST COUNT
2072023-12-06T05:03:28.707Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-12-06T05:03:28.707Z++ bmat address ls -f extra -Ho first,last,count
2092023-12-06T05:03:28.736Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-12-06T05:03:28.736Z+ EXTRA_IP_START=10.151.2.164
2112023-12-06T05:03:28.736Z+ EXTRA_IP_END=10.151.2.227
2122023-12-06T05:03:28.737Z+ EXTRA_IP_COUNT=64
2132023-12-06T05:03:28.737Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-12-06T05:03:28.737Z+ EXTRA_IP_BASE=10.151.2
2152023-12-06T05:03:28.737Z+ EXTRA_IP_FOCTET=164
2162023-12-06T05:03:28.737Z+ EXTRA_IP_LOCTET=227
2172023-12-06T05:03:28.737Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-12-06T05:03:28.737Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-12-06T05:03:28.737Z+ DNS_IP1=10.151.2.164
2202023-12-06T05:03:28.737Z+ DNS_IP2=10.151.2.165
2212023-12-06T05:03:28.737Z+ UPLINK_IP=10.151.2.174
2222023-12-06T05:03:28.737Z+ IPPOOL_START=10.151.2.175
2232023-12-06T05:03:28.737Z+ IPPOOL_END=10.151.2.227
2242023-12-06T05:03:28.737Z++ ipadm show-addr -po type,addr
2252023-12-06T05:03:28.737Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-12-06T05:03:28.737Z+ GATEWAY_IP=10.151.2.100
2272023-12-06T05:03:28.737Z+ [[ -n 10.151.2.100 ]]
2282023-12-06T05:03:28.737Z+ ping -s 10.151.2.100 56 1
2292023-12-06T05:03:28.737ZPING 10.151.2.100: 56 data bytes
2302023-12-06T05:03:28.737Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2312023-12-06T05:03:28.737Z
2322023-12-06T05:03:28.737Z----10.151.2.100 PING Statistics----
2332023-12-06T05:03:28.738Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-12-06T05:03:28.738Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-12-06T05:03:28.738Z++ arp -an
2362023-12-06T05:03:28.738Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-12-06T05:03:28.762Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-12-06T05:03:28.762Z+ routeadm -e ipv4-forwarding -u
2392023-12-06T05:03:28.850Z+ PXA_START=10.151.2.164
2402023-12-06T05:03:28.850Z+ PXA_END=10.151.2.227
2412023-12-06T05:03:28.850Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-12-06T05:03:28.874Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-12-06T05:03:28.984Z+ ZPOOL_VDEV_DIR=/scratch
2442023-12-06T05:03:28.984Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-12-06T05:03:29.008Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-12-06T05:03:29.008Z++ cd ./tools
2472023-12-06T05:03:29.009Z++ pwd
2482023-12-06T05:03:29.009Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-12-06T05:03:29.009Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-12-06T05:03:29.009Z+ SOFTNPU_MODE=zone
2512023-12-06T05:03:29.009Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-12-06T05:03:29.009Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-12-06T05:03:29.009Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-12-06T05:03:29.009Z++ dladm show-phys -p -o LINK
2552023-12-06T05:03:29.009Z++ head -1
2562023-12-06T05:03:29.066Z+ PHYSICAL_LINK=igb0
2572023-12-06T05:03:29.091Z+ echo 'Using igb0 as physical link'
2582023-12-06T05:03:29.091ZUsing igb0 as physical link
2592023-12-06T05:03:29.092Z+ ensure_run_as_root
2602023-12-06T05:03:29.092Z++ id -u
2612023-12-06T05:03:29.092Z+ [[ 0 -ne 0 ]]
2622023-12-06T05:03:29.092Z+ ensure_zpools
2632023-12-06T05:03:29.092Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-12-06T05:03:29.092Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-12-06T05:03:29.092Z+ readarray -t ZPOOLS
2662023-12-06T05:03:29.092Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-12-06T05:03:29.092Z++ sed 's/[ ",]//g'
2682023-12-06T05:03:29.092Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-12-06T05:03:29.092Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-12-06T05:03:29.092ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-12-06T05:03:29.093Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-12-06T05:03:29.093Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-12-06T05:03:29.093Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-12-06T05:03:29.153Z0+0 records in
2752023-12-06T05:03:29.153Z0+0 records out
2762023-12-06T05:03:29.153Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2772023-12-06T05:03:29.153Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-12-06T05:03:29.153Z+ set +x
2792023-12-06T05:03:29.153ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-12-06T05:03:29.153Z++ zpool list -o name
2812023-12-06T05:03:29.153Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-12-06T05:03:29.178Z+ [[ -z '' ]]
2832023-12-06T05:03:29.178Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-12-06T05:03:29.178Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-12-06T05:03:29.178Z+ set +x
2862023-12-06T05:03:29.178ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-12-06T05:03:29.178Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-12-06T05:03:29.178Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-12-06T05:03:29.179ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-12-06T05:03:29.179Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-12-06T05:03:29.179Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-12-06T05:03:29.179Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-12-06T05:03:29.179Z0+0 records in
2942023-12-06T05:03:29.179Z0+0 records out
2952023-12-06T05:03:29.179Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-12-06T05:03:29.179Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-12-06T05:03:29.179Z+ set +x
2982023-12-06T05:03:29.179ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-12-06T05:03:29.179Z++ zpool list -o name
3002023-12-06T05:03:29.179Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-12-06T05:03:29.179Z+ [[ -z '' ]]
3022023-12-06T05:03:29.180Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-12-06T05:03:29.204Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-12-06T05:03:29.204Z+ set +x
3052023-12-06T05:03:29.204ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-12-06T05:03:29.204Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-12-06T05:03:29.204Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-12-06T05:03:29.204ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3092023-12-06T05:03:29.205Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-12-06T05:03:29.205Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-12-06T05:03:29.205Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-12-06T05:03:29.205Z0+0 records in
3132023-12-06T05:03:29.205Z0+0 records out
3142023-12-06T05:03:29.205Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3152023-12-06T05:03:29.205Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-12-06T05:03:29.205Z+ set +x
3172023-12-06T05:03:29.205ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-12-06T05:03:29.205Z++ zpool list -o name
3192023-12-06T05:03:29.205Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-12-06T05:03:29.230Z+ [[ -z '' ]]
3212023-12-06T05:03:29.230Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-12-06T05:03:29.254Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-12-06T05:03:29.254Z+ set +x
3242023-12-06T05:03:29.254ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-12-06T05:03:29.254Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-12-06T05:03:29.254Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-12-06T05:03:29.255ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-12-06T05:03:29.255Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-12-06T05:03:29.255Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-12-06T05:03:29.255Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-12-06T05:03:29.255Z0+0 records in
3322023-12-06T05:03:29.255Z0+0 records out
3332023-12-06T05:03:29.255Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-12-06T05:03:29.255Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-12-06T05:03:29.255Z+ set +x
3362023-12-06T05:03:29.255ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-12-06T05:03:29.255Z++ zpool list -o name
3382023-12-06T05:03:29.255Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-12-06T05:03:29.255Z+ [[ -z '' ]]
3402023-12-06T05:03:29.256Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-12-06T05:03:29.282Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-12-06T05:03:29.282Z+ set +x
3432023-12-06T05:03:29.306ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-12-06T05:03:29.306Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-12-06T05:03:29.306Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-12-06T05:03:29.306ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-12-06T05:03:29.306Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-12-06T05:03:29.306Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-12-06T05:03:29.307Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-12-06T05:03:29.307Z0+0 records in
3512023-12-06T05:03:29.307Z0+0 records out
3522023-12-06T05:03:29.307Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3532023-12-06T05:03:29.307Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-12-06T05:03:29.307Z+ set +x
3552023-12-06T05:03:29.307ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-12-06T05:03:29.307Z++ zpool list -o name
3572023-12-06T05:03:29.307Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-12-06T05:03:29.307Z+ [[ -z '' ]]
3592023-12-06T05:03:29.307Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-12-06T05:03:29.342Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-12-06T05:03:29.342Z+ set +x
3622023-12-06T05:03:29.342ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-12-06T05:03:29.342Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-12-06T05:03:29.342Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-12-06T05:03:29.342ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-12-06T05:03:29.342Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-12-06T05:03:29.342Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-12-06T05:03:29.342Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-12-06T05:03:29.342Z0+0 records in
3702023-12-06T05:03:29.342Z0+0 records out
3712023-12-06T05:03:29.342Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3722023-12-06T05:03:29.342Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-12-06T05:03:29.342Z+ set +x
3742023-12-06T05:03:29.342ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-12-06T05:03:29.342Z++ zpool list -o name
3762023-12-06T05:03:29.342Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-12-06T05:03:29.369Z+ [[ -z '' ]]
3782023-12-06T05:03:29.369Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-12-06T05:03:29.394Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-12-06T05:03:29.394Z+ set +x
3812023-12-06T05:03:29.394ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-12-06T05:03:29.394Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-12-06T05:03:29.394Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-12-06T05:03:29.394ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-12-06T05:03:29.394Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-12-06T05:03:29.394Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-12-06T05:03:29.394Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-12-06T05:03:29.394Z0+0 records in
3892023-12-06T05:03:29.394Z0+0 records out
3902023-12-06T05:03:29.394Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-12-06T05:03:29.395Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-12-06T05:03:29.395Z+ set +x
3932023-12-06T05:03:29.395ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-12-06T05:03:29.395Z++ zpool list -o name
3952023-12-06T05:03:29.395Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-12-06T05:03:29.395Z+ [[ -z '' ]]
3972023-12-06T05:03:29.395Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-12-06T05:03:29.420Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-12-06T05:03:29.420Z+ set +x
4002023-12-06T05:03:29.420ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-12-06T05:03:29.420Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-12-06T05:03:29.420Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-12-06T05:03:29.420ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-12-06T05:03:29.420Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-12-06T05:03:29.420Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-12-06T05:03:29.420Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-12-06T05:03:29.420Z0+0 records in
4082023-12-06T05:03:29.421Z0+0 records out
4092023-12-06T05:03:29.421Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-12-06T05:03:29.421Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-12-06T05:03:29.421Z+ set +x
4122023-12-06T05:03:29.421ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-12-06T05:03:29.421Z++ zpool list -o name
4142023-12-06T05:03:29.421Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-12-06T05:03:29.446Z+ [[ -z '' ]]
4162023-12-06T05:03:29.446Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-12-06T05:03:29.471Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-12-06T05:03:29.471Z+ set +x
4192023-12-06T05:03:29.471ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-12-06T05:03:29.471Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-12-06T05:03:29.471Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-12-06T05:03:29.471ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-12-06T05:03:29.471Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-12-06T05:03:29.471Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-12-06T05:03:29.471Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-12-06T05:03:29.471Z0+0 records in
4272023-12-06T05:03:29.471Z0+0 records out
4282023-12-06T05:03:29.471Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4292023-12-06T05:03:29.471Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-12-06T05:03:29.471Z+ set +x
4312023-12-06T05:03:29.472ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-12-06T05:03:29.472Z++ zpool list -o name
4332023-12-06T05:03:29.472Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-12-06T05:03:29.498Z+ [[ -z '' ]]
4352023-12-06T05:03:29.498Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-12-06T05:03:29.523Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-12-06T05:03:29.523Z+ set +x
4382023-12-06T05:03:29.523ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-12-06T05:03:29.523Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-12-06T05:03:29.523Z+ readarray -t ZPOOLS
4412023-12-06T05:03:29.523Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-12-06T05:03:29.523Z++ sed 's/[ ",]//g'
4432023-12-06T05:03:29.523Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-12-06T05:03:29.523Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-12-06T05:03:29.523ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-12-06T05:03:29.523Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-12-06T05:03:29.523Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-12-06T05:03:29.523Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-12-06T05:03:29.523Z0+0 records in
4502023-12-06T05:03:29.524Z0+0 records out
4512023-12-06T05:03:29.524Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4522023-12-06T05:03:29.524Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-12-06T05:03:29.524Z+ set +x
4542023-12-06T05:03:29.524ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-12-06T05:03:29.524Z++ zpool list -o name
4562023-12-06T05:03:29.524Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-12-06T05:03:29.525Z+ [[ -z '' ]]
4582023-12-06T05:03:29.526Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-12-06T05:03:29.551Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-12-06T05:03:29.551Z+ set +x
4612023-12-06T05:03:29.551ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-12-06T05:03:29.551Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-12-06T05:03:29.551Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-12-06T05:03:29.551ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-12-06T05:03:29.551Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-12-06T05:03:29.551Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-12-06T05:03:29.551Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-12-06T05:03:29.576Z0+0 records in
4692023-12-06T05:03:29.576Z0+0 records out
4702023-12-06T05:03:29.576Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4712023-12-06T05:03:29.576Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-12-06T05:03:29.576Z+ set +x
4732023-12-06T05:03:29.576ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-12-06T05:03:29.577Z++ zpool list -o name
4752023-12-06T05:03:29.577Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-12-06T05:03:29.577Z+ [[ -z '' ]]
4772023-12-06T05:03:29.577Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-12-06T05:03:29.602Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-12-06T05:03:29.602Z+ set +x
4802023-12-06T05:03:29.602ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-12-06T05:03:29.602Z+ [[ zone == \z\o\n\e ]]
4822023-12-06T05:03:29.602Z+ ensure_simulated_links igb0
4832023-12-06T05:03:29.602Z+ local PHYSICAL_LINK=igb0
4842023-12-06T05:03:29.602Z+ INDICES=("0" "1")
4852023-12-06T05:03:29.602Z+ for I in "${INDICES[@]}"
4862023-12-06T05:03:29.602Z++ get_simnet_name_if_exists net0
4872023-12-06T05:03:29.603Z++ dladm show-simnet -p -o LINK net0
4882023-12-06T05:03:29.628Z++ echo ''
4892023-12-06T05:03:29.628Z+ [[ -z '' ]]
4902023-12-06T05:03:29.628Z+ dladm create-simnet -t net0
4912023-12-06T05:03:29.628Z+ dladm create-simnet -t sc0_0
4922023-12-06T05:03:29.628Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-12-06T05:03:29.628Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-12-06T05:03:29.628Z+ success 'Simnet net0/sc0_0 exists'
4952023-12-06T05:03:29.628Z+ set +x
4962023-12-06T05:03:29.628ZSimnet net0/sc0_0 exists
4972023-12-06T05:03:29.628Z+ for I in "${INDICES[@]}"
4982023-12-06T05:03:29.629Z++ get_simnet_name_if_exists net1
4992023-12-06T05:03:29.629Z++ dladm show-simnet -p -o LINK net1
5002023-12-06T05:03:29.629Z++ echo ''
5012023-12-06T05:03:29.629Z+ [[ -z '' ]]
5022023-12-06T05:03:29.629Z+ dladm create-simnet -t net1
5032023-12-06T05:03:29.656Z+ dladm create-simnet -t sc1_0
5042023-12-06T05:03:29.656Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-12-06T05:03:29.656Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-12-06T05:03:29.656Z+ success 'Simnet net1/sc1_0 exists'
5072023-12-06T05:03:29.656Z+ set +x
5082023-12-06T05:03:29.656ZSimnet net1/sc1_0 exists
5092023-12-06T05:03:29.657Z++ get_vnic_name_if_exists sc0_1
5102023-12-06T05:03:29.657Z++ dladm show-vnic -p -o LINK sc0_1
5112023-12-06T05:03:29.657Z++ echo ''
5122023-12-06T05:03:29.657Z+ [[ -z '' ]]
5132023-12-06T05:03:29.657Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-12-06T05:03:29.682Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-12-06T05:03:29.683Z+ success 'Vnic sc0_1 exists'
5162023-12-06T05:03:29.683Z+ set +x
5172023-12-06T05:03:29.683ZVnic sc0_1 exists
5182023-12-06T05:03:29.683Z+ warn_if_no_proxy_arp
5192023-12-06T05:03:29.683Z+ [[ -v PXA_START ]]
5202023-12-06T05:03:29.683Z+ [[ -v PXA_END ]]
5212023-12-06T05:03:29.683Z+ ensure_softnpu_zone
5222023-12-06T05:03:29.683Z+ zoneadm list
5232023-12-06T05:03:29.683Z+ grep -q sidecar_softnpu
5242023-12-06T05:03:29.683Z+ [[ -f out/npuzone/npuzone ]]
5252023-12-06T05:03:29.683Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5262023-12-06T05:03:29.683Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
5272023-12-06T05:03:34.671Zdownloading softnpu ...done
5282023-12-06T05:03:34.695Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5292023-12-06T05:03:40.848Zdownloading asic_program.so ...done
5302023-12-06T05:03:40.873Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-12-06T05:03:46.009Zdownloading scadm ...done
5322023-12-06T05:03:46.033Zcreating port tfportrear0_0 for sc0_0
5332023-12-06T05:03:46.033Zcreating port tfportqsfp0_0 for sc0_1
5342023-12-06T05:03:46.057Zsoftnpu zone setup
5352023-12-06T05:03:46.057Zconfigure zone
5362023-12-06T05:03:46.057Zinstall zone
5372023-12-06T05:03:47.102Zboot zone
5382023-12-06T05:03:50.210Z[sidecar_softnpu] svcs milestone/network
5392023-12-06T05:03:50.240ZSTATE STIME FMRI
5402023-12-06T05:03:50.240Zonline 5:03:48 svc:/milestone/network:default
5412023-12-06T05:03:50.240Z[sidecar_softnpu] /softnpu/npu start
5422023-12-06T05:03:50.265Z
5432023-12-06T05:03:50.289Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-12-06T05:03:50.290Z+ GATEWAY_IP=10.151.2.100
5452023-12-06T05:03:50.290Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-12-06T05:03:50.290ZUsing 10.151.2.100 as gateway ip
5472023-12-06T05:03:50.290Z+ [[ ! -v GATEWAY_MAC ]]
5482023-12-06T05:03:50.290Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-12-06T05:03:50.290Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-12-06T05:03:50.290Z+++ echo 18:c0:4d:0d:a0:2a
5512023-12-06T05:03:50.290Z+ COUNT=1
5522023-12-06T05:03:50.290Z+ [[ 1 -ne 1 ]]
5532023-12-06T05:03:50.290Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-12-06T05:03:50.290ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-12-06T05:03:50.290Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-12-06T05:03:50.290Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-12-06T05:03:50.314Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-12-06T05:03:50.314Z+ [[ -v PXA_START ]]
5592023-12-06T05:03:50.314Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-12-06T05:03:50.314Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-12-06T05:03:50.339Z+ z_scadm dump-state
5622023-12-06T05:03:50.339Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-12-06T05:03:50.364Zlocal v6:
5642023-12-06T05:03:50.364Zlocal v4:
5652023-12-06T05:03:50.364Zrouter v6:
5662023-12-06T05:03:50.364Zrouter v4_idx:
5672023-12-06T05:03:50.364Zrouter v4_routes:
5682023-12-06T05:03:50.364Zresolver v4:
5692023-12-06T05:03:50.364Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-12-06T05:03:50.364Zresolver v6:
5712023-12-06T05:03:50.364Znat v4:
5722023-12-06T05:03:50.364Znat_v6:
5732023-12-06T05:03:50.364Zport_mac:
5742023-12-06T05:03:50.364Zproxy arp:
5752023-12-06T05:03:50.364Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-12-06T05:03:50.388Z+ success 'softnpu zone exists'
5772023-12-06T05:03:50.388Z+ set +x
5782023-12-06T05:03:50.389Zsoftnpu zone exists
5792023-12-06T05:03:50.389Z
5802023-12-06T05:03:50.389Zreal 21.381163713
5812023-12-06T05:03:50.389Zuser 0.735646011
5822023-12-06T05:03:50.389Zsys 1.647334577
5832023-12-06T05:03:50.389Ztrap 0.000099586
5842023-12-06T05:03:50.389Ztflt 0.000250936
5852023-12-06T05:03:50.389Zdflt 0.001544118
5862023-12-06T05:03:50.389Zkflt 0.000000000
5872023-12-06T05:03:50.389Zlock 16:27.044688365
5882023-12-06T05:03:50.389Zslp 1:03.964082526
5892023-12-06T05:03:50.389Zlat 0.087304992
5902023-12-06T05:03:50.389Zstop 0.346674857
5912023-12-06T05:03:50.389Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-12-06T05:03:50.413Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-12-06T05:03:50.414Z+ SILO_NAME=recovery
5942023-12-06T05:03:50.414Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-06T05:03:50.414Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-12-06T05:03:50.441Z+ sed -i~ '
5972023-12-06T05:03:50.441Z /^external_dns_ips/c\
5982023-12-06T05:03:50.441Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-12-06T05:03:50.441Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-12-06T05:03:50.441Z /^first/c\
6012023-12-06T05:03:50.441Zfirst = "10.151.2.164"
6022023-12-06T05:03:50.441Z /^last/c\
6032023-12-06T05:03:50.441Zlast = "10.151.2.173"
6042023-12-06T05:03:50.441Z }
6052023-12-06T05:03:50.441Z /^\[rack_network_config/,/^$/ {
6062023-12-06T05:03:50.441Z /^infra_ip_first/c\
6072023-12-06T05:03:50.441Zinfra_ip_first = "10.151.2.174"
6082023-12-06T05:03:50.442Z /^infra_ip_last/c\
6092023-12-06T05:03:50.442Zinfra_ip_last = "10.151.2.174"
6102023-12-06T05:03:50.442Z }
6112023-12-06T05:03:50.442Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-12-06T05:03:50.442Z /^routes/c\
6132023-12-06T05:03:50.442Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6142023-12-06T05:03:50.442Z /^addresses/c\
6152023-12-06T05:03:50.442Zaddresses = \["10.151.2.174/32"\]
6162023-12-06T05:03:50.442Z }
6172023-12-06T05:03:50.442Z' pkg/config-rss.toml
6182023-12-06T05:03:50.442Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-12-06T05:03:50.442Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-12-06T05:03:50.442Z+++ pkg/config-rss.toml Wed Dec 6 05:03:50 2023
6212023-12-06T05:03:50.443Z@@ -41,7 +41,7 @@
6222023-12-06T05:03:50.443Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-12-06T05:03:50.443Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-12-06T05:03:50.443Z # below.
6252023-12-06T05:03:50.443Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-12-06T05:03:50.443Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6272023-12-06T05:03:50.443Z
6282023-12-06T05:03:50.443Z # Initial TLS certificates for the external API
6292023-12-06T05:03:50.443Z #
6302023-12-06T05:03:50.443Z@@ -77,8 +77,8 @@
6312023-12-06T05:03:50.443Z #
6322023-12-06T05:03:50.443Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-12-06T05:03:50.443Z [[internal_services_ip_pool_ranges]]
6342023-12-06T05:03:50.443Z-first = "192.168.1.20"
6352023-12-06T05:03:50.443Z-last = "192.168.1.29"
6362023-12-06T05:03:50.444Z+first = "10.151.2.164"
6372023-12-06T05:03:50.444Z+last = "10.151.2.173"
6382023-12-06T05:03:50.444Z
6392023-12-06T05:03:50.444Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-12-06T05:03:50.444Z # parameters should be provided to the control plane via wicket, but we need to
6412023-12-06T05:03:50.444Z@@ -92,8 +92,8 @@
6422023-12-06T05:03:50.444Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-12-06T05:03:50.444Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-12-06T05:03:50.444Z # With softnpu, only one address is used.
6452023-12-06T05:03:50.444Z-infra_ip_first = "192.168.1.30"
6462023-12-06T05:03:50.444Z-infra_ip_last = "192.168.1.30"
6472023-12-06T05:03:50.444Z+infra_ip_first = "10.151.2.174"
6482023-12-06T05:03:50.444Z+infra_ip_last = "10.151.2.174"
6492023-12-06T05:03:50.444Z
6502023-12-06T05:03:50.445Z # Configurations for BGP routers to run on the scrimlets.
6512023-12-06T05:03:50.445Z bgp = []
6522023-12-06T05:03:50.445Z@@ -101,9 +101,9 @@
6532023-12-06T05:03:50.445Z # You can configure multiple uplinks by repeating the following stanza
6542023-12-06T05:03:50.445Z [[rack_network_config.ports]]
6552023-12-06T05:03:50.445Z # Routes associated with this port.
6562023-12-06T05:03:50.445Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-12-06T05:03:50.445Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6582023-12-06T05:03:50.445Z # Addresses associated with this port.
6592023-12-06T05:03:50.445Z-addresses = ["192.168.1.30/32"]
6602023-12-06T05:03:50.445Z+addresses = ["10.151.2.174/32"]
6612023-12-06T05:03:50.445Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-12-06T05:03:50.445Z port = "qsfp0"
6632023-12-06T05:03:50.445Z # The speed of this port.
6642023-12-06T05:03:50.446Z+ true
6652023-12-06T05:03:50.446Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-12-06T05:03:50.446Za pkg/config-rss.toml 7K
6672023-12-06T05:03:50.446Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-12-06T05:03:50.446Z+ TLS_NAME=recovery.sys.oxide.test
6692023-12-06T05:03:50.446Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-12-06T05:03:50.859Z........+......+...+....+...+..+.......+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...+..+.+......+......+..+...........................+...+...+...+......+.+...+...........+.+..............+...+.+.....+....+...+..+.+..+............+.+..............+......+.+..............+.......+........+....+...+..+.......+...+.....+.......+.........+......+.....+...............+.+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....................+......................+............+.....+...............+................+.....+..........+..+............+.............+......+.........+.....+.............+...+......+.....+................+.....+..........+..+.+.........+......+...............+......+........+................+.........+..+.............+...........+....+...+.................+.+......+......+..+............+...+.......+......+...+......+........+.......+...+........+...+....+..+..........+...+.....+...+..........+...+.....+......+.....................+.........................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-12-06T05:03:51.679Z..+....+..+.+.....+...+....+..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...............+....+......+......+........+...+...+.+........+.+..+.........+.+.........+.........+..+.........+.+..+...+...+......+.........+....+..+...................+.........+...+..+...+.......+..+...............+.+..+......+......+......+...+.+......+...+..+...+....+...+..............+......+............+......................+........+......+.+...........+............................+.........+......+.....+............+....+..............+......+.+........+............+.......+...+...+..............................+..+....+........+.+.....................+...............+..................+...............+.....+.+.........+........+.............+.........+..+...+......+...........................+....+....................+.+..+.......+.....+...+...............+.+......+...+.....................+..+....+...........+..................+....+..+......+..................+.+...+............+......+...+..............+...+...............+.........+......+.........+.+.........+...+.....+.+.................+...+.+...+..+..................+.+..+...+......+...+..........+.................+.+...+......+...........+...+....+...........+.+...+...........+......+.+........+.......+..................+..+...+.......+...+.........+......+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-06T05:03:51.706Z-----
6732023-12-06T05:03:51.706Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-12-06T05:03:51.731Za pkg/initial-tls-cert.pem 2K
6752023-12-06T05:03:51.731Za pkg/initial-tls-key.pem 4K
6762023-12-06T05:03:51.731Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-12-06T05:03:51.731Z+ rmdir pkg
6782023-12-06T05:03:51.731Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-12-06T05:03:51.731Z+ pfexec mkdir -p /usr/oxide
6802023-12-06T05:03:51.731Z+ pfexec rm -f /usr/oxide/tcpproxy
6812023-12-06T05:03:51.731Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6822023-12-06T05:03:52.011Z+ pfexec chmod +x /usr/oxide/tcpproxy
6832023-12-06T05:03:52.036Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6842023-12-06T05:03:52.036Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6852023-12-06T05:03:52.036Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6862023-12-06T05:03:52.134Z+ OMICRON_NO_UNINSTALL=1
6872023-12-06T05:03:52.134Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6882023-12-06T05:03:52.158ZDec 06 05:03:52.271 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6892023-12-06T05:03:52.158ZDec 06 05:03:52.273 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6902023-12-06T05:03:52.159ZDec 06 05:03:52.273 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6912023-12-06T05:03:52.159ZDec 06 05:03:52.273 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6922023-12-06T05:03:52.159ZDec 06 05:03:52.273 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6932023-12-06T05:03:52.159ZDec 06 05:03:52.273 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6942023-12-06T05:03:52.159ZDec 06 05:03:52.273 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6952023-12-06T05:03:52.159ZDec 06 05:03:52.273 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6962023-12-06T05:03:52.159ZDec 06 05:03:52.274 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6972023-12-06T05:03:52.159ZDec 06 05:03:52.274 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6982023-12-06T05:03:52.159ZDec 06 05:03:52.274 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6992023-12-06T05:03:52.159ZDec 06 05:03:52.274 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7002023-12-06T05:03:52.159ZDec 06 05:03:52.274 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7012023-12-06T05:03:52.159ZDec 06 05:03:52.274 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7022023-12-06T05:03:52.159ZDec 06 05:03:52.274 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
7032023-12-06T05:03:52.159ZDec 06 05:03:52.274 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7042023-12-06T05:03:52.159ZDec 06 05:03:52.274 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7052023-12-06T05:03:52.397ZDec 06 05:03:52.520 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-12-06T05:03:52.434ZDec 06 05:03:52.556 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-12-06T05:03:52.492ZDec 06 05:03:52.615 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-12-06T05:03:52.621Z
7092023-12-06T05:03:52.645Zreal 0.482177760
7102023-12-06T05:03:52.645Zuser 0.100532526
7112023-12-06T05:03:52.645Zsys 2.011422434
7122023-12-06T05:03:52.646Ztrap 0.000012688
7132023-12-06T05:03:52.646Ztflt 0.000044279
7142023-12-06T05:03:52.646Zdflt 0.000342495
7152023-12-06T05:03:52.646Zkflt 0.000000000
7162023-12-06T05:03:52.646Zlock 43.249563381
7172023-12-06T05:03:52.646Zslp 0.687473266
7182023-12-06T05:03:52.646Zlat 0.002962275
7192023-12-06T05:03:52.646Zstop 0.206532191
7202023-12-06T05:03:52.646Z+ retry=0
7212023-12-06T05:03:52.646Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-12-06T05:03:52.646Z+ [[ 0 -gt 30 ]]
7232023-12-06T05:03:52.646Z+ sleep 1
7242023-12-06T05:03:53.635Z+ retry=1
7252023-12-06T05:03:53.635Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-12-06T05:03:53.659Z+ [[ 1 -gt 30 ]]
7272023-12-06T05:03:53.659Z+ sleep 1
7282023-12-06T05:03:54.648Z+ retry=2
7292023-12-06T05:03:54.648Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-12-06T05:03:54.672Z+ [[ 2 -gt 30 ]]
7312023-12-06T05:03:54.672Z+ sleep 1
7322023-12-06T05:03:55.663Z+ retry=3
7332023-12-06T05:03:55.664Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-12-06T05:03:55.688Z+ [[ 3 -gt 30 ]]
7352023-12-06T05:03:55.689Z+ sleep 1
7362023-12-06T05:03:56.680Z+ retry=4
7372023-12-06T05:03:56.680Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-12-06T05:04:21.857Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7392023-12-06T05:04:21.885Zlocal v6:
7402023-12-06T05:04:21.885Zfe80::94fa:6dff:fe2d:d7df
7412023-12-06T05:04:21.885Zfd00:99::1
7422023-12-06T05:04:21.885Zlocal v4:
7432023-12-06T05:04:21.885Z10.151.2.174
7442023-12-06T05:04:21.885Zrouter v6:
7452023-12-06T05:04:21.885Zfd00:1122:3344:3::/64 -> fe80::d472:32ff:fe47:acdd (1)
7462023-12-06T05:04:21.885Zfdb0:18c0:4d0d:a02a::/64 -> fe80::d472:32ff:fe47:acdd (1)
7472023-12-06T05:04:21.885Zfd00:1122:3344:1::/64 -> fe80::d472:32ff:fe47:acdd (1)
7482023-12-06T05:04:21.885Zfd00:1122:3344:2::/64 -> fe80::d472:32ff:fe47:acdd (1)
7492023-12-06T05:04:21.885Zfd00:1122:3344:101::/64 -> fe80::d472:32ff:fe47:acdd (1)
7502023-12-06T05:04:21.885Zrouter v4_idx:
7512023-12-06T05:04:21.885Z0.0.0.0/0 -> 0
7522023-12-06T05:04:21.885Zrouter v4_routes:
7532023-12-06T05:04:21.886Z0 -> 10.151.2.100 (2)
7542023-12-06T05:04:21.886Zresolver v4:
7552023-12-06T05:04:21.939Z10.151.2.174 -> 1a:39:c3:36:77:82
7562023-12-06T05:04:21.939Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7572023-12-06T05:04:21.939Zresolver v6:
7582023-12-06T05:04:21.939Zfe80::94fa:6dff:fe2d:d7df -> 96:fa:6d:2d:d7:df
7592023-12-06T05:04:21.939Zfe80::d472:32ff:fe47:acdd -> d6:72:32:47:ac:dd
7602023-12-06T05:04:21.939Znat v4:
7612023-12-06T05:04:21.939Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:b1:0d
7622023-12-06T05:04:21.939Znat_v6:
7632023-12-06T05:04:21.940Zport_mac:
7642023-12-06T05:04:21.940Z1: a8:40:25:47:c8:87
7652023-12-06T05:04:21.940Z1000: a8:40:25:47:c8:86
7662023-12-06T05:04:21.940Zproxy arp:
7672023-12-06T05:04:21.940Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7682023-12-06T05:04:21.940Z+ retry=0
7692023-12-06T05:04:21.940Z+++ zoneadm list -n
7702023-12-06T05:04:21.940Z+++ grep oxz_ntp
7712023-12-06T05:04:21.940Z++ pfexec svcs -z oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef -Hostate oxide/ntp
7722023-12-06T05:04:21.967Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7732023-12-06T05:04:21.967Z++ true
7742023-12-06T05:04:21.967Z+ [[ '' != online ]]
7752023-12-06T05:04:21.968Z+ [[ 0 -gt 60 ]]
7762023-12-06T05:04:21.968Z+ sleep 1
7772023-12-06T05:04:22.923Z+ retry=1
7782023-12-06T05:04:22.985Z+++ zoneadm list -n
7792023-12-06T05:04:22.986Z+++ grep oxz_ntp
7802023-12-06T05:04:22.986Z++ pfexec svcs -z oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef -Hostate oxide/ntp
7812023-12-06T05:04:22.986Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7822023-12-06T05:04:22.986Z++ true
7832023-12-06T05:04:22.986Z+ [[ '' != online ]]
7842023-12-06T05:04:22.986Z+ [[ 1 -gt 60 ]]
7852023-12-06T05:04:22.986Z+ sleep 1
7862023-12-06T05:04:23.954Z+ retry=2
7872023-12-06T05:04:24.060Z+++ zoneadm list -n
7882023-12-06T05:04:24.060Z+++ grep oxz_ntp
7892023-12-06T05:04:24.060Z++ pfexec svcs -z oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef -Hostate oxide/ntp
7902023-12-06T05:04:24.061Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7912023-12-06T05:04:24.061Z++ true
7922023-12-06T05:04:24.061Z+ [[ '' != online ]]
7932023-12-06T05:04:24.061Z+ [[ 2 -gt 60 ]]
7942023-12-06T05:04:24.061Z+ sleep 1
7952023-12-06T05:04:24.979Z+ retry=3
7962023-12-06T05:04:25.004Z+++ zoneadm list -n
7972023-12-06T05:04:25.004Z+++ grep oxz_ntp
7982023-12-06T05:04:25.004Z++ pfexec svcs -z oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef -Hostate oxide/ntp
7992023-12-06T05:04:25.004Z+ [[ disabled != online ]]
8002023-12-06T05:04:25.004Z+ [[ 3 -gt 60 ]]
8012023-12-06T05:04:25.004Z+ sleep 1
8022023-12-06T05:04:26.006Z+ retry=4
8032023-12-06T05:04:26.031Z+++ zoneadm list -n
8042023-12-06T05:04:26.031Z+++ grep oxz_ntp
8052023-12-06T05:04:26.031Z++ pfexec svcs -z oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef -Hostate oxide/ntp
8062023-12-06T05:04:26.031Z+ [[ disabled != online ]]
8072023-12-06T05:04:26.031Z+ [[ 4 -gt 60 ]]
8082023-12-06T05:04:26.031Z+ sleep 1
8092023-12-06T05:04:27.025Z+ retry=5
8102023-12-06T05:04:27.050Z+++ zoneadm list -n
8112023-12-06T05:04:27.050Z+++ grep oxz_ntp
8122023-12-06T05:04:27.051Z++ pfexec svcs -z oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef -Hostate oxide/ntp
8132023-12-06T05:04:27.051Z+ [[ disabled != online ]]
8142023-12-06T05:04:27.051Z+ [[ 5 -gt 60 ]]
8152023-12-06T05:04:27.051Z+ sleep 1
8162023-12-06T05:04:28.068Z+ retry=6
8172023-12-06T05:04:28.093Z+++ zoneadm list -n
8182023-12-06T05:04:28.093Z+++ grep oxz_ntp
8192023-12-06T05:04:28.093Z++ pfexec svcs -z oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef -Hostate oxide/ntp
8202023-12-06T05:04:28.093Z+ [[ disabled != online ]]
8212023-12-06T05:04:28.093Z+ [[ 6 -gt 60 ]]
8222023-12-06T05:04:28.093Z+ sleep 1
8232023-12-06T05:04:29.082Z+ retry=7
8242023-12-06T05:04:29.110Z+++ zoneadm list -n
8252023-12-06T05:04:29.111Z+++ grep oxz_ntp
8262023-12-06T05:04:29.111Z++ pfexec svcs -z oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef -Hostate oxide/ntp
8272023-12-06T05:04:29.111Z+ [[ disabled != online ]]
8282023-12-06T05:04:29.112Z+ [[ 7 -gt 60 ]]
8292023-12-06T05:04:29.112Z+ sleep 1
8302023-12-06T05:04:30.089Z+ retry=8
8312023-12-06T05:04:30.114Z+++ zoneadm list -n
8322023-12-06T05:04:30.114Z+++ grep oxz_ntp
8332023-12-06T05:04:30.114Z++ pfexec svcs -z oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef -Hostate oxide/ntp
8342023-12-06T05:04:30.114Z+ [[ disabled != online ]]
8352023-12-06T05:04:30.114Z+ [[ 8 -gt 60 ]]
8362023-12-06T05:04:30.114Z+ sleep 1
8372023-12-06T05:04:31.110Z+ retry=9
8382023-12-06T05:04:31.134Z+++ zoneadm list -n
8392023-12-06T05:04:31.135Z+++ grep oxz_ntp
8402023-12-06T05:04:31.135Z++ pfexec svcs -z oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef -Hostate oxide/ntp
8412023-12-06T05:04:31.135Z+ [[ disabled != online ]]
8422023-12-06T05:04:31.135Z+ [[ 9 -gt 60 ]]
8432023-12-06T05:04:31.135Z+ sleep 1
 
8442023-12-06T05:04:31.525Zsock console login: Dec 6 05:04:31 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8452023-12-06T05:04:32.134Z+ retry=10
8462023-12-06T05:04:32.182Z+++ zoneadm list -n
8472023-12-06T05:04:32.182Z+++ grep oxz_ntp
8482023-12-06T05:04:32.182Z++ pfexec svcs -z oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef -Hostate oxide/ntp
8492023-12-06T05:04:32.182Z+ [[ online != online ]]
8502023-12-06T05:04:32.182Z+ echo 'Waited for chrony: 10s'
8512023-12-06T05:04:32.183ZWaited for chrony: 10s
8522023-12-06T05:04:32.183Z+ export RUST_BACKTRACE=1
8532023-12-06T05:04:32.183Z+ RUST_BACKTRACE=1
8542023-12-06T05:04:32.183Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8552023-12-06T05:04:32.183Z+ ./tests/bootstrap
8562023-12-06T05:04:32.622ZUsing external DNS server at 10.151.2.164:53
8572023-12-06T05:04:32.622Z2023-12-06 05:04:32.303417222 UTC: attempting to log into API
8582023-12-06T05:04:47.217Z2023-12-06 05:04:46.833041152 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8592023-12-06T05:04:48.220Z2023-12-06 05:04:47.833942854 UTC: attempting to log into API
8602023-12-06T05:05:03.269Z2023-12-06 05:05:02.866262779 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8612023-12-06T05:05:04.258Z2023-12-06 05:05:03.867075670 UTC: attempting to log into API
8622023-12-06T05:05:19.299Z2023-12-06 05:05:19.226184615 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8632023-12-06T05:05:20.290Z2023-12-06 05:05:20.230414476 UTC: attempting to log into API
8642023-12-06T05:05:35.348Z2023-12-06 05:05:35.332239129 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8652023-12-06T05:05:36.351Z2023-12-06 05:05:36.334879864 UTC: attempting to log into API
 
8662023-12-06T05:05:38.753ZDec 6 05:05:38 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
8672023-12-06T05:05:45.891ZDec 6 05:05:45 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
8682023-12-06T05:05:51.393Z2023-12-06 05:05:51.369046353 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8692023-12-06T05:05:52.383Z2023-12-06 05:05:52.370117208 UTC: attempting to log into API
 
8702023-12-06T05:05:53.056ZDec 6 05:05:52 sok ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
8712023-12-06T05:06:02.852ZDec 6 05:06:02 sok ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
8722023-12-06T05:06:07.420Z2023-12-06 05:06:07.404558809 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8732023-12-06T05:06:08.420Z2023-12-06 05:06:08.406633809 UTC: attempting to log into API
 
8742023-12-06T05:06:08.475ZDec 6 05:06:07 sok ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8752023-12-06T05:06:23.451Z2023-12-06 05:06:23.440122348 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8762023-12-06T05:06:24.453Z2023-12-06 05:06:24.441064850 UTC: attempting to log into API
8772023-12-06T05:06:39.486Z2023-12-06 05:06:39.475675573 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8782023-12-06T05:06:40.488Z2023-12-06 05:06:40.477212895 UTC: attempting to log into API
8792023-12-06T05:06:51.283Z2023-12-06 05:06:51.273090733 UTC: login succeeded
8802023-12-06T05:06:51.315Zwaiting for nexus to come up...
8812023-12-06T05:06:51.386Zcreating IP pool... 10.151.2.175 - 10.151.2.227
8822023-12-06T05:06:51.435Zensuring datasets are ready...
8832023-12-06T05:06:54.166Zlet's roll.
8842023-12-06T05:06:54.192Z+ rm ./tests/bootstrap
8852023-12-06T05:06:54.250Z+ for test_bin in tests/*
8862023-12-06T05:06:54.250Z+ ./tests/bootstrap-be9c817b94af4e8a
8872023-12-06T05:06:54.275Z
8882023-12-06T05:06:54.275Zrunning 0 tests
8892023-12-06T05:06:54.275Z
8902023-12-06T05:06:54.275Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
8912023-12-06T05:06:54.275Z
8922023-12-06T05:06:54.275Z+ for test_bin in tests/*
8932023-12-06T05:06:54.275Z+ ./tests/end_to_end_tests-ff923d3f5c6228d4
8942023-12-06T05:06:54.331Z
8952023-12-06T05:06:54.331Zrunning 2 tests
8962023-12-06T05:06:54.508Ztest no_spoof::no_spoof_here ... ok
8972023-12-06T05:07:54.277Ztest instance_launch::instance_launch has been running for over 60 seconds
8982023-12-06T05:09:52.011Ztest instance_launch::instance_launch ... ok
8992023-12-06T05:09:52.011Z
9002023-12-06T05:09:52.011Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 177.73s
9012023-12-06T05:09:52.011Z
9022023-12-06T05:09:52.038Z+ _exit_trap
9032023-12-06T05:09:52.038Z+ local status=0
9042023-12-06T05:09:52.038Z+ [[ 0 -eq 0 ]]
9052023-12-06T05:09:52.038Z+ exit 0
9062023-12-06T05:09:52.038Zprocess exited: duration 390889 ms, exit code 0
 
9072023-12-06T05:09:52.149Zfound 47 output files
9082023-12-06T05:09:52.150Zuploading: /var/svc/log/oxide-sled-agent:default.log (643961 bytes)
9092023-12-06T05:09:54.062Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9102023-12-06T05:09:54.062Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b/root/var/svc/log/oxide-clickhouse:default.log (558642 bytes)
9112023-12-06T05:09:55.929Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b/root/var/svc/log/oxide-clickhouse:default.log
9122023-12-06T05:09:55.929Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
9132023-12-06T05:09:57.036Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb/root/var/svc/log/oxide-cockroachdb:default.log
9142023-12-06T05:09:57.036Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9152023-12-06T05:09:58.130Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa/root/var/svc/log/oxide-crucible-agent:default.log
9162023-12-06T05:09:58.130Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9172023-12-06T05:09:59.224Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa/root/var/svc/log/oxide-crucible-downstairs:default.log
9182023-12-06T05:09:59.224Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0/root/var/svc/log/oxide-crucible-agent:default.log (13684 bytes)
9192023-12-06T05:10:00.368Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0/root/var/svc/log/oxide-crucible-agent:default.log
9202023-12-06T05:10:00.369Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9212023-12-06T05:10:01.464Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0/root/var/svc/log/oxide-crucible-downstairs:default.log
9222023-12-06T05:10:01.464Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0/root/var/svc/log/oxide-crucible-downstairs:downstairs-f56fb970-13c6-46c9-9c82-377f6730cbe8.log (3049 bytes)
9232023-12-06T05:10:02.565Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0/root/var/svc/log/oxide-crucible-downstairs:downstairs-f56fb970-13c6-46c9-9c82-377f6730cbe8.log
9242023-12-06T05:10:02.566Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9252023-12-06T05:10:03.665Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4/root/var/svc/log/oxide-crucible-pantry:default.log
9262023-12-06T05:10:03.665Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c/root/var/svc/log/oxide-internal_dns:default.log (99887 bytes)
9272023-12-06T05:10:05.301Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c/root/var/svc/log/oxide-internal_dns:default.log
9282023-12-06T05:10:05.301Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
9292023-12-06T05:10:06.399Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b/root/var/svc/log/oxide-cockroachdb:default.log
9302023-12-06T05:10:06.399Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9312023-12-06T05:10:07.497Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25/root/var/svc/log/oxide-crucible-agent:default.log
9322023-12-06T05:10:07.497Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9332023-12-06T05:10:08.596Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25/root/var/svc/log/oxide-crucible-downstairs:default.log
9342023-12-06T05:10:08.596Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9/root/var/svc/log/oxide-external_dns:default.log (15613 bytes)
9352023-12-06T05:10:09.730Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9/root/var/svc/log/oxide-external_dns:default.log
9362023-12-06T05:10:09.730Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087/root/var/svc/log/oxide-nexus:default.log (560960 bytes)
9372023-12-06T05:10:09.995Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 560960 -> 625412
9382023-12-06T05:10:11.081Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087/root/var/svc/log/oxide-nexus:default.log
9392023-12-06T05:10:11.081Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065/root/var/svc/log/oxide-nexus:default.log (565687 bytes)
9402023-12-06T05:10:11.339Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 565687 -> 630141
9412023-12-06T05:10:12.513Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065/root/var/svc/log/oxide-nexus:default.log
9422023-12-06T05:10:12.513Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a/root/var/svc/log/oxide-cockroachdb:default.log (5341 bytes)
9432023-12-06T05:10:13.633Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a/root/var/svc/log/oxide-cockroachdb:default.log
9442023-12-06T05:10:13.634Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
9452023-12-06T05:10:14.731Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35/root/var/svc/log/oxide-cockroachdb:default.log
9462023-12-06T05:10:14.731Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9472023-12-06T05:10:15.831Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff/root/var/svc/log/oxide-crucible-pantry:default.log
9482023-12-06T05:10:15.855Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35/root/var/svc/log/oxide-internal_dns:default.log (98263 bytes)
9492023-12-06T05:10:15.999Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 98263 -> 102019
9502023-12-06T05:10:17.082Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35/root/var/svc/log/oxide-internal_dns:default.log
9512023-12-06T05:10:17.082Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef/root/var/svc/log/oxide-ntp:default.log (1310 bytes)
9522023-12-06T05:10:18.178Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef/root/var/svc/log/oxide-ntp:default.log
9532023-12-06T05:10:18.178Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d/root/var/svc/log/oxide-oximeter:default.log (160042 bytes)
9542023-12-06T05:10:18.362Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 160042 -> 174463
9552023-12-06T05:10:19.434Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d/root/var/svc/log/oxide-oximeter:default.log
9562023-12-06T05:10:19.434Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35/root/var/svc/log/oxide-crucible-agent:default.log (14119 bytes)
9572023-12-06T05:10:20.557Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35/root/var/svc/log/oxide-crucible-agent:default.log
9582023-12-06T05:10:20.557Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9592023-12-06T05:10:21.658Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35/root/var/svc/log/oxide-crucible-downstairs:default.log
9602023-12-06T05:10:21.658Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35/root/var/svc/log/oxide-crucible-downstairs:downstairs-6d4a13ff-b0d7-4144-8087-5042e3a49a15.log (3049 bytes)
9612023-12-06T05:10:22.755Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35/root/var/svc/log/oxide-crucible-downstairs:downstairs-6d4a13ff-b0d7-4144-8087-5042e3a49a15.log
9622023-12-06T05:10:22.755Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
9632023-12-06T05:10:23.878Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a/root/var/svc/log/oxide-crucible-pantry:default.log
9642023-12-06T05:10:23.878Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
9652023-12-06T05:10:24.977Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1/root/var/svc/log/oxide-cockroachdb:default.log
9662023-12-06T05:10:24.977Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4/root/var/svc/log/oxide-external_dns:default.log (15612 bytes)
9672023-12-06T05:10:25.072Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15612 -> 19392
9682023-12-06T05:10:26.158Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4/root/var/svc/log/oxide-external_dns:default.log
9692023-12-06T05:10:26.158Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80/root/var/svc/log/oxide-nexus:default.log (4799110 bytes)
9702023-12-06T05:10:27.449Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4799110 -> 4866574
9712023-12-06T05:10:28.525Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80/root/var/svc/log/oxide-nexus:default.log
9722023-12-06T05:10:28.525Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d/root/var/svc/log/oxide-crucible-agent:default.log (14122 bytes)
9732023-12-06T05:10:29.644Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d/root/var/svc/log/oxide-crucible-agent:default.log
9742023-12-06T05:10:29.644Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9752023-12-06T05:10:32.114Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d/root/var/svc/log/oxide-crucible-downstairs:default.log
9762023-12-06T05:10:32.114Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d/root/var/svc/log/oxide-crucible-downstairs:downstairs-5631cff5-14d2-43d5-b7d8-4be9dbc854b8.log (5535 bytes)
9772023-12-06T05:10:33.270Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d/root/var/svc/log/oxide-crucible-downstairs:downstairs-5631cff5-14d2-43d5-b7d8-4be9dbc854b8.log
9782023-12-06T05:10:33.270Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a/root/var/svc/log/oxide-crucible-agent:default.log (24366 bytes)
9792023-12-06T05:10:34.411Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a/root/var/svc/log/oxide-crucible-agent:default.log
9802023-12-06T05:10:34.411Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9812023-12-06T05:10:35.508Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a/root/var/svc/log/oxide-crucible-downstairs:default.log
9822023-12-06T05:10:35.508Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a/root/var/svc/log/oxide-crucible-downstairs:downstairs-9b6a58a0-215c-4117-a6e0-f5551ca931c3.log (5533 bytes)
9832023-12-06T05:10:36.606Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a/root/var/svc/log/oxide-crucible-downstairs:downstairs-9b6a58a0-215c-4117-a6e0-f5551ca931c3.log
9842023-12-06T05:10:36.606Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a/root/var/svc/log/oxide-crucible-downstairs:downstairs-db315263-1e4b-4db9-8103-4b0c40456866.log (3048 bytes)
9852023-12-06T05:10:37.703Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a/root/var/svc/log/oxide-crucible-downstairs:downstairs-db315263-1e4b-4db9-8103-4b0c40456866.log
9862023-12-06T05:10:37.703Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9872023-12-06T05:10:38.808Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a/root/var/svc/log/oxide-crucible-agent:default.log
9882023-12-06T05:10:38.808Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9892023-12-06T05:10:39.905Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a/root/var/svc/log/oxide-crucible-downstairs:default.log
9902023-12-06T05:10:39.905Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
9912023-12-06T05:10:41.029Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e/root/var/svc/log/oxide-crucible-agent:default.log
9922023-12-06T05:10:41.029Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9932023-12-06T05:10:42.125Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e/root/var/svc/log/oxide-crucible-downstairs:default.log
9942023-12-06T05:10:42.125Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e/root/var/svc/log/oxide-crucible-downstairs:downstairs-401d694a-ccb7-4877-aed6-c125ad09a3f8.log (5535 bytes)
9952023-12-06T05:10:43.251Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e/root/var/svc/log/oxide-crucible-downstairs:downstairs-401d694a-ccb7-4877-aed6-c125ad09a3f8.log
9962023-12-06T05:10:43.251Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9972023-12-06T05:10:44.355Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389/root/var/svc/log/oxide-crucible-agent:default.log
9982023-12-06T05:10:44.355Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9992023-12-06T05:10:45.451Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389/root/var/svc/log/oxide-crucible-downstairs:default.log
10002023-12-06T05:10:45.451Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894/root/var/svc/log/oxide-internal_dns:default.log (103535 bytes)
10012023-12-06T05:10:45.780Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 103535 -> 107291
10022023-12-06T05:10:46.849Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894/root/var/svc/log/oxide-internal_dns:default.log
10032023-12-06T05:10:46.849Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef/root/var/log/chrony/measurements.log (14385 bytes)
10042023-12-06T05:10:46.939Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef/root/var/log/chrony/measurements.log" changed size mid upload: 14385 -> 16714
10052023-12-06T05:10:48.024Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef/root/var/log/chrony/measurements.log
10062023-12-06T05:10:48.024Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef/root/var/log/chrony/statistics.log (8658 bytes)
10072023-12-06T05:10:48.093Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef/root/var/log/chrony/statistics.log" changed size mid upload: 8658 -> 10296
10082023-12-06T05:10:49.165Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef/root/var/log/chrony/statistics.log
10092023-12-06T05:10:49.165Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef/root/var/log/chrony/tracking.log (1981 bytes)
10102023-12-06T05:10:49.209Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef/root/var/log/chrony/tracking.log" changed size mid upload: 1981 -> 2113
10112023-12-06T05:10:50.278Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef/root/var/log/chrony/tracking.log