01HMZB3XRK2A0665178736PTNS: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.28",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HMZB4CQFGBBGSWH6W9SC4EP1

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12024-01-25T04:32:48.960Zjob dependencies complete; ready to run (waiting for 54 m 46 s)
22024-01-25T04:33:44.000Zjob assigned to worker 01HMZE8WTP82FW0Q7QH9SPMXBZ (queued for 55 s)
32024-01-25T04:33:44.140Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42024-01-25T04:33:44.202Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52024-01-25T04:33:44.255Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62024-01-25T04:33:44.302Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72024-01-25T04:33:44.348Zhttp://172.20.0.5:9969/postboot/buskin... ok
82024-01-25T04:33:44.449ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92024-01-25T04:33:44.503Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102024-01-25T04:33:44.556ZOxide Helios Version helios-2.0.22418 64-bit
112024-01-25T04:33:44.603ZConfiguring devices.
122024-01-25T04:33:44.656Z##################################################
132024-01-25T04:33:44.703Z#### ############# ##
142024-01-25T04:33:44.756Z### ### ############ ##
152024-01-25T04:33:44.804Z## ### # ## ### ## Oxide Computer Company ##
162024-01-25T04:33:44.857Z## ## ## ### # ### ##
172024-01-25T04:33:44.912Z## # ### #### #### This Station Under ##
182024-01-25T04:33:44.959Z### ### #### # ### Computer Control ##
192024-01-25T04:33:45.037Z#### #### ### ## ##
202024-01-25T04:33:45.108Z##################################################
212024-01-25T04:33:45.159ZHostname: buskin
222024-01-25T04:33:45.212ZLoading smf(7) service descriptions:
232024-01-25T04:33:46.421Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-bb8a8abc67437a2f.gz
242024-01-25T04:33:48.638Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-bb8a8abc67437a2f.gz
252024-01-25T04:33:48.639Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262024-01-25T04:34:20.790Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272024-01-25T04:34:20.790Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-c4605ef041dbbb6b.gz
282024-01-25T04:34:59.561Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-c4605ef041dbbb6b.gz
292024-01-25T04:34:59.561Zdownloading input: /input/ci-tools/work/caboose-util.gz
302024-01-25T04:35:00.847Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312024-01-25T04:35:00.847Zdownloading input: /input/ci-tools/work/tufaceous.gz
322024-01-25T04:35:05.787Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332024-01-25T04:35:05.827Zdownloading input: /input/package/work/version.txt
342024-01-25T04:35:05.868Zdownloaded input: /input/package/work/version.txt
352024-01-25T04:35:05.868Zdownloading input: /input/package/work/package.tar.gz
362024-01-25T04:35:26.948Zdownloaded input: /input/package/work/package.tar.gz
372024-01-25T04:35:26.948Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382024-01-25T04:35:58.275Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392024-01-25T04:35:58.275Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402024-01-25T04:36:07.516Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412024-01-25T04:36:07.554Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422024-01-25T04:37:03.432Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432024-01-25T04:37:03.432Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442024-01-25T04:37:59.977Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452024-01-25T04:37:59.978Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462024-01-25T04:38:38.255Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472024-01-25T04:38:38.255Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482024-01-25T04:38:42.928Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492024-01-25T04:38:42.928Zdownloading input: /input/package/work/zones/crucible.tar.gz
502024-01-25T04:38:51.604Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512024-01-25T04:38:51.604Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522024-01-25T04:38:58.819Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532024-01-25T04:38:58.820Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542024-01-25T04:39:05.754Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552024-01-25T04:39:05.754Zdownloading input: /input/package/work/zones/ntp.tar.gz
562024-01-25T04:39:05.840Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572024-01-25T04:39:05.879Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582024-01-25T04:39:11.105Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592024-01-25T04:39:11.105Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602024-01-25T04:39:16.368Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612024-01-25T04:39:16.368Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622024-01-25T04:39:40.964Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632024-01-25T04:39:40.964Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642024-01-25T04:40:05.362Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652024-01-25T04:40:05.362Zdownloading input: /input/package/work/zones/overlay.tar.gz
662024-01-25T04:40:05.447Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672024-01-25T04:40:05.448Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682024-01-25T04:40:17.796Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692024-01-25T04:40:17.797Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702024-01-25T04:40:39.706Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712024-01-25T04:40:39.706Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722024-01-25T04:42:07.662Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732024-01-25T04:42:07.662Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742024-01-25T04:43:10.785Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752024-01-25T04:43:10.785Zstarting task 0: "setup"
762024-01-25T04:43:10.825Z++ uname -s
772024-01-25T04:43:10.825Z+ kern=SunOS
782024-01-25T04:43:10.825Z+ case "$kern" in
792024-01-25T04:43:10.825Z+ groupadd -g 12345 build
802024-01-25T04:43:10.825Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812024-01-25T04:43:12.803Z+ zfs create -o mountpoint=/work rpool/work
822024-01-25T04:43:12.843Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832024-01-25T04:43:12.843Z+ home_fs=zfs
842024-01-25T04:43:12.843Z+ [[ zfs == autofs ]]
852024-01-25T04:43:12.843Z+ mkdir -p /home/build
862024-01-25T04:43:12.843Z+ chown build:build /home/build /work
872024-01-25T04:43:14.846Z+ chmod 0700 /home/build /work
882024-01-25T04:43:14.886Zprocess exited: duration 4064 ms, exit code 0
 
892024-01-25T04:43:14.966Zstarting task 1: "authentication"
902024-01-25T04:43:15.025Zprocess exited: duration 58 ms, exit code 0
 
912024-01-25T04:43:15.103Zstarting task 2: "build"
922024-01-25T04:43:15.142Z+ trap _exit_trap EXIT
932024-01-25T04:43:15.142Z+ set +x
942024-01-25T04:43:53.173Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
952024-01-25T04:43:53.422Zsecpolicy_fs_linkdir: 0x55 = 0x55
962024-01-25T04:43:53.422Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
972024-01-25T04:43:53.422Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
982024-01-25T04:43:53.422Zsecpolicy_fs_linkdir+3: 0 = 0xe5
992024-01-25T04:43:53.462Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1002024-01-25T04:43:53.462Zsecpolicy_fs_linkdir+5: 0 = 0x1
1012024-01-25T04:43:53.462Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1022024-01-25T04:43:53.462Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1032024-01-25T04:43:53.462Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1042024-01-25T04:43:53.462Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1052024-01-25T04:43:53.462Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1062024-01-25T04:43:53.462Z+ [[ -d /opt/oxide ]]
1072024-01-25T04:43:53.462Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1082024-01-25T04:43:53.462Z+ pfexec mkdir /opt/oxide
1092024-01-25T04:43:53.463Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1102024-01-25T04:43:53.463Z+ [[ -d /opt/oxide-underneath ]]
1112024-01-25T04:43:53.463Z+ cd /opt/oxide-underneath
1122024-01-25T04:43:53.463Z+ pfexec tar ceEp@/f - .
1132024-01-25T04:43:53.463Z+ cd /opt/oxide
1142024-01-25T04:43:53.463Z+ pfexec tar xveEp@/f -
1152024-01-25T04:43:53.503Zx ., 0 bytes, 0 tape blocks
1162024-01-25T04:43:53.503Zx ./opte, 0 bytes, 0 tape blocks
1172024-01-25T04:43:53.503Zx ./opte/bin, 0 bytes, 0 tape blocks
1182024-01-25T04:43:53.503Zx ./opte/bin/opteadm, 51799224 bytes, 101171 tape blocks
1192024-01-25T04:43:53.503Z+ pfexec rm -rf /opt/oxide-underneath
1202024-01-25T04:43:53.503Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1212024-01-25T04:43:53.543Z+ pfexec mkdir /opt/oxide/work
1222024-01-25T04:43:53.543Z+ pfexec chown build:build /opt/oxide/work
1232024-01-25T04:43:53.543Z+ cd /opt/oxide/work
1242024-01-25T04:43:53.543Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1252024-01-25T04:43:55.176ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1262024-01-25T04:43:55.176Zx out/mg-ddm-gz.tar, 62812160 bytes, 122680 tape blocks
1272024-01-25T04:43:55.176Zx out/omicron-sled-agent.tar, 121153024 bytes, 236627 tape blocks
1282024-01-25T04:43:55.176Zx out/oxlog.tar, 8382464 bytes, 16372 tape blocks
1292024-01-25T04:43:55.176Zx out/target/test, 75 bytes, 1 tape blocks
1302024-01-25T04:43:55.176Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1312024-01-25T04:43:55.176Zx package-manifest.toml, 23482 bytes, 46 tape blocks
1322024-01-25T04:43:55.176Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1332024-01-25T04:43:55.176Zx target/release/omicron-package, 44605632 bytes, 87121 tape blocks
1342024-01-25T04:43:55.176Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1352024-01-25T04:43:55.176Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1362024-01-25T04:43:55.176Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1372024-01-25T04:43:55.176Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1382024-01-25T04:43:55.176Zx tools/scrimlet/softnpu-init.sh, 2149 bytes, 5 tape blocks
1392024-01-25T04:43:55.176Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1402024-01-25T04:43:55.177Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1412024-01-25T04:43:55.216Z
1422024-01-25T04:43:55.216Zreal 1.649329305
1432024-01-25T04:43:55.216Zuser 1.487612697
1442024-01-25T04:43:55.217Zsys 0.283871107
1452024-01-25T04:43:55.217Ztrap 0.000070261
1462024-01-25T04:43:55.217Ztflt 0.000000000
1472024-01-25T04:43:55.217Zdflt 0.000000000
1482024-01-25T04:43:55.217Zkflt 0.000000000
1492024-01-25T04:43:55.217Zlock 0.000000000
1502024-01-25T04:43:55.217Zslp 1.486915492
1512024-01-25T04:43:55.217Zlat 0.038341863
1522024-01-25T04:43:55.217Zstop 0.000124293
1532024-01-25T04:43:55.217Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1542024-01-25T04:43:56.368Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1552024-01-25T04:43:56.368Z+ mkdir tests
1562024-01-25T04:43:56.368Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1572024-01-25T04:43:56.368Z+ ptime -m gunzip
1582024-01-25T04:43:56.368Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-bb8a8abc67437a2f
1592024-01-25T04:43:56.500Z
1602024-01-25T04:43:56.541Zreal 0.139429326
1612024-01-25T04:43:56.541Zuser 0.126904748
1622024-01-25T04:43:56.541Zsys 0.012273612
1632024-01-25T04:43:56.541Ztrap 0.000011910
1642024-01-25T04:43:56.541Ztflt 0.000000000
1652024-01-25T04:43:56.541Zdflt 0.000000000
1662024-01-25T04:43:56.541Zkflt 0.000000000
1672024-01-25T04:43:56.541Zlock 0.000000000
1682024-01-25T04:43:56.541Zslp 0.000021310
1692024-01-25T04:43:56.541Zlat 0.000022671
1702024-01-25T04:43:56.541Zstop 0.000131152
1712024-01-25T04:43:56.541Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-bb8a8abc67437a2f
1722024-01-25T04:43:56.541Z+ chmod a+x tests/bootstrap-bb8a8abc67437a2f
1732024-01-25T04:43:56.541Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1742024-01-25T04:43:56.541Z+ ptime -m gunzip
1752024-01-25T04:43:56.541Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1762024-01-25T04:43:58.930Z
1772024-01-25T04:43:58.969Zreal 2.419358047
1782024-01-25T04:43:58.969Zuser 2.261990102
1792024-01-25T04:43:58.969Zsys 0.156964798
1802024-01-25T04:43:58.970Ztrap 0.000176472
1812024-01-25T04:43:58.970Ztflt 0.000000000
1822024-01-25T04:43:58.970Zdflt 0.000000000
1832024-01-25T04:43:58.970Zkflt 0.000000000
1842024-01-25T04:43:58.970Zlock 0.000000000
1852024-01-25T04:43:58.970Zslp 0.000012600
1862024-01-25T04:43:58.970Zlat 0.000031870
1872024-01-25T04:43:58.970Zstop 0.000130053
1882024-01-25T04:43:58.970Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1892024-01-25T04:43:58.970Z+ chmod a+x tests/bootstrap
1902024-01-25T04:43:58.970Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1912024-01-25T04:43:58.970Z+ ptime -m gunzip
1922024-01-25T04:43:58.970Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-c4605ef041dbbb6b
1932024-01-25T04:44:01.416Z
1942024-01-25T04:44:01.457Zreal 2.475053199
1952024-01-25T04:44:01.457Zuser 2.329223011
1962024-01-25T04:44:01.457Zsys 0.145412119
1972024-01-25T04:44:01.457Ztrap 0.000177043
1982024-01-25T04:44:01.457Ztflt 0.000000000
1992024-01-25T04:44:01.457Zdflt 0.000000000
2002024-01-25T04:44:01.457Zkflt 0.000000000
2012024-01-25T04:44:01.457Zlock 0.000000000
2022024-01-25T04:44:01.457Zslp 0.000024690
2032024-01-25T04:44:01.457Zlat 0.000023251
2042024-01-25T04:44:01.458Zstop 0.000127382
2052024-01-25T04:44:01.458Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-c4605ef041dbbb6b
2062024-01-25T04:44:01.458Z+ chmod a+x tests/end_to_end_tests-c4605ef041dbbb6b
2072024-01-25T04:44:01.458Z+ bmat address ls
2082024-01-25T04:44:01.458ZNAME CIDR FIRST LAST COUNT
2092024-01-25T04:44:01.458Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2102024-01-25T04:44:01.458Z++ bmat address ls -f extra -Ho first,last,count
2112024-01-25T04:44:01.458Z+ set -- 10.151.1.164 10.151.1.227 64
2122024-01-25T04:44:01.458Z+ EXTRA_IP_START=10.151.1.164
2132024-01-25T04:44:01.458Z+ EXTRA_IP_END=10.151.1.227
2142024-01-25T04:44:01.458Z+ EXTRA_IP_COUNT=64
2152024-01-25T04:44:01.458Z+ (( EXTRA_IP_COUNT >= 32 ))
2162024-01-25T04:44:01.458Z+ EXTRA_IP_BASE=10.151.1
2172024-01-25T04:44:01.458Z+ EXTRA_IP_FOCTET=164
2182024-01-25T04:44:01.458Z+ EXTRA_IP_LOCTET=227
2192024-01-25T04:44:01.458Z+ SERVICE_IP_POOL_START=10.151.1.164
2202024-01-25T04:44:01.458Z+ SERVICE_IP_POOL_END=10.151.1.173
2212024-01-25T04:44:01.458Z+ DNS_IP1=10.151.1.164
2222024-01-25T04:44:01.458Z+ DNS_IP2=10.151.1.165
2232024-01-25T04:44:01.458Z+ UPLINK_IP=10.151.1.174
2242024-01-25T04:44:01.458Z+ IPPOOL_START=10.151.1.175
2252024-01-25T04:44:01.458Z+ IPPOOL_END=10.151.1.227
2262024-01-25T04:44:01.458Z++ ipadm show-addr -po type,addr
2272024-01-25T04:44:01.459Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2282024-01-25T04:44:01.459Z+ GATEWAY_IP=10.151.1.100
2292024-01-25T04:44:01.459Z+ [[ -n 10.151.1.100 ]]
2302024-01-25T04:44:01.459Z+ ping -s 10.151.1.100 56 1
2312024-01-25T04:44:01.459ZPING 10.151.1.100: 56 data bytes
2322024-01-25T04:44:01.498Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
2332024-01-25T04:44:01.499Z
2342024-01-25T04:44:01.499Z----10.151.1.100 PING Statistics----
2352024-01-25T04:44:01.499Z1 packets transmitted, 1 packets received, 0% packet loss
2362024-01-25T04:44:01.499Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2372024-01-25T04:44:01.499Z++ arp -an
2382024-01-25T04:44:01.499Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2392024-01-25T04:44:01.499Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2402024-01-25T04:44:01.499Z+ routeadm -e ipv4-forwarding -u
2412024-01-25T04:44:01.580Z+ PXA_START=10.151.1.164
2422024-01-25T04:44:01.580Z+ PXA_END=10.151.1.227
2432024-01-25T04:44:01.580Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2442024-01-25T04:44:01.622Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2452024-01-25T04:44:01.782Z+ ZPOOL_VDEV_DIR=/scratch
2462024-01-25T04:44:01.782Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2472024-01-25T04:44:01.822Z+++ dirname ./tools/create_virtual_hardware.sh
2482024-01-25T04:44:01.822Z++ cd ./tools
2492024-01-25T04:44:01.822Z++ pwd
2502024-01-25T04:44:01.823Z+ SOURCE_DIR=/opt/oxide/work/tools
2512024-01-25T04:44:01.823Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2522024-01-25T04:44:01.823Z+ SOFTNPU_MODE=zone
2532024-01-25T04:44:01.823Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2542024-01-25T04:44:01.823Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2552024-01-25T04:44:01.823Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2562024-01-25T04:44:01.823Z++ dladm show-phys -p -o LINK
2572024-01-25T04:44:01.823Z++ head -1
2582024-01-25T04:44:01.823Z+ PHYSICAL_LINK=igb0
2592024-01-25T04:44:01.823Z+ echo 'Using igb0 as physical link'
2602024-01-25T04:44:01.823ZUsing igb0 as physical link
2612024-01-25T04:44:01.823Z+ ensure_run_as_root
2622024-01-25T04:44:01.823Z++ id -u
2632024-01-25T04:44:01.823Z+ [[ 0 -ne 0 ]]
2642024-01-25T04:44:01.823Z+ ensure_zpools
2652024-01-25T04:44:01.823Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2662024-01-25T04:44:01.823Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2672024-01-25T04:44:01.823Z+ readarray -t ZPOOLS
2682024-01-25T04:44:01.823Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2692024-01-25T04:44:01.823Z++ sed 's/[ ",]//g'
2702024-01-25T04:44:01.824Z+ for ZPOOL in "${ZPOOLS[@]}"
2712024-01-25T04:44:01.824Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2722024-01-25T04:44:01.824ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2732024-01-25T04:44:01.824Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2742024-01-25T04:44:01.824Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2752024-01-25T04:44:01.825Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2762024-01-25T04:44:01.825Z0+0 records in
2772024-01-25T04:44:01.825Z0+0 records out
2782024-01-25T04:44:01.825Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2792024-01-25T04:44:01.825Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2802024-01-25T04:44:01.825Z+ set +x
2812024-01-25T04:44:01.825ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2822024-01-25T04:44:01.825Z++ zpool list -o name
2832024-01-25T04:44:01.825Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2842024-01-25T04:44:01.865Z+ [[ -z '' ]]
2852024-01-25T04:44:01.865Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2862024-01-25T04:44:01.865Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2872024-01-25T04:44:01.865Z+ set +x
2882024-01-25T04:44:01.865ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2892024-01-25T04:44:01.865Z+ for ZPOOL in "${ZPOOLS[@]}"
2902024-01-25T04:44:01.865Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2912024-01-25T04:44:01.866ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2922024-01-25T04:44:01.866Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2932024-01-25T04:44:01.866Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2942024-01-25T04:44:01.866Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2952024-01-25T04:44:01.881Z0+0 records in
2962024-01-25T04:44:01.881Z0+0 records out
2972024-01-25T04:44:01.881Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2982024-01-25T04:44:01.881Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2992024-01-25T04:44:01.881Z+ set +x
3002024-01-25T04:44:01.881ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3012024-01-25T04:44:01.922Z++ zpool list -o name
3022024-01-25T04:44:01.922Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3032024-01-25T04:44:01.922Z+ [[ -z '' ]]
3042024-01-25T04:44:01.922Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3052024-01-25T04:44:01.922Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3062024-01-25T04:44:01.922Z+ set +x
3072024-01-25T04:44:01.922ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3082024-01-25T04:44:01.922Z+ for ZPOOL in "${ZPOOLS[@]}"
3092024-01-25T04:44:01.922Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3102024-01-25T04:44:01.922ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3112024-01-25T04:44:01.922Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3122024-01-25T04:44:01.922Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3132024-01-25T04:44:01.922Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3142024-01-25T04:44:01.922Z0+0 records in
3152024-01-25T04:44:01.922Z0+0 records out
3162024-01-25T04:44:01.922Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3172024-01-25T04:44:01.922Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3182024-01-25T04:44:01.922Z+ set +x
3192024-01-25T04:44:01.922ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3202024-01-25T04:44:01.923Z++ zpool list -o name
3212024-01-25T04:44:01.923Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3222024-01-25T04:44:01.923Z+ [[ -z '' ]]
3232024-01-25T04:44:01.923Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3242024-01-25T04:44:01.964Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3252024-01-25T04:44:01.964Z+ set +x
3262024-01-25T04:44:01.964ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3272024-01-25T04:44:01.964ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282024-01-25T04:44:01.964Z+ for ZPOOL in "${ZPOOLS[@]}"
3292024-01-25T04:44:01.964Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3302024-01-25T04:44:01.965Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3312024-01-25T04:44:01.965Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3322024-01-25T04:44:01.965Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3332024-01-25T04:44:01.965Z0+0 records in
3342024-01-25T04:44:01.965Z0+0 records out
3352024-01-25T04:44:01.965Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3362024-01-25T04:44:01.965Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3372024-01-25T04:44:01.965Z+ set +x
3382024-01-25T04:44:01.965ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3392024-01-25T04:44:01.965Z++ zpool list -o name
3402024-01-25T04:44:01.965Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3412024-01-25T04:44:01.965Z+ [[ -z '' ]]
3422024-01-25T04:44:01.965Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3432024-01-25T04:44:02.011Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3442024-01-25T04:44:02.011Z+ set +x
3452024-01-25T04:44:02.011ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3462024-01-25T04:44:02.011Z+ for ZPOOL in "${ZPOOLS[@]}"
3472024-01-25T04:44:02.011Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3482024-01-25T04:44:02.011Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3492024-01-25T04:44:02.011Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3502024-01-25T04:44:02.011ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3512024-01-25T04:44:02.011Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3522024-01-25T04:44:02.011Z0+0 records in
3532024-01-25T04:44:02.011Z0+0 records out
3542024-01-25T04:44:02.011Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3552024-01-25T04:44:02.011Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3562024-01-25T04:44:02.011Z+ set +x
3572024-01-25T04:44:02.011ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3582024-01-25T04:44:02.011Z++ zpool list -o name
3592024-01-25T04:44:02.011Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3602024-01-25T04:44:02.012Z+ [[ -z '' ]]
3612024-01-25T04:44:02.012Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3622024-01-25T04:44:02.062Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3632024-01-25T04:44:02.062Z+ set +x
3642024-01-25T04:44:02.062ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3652024-01-25T04:44:02.062Z+ for ZPOOL in "${ZPOOLS[@]}"
3662024-01-25T04:44:02.062Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3672024-01-25T04:44:02.062ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3682024-01-25T04:44:02.062Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3692024-01-25T04:44:02.062Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3702024-01-25T04:44:02.062Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3712024-01-25T04:44:02.062Z0+0 records in
3722024-01-25T04:44:02.062Z0+0 records out
3732024-01-25T04:44:02.062Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3742024-01-25T04:44:02.062Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3752024-01-25T04:44:02.062Z+ set +x
3762024-01-25T04:44:02.063ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3772024-01-25T04:44:02.063Z++ zpool list -o name
3782024-01-25T04:44:02.063Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3792024-01-25T04:44:02.063Z+ [[ -z '' ]]
3802024-01-25T04:44:02.063Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3812024-01-25T04:44:02.105Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3822024-01-25T04:44:02.105Z+ set +x
3832024-01-25T04:44:02.105ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3842024-01-25T04:44:02.105Z+ for ZPOOL in "${ZPOOLS[@]}"
3852024-01-25T04:44:02.105ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3862024-01-25T04:44:02.105Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3872024-01-25T04:44:02.105Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3882024-01-25T04:44:02.105Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3892024-01-25T04:44:02.105Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3902024-01-25T04:44:02.105Z0+0 records in
3912024-01-25T04:44:02.106Z0+0 records out
3922024-01-25T04:44:02.106Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3932024-01-25T04:44:02.107Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3942024-01-25T04:44:02.107Z+ set +x
3952024-01-25T04:44:02.107ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3962024-01-25T04:44:02.107Z++ zpool list -o name
3972024-01-25T04:44:02.107Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3982024-01-25T04:44:02.107Z+ [[ -z '' ]]
3992024-01-25T04:44:02.107Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4002024-01-25T04:44:02.147Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4012024-01-25T04:44:02.147Z+ set +x
4022024-01-25T04:44:02.147ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4032024-01-25T04:44:02.148Z+ for ZPOOL in "${ZPOOLS[@]}"
4042024-01-25T04:44:02.148Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4052024-01-25T04:44:02.148ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4062024-01-25T04:44:02.148Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4072024-01-25T04:44:02.148Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4082024-01-25T04:44:02.148Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4092024-01-25T04:44:02.148Z0+0 records in
4102024-01-25T04:44:02.148Z0+0 records out
4112024-01-25T04:44:02.148Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4122024-01-25T04:44:02.148Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4132024-01-25T04:44:02.148Z+ set +x
4142024-01-25T04:44:02.148ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4152024-01-25T04:44:02.148Z++ zpool list -o name
4162024-01-25T04:44:02.148Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4172024-01-25T04:44:02.148Z+ [[ -z '' ]]
4182024-01-25T04:44:02.148Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192024-01-25T04:44:02.190Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4202024-01-25T04:44:02.190Z+ set +x
4212024-01-25T04:44:02.190ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4222024-01-25T04:44:02.191Z+ for ZPOOL in "${ZPOOLS[@]}"
4232024-01-25T04:44:02.191Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4242024-01-25T04:44:02.191ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4252024-01-25T04:44:02.191Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4262024-01-25T04:44:02.191Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4272024-01-25T04:44:02.191Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4282024-01-25T04:44:02.191Z0+0 records in
4292024-01-25T04:44:02.191Z0+0 records out
4302024-01-25T04:44:02.191Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4312024-01-25T04:44:02.191Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4322024-01-25T04:44:02.191Z+ set +x
4332024-01-25T04:44:02.191ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4342024-01-25T04:44:02.191Z++ zpool list -o name
4352024-01-25T04:44:02.191Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4362024-01-25T04:44:02.191Z+ [[ -z '' ]]
4372024-01-25T04:44:02.191Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4382024-01-25T04:44:02.232Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4392024-01-25T04:44:02.232Z+ set +x
4402024-01-25T04:44:02.232ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4412024-01-25T04:44:02.232Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4422024-01-25T04:44:02.236Z+ readarray -t ZPOOLS
4432024-01-25T04:44:02.236Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4442024-01-25T04:44:02.236Z++ sed 's/[ ",]//g'
4452024-01-25T04:44:02.236Z+ for ZPOOL in "${ZPOOLS[@]}"
4462024-01-25T04:44:02.236Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4472024-01-25T04:44:02.236ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4482024-01-25T04:44:02.236Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4492024-01-25T04:44:02.236Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4502024-01-25T04:44:02.236Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4512024-01-25T04:44:02.236Z0+0 records in
4522024-01-25T04:44:02.237Z0+0 records out
4532024-01-25T04:44:02.237Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4542024-01-25T04:44:02.237Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4552024-01-25T04:44:02.237Z+ set +x
4562024-01-25T04:44:02.237ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4572024-01-25T04:44:02.237Z++ zpool list -o name
4582024-01-25T04:44:02.237Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4592024-01-25T04:44:02.237Z+ [[ -z '' ]]
4602024-01-25T04:44:02.237Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4612024-01-25T04:44:02.278Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4622024-01-25T04:44:02.278Z+ set +x
4632024-01-25T04:44:02.278ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4642024-01-25T04:44:02.278Z+ for ZPOOL in "${ZPOOLS[@]}"
4652024-01-25T04:44:02.278Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4662024-01-25T04:44:02.279ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4672024-01-25T04:44:02.298Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4682024-01-25T04:44:02.298Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4692024-01-25T04:44:02.298Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4702024-01-25T04:44:02.298Z0+0 records in
4712024-01-25T04:44:02.298Z0+0 records out
4722024-01-25T04:44:02.298Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4732024-01-25T04:44:02.298Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4742024-01-25T04:44:02.298Z+ set +x
4752024-01-25T04:44:02.298ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4762024-01-25T04:44:02.298Z++ zpool list -o name
4772024-01-25T04:44:02.298Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4782024-01-25T04:44:02.298Z+ [[ -z '' ]]
4792024-01-25T04:44:02.298Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4802024-01-25T04:44:02.348Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4812024-01-25T04:44:02.348Z+ set +x
4822024-01-25T04:44:02.348ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4832024-01-25T04:44:02.348Z+ [[ zone == \z\o\n\e ]]
4842024-01-25T04:44:02.348Z+ ensure_simulated_links igb0
4852024-01-25T04:44:02.348Z+ local PHYSICAL_LINK=igb0
4862024-01-25T04:44:02.348Z+ INDICES=("0" "1")
4872024-01-25T04:44:02.349Z+ for I in "${INDICES[@]}"
4882024-01-25T04:44:02.349Z++ get_simnet_name_if_exists net0
4892024-01-25T04:44:02.349Z++ dladm show-simnet -p -o LINK net0
4902024-01-25T04:44:02.349Z++ echo ''
4912024-01-25T04:44:02.349Z+ [[ -z '' ]]
4922024-01-25T04:44:02.349Z+ dladm create-simnet -t net0
4932024-01-25T04:44:02.349Z+ dladm create-simnet -t sc0_0
4942024-01-25T04:44:02.349Z+ dladm modify-simnet -t -p net0 sc0_0
4952024-01-25T04:44:02.349Z+ dladm set-linkprop -p mtu=9000 sc0_0
4962024-01-25T04:44:02.349Z+ success 'Simnet net0/sc0_0 exists'
4972024-01-25T04:44:02.349Z+ set +x
4982024-01-25T04:44:02.349ZSimnet net0/sc0_0 exists
4992024-01-25T04:44:02.349Z+ for I in "${INDICES[@]}"
5002024-01-25T04:44:02.349Z++ get_simnet_name_if_exists net1
5012024-01-25T04:44:02.349Z++ dladm show-simnet -p -o LINK net1
5022024-01-25T04:44:02.349Z++ echo ''
5032024-01-25T04:44:02.349Z+ [[ -z '' ]]
5042024-01-25T04:44:02.349Z+ dladm create-simnet -t net1
5052024-01-25T04:44:02.349Z+ dladm create-simnet -t sc1_0
5062024-01-25T04:44:02.350Z+ dladm modify-simnet -t -p net1 sc1_0
5072024-01-25T04:44:02.350Z+ dladm set-linkprop -p mtu=9000 sc1_0
5082024-01-25T04:44:02.350Z+ success 'Simnet net1/sc1_0 exists'
5092024-01-25T04:44:02.350Z+ set +x
5102024-01-25T04:44:02.350ZSimnet net1/sc1_0 exists
5112024-01-25T04:44:02.350Z++ get_vnic_name_if_exists sc0_1
5122024-01-25T04:44:02.350Z++ dladm show-vnic -p -o LINK sc0_1
5132024-01-25T04:44:02.350Z++ echo ''
5142024-01-25T04:44:02.350Z+ [[ -z '' ]]
5152024-01-25T04:44:02.350Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162024-01-25T04:44:02.390Z+ [[ -v PROMISC_FILT_OFF ]]
5172024-01-25T04:44:02.391Z+ success 'Vnic sc0_1 exists'
5182024-01-25T04:44:02.391Z+ set +x
5192024-01-25T04:44:02.391ZVnic sc0_1 exists
5202024-01-25T04:44:02.391Z+ warn_if_no_proxy_arp
5212024-01-25T04:44:02.391Z+ [[ -v PXA_START ]]
5222024-01-25T04:44:02.391Z+ [[ -v PXA_END ]]
5232024-01-25T04:44:02.391Z+ ensure_softnpu_zone
5242024-01-25T04:44:02.391Z+ zoneadm list
5252024-01-25T04:44:02.391Z+ grep -q sidecar_softnpu
5262024-01-25T04:44:02.391Z+ [[ -f out/npuzone/npuzone ]]
5272024-01-25T04:44:02.391Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit e3ea4b495ba0a71801ded0776ae4bbd31df57e26 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5282024-01-25T04:44:02.391Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5292024-01-25T04:44:09.983Zdownloading softnpu ...done
5302024-01-25T04:44:10.023Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/libsidecar_lite.so.sha256.txt
5312024-01-25T04:44:17.003Zdownloading asic_program.so ...done
5322024-01-25T04:44:17.045Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/scadm.sha256.txt
5332024-01-25T04:44:24.994Zdownloading scadm ...done
5342024-01-25T04:44:25.034Zcreating port tfportrear0_0 for sc0_0
5352024-01-25T04:44:25.034Zcreating port tfportqsfp0_0 for sc0_1
5362024-01-25T04:44:25.034Zsoftnpu zone setup
5372024-01-25T04:44:25.034Zconfigure zone
5382024-01-25T04:44:25.089Zinstall zone
5392024-01-25T04:44:26.104Zboot zone
5402024-01-25T04:44:29.210Z[sidecar_softnpu] svcs milestone/network
5412024-01-25T04:44:29.250ZSTATE STIME FMRI
5422024-01-25T04:44:29.250Zonline 4:44:27 svc:/milestone/network:default
5432024-01-25T04:44:29.250Z[sidecar_softnpu] /softnpu/npu start
5442024-01-25T04:44:29.294Z
5452024-01-25T04:44:29.294Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5462024-01-25T04:44:29.294Z+ GATEWAY_IP=10.151.1.100
5472024-01-25T04:44:29.294Z+ echo 'Using 10.151.1.100 as gateway ip'
5482024-01-25T04:44:29.294ZUsing 10.151.1.100 as gateway ip
5492024-01-25T04:44:29.294Z+ [[ ! -v GATEWAY_MAC ]]
5502024-01-25T04:44:29.295Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5512024-01-25T04:44:29.295Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5522024-01-25T04:44:29.295Z+++ echo 18:c0:4d:0d:9f:b2
5532024-01-25T04:44:29.295Z+ COUNT=1
5542024-01-25T04:44:29.295Z+ [[ 1 -ne 1 ]]
5552024-01-25T04:44:29.295Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5562024-01-25T04:44:29.295ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5572024-01-25T04:44:29.295Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5582024-01-25T04:44:29.295Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5592024-01-25T04:44:29.338Z+ PXA_MAC=a8:e1:de:01:70:1d
5602024-01-25T04:44:29.339Z+ [[ -v PXA_START ]]
5612024-01-25T04:44:29.339Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5622024-01-25T04:44:29.339Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5632024-01-25T04:44:29.339Z+ z_scadm dump-state
5642024-01-25T04:44:29.339Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5652024-01-25T04:44:29.380Zlocal v6:
5662024-01-25T04:44:29.380Zlocal v4:
5672024-01-25T04:44:29.380Zrouter v6:
5682024-01-25T04:44:29.381Zrouter v4_idx:
5692024-01-25T04:44:29.381Zrouter v4_routes:
5702024-01-25T04:44:29.383Zresolver v4:
5712024-01-25T04:44:29.383Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5722024-01-25T04:44:29.383Zresolver v6:
5732024-01-25T04:44:29.384Znat v4:
5742024-01-25T04:44:29.384Znat_v6:
5752024-01-25T04:44:29.384Zport_mac:
5762024-01-25T04:44:29.384Zproxy arp:
5772024-01-25T04:44:29.384Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5782024-01-25T04:44:29.384Z+ success 'softnpu zone exists'
5792024-01-25T04:44:29.384Z+ set +x
5802024-01-25T04:44:29.384Zsoftnpu zone exists
5812024-01-25T04:44:29.384Z
5822024-01-25T04:44:29.384Zreal 27.578781497
5832024-01-25T04:44:29.384Zuser 0.697092220
5842024-01-25T04:44:29.384Zsys 1.558400426
5852024-01-25T04:44:29.384Ztrap 0.000074188
5862024-01-25T04:44:29.385Ztflt 0.000148187
5872024-01-25T04:44:29.385Zdflt 0.001271645
5882024-01-25T04:44:29.385Zkflt 0.000000000
5892024-01-25T04:44:29.385Zlock 21:29.218377171
5902024-01-25T04:44:29.385Zslp 1:22.501105834
5912024-01-25T04:44:29.385Zlat 0.074951918
5922024-01-25T04:44:29.385Zstop 0.381291481
5932024-01-25T04:44:29.385Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5942024-01-25T04:44:29.439Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952024-01-25T04:44:29.439Z+ SILO_NAME=recovery
5962024-01-25T04:44:29.440Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5972024-01-25T04:44:29.440Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5982024-01-25T04:44:29.440Z+ sed -i~ '
5992024-01-25T04:44:29.440Z /^external_dns_ips/c\
6002024-01-25T04:44:29.440Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6012024-01-25T04:44:29.440Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6022024-01-25T04:44:29.440Z /^first/c\
6032024-01-25T04:44:29.440Zfirst = "10.151.1.164"
6042024-01-25T04:44:29.440Z /^last/c\
6052024-01-25T04:44:29.440Zlast = "10.151.1.173"
6062024-01-25T04:44:29.440Z }
6072024-01-25T04:44:29.440Z /^\[rack_network_config/,/^$/ {
6082024-01-25T04:44:29.441Z /^infra_ip_first/c\
6092024-01-25T04:44:29.441Zinfra_ip_first = "10.151.1.174"
6102024-01-25T04:44:29.441Z /^infra_ip_last/c\
6112024-01-25T04:44:29.441Zinfra_ip_last = "10.151.1.174"
6122024-01-25T04:44:29.441Z }
6132024-01-25T04:44:29.441Z /^\[\[rack_network_config.ports/,/^$/ {
6142024-01-25T04:44:29.441Z /^routes/c\
6152024-01-25T04:44:29.441Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6162024-01-25T04:44:29.441Z /^addresses/c\
6172024-01-25T04:44:29.441Zaddresses = \["10.151.1.174/24"\]
6182024-01-25T04:44:29.441Z }
6192024-01-25T04:44:29.442Z' pkg/config-rss.toml
6202024-01-25T04:44:29.442Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6212024-01-25T04:44:29.442Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6222024-01-25T04:44:29.442Z+++ pkg/config-rss.toml Thu Jan 25 04:44:28 2024
6232024-01-25T04:44:29.442Z@@ -41,7 +41,7 @@
6242024-01-25T04:44:29.442Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6252024-01-25T04:44:29.442Z # must be contained in one of the "internal services" IP Pool ranges listed
6262024-01-25T04:44:29.442Z # below.
6272024-01-25T04:44:29.442Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6282024-01-25T04:44:29.442Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6292024-01-25T04:44:29.442Z
6302024-01-25T04:44:29.442Z # Initial TLS certificates for the external API
6312024-01-25T04:44:29.442Z #
6322024-01-25T04:44:29.442Z@@ -77,8 +77,8 @@
6332024-01-25T04:44:29.443Z #
6342024-01-25T04:44:29.443Z # For more on this and what to put here, see docs/how-to-run.adoc.
6352024-01-25T04:44:29.443Z [[internal_services_ip_pool_ranges]]
6362024-01-25T04:44:29.443Z-first = "192.168.1.20"
6372024-01-25T04:44:29.443Z-last = "192.168.1.29"
6382024-01-25T04:44:29.443Z+first = "10.151.1.164"
6392024-01-25T04:44:29.443Z+last = "10.151.1.173"
6402024-01-25T04:44:29.443Z
6412024-01-25T04:44:29.443Z # TODO - this configuration is subject to change going forward. Ultimately these
6422024-01-25T04:44:29.443Z # parameters should be provided to the control plane via wicket, but we need to
6432024-01-25T04:44:29.444Z@@ -92,8 +92,8 @@
6442024-01-25T04:44:29.444Z # A range of IP addresses used by Boundary Services on the external network. In
6452024-01-25T04:44:29.444Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6462024-01-25T04:44:29.444Z # With softnpu, only one address is used.
6472024-01-25T04:44:29.444Z-infra_ip_first = "192.168.1.30"
6482024-01-25T04:44:29.444Z-infra_ip_last = "192.168.1.30"
6492024-01-25T04:44:29.444Z+infra_ip_first = "10.151.1.174"
6502024-01-25T04:44:29.444Z+infra_ip_last = "10.151.1.174"
6512024-01-25T04:44:29.444Z
6522024-01-25T04:44:29.444Z # Configurations for BGP routers to run on the scrimlets.
6532024-01-25T04:44:29.458Z bgp = []
6542024-01-25T04:44:29.458Z@@ -101,9 +101,9 @@
6552024-01-25T04:44:29.458Z # You can configure multiple uplinks by repeating the following stanza
6562024-01-25T04:44:29.458Z [[rack_network_config.ports]]
6572024-01-25T04:44:29.458Z # Routes associated with this port.
6582024-01-25T04:44:29.458Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6592024-01-25T04:44:29.458Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6602024-01-25T04:44:29.459Z # Addresses associated with this port.
6612024-01-25T04:44:29.459Z-addresses = ["192.168.1.30/24"]
6622024-01-25T04:44:29.459Z+addresses = ["10.151.1.174/24"]
6632024-01-25T04:44:29.460Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6642024-01-25T04:44:29.460Z port = "qsfp0"
6652024-01-25T04:44:29.460Z # The speed of this port.
6662024-01-25T04:44:29.460Z+ true
6672024-01-25T04:44:29.460Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6682024-01-25T04:44:29.460Za pkg/config-rss.toml 7K
6692024-01-25T04:44:29.461Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6702024-01-25T04:44:29.461Z+ TLS_NAME=recovery.sys.oxide.test
6712024-01-25T04:44:29.461Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6722024-01-25T04:44:29.727Z..+..+......+....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+..........+......+..............+....+..+.............+..+...+.+..+............+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+...+..+...+....+...+..+............+.............+.....+.+..+......................+........+...+.+........+.......+......+.................+.......+......+............+........+....+...+.....+...+............+.......+.........+.................+.........+.........+....+...+.....+.........+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6732024-01-25T04:44:30.985Z.........+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+....+...+..+.+..............+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+.......+....................+...+.......+..+......+.............+.........+...+........+.......+............+...........+.......+.................+.+...........+...+...+..........+...+.....+.......+.........+...........+...+......+.........+.+...+...............+..+.+..+...............+..........+.....+.+......+........+.......+.....+..................+...+.........+....+.....+.......+.....+............+...+...+................+......+.................+.+..+.........+........................+.......+...+........................+.....+.......+......+...+............+..+......+.......+.........+...........+.......+......+..............+..........+............+........+.+.....+.+.....+...+.+..+....+......+.....+...............+...+.......+.........+.....+................+..+.+.....+.+...+...+...+.....................+...+..+...................+.........+...+..............+......+.......+......+..+.........+.........+......+.............+..+.+.....+....+......+...+.....+....+...............+......+.........+...+..+...............+.............+.....+....+........+...+....+.....+......+...+......+......+.......+.....+..........+............+......+........+.........+...+.................................+..........+...............+.....+.........+.+..+............+.......+........+.+......+..................+..+......+...+.........+...+............+................+.........+..+...+............+...+.......+...+..+...+......+......+.............+......+..+....+........+...+...+....+.....+...+.......+..+..........+......+...+..+.......+...........+.+.........+...........+.+...+...........+.............+......+............+........................+...+..+.........+.......+........+.......+............+.....+...+..........+........+......+................+...+.....+.........+................+..+...+......+...................+.....+.+........+.+......+......+...............+.........+...........+...+......................+............+..+....+.....+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6742024-01-25T04:44:31.030Z-----
6752024-01-25T04:44:31.030Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6762024-01-25T04:44:31.031Za pkg/initial-tls-cert.pem 2K
6772024-01-25T04:44:31.031Za pkg/initial-tls-key.pem 4K
6782024-01-25T04:44:31.031Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6792024-01-25T04:44:31.031Z+ rmdir pkg
6802024-01-25T04:44:31.031Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6812024-01-25T04:44:31.031Z+ pfexec mkdir -p /usr/oxide
6822024-01-25T04:44:31.031Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6832024-01-25T04:44:31.746Z+ pfexec chmod +x /usr/oxide/oxide
6842024-01-25T04:44:31.789Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6852024-01-25T04:44:49.977Z+ OMICRON_NO_UNINSTALL=1
6862024-01-25T04:44:49.978Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6872024-01-25T04:44:50.545ZJan 25 04:44:49.577 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6882024-01-25T04:44:50.545ZJan 25 04:44:49.579 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6892024-01-25T04:44:50.553ZJan 25 04:44:49.579 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6902024-01-25T04:44:50.553ZJan 25 04:44:49.579 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6912024-01-25T04:44:50.554ZJan 25 04:44:49.579 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6922024-01-25T04:44:50.554ZJan 25 04:44:49.579 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6932024-01-25T04:44:50.554ZJan 25 04:44:49.579 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6942024-01-25T04:44:50.554ZJan 25 04:44:49.579 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6952024-01-25T04:44:50.554ZJan 25 04:44:49.579 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6962024-01-25T04:44:50.555ZJan 25 04:44:49.579 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6972024-01-25T04:44:50.555ZJan 25 04:44:49.580 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6982024-01-25T04:44:50.555ZJan 25 04:44:49.580 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6992024-01-25T04:44:50.555ZJan 25 04:44:49.580 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7002024-01-25T04:44:50.555ZJan 25 04:44:49.580 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7012024-01-25T04:44:50.555ZJan 25 04:44:49.580 INFO Installing service, dst: /opt/oxide/oxlog.tar, src: out/oxlog.tar
7022024-01-25T04:44:50.555ZJan 25 04:44:49.580 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7032024-01-25T04:44:50.555ZJan 25 04:44:49.580 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7042024-01-25T04:44:50.556ZJan 25 04:44:49.580 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7052024-01-25T04:44:50.678ZJan 25 04:44:49.841 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062024-01-25T04:44:50.678ZJan 25 04:44:49.875 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072024-01-25T04:44:50.697ZJan 25 04:44:49.939 INFO Unpacking service tarball, service_path: /opt/oxide/oxlog, tar_path: /opt/oxide/oxlog.tar
7082024-01-25T04:44:50.697ZJan 25 04:44:49.943 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7092024-01-25T04:44:50.697Z
7102024-01-25T04:44:50.697Zreal 0.507610885
7112024-01-25T04:44:50.698Zuser 0.107422527
7122024-01-25T04:44:50.698Zsys 2.099523251
7132024-01-25T04:44:50.698Ztrap 0.000008128
7142024-01-25T04:44:50.698Ztflt 0.000073958
7152024-01-25T04:44:50.698Zdflt 0.001038552
7162024-01-25T04:44:50.698Zkflt 0.000008929
7172024-01-25T04:44:50.698Zlock 45.558748871
7182024-01-25T04:44:50.698Zslp 0.721233390
7192024-01-25T04:44:50.699Zlat 0.003086793
7202024-01-25T04:44:50.699Zstop 0.205406432
7212024-01-25T04:44:50.699Z+ retry=0
7222024-01-25T04:44:50.699Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232024-01-25T04:44:50.700Z+ [[ 0 -gt 30 ]]
7242024-01-25T04:44:50.700Z+ sleep 1
7252024-01-25T04:44:51.509Z+ retry=1
7262024-01-25T04:44:51.509Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272024-01-25T04:44:51.695Z+ [[ 1 -gt 30 ]]
7282024-01-25T04:44:51.695Z+ sleep 1
7292024-01-25T04:44:52.523Z+ retry=2
7302024-01-25T04:44:52.523Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312024-01-25T04:44:52.592Z+ [[ 2 -gt 30 ]]
7322024-01-25T04:44:52.592Z+ sleep 1
7332024-01-25T04:44:53.938Z+ retry=3
7342024-01-25T04:44:53.938Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352024-01-25T04:44:53.982Z+ [[ 3 -gt 30 ]]
7362024-01-25T04:44:53.982Z+ sleep 1
7372024-01-25T04:44:54.582Z+ retry=4
7382024-01-25T04:44:54.582Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7392024-01-25T04:45:19.718Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7402024-01-25T04:45:19.771Zlocal v6:
7412024-01-25T04:45:19.771Zfd1a:a7aa:8c4c:73c::1
7422024-01-25T04:45:19.771Zfe80::e4a8:eeff:fe03:2e25
7432024-01-25T04:45:19.771Zlocal v4:
7442024-01-25T04:45:19.771Z10.151.1.174
7452024-01-25T04:45:19.771Zrouter v6:
7462024-01-25T04:45:19.771Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::a4e3:4bff:fec9:cae7 (1)
7472024-01-25T04:45:19.771Zfd00:1122:3344:1::/64 -> fe80::a4e3:4bff:fec9:cae7 (1)
7482024-01-25T04:45:19.771Zfd00:1122:3344:101::/64 -> fe80::a4e3:4bff:fec9:cae7 (1)
7492024-01-25T04:45:19.772Zfd00:1122:3344:3::/64 -> fe80::a4e3:4bff:fec9:cae7 (1)
7502024-01-25T04:45:19.772Zfd00:1122:3344:2::/64 -> fe80::a4e3:4bff:fec9:cae7 (1)
7512024-01-25T04:45:19.772Zrouter v4_idx:
7522024-01-25T04:45:19.772Z0.0.0.0/0 -> 0
7532024-01-25T04:45:19.772Zrouter v4_routes:
7542024-01-25T04:45:19.772Z0 -> 10.151.1.100 (2)
7552024-01-25T04:45:19.772Zresolver v4:
7562024-01-25T04:45:19.772Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7572024-01-25T04:45:19.772Z10.151.1.174 -> e6:4c:93:20:ff:58
7582024-01-25T04:45:19.772Zresolver v6:
7592024-01-25T04:45:19.772Zfe80::e4a8:eeff:fe03:2e25 -> e6:a8:ee:03:2e:25
7602024-01-25T04:45:19.772Zfe80::a4e3:4bff:fec9:cae7 -> a6:e3:4b:c9:ca:e7
7612024-01-25T04:45:19.772Znat v4:
7622024-01-25T04:45:19.772Z10.151.1.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:bb:21
7632024-01-25T04:45:19.772Znat_v6:
7642024-01-25T04:45:19.772Zport_mac:
7652024-01-25T04:45:19.772Z1000: a8:40:25:0a:43:76
7662024-01-25T04:45:19.772Z1: a8:40:25:0a:43:77
7672024-01-25T04:45:19.772Zproxy arp:
7682024-01-25T04:45:19.772Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7692024-01-25T04:45:19.772Z+ retry=0
7702024-01-25T04:45:19.772Z+++ zoneadm list -n
7712024-01-25T04:45:19.772Z+++ grep oxz_ntp
7722024-01-25T04:45:19.773Z++ pfexec svcs -z oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073 -Hostate oxide/ntp
7732024-01-25T04:45:19.812Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7742024-01-25T04:45:19.812Z++ true
7752024-01-25T04:45:19.812Z+ [[ '' != online ]]
7762024-01-25T04:45:19.812Z+ [[ 0 -gt 60 ]]
7772024-01-25T04:45:19.812Z+ sleep 1
7782024-01-25T04:45:20.778Z+ retry=1
7792024-01-25T04:45:20.832Z+++ zoneadm list -n
7802024-01-25T04:45:20.832Z+++ grep oxz_ntp
7812024-01-25T04:45:20.832Z++ pfexec svcs -z oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073 -Hostate oxide/ntp
7822024-01-25T04:45:20.833Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7832024-01-25T04:45:20.833Z++ true
7842024-01-25T04:45:20.833Z+ [[ '' != online ]]
7852024-01-25T04:45:20.833Z+ [[ 1 -gt 60 ]]
7862024-01-25T04:45:20.833Z+ sleep 1
7872024-01-25T04:45:21.860Z+ retry=2
7882024-01-25T04:45:21.900Z+++ zoneadm list -n
7892024-01-25T04:45:21.901Z+++ grep oxz_ntp
7902024-01-25T04:45:21.901Z++ pfexec svcs -z oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073 -Hostate oxide/ntp
7912024-01-25T04:45:21.901Z+ [[ disabled != online ]]
7922024-01-25T04:45:21.901Z+ [[ 2 -gt 60 ]]
7932024-01-25T04:45:21.901Z+ sleep 1
7942024-01-25T04:45:22.832Z+ retry=3
7952024-01-25T04:45:22.877Z+++ zoneadm list -n
7962024-01-25T04:45:22.877Z+++ grep oxz_ntp
7972024-01-25T04:45:22.877Z++ pfexec svcs -z oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073 -Hostate oxide/ntp
7982024-01-25T04:45:22.877Z+ [[ disabled != online ]]
7992024-01-25T04:45:22.877Z+ [[ 3 -gt 60 ]]
8002024-01-25T04:45:22.877Z+ sleep 1
8012024-01-25T04:45:23.855Z+ retry=4
8022024-01-25T04:45:23.897Z+++ zoneadm list -n
8032024-01-25T04:45:23.897Z+++ grep oxz_ntp
8042024-01-25T04:45:23.897Z++ pfexec svcs -z oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073 -Hostate oxide/ntp
8052024-01-25T04:45:23.897Z+ [[ disabled != online ]]
8062024-01-25T04:45:23.898Z+ [[ 4 -gt 60 ]]
8072024-01-25T04:45:23.898Z+ sleep 1
8082024-01-25T04:45:24.876Z+ retry=5
8092024-01-25T04:45:24.940Z+++ zoneadm list -n
8102024-01-25T04:45:24.940Z+++ grep oxz_ntp
8112024-01-25T04:45:24.940Z++ pfexec svcs -z oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073 -Hostate oxide/ntp
8122024-01-25T04:45:24.940Z+ [[ disabled != online ]]
8132024-01-25T04:45:24.941Z+ [[ 5 -gt 60 ]]
8142024-01-25T04:45:24.941Z+ sleep 1
8152024-01-25T04:45:25.899Z+ retry=6
8162024-01-25T04:45:25.947Z+++ zoneadm list -n
8172024-01-25T04:45:25.947Z+++ grep oxz_ntp
8182024-01-25T04:45:25.947Z++ pfexec svcs -z oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073 -Hostate oxide/ntp
8192024-01-25T04:45:25.947Z+ [[ disabled != online ]]
8202024-01-25T04:45:25.947Z+ [[ 6 -gt 60 ]]
8212024-01-25T04:45:25.947Z+ sleep 1
8222024-01-25T04:45:26.924Z+ retry=7
8232024-01-25T04:45:26.970Z+++ zoneadm list -n
8242024-01-25T04:45:26.971Z+++ grep oxz_ntp
8252024-01-25T04:45:26.971Z++ pfexec svcs -z oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073 -Hostate oxide/ntp
8262024-01-25T04:45:26.971Z+ [[ disabled != online ]]
8272024-01-25T04:45:26.971Z+ [[ 7 -gt 60 ]]
8282024-01-25T04:45:26.971Z+ sleep 1
8292024-01-25T04:45:27.937Z+ retry=8
8302024-01-25T04:45:27.976Z+++ zoneadm list -n
8312024-01-25T04:45:27.976Z+++ grep oxz_ntp
8322024-01-25T04:45:27.976Z++ pfexec svcs -z oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073 -Hostate oxide/ntp
8332024-01-25T04:45:27.976Z+ [[ disabled != online ]]
8342024-01-25T04:45:27.977Z+ [[ 8 -gt 60 ]]
8352024-01-25T04:45:27.977Z+ sleep 1
8362024-01-25T04:45:28.959Z+ retry=9
8372024-01-25T04:45:28.998Z+++ zoneadm list -n
8382024-01-25T04:45:28.999Z+++ grep oxz_ntp
8392024-01-25T04:45:28.999Z++ pfexec svcs -z oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073 -Hostate oxide/ntp
8402024-01-25T04:45:28.999Z+ [[ disabled != online ]]
8412024-01-25T04:45:28.999Z+ [[ 9 -gt 60 ]]
8422024-01-25T04:45:28.999Z+ sleep 1
 
8432024-01-25T04:45:29.330Zbuskin console login: Jan 25 04:45:28 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8442024-01-25T04:45:29.981Z+ retry=10
8452024-01-25T04:45:30.020Z+++ zoneadm list -n
8462024-01-25T04:45:30.021Z+++ grep oxz_ntp
8472024-01-25T04:45:30.021Z++ pfexec svcs -z oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073 -Hostate oxide/ntp
8482024-01-25T04:45:30.021Z+ [[ online != online ]]
8492024-01-25T04:45:30.021Z+ echo 'Waited for chrony: 10s'
8502024-01-25T04:45:30.021ZWaited for chrony: 10s
8512024-01-25T04:45:30.021Z+ export RUST_BACKTRACE=1
8522024-01-25T04:45:30.021Z+ RUST_BACKTRACE=1
8532024-01-25T04:45:30.021Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8542024-01-25T04:45:30.021Z++ ./tests/bootstrap
8552024-01-25T04:45:30.060ZUsing external DNS server at 10.151.1.164:53
8562024-01-25T04:45:30.060Z2024-01-25 04:45:29.624772153 UTC: attempting to log into API
8572024-01-25T04:45:45.069Z2024-01-25 04:45:45.037675877 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8582024-01-25T04:45:46.070Z2024-01-25 04:45:46.041468959 UTC: attempting to log into API
8592024-01-25T04:46:01.104Z2024-01-25 04:46:01.085706664 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8602024-01-25T04:46:02.164Z2024-01-25 04:46:02.143757202 UTC: attempting to log into API
8612024-01-25T04:46:17.242Z2024-01-25 04:46:17.224039093 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8622024-01-25T04:46:18.243Z2024-01-25 04:46:18.225151896 UTC: attempting to log into API
8632024-01-25T04:46:33.275Z2024-01-25 04:46:33.257250305 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8642024-01-25T04:46:35.479Z2024-01-25 04:46:34.258312777 UTC: attempting to log into API
8652024-01-25T04:46:49.313Z2024-01-25 04:46:49.291923915 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8662024-01-25T04:46:50.311Z2024-01-25 04:46:50.293033018 UTC: attempting to log into API
8672024-01-25T04:47:05.345Z2024-01-25 04:47:05.326648156 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8682024-01-25T04:47:06.370Z2024-01-25 04:47:06.327715648 UTC: attempting to log into API
8692024-01-25T04:47:21.379Z2024-01-25 04:47:21.361029227 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8702024-01-25T04:47:22.381Z2024-01-25 04:47:22.363021202 UTC: attempting to log into API
8712024-01-25T04:47:37.415Z2024-01-25 04:47:37.396942008 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8722024-01-25T04:47:38.416Z2024-01-25 04:47:38.397989099 UTC: attempting to log into API
8732024-01-25T04:47:53.452Z2024-01-25 04:47:53.432894402 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8742024-01-25T04:47:54.628Z2024-01-25 04:47:54.552061920 UTC: attempting to log into API
8752024-01-25T04:48:09.720Z2024-01-25 04:48:09.697278768 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8762024-01-25T04:48:10.718Z2024-01-25 04:48:10.698443012 UTC: attempting to log into API
8772024-01-25T04:48:25.830Z2024-01-25 04:48:25.811347771 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8782024-01-25T04:48:26.852Z2024-01-25 04:48:26.812448273 UTC: attempting to log into API
 
8792024-01-25T04:48:39.101ZJan 25 04:48:38 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
8802024-01-25T04:48:41.868Z2024-01-25 04:48:41.848687107 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8812024-01-25T04:48:42.871Z2024-01-25 04:48:42.849455382 UTC: attempting to log into API
 
8822024-01-25T04:48:46.368ZJan 25 04:48:45 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
8832024-01-25T04:48:52.926ZJan 25 04:48:51 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8842024-01-25T04:48:57.902Z2024-01-25 04:48:57.883240173 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8852024-01-25T04:48:58.904Z2024-01-25 04:48:58.885196558 UTC: attempting to log into API
 
8862024-01-25T04:48:59.038ZJan 25 04:48:58 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
8872024-01-25T04:49:06.382ZJan 25 04:49:05 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
8882024-01-25T04:49:18.581Z2024-01-25 04:49:13.917451711 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8892024-01-25T04:49:18.662Z2024-01-25 04:49:14.918515333 UTC: attempting to log into API
8902024-01-25T04:49:29.970Z2024-01-25 04:49:29.951316109 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8912024-01-25T04:49:30.972Z2024-01-25 04:49:30.953024118 UTC: attempting to log into API
8922024-01-25T04:49:31.776Z2024-01-25 04:49:31.750033364 UTC: login succeeded
8932024-01-25T04:49:31.817Zwaiting for nexus to come up...
8942024-01-25T04:49:31.876Zcreating IP pool... 10.151.1.175 - 10.151.1.227
8952024-01-25T04:49:32.064Zsetting up quotas...
8962024-01-25T04:49:32.108Zensuring datasets are ready...
8972024-01-25T04:49:34.629Zlet's roll.
8982024-01-25T04:49:34.668Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
8992024-01-25T04:49:34.668ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
9002024-01-25T04:49:34.668ZOXIDE_TOKEN=oxide-token-c8fae551ee07a36dc885149aa15e10e7c1046beb'
9012024-01-25T04:49:34.668Z++ OXIDE_HOST=https://recovery.sys.oxide.test
9022024-01-25T04:49:34.669Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
9032024-01-25T04:49:34.669Z++ OXIDE_TOKEN=oxide-token-c8fae551ee07a36dc885149aa15e10e7c1046beb
9042024-01-25T04:49:34.669Z+ export OXIDE_HOST OXIDE_TOKEN
9052024-01-25T04:49:34.669Z+ retry=0
9062024-01-25T04:49:34.669Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9072024-01-25T04:49:34.669Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9082024-01-25T04:49:34.669Z+ [[ 0 -gt 60 ]]
9092024-01-25T04:49:34.669Z+ sleep 1
9102024-01-25T04:49:35.667Z+ retry=1
9112024-01-25T04:49:35.667Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9122024-01-25T04:49:35.706Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9132024-01-25T04:49:35.707Z+ [[ 1 -gt 60 ]]
9142024-01-25T04:49:35.707Z+ sleep 1
9152024-01-25T04:49:36.683Z+ retry=2
9162024-01-25T04:49:36.683Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9172024-01-25T04:49:36.723Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9182024-01-25T04:49:36.723Z+ [[ 2 -gt 60 ]]
9192024-01-25T04:49:36.723Z+ sleep 1
9202024-01-25T04:49:37.699Z+ retry=3
9212024-01-25T04:49:37.699Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9222024-01-25T04:49:37.738Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9232024-01-25T04:49:37.738Z+ [[ 3 -gt 60 ]]
9242024-01-25T04:49:37.739Z+ sleep 1
9252024-01-25T04:49:38.714Z+ retry=4
9262024-01-25T04:49:38.714Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9272024-01-25T04:49:38.754Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9282024-01-25T04:49:38.754Z+ [[ 4 -gt 60 ]]
9292024-01-25T04:49:38.754Z+ sleep 1
9302024-01-25T04:49:39.729Z+ retry=5
9312024-01-25T04:49:39.729Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9322024-01-25T04:49:39.768Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9332024-01-25T04:49:39.768Z+ [[ 5 -gt 60 ]]
9342024-01-25T04:49:39.768Z+ sleep 1
9352024-01-25T04:49:40.745Z+ retry=6
9362024-01-25T04:49:40.745Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9372024-01-25T04:49:41.201Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9382024-01-25T04:49:41.201Z+ [[ 6 -gt 60 ]]
9392024-01-25T04:49:41.201Z+ sleep 1
9402024-01-25T04:49:41.760Z+ retry=7
9412024-01-25T04:49:41.760Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9422024-01-25T04:49:41.799Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9432024-01-25T04:49:41.799Z+ [[ 7 -gt 60 ]]
9442024-01-25T04:49:41.799Z+ sleep 1
9452024-01-25T04:49:42.774Z+ retry=8
9462024-01-25T04:49:42.774Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9472024-01-25T04:49:42.814Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9482024-01-25T04:49:42.814Z+ [[ 8 -gt 60 ]]
9492024-01-25T04:49:42.814Z+ sleep 1
9502024-01-25T04:49:43.789Z+ retry=9
9512024-01-25T04:49:43.790Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9522024-01-25T04:49:43.829Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9532024-01-25T04:49:43.829Z+ [[ 9 -gt 60 ]]
9542024-01-25T04:49:43.829Z+ sleep 1
9552024-01-25T04:49:44.804Z+ retry=10
9562024-01-25T04:49:44.804Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9572024-01-25T04:49:44.843Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9582024-01-25T04:49:44.844Z+ [[ 10 -gt 60 ]]
9592024-01-25T04:49:44.844Z+ sleep 1
9602024-01-25T04:49:45.818Z+ retry=11
9612024-01-25T04:49:45.819Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9622024-01-25T04:49:45.858Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9632024-01-25T04:49:45.858Z+ [[ 11 -gt 60 ]]
9642024-01-25T04:49:45.858Z+ sleep 1
9652024-01-25T04:49:46.833Z+ retry=12
9662024-01-25T04:49:46.873Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9672024-01-25T04:49:46.873Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9682024-01-25T04:49:46.873Z+ [[ 12 -gt 60 ]]
9692024-01-25T04:49:46.873Z+ sleep 1
9702024-01-25T04:49:47.848Z+ retry=13
9712024-01-25T04:49:47.848Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9722024-01-25T04:49:47.887Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9732024-01-25T04:49:47.887Z+ [[ 13 -gt 60 ]]
9742024-01-25T04:49:47.887Z+ sleep 1
9752024-01-25T04:49:48.863Z+ retry=14
9762024-01-25T04:49:48.863Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9772024-01-25T04:49:48.905Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9782024-01-25T04:49:48.905Z+ [[ 14 -gt 60 ]]
9792024-01-25T04:49:48.905Z+ sleep 1
9802024-01-25T04:49:49.878Z+ retry=15
9812024-01-25T04:49:49.878Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9822024-01-25T04:49:49.917Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9832024-01-25T04:49:49.918Z+ [[ 15 -gt 60 ]]
9842024-01-25T04:49:49.918Z+ sleep 1
9852024-01-25T04:49:50.896Z+ retry=16
9862024-01-25T04:49:50.896Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9872024-01-25T04:49:50.935Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9882024-01-25T04:49:50.935Z+ [[ 16 -gt 60 ]]
9892024-01-25T04:49:50.935Z+ sleep 1
9902024-01-25T04:49:51.908Z+ retry=17
9912024-01-25T04:49:51.908Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9922024-01-25T04:49:51.952Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9932024-01-25T04:49:51.952Z+ [[ 17 -gt 60 ]]
9942024-01-25T04:49:51.952Z+ sleep 1
9952024-01-25T04:49:52.924Z+ retry=18
9962024-01-25T04:49:52.925Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9972024-01-25T04:49:52.964Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9982024-01-25T04:49:52.964Z+ [[ 18 -gt 60 ]]
9992024-01-25T04:49:52.964Z+ sleep 1
10002024-01-25T04:49:53.939Z+ retry=19
10012024-01-25T04:49:53.939Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10022024-01-25T04:49:53.978Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10032024-01-25T04:49:53.978Z+ [[ 19 -gt 60 ]]
10042024-01-25T04:49:53.978Z+ sleep 1
10052024-01-25T04:49:54.953Z+ retry=20
10062024-01-25T04:49:54.954Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10072024-01-25T04:49:54.993Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10082024-01-25T04:49:54.993Z+ [[ 20 -gt 60 ]]
10092024-01-25T04:49:54.993Z+ sleep 1
10102024-01-25T04:49:55.969Z+ retry=21
10112024-01-25T04:49:55.969Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10122024-01-25T04:49:56.017Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10132024-01-25T04:49:56.017Z+ [[ 21 -gt 60 ]]
10142024-01-25T04:49:56.017Z+ sleep 1
10152024-01-25T04:49:56.984Z+ retry=22
10162024-01-25T04:49:56.984Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10172024-01-25T04:49:57.023Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10182024-01-25T04:49:57.023Z+ [[ 22 -gt 60 ]]
10192024-01-25T04:49:57.023Z+ sleep 1
10202024-01-25T04:49:57.999Z+ retry=23
10212024-01-25T04:49:57.999Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10222024-01-25T04:49:58.043Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10232024-01-25T04:49:58.043Z+ [[ 23 -gt 60 ]]
10242024-01-25T04:49:58.043Z+ sleep 1
10252024-01-25T04:49:59.012Z+ retry=24
10262024-01-25T04:49:59.012Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10272024-01-25T04:49:59.055Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10282024-01-25T04:49:59.055Z+ [[ 24 -gt 60 ]]
10292024-01-25T04:49:59.055Z+ sleep 1
10302024-01-25T04:50:00.030Z+ retry=25
10312024-01-25T04:50:00.030Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10322024-01-25T04:50:00.078Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10332024-01-25T04:50:00.078Z+ [[ 25 -gt 60 ]]
10342024-01-25T04:50:00.078Z+ sleep 1
10352024-01-25T04:50:01.043Z+ retry=26
10362024-01-25T04:50:01.043Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10372024-01-25T04:50:01.082Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10382024-01-25T04:50:01.082Z+ [[ 26 -gt 60 ]]
10392024-01-25T04:50:01.082Z+ sleep 1
10402024-01-25T04:50:02.061Z+ retry=27
10412024-01-25T04:50:02.061Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10422024-01-25T04:50:02.102Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10432024-01-25T04:50:02.102Z+ [[ 27 -gt 60 ]]
10442024-01-25T04:50:02.102Z+ sleep 1
10452024-01-25T04:50:03.086Z+ retry=28
10462024-01-25T04:50:03.086Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10472024-01-25T04:50:03.128Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10482024-01-25T04:50:03.128Z+ [[ 28 -gt 60 ]]
10492024-01-25T04:50:03.128Z+ sleep 1
10502024-01-25T04:50:07.303Z+ retry=29
10512024-01-25T04:50:07.303Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10522024-01-25T04:50:07.404Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10532024-01-25T04:50:07.404Z+ [[ 29 -gt 60 ]]
10542024-01-25T04:50:07.404Z+ sleep 1
10552024-01-25T04:50:07.404Z+ retry=30
10562024-01-25T04:50:07.404Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10572024-01-25T04:50:07.404Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10582024-01-25T04:50:07.405Z+ [[ 30 -gt 60 ]]
10592024-01-25T04:50:07.405Z+ sleep 1
10602024-01-25T04:50:07.405Z+ retry=31
10612024-01-25T04:50:07.405Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10622024-01-25T04:50:07.405Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10632024-01-25T04:50:07.405Z+ [[ 31 -gt 60 ]]
10642024-01-25T04:50:07.405Z+ sleep 1
10652024-01-25T04:50:07.497Z+ retry=32
10662024-01-25T04:50:07.497Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10672024-01-25T04:50:07.497Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
10682024-01-25T04:50:07.546Z{"status":"ok"}success
10692024-01-25T04:50:07.546ZProject {
10702024-01-25T04:50:07.546Z description: "some images",
10712024-01-25T04:50:07.547Z id: d1d47c48-f377-41e4-bfa8-905599fae9f2,
10722024-01-25T04:50:07.547Z name: Name(
10732024-01-25T04:50:07.547Z "images",
10742024-01-25T04:50:07.547Z ),
10752024-01-25T04:50:07.547Z time_created: 2024-01-25T04:50:07.282415Z,
10762024-01-25T04:50:07.547Z time_modified: 2024-01-25T04:50:07.282415Z,
10772024-01-25T04:50:07.547Z}
10782024-01-25T04:50:07.547Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
10792024-01-25T04:55:36.035Zdone!
10802024-01-25T04:55:36.035Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
10812024-01-25T04:55:36.035Zsuccess
10822024-01-25T04:55:36.035ZImage {
10832024-01-25T04:55:36.035Z block_size: ByteCount(
10842024-01-25T04:55:36.035Z 512,
10852024-01-25T04:55:36.035Z ),
10862024-01-25T04:55:36.035Z description: "debian 11 original base image",
10872024-01-25T04:55:36.035Z digest: None,
10882024-01-25T04:55:36.035Z id: 3cabc737-a5a4-4fbe-9d25-6777f6e8e7db,
10892024-01-25T04:55:36.035Z name: Name(
10902024-01-25T04:55:36.035Z "debian11",
10912024-01-25T04:55:36.035Z ),
10922024-01-25T04:55:36.036Z os: "debian",
10932024-01-25T04:55:36.036Z project_id: None,
10942024-01-25T04:55:36.036Z size: ByteCount(
10952024-01-25T04:55:36.036Z 2147483648,
10962024-01-25T04:55:36.036Z ),
10972024-01-25T04:55:36.036Z time_created: 2024-01-25T04:55:32.820300Z,
10982024-01-25T04:55:36.036Z time_modified: 2024-01-25T04:55:33.049917Z,
10992024-01-25T04:55:36.036Z url: None,
11002024-01-25T04:55:36.036Z version: "11",
11012024-01-25T04:55:36.036Z}
11022024-01-25T04:55:36.036Z+ rm ./tests/bootstrap
11032024-01-25T04:55:36.036Z+ for test_bin in tests/*
11042024-01-25T04:55:36.036Z+ ./tests/bootstrap-bb8a8abc67437a2f
11052024-01-25T04:55:36.036Z
11062024-01-25T04:55:36.036Zrunning 0 tests
11072024-01-25T04:55:36.036Z
11082024-01-25T04:55:36.036Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
11092024-01-25T04:55:36.036Z
11102024-01-25T04:55:36.036Z+ for test_bin in tests/*
11112024-01-25T04:55:36.036Z+ ./tests/end_to_end_tests-c4605ef041dbbb6b
11122024-01-25T04:55:36.036Z
11132024-01-25T04:55:36.040Zrunning 2 tests
11142024-01-25T04:55:36.040Ztest no_spoof::no_spoof_here ... ok
11152024-01-25T04:56:05.737Ztest instance_launch::instance_launch ... ok
11162024-01-25T04:56:05.737Z
11172024-01-25T04:56:05.737Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 32.37s
11182024-01-25T04:56:05.737Z
11192024-01-25T04:56:05.783Z+ _exit_trap
11202024-01-25T04:56:05.784Z+ local status=0
11212024-01-25T04:56:05.784Z+ [[ 0 -eq 0 ]]
11222024-01-25T04:56:05.784Z+ exit 0
11232024-01-25T04:56:05.825Zprocess exited: duration 770633 ms, exit code 0
 
11242024-01-25T04:56:05.943Zfound 62 output files
11252024-01-25T04:56:05.943Zuploading: /var/svc/log/oxide-sled-agent:default.log (1731756 bytes)
11262024-01-25T04:56:07.967Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 1731756 -> 1732168
11272024-01-25T04:56:09.094Zuploaded: /var/svc/log/oxide-sled-agent:default.log
11282024-01-25T04:56:09.094Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_3892497f-2613-42ea-8327-99b64b8943c8/root/var/svc/log/oxide-clickhouse:default.log (869991 bytes)
11292024-01-25T04:56:11.376Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_3892497f-2613-42ea-8327-99b64b8943c8/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 869991 -> 888758
11302024-01-25T04:56:12.506Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_3892497f-2613-42ea-8327-99b64b8943c8/root/var/svc/log/oxide-clickhouse:default.log
11312024-01-25T04:56:12.506Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_3892497f-2613-42ea-8327-99b64b8943c8/root/var/svc/log/oxide-zone-network-setup:default.log (1709 bytes)
11322024-01-25T04:56:13.667Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_3892497f-2613-42ea-8327-99b64b8943c8/root/var/svc/log/oxide-zone-network-setup:default.log
11332024-01-25T04:56:13.668Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_214511e8-4b09-4dc2-a977-4413f4f36586/root/var/svc/log/oxide-cockroachdb:default.log (4380 bytes)
11342024-01-25T04:56:14.826Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_214511e8-4b09-4dc2-a977-4413f4f36586/root/var/svc/log/oxide-cockroachdb:default.log
11352024-01-25T04:56:14.866Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_214511e8-4b09-4dc2-a977-4413f4f36586/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
11362024-01-25T04:56:15.995Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_214511e8-4b09-4dc2-a977-4413f4f36586/root/var/svc/log/oxide-zone-network-setup:default.log
11372024-01-25T04:56:15.995Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fd13a787-b246-4e37-8fc2-29e785613676/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
11382024-01-25T04:56:17.170Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fd13a787-b246-4e37-8fc2-29e785613676/root/var/svc/log/oxide-crucible-agent:default.log
11392024-01-25T04:56:17.210Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fd13a787-b246-4e37-8fc2-29e785613676/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11402024-01-25T04:56:18.345Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fd13a787-b246-4e37-8fc2-29e785613676/root/var/svc/log/oxide-crucible-downstairs:default.log
11412024-01-25T04:56:18.345Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_109c23d6-8f56-47c9-a88a-bb521761b808/root/var/svc/log/oxide-external_dns:default.log (26939 bytes)
11422024-01-25T04:56:18.560Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_109c23d6-8f56-47c9-a88a-bb521761b808/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 26939 -> 30712
11432024-01-25T04:56:19.692Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_109c23d6-8f56-47c9-a88a-bb521761b808/root/var/svc/log/oxide-external_dns:default.log
11442024-01-25T04:56:19.737Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b13557e6-d58f-4486-b06a-55372e116a62/root/var/svc/log/oxide-internal_dns:default.log (113134 bytes)
11452024-01-25T04:56:20.894Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b13557e6-d58f-4486-b06a-55372e116a62/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 113134 -> 116727
11462024-01-25T04:56:22.021Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b13557e6-d58f-4486-b06a-55372e116a62/root/var/svc/log/oxide-internal_dns:default.log
11472024-01-25T04:56:22.021Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2fb73d84-51e0-475f-a37a-cdd68817427e/root/var/svc/log/oxide-cockroachdb:default.log (4406 bytes)
11482024-01-25T04:56:26.930Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2fb73d84-51e0-475f-a37a-cdd68817427e/root/var/svc/log/oxide-cockroachdb:default.log
11492024-01-25T04:56:26.930Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2fb73d84-51e0-475f-a37a-cdd68817427e/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
11502024-01-25T04:56:32.193Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2fb73d84-51e0-475f-a37a-cdd68817427e/root/var/svc/log/oxide-zone-network-setup:default.log
11512024-01-25T04:56:32.193Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9e93968b-e2b3-4a49-a0e9-24cc01d9741b/root/var/svc/log/oxide-crucible-agent:default.log (27793 bytes)
11522024-01-25T04:56:38.854Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9e93968b-e2b3-4a49-a0e9-24cc01d9741b/root/var/svc/log/oxide-crucible-agent:default.log
11532024-01-25T04:56:38.854Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9e93968b-e2b3-4a49-a0e9-24cc01d9741b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11542024-01-25T04:56:39.948Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9e93968b-e2b3-4a49-a0e9-24cc01d9741b/root/var/svc/log/oxide-crucible-downstairs:default.log
11552024-01-25T04:56:39.948Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9e93968b-e2b3-4a49-a0e9-24cc01d9741b/root/var/svc/log/oxide-crucible-downstairs:downstairs-5fbf2bed-d906-4e58-9d18-f9ec1d056b4b.log (5617 bytes)
11562024-01-25T04:56:41.163Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9e93968b-e2b3-4a49-a0e9-24cc01d9741b/root/var/svc/log/oxide-crucible-downstairs:downstairs-5fbf2bed-d906-4e58-9d18-f9ec1d056b4b.log
11572024-01-25T04:56:41.163Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9e93968b-e2b3-4a49-a0e9-24cc01d9741b/root/var/svc/log/oxide-crucible-downstairs:downstairs-b546d202-3730-4095-8b35-7dc8cd5d5872.log (5537 bytes)
11582024-01-25T04:56:42.371Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9e93968b-e2b3-4a49-a0e9-24cc01d9741b/root/var/svc/log/oxide-crucible-downstairs:downstairs-b546d202-3730-4095-8b35-7dc8cd5d5872.log
11592024-01-25T04:56:42.371Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9e93968b-e2b3-4a49-a0e9-24cc01d9741b/root/var/svc/log/oxide-crucible-downstairs:snapshot-5fbf2bed-d906-4e58-9d18-f9ec1d056b4b-395a799e-3df0-44f7-a272-b33a4277dcc0.log (5177 bytes)
11602024-01-25T04:56:44.090Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9e93968b-e2b3-4a49-a0e9-24cc01d9741b/root/var/svc/log/oxide-crucible-downstairs:snapshot-5fbf2bed-d906-4e58-9d18-f9ec1d056b4b-395a799e-3df0-44f7-a272-b33a4277dcc0.log
11612024-01-25T04:56:44.090Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_bd8ee388-dd0b-4f06-9a74-03b06eb53f57/root/var/svc/log/oxide-external_dns:default.log (26941 bytes)
11622024-01-25T04:56:44.372Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_bd8ee388-dd0b-4f06-9a74-03b06eb53f57/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 26941 -> 30715
11632024-01-25T04:56:45.489Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_bd8ee388-dd0b-4f06-9a74-03b06eb53f57/root/var/svc/log/oxide-external_dns:default.log
11642024-01-25T04:56:45.489Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_03bd4301-c6c7-4002-9ce6-61ed64616157/root/var/svc/log/oxide-internal_dns:default.log (109143 bytes)
11652024-01-25T04:56:46.055Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_03bd4301-c6c7-4002-9ce6-61ed64616157/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 109143 -> 112898
11662024-01-25T04:56:47.185Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_03bd4301-c6c7-4002-9ce6-61ed64616157/root/var/svc/log/oxide-internal_dns:default.log
11672024-01-25T04:56:47.185Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9361554f-f45d-4ca7-aea8-33a993c79ac7/root/var/svc/log/oxide-nexus:default.log (34131095 bytes)
11682024-01-25T04:57:15.827Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9361554f-f45d-4ca7-aea8-33a993c79ac7/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 34131095 -> 34266974
11692024-01-25T04:57:16.959Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9361554f-f45d-4ca7-aea8-33a993c79ac7/root/var/svc/log/oxide-nexus:default.log
11702024-01-25T04:57:16.959Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_9b62c403-061e-4662-bb6a-c1f39e2a3633/root/var/svc/log/oxide-oximeter:default.log (264454 bytes)
11712024-01-25T04:57:17.476Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_9b62c403-061e-4662-bb6a-c1f39e2a3633/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 264454 -> 312108
11722024-01-25T04:57:18.653Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_9b62c403-061e-4662-bb6a-c1f39e2a3633/root/var/svc/log/oxide-oximeter:default.log
11732024-01-25T04:57:18.653Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_3fb5a01a-285f-4adf-b88c-1c5cdef3d09f/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
11742024-01-25T04:57:19.828Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_3fb5a01a-285f-4adf-b88c-1c5cdef3d09f/root/var/svc/log/oxide-cockroachdb:default.log
11752024-01-25T04:57:19.869Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_3fb5a01a-285f-4adf-b88c-1c5cdef3d09f/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
11762024-01-25T04:57:20.982Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_3fb5a01a-285f-4adf-b88c-1c5cdef3d09f/root/var/svc/log/oxide-zone-network-setup:default.log
11772024-01-25T04:57:20.982Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7728ac6e-c6b4-42dc-99ac-fcd1067f6c17/root/var/svc/log/oxide-crucible-agent:default.log (14113 bytes)
11782024-01-25T04:57:22.191Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7728ac6e-c6b4-42dc-99ac-fcd1067f6c17/root/var/svc/log/oxide-crucible-agent:default.log
11792024-01-25T04:57:22.192Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7728ac6e-c6b4-42dc-99ac-fcd1067f6c17/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11802024-01-25T04:57:24.379Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7728ac6e-c6b4-42dc-99ac-fcd1067f6c17/root/var/svc/log/oxide-crucible-downstairs:default.log
11812024-01-25T04:57:24.434Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7728ac6e-c6b4-42dc-99ac-fcd1067f6c17/root/var/svc/log/oxide-crucible-downstairs:downstairs-88f1a49b-f5be-4c7c-a66c-6aa838075cb4.log (3047 bytes)
11822024-01-25T04:57:25.538Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7728ac6e-c6b4-42dc-99ac-fcd1067f6c17/root/var/svc/log/oxide-crucible-downstairs:downstairs-88f1a49b-f5be-4c7c-a66c-6aa838075cb4.log
11832024-01-25T04:57:25.538Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_c40bf647-0593-471d-9f78-92f11c1aad6e/root/var/svc/log/oxide-internal_dns:default.log (113314 bytes)
11842024-01-25T04:57:25.802Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_c40bf647-0593-471d-9f78-92f11c1aad6e/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 113314 -> 120825
11852024-01-25T04:57:26.929Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_c40bf647-0593-471d-9f78-92f11c1aad6e/root/var/svc/log/oxide-internal_dns:default.log
11862024-01-25T04:57:26.929Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_782354d7-7e09-44f8-8952-b34facd1b711/root/var/svc/log/oxide-cockroachdb:default.log (4406 bytes)
11872024-01-25T04:57:28.098Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_782354d7-7e09-44f8-8952-b34facd1b711/root/var/svc/log/oxide-cockroachdb:default.log
11882024-01-25T04:57:28.098Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_782354d7-7e09-44f8-8952-b34facd1b711/root/var/svc/log/oxide-zone-network-setup:default.log (1705 bytes)
11892024-01-25T04:57:29.268Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_782354d7-7e09-44f8-8952-b34facd1b711/root/var/svc/log/oxide-zone-network-setup:default.log
11902024-01-25T04:57:29.268Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f04a5964-e16c-4cb5-a05c-5cd5b32aaef6/root/var/svc/log/oxide-crucible-agent:default.log (20154 bytes)
11912024-01-25T04:57:30.607Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f04a5964-e16c-4cb5-a05c-5cd5b32aaef6/root/var/svc/log/oxide-crucible-agent:default.log
11922024-01-25T04:57:30.607Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f04a5964-e16c-4cb5-a05c-5cd5b32aaef6/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11932024-01-25T04:57:31.795Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f04a5964-e16c-4cb5-a05c-5cd5b32aaef6/root/var/svc/log/oxide-crucible-downstairs:default.log
11942024-01-25T04:57:31.795Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f04a5964-e16c-4cb5-a05c-5cd5b32aaef6/root/var/svc/log/oxide-crucible-downstairs:downstairs-6b6e0904-2942-4d0a-8658-3cd0dbe1fac8.log (5622 bytes)
11952024-01-25T04:57:34.297Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f04a5964-e16c-4cb5-a05c-5cd5b32aaef6/root/var/svc/log/oxide-crucible-downstairs:downstairs-6b6e0904-2942-4d0a-8658-3cd0dbe1fac8.log
11962024-01-25T04:57:34.297Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f04a5964-e16c-4cb5-a05c-5cd5b32aaef6/root/var/svc/log/oxide-crucible-downstairs:snapshot-6b6e0904-2942-4d0a-8658-3cd0dbe1fac8-395a799e-3df0-44f7-a272-b33a4277dcc0.log (5177 bytes)
11972024-01-25T04:57:35.463Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_f04a5964-e16c-4cb5-a05c-5cd5b32aaef6/root/var/svc/log/oxide-crucible-downstairs:snapshot-6b6e0904-2942-4d0a-8658-3cd0dbe1fac8-395a799e-3df0-44f7-a272-b33a4277dcc0.log
11982024-01-25T04:57:35.463Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_cb8a177f-9e71-4be9-933b-a6ef11f3afa4/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
11992024-01-25T04:57:36.623Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_cb8a177f-9e71-4be9-933b-a6ef11f3afa4/root/var/svc/log/oxide-crucible-pantry:default.log
12002024-01-25T04:57:36.623Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_5226e3b8-f622-4981-afd2-54526c8528c0/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
12012024-01-25T04:57:42.101Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_5226e3b8-f622-4981-afd2-54526c8528c0/root/var/svc/log/oxide-cockroachdb:default.log
12022024-01-25T04:57:42.101Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_5226e3b8-f622-4981-afd2-54526c8528c0/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
12032024-01-25T04:57:43.643Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_5226e3b8-f622-4981-afd2-54526c8528c0/root/var/svc/log/oxide-zone-network-setup:default.log
12042024-01-25T04:57:43.643Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d3183930-1278-4b23-a29e-fc7b8d26f6cb/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
12052024-01-25T04:57:45.342Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d3183930-1278-4b23-a29e-fc7b8d26f6cb/root/var/svc/log/oxide-crucible-agent:default.log
12062024-01-25T04:57:45.344Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d3183930-1278-4b23-a29e-fc7b8d26f6cb/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12072024-01-25T04:57:46.497Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d3183930-1278-4b23-a29e-fc7b8d26f6cb/root/var/svc/log/oxide-crucible-downstairs:default.log
12082024-01-25T04:57:46.497Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d3183930-1278-4b23-a29e-fc7b8d26f6cb/root/var/svc/log/oxide-crucible-downstairs:downstairs-895198a5-122a-4139-ac74-8ab2eb06c4ff.log (3049 bytes)
12092024-01-25T04:57:47.657Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d3183930-1278-4b23-a29e-fc7b8d26f6cb/root/var/svc/log/oxide-crucible-downstairs:downstairs-895198a5-122a-4139-ac74-8ab2eb06c4ff.log
12102024-01-25T04:57:47.657Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_10d53d00-9021-4bf1-9e8a-0df16db7fe1e/root/var/svc/log/oxide-crucible-pantry:default.log (1221765 bytes)
12112024-01-25T04:57:50.982Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_10d53d00-9021-4bf1-9e8a-0df16db7fe1e/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 1221765 -> 1253463
12122024-01-25T04:57:52.117Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_10d53d00-9021-4bf1-9e8a-0df16db7fe1e/root/var/svc/log/oxide-crucible-pantry:default.log
12132024-01-25T04:57:52.117Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a6a3414f-45de-44f5-8a51-0b47a01ebf66/root/var/svc/log/oxide-crucible-agent:default.log (9794 bytes)
12142024-01-25T04:57:53.342Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a6a3414f-45de-44f5-8a51-0b47a01ebf66/root/var/svc/log/oxide-crucible-agent:default.log
12152024-01-25T04:57:53.383Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a6a3414f-45de-44f5-8a51-0b47a01ebf66/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12162024-01-25T04:57:54.503Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a6a3414f-45de-44f5-8a51-0b47a01ebf66/root/var/svc/log/oxide-crucible-downstairs:default.log
12172024-01-25T04:57:54.503Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a6a3414f-45de-44f5-8a51-0b47a01ebf66/root/var/svc/log/oxide-crucible-downstairs:downstairs-ee660f16-5bde-40c4-b2bd-82eeaf2f6e6c.log (2944 bytes)
12182024-01-25T04:57:56.085Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a6a3414f-45de-44f5-8a51-0b47a01ebf66/root/var/svc/log/oxide-crucible-downstairs:downstairs-ee660f16-5bde-40c4-b2bd-82eeaf2f6e6c.log
12192024-01-25T04:57:56.085Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2f44dfd6-28a7-439a-ab37-9464a5d0fd4e/root/var/svc/log/oxide-crucible-agent:default.log (9794 bytes)
12202024-01-25T04:57:57.430Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2f44dfd6-28a7-439a-ab37-9464a5d0fd4e/root/var/svc/log/oxide-crucible-agent:default.log
12212024-01-25T04:57:57.469Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2f44dfd6-28a7-439a-ab37-9464a5d0fd4e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12222024-01-25T04:57:58.556Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2f44dfd6-28a7-439a-ab37-9464a5d0fd4e/root/var/svc/log/oxide-crucible-downstairs:default.log
12232024-01-25T04:57:58.596Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2f44dfd6-28a7-439a-ab37-9464a5d0fd4e/root/var/svc/log/oxide-crucible-downstairs:downstairs-75643494-49d3-4880-92ac-b9421394e812.log (2945 bytes)
12242024-01-25T04:57:59.718Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2f44dfd6-28a7-439a-ab37-9464a5d0fd4e/root/var/svc/log/oxide-crucible-downstairs:downstairs-75643494-49d3-4880-92ac-b9421394e812.log
12252024-01-25T04:57:59.718Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_9769b656-0e46-4cd4-9109-dc7e3d87c603/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
12262024-01-25T04:58:00.967Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_9769b656-0e46-4cd4-9109-dc7e3d87c603/root/var/svc/log/oxide-crucible-pantry:default.log
12272024-01-25T04:58:00.968Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_255a68af-7ea5-4efa-8feb-febcb49e2ab0/root/var/svc/log/oxide-nexus:default.log (2541207 bytes)
12282024-01-25T04:58:03.374Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_255a68af-7ea5-4efa-8feb-febcb49e2ab0/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 2541207 -> 2690572
12292024-01-25T04:58:04.519Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_255a68af-7ea5-4efa-8feb-febcb49e2ab0/root/var/svc/log/oxide-nexus:default.log
12302024-01-25T04:58:04.519Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073/root/var/svc/log/oxide-ntp:default.log (1120 bytes)
12312024-01-25T04:58:05.685Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073/root/var/svc/log/oxide-ntp:default.log
12322024-01-25T04:58:05.685Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bda361e-3b14-444f-b916-eb7a5e415d33/root/var/svc/log/oxide-crucible-agent:default.log (33721 bytes)
12332024-01-25T04:58:08.795Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bda361e-3b14-444f-b916-eb7a5e415d33/root/var/svc/log/oxide-crucible-agent:default.log
12342024-01-25T04:58:08.795Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bda361e-3b14-444f-b916-eb7a5e415d33/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12352024-01-25T04:58:10.031Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bda361e-3b14-444f-b916-eb7a5e415d33/root/var/svc/log/oxide-crucible-downstairs:default.log
12362024-01-25T04:58:10.031Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bda361e-3b14-444f-b916-eb7a5e415d33/root/var/svc/log/oxide-crucible-downstairs:downstairs-60aa8ce0-0e2b-406e-b782-72fdee94485f.log (2943 bytes)
12372024-01-25T04:58:11.151Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bda361e-3b14-444f-b916-eb7a5e415d33/root/var/svc/log/oxide-crucible-downstairs:downstairs-60aa8ce0-0e2b-406e-b782-72fdee94485f.log
12382024-01-25T04:58:11.151Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bda361e-3b14-444f-b916-eb7a5e415d33/root/var/svc/log/oxide-crucible-downstairs:downstairs-9b4f01c6-a7f8-4fa5-a3af-b5730225b661.log (5619 bytes)
12392024-01-25T04:58:12.808Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bda361e-3b14-444f-b916-eb7a5e415d33/root/var/svc/log/oxide-crucible-downstairs:downstairs-9b4f01c6-a7f8-4fa5-a3af-b5730225b661.log
12402024-01-25T04:58:12.808Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bda361e-3b14-444f-b916-eb7a5e415d33/root/var/svc/log/oxide-crucible-downstairs:downstairs-f4153451-b741-4cf4-bd93-a4418b5cc6ab.log (5533 bytes)
12412024-01-25T04:58:13.998Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bda361e-3b14-444f-b916-eb7a5e415d33/root/var/svc/log/oxide-crucible-downstairs:downstairs-f4153451-b741-4cf4-bd93-a4418b5cc6ab.log
12422024-01-25T04:58:13.998Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bda361e-3b14-444f-b916-eb7a5e415d33/root/var/svc/log/oxide-crucible-downstairs:snapshot-9b4f01c6-a7f8-4fa5-a3af-b5730225b661-395a799e-3df0-44f7-a272-b33a4277dcc0.log (5176 bytes)
12432024-01-25T04:58:15.181Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bda361e-3b14-444f-b916-eb7a5e415d33/root/var/svc/log/oxide-crucible-downstairs:snapshot-9b4f01c6-a7f8-4fa5-a3af-b5730225b661-395a799e-3df0-44f7-a272-b33a4277dcc0.log
12442024-01-25T04:58:15.181Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_7acb0330-12c6-4093-bba1-46f298978394/root/var/svc/log/oxide-nexus:default.log (892223 bytes)
12452024-01-25T04:58:17.435Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_7acb0330-12c6-4093-bba1-46f298978394/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 892223 -> 1101291
12462024-01-25T04:58:18.990Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_7acb0330-12c6-4093-bba1-46f298978394/root/var/svc/log/oxide-nexus:default.log
12472024-01-25T04:58:18.990Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e8704f4-141a-44f9-b744-c5abb7c63a43/root/var/svc/log/oxide-crucible-agent:default.log (24361 bytes)
12482024-01-25T04:58:22.234Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e8704f4-141a-44f9-b744-c5abb7c63a43/root/var/svc/log/oxide-crucible-agent:default.log
12492024-01-25T04:58:22.234Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e8704f4-141a-44f9-b744-c5abb7c63a43/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12502024-01-25T04:58:23.389Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e8704f4-141a-44f9-b744-c5abb7c63a43/root/var/svc/log/oxide-crucible-downstairs:default.log
12512024-01-25T04:58:23.389Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e8704f4-141a-44f9-b744-c5abb7c63a43/root/var/svc/log/oxide-crucible-downstairs:downstairs-48120578-5948-4902-9865-0e5a2b8f6782.log (5539 bytes)
12522024-01-25T04:58:24.740Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e8704f4-141a-44f9-b744-c5abb7c63a43/root/var/svc/log/oxide-crucible-downstairs:downstairs-48120578-5948-4902-9865-0e5a2b8f6782.log
12532024-01-25T04:58:24.740Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e8704f4-141a-44f9-b744-c5abb7c63a43/root/var/svc/log/oxide-crucible-downstairs:downstairs-6be0608d-76b1-4ea4-9ae9-4bd6c088fc60.log (3051 bytes)
12542024-01-25T04:58:26.110Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e8704f4-141a-44f9-b744-c5abb7c63a43/root/var/svc/log/oxide-crucible-downstairs:downstairs-6be0608d-76b1-4ea4-9ae9-4bd6c088fc60.log
12552024-01-25T04:58:26.111Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073/root/var/log/chrony/measurements.log (18769 bytes)
12562024-01-25T04:58:26.792Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073/root/var/log/chrony/measurements.log" changed size mid upload: 18769 -> 22605
12572024-01-25T04:58:28.360Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073/root/var/log/chrony/measurements.log
12582024-01-25T04:58:28.360Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073/root/var/log/chrony/statistics.log (12987 bytes)
12592024-01-25T04:58:28.522Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073/root/var/log/chrony/statistics.log" changed size mid upload: 12987 -> 15912
12602024-01-25T04:58:29.649Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073/root/var/log/chrony/statistics.log
12612024-01-25T04:58:29.649Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073/root/var/log/chrony/tracking.log (2250 bytes)
12622024-01-25T04:58:29.732Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073/root/var/log/chrony/tracking.log" changed size mid upload: 2250 -> 2646
12632024-01-25T04:58:30.859Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_d40e9117-8bd4-45ea-80cd-ee201504a073/root/var/log/chrony/tracking.log