01HHE3CSYPWGZNKA6KVM62Z5WY: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.27",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HHE3FRJF8D1YHF87MB9PV3TK

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-12T04:57:50.877Zjob dependencies complete; ready to run (waiting for 47 m 35 s)
22023-12-12T05:27:56.025Zjob assigned to worker 01HHE7KBBKX3XGB7S8286HB05S (queued for 30 m 5 s)
32023-12-12T05:27:56.373Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-12-12T05:27:56.422Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-12-12T05:27:56.460Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-12-12T05:27:56.494Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-12T05:27:56.527Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-12-12T05:27:56.560ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-12T05:27:56.594Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-12T05:27:56.627ZOxide Helios Version helios-2.0.22333 64-bit
112023-12-12T05:27:56.661ZConfiguring devices.
122023-12-12T05:27:56.699Z##################################################
132023-12-12T05:27:56.739Z#### ############# ##
142023-12-12T05:27:56.772Z### ### ############ ##
152023-12-12T05:27:56.807Z## ### # ## ### ## Oxide Computer Company ##
162023-12-12T05:27:56.839Z## ## ## ### # ### ##
172023-12-12T05:27:56.873Z## # ### #### #### This Station Under ##
182023-12-12T05:27:56.906Z### ### #### # ### Computer Control ##
192023-12-12T05:27:56.941Z#### #### ### ## ##
202023-12-12T05:27:56.973Z##################################################
212023-12-12T05:27:57.013ZHostname: buskin
222023-12-12T05:27:57.053ZLoading smf(7) service descriptions:
232023-12-12T05:27:57.693Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-8aed3472db5f37d6.gz
242023-12-12T05:27:59.020Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-8aed3472db5f37d6.gz
252023-12-12T05:27:59.020Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-12T05:28:16.797Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-12T05:28:16.797Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-fd6b950b6a51f2ac.gz
282023-12-12T05:28:39.713Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-fd6b950b6a51f2ac.gz
292023-12-12T05:28:39.713Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-12T05:28:40.585Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-12T05:28:40.585Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-12T05:28:55.157Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-12T05:28:55.515Zdownloading input: /input/package/work/version.txt
342023-12-12T05:28:55.672Zdownloaded input: /input/package/work/version.txt
352023-12-12T05:28:55.672Zdownloading input: /input/package/work/package.tar.gz
362023-12-12T05:29:11.087Zdownloaded input: /input/package/work/package.tar.gz
372023-12-12T05:29:11.087Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-12T05:29:45.100Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-12T05:29:45.141Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-12T05:29:53.248Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-12T05:29:53.276Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-12T05:30:34.986Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-12T05:30:35.018Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-12T05:31:16.211Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-12T05:31:16.213Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-12T05:31:48.799Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-12T05:31:48.799Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-12T05:31:56.038Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-12T05:31:56.065Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-12T05:32:25.073Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-12T05:32:25.073Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-12T05:32:34.332Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-12T05:32:34.332Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-12T05:32:40.353Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-12T05:32:40.353Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-12T05:32:40.421Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-12T05:32:40.422Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-12T05:32:46.475Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-12T05:32:46.475Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-12T05:32:57.352Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-12T05:32:57.353Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-12T05:33:12.461Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-12T05:33:12.462Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-12T05:33:28.184Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-12T05:33:28.184Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-12T05:33:28.277Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-12T05:33:28.341Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-12T05:33:36.213Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-12T05:33:36.238Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-12T05:33:54.498Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-12T05:33:54.498Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-12T05:35:13.699Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-12T05:35:13.738Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-12T05:36:06.188Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-12T05:36:06.222Zstarting task 0: "setup"
762023-12-12T05:36:06.222Z++ uname -s
772023-12-12T05:36:06.223Z+ kern=SunOS
782023-12-12T05:36:06.223Z+ case "$kern" in
792023-12-12T05:36:06.223Z+ groupadd -g 12345 build
802023-12-12T05:36:06.223Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-12T05:36:08.189Z+ zfs create -o mountpoint=/work rpool/work
822023-12-12T05:36:08.636Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-12T05:36:08.636Z+ home_fs=zfs
842023-12-12T05:36:08.636Z+ [[ zfs == autofs ]]
852023-12-12T05:36:08.636Z+ mkdir -p /home/build
862023-12-12T05:36:08.636Z+ chown build:build /home/build /work
872023-12-12T05:36:10.222Z+ chmod 0700 /home/build /work
882023-12-12T05:36:10.255Zprocess exited: duration 4052 ms, exit code 0
 
892023-12-12T05:36:10.316Zstarting task 1: "authentication"
902023-12-12T05:36:10.364Zprocess exited: duration 47 ms, exit code 0
 
912023-12-12T05:36:10.425Zstarting task 2: "build"
922023-12-12T05:36:10.509Z+ trap _exit_trap EXIT
932023-12-12T05:36:10.510Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-12T05:36:10.648Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-12T05:36:10.648Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-12T05:36:10.648Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-12T05:36:10.649Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-12T05:36:10.649Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-12T05:36:10.649Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-12T05:36:10.649Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-12T05:36:10.649Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-12T05:36:10.657Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-12T05:36:10.657Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-12T05:36:10.691Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-12T05:36:10.692Z+ [[ -d /opt/oxide ]]
1062023-12-12T05:36:10.692Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-12T05:36:10.693Z+ pfexec mkdir /opt/oxide
1082023-12-12T05:36:10.694Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-12T05:36:10.694Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-12T05:36:10.694Z+ cd /opt/oxide-underneath
1112023-12-12T05:36:10.694Z+ pfexec tar ceEp@/f - .
1122023-12-12T05:36:10.694Z+ cd /opt/oxide
1132023-12-12T05:36:10.695Z+ pfexec tar xveEp@/f -
1142023-12-12T05:36:10.758Zx ., 0 bytes, 0 tape blocks
1152023-12-12T05:36:10.758Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-12T05:36:10.758Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-12T05:36:10.758Zx ./opte/bin/opteadm, 52567720 bytes, 102672 tape blocks
1182023-12-12T05:36:10.783Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-12T05:36:10.783Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-12T05:36:10.808Z+ pfexec mkdir /opt/oxide/work
1212023-12-12T05:36:10.808Z+ pfexec chown build:build /opt/oxide/work
1222023-12-12T05:36:10.808Z+ cd /opt/oxide/work
1232023-12-12T05:36:10.808Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-12T05:36:12.335ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-12-12T05:36:12.335Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-12-12T05:36:12.336Zx out/omicron-sled-agent.tar, 111837696 bytes, 218433 tape blocks
1272023-12-12T05:36:12.336Zx out/target/test, 75 bytes, 1 tape blocks
1282023-12-12T05:36:12.336Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1292023-12-12T05:36:12.336Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-12-12T05:36:12.337Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1312023-12-12T05:36:12.337Zx target/release/omicron-package, 42513136 bytes, 83034 tape blocks
1322023-12-12T05:36:12.337Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-12-12T05:36:12.337Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-12-12T05:36:12.337Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-12-12T05:36:12.338Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-12-12T05:36:12.338Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-12-12T05:36:12.338Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-12-12T05:36:12.338Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-12-12T05:36:12.368Z
1402023-12-12T05:36:12.368Zreal 1.532208432
1412023-12-12T05:36:12.369Zuser 1.384055791
1422023-12-12T05:36:12.369Zsys 0.261151098
1432023-12-12T05:36:12.369Ztrap 0.000063381
1442023-12-12T05:36:12.369Ztflt 0.000000000
1452023-12-12T05:36:12.369Zdflt 0.000000000
1462023-12-12T05:36:12.369Zkflt 0.000000000
1472023-12-12T05:36:12.370Zlock 0.000000000
1482023-12-12T05:36:12.370Zslp 1.383886477
1492023-12-12T05:36:12.370Zlat 0.033368001
1502023-12-12T05:36:12.370Zstop 0.000117521
1512023-12-12T05:36:12.370Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-12-12T05:36:13.397Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-12-12T05:36:13.480Z+ mkdir tests
1542023-12-12T05:36:13.480Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-12-12T05:36:13.480Z+ ptime -m gunzip
1562023-12-12T05:36:13.481Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8aed3472db5f37d6
1572023-12-12T05:36:13.576Z
1582023-12-12T05:36:13.607Zreal 0.141002459
1592023-12-12T05:36:13.607Zuser 0.128211234
1602023-12-12T05:36:13.607Zsys 0.012547861
1612023-12-12T05:36:13.608Ztrap 0.000012589
1622023-12-12T05:36:13.608Ztflt 0.000000000
1632023-12-12T05:36:13.609Zdflt 0.000000000
1642023-12-12T05:36:13.609Zkflt 0.000000000
1652023-12-12T05:36:13.609Zlock 0.000000000
1662023-12-12T05:36:13.610Zslp 0.000025400
1672023-12-12T05:36:13.610Zlat 0.000020920
1682023-12-12T05:36:13.610Zstop 0.000122811
1692023-12-12T05:36:13.610Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8aed3472db5f37d6
1702023-12-12T05:36:13.611Z+ chmod a+x tests/bootstrap-8aed3472db5f37d6
1712023-12-12T05:36:13.611Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-12-12T05:36:13.611Z+ ptime -m gunzip
1732023-12-12T05:36:13.611Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-12-12T05:36:15.863Z
1752023-12-12T05:36:15.892Zreal 2.251393420
1762023-12-12T05:36:15.892Zuser 2.100846337
1772023-12-12T05:36:15.892Zsys 0.150137249
1782023-12-12T05:36:15.892Ztrap 0.000172212
1792023-12-12T05:36:15.892Ztflt 0.000000000
1802023-12-12T05:36:15.892Zdflt 0.000000000
1812023-12-12T05:36:15.892Zkflt 0.000000000
1822023-12-12T05:36:15.893Zlock 0.000000000
1832023-12-12T05:36:15.893Zslp 0.000022480
1842023-12-12T05:36:15.893Zlat 0.000023350
1852023-12-12T05:36:15.893Zstop 0.000126041
1862023-12-12T05:36:15.893Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-12-12T05:36:15.893Z+ chmod a+x tests/bootstrap
1882023-12-12T05:36:15.893Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-12-12T05:36:15.893Z+ ptime -m gunzip
1902023-12-12T05:36:15.894Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-fd6b950b6a51f2ac
1912023-12-12T05:36:18.149Z
1922023-12-12T05:36:18.175Zreal 2.314124159
1932023-12-12T05:36:18.176Zuser 2.156897681
1942023-12-12T05:36:18.176Zsys 0.156832925
1952023-12-12T05:36:18.176Ztrap 0.000163550
1962023-12-12T05:36:18.176Ztflt 0.000000000
1972023-12-12T05:36:18.176Zdflt 0.000000000
1982023-12-12T05:36:18.177Zkflt 0.000000000
1992023-12-12T05:36:18.177Zlock 0.000000000
2002023-12-12T05:36:18.177Zslp 0.000023350
2012023-12-12T05:36:18.177Zlat 0.000022700
2022023-12-12T05:36:18.178Zstop 0.000124171
2032023-12-12T05:36:18.178Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-fd6b950b6a51f2ac
2042023-12-12T05:36:18.178Z+ chmod a+x tests/end_to_end_tests-fd6b950b6a51f2ac
2052023-12-12T05:36:18.178Z+ bmat address ls
2062023-12-12T05:36:18.178ZNAME CIDR FIRST LAST COUNT
2072023-12-12T05:36:18.180Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-12-12T05:36:18.181Z++ bmat address ls -f extra -Ho first,last,count
2092023-12-12T05:36:18.181Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-12-12T05:36:18.182Z+ EXTRA_IP_START=10.151.1.164
2112023-12-12T05:36:18.182Z+ EXTRA_IP_END=10.151.1.227
2122023-12-12T05:36:18.182Z+ EXTRA_IP_COUNT=64
2132023-12-12T05:36:18.182Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-12-12T05:36:18.182Z+ EXTRA_IP_BASE=10.151.1
2152023-12-12T05:36:18.184Z+ EXTRA_IP_FOCTET=164
2162023-12-12T05:36:18.185Z+ EXTRA_IP_LOCTET=227
2172023-12-12T05:36:18.185Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-12-12T05:36:18.185Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-12-12T05:36:18.185Z+ DNS_IP1=10.151.1.164
2202023-12-12T05:36:18.186Z+ DNS_IP2=10.151.1.165
2212023-12-12T05:36:18.212Z+ UPLINK_IP=10.151.1.174
2222023-12-12T05:36:18.213Z+ IPPOOL_START=10.151.1.175
2232023-12-12T05:36:18.213Z+ IPPOOL_END=10.151.1.227
2242023-12-12T05:36:18.213Z++ ipadm show-addr -po type,addr
2252023-12-12T05:36:18.213Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-12-12T05:36:18.213Z+ GATEWAY_IP=10.151.1.100
2272023-12-12T05:36:18.213Z+ [[ -n 10.151.1.100 ]]
2282023-12-12T05:36:18.213Z+ ping -s 10.151.1.100 56 1
2292023-12-12T05:36:18.213ZPING 10.151.1.100: 56 data bytes
2302023-12-12T05:36:18.214Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
2312023-12-12T05:36:18.214Z
2322023-12-12T05:36:18.214Z----10.151.1.100 PING Statistics----
2332023-12-12T05:36:18.214Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-12-12T05:36:18.214Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-12-12T05:36:18.214Z++ arp -an
2362023-12-12T05:36:18.214Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-12-12T05:36:18.215Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-12-12T05:36:18.215Z+ routeadm -e ipv4-forwarding -u
2392023-12-12T05:36:18.312Z+ PXA_START=10.151.1.164
2402023-12-12T05:36:18.313Z+ PXA_END=10.151.1.227
2412023-12-12T05:36:18.338Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-12-12T05:36:18.338Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-12-12T05:36:18.453Z+ ZPOOL_VDEV_DIR=/scratch
2442023-12-12T05:36:18.453Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-12-12T05:36:18.484Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-12-12T05:36:18.484Z++ cd ./tools
2472023-12-12T05:36:18.484Z++ pwd
2482023-12-12T05:36:18.484Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-12-12T05:36:18.485Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-12-12T05:36:18.485Z+ SOFTNPU_MODE=zone
2512023-12-12T05:36:18.485Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-12-12T05:36:18.485Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-12-12T05:36:18.485Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-12-12T05:36:18.486Z++ dladm show-phys -p -o LINK
2552023-12-12T05:36:18.486Z++ head -1
2562023-12-12T05:36:18.543Z+ PHYSICAL_LINK=igb0
2572023-12-12T05:36:18.544Z+ echo 'Using igb0 as physical link'
2582023-12-12T05:36:18.580ZUsing igb0 as physical link
2592023-12-12T05:36:18.580Z+ ensure_run_as_root
2602023-12-12T05:36:18.581Z++ id -u
2612023-12-12T05:36:18.581Z+ [[ 0 -ne 0 ]]
2622023-12-12T05:36:18.581Z+ ensure_zpools
2632023-12-12T05:36:18.581Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-12-12T05:36:18.581Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-12-12T05:36:18.581Z+ readarray -t ZPOOLS
2662023-12-12T05:36:18.581Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-12-12T05:36:18.581Z++ sed 's/[ ",]//g'
2682023-12-12T05:36:18.582Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-12-12T05:36:18.584Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-12-12T05:36:18.584ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-12-12T05:36:18.584Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-12-12T05:36:18.584Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-12-12T05:36:18.584Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-12-12T05:36:18.584Z0+0 records in
2752023-12-12T05:36:18.585Z0+0 records out
2762023-12-12T05:36:18.585Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2772023-12-12T05:36:18.585Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-12-12T05:36:18.585Z+ set +x
2792023-12-12T05:36:18.585ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-12-12T05:36:18.622Z++ zpool list -o name
2812023-12-12T05:36:18.622Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-12-12T05:36:18.622Z+ [[ -z '' ]]
2832023-12-12T05:36:18.622Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-12-12T05:36:18.649Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-12-12T05:36:18.649Z+ set +x
2862023-12-12T05:36:18.649ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-12-12T05:36:18.649Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-12-12T05:36:18.650Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-12-12T05:36:18.650ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-12-12T05:36:18.650Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-12-12T05:36:18.650Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-12-12T05:36:18.650Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-12-12T05:36:18.651Z0+0 records in
2942023-12-12T05:36:18.651Z0+0 records out
2952023-12-12T05:36:18.651Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-12-12T05:36:18.651Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-12-12T05:36:18.651Z+ set +x
2982023-12-12T05:36:18.652ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-12-12T05:36:18.652Z++ zpool list -o name
3002023-12-12T05:36:18.652Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-12-12T05:36:18.652Z+ [[ -z '' ]]
3022023-12-12T05:36:18.652Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-12-12T05:36:19.137Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-12-12T05:36:19.137Z+ set +x
3052023-12-12T05:36:19.138ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-12-12T05:36:19.138Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-12-12T05:36:19.138Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-12-12T05:36:19.138Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3092023-12-12T05:36:19.138ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3102023-12-12T05:36:19.138Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-12-12T05:36:19.138Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-12-12T05:36:19.138Z0+0 records in
3132023-12-12T05:36:19.138Z0+0 records out
3142023-12-12T05:36:19.138Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3152023-12-12T05:36:19.139Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-12-12T05:36:19.139Z+ set +x
3172023-12-12T05:36:19.139ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-12-12T05:36:19.139Z++ zpool list -o name
3192023-12-12T05:36:19.139Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-12-12T05:36:19.581Z+ [[ -z '' ]]
3212023-12-12T05:36:19.582Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-12-12T05:36:19.582Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-12-12T05:36:19.582Z+ set +x
3242023-12-12T05:36:19.582ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-12-12T05:36:19.582Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-12-12T05:36:19.582Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-12-12T05:36:19.582ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-12-12T05:36:19.583Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-12-12T05:36:19.583Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-12-12T05:36:19.583Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-12-12T05:36:19.583Z0+0 records in
3322023-12-12T05:36:19.583Z0+0 records out
3332023-12-12T05:36:19.583Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-12-12T05:36:19.584Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-12-12T05:36:19.584Z+ set +x
3362023-12-12T05:36:19.584ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-12-12T05:36:19.584Z++ zpool list -o name
3382023-12-12T05:36:19.584Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-12-12T05:36:19.584Z+ [[ -z '' ]]
3402023-12-12T05:36:19.585Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-12-12T05:36:19.585Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-12-12T05:36:19.585Z+ set +x
3432023-12-12T05:36:19.585ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-12-12T05:36:19.585Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-12-12T05:36:19.585Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-12-12T05:36:19.585ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-12-12T05:36:19.585Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-12-12T05:36:19.586Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-12-12T05:36:19.586Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-12-12T05:36:19.586Z0+0 records in
3512023-12-12T05:36:19.586Z0+0 records out
3522023-12-12T05:36:19.587Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3532023-12-12T05:36:19.587Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-12-12T05:36:19.587Z+ set +x
3552023-12-12T05:36:19.587ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-12-12T05:36:19.587Z++ zpool list -o name
3572023-12-12T05:36:19.587Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-12-12T05:36:19.587Z+ [[ -z '' ]]
3592023-12-12T05:36:19.587Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-12-12T05:36:19.587Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-12-12T05:36:19.587Z+ set +x
3622023-12-12T05:36:19.587ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-12-12T05:36:19.587Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-12-12T05:36:19.588Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-12-12T05:36:19.589ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-12-12T05:36:19.589Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-12-12T05:36:19.589Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-12-12T05:36:19.589Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-12-12T05:36:19.589Z0+0 records in
3702023-12-12T05:36:19.589Z0+0 records out
3712023-12-12T05:36:19.589Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3722023-12-12T05:36:19.589Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-12-12T05:36:19.589Z+ set +x
3742023-12-12T05:36:19.589ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-12-12T05:36:19.589Z++ zpool list -o name
3762023-12-12T05:36:19.589Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-12-12T05:36:19.589Z+ [[ -z '' ]]
3782023-12-12T05:36:19.590Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-12-12T05:36:19.590Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-12-12T05:36:19.590Z+ set +x
3812023-12-12T05:36:19.590ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-12-12T05:36:19.590Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-12-12T05:36:19.590Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-12-12T05:36:19.590ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-12-12T05:36:19.590Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-12-12T05:36:19.590Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-12-12T05:36:19.590Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-12-12T05:36:19.590Z0+0 records in
3892023-12-12T05:36:19.590Z0+0 records out
3902023-12-12T05:36:19.590Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3912023-12-12T05:36:19.591Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-12-12T05:36:19.591Z+ set +x
3932023-12-12T05:36:19.591ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-12-12T05:36:19.591Z++ zpool list -o name
3952023-12-12T05:36:19.591Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-12-12T05:36:19.591Z+ [[ -z '' ]]
3972023-12-12T05:36:19.591Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-12-12T05:36:19.591Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-12-12T05:36:19.591Z+ set +x
4002023-12-12T05:36:19.591ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-12-12T05:36:19.591Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-12-12T05:36:19.591Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-12-12T05:36:19.592ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-12-12T05:36:19.592Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-12-12T05:36:19.592Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-12-12T05:36:19.592Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-12-12T05:36:19.592Z0+0 records in
4082023-12-12T05:36:19.592Z0+0 records out
4092023-12-12T05:36:19.592Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4102023-12-12T05:36:19.592Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-12-12T05:36:19.592Z+ set +x
4122023-12-12T05:36:19.592ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-12-12T05:36:19.592Z++ zpool list -o name
4142023-12-12T05:36:19.593Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-12-12T05:36:19.593Z+ [[ -z '' ]]
4162023-12-12T05:36:19.593Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-12-12T05:36:19.593Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-12-12T05:36:19.593ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4192023-12-12T05:36:19.593ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4202023-12-12T05:36:20.053Z+ set +x
4212023-12-12T05:36:20.053Z+ for ZPOOL in "${ZPOOLS[@]}"
4222023-12-12T05:36:20.054Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4232023-12-12T05:36:20.054Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-12-12T05:36:20.054Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-12-12T05:36:20.054Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-12-12T05:36:20.054Z0+0 records in
4272023-12-12T05:36:20.054Z0+0 records out
4282023-12-12T05:36:20.054Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4292023-12-12T05:36:20.054Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-12-12T05:36:20.054Z+ set +x
4312023-12-12T05:36:20.055ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-12-12T05:36:20.055Z++ zpool list -o name
4332023-12-12T05:36:20.055Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-12-12T05:36:20.055Z+ [[ -z '' ]]
4352023-12-12T05:36:20.056Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-12-12T05:36:20.056Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-12-12T05:36:20.057Z+ set +x
4382023-12-12T05:36:20.057ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-12-12T05:36:20.057Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-12-12T05:36:20.057Z+ readarray -t ZPOOLS
4412023-12-12T05:36:20.057Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-12-12T05:36:20.057Z++ sed 's/[ ",]//g'
4432023-12-12T05:36:20.057Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-12-12T05:36:20.058Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-12-12T05:36:20.058ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-12-12T05:36:20.058Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-12-12T05:36:20.058Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-12-12T05:36:20.058Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-12-12T05:36:20.058Z0+0 records in
4502023-12-12T05:36:20.058Z0+0 records out
4512023-12-12T05:36:20.058Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4522023-12-12T05:36:20.059Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-12-12T05:36:20.059Z+ set +x
4542023-12-12T05:36:20.059ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-12-12T05:36:20.059Z++ zpool list -o name
4562023-12-12T05:36:20.059Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-12-12T05:36:20.059Z+ [[ -z '' ]]
4582023-12-12T05:36:20.059Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-12-12T05:36:20.059Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-12-12T05:36:20.060Z+ set +x
4612023-12-12T05:36:20.060ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-12-12T05:36:20.061ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4632023-12-12T05:36:20.061Z+ for ZPOOL in "${ZPOOLS[@]}"
4642023-12-12T05:36:20.061Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4652023-12-12T05:36:20.061Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-12-12T05:36:20.061Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-12-12T05:36:20.061Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-12-12T05:36:20.061Z0+0 records in
4692023-12-12T05:36:20.061Z0+0 records out
4702023-12-12T05:36:20.061Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4712023-12-12T05:36:20.061Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-12-12T05:36:20.061Z+ set +x
4732023-12-12T05:36:20.062ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-12-12T05:36:20.062Z++ zpool list -o name
4752023-12-12T05:36:20.062Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-12-12T05:36:20.062Z+ [[ -z '' ]]
4772023-12-12T05:36:20.062Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-12-12T05:36:20.062Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-12-12T05:36:20.062Z+ set +x
4802023-12-12T05:36:20.062ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-12-12T05:36:20.062Z+ [[ zone == \z\o\n\e ]]
4822023-12-12T05:36:20.062Z+ ensure_simulated_links igb0
4832023-12-12T05:36:20.063Z+ local PHYSICAL_LINK=igb0
4842023-12-12T05:36:20.063Z+ INDICES=("0" "1")
4852023-12-12T05:36:20.063Z+ for I in "${INDICES[@]}"
4862023-12-12T05:36:20.063Z++ get_simnet_name_if_exists net0
4872023-12-12T05:36:20.064Z++ dladm show-simnet -p -o LINK net0
4882023-12-12T05:36:20.064Z++ echo ''
4892023-12-12T05:36:20.064Z+ [[ -z '' ]]
4902023-12-12T05:36:20.064Z+ dladm create-simnet -t net0
4912023-12-12T05:36:20.064Z+ dladm create-simnet -t sc0_0
4922023-12-12T05:36:20.064Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-12-12T05:36:20.065Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-12-12T05:36:20.065Z+ success 'Simnet net0/sc0_0 exists'
4952023-12-12T05:36:20.065Z+ set +x
4962023-12-12T05:36:20.065ZSimnet net0/sc0_0 exists
4972023-12-12T05:36:20.065Z+ for I in "${INDICES[@]}"
4982023-12-12T05:36:20.065Z++ get_simnet_name_if_exists net1
4992023-12-12T05:36:20.065Z++ dladm show-simnet -p -o LINK net1
5002023-12-12T05:36:20.065Z++ echo ''
5012023-12-12T05:36:20.065Z+ [[ -z '' ]]
5022023-12-12T05:36:20.066Z+ dladm create-simnet -t net1
5032023-12-12T05:36:20.066Z+ dladm create-simnet -t sc1_0
5042023-12-12T05:36:20.067Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-12-12T05:36:20.067Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-12-12T05:36:20.067Z+ success 'Simnet net1/sc1_0 exists'
5072023-12-12T05:36:20.067Z+ set +x
5082023-12-12T05:36:20.067ZSimnet net1/sc1_0 exists
5092023-12-12T05:36:20.067Z++ get_vnic_name_if_exists sc0_1
5102023-12-12T05:36:20.067Z++ dladm show-vnic -p -o LINK sc0_1
5112023-12-12T05:36:20.067Z++ echo ''
5122023-12-12T05:36:20.068Z+ [[ -z '' ]]
5132023-12-12T05:36:20.068Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-12-12T05:36:20.068Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-12-12T05:36:20.068Z+ success 'Vnic sc0_1 exists'
5162023-12-12T05:36:20.069Z+ set +x
5172023-12-12T05:36:20.069ZVnic sc0_1 exists
5182023-12-12T05:36:20.069Z+ warn_if_no_proxy_arp
5192023-12-12T05:36:20.069Z+ [[ -v PXA_START ]]
5202023-12-12T05:36:20.096Z+ [[ -v PXA_END ]]
5212023-12-12T05:36:20.096Z+ ensure_softnpu_zone
5222023-12-12T05:36:20.096Z+ zoneadm list
5232023-12-12T05:36:20.096Z+ grep -q sidecar_softnpu
5242023-12-12T05:36:20.096Z+ [[ -f out/npuzone/npuzone ]]
5252023-12-12T05:36:20.097Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit 45ed98fea5824feb4d42f45bbf218e597dc9fc58 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5262023-12-12T05:36:20.097Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5272023-12-12T05:36:31.145Zdownloading softnpu ...done
5282023-12-12T05:36:31.178Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/libsidecar_lite.so.sha256.txt
5292023-12-12T05:36:40.312Zdownloading asic_program.so ...done
5302023-12-12T05:36:40.752Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/scadm.sha256.txt
5312023-12-12T05:36:50.818Zdownloading scadm ...done
5322023-12-12T05:36:50.891Zcreating port tfportrear0_0 for sc0_0
5332023-12-12T05:36:50.891Zcreating port tfportqsfp0_0 for sc0_1
5342023-12-12T05:36:50.891Zsoftnpu zone setup
5352023-12-12T05:36:50.891Zconfigure zone
5362023-12-12T05:36:50.921Zinstall zone
5372023-12-12T05:36:51.950Zboot zone
5382023-12-12T05:36:54.990Z[sidecar_softnpu] svcs milestone/network
5392023-12-12T05:36:55.021ZSTATE STIME FMRI
5402023-12-12T05:36:55.021Zonline 5:36:53 svc:/milestone/network:default
5412023-12-12T05:36:55.021Z[sidecar_softnpu] /softnpu/npu start
5422023-12-12T05:36:55.477Z
5432023-12-12T05:36:55.478Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-12-12T05:36:55.478Z+ GATEWAY_IP=10.151.1.100
5452023-12-12T05:36:55.489Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-12-12T05:36:55.489ZUsing 10.151.1.100 as gateway ip
5472023-12-12T05:36:55.490Z+ [[ ! -v GATEWAY_MAC ]]
5482023-12-12T05:36:55.490Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-12-12T05:36:55.490Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-12-12T05:36:55.491Z+++ echo 18:c0:4d:0d:9f:b2
5512023-12-12T05:36:55.491Z+ COUNT=1
5522023-12-12T05:36:55.491Z+ [[ 1 -ne 1 ]]
5532023-12-12T05:36:55.491Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-12-12T05:36:55.491ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-12-12T05:36:55.492Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-12-12T05:36:55.492Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-12-12T05:36:55.492Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-12-12T05:36:55.492Z+ [[ -v PXA_START ]]
5592023-12-12T05:36:55.492Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-12-12T05:36:55.493Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-12-12T05:36:55.493Z+ z_scadm dump-state
5622023-12-12T05:36:55.493Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-12-12T05:36:55.494Zlocal v6:
5642023-12-12T05:36:55.494Zlocal v4:
5652023-12-12T05:36:55.494Zrouter v6:
5662023-12-12T05:36:55.494Zrouter v4_idx:
5672023-12-12T05:36:55.495Zrouter v4_routes:
5682023-12-12T05:36:55.495Zresolver v4:
5692023-12-12T05:36:55.495Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-12-12T05:36:55.495Zresolver v6:
5712023-12-12T05:36:55.496Znat v4:
5722023-12-12T05:36:55.496Znat_v6:
5732023-12-12T05:36:55.496Zport_mac:
5742023-12-12T05:36:55.497Zproxy arp:
5752023-12-12T05:36:55.497Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-12-12T05:36:55.497Z+ success 'softnpu zone exists'
5772023-12-12T05:36:55.497Z+ set +x
5782023-12-12T05:36:55.498Zsoftnpu zone exists
5792023-12-12T05:36:55.498Z
5802023-12-12T05:36:55.498Zreal 36.684643635
5812023-12-12T05:36:55.499Zuser 0.713796332
5822023-12-12T05:36:55.499Zsys 1.606540000
5832023-12-12T05:36:55.499Ztrap 0.000098243
5842023-12-12T05:36:55.499Ztflt 0.000149378
5852023-12-12T05:36:55.499Zdflt 0.001049255
5862023-12-12T05:36:55.500Zkflt 0.000000000
5872023-12-12T05:36:55.500Zlock 28:40.621018915
5882023-12-12T05:36:55.500Zslp 1:49.777486363
5892023-12-12T05:36:55.500Zlat 0.083164283
5902023-12-12T05:36:55.501Zstop 0.377151195
5912023-12-12T05:36:55.501Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-12-12T05:36:55.501Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-12-12T05:36:55.501Z+ SILO_NAME=recovery
5942023-12-12T05:36:55.502Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-12T05:36:55.502Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-12-12T05:36:55.502Z+ sed -i~ '
5972023-12-12T05:36:55.502Z /^external_dns_ips/c\
5982023-12-12T05:36:55.503Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-12-12T05:36:55.503Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-12-12T05:36:55.504Z /^first/c\
6012023-12-12T05:36:55.504Zfirst = "10.151.1.164"
6022023-12-12T05:36:55.505Z /^last/c\
6032023-12-12T05:36:55.505Zlast = "10.151.1.173"
6042023-12-12T05:36:55.505Z }
6052023-12-12T05:36:55.505Z /^\[rack_network_config/,/^$/ {
6062023-12-12T05:36:55.506Z /^infra_ip_first/c\
6072023-12-12T05:36:55.506Zinfra_ip_first = "10.151.1.174"
6082023-12-12T05:36:55.506Z /^infra_ip_last/c\
6092023-12-12T05:36:55.507Zinfra_ip_last = "10.151.1.174"
6102023-12-12T05:36:55.507Z }
6112023-12-12T05:36:55.507Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-12-12T05:36:55.507Z /^routes/c\
6132023-12-12T05:36:55.508Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6142023-12-12T05:36:55.508Z /^addresses/c\
6152023-12-12T05:36:55.508Zaddresses = \["10.151.1.174/32"\]
6162023-12-12T05:36:55.508Z }
6172023-12-12T05:36:55.508Z' pkg/config-rss.toml
6182023-12-12T05:36:55.509Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-12-12T05:36:55.509Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-12-12T05:36:55.509Z+++ pkg/config-rss.toml Tue Dec 12 05:36:55 2023
6212023-12-12T05:36:55.510Z@@ -41,7 +41,7 @@
6222023-12-12T05:36:55.510Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-12-12T05:36:55.510Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-12-12T05:36:55.510Z # below.
6252023-12-12T05:36:55.511Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-12-12T05:36:55.511Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6272023-12-12T05:36:55.511Z
6282023-12-12T05:36:55.512Z # Initial TLS certificates for the external API
6292023-12-12T05:36:55.512Z #
6302023-12-12T05:36:55.512Z@@ -77,8 +77,8 @@
6312023-12-12T05:36:55.512Z #
6322023-12-12T05:36:55.513Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-12-12T05:36:55.513Z [[internal_services_ip_pool_ranges]]
6342023-12-12T05:36:55.513Z-first = "192.168.1.20"
6352023-12-12T05:36:55.513Z-last = "192.168.1.29"
6362023-12-12T05:36:55.514Z+first = "10.151.1.164"
6372023-12-12T05:36:55.514Z+last = "10.151.1.173"
6382023-12-12T05:36:55.514Z
6392023-12-12T05:36:55.514Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-12-12T05:36:55.514Z # parameters should be provided to the control plane via wicket, but we need to
6412023-12-12T05:36:55.515Z@@ -92,8 +92,8 @@
6422023-12-12T05:36:55.957Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-12-12T05:36:55.957Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-12-12T05:36:55.957Z # With softnpu, only one address is used.
6452023-12-12T05:36:55.957Z-infra_ip_first = "192.168.1.30"
6462023-12-12T05:36:55.957Z-infra_ip_last = "192.168.1.30"
6472023-12-12T05:36:55.957Z+infra_ip_first = "10.151.1.174"
6482023-12-12T05:36:55.957Z+infra_ip_last = "10.151.1.174"
6492023-12-12T05:36:55.958Z
6502023-12-12T05:36:55.958Z # Configurations for BGP routers to run on the scrimlets.
6512023-12-12T05:36:55.958Z bgp = []
6522023-12-12T05:36:55.958Z@@ -101,9 +101,9 @@
6532023-12-12T05:36:55.958Z # You can configure multiple uplinks by repeating the following stanza
6542023-12-12T05:36:55.959Z [[rack_network_config.ports]]
6552023-12-12T05:36:55.959Z # Routes associated with this port.
6562023-12-12T05:36:55.959Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-12-12T05:36:55.959Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6582023-12-12T05:36:55.959Z # Addresses associated with this port.
6592023-12-12T05:36:55.959Z-addresses = ["192.168.1.30/32"]
6602023-12-12T05:36:55.959Z+addresses = ["10.151.1.174/32"]
6612023-12-12T05:36:55.959Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-12-12T05:36:55.959Z port = "qsfp0"
6632023-12-12T05:36:55.959Z # The speed of this port.
6642023-12-12T05:36:55.960Z+ true
6652023-12-12T05:36:55.960Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-12-12T05:36:55.960Za pkg/config-rss.toml 7K
6672023-12-12T05:36:55.960Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-12-12T05:36:55.960Z+ TLS_NAME=recovery.sys.oxide.test
6692023-12-12T05:36:55.960Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-12-12T05:36:55.988Z................+.+.........+...........+....+..+...+.+...........+....+.....+.+...+..+..........+.........+.........+........+.............+..+....+...+...+.........+......+........+.+........+...+...+.+......+.....+..........+...+.........+..+...+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+......+.+....................+.......+..+.+...+.....+..........+..+.............+.....+....+.....+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..+.......+......+......+.........+...........+.+............+.....+......+.+.....+.+...+..+...+......+.+.................+...+...+....+............+......+.........+........+...+....+...+.....+...+..................+....+.....+.......+...+........+...+.+........+......+..............................+.+.....+.+........+......+.........+....+...+...+......+.....+.....................+...+....+..+....+......+......+...+..+..........+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-12-12T05:36:55.988Z...+.........+...+.....+....+.....+...+.+.....+.......+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..+....+...+............+..............+.+......+........+....+..+.+........+......+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...........................+...............+....+.....+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-12T05:36:55.989Z-----
6732023-12-12T05:36:55.989Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-12-12T05:36:55.989Za pkg/initial-tls-cert.pem 2K
6752023-12-12T05:36:55.989Za pkg/initial-tls-key.pem 4K
6762023-12-12T05:36:55.989Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-12-12T05:36:55.989Z+ rmdir pkg
6782023-12-12T05:36:55.990Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-12-12T05:36:55.990Z+ pfexec mkdir -p /usr/oxide
6802023-12-12T05:36:55.990Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6812023-12-12T05:36:56.559Z+ pfexec chmod +x /usr/oxide/oxide
6822023-12-12T05:36:56.584Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6832023-12-12T05:37:14.885Z+ OMICRON_NO_UNINSTALL=1
6842023-12-12T05:37:14.886Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6852023-12-12T05:37:14.931ZDec 12 05:37:15.119 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6862023-12-12T05:37:14.931ZDec 12 05:37:15.121 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6872023-12-12T05:37:14.931ZDec 12 05:37:15.121 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6882023-12-12T05:37:14.931ZDec 12 05:37:15.121 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6892023-12-12T05:37:14.932ZDec 12 05:37:15.121 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6902023-12-12T05:37:14.932ZDec 12 05:37:15.121 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6912023-12-12T05:37:14.932ZDec 12 05:37:15.121 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6922023-12-12T05:37:14.933ZDec 12 05:37:15.121 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6932023-12-12T05:37:14.933ZDec 12 05:37:15.121 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6942023-12-12T05:37:14.933ZDec 12 05:37:15.121 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6952023-12-12T05:37:14.933ZDec 12 05:37:15.121 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6962023-12-12T05:37:14.933ZDec 12 05:37:15.121 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6972023-12-12T05:37:14.933ZDec 12 05:37:15.121 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6982023-12-12T05:37:14.933ZDec 12 05:37:15.121 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6992023-12-12T05:37:14.933ZDec 12 05:37:15.122 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7002023-12-12T05:37:14.933ZDec 12 05:37:15.122 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7012023-12-12T05:37:14.933ZDec 12 05:37:15.122 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7022023-12-12T05:37:15.134ZDec 12 05:37:15.354 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7032023-12-12T05:37:15.168ZDec 12 05:37:15.389 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7042023-12-12T05:37:15.238ZDec 12 05:37:15.459 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7052023-12-12T05:37:15.370Z
7062023-12-12T05:37:15.400Zreal 0.481397643
7072023-12-12T05:37:15.400Zuser 0.099242461
7082023-12-12T05:37:15.400Zsys 1.927067643
7092023-12-12T05:37:15.400Ztrap 0.000009859
7102023-12-12T05:37:15.400Ztflt 0.000034839
7112023-12-12T05:37:15.401Zdflt 0.000315166
7122023-12-12T05:37:15.401Zkflt 0.000000000
7132023-12-12T05:37:15.401Zlock 43.217762318
7142023-12-12T05:37:15.401Zslp 0.699756127
7152023-12-12T05:37:15.401Zlat 0.003241005
7162023-12-12T05:37:15.401Zstop 0.209376053
7172023-12-12T05:37:15.401Z+ retry=0
7182023-12-12T05:37:15.401Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7192023-12-12T05:37:15.401Z+ [[ 0 -gt 30 ]]
7202023-12-12T05:37:15.401Z+ sleep 1
7212023-12-12T05:37:16.388Z+ retry=1
7222023-12-12T05:37:16.389Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232023-12-12T05:37:16.426Z+ [[ 1 -gt 30 ]]
7242023-12-12T05:37:16.426Z+ sleep 1
7252023-12-12T05:37:17.406Z+ retry=2
7262023-12-12T05:37:17.406Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272023-12-12T05:37:17.440Z+ [[ 2 -gt 30 ]]
7282023-12-12T05:37:17.441Z+ sleep 1
7292023-12-12T05:37:18.419Z+ retry=3
7302023-12-12T05:37:18.419Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312023-12-12T05:37:18.452Z+ [[ 3 -gt 30 ]]
7322023-12-12T05:37:18.452Z+ sleep 1
7332023-12-12T05:37:19.435Z+ retry=4
7342023-12-12T05:37:19.435Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352023-12-12T05:37:39.262Z+ [[ 4 -gt 30 ]]
7362023-12-12T05:37:39.262Z+ sleep 1
7372023-12-12T05:37:40.268Z+ retry=5
7382023-12-12T05:37:40.268Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7392023-12-12T05:37:40.308Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7402023-12-12T05:37:40.338Zlocal v6:
7412023-12-12T05:37:40.339Zfd00:99::1
7422023-12-12T05:37:40.339Zfe80::4068:feff:fe7f:72e3
7432023-12-12T05:37:40.340Zlocal v4:
7442023-12-12T05:37:40.340Z10.151.1.174
7452023-12-12T05:37:40.340Zrouter v6:
7462023-12-12T05:37:40.340Zrouter v4_idx:
7472023-12-12T05:37:40.341Z0.0.0.0/0 -> 0
7482023-12-12T05:37:40.341Zrouter v4_routes:
7492023-12-12T05:37:40.341Z0 -> 10.151.1.100 (2)
7502023-12-12T05:37:40.342Zresolver v4:
7512023-12-12T05:37:40.342Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7522023-12-12T05:37:40.342Zresolver v6:
7532023-12-12T05:37:40.342Zfe80::4068:feff:fe7f:72e3 -> 42:68:fe:7f:72:e3
7542023-12-12T05:37:40.342Zfe80::10f3:a1ff:fe12:a91f -> 12:f3:a1:12:a9:1f
7552023-12-12T05:37:40.342Znat v4:
7562023-12-12T05:37:40.343Znat_v6:
7572023-12-12T05:37:40.343Zport_mac:
7582023-12-12T05:37:40.343Z1: a8:40:25:39:9b:3e
7592023-12-12T05:37:40.344Z1000: a8:40:25:39:9b:3d
7602023-12-12T05:37:40.344Zproxy arp:
7612023-12-12T05:37:40.344Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7622023-12-12T05:37:40.344Z+ retry=0
7632023-12-12T05:37:40.345Z+++ zoneadm list -n
7642023-12-12T05:37:40.345Z+++ grep oxz_ntp
7652023-12-12T05:37:40.345Z++ pfexec svcs -z -Hostate oxide/ntp
7662023-12-12T05:37:40.346Zsvcs: zone '-Hostate': entity not found
7672023-12-12T05:37:40.346Z++ true
7682023-12-12T05:37:40.346Z+ [[ '' != online ]]
7692023-12-12T05:37:40.347Z+ [[ 0 -gt 60 ]]
7702023-12-12T05:37:40.347Z+ sleep 1
7712023-12-12T05:37:41.332Z+ retry=1
7722023-12-12T05:37:41.365Z+++ zoneadm list -n
7732023-12-12T05:37:41.365Z+++ grep oxz_ntp
7742023-12-12T05:37:41.366Z++ pfexec svcs -z -Hostate oxide/ntp
7752023-12-12T05:37:41.366Zsvcs: zone '-Hostate': entity not found
7762023-12-12T05:37:41.366Z++ true
7772023-12-12T05:37:41.366Z+ [[ '' != online ]]
7782023-12-12T05:37:41.366Z+ [[ 1 -gt 60 ]]
7792023-12-12T05:37:41.366Z+ sleep 1
7802023-12-12T05:37:42.370Z+ retry=2
7812023-12-12T05:37:42.395Z+++ zoneadm list -n
7822023-12-12T05:37:42.395Z+++ grep oxz_ntp
7832023-12-12T05:37:42.395Z++ pfexec svcs -z -Hostate oxide/ntp
7842023-12-12T05:37:42.395Zsvcs: zone '-Hostate': entity not found
7852023-12-12T05:37:42.395Z++ true
7862023-12-12T05:37:42.395Z+ [[ '' != online ]]
7872023-12-12T05:37:42.395Z+ [[ 2 -gt 60 ]]
7882023-12-12T05:37:42.395Z+ sleep 1
7892023-12-12T05:37:43.377Z+ retry=3
7902023-12-12T05:37:43.409Z+++ zoneadm list -n
7912023-12-12T05:37:43.409Z+++ grep oxz_ntp
7922023-12-12T05:37:43.409Z++ pfexec svcs -z oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3 -Hostate oxide/ntp
7932023-12-12T05:37:43.409Zsvcs: Could not bind to repository server for zone oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3: repository server unavailable
7942023-12-12T05:37:43.409Z++ true
7952023-12-12T05:37:43.409Z+ [[ '' != online ]]
7962023-12-12T05:37:43.409Z+ [[ 3 -gt 60 ]]
7972023-12-12T05:37:43.409Z+ sleep 1
7982023-12-12T05:37:44.416Z+ retry=4
7992023-12-12T05:37:44.457Z+++ zoneadm list -n
8002023-12-12T05:37:44.458Z+++ grep oxz_ntp
8012023-12-12T05:37:44.459Z++ pfexec svcs -z oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3 -Hostate oxide/ntp
8022023-12-12T05:37:44.459Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8032023-12-12T05:37:44.459Z++ true
8042023-12-12T05:37:44.460Z+ [[ '' != online ]]
8052023-12-12T05:37:44.460Z+ [[ 4 -gt 60 ]]
8062023-12-12T05:37:44.460Z+ sleep 1
8072023-12-12T05:37:45.491Z+ retry=5
8082023-12-12T05:37:45.529Z+++ zoneadm list -n
8092023-12-12T05:37:45.529Z+++ grep oxz_ntp
8102023-12-12T05:37:45.529Z++ pfexec svcs -z oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3 -Hostate oxide/ntp
8112023-12-12T05:37:45.529Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8122023-12-12T05:37:45.530Z++ true
8132023-12-12T05:37:45.530Z+ [[ '' != online ]]
8142023-12-12T05:37:45.530Z+ [[ 5 -gt 60 ]]
8152023-12-12T05:37:45.530Z+ sleep 1
8162023-12-12T05:37:46.454Z+ retry=6
8172023-12-12T05:37:46.553Z+++ zoneadm list -n
8182023-12-12T05:37:46.553Z+++ grep oxz_ntp
8192023-12-12T05:37:46.553Z++ pfexec svcs -z oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3 -Hostate oxide/ntp
8202023-12-12T05:37:46.553Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8212023-12-12T05:37:46.553Z++ true
8222023-12-12T05:37:46.553Z+ [[ '' != online ]]
8232023-12-12T05:37:46.553Z+ [[ 6 -gt 60 ]]
8242023-12-12T05:37:46.553Z+ sleep 1
8252023-12-12T05:37:47.485Z+ retry=7
8262023-12-12T05:37:47.920Z+++ zoneadm list -n
8272023-12-12T05:37:47.920Z+++ grep oxz_ntp
8282023-12-12T05:37:47.920Z++ pfexec svcs -z oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3 -Hostate oxide/ntp
8292023-12-12T05:37:47.920Z+ [[ disabled != online ]]
8302023-12-12T05:37:47.920Z+ [[ 7 -gt 60 ]]
8312023-12-12T05:37:47.921Z+ sleep 1
8322023-12-12T05:37:48.506Z+ retry=8
8332023-12-12T05:37:48.536Z+++ zoneadm list -n
8342023-12-12T05:37:48.537Z+++ grep oxz_ntp
8352023-12-12T05:37:48.537Z++ pfexec svcs -z oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3 -Hostate oxide/ntp
8362023-12-12T05:37:48.537Z+ [[ disabled != online ]]
8372023-12-12T05:37:48.537Z+ [[ 8 -gt 60 ]]
8382023-12-12T05:37:48.537Z+ sleep 1
8392023-12-12T05:37:49.533Z+ retry=9
8402023-12-12T05:37:49.569Z+++ zoneadm list -n
8412023-12-12T05:37:49.570Z+++ grep oxz_ntp
8422023-12-12T05:37:49.570Z++ pfexec svcs -z oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3 -Hostate oxide/ntp
8432023-12-12T05:37:49.570Z+ [[ disabled != online ]]
8442023-12-12T05:37:49.570Z+ [[ 9 -gt 60 ]]
8452023-12-12T05:37:49.570Z+ sleep 1
8462023-12-12T05:37:50.550Z+ retry=10
8472023-12-12T05:37:50.578Z+++ zoneadm list -n
8482023-12-12T05:37:50.578Z+++ grep oxz_ntp
8492023-12-12T05:37:50.579Z++ pfexec svcs -z oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3 -Hostate oxide/ntp
8502023-12-12T05:37:50.579Z+ [[ disabled != online ]]
8512023-12-12T05:37:50.583Z+ [[ 10 -gt 60 ]]
8522023-12-12T05:37:50.583Z+ sleep 1
8532023-12-12T05:37:51.567Z+ retry=11
8542023-12-12T05:37:51.594Z+++ zoneadm list -n
8552023-12-12T05:37:51.594Z+++ grep oxz_ntp
8562023-12-12T05:37:51.595Z++ pfexec svcs -z oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3 -Hostate oxide/ntp
8572023-12-12T05:37:51.595Z+ [[ disabled != online ]]
8582023-12-12T05:37:51.595Z+ [[ 11 -gt 60 ]]
8592023-12-12T05:37:51.595Z+ sleep 1
8602023-12-12T05:37:52.588Z+ retry=12
8612023-12-12T05:37:52.612Z+++ zoneadm list -n
8622023-12-12T05:37:52.612Z+++ grep oxz_ntp
8632023-12-12T05:37:52.612Z++ pfexec svcs -z oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3 -Hostate oxide/ntp
8642023-12-12T05:37:52.612Z+ [[ disabled != online ]]
8652023-12-12T05:37:52.612Z+ [[ 12 -gt 60 ]]
8662023-12-12T05:37:52.612Z+ sleep 1
8672023-12-12T05:37:53.625Z+ retry=13
 
8682023-12-12T05:37:53.657Zbuskin console login: Dec 12 05:37:53 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8692023-12-12T05:37:54.073Z+++ zoneadm list -n
8702023-12-12T05:37:54.074Z+++ grep oxz_ntp
8712023-12-12T05:37:54.074Z++ pfexec svcs -z oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3 -Hostate oxide/ntp
8722023-12-12T05:37:54.074Z+ [[ disabled != online ]]
8732023-12-12T05:37:54.074Z+ [[ 13 -gt 60 ]]
8742023-12-12T05:37:54.074Z+ sleep 1
8752023-12-12T05:37:54.633Z+ retry=14
8762023-12-12T05:37:54.668Z+++ zoneadm list -n
8772023-12-12T05:37:54.669Z+++ grep oxz_ntp
8782023-12-12T05:37:54.669Z++ pfexec svcs -z oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3 -Hostate oxide/ntp
8792023-12-12T05:37:54.669Z+ [[ online != online ]]
8802023-12-12T05:37:54.669Z+ echo 'Waited for chrony: 14s'
8812023-12-12T05:37:54.669Z+ export RUST_BACKTRACE=1
8822023-12-12T05:37:54.669ZWaited for chrony: 14s
8832023-12-12T05:37:54.670Z+ RUST_BACKTRACE=1
8842023-12-12T05:37:54.670Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8852023-12-12T05:37:54.670Z++ ./tests/bootstrap
8862023-12-12T05:37:55.115ZUsing external DNS server at 10.151.1.164:53
8872023-12-12T05:37:55.115Z2023-12-12 05:37:54.905287833 UTC: attempting to log into API
8882023-12-12T05:38:09.738Z2023-12-12 05:38:09.472365443 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8892023-12-12T05:38:10.726Z2023-12-12 05:38:10.473334567 UTC: attempting to log into API
8902023-12-12T05:38:25.759Z2023-12-12 05:38:25.505615996 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8912023-12-12T05:38:26.758Z2023-12-12 05:38:26.506415468 UTC: attempting to log into API
8922023-12-12T05:38:41.789Z2023-12-12 05:38:41.539248324 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8932023-12-12T05:38:43.218Z2023-12-12 05:38:42.541146108 UTC: attempting to log into API
8942023-12-12T05:38:57.837Z2023-12-12 05:38:57.574151567 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8952023-12-12T05:38:58.834Z2023-12-12 05:38:58.575044909 UTC: attempting to log into API
8962023-12-12T05:39:13.864Z2023-12-12 05:39:14.025738881 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8972023-12-12T05:39:14.860Z2023-12-12 05:39:15.026731714 UTC: attempting to log into API
8982023-12-12T05:39:29.895Z2023-12-12 05:39:30.059284137 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8992023-12-12T05:39:30.892Z2023-12-12 05:39:31.061266923 UTC: attempting to log into API
9002023-12-12T05:39:45.925Z2023-12-12 05:39:46.093281919 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9012023-12-12T05:39:46.938Z2023-12-12 05:39:47.094930601 UTC: attempting to log into API
9022023-12-12T05:40:01.960Z2023-12-12 05:40:02.127711046 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9032023-12-12T05:40:02.969Z2023-12-12 05:40:03.128469627 UTC: attempting to log into API
9042023-12-12T05:40:18.003Z2023-12-12 05:40:18.126234941 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9052023-12-12T05:40:18.998Z2023-12-12 05:40:19.124389406 UTC: attempting to log into API
9062023-12-12T05:40:34.030Z2023-12-12 05:40:34.112925472 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9072023-12-12T05:40:35.028Z2023-12-12 05:40:35.110779364 UTC: attempting to log into API
9082023-12-12T05:40:50.061Z2023-12-12 05:40:50.098458990 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9092023-12-12T05:40:51.061Z2023-12-12 05:40:51.096519144 UTC: attempting to log into API
9102023-12-12T05:41:06.094Z2023-12-12 05:41:06.084224940 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9112023-12-12T05:41:07.094Z2023-12-12 05:41:07.081932400 UTC: attempting to log into API
9122023-12-12T05:41:22.126Z2023-12-12 05:41:22.102633356 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9132023-12-12T05:41:23.127Z2023-12-12 05:41:23.104416651 UTC: attempting to log into API
9142023-12-12T05:41:38.162Z2023-12-12 05:41:38.147160848 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9152023-12-12T05:41:39.164Z2023-12-12 05:41:39.148992601 UTC: attempting to log into API
9162023-12-12T05:41:54.212Z2023-12-12 05:41:54.181948939 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9172023-12-12T05:41:55.198Z2023-12-12 05:41:55.183033334 UTC: attempting to log into API
9182023-12-12T05:42:10.232Z2023-12-12 05:42:10.216121553 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9192023-12-12T05:42:11.233Z2023-12-12 05:42:11.217915825 UTC: attempting to log into API
9202023-12-12T05:42:26.274Z2023-12-12 05:42:26.259573692 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9212023-12-12T05:42:27.283Z2023-12-12 05:42:27.260735597 UTC: attempting to log into API
 
9222023-12-12T05:42:36.340ZDec 12 05:42:35 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9232023-12-12T05:42:42.309Z2023-12-12 05:42:42.294875732 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9242023-12-12T05:42:42.957ZDec 12 05:42:42 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9252023-12-12T05:42:43.310Z2023-12-12 05:42:43.295725334 UTC: attempting to log into API
 
9262023-12-12T05:42:48.544ZDec 12 05:42:48 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9272023-12-12T05:42:55.163ZDec 12 05:42:54 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9282023-12-12T05:42:58.344Z2023-12-12 05:42:58.329713947 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9292023-12-12T05:42:59.346Z2023-12-12 05:42:59.331399848 UTC: attempting to log into API
 
9302023-12-12T05:43:01.250ZDec 12 05:43:00 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9312023-12-12T05:43:14.378Z2023-12-12 05:43:14.364125726 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9322023-12-12T05:43:15.379Z2023-12-12 05:43:15.364892977 UTC: attempting to log into API
9332023-12-12T05:43:26.170Z2023-12-12 05:43:26.156952358 UTC: login succeeded
9342023-12-12T05:43:26.222Zwaiting for nexus to come up...
9352023-12-12T05:43:26.273Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9362023-12-12T05:43:26.318Zensuring datasets are ready...
9372023-12-12T05:43:29.070Zlet's roll.
9382023-12-12T05:43:29.096Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
9392023-12-12T05:43:29.096ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
9402023-12-12T05:43:29.096ZOXIDE_TOKEN=oxide-token-9ee252bb99a943875629298757ee69530389a4fa'
9412023-12-12T05:43:29.096Z++ OXIDE_HOST=https://recovery.sys.oxide.test
9422023-12-12T05:43:29.096Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
9432023-12-12T05:43:29.096Z++ OXIDE_TOKEN=oxide-token-9ee252bb99a943875629298757ee69530389a4fa
9442023-12-12T05:43:29.096Z+ export OXIDE_HOST OXIDE_TOKEN
9452023-12-12T05:43:29.096Z+ retry=0
9462023-12-12T05:43:29.096Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9472023-12-12T05:43:29.096Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9482023-12-12T05:43:29.096Z+ [[ 0 -gt 60 ]]
9492023-12-12T05:43:29.096Z+ sleep 1
9502023-12-12T05:43:30.099Z+ retry=1
9512023-12-12T05:43:30.100Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9522023-12-12T05:43:30.124Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9532023-12-12T05:43:30.124Z+ [[ 1 -gt 60 ]]
9542023-12-12T05:43:30.124Z+ sleep 1
9552023-12-12T05:43:31.114Z+ retry=2
9562023-12-12T05:43:31.114Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9572023-12-12T05:43:31.139Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9582023-12-12T05:43:31.139Z+ [[ 2 -gt 60 ]]
9592023-12-12T05:43:31.139Z+ sleep 1
9602023-12-12T05:43:32.130Z+ retry=3
9612023-12-12T05:43:32.130Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9622023-12-12T05:43:32.154Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9632023-12-12T05:43:32.154Z+ [[ 3 -gt 60 ]]
9642023-12-12T05:43:32.154Z+ sleep 1
9652023-12-12T05:43:33.147Z+ retry=4
9662023-12-12T05:43:33.147Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9672023-12-12T05:43:33.173Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9682023-12-12T05:43:33.173Z+ [[ 4 -gt 60 ]]
9692023-12-12T05:43:33.173Z+ sleep 1
9702023-12-12T05:43:34.168Z+ retry=5
9712023-12-12T05:43:34.168Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9722023-12-12T05:43:34.198Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9732023-12-12T05:43:34.198Z+ [[ 5 -gt 60 ]]
9742023-12-12T05:43:34.198Z+ sleep 1
9752023-12-12T05:43:35.178Z+ retry=6
9762023-12-12T05:43:35.178Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9772023-12-12T05:43:35.212Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9782023-12-12T05:43:35.213Z+ [[ 6 -gt 60 ]]
9792023-12-12T05:43:35.213Z+ sleep 1
9802023-12-12T05:43:36.193Z+ retry=7
9812023-12-12T05:43:36.193Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9822023-12-12T05:43:36.217Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9832023-12-12T05:43:36.217Z+ [[ 7 -gt 60 ]]
9842023-12-12T05:43:36.218Z+ sleep 1
9852023-12-12T05:43:37.218Z+ retry=8
9862023-12-12T05:43:37.218Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9872023-12-12T05:43:37.251Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9882023-12-12T05:43:37.252Z+ [[ 8 -gt 60 ]]
9892023-12-12T05:43:37.252Z+ sleep 1
9902023-12-12T05:43:38.225Z+ retry=9
9912023-12-12T05:43:38.225Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9922023-12-12T05:43:38.249Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9932023-12-12T05:43:38.249Z+ [[ 9 -gt 60 ]]
9942023-12-12T05:43:38.249Z+ sleep 1
9952023-12-12T05:43:39.252Z+ retry=10
9962023-12-12T05:43:39.252Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9972023-12-12T05:43:39.285Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9982023-12-12T05:43:39.285Z+ [[ 10 -gt 60 ]]
9992023-12-12T05:43:39.287Z+ sleep 1
10002023-12-12T05:43:40.259Z+ retry=11
10012023-12-12T05:43:40.260Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10022023-12-12T05:43:40.288Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10032023-12-12T05:43:40.289Z+ [[ 11 -gt 60 ]]
10042023-12-12T05:43:40.289Z+ sleep 1
10052023-12-12T05:43:41.683Z+ retry=12
10062023-12-12T05:43:41.684Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10072023-12-12T05:43:41.716Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10082023-12-12T05:43:41.716Z+ [[ 12 -gt 60 ]]
10092023-12-12T05:43:41.716Z+ sleep 1
10102023-12-12T05:43:42.288Z+ retry=13
10112023-12-12T05:43:42.288Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10122023-12-12T05:43:42.312Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10132023-12-12T05:43:42.312Z+ [[ 13 -gt 60 ]]
10142023-12-12T05:43:42.312Z+ sleep 1
10152023-12-12T05:43:43.307Z+ retry=14
10162023-12-12T05:43:43.307Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10172023-12-12T05:43:43.333Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10182023-12-12T05:43:43.333Z+ [[ 14 -gt 60 ]]
10192023-12-12T05:43:43.333Z+ sleep 1
10202023-12-12T05:43:44.327Z+ retry=15
10212023-12-12T05:43:44.327Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10222023-12-12T05:43:44.352Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10232023-12-12T05:43:44.352Z+ [[ 15 -gt 60 ]]
10242023-12-12T05:43:44.352Z+ sleep 1
10252023-12-12T05:43:45.335Z+ retry=16
10262023-12-12T05:43:45.336Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10272023-12-12T05:43:45.360Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10282023-12-12T05:43:45.360Z+ [[ 16 -gt 60 ]]
10292023-12-12T05:43:45.360Z+ sleep 1
10302023-12-12T05:43:46.353Z+ retry=17
10312023-12-12T05:43:46.353Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10322023-12-12T05:43:46.381Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10332023-12-12T05:43:46.381Z+ [[ 17 -gt 60 ]]
10342023-12-12T05:43:46.381Z+ sleep 1
10352023-12-12T05:43:47.367Z+ retry=18
10362023-12-12T05:43:47.367Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10372023-12-12T05:43:47.396Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10382023-12-12T05:43:47.397Z+ [[ 18 -gt 60 ]]
10392023-12-12T05:43:47.397Z+ sleep 1
10402023-12-12T05:43:48.392Z+ retry=19
10412023-12-12T05:43:48.392Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10422023-12-12T05:43:48.425Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10432023-12-12T05:43:48.425Z+ [[ 19 -gt 60 ]]
10442023-12-12T05:43:48.425Z+ sleep 1
10452023-12-12T05:43:49.399Z+ retry=20
10462023-12-12T05:43:49.399Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10472023-12-12T05:43:49.428Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10482023-12-12T05:43:49.428Z+ [[ 20 -gt 60 ]]
10492023-12-12T05:43:49.428Z+ sleep 1
10502023-12-12T05:43:50.420Z+ retry=21
10512023-12-12T05:43:50.420Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10522023-12-12T05:43:50.457Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10532023-12-12T05:43:50.457Z+ [[ 21 -gt 60 ]]
10542023-12-12T05:43:50.457Z+ sleep 1
10552023-12-12T05:43:51.430Z+ retry=22
10562023-12-12T05:43:51.430Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10572023-12-12T05:43:51.454Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10582023-12-12T05:43:51.454Z+ [[ 22 -gt 60 ]]
10592023-12-12T05:43:51.454Z+ sleep 1
10602023-12-12T05:43:52.445Z+ retry=23
10612023-12-12T05:43:52.445Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10622023-12-12T05:43:52.470Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10632023-12-12T05:43:52.470Z+ [[ 23 -gt 60 ]]
10642023-12-12T05:43:52.470Z+ sleep 1
10652023-12-12T05:43:53.460Z+ retry=24
10662023-12-12T05:43:53.460Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10672023-12-12T05:43:53.485Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10682023-12-12T05:43:53.485Z+ [[ 24 -gt 60 ]]
10692023-12-12T05:43:53.485Z+ sleep 1
10702023-12-12T05:43:54.488Z+ retry=25
10712023-12-12T05:43:54.489Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10722023-12-12T05:43:54.963Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10732023-12-12T05:43:54.963Z+ [[ 25 -gt 60 ]]
10742023-12-12T05:43:54.963Z+ sleep 1
10752023-12-12T05:43:55.492Z+ retry=26
10762023-12-12T05:43:55.492Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10772023-12-12T05:43:55.517Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10782023-12-12T05:43:55.517Z+ [[ 26 -gt 60 ]]
10792023-12-12T05:43:55.517Z+ sleep 1
10802023-12-12T05:43:56.507Z+ retry=27
10812023-12-12T05:43:56.507Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10822023-12-12T05:43:56.532Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10832023-12-12T05:43:56.533Z+ [[ 27 -gt 60 ]]
10842023-12-12T05:43:56.533Z+ sleep 1
10852023-12-12T05:43:57.532Z+ retry=28
10862023-12-12T05:43:57.533Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10872023-12-12T05:43:57.567Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10882023-12-12T05:43:57.567Z+ [[ 28 -gt 60 ]]
10892023-12-12T05:43:57.567Z+ sleep 1
10902023-12-12T05:43:58.537Z+ retry=29
10912023-12-12T05:43:58.538Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10922023-12-12T05:43:58.562Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10932023-12-12T05:43:58.562Z+ [[ 29 -gt 60 ]]
10942023-12-12T05:43:58.562Z+ sleep 1
10952023-12-12T05:43:59.553Z+ retry=30
10962023-12-12T05:43:59.553Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10972023-12-12T05:43:59.578Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10982023-12-12T05:43:59.579Z+ [[ 30 -gt 60 ]]
10992023-12-12T05:43:59.579Z+ sleep 1
11002023-12-12T05:44:00.569Z+ retry=31
11012023-12-12T05:44:00.569Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11022023-12-12T05:44:00.593Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11032023-12-12T05:44:00.593Z+ [[ 31 -gt 60 ]]
11042023-12-12T05:44:00.594Z+ sleep 1
11052023-12-12T05:44:01.585Z+ retry=32
11062023-12-12T05:44:01.585Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11072023-12-12T05:44:01.609Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11082023-12-12T05:44:01.609Z+ [[ 32 -gt 60 ]]
11092023-12-12T05:44:01.610Z+ sleep 1
11102023-12-12T05:44:02.598Z+ retry=33
11112023-12-12T05:44:02.598Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11122023-12-12T05:44:02.675Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
11132023-12-12T05:44:03.127Z{"status":"ok"}success
11142023-12-12T05:44:03.128ZProject {
11152023-12-12T05:44:03.128Z description: "some images",
11162023-12-12T05:44:03.128Z id: ef3a77b2-ab6c-450b-9102-d60551918a43,
11172023-12-12T05:44:03.129Z name: Name(
11182023-12-12T05:44:03.129Z "images",
11192023-12-12T05:44:03.129Z ),
11202023-12-12T05:44:03.159Z time_created: 2023-12-12T05:44:02.771162Z,
11212023-12-12T05:44:03.159Z time_modified: 2023-12-12T05:44:02.771162Z,
11222023-12-12T05:44:03.160Z}
11232023-12-12T05:44:03.160Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
11242023-12-12T05:49:44.454Zdone!
11252023-12-12T05:49:44.454Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
11262023-12-12T05:49:44.455Zsuccess
11272023-12-12T05:49:44.455ZImage {
11282023-12-12T05:49:44.455Z block_size: ByteCount(
11292023-12-12T05:49:44.455Z 512,
11302023-12-12T05:49:44.455Z ),
11312023-12-12T05:49:44.455Z description: "debian 11 original base image",
11322023-12-12T05:49:44.455Z digest: None,
11332023-12-12T05:49:44.455Z id: cd31b07b-4891-429b-85d4-cd86bc2facd5,
11342023-12-12T05:49:44.455Z name: Name(
11352023-12-12T05:49:44.455Z "debian11",
11362023-12-12T05:49:44.455Z ),
11372023-12-12T05:49:44.455Z os: "debian",
11382023-12-12T05:49:44.456Z project_id: None,
11392023-12-12T05:49:44.456Z size: ByteCount(
11402023-12-12T05:49:44.456Z 2147483648,
11412023-12-12T05:49:44.456Z ),
11422023-12-12T05:49:44.456Z time_created: 2023-12-12T05:49:35.304505Z,
11432023-12-12T05:49:44.456Z time_modified: 2023-12-12T05:49:35.560881Z,
11442023-12-12T05:49:44.456Z url: None,
11452023-12-12T05:49:44.456Z version: "11",
11462023-12-12T05:49:44.456Z}
11472023-12-12T05:49:44.456Z+ rm ./tests/bootstrap
11482023-12-12T05:49:44.456Z+ for test_bin in tests/*
11492023-12-12T05:49:44.456Z+ ./tests/bootstrap-8aed3472db5f37d6
11502023-12-12T05:49:44.456Z
11512023-12-12T05:49:44.456Zrunning 0 tests
11522023-12-12T05:49:44.456Z
11532023-12-12T05:49:44.456Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
11542023-12-12T05:49:44.457Z
11552023-12-12T05:49:44.457Z+ for test_bin in tests/*
11562023-12-12T05:49:44.457Z+ ./tests/end_to_end_tests-fd6b950b6a51f2ac
11572023-12-12T05:49:44.457Z
11582023-12-12T05:49:44.457Zrunning 2 tests
11592023-12-12T05:49:44.457Ztest no_spoof::no_spoof_here ... ok
11602023-12-12T05:50:08.695Ztest instance_launch::instance_launch ... ok
11612023-12-12T05:50:08.695Z
11622023-12-12T05:50:08.695Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 33.01s
11632023-12-12T05:50:08.695Z
11642023-12-12T05:50:08.724Z+ _exit_trap
11652023-12-12T05:50:08.724Z+ local status=0
11662023-12-12T05:50:08.724Z+ [[ 0 -eq 0 ]]
11672023-12-12T05:50:08.724Z+ exit 0
11682023-12-12T05:50:08.727Zprocess exited: duration 838265 ms, exit code 0
 
11692023-12-12T05:50:08.848Zfound 99 output files
11702023-12-12T05:50:08.848Zuploading: /var/svc/log/oxide-sled-agent:default.log (759549 bytes)
11712023-12-12T05:50:11.451Zuploaded: /var/svc/log/oxide-sled-agent:default.log
11722023-12-12T05:50:11.451Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c2d6f4ae-d752-4986-b885-7ca297276b3b/root/var/svc/log/oxide-internal_dns:default.log (22524 bytes)
11732023-12-12T05:50:12.607Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c2d6f4ae-d752-4986-b885-7ca297276b3b/root/var/svc/log/oxide-internal_dns:default.log
11742023-12-12T05:50:12.608Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c2d6f4ae-d752-4986-b885-7ca297276b3b/root/var/svc/log/oxide-internal_dns:default.log.0 (109119 bytes)
11752023-12-12T05:50:13.955Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c2d6f4ae-d752-4986-b885-7ca297276b3b/root/var/svc/log/oxide-internal_dns:default.log.0
11762023-12-12T05:50:13.955Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_7fe72949-2989-44a1-b3f9-3debc7e60123/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11772023-12-12T05:50:15.049Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_7fe72949-2989-44a1-b3f9-3debc7e60123/root/var/svc/log/oxide-cockroachdb:default.log
11782023-12-12T05:50:15.049Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_7fe72949-2989-44a1-b3f9-3debc7e60123/root/var/svc/log/oxide-cockroachdb:default.log.0 (5413 bytes)
11792023-12-12T05:50:16.154Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_7fe72949-2989-44a1-b3f9-3debc7e60123/root/var/svc/log/oxide-cockroachdb:default.log.0
11802023-12-12T05:50:16.154Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3ad918bf-2e0d-4e65-b210-599e922b6529/root/var/svc/log/oxide-crucible-agent:default.log (7755 bytes)
11812023-12-12T05:50:17.291Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3ad918bf-2e0d-4e65-b210-599e922b6529/root/var/svc/log/oxide-crucible-agent:default.log
11822023-12-12T05:50:17.291Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3ad918bf-2e0d-4e65-b210-599e922b6529/root/var/svc/log/oxide-crucible-agent:default.log.0 (9796 bytes)
11832023-12-12T05:50:18.421Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3ad918bf-2e0d-4e65-b210-599e922b6529/root/var/svc/log/oxide-crucible-agent:default.log.0
11842023-12-12T05:50:18.421Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3ad918bf-2e0d-4e65-b210-599e922b6529/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11852023-12-12T05:50:19.489Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3ad918bf-2e0d-4e65-b210-599e922b6529/root/var/svc/log/oxide-crucible-downstairs:default.log
11862023-12-12T05:50:19.490Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3ad918bf-2e0d-4e65-b210-599e922b6529/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11872023-12-12T05:50:20.589Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3ad918bf-2e0d-4e65-b210-599e922b6529/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11882023-12-12T05:50:20.590Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3ad918bf-2e0d-4e65-b210-599e922b6529/root/var/svc/log/oxide-crucible-downstairs:downstairs-6db6d6b9-0475-43c6-a332-32b4a80fc332.log (1905 bytes)
11892023-12-12T05:50:21.685Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3ad918bf-2e0d-4e65-b210-599e922b6529/root/var/svc/log/oxide-crucible-downstairs:downstairs-6db6d6b9-0475-43c6-a332-32b4a80fc332.log
11902023-12-12T05:50:21.710Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3ad918bf-2e0d-4e65-b210-599e922b6529/root/var/svc/log/oxide-crucible-downstairs:downstairs-6db6d6b9-0475-43c6-a332-32b4a80fc332.log.0 (4529 bytes)
11912023-12-12T05:50:22.784Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3ad918bf-2e0d-4e65-b210-599e922b6529/root/var/svc/log/oxide-crucible-downstairs:downstairs-6db6d6b9-0475-43c6-a332-32b4a80fc332.log.0
11922023-12-12T05:50:22.809Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3ad918bf-2e0d-4e65-b210-599e922b6529/root/var/svc/log/oxide-crucible-downstairs:snapshot-6db6d6b9-0475-43c6-a332-32b4a80fc332-cccef0db-08f2-410e-91ab-2e88098b6d5f.log (5175 bytes)
11932023-12-12T05:50:23.878Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3ad918bf-2e0d-4e65-b210-599e922b6529/root/var/svc/log/oxide-crucible-downstairs:snapshot-6db6d6b9-0475-43c6-a332-32b4a80fc332-cccef0db-08f2-410e-91ab-2e88098b6d5f.log
11942023-12-12T05:50:23.879Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_9623ebca-4de2-4562-9c26-539b449cf273/root/var/svc/log/oxide-internal_dns:default.log (22523 bytes)
11952023-12-12T05:50:25.024Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_9623ebca-4de2-4562-9c26-539b449cf273/root/var/svc/log/oxide-internal_dns:default.log
11962023-12-12T05:50:25.025Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_9623ebca-4de2-4562-9c26-539b449cf273/root/var/svc/log/oxide-internal_dns:default.log.0 (111147 bytes)
11972023-12-12T05:50:26.396Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_9623ebca-4de2-4562-9c26-539b449cf273/root/var/svc/log/oxide-internal_dns:default.log.0
11982023-12-12T05:50:26.396Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_4a4978ab-e3a0-4cd4-84ec-b237e1555aa7/root/var/svc/log/oxide-crucible-agent:default.log (5927 bytes)
11992023-12-12T05:50:27.515Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_4a4978ab-e3a0-4cd4-84ec-b237e1555aa7/root/var/svc/log/oxide-crucible-agent:default.log
12002023-12-12T05:50:27.542Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_4a4978ab-e3a0-4cd4-84ec-b237e1555aa7/root/var/svc/log/oxide-crucible-agent:default.log.0 (14118 bytes)
12012023-12-12T05:50:28.672Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_4a4978ab-e3a0-4cd4-84ec-b237e1555aa7/root/var/svc/log/oxide-crucible-agent:default.log.0
12022023-12-12T05:50:28.706Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_4a4978ab-e3a0-4cd4-84ec-b237e1555aa7/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12032023-12-12T05:50:29.756Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_4a4978ab-e3a0-4cd4-84ec-b237e1555aa7/root/var/svc/log/oxide-crucible-downstairs:default.log
12042023-12-12T05:50:29.756Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_4a4978ab-e3a0-4cd4-84ec-b237e1555aa7/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12052023-12-12T05:50:31.307Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_4a4978ab-e3a0-4cd4-84ec-b237e1555aa7/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12062023-12-12T05:50:31.307Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_4a4978ab-e3a0-4cd4-84ec-b237e1555aa7/root/var/svc/log/oxide-crucible-downstairs:downstairs-67b12248-bd65-49e2-b718-e9dcb7a708e9.log (0 bytes)
12072023-12-12T05:50:32.391Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_4a4978ab-e3a0-4cd4-84ec-b237e1555aa7/root/var/svc/log/oxide-crucible-downstairs:downstairs-67b12248-bd65-49e2-b718-e9dcb7a708e9.log
12082023-12-12T05:50:32.391Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_4a4978ab-e3a0-4cd4-84ec-b237e1555aa7/root/var/svc/log/oxide-crucible-downstairs:downstairs-67b12248-bd65-49e2-b718-e9dcb7a708e9.log.0 (3046 bytes)
12092023-12-12T05:50:33.500Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_4a4978ab-e3a0-4cd4-84ec-b237e1555aa7/root/var/svc/log/oxide-crucible-downstairs:downstairs-67b12248-bd65-49e2-b718-e9dcb7a708e9.log.0
12102023-12-12T05:50:33.500Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_4a4978ab-e3a0-4cd4-84ec-b237e1555aa7/root/var/svc/log/oxide-crucible-downstairs:downstairs-d8e6616b-21c7-4723-8a9a-ead64432f15f.log (2943 bytes)
12112023-12-12T05:50:34.603Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_4a4978ab-e3a0-4cd4-84ec-b237e1555aa7/root/var/svc/log/oxide-crucible-downstairs:downstairs-d8e6616b-21c7-4723-8a9a-ead64432f15f.log
12122023-12-12T05:50:34.628Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_a1c2465c-cc25-41dd-8119-4fbb722637ec/root/var/svc/log/oxide-crucible-agent:default.log (10248 bytes)
12132023-12-12T05:50:35.721Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_a1c2465c-cc25-41dd-8119-4fbb722637ec/root/var/svc/log/oxide-crucible-agent:default.log
12142023-12-12T05:50:35.721Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_a1c2465c-cc25-41dd-8119-4fbb722637ec/root/var/svc/log/oxide-crucible-agent:default.log.0 (13687 bytes)
12152023-12-12T05:50:36.863Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_a1c2465c-cc25-41dd-8119-4fbb722637ec/root/var/svc/log/oxide-crucible-agent:default.log.0
12162023-12-12T05:50:36.902Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_a1c2465c-cc25-41dd-8119-4fbb722637ec/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12172023-12-12T05:50:37.953Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_a1c2465c-cc25-41dd-8119-4fbb722637ec/root/var/svc/log/oxide-crucible-downstairs:default.log
12182023-12-12T05:50:37.954Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_a1c2465c-cc25-41dd-8119-4fbb722637ec/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12192023-12-12T05:50:39.058Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_a1c2465c-cc25-41dd-8119-4fbb722637ec/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12202023-12-12T05:50:39.082Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_a1c2465c-cc25-41dd-8119-4fbb722637ec/root/var/svc/log/oxide-crucible-downstairs:downstairs-c80ae781-02cb-4dd2-b79a-f1f1aa6d263c.log (5533 bytes)
12212023-12-12T05:50:40.152Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_a1c2465c-cc25-41dd-8119-4fbb722637ec/root/var/svc/log/oxide-crucible-downstairs:downstairs-c80ae781-02cb-4dd2-b79a-f1f1aa6d263c.log
12222023-12-12T05:50:40.152Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_a1c2465c-cc25-41dd-8119-4fbb722637ec/root/var/svc/log/oxide-crucible-downstairs:downstairs-ed59e00b-eeb4-4c44-b171-0d42e7aa65fd.log (0 bytes)
12232023-12-12T05:50:41.224Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_a1c2465c-cc25-41dd-8119-4fbb722637ec/root/var/svc/log/oxide-crucible-downstairs:downstairs-ed59e00b-eeb4-4c44-b171-0d42e7aa65fd.log
12242023-12-12T05:50:41.249Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_a1c2465c-cc25-41dd-8119-4fbb722637ec/root/var/svc/log/oxide-crucible-downstairs:downstairs-ed59e00b-eeb4-4c44-b171-0d42e7aa65fd.log.0 (3050 bytes)
12252023-12-12T05:50:42.320Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_a1c2465c-cc25-41dd-8119-4fbb722637ec/root/var/svc/log/oxide-crucible-downstairs:downstairs-ed59e00b-eeb4-4c44-b171-0d42e7aa65fd.log.0
12262023-12-12T05:50:42.347Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_48e56b92-afbc-4419-abb9-8326f6ceb958/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
12272023-12-12T05:50:43.394Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_48e56b92-afbc-4419-abb9-8326f6ceb958/root/var/svc/log/oxide-crucible-pantry:default.log
12282023-12-12T05:50:43.395Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_48e56b92-afbc-4419-abb9-8326f6ceb958/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
12292023-12-12T05:50:44.497Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_48e56b92-afbc-4419-abb9-8326f6ceb958/root/var/svc/log/oxide-crucible-pantry:default.log.0
12302023-12-12T05:50:44.497Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_47e44ab6-ee0b-4407-95b8-c3af199399c8/root/var/svc/log/oxide-nexus:default.log (454355 bytes)
12312023-12-12T05:50:46.228Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_47e44ab6-ee0b-4407-95b8-c3af199399c8/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 454355 -> 461848
12322023-12-12T05:50:46.924Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_47e44ab6-ee0b-4407-95b8-c3af199399c8/root/var/svc/log/oxide-nexus:default.log
12332023-12-12T05:50:46.924Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_47e44ab6-ee0b-4407-95b8-c3af199399c8/root/var/svc/log/oxide-nexus:default.log.0 (2026636 bytes)
12342023-12-12T05:50:49.593Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_47e44ab6-ee0b-4407-95b8-c3af199399c8/root/var/svc/log/oxide-nexus:default.log.0
12352023-12-12T05:50:49.593Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3/root/var/svc/log/oxide-ntp:default.log (67 bytes)
12362023-12-12T05:50:51.105Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3/root/var/svc/log/oxide-ntp:default.log
12372023-12-12T05:50:51.106Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3/root/var/svc/log/oxide-ntp:default.log.0 (1190 bytes)
12382023-12-12T05:50:52.201Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3/root/var/svc/log/oxide-ntp:default.log.0
12392023-12-12T05:50:52.202Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_b4f6dc6a-2b4b-4f27-bff6-af6b121fce6c/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12402023-12-12T05:50:53.273Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_b4f6dc6a-2b4b-4f27-bff6-af6b121fce6c/root/var/svc/log/oxide-cockroachdb:default.log
12412023-12-12T05:50:53.278Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_b4f6dc6a-2b4b-4f27-bff6-af6b121fce6c/root/var/svc/log/oxide-cockroachdb:default.log.0 (5411 bytes)
12422023-12-12T05:50:54.370Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_b4f6dc6a-2b4b-4f27-bff6-af6b121fce6c/root/var/svc/log/oxide-cockroachdb:default.log.0
12432023-12-12T05:50:54.370Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_a16b6262-a3a3-4964-923d-ab85f6104ee0/root/var/svc/log/oxide-external_dns:default.log (20139 bytes)
12442023-12-12T05:50:55.510Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_a16b6262-a3a3-4964-923d-ab85f6104ee0/root/var/svc/log/oxide-external_dns:default.log
12452023-12-12T05:50:55.510Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_a16b6262-a3a3-4964-923d-ab85f6104ee0/root/var/svc/log/oxide-external_dns:default.log.0 (10575 bytes)
12462023-12-12T05:50:56.641Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_a16b6262-a3a3-4964-923d-ab85f6104ee0/root/var/svc/log/oxide-external_dns:default.log.0
12472023-12-12T05:50:56.641Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_4d2f517b-b57d-462a-b53b-76945005e608/root/var/svc/log/oxide-internal_dns:default.log (22523 bytes)
12482023-12-12T05:50:58.067Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_4d2f517b-b57d-462a-b53b-76945005e608/root/var/svc/log/oxide-internal_dns:default.log
12492023-12-12T05:50:58.068Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_4d2f517b-b57d-462a-b53b-76945005e608/root/var/svc/log/oxide-internal_dns:default.log.0 (107078 bytes)
12502023-12-12T05:50:59.388Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_4d2f517b-b57d-462a-b53b-76945005e608/root/var/svc/log/oxide-internal_dns:default.log.0
12512023-12-12T05:50:59.388Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa1eecb-1b7c-4142-b740-a09c3168c58c/root/var/svc/log/oxide-crucible-agent:default.log (6358 bytes)
12522023-12-12T05:51:00.592Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa1eecb-1b7c-4142-b740-a09c3168c58c/root/var/svc/log/oxide-crucible-agent:default.log
12532023-12-12T05:51:00.592Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa1eecb-1b7c-4142-b740-a09c3168c58c/root/var/svc/log/oxide-crucible-agent:default.log.0 (3434 bytes)
12542023-12-12T05:51:02.351Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa1eecb-1b7c-4142-b740-a09c3168c58c/root/var/svc/log/oxide-crucible-agent:default.log.0
12552023-12-12T05:51:02.351Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa1eecb-1b7c-4142-b740-a09c3168c58c/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12562023-12-12T05:51:14.518Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa1eecb-1b7c-4142-b740-a09c3168c58c/root/var/svc/log/oxide-crucible-downstairs:default.log
12572023-12-12T05:51:14.518Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa1eecb-1b7c-4142-b740-a09c3168c58c/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12582023-12-12T05:51:15.970Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa1eecb-1b7c-4142-b740-a09c3168c58c/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12592023-12-12T05:51:15.970Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa1eecb-1b7c-4142-b740-a09c3168c58c/root/var/svc/log/oxide-crucible-downstairs:downstairs-5b474e5f-992d-4140-ad23-688a10707d81.log (2944 bytes)
12602023-12-12T05:51:17.496Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa1eecb-1b7c-4142-b740-a09c3168c58c/root/var/svc/log/oxide-crucible-downstairs:downstairs-5b474e5f-992d-4140-ad23-688a10707d81.log
12612023-12-12T05:51:17.658Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_f2d620ac-3e6a-430a-906d-1392095be38d/root/var/svc/log/oxide-nexus:default.log (27681812 bytes)
12622023-12-12T05:51:42.734Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_f2d620ac-3e6a-430a-906d-1392095be38d/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 27681812 -> 27761264
12632023-12-12T05:51:43.811Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_f2d620ac-3e6a-430a-906d-1392095be38d/root/var/svc/log/oxide-nexus:default.log
12642023-12-12T05:51:43.811Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_f2d620ac-3e6a-430a-906d-1392095be38d/root/var/svc/log/oxide-nexus:default.log.0 (6367750 bytes)
12652023-12-12T05:51:48.053Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_f2d620ac-3e6a-430a-906d-1392095be38d/root/var/svc/log/oxide-nexus:default.log.0
12662023-12-12T05:51:48.087Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_ddf49ac7-6869-4741-91f6-e65bc6cc1af2/root/var/svc/log/oxide-clickhouse:default.log (521979 bytes)
12672023-12-12T05:51:48.566Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_ddf49ac7-6869-4741-91f6-e65bc6cc1af2/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 521979 -> 697339
12682023-12-12T05:51:49.206Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_ddf49ac7-6869-4741-91f6-e65bc6cc1af2/root/var/svc/log/oxide-clickhouse:default.log
12692023-12-12T05:51:49.232Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_ddf49ac7-6869-4741-91f6-e65bc6cc1af2/root/var/svc/log/oxide-clickhouse:default.log.0 (288728 bytes)
12702023-12-12T05:51:50.838Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_ddf49ac7-6869-4741-91f6-e65bc6cc1af2/root/var/svc/log/oxide-clickhouse:default.log.0
12712023-12-12T05:51:50.840Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5ea4a5dc-ab72-4d7b-8551-1ee6e68f5a53/root/var/svc/log/oxide-crucible-agent:default.log (7760 bytes)
12722023-12-12T05:51:51.959Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5ea4a5dc-ab72-4d7b-8551-1ee6e68f5a53/root/var/svc/log/oxide-crucible-agent:default.log
12732023-12-12T05:51:51.959Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5ea4a5dc-ab72-4d7b-8551-1ee6e68f5a53/root/var/svc/log/oxide-crucible-agent:default.log.0 (9793 bytes)
12742023-12-12T05:51:53.098Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5ea4a5dc-ab72-4d7b-8551-1ee6e68f5a53/root/var/svc/log/oxide-crucible-agent:default.log.0
12752023-12-12T05:51:53.099Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5ea4a5dc-ab72-4d7b-8551-1ee6e68f5a53/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12762023-12-12T05:51:54.166Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5ea4a5dc-ab72-4d7b-8551-1ee6e68f5a53/root/var/svc/log/oxide-crucible-downstairs:default.log
12772023-12-12T05:51:54.167Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5ea4a5dc-ab72-4d7b-8551-1ee6e68f5a53/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12782023-12-12T05:51:55.281Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5ea4a5dc-ab72-4d7b-8551-1ee6e68f5a53/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12792023-12-12T05:51:55.305Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5ea4a5dc-ab72-4d7b-8551-1ee6e68f5a53/root/var/svc/log/oxide-crucible-downstairs:downstairs-aecf8985-7d2f-45ed-8779-492377e310cb.log (1907 bytes)
12802023-12-12T05:51:56.378Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5ea4a5dc-ab72-4d7b-8551-1ee6e68f5a53/root/var/svc/log/oxide-crucible-downstairs:downstairs-aecf8985-7d2f-45ed-8779-492377e310cb.log
12812023-12-12T05:51:56.378Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5ea4a5dc-ab72-4d7b-8551-1ee6e68f5a53/root/var/svc/log/oxide-crucible-downstairs:downstairs-aecf8985-7d2f-45ed-8779-492377e310cb.log.0 (4528 bytes)
12822023-12-12T05:51:57.481Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5ea4a5dc-ab72-4d7b-8551-1ee6e68f5a53/root/var/svc/log/oxide-crucible-downstairs:downstairs-aecf8985-7d2f-45ed-8779-492377e310cb.log.0
12832023-12-12T05:51:57.481Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5ea4a5dc-ab72-4d7b-8551-1ee6e68f5a53/root/var/svc/log/oxide-crucible-downstairs:snapshot-aecf8985-7d2f-45ed-8779-492377e310cb-cccef0db-08f2-410e-91ab-2e88098b6d5f.log (5178 bytes)
12842023-12-12T05:51:58.581Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5ea4a5dc-ab72-4d7b-8551-1ee6e68f5a53/root/var/svc/log/oxide-crucible-downstairs:snapshot-aecf8985-7d2f-45ed-8779-492377e310cb-cccef0db-08f2-410e-91ab-2e88098b6d5f.log
12852023-12-12T05:51:58.584Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f0c002ea-f5d3-4825-97d6-df84b533198a/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
12862023-12-12T05:51:59.656Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f0c002ea-f5d3-4825-97d6-df84b533198a/root/var/svc/log/oxide-crucible-agent:default.log
12872023-12-12T05:51:59.656Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f0c002ea-f5d3-4825-97d6-df84b533198a/root/var/svc/log/oxide-crucible-agent:default.log.0 (3434 bytes)
12882023-12-12T05:52:00.761Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f0c002ea-f5d3-4825-97d6-df84b533198a/root/var/svc/log/oxide-crucible-agent:default.log.0
12892023-12-12T05:52:00.795Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f0c002ea-f5d3-4825-97d6-df84b533198a/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12902023-12-12T05:52:01.846Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f0c002ea-f5d3-4825-97d6-df84b533198a/root/var/svc/log/oxide-crucible-downstairs:default.log
12912023-12-12T05:52:01.846Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f0c002ea-f5d3-4825-97d6-df84b533198a/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12922023-12-12T05:52:03.052Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f0c002ea-f5d3-4825-97d6-df84b533198a/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12932023-12-12T05:52:03.052Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_5b57b4f5-b35a-4cda-8002-d40988620faa/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
12942023-12-12T05:52:04.149Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_5b57b4f5-b35a-4cda-8002-d40988620faa/root/var/svc/log/oxide-crucible-pantry:default.log
12952023-12-12T05:52:04.149Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_5b57b4f5-b35a-4cda-8002-d40988620faa/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
12962023-12-12T05:52:05.256Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_5b57b4f5-b35a-4cda-8002-d40988620faa/root/var/svc/log/oxide-crucible-pantry:default.log.0
12972023-12-12T05:52:05.281Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_28b8d1c1-5b3f-4cd0-a8e8-d7083daf893b/root/var/svc/log/oxide-crucible-agent:default.log (6357 bytes)
12982023-12-12T05:52:06.374Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_28b8d1c1-5b3f-4cd0-a8e8-d7083daf893b/root/var/svc/log/oxide-crucible-agent:default.log
12992023-12-12T05:52:06.374Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_28b8d1c1-5b3f-4cd0-a8e8-d7083daf893b/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
13002023-12-12T05:52:07.476Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_28b8d1c1-5b3f-4cd0-a8e8-d7083daf893b/root/var/svc/log/oxide-crucible-agent:default.log.0
13012023-12-12T05:52:07.476Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_28b8d1c1-5b3f-4cd0-a8e8-d7083daf893b/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
13022023-12-12T05:52:08.554Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_28b8d1c1-5b3f-4cd0-a8e8-d7083daf893b/root/var/svc/log/oxide-crucible-downstairs:default.log
13032023-12-12T05:52:08.554Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_28b8d1c1-5b3f-4cd0-a8e8-d7083daf893b/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
13042023-12-12T05:52:09.648Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_28b8d1c1-5b3f-4cd0-a8e8-d7083daf893b/root/var/svc/log/oxide-crucible-downstairs:default.log.0
13052023-12-12T05:52:09.648Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_28b8d1c1-5b3f-4cd0-a8e8-d7083daf893b/root/var/svc/log/oxide-crucible-downstairs:downstairs-d8ac6200-8937-4c67-9bab-78379cc47aeb.log (2945 bytes)
13062023-12-12T05:52:10.748Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_28b8d1c1-5b3f-4cd0-a8e8-d7083daf893b/root/var/svc/log/oxide-crucible-downstairs:downstairs-d8ac6200-8937-4c67-9bab-78379cc47aeb.log
13072023-12-12T05:52:10.748Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_44fc3102-80fe-4811-aaa4-22df83ddad3a/root/var/svc/log/oxide-crucible-pantry:default.log (979754 bytes)
13082023-12-12T05:52:13.234Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_44fc3102-80fe-4811-aaa4-22df83ddad3a/root/var/svc/log/oxide-crucible-pantry:default.log
13092023-12-12T05:52:13.234Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_44fc3102-80fe-4811-aaa4-22df83ddad3a/root/var/svc/log/oxide-crucible-pantry:default.log.0 (248367 bytes)
13102023-12-12T05:52:24.026Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_44fc3102-80fe-4811-aaa4-22df83ddad3a/root/var/svc/log/oxide-crucible-pantry:default.log.0
13112023-12-12T05:52:24.027Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_92bb13c6-8029-496f-a1c1-11eaabb18afa/root/var/svc/log/oxide-nexus:default.log (452348 bytes)
13122023-12-12T05:52:25.516Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_92bb13c6-8029-496f-a1c1-11eaabb18afa/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 452348 -> 604634
13132023-12-12T05:52:26.707Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_92bb13c6-8029-496f-a1c1-11eaabb18afa/root/var/svc/log/oxide-nexus:default.log
13142023-12-12T05:52:26.707Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_92bb13c6-8029-496f-a1c1-11eaabb18afa/root/var/svc/log/oxide-nexus:default.log.0 (629508 bytes)
13152023-12-12T05:52:28.680Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_92bb13c6-8029-496f-a1c1-11eaabb18afa/root/var/svc/log/oxide-nexus:default.log.0
13162023-12-12T05:52:28.680Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_8c5057a0-d3bf-418d-9fa5-e6dc3ef919a3/root/var/svc/log/oxide-oximeter:default.log (176975 bytes)
13172023-12-12T05:52:29.188Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_8c5057a0-d3bf-418d-9fa5-e6dc3ef919a3/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 176975 -> 256682
13182023-12-12T05:52:30.351Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_8c5057a0-d3bf-418d-9fa5-e6dc3ef919a3/root/var/svc/log/oxide-oximeter:default.log
13192023-12-12T05:52:30.351Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_8c5057a0-d3bf-418d-9fa5-e6dc3ef919a3/root/var/svc/log/oxide-oximeter:default.log.0 (72500 bytes)
13202023-12-12T05:52:31.767Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_8c5057a0-d3bf-418d-9fa5-e6dc3ef919a3/root/var/svc/log/oxide-oximeter:default.log.0
13212023-12-12T05:52:31.768Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_00c8e725-23ce-4166-aeae-8eb98d24cc24/root/var/svc/log/oxide-external_dns:default.log (18882 bytes)
13222023-12-12T05:52:32.423Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_00c8e725-23ce-4166-aeae-8eb98d24cc24/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 18882 -> 26435
13232023-12-12T05:52:33.021Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_00c8e725-23ce-4166-aeae-8eb98d24cc24/root/var/svc/log/oxide-external_dns:default.log
13242023-12-12T05:52:33.021Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_00c8e725-23ce-4166-aeae-8eb98d24cc24/root/var/svc/log/oxide-external_dns:default.log.0 (11833 bytes)
13252023-12-12T05:52:34.190Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_00c8e725-23ce-4166-aeae-8eb98d24cc24/root/var/svc/log/oxide-external_dns:default.log.0
13262023-12-12T05:52:34.190Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0b7a69c2-9a41-4537-8196-39483af5a1e6/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
13272023-12-12T05:52:35.264Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0b7a69c2-9a41-4537-8196-39483af5a1e6/root/var/svc/log/oxide-cockroachdb:default.log
13282023-12-12T05:52:35.269Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0b7a69c2-9a41-4537-8196-39483af5a1e6/root/var/svc/log/oxide-cockroachdb:default.log.0 (5387 bytes)
13292023-12-12T05:52:36.486Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0b7a69c2-9a41-4537-8196-39483af5a1e6/root/var/svc/log/oxide-cockroachdb:default.log.0
13302023-12-12T05:52:36.486Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6099c111-2b3c-4eb2-a92c-5f855bc68286/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
13312023-12-12T05:52:37.820Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6099c111-2b3c-4eb2-a92c-5f855bc68286/root/var/svc/log/oxide-cockroachdb:default.log
13322023-12-12T05:52:37.820Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6099c111-2b3c-4eb2-a92c-5f855bc68286/root/var/svc/log/oxide-cockroachdb:default.log.0 (5412 bytes)
13332023-12-12T05:52:38.918Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6099c111-2b3c-4eb2-a92c-5f855bc68286/root/var/svc/log/oxide-cockroachdb:default.log.0
13342023-12-12T05:52:38.919Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_94f0ae50-0515-4bf3-ad80-f175a53f699f/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
13352023-12-12T05:52:39.984Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_94f0ae50-0515-4bf3-ad80-f175a53f699f/root/var/svc/log/oxide-cockroachdb:default.log
13362023-12-12T05:52:39.984Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_94f0ae50-0515-4bf3-ad80-f175a53f699f/root/var/svc/log/oxide-cockroachdb:default.log.0 (5412 bytes)
13372023-12-12T05:52:41.550Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_94f0ae50-0515-4bf3-ad80-f175a53f699f/root/var/svc/log/oxide-cockroachdb:default.log.0
13382023-12-12T05:52:41.550Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-agent:default.log (18002 bytes)
13392023-12-12T05:52:42.791Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-agent:default.log
13402023-12-12T05:52:42.792Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-agent:default.log.0 (20041 bytes)
13412023-12-12T05:52:43.960Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-agent:default.log.0
13422023-12-12T05:52:43.960Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
13432023-12-12T05:52:45.452Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:default.log
13442023-12-12T05:52:45.452Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
13452023-12-12T05:52:46.160Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:default.log.0
13462023-12-12T05:52:46.160Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:downstairs-ae967c83-b4a0-4d4f-af70-41888f233f0a.log (1906 bytes)
13472023-12-12T05:52:47.293Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:downstairs-ae967c83-b4a0-4d4f-af70-41888f233f0a.log
13482023-12-12T05:52:47.293Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:downstairs-ae967c83-b4a0-4d4f-af70-41888f233f0a.log.0 (4526 bytes)
13492023-12-12T05:52:48.391Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:downstairs-ae967c83-b4a0-4d4f-af70-41888f233f0a.log.0
13502023-12-12T05:52:48.391Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:downstairs-d51ccede-9354-4153-a8cc-1f309a8dc1cb.log (0 bytes)
13512023-12-12T05:52:49.478Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:downstairs-d51ccede-9354-4153-a8cc-1f309a8dc1cb.log
13522023-12-12T05:52:49.478Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:downstairs-d51ccede-9354-4153-a8cc-1f309a8dc1cb.log.0 (3050 bytes)
13532023-12-12T05:52:50.590Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:downstairs-d51ccede-9354-4153-a8cc-1f309a8dc1cb.log.0
13542023-12-12T05:52:50.591Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:downstairs-e9cc1a02-f706-4143-9c1f-7039085eeba1.log (5533 bytes)
13552023-12-12T05:52:51.684Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:downstairs-e9cc1a02-f706-4143-9c1f-7039085eeba1.log
13562023-12-12T05:52:51.684Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:snapshot-ae967c83-b4a0-4d4f-af70-41888f233f0a-cccef0db-08f2-410e-91ab-2e88098b6d5f.log (5177 bytes)
13572023-12-12T05:52:53.202Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_02bdd6b5-f7bf-4e6b-bbae-9979ce81571d/root/var/svc/log/oxide-crucible-downstairs:snapshot-ae967c83-b4a0-4d4f-af70-41888f233f0a-cccef0db-08f2-410e-91ab-2e88098b6d5f.log
13582023-12-12T05:52:53.202Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69ce5d0a-dc79-462e-adfb-33d42865fabc/root/var/svc/log/oxide-crucible-agent:default.log (10678 bytes)
13592023-12-12T05:52:54.317Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69ce5d0a-dc79-462e-adfb-33d42865fabc/root/var/svc/log/oxide-crucible-agent:default.log
13602023-12-12T05:52:54.318Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69ce5d0a-dc79-462e-adfb-33d42865fabc/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
13612023-12-12T05:52:55.489Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69ce5d0a-dc79-462e-adfb-33d42865fabc/root/var/svc/log/oxide-crucible-agent:default.log.0
13622023-12-12T05:52:55.491Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69ce5d0a-dc79-462e-adfb-33d42865fabc/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
13632023-12-12T05:52:56.563Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69ce5d0a-dc79-462e-adfb-33d42865fabc/root/var/svc/log/oxide-crucible-downstairs:default.log
13642023-12-12T05:52:56.563Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69ce5d0a-dc79-462e-adfb-33d42865fabc/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
13652023-12-12T05:52:57.691Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69ce5d0a-dc79-462e-adfb-33d42865fabc/root/var/svc/log/oxide-crucible-downstairs:default.log.0
13662023-12-12T05:52:57.691Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69ce5d0a-dc79-462e-adfb-33d42865fabc/root/var/svc/log/oxide-crucible-downstairs:downstairs-43d7d6d0-b6b5-4f27-be09-ff5e219465b9.log (5535 bytes)
13672023-12-12T05:52:58.754Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_69ce5d0a-dc79-462e-adfb-33d42865fabc/root/var/svc/log/oxide-crucible-downstairs:downstairs-43d7d6d0-b6b5-4f27-be09-ff5e219465b9.log
13682023-12-12T05:52:58.754Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3/root/var/log/chrony/measurements.log (18632 bytes)
13692023-12-12T05:52:58.856Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3/root/var/log/chrony/measurements.log" changed size mid upload: 18632 -> 21783
13702023-12-12T05:52:59.936Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3/root/var/log/chrony/measurements.log
13712023-12-12T05:52:59.936Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3/root/var/log/chrony/statistics.log (13338 bytes)
13722023-12-12T05:53:00.027Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3/root/var/log/chrony/statistics.log" changed size mid upload: 13338 -> 15678
13732023-12-12T05:53:01.107Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3/root/var/log/chrony/statistics.log
13742023-12-12T05:53:01.107Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3/root/var/log/chrony/tracking.log (2646 bytes)
13752023-12-12T05:53:01.157Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3/root/var/log/chrony/tracking.log" changed size mid upload: 2646 -> 2910
13762023-12-12T05:53:02.228Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_30d4d716-57a3-4b0f-b96a-d7ea534dfab3/root/var/log/chrony/tracking.log