01HNK4Y113EHV9XS6WB3XXSNV7: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.28",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HNK500EGM0QCKPM19P5C4FPV

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12024-02-01T21:41:30.936Zjob dependencies complete; ready to run (waiting for 1 h 25 m 55 s)
22024-02-01T22:48:19.221Zjob assigned to worker 01HNKDCQ458K11A7EC9QJ1FXX3 (queued for 1 h 6 m 48 s)
32024-02-01T22:48:19.233Zhttp://172.20.0.5:9969/ipxe/sock... ok
42024-02-01T22:48:19.313Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52024-02-01T22:48:19.368Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... 21%Error: No response to keepalive - Terminating session
62024-02-01T22:48:19.411ZError: No response de-activating SOL payload
72024-02-01T22:48:19.445Z[SOL Session operational. Use ~? for help]
82024-02-01T22:48:19.492Z 81%Error: No response to keepalive - Terminating session
92024-02-01T22:48:19.957ZError: No response de-activating SOL payload
102024-02-01T22:48:19.997Z[SOL Session operational. Use ~? for help]
112024-02-01T22:48:21.188Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-d85755d230534e96.gz
122024-02-01T22:48:25.744Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-d85755d230534e96.gz
132024-02-01T22:48:25.744Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
142024-02-01T22:48:52.339Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
152024-02-01T22:48:52.788Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2bc5efa67eecf70d.gz
162024-02-01T22:49:25.683Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2bc5efa67eecf70d.gz
172024-02-01T22:49:25.684Zdownloading input: /input/ci-tools/work/caboose-util.gz
182024-02-01T22:49:27.159Zdownloaded input: /input/ci-tools/work/caboose-util.gz
192024-02-01T22:49:27.159Zdownloading input: /input/ci-tools/work/tufaceous.gz
202024-02-01T22:49:31.398Zdownloaded input: /input/ci-tools/work/tufaceous.gz
212024-02-01T22:49:31.424Zdownloading input: /input/package/work/version.txt
222024-02-01T22:49:31.450Zdownloaded input: /input/package/work/version.txt
232024-02-01T22:49:31.450Zdownloading input: /input/package/work/package.tar.gz
242024-02-01T22:49:58.750Zdownloaded input: /input/package/work/package.tar.gz
252024-02-01T22:49:58.750Zdownloading input: /input/package/work/global-zone-packages.tar.gz
262024-02-01T22:50:42.369Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
272024-02-01T22:50:42.369Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
282024-02-01T22:50:54.597Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
292024-02-01T22:50:55.830Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
302024-02-01T22:51:16.519ZError: No response to keepalive - Terminating session
312024-02-01T22:51:26.726ZError: No response de-activating SOL payload
322024-02-01T22:51:27.841Z[SOL Session operational. Use ~? for help]
332024-02-01T22:51:48.305Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
342024-02-01T22:51:48.305Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
352024-02-01T22:52:34.564Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
362024-02-01T22:52:34.564Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
372024-02-01T22:53:17.633Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
382024-02-01T22:53:17.633Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
392024-02-01T22:53:32.099Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
402024-02-01T22:53:32.099Zdownloading input: /input/package/work/zones/crucible.tar.gz
412024-02-01T22:53:42.812Zdownloaded input: /input/package/work/zones/crucible.tar.gz
422024-02-01T22:53:42.845Zdownloading input: /input/package/work/zones/external-dns.tar.gz
432024-02-01T22:53:49.907Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
442024-02-01T22:53:49.907Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
452024-02-01T22:53:57.005Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
462024-02-01T22:53:57.006Zdownloading input: /input/package/work/zones/ntp.tar.gz
472024-02-01T22:53:57.100Zdownloaded input: /input/package/work/zones/ntp.tar.gz
482024-02-01T22:53:57.100Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
492024-02-01T22:54:01.124Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
502024-02-01T22:54:01.124Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
512024-02-01T22:54:06.488Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
522024-02-01T22:54:06.488Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
532024-02-01T22:54:16.513ZError: No response to keepalive - Terminating session
542024-02-01T22:54:26.473ZError: No response de-activating SOL payload
552024-02-01T22:54:27.851Z[SOL Session operational. Use ~? for help]
562024-02-01T22:54:31.891Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
572024-02-01T22:54:31.891Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
582024-02-01T22:54:59.597Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
592024-02-01T22:54:59.597Zdownloading input: /input/package/work/zones/overlay.tar.gz
602024-02-01T22:54:59.710Zdownloaded input: /input/package/work/zones/overlay.tar.gz
612024-02-01T22:54:59.710Zdownloading input: /input/package/work/zones/oximeter.tar.gz
622024-02-01T22:55:28.981Zdownloaded input: /input/package/work/zones/oximeter.tar.gz
632024-02-01T22:55:28.981Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
642024-02-01T22:55:58.419Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
652024-02-01T22:55:58.419Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
662024-02-01T22:57:21.551ZError: No response to keepalive - Terminating session
672024-02-01T22:57:21.843Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
682024-02-01T22:57:21.877Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
692024-02-01T22:57:26.688ZError: No response de-activating SOL payload
702024-02-01T22:57:27.806Z[SOL Session operational. Use ~? for help]
712024-02-01T22:58:42.496Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
722024-02-01T22:58:42.670Zstarting task 0: "setup"
732024-02-01T22:58:42.670Z++ uname -s
742024-02-01T22:58:42.670Z+ kern=SunOS
752024-02-01T22:58:42.670Z+ case "$kern" in
762024-02-01T22:58:42.670Z+ groupadd -g 12345 build
772024-02-01T22:58:42.671Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
782024-02-01T22:58:44.512Z+ zfs create -o mountpoint=/work rpool/work
792024-02-01T22:58:44.541Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
802024-02-01T22:58:44.541Z+ home_fs=zfs
812024-02-01T22:58:44.541Z+ [[ zfs == autofs ]]
822024-02-01T22:58:44.541Z+ mkdir -p /home/build
832024-02-01T22:58:44.542Z+ chown build:build /home/build /work
842024-02-01T22:58:46.544Z+ chmod 0700 /home/build /work
852024-02-01T22:58:46.585Zprocess exited: duration 4051 ms, exit code 0
 
862024-02-01T22:58:46.644Zstarting task 1: "authentication"
872024-02-01T22:58:46.695Zprocess exited: duration 48 ms, exit code 0
 
882024-02-01T22:58:46.762Zstarting task 2: "build"
892024-02-01T22:58:47.200Z+ trap _exit_trap EXIT
902024-02-01T22:58:47.200Z+ set +x
912024-02-01T22:59:33.605Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
922024-02-01T22:59:33.605Zsecpolicy_fs_linkdir: 0x55 = 0x55
932024-02-01T22:59:33.606Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
942024-02-01T22:59:33.606Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
952024-02-01T22:59:33.606Zsecpolicy_fs_linkdir+3: 0 = 0xe5
962024-02-01T22:59:33.606Zsecpolicy_fs_linkdir+4: 0 = 0xb8
972024-02-01T22:59:33.606Zsecpolicy_fs_linkdir+5: 0 = 0x1
982024-02-01T22:59:33.606Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
992024-02-01T22:59:33.606Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1002024-02-01T22:59:33.606Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1012024-02-01T22:59:33.606Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1022024-02-01T22:59:33.606Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1032024-02-01T22:59:33.606Z+ [[ -d /opt/oxide ]]
1042024-02-01T22:59:33.606Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1052024-02-01T22:59:33.606Z+ pfexec mkdir /opt/oxide
1062024-02-01T22:59:33.606Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1072024-02-01T22:59:33.606Z+ [[ -d /opt/oxide-underneath ]]
1082024-02-01T22:59:33.606Z+ cd /opt/oxide-underneath
1092024-02-01T22:59:33.606Z+ pfexec tar ceEp@/f - .
1102024-02-01T22:59:33.606Z+ cd /opt/oxide
1112024-02-01T22:59:33.606Z+ pfexec tar xveEp@/f -
1122024-02-01T22:59:33.644Zx ., 0 bytes, 0 tape blocks
1132024-02-01T22:59:33.644Zx ./opte, 0 bytes, 0 tape blocks
1142024-02-01T22:59:33.644Zx ./opte/bin, 0 bytes, 0 tape blocks
1152024-02-01T22:59:33.644Zx ./opte/bin/opteadm, 51799224 bytes, 101171 tape blocks
1162024-02-01T22:59:33.644Z+ pfexec rm -rf /opt/oxide-underneath
1172024-02-01T22:59:33.644Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1182024-02-01T22:59:33.644Z+ pfexec mkdir /opt/oxide/work
1192024-02-01T22:59:33.644Z+ pfexec chown build:build /opt/oxide/work
1202024-02-01T22:59:33.644Z+ cd /opt/oxide/work
1212024-02-01T22:59:33.644Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1222024-02-01T22:59:35.297ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1232024-02-01T22:59:35.297Zx out/mg-ddm-gz.tar, 63076864 bytes, 123197 tape blocks
1242024-02-01T22:59:35.298Zx out/omicron-sled-agent.tar, 122163712 bytes, 238601 tape blocks
1252024-02-01T22:59:35.298Zx out/oxlog.tar, 8382976 bytes, 16373 tape blocks
1262024-02-01T22:59:35.298Zx out/target/test, 75 bytes, 1 tape blocks
1272024-02-01T22:59:35.298Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1282024-02-01T22:59:35.298Zx package-manifest.toml, 23732 bytes, 47 tape blocks
1292024-02-01T22:59:35.299Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1302024-02-01T22:59:35.299Zx target/release/omicron-package, 44570944 bytes, 87053 tape blocks
1312024-02-01T22:59:35.299Zx tools/create_virtual_hardware.sh, 3184 bytes, 7 tape blocks
1322024-02-01T22:59:35.299Zx tools/virtual_hardware.sh, 3416 bytes, 7 tape blocks
1332024-02-01T22:59:35.299Zx tools/scrimlet/create-softnpu-zone.sh, 627 bytes, 2 tape blocks
1342024-02-01T22:59:35.299Zx tools/scrimlet/destroy-softnpu-zone.sh, 203 bytes, 1 tape blocks
1352024-02-01T22:59:35.300Zx tools/scrimlet/softnpu-init.sh, 2157 bytes, 5 tape blocks
1362024-02-01T22:59:35.300Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1372024-02-01T22:59:35.300Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1382024-02-01T22:59:35.336Z
1392024-02-01T22:59:35.336Zreal 1.669756747
1402024-02-01T22:59:35.337Zuser 1.501547108
1412024-02-01T22:59:35.337Zsys 0.294729063
1422024-02-01T22:59:35.337Ztrap 0.000073722
1432024-02-01T22:59:35.337Ztflt 0.000000000
1442024-02-01T22:59:35.337Zdflt 0.000000000
1452024-02-01T22:59:35.337Zkflt 0.000000000
1462024-02-01T22:59:35.337Zlock 0.000000000
1472024-02-01T22:59:35.337Zslp 1.502169308
1482024-02-01T22:59:35.337Zlat 0.039194896
1492024-02-01T22:59:35.337Zstop 0.000117302
1502024-02-01T22:59:35.337Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1512024-02-01T22:59:36.759Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1522024-02-01T22:59:36.759Z+ mkdir tests
1532024-02-01T22:59:36.759Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1542024-02-01T22:59:36.759Z+ ptime -m gunzip
1552024-02-01T22:59:36.759Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-d85755d230534e96
1562024-02-01T22:59:36.790Z
1572024-02-01T22:59:36.790Zreal 0.236223472
1582024-02-01T22:59:36.790Zuser 0.215490524
1592024-02-01T22:59:36.790Zsys 0.020506893
1602024-02-01T22:59:36.790Ztrap 0.000021450
1612024-02-01T22:59:36.790Ztflt 0.000000000
1622024-02-01T22:59:36.791Zdflt 0.000000000
1632024-02-01T22:59:36.791Zkflt 0.000000000
1642024-02-01T22:59:36.791Zlock 0.000000000
1652024-02-01T22:59:36.791Zslp 0.000018750
1662024-02-01T22:59:36.791Zlat 0.000016860
1672024-02-01T22:59:36.791Zstop 0.000112762
1682024-02-01T22:59:36.791Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-d85755d230534e96
1692024-02-01T22:59:36.791Z+ chmod a+x tests/bootstrap-d85755d230534e96
1702024-02-01T22:59:36.791Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1712024-02-01T22:59:36.792Z+ ptime -m gunzip
1722024-02-01T22:59:36.792Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1732024-02-01T22:59:39.326Z
1742024-02-01T22:59:39.358Zreal 2.538348503
1752024-02-01T22:59:39.358Zuser 2.391669657
1762024-02-01T22:59:39.359Zsys 0.146252718
1772024-02-01T22:59:39.359Ztrap 0.000212704
1782024-02-01T22:59:39.359Ztflt 0.000000000
1792024-02-01T22:59:39.359Zdflt 0.000000000
1802024-02-01T22:59:39.359Zkflt 0.000000000
1812024-02-01T22:59:39.359Zlock 0.000000000
1822024-02-01T22:59:39.359Zslp 0.000020560
1832024-02-01T22:59:39.359Zlat 0.000017900
1842024-02-01T22:59:39.359Zstop 0.000112262
1852024-02-01T22:59:39.360Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1862024-02-01T22:59:39.360Z+ chmod a+x tests/bootstrap
1872024-02-01T22:59:39.360Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1882024-02-01T22:59:39.360Z+ ptime -m gunzip
1892024-02-01T22:59:39.360Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2bc5efa67eecf70d
1902024-02-01T22:59:41.914Z
1912024-02-01T22:59:41.944Zreal 2.576368228
1922024-02-01T22:59:41.944Zuser 2.402256989
1932024-02-01T22:59:41.944Zsys 0.173688330
1942024-02-01T22:59:41.944Ztrap 0.000183032
1952024-02-01T22:59:41.945Ztflt 0.000000000
1962024-02-01T22:59:41.945Zdflt 0.000000000
1972024-02-01T22:59:41.945Zkflt 0.000000000
1982024-02-01T22:59:41.945Zlock 0.000000000
1992024-02-01T22:59:41.946Zslp 0.000022540
2002024-02-01T22:59:41.946Zlat 0.000022639
2012024-02-01T22:59:41.946Zstop 0.000135433
2022024-02-01T22:59:41.946Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-2bc5efa67eecf70d
2032024-02-01T22:59:41.947Z+ chmod a+x tests/end_to_end_tests-2bc5efa67eecf70d
2042024-02-01T22:59:41.947Z+ bmat address ls
2052024-02-01T22:59:41.947ZNAME CIDR FIRST LAST COUNT
2062024-02-01T22:59:41.947Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2072024-02-01T22:59:41.947Z++ bmat address ls -f extra -Ho first,last,count
2082024-02-01T22:59:41.947Z+ set -- 10.151.2.164 10.151.2.227 64
2092024-02-01T22:59:41.948Z+ EXTRA_IP_START=10.151.2.164
2102024-02-01T22:59:41.948Z+ EXTRA_IP_END=10.151.2.227
2112024-02-01T22:59:41.948Z+ EXTRA_IP_COUNT=64
2122024-02-01T22:59:41.948Z+ (( EXTRA_IP_COUNT >= 32 ))
2132024-02-01T22:59:41.948Z+ EXTRA_IP_BASE=10.151.2
2142024-02-01T22:59:41.948Z+ EXTRA_IP_FOCTET=164
2152024-02-01T22:59:41.948Z+ EXTRA_IP_LOCTET=227
2162024-02-01T22:59:41.949Z+ SERVICE_IP_POOL_START=10.151.2.164
2172024-02-01T22:59:41.949Z+ SERVICE_IP_POOL_END=10.151.2.173
2182024-02-01T22:59:41.949Z+ DNS_IP1=10.151.2.164
2192024-02-01T22:59:41.949Z+ DNS_IP2=10.151.2.165
2202024-02-01T22:59:41.949Z+ UPLINK_IP=10.151.2.174
2212024-02-01T22:59:41.949Z+ IPPOOL_START=10.151.2.175
2222024-02-01T22:59:41.949Z+ IPPOOL_END=10.151.2.227
2232024-02-01T22:59:41.981Z++ ipadm show-addr -po type,addr
2242024-02-01T22:59:41.981Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2252024-02-01T22:59:41.981Z+ GATEWAY_IP=10.151.2.100
2262024-02-01T22:59:41.982Z+ [[ -n 10.151.2.100 ]]
2272024-02-01T22:59:41.982Z+ ping -s 10.151.2.100 56 1
2282024-02-01T22:59:41.982ZPING 10.151.2.100: 56 data bytes
2292024-02-01T22:59:41.982Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2302024-02-01T22:59:41.982Z
2312024-02-01T22:59:41.983Z----10.151.2.100 PING Statistics----
2322024-02-01T22:59:41.983Z1 packets transmitted, 1 packets received, 0% packet loss
2332024-02-01T22:59:41.983Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2342024-02-01T22:59:41.984Z++ arp -an
2352024-02-01T22:59:41.984Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2362024-02-01T22:59:41.984Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2372024-02-01T22:59:41.984Z+ routeadm -e ipv4-forwarding -u
2382024-02-01T22:59:42.077Z+ PXA_START=10.151.2.164
2392024-02-01T22:59:42.078Z+ PXA_END=10.151.2.227
2402024-02-01T22:59:42.102Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2412024-02-01T22:59:42.102Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2422024-02-01T22:59:42.294Z+ ZPOOL_VDEV_DIR=/scratch
2432024-02-01T22:59:42.294Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2442024-02-01T22:59:42.319Z+++ dirname ./tools/create_virtual_hardware.sh
2452024-02-01T22:59:42.319Z++ cd ./tools
2462024-02-01T22:59:42.319Z++ pwd
2472024-02-01T22:59:42.319Z+ SOURCE_DIR=/opt/oxide/work/tools
2482024-02-01T22:59:42.319Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2492024-02-01T22:59:42.319Z+ SOFTNPU_MODE=zone
2502024-02-01T22:59:42.319Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512024-02-01T22:59:42.319Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522024-02-01T22:59:42.319Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532024-02-01T22:59:42.319Z++ dladm show-phys -p -o LINK
2542024-02-01T22:59:42.319Z++ head -1
2552024-02-01T22:59:42.319Z+ PHYSICAL_LINK=igb0
2562024-02-01T22:59:42.319Z+ echo 'Using igb0 as physical link'
2572024-02-01T22:59:42.319ZUsing igb0 as physical link
2582024-02-01T22:59:42.319Z+ ensure_run_as_root
2592024-02-01T22:59:42.319Z++ id -u
2602024-02-01T22:59:42.319Z+ [[ 0 -ne 0 ]]
2612024-02-01T22:59:42.319Z+ ensure_zpools
2622024-02-01T22:59:42.319Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632024-02-01T22:59:42.319Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642024-02-01T22:59:42.319Z+ readarray -t ZPOOLS
2652024-02-01T22:59:42.348Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662024-02-01T22:59:42.348Z++ sed 's/[ ",]//g'
2672024-02-01T22:59:42.348Z+ for ZPOOL in "${ZPOOLS[@]}"
2682024-02-01T22:59:42.348Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2692024-02-01T22:59:42.348ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2702024-02-01T22:59:42.348Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2712024-02-01T22:59:42.348Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2722024-02-01T22:59:42.348Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2732024-02-01T22:59:42.348Z0+0 records in
2742024-02-01T22:59:42.348Z0+0 records out
2752024-02-01T22:59:42.348Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2762024-02-01T22:59:42.348Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2772024-02-01T22:59:42.348Z+ set +x
2782024-02-01T22:59:42.348ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2792024-02-01T22:59:42.348Z++ zpool list -o name
2802024-02-01T22:59:42.348Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2812024-02-01T22:59:42.349Z+ [[ -z '' ]]
2822024-02-01T22:59:42.349Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2832024-02-01T22:59:42.383Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2842024-02-01T22:59:42.422Z+ set +x
2852024-02-01T22:59:42.422ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2862024-02-01T22:59:42.422Z+ for ZPOOL in "${ZPOOLS[@]}"
2872024-02-01T22:59:42.422Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2882024-02-01T22:59:42.422ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2892024-02-01T22:59:42.423Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2902024-02-01T22:59:42.423Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2912024-02-01T22:59:42.423Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2922024-02-01T22:59:42.423Z0+0 records in
2932024-02-01T22:59:42.423Z0+0 records out
2942024-02-01T22:59:42.423Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2952024-02-01T22:59:42.423Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2962024-02-01T22:59:42.423Z+ set +x
2972024-02-01T22:59:42.423ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2982024-02-01T22:59:42.423Z++ zpool list -o name
2992024-02-01T22:59:42.423Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3002024-02-01T22:59:42.423Z+ [[ -z '' ]]
3012024-02-01T22:59:42.423Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3022024-02-01T22:59:42.462Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3032024-02-01T22:59:42.462Z+ set +x
3042024-02-01T22:59:42.462ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3052024-02-01T22:59:42.462Z+ for ZPOOL in "${ZPOOLS[@]}"
3062024-02-01T22:59:42.462ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3072024-02-01T22:59:42.462Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082024-02-01T22:59:42.462Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3092024-02-01T22:59:42.462Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3102024-02-01T22:59:42.462Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3112024-02-01T22:59:42.463Z0+0 records in
3122024-02-01T22:59:42.463Z0+0 records out
3132024-02-01T22:59:42.463Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3142024-02-01T22:59:42.463Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3152024-02-01T22:59:42.463Z+ set +x
3162024-02-01T22:59:42.463ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3172024-02-01T22:59:42.463Z++ zpool list -o name
3182024-02-01T22:59:42.463Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3192024-02-01T22:59:42.464Z+ [[ -z '' ]]
3202024-02-01T22:59:42.464Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3212024-02-01T22:59:42.502Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3222024-02-01T22:59:42.503Z+ set +x
3232024-02-01T22:59:42.503ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3242024-02-01T22:59:42.503Z+ for ZPOOL in "${ZPOOLS[@]}"
3252024-02-01T22:59:42.503Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3262024-02-01T22:59:42.503ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3272024-02-01T22:59:42.503Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3282024-02-01T22:59:42.504Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3292024-02-01T22:59:42.504Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3302024-02-01T22:59:42.504Z0+0 records in
3312024-02-01T22:59:42.504Z0+0 records out
3322024-02-01T22:59:42.504Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3332024-02-01T22:59:42.504Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3342024-02-01T22:59:42.505Z+ set +x
3352024-02-01T22:59:42.505ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3362024-02-01T22:59:42.505Z++ zpool list -o name
3372024-02-01T22:59:42.505Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3382024-02-01T22:59:42.505Z+ [[ -z '' ]]
3392024-02-01T22:59:42.505Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3402024-02-01T22:59:42.544Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3412024-02-01T22:59:42.544Z+ set +x
3422024-02-01T22:59:42.544ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3432024-02-01T22:59:42.544Z+ for ZPOOL in "${ZPOOLS[@]}"
3442024-02-01T22:59:42.544Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3452024-02-01T22:59:42.544ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3462024-02-01T22:59:42.544Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3472024-02-01T22:59:42.544Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3482024-02-01T22:59:42.545Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3492024-02-01T22:59:42.545Z0+0 records in
3502024-02-01T22:59:42.545Z0+0 records out
3512024-02-01T22:59:42.545Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3522024-02-01T22:59:42.545Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3532024-02-01T22:59:42.545Z+ set +x
3542024-02-01T22:59:42.545ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3552024-02-01T22:59:42.545Z++ zpool list -o name
3562024-02-01T22:59:42.545Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3572024-02-01T22:59:42.545Z+ [[ -z '' ]]
3582024-02-01T22:59:42.545Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3592024-02-01T22:59:42.577Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3602024-02-01T22:59:42.577Z+ set +x
3612024-02-01T22:59:42.577ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3622024-02-01T22:59:42.577Z+ for ZPOOL in "${ZPOOLS[@]}"
3632024-02-01T22:59:42.577Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3642024-02-01T22:59:42.577ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3652024-02-01T22:59:42.577Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3662024-02-01T22:59:42.577Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3672024-02-01T22:59:42.577Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3682024-02-01T22:59:42.577Z0+0 records in
3692024-02-01T22:59:42.577Z0+0 records out
3702024-02-01T22:59:42.578Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3712024-02-01T22:59:42.578Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3722024-02-01T22:59:42.578Z+ set +x
3732024-02-01T22:59:42.578ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3742024-02-01T22:59:42.578Z++ zpool list -o name
3752024-02-01T22:59:42.578Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3762024-02-01T22:59:42.611Z+ [[ -z '' ]]
3772024-02-01T22:59:42.611Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3782024-02-01T22:59:42.637Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3792024-02-01T22:59:42.637Z+ set +x
3802024-02-01T22:59:42.637ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3812024-02-01T22:59:42.637Z+ for ZPOOL in "${ZPOOLS[@]}"
3822024-02-01T22:59:42.637Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3832024-02-01T22:59:42.637ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3842024-02-01T22:59:42.637Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3852024-02-01T22:59:42.637Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3862024-02-01T22:59:42.637Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3872024-02-01T22:59:42.637Z0+0 records in
3882024-02-01T22:59:42.637Z0+0 records out
3892024-02-01T22:59:42.637Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3902024-02-01T22:59:42.637Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3912024-02-01T22:59:42.637Z+ set +x
3922024-02-01T22:59:42.637ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3932024-02-01T22:59:42.637Z++ zpool list -o name
3942024-02-01T22:59:42.637Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3952024-02-01T22:59:42.638Z+ [[ -z '' ]]
3962024-02-01T22:59:42.638Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3972024-02-01T22:59:42.662Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3982024-02-01T22:59:42.662Z+ set +x
3992024-02-01T22:59:42.662ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4002024-02-01T22:59:42.662Z+ for ZPOOL in "${ZPOOLS[@]}"
4012024-02-01T22:59:42.662Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4022024-02-01T22:59:42.662ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4032024-02-01T22:59:42.662Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4042024-02-01T22:59:42.662Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4052024-02-01T22:59:42.662Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4062024-02-01T22:59:42.662Z0+0 records in
4072024-02-01T22:59:42.662Z0+0 records out
4082024-02-01T22:59:42.662Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4092024-02-01T22:59:42.662Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4102024-02-01T22:59:42.663Z+ set +x
4112024-02-01T22:59:42.663ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4122024-02-01T22:59:42.687Z++ zpool list -o name
4132024-02-01T22:59:42.687Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4142024-02-01T22:59:42.687Z+ [[ -z '' ]]
4152024-02-01T22:59:42.687Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4162024-02-01T22:59:42.757Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4172024-02-01T22:59:42.757Z+ set +x
4182024-02-01T22:59:42.758ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4192024-02-01T22:59:42.758Z+ for ZPOOL in "${ZPOOLS[@]}"
4202024-02-01T22:59:42.758Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4212024-02-01T22:59:42.758ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4222024-02-01T22:59:42.758Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4232024-02-01T22:59:42.758Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4242024-02-01T22:59:42.759Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4252024-02-01T22:59:42.759Z0+0 records in
4262024-02-01T22:59:42.759Z0+0 records out
4272024-02-01T22:59:42.759Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4282024-02-01T22:59:42.759Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4292024-02-01T22:59:42.759Z+ set +x
4302024-02-01T22:59:42.760ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4312024-02-01T22:59:42.760Z++ zpool list -o name
4322024-02-01T22:59:42.760Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4332024-02-01T22:59:42.790Z+ [[ -z '' ]]
4342024-02-01T22:59:42.790Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4352024-02-01T22:59:42.790Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4362024-02-01T22:59:42.790Z+ set +x
4372024-02-01T22:59:42.790ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4382024-02-01T22:59:42.790Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4392024-02-01T22:59:42.790Z+ readarray -t ZPOOLS
4402024-02-01T22:59:42.790Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4412024-02-01T22:59:42.790Z++ sed 's/[ ",]//g'
4422024-02-01T22:59:42.790Z+ for ZPOOL in "${ZPOOLS[@]}"
4432024-02-01T22:59:42.790Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4442024-02-01T22:59:42.791ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4452024-02-01T22:59:42.791Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4462024-02-01T22:59:42.791Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4472024-02-01T22:59:42.791Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4482024-02-01T22:59:42.791Z0+0 records in
4492024-02-01T22:59:42.791Z0+0 records out
4502024-02-01T22:59:42.791Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4512024-02-01T22:59:42.791Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4522024-02-01T22:59:42.791Z+ set +x
4532024-02-01T22:59:42.792ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4542024-02-01T22:59:42.792Z++ zpool list -o name
4552024-02-01T22:59:42.792Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4562024-02-01T22:59:42.792Z+ [[ -z '' ]]
4572024-02-01T22:59:42.792Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4582024-02-01T22:59:42.817Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4592024-02-01T22:59:42.817Z+ set +x
4602024-02-01T22:59:42.817ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4612024-02-01T22:59:42.818Z+ for ZPOOL in "${ZPOOLS[@]}"
4622024-02-01T22:59:42.818Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4632024-02-01T22:59:42.818ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4642024-02-01T22:59:42.818Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4652024-02-01T22:59:42.818Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4662024-02-01T22:59:42.818Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4672024-02-01T22:59:42.818Z0+0 records in
4682024-02-01T22:59:42.818Z0+0 records out
4692024-02-01T22:59:42.819Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4702024-02-01T22:59:42.819Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4712024-02-01T22:59:42.819Z+ set +x
4722024-02-01T22:59:42.819ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4732024-02-01T22:59:42.819Z++ zpool list -o name
4742024-02-01T22:59:42.819Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4752024-02-01T22:59:42.846Z+ [[ -z '' ]]
4762024-02-01T22:59:42.847Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4772024-02-01T22:59:42.877Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4782024-02-01T22:59:42.877Z+ set +x
4792024-02-01T22:59:42.877ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4802024-02-01T22:59:42.877Z+ [[ zone == \z\o\n\e ]]
4812024-02-01T22:59:42.877Z+ ensure_simulated_links igb0
4822024-02-01T22:59:42.877Z+ local PHYSICAL_LINK=igb0
4832024-02-01T22:59:42.877Z+ INDICES=("0" "1")
4842024-02-01T22:59:42.877Z+ for I in "${INDICES[@]}"
4852024-02-01T22:59:42.877Z++ get_simnet_name_if_exists net0
4862024-02-01T22:59:42.877Z++ dladm show-simnet -p -o LINK net0
4872024-02-01T22:59:42.877Z++ echo ''
4882024-02-01T22:59:42.877Z+ [[ -z '' ]]
4892024-02-01T22:59:42.877Z+ dladm create-simnet -t net0
4902024-02-01T22:59:42.877Z+ dladm create-simnet -t sc0_0
4912024-02-01T22:59:42.901Z+ dladm modify-simnet -t -p net0 sc0_0
4922024-02-01T22:59:42.901Z+ dladm set-linkprop -p mtu=9000 sc0_0
4932024-02-01T22:59:42.902Z+ success 'Simnet net0/sc0_0 exists'
4942024-02-01T22:59:42.902Z+ set +x
4952024-02-01T22:59:42.902ZSimnet net0/sc0_0 exists
4962024-02-01T22:59:42.902Z+ for I in "${INDICES[@]}"
4972024-02-01T22:59:42.902Z++ get_simnet_name_if_exists net1
4982024-02-01T22:59:42.902Z++ dladm show-simnet -p -o LINK net1
4992024-02-01T22:59:42.902Z++ echo ''
5002024-02-01T22:59:42.902Z+ [[ -z '' ]]
5012024-02-01T22:59:42.902Z+ dladm create-simnet -t net1
5022024-02-01T22:59:42.902Z+ dladm create-simnet -t sc1_0
5032024-02-01T22:59:42.902Z+ dladm modify-simnet -t -p net1 sc1_0
5042024-02-01T22:59:42.927Z+ dladm set-linkprop -p mtu=9000 sc1_0
5052024-02-01T22:59:42.927Z+ success 'Simnet net1/sc1_0 exists'
5062024-02-01T22:59:42.927Z+ set +x
5072024-02-01T22:59:42.927ZSimnet net1/sc1_0 exists
5082024-02-01T22:59:42.928Z++ get_vnic_name_if_exists sc0_1
5092024-02-01T22:59:42.928Z++ dladm show-vnic -p -o LINK sc0_1
5102024-02-01T22:59:42.928Z++ echo ''
5112024-02-01T22:59:42.928Z+ [[ -z '' ]]
5122024-02-01T22:59:42.928Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5132024-02-01T22:59:42.928Z+ [[ -v PROMISC_FILT_OFF ]]
5142024-02-01T22:59:42.928Z+ success 'Vnic sc0_1 exists'
5152024-02-01T22:59:42.928Z+ set +x
5162024-02-01T22:59:42.928ZVnic sc0_1 exists
5172024-02-01T22:59:42.928Z+ warn_if_no_proxy_arp
5182024-02-01T22:59:42.928Z+ [[ -v PXA_START ]]
5192024-02-01T22:59:42.928Z+ [[ -v PXA_END ]]
5202024-02-01T22:59:42.928Z+ ensure_softnpu_zone
5212024-02-01T22:59:42.928Z+ zoneadm list
5222024-02-01T22:59:42.928Z+ grep -q sidecar_softnpu
5232024-02-01T22:59:42.929Z+ [[ -f out/npuzone/npuzone ]]
5242024-02-01T22:59:42.929Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit e3ea4b495ba0a71801ded0776ae4bbd31df57e26 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5252024-02-01T22:59:42.955Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5262024-02-01T22:59:58.512Zdownloading softnpu ...done
5272024-02-01T22:59:58.558Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/libsidecar_lite.so.sha256.txt
 
5282024-02-01T23:00:16.485ZError: No response to keepalive - Terminating session
 
5292024-02-01T23:00:17.806Zdownloading asic_program.so ...done
5302024-02-01T23:00:17.831Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/scadm.sha256.txt
 
5312024-02-01T23:00:26.683ZError: No response de-activating SOL payload
5322024-02-01T23:00:27.782Z[SOL Session operational. Use ~? for help]
 
5332024-02-01T23:00:31.813Zdownloading scadm ...done
5342024-02-01T23:00:31.853Zcreating port tfportrear0_0 for sc0_0
5352024-02-01T23:00:31.853Zcreating port tfportqsfp0_0 for sc0_1
5362024-02-01T23:00:31.853Zsoftnpu zone setup
5372024-02-01T23:00:31.853Zconfigure zone
5382024-02-01T23:00:31.890Zinstall zone
5392024-02-01T23:00:32.925Zboot zone
5402024-02-01T23:00:36.025Z[sidecar_softnpu] svcs milestone/network
5412024-02-01T23:00:36.052ZSTATE STIME FMRI
5422024-02-01T23:00:36.053Zonline 23:00:34 svc:/milestone/network:default
5432024-02-01T23:00:36.053Z[sidecar_softnpu] /softnpu/npu start
5442024-02-01T23:00:36.088Z
5452024-02-01T23:00:36.125Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5462024-02-01T23:00:36.125Z+ GATEWAY_IP=10.151.2.100
5472024-02-01T23:00:36.125ZUsing 10.151.2.100 as gateway ip
5482024-02-01T23:00:36.125Z+ echo 'Using 10.151.2.100 as gateway ip'
5492024-02-01T23:00:36.125Z+ [[ ! -v GATEWAY_MAC ]]
5502024-02-01T23:00:36.125Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5512024-02-01T23:00:36.126Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5522024-02-01T23:00:36.126Z+++ echo 18:c0:4d:0d:a0:2a
5532024-02-01T23:00:36.126Z+ COUNT=1
5542024-02-01T23:00:36.126Z+ [[ 1 -ne 1 ]]
5552024-02-01T23:00:36.126Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5562024-02-01T23:00:36.126ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5572024-02-01T23:00:36.126Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5582024-02-01T23:00:36.126Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5592024-02-01T23:00:36.152Z+ PXA_MAC=a8:e1:de:01:70:1d
5602024-02-01T23:00:36.152Z+ [[ -v PXA_START ]]
5612024-02-01T23:00:36.152Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5622024-02-01T23:00:36.152Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5632024-02-01T23:00:36.152Z+ z_scadm dump-state
5642024-02-01T23:00:36.153Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5652024-02-01T23:00:36.183Zlocal v6:
5662024-02-01T23:00:36.183Zlocal v4:
5672024-02-01T23:00:36.184Zrouter v6:
5682024-02-01T23:00:36.184Zrouter v4_idx:
5692024-02-01T23:00:36.184Zrouter v4_routes:
5702024-02-01T23:00:36.184Zresolver v4:
5712024-02-01T23:00:36.184Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5722024-02-01T23:00:36.184Zresolver v6:
5732024-02-01T23:00:36.184Znat v4:
5742024-02-01T23:00:36.184Znat_v6:
5752024-02-01T23:00:36.184Zport_mac:
5762024-02-01T23:00:36.184Zproxy arp:
5772024-02-01T23:00:36.184Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5782024-02-01T23:00:36.209Z+ success 'softnpu zone exists'
5792024-02-01T23:00:36.210Z+ set +x
5802024-02-01T23:00:36.210Zsoftnpu zone exists
5812024-02-01T23:00:36.210Z
5822024-02-01T23:00:36.210Zreal 53.884918872
5832024-02-01T23:00:36.210Zuser 0.684760704
5842024-02-01T23:00:36.210Zsys 1.548939600
5852024-02-01T23:00:36.210Ztrap 0.000078548
5862024-02-01T23:00:36.210Ztflt 0.000155397
5872024-02-01T23:00:36.210Zdflt 0.001136952
5882024-02-01T23:00:36.210Zkflt 0.000001090
5892024-02-01T23:00:36.210Zlock 42:26.839473841
5902024-02-01T23:00:36.210Zslp 2:41.385846368
5912024-02-01T23:00:36.210Zlat 0.068631691
5922024-02-01T23:00:36.210Zstop 0.353233776
5932024-02-01T23:00:36.210Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5942024-02-01T23:00:36.275Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952024-02-01T23:00:36.275Z+ SILO_NAME=recovery
5962024-02-01T23:00:36.275Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5972024-02-01T23:00:36.275Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5982024-02-01T23:00:36.275Z+ sed -i~ '
5992024-02-01T23:00:36.275Z /^external_dns_ips/c\
6002024-02-01T23:00:36.275Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6012024-02-01T23:00:36.275Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6022024-02-01T23:00:36.275Z /^first/c\
6032024-02-01T23:00:36.275Zfirst = "10.151.2.164"
6042024-02-01T23:00:36.275Z /^last/c\
6052024-02-01T23:00:36.275Zlast = "10.151.2.173"
6062024-02-01T23:00:36.276Z }
6072024-02-01T23:00:36.276Z /^\[rack_network_config/,/^$/ {
6082024-02-01T23:00:36.276Z /^infra_ip_first/c\
6092024-02-01T23:00:36.276Zinfra_ip_first = "10.151.2.174"
6102024-02-01T23:00:36.276Z /^infra_ip_last/c\
6112024-02-01T23:00:36.276Zinfra_ip_last = "10.151.2.174"
6122024-02-01T23:00:36.276Z }
6132024-02-01T23:00:36.276Z /^\[\[rack_network_config.ports/,/^$/ {
6142024-02-01T23:00:36.276Z /^routes/c\
6152024-02-01T23:00:36.276Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6162024-02-01T23:00:36.276Z /^addresses/c\
6172024-02-01T23:00:36.276Zaddresses = \["10.151.2.174/24"\]
6182024-02-01T23:00:36.276Z }
6192024-02-01T23:00:36.276Z' pkg/config-rss.toml
6202024-02-01T23:00:36.276Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6212024-02-01T23:00:36.276Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6222024-02-01T23:00:36.276Z+++ pkg/config-rss.toml Thu Feb 1 23:00:36 2024
6232024-02-01T23:00:36.276Z@@ -41,7 +41,7 @@
6242024-02-01T23:00:36.276Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6252024-02-01T23:00:36.276Z # must be contained in one of the "internal services" IP Pool ranges listed
6262024-02-01T23:00:36.276Z # below.
6272024-02-01T23:00:36.277Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6282024-02-01T23:00:36.277Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6292024-02-01T23:00:36.277Z
6302024-02-01T23:00:36.277Z # Initial TLS certificates for the external API
6312024-02-01T23:00:36.277Z #
6322024-02-01T23:00:36.277Z@@ -77,8 +77,8 @@
6332024-02-01T23:00:36.277Z #
6342024-02-01T23:00:36.277Z # For more on this and what to put here, see docs/how-to-run.adoc.
6352024-02-01T23:00:36.277Z [[internal_services_ip_pool_ranges]]
6362024-02-01T23:00:36.277Z-first = "192.168.1.20"
6372024-02-01T23:00:36.277Z-last = "192.168.1.29"
6382024-02-01T23:00:36.277Z+first = "10.151.2.164"
6392024-02-01T23:00:36.277Z+last = "10.151.2.173"
6402024-02-01T23:00:36.277Z
6412024-02-01T23:00:36.277Z # TODO - this configuration is subject to change going forward. Ultimately these
6422024-02-01T23:00:36.277Z # parameters should be provided to the control plane via wicket, but we need to
6432024-02-01T23:00:36.277Z@@ -92,8 +92,8 @@
6442024-02-01T23:00:36.277Z # A range of IP addresses used by Boundary Services on the external network. In
6452024-02-01T23:00:36.277Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6462024-02-01T23:00:36.277Z # With softnpu, only one address is used.
6472024-02-01T23:00:36.277Z-infra_ip_first = "192.168.1.30"
6482024-02-01T23:00:36.277Z-infra_ip_last = "192.168.1.30"
6492024-02-01T23:00:36.277Z+infra_ip_first = "10.151.2.174"
6502024-02-01T23:00:36.277Z+infra_ip_last = "10.151.2.174"
6512024-02-01T23:00:36.277Z
6522024-02-01T23:00:36.278Z # Configurations for BGP routers to run on the scrimlets.
6532024-02-01T23:00:36.278Z bgp = []
6542024-02-01T23:00:36.278Z@@ -101,9 +101,9 @@
6552024-02-01T23:00:36.278Z # You can configure multiple uplinks by repeating the following stanza
6562024-02-01T23:00:36.278Z [[rack_network_config.ports]]
6572024-02-01T23:00:36.278Z # Routes associated with this port.
6582024-02-01T23:00:36.278Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6592024-02-01T23:00:36.278Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6602024-02-01T23:00:36.278Z # Addresses associated with this port.
6612024-02-01T23:00:36.278Z-addresses = ["192.168.1.30/24"]
6622024-02-01T23:00:36.278Z+addresses = ["10.151.2.174/24"]
6632024-02-01T23:00:36.278Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6642024-02-01T23:00:36.278Z port = "qsfp0"
6652024-02-01T23:00:36.278Z # The speed of this port.
6662024-02-01T23:00:36.278Z+ true
6672024-02-01T23:00:36.278Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6682024-02-01T23:00:36.278Za pkg/config-rss.toml 7K
6692024-02-01T23:00:36.278Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6702024-02-01T23:00:36.278Z+ TLS_NAME=recovery.sys.oxide.test
6712024-02-01T23:00:36.278Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6722024-02-01T23:00:36.714Z..+.+..+....+...+..+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*................+..+...............+....+.....+....+.....+...+..........+........+..........+......+...........+....+..............+....+...+.................+.+......+.....+.+..+.......+...........+.......+..+......+...........................+...+....+..............+.+...........+....+.....+.........+.+.....+....+.........+...+..+...............+.+............+...+......+........+.......+.....+...+...+...............+..........+...+..+...+....+....................+...+....+........+......................+........+............+..........+......+...+...........+...+.+....................+..........+.....+.+...+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6732024-02-01T23:00:37.120Z..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+....+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..................+...........+.......+.........+.....+...+.+.....+.........+...+...+......................+.....+....+..............+.+...........+....+...+..+....+.....+.........+.+..+..................+.+.........+.....+......+......+...........................+......+.+........+...............+...+..........+..................+...........+....+......+...+.....+.............+...+...........+...+......................+.....+......+....+..............+....+..+...............+...+......+.........+......+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6742024-02-01T23:00:37.145Z-----
6752024-02-01T23:00:37.145Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6762024-02-01T23:00:37.145Za pkg/initial-tls-cert.pem 2K
6772024-02-01T23:00:37.145Za pkg/initial-tls-key.pem 4K
6782024-02-01T23:00:37.145Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6792024-02-01T23:00:37.170Z+ rmdir pkg
6802024-02-01T23:00:37.170Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6812024-02-01T23:00:37.170Z+ pfexec mkdir -p /usr/oxide
6822024-02-01T23:00:37.170Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6832024-02-01T23:00:37.810Z+ pfexec chmod +x /usr/oxide/oxide
6842024-02-01T23:00:37.837Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6852024-02-01T23:00:57.274Z+ OMICRON_NO_UNINSTALL=1
6862024-02-01T23:00:57.274Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6872024-02-01T23:00:57.274ZFeb 01 23:00:56.525 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6882024-02-01T23:00:57.274ZFeb 01 23:00:56.527 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6892024-02-01T23:00:57.274ZFeb 01 23:00:56.527 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6902024-02-01T23:00:57.274ZFeb 01 23:00:56.527 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6912024-02-01T23:00:57.274ZFeb 01 23:00:56.527 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6922024-02-01T23:00:57.274ZFeb 01 23:00:56.528 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6932024-02-01T23:00:57.275ZFeb 01 23:00:56.528 INFO Installing service, dst: /opt/oxide/oxlog.tar, src: out/oxlog.tar
6942024-02-01T23:00:57.275ZFeb 01 23:00:56.528 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6952024-02-01T23:00:57.275ZFeb 01 23:00:56.528 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6962024-02-01T23:00:57.275ZFeb 01 23:00:56.528 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6972024-02-01T23:00:57.275ZFeb 01 23:00:56.528 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6982024-02-01T23:00:57.275ZFeb 01 23:00:56.528 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6992024-02-01T23:00:57.275ZFeb 01 23:00:56.528 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
7002024-02-01T23:00:57.275ZFeb 01 23:00:56.528 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7012024-02-01T23:00:57.275ZFeb 01 23:00:56.528 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7022024-02-01T23:00:57.275ZFeb 01 23:00:56.528 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7032024-02-01T23:00:57.275ZFeb 01 23:00:56.528 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter.tar.gz
7042024-02-01T23:00:57.275ZFeb 01 23:00:56.528 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7052024-02-01T23:00:57.275ZFeb 01 23:00:56.771 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062024-02-01T23:00:57.275ZFeb 01 23:00:56.818 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072024-02-01T23:00:57.275ZFeb 01 23:00:56.881 INFO Unpacking service tarball, service_path: /opt/oxide/oxlog, tar_path: /opt/oxide/oxlog.tar
7082024-02-01T23:00:57.275ZFeb 01 23:00:56.886 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7092024-02-01T23:00:57.275Z
7102024-02-01T23:00:57.276Zreal 0.502482304
7112024-02-01T23:00:57.276Zuser 0.111764534
7122024-02-01T23:00:57.276Zsys 1.954545151
7132024-02-01T23:00:57.305Ztrap 0.000003929
7142024-02-01T23:00:57.305Ztflt 0.000122320
7152024-02-01T23:00:57.305Zdflt 0.000762685
7162024-02-01T23:00:57.305Zkflt 0.000000000
7172024-02-01T23:00:57.305Zlock 45.193448709
7182024-02-01T23:00:57.305Zslp 0.720609997
7192024-02-01T23:00:57.305Zlat 0.003421738
7202024-02-01T23:00:57.305Zstop 0.209277988
7212024-02-01T23:00:57.305Z+ retry=0
7222024-02-01T23:00:57.305Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232024-02-01T23:00:57.305Z+ [[ 0 -gt 30 ]]
7242024-02-01T23:00:57.305Z+ sleep 1
7252024-02-01T23:00:57.648Z+ retry=1
7262024-02-01T23:00:57.648Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272024-02-01T23:00:57.699Z+ [[ 1 -gt 30 ]]
7282024-02-01T23:00:57.699Z+ sleep 1
7292024-02-01T23:00:58.653Z+ retry=2
7302024-02-01T23:00:58.654Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312024-02-01T23:00:58.685Z+ [[ 2 -gt 30 ]]
7322024-02-01T23:00:58.685Z+ sleep 1
7332024-02-01T23:00:59.663Z+ retry=3
7342024-02-01T23:00:59.663Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352024-02-01T23:00:59.691Z+ [[ 3 -gt 30 ]]
7362024-02-01T23:00:59.691Z+ sleep 1
7372024-02-01T23:01:00.683Z+ retry=4
7382024-02-01T23:01:00.683Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7392024-02-01T23:01:25.860Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7402024-02-01T23:01:25.900Zlocal v6:
7412024-02-01T23:01:25.900Zfd45:6596:da46:1549::1
7422024-02-01T23:01:25.900Zfe80::14bd:eeff:fe4c:ec15
7432024-02-01T23:01:25.900Zlocal v4:
7442024-02-01T23:01:25.900Z10.151.2.174
7452024-02-01T23:01:25.900Zrouter v6:
7462024-02-01T23:01:25.900Zfd00:1122:3344:2::/64 -> fe80::643a:45ff:fe91:b08 (1)
7472024-02-01T23:01:25.900Zfd00:1122:3344:101::/64 -> fe80::643a:45ff:fe91:b08 (1)
7482024-02-01T23:01:25.900Zfd00:1122:3344:1::/64 -> fe80::643a:45ff:fe91:b08 (1)
7492024-02-01T23:01:25.900Zfdb0:18c0:4d0d:a02a::/64 -> fe80::643a:45ff:fe91:b08 (1)
7502024-02-01T23:01:25.900Zfd00:1122:3344:3::/64 -> fe80::643a:45ff:fe91:b08 (1)
7512024-02-01T23:01:25.900Zrouter v4_idx:
7522024-02-01T23:01:25.900Z0.0.0.0/0 -> 0
7532024-02-01T23:01:25.900Zrouter v4_routes:
7542024-02-01T23:01:25.900Z0 -> 10.151.2.100 (2)
7552024-02-01T23:01:25.900Zresolver v4:
7562024-02-01T23:01:25.900Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7572024-02-01T23:01:25.900Z10.151.2.174 -> d2:8d:ae:82:a7:dd
7582024-02-01T23:01:25.900Zresolver v6:
7592024-02-01T23:01:25.900Zfe80::14bd:eeff:fe4c:ec15 -> 16:bd:ee:4c:ec:15
7602024-02-01T23:01:25.901Zfe80::643a:45ff:fe91:b08 -> 66:3a:45:91:0b:08
7612024-02-01T23:01:25.901Znat v4:
7622024-02-01T23:01:25.901Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:97:21
7632024-02-01T23:01:25.901Znat_v6:
7642024-02-01T23:01:25.901Zport_mac:
7652024-02-01T23:01:25.901Z1: a8:40:25:1c:3c:02
7662024-02-01T23:01:25.901Z1000: a8:40:25:1c:3c:01
7672024-02-01T23:01:25.901Zproxy arp:
7682024-02-01T23:01:25.901Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7692024-02-01T23:01:25.928Z+ retry=0
7702024-02-01T23:01:25.928Z+++ zoneadm list -n
7712024-02-01T23:01:25.928Z+++ grep oxz_ntp
7722024-02-01T23:01:25.928Z++ pfexec svcs -z oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7 -Hostate oxide/ntp
7732024-02-01T23:01:25.978Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7742024-02-01T23:01:26.074Z++ true
7752024-02-01T23:01:26.074Z+ [[ '' != online ]]
7762024-02-01T23:01:26.074Z+ [[ 0 -gt 60 ]]
7772024-02-01T23:01:26.074Z+ sleep 1
7782024-02-01T23:01:26.975Z+ retry=1
7792024-02-01T23:01:27.009Z+++ zoneadm list -n
7802024-02-01T23:01:27.009Z+++ grep oxz_ntp
7812024-02-01T23:01:27.009Z++ pfexec svcs -z oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7 -Hostate oxide/ntp
7822024-02-01T23:01:27.040Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7832024-02-01T23:01:27.040Z++ true
7842024-02-01T23:01:27.041Z+ [[ '' != online ]]
7852024-02-01T23:01:27.041Z+ [[ 1 -gt 60 ]]
7862024-02-01T23:01:27.041Z+ sleep 1
7872024-02-01T23:01:28.009Z+ retry=2
7882024-02-01T23:01:28.042Z+++ zoneadm list -n
7892024-02-01T23:01:28.042Z+++ grep oxz_ntp
7902024-02-01T23:01:28.042Z++ pfexec svcs -z oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7 -Hostate oxide/ntp
7912024-02-01T23:01:28.042Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7922024-02-01T23:01:28.043Z++ true
7932024-02-01T23:01:28.101Z+ [[ '' != online ]]
7942024-02-01T23:01:28.101Z+ [[ 2 -gt 60 ]]
7952024-02-01T23:01:28.102Z+ sleep 1
7962024-02-01T23:01:29.040Z+ retry=3
7972024-02-01T23:01:29.076Z+++ zoneadm list -n
7982024-02-01T23:01:29.077Z+++ grep oxz_ntp
7992024-02-01T23:01:29.077Z++ pfexec svcs -z oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7 -Hostate oxide/ntp
8002024-02-01T23:01:29.077Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8012024-02-01T23:01:29.077Z++ true
8022024-02-01T23:01:29.078Z+ [[ '' != online ]]
8032024-02-01T23:01:29.078Z+ [[ 3 -gt 60 ]]
8042024-02-01T23:01:29.078Z+ sleep 1
8052024-02-01T23:01:30.074Z+ retry=4
8062024-02-01T23:01:30.120Z+++ zoneadm list -n
8072024-02-01T23:01:30.120Z+++ grep oxz_ntp
8082024-02-01T23:01:30.120Z++ pfexec svcs -z oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7 -Hostate oxide/ntp
8092024-02-01T23:01:30.120Z+ [[ disabled != online ]]
8102024-02-01T23:01:30.120Z+ [[ 4 -gt 60 ]]
8112024-02-01T23:01:30.120Z+ sleep 1
8122024-02-01T23:01:31.098Z+ retry=5
8132024-02-01T23:01:31.133Z+++ zoneadm list -n
8142024-02-01T23:01:31.133Z+++ grep oxz_ntp
8152024-02-01T23:01:31.133Z++ pfexec svcs -z oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7 -Hostate oxide/ntp
8162024-02-01T23:01:31.133Z+ [[ disabled != online ]]
8172024-02-01T23:01:31.134Z+ [[ 5 -gt 60 ]]
8182024-02-01T23:01:31.134Z+ sleep 1
8192024-02-01T23:01:32.264Z+ retry=6
8202024-02-01T23:01:32.289Z+++ zoneadm list -n
8212024-02-01T23:01:32.289Z+++ grep oxz_ntp
8222024-02-01T23:01:32.290Z++ pfexec svcs -z oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7 -Hostate oxide/ntp
8232024-02-01T23:01:32.290Z+ [[ disabled != online ]]
8242024-02-01T23:01:32.290Z+ [[ 6 -gt 60 ]]
8252024-02-01T23:01:32.290Z+ sleep 1
8262024-02-01T23:01:33.137Z+ retry=7
8272024-02-01T23:01:33.171Z+++ zoneadm list -n
8282024-02-01T23:01:33.172Z+++ grep oxz_ntp
8292024-02-01T23:01:33.172Z++ pfexec svcs -z oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7 -Hostate oxide/ntp
8302024-02-01T23:01:33.172Z+ [[ disabled != online ]]
8312024-02-01T23:01:33.172Z+ [[ 7 -gt 60 ]]
8322024-02-01T23:01:33.172Z+ sleep 1
8332024-02-01T23:01:34.168Z+ retry=8
8342024-02-01T23:01:34.198Z+++ zoneadm list -n
8352024-02-01T23:01:34.198Z+++ grep oxz_ntp
8362024-02-01T23:01:34.199Z++ pfexec svcs -z oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7 -Hostate oxide/ntp
8372024-02-01T23:01:34.199Z+ [[ disabled != online ]]
8382024-02-01T23:01:34.199Z+ [[ 8 -gt 60 ]]
8392024-02-01T23:01:34.199Z+ sleep 1
8402024-02-01T23:01:35.188Z+ retry=9
8412024-02-01T23:01:35.228Z+++ zoneadm list -n
8422024-02-01T23:01:35.228Z+++ grep oxz_ntp
8432024-02-01T23:01:35.229Z++ pfexec svcs -z oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7 -Hostate oxide/ntp
8442024-02-01T23:01:35.229Z+ [[ disabled != online ]]
8452024-02-01T23:01:35.229Z+ [[ 9 -gt 60 ]]
8462024-02-01T23:01:35.230Z+ sleep 1
8472024-02-01T23:01:36.209Z+ retry=10
8482024-02-01T23:01:36.257Z+++ zoneadm list -n
8492024-02-01T23:01:36.257Z+++ grep oxz_ntp
8502024-02-01T23:01:36.257Z++ pfexec svcs -z oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7 -Hostate oxide/ntp
8512024-02-01T23:01:36.257Z+ [[ disabled != online ]]
8522024-02-01T23:01:36.257Z+ [[ 10 -gt 60 ]]
8532024-02-01T23:01:36.257Z+ sleep 1
8542024-02-01T23:01:37.248Z+ retry=11
8552024-02-01T23:01:37.275Z+++ zoneadm list -n
8562024-02-01T23:01:37.276Z+++ grep oxz_ntp
8572024-02-01T23:01:37.276Z++ pfexec svcs -z oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7 -Hostate oxide/ntp
8582024-02-01T23:01:37.276Z+ [[ disabled != online ]]
8592024-02-01T23:01:37.276Z+ [[ 11 -gt 60 ]]
8602024-02-01T23:01:37.276Z+ sleep 1
 
8612024-02-01T23:01:37.638ZFeb 1 23:01:37 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8622024-02-01T23:01:38.249Z+ retry=12
8632024-02-01T23:01:38.277Z+++ zoneadm list -n
8642024-02-01T23:01:38.277Z+++ grep oxz_ntp
8652024-02-01T23:01:38.278Z++ pfexec svcs -z oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7 -Hostate oxide/ntp
8662024-02-01T23:01:38.278Z+ [[ online != online ]]
8672024-02-01T23:01:38.278Z+ echo 'Waited for chrony: 12s'
8682024-02-01T23:01:38.278ZWaited for chrony: 12s
8692024-02-01T23:01:38.278Z+ export RUST_BACKTRACE=1
8702024-02-01T23:01:38.278Z+ RUST_BACKTRACE=1
8712024-02-01T23:01:38.278Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8722024-02-01T23:01:38.278Z++ ./tests/bootstrap
8732024-02-01T23:01:38.304ZUsing external DNS server at 10.151.2.164:53
8742024-02-01T23:01:38.304Z2024-02-01 23:01:38.701264177 UTC: attempting to log into API
8752024-02-01T23:01:53.336Z2024-02-01 23:01:53.563378053 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8762024-02-01T23:01:54.344Z2024-02-01 23:01:54.564365838 UTC: attempting to log into API
8772024-02-01T23:02:09.378Z2024-02-01 23:02:09.600408787 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8782024-02-01T23:02:10.386Z2024-02-01 23:02:10.601137207 UTC: attempting to log into API
8792024-02-01T23:02:25.413Z2024-02-01 23:02:25.633352847 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8802024-02-01T23:02:26.475Z2024-02-01 23:02:26.634418554 UTC: attempting to log into API
8812024-02-01T23:02:41.441Z2024-02-01 23:02:41.667587407 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8822024-02-01T23:02:42.516Z2024-02-01 23:02:42.668578233 UTC: attempting to log into API
8832024-02-01T23:02:57.489Z2024-02-01 23:02:57.677806878 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8842024-02-01T23:02:58.476Z2024-02-01 23:02:58.676102944 UTC: attempting to log into API
8852024-02-01T23:03:13.982Z2024-02-01 23:03:13.668458903 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8862024-02-01T23:03:14.513Z2024-02-01 23:03:14.666665317 UTC: attempting to log into API
8872024-02-01T23:03:29.565Z2024-02-01 23:03:29.659414990 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8882024-02-01T23:03:31.003Z2024-02-01 23:03:30.657403398 UTC: attempting to log into API
 
8892024-02-01T23:03:32.650ZError: No response to keepalive - Terminating session
8902024-02-01T23:03:42.823ZError: No response de-activating SOL payload
8912024-02-01T23:03:43.875Z[SOL Session operational. Use ~? for help]
 
8922024-02-01T23:03:45.703Z2024-02-01 23:03:45.649984887 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8932024-02-01T23:03:46.585Z2024-02-01 23:03:46.649025141 UTC: attempting to log into API
8942024-02-01T23:04:01.613Z2024-02-01 23:04:01.642066372 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8952024-02-01T23:04:02.618Z2024-02-01 23:04:02.640032641 UTC: attempting to log into API
8962024-02-01T23:04:20.232Z2024-02-01 23:04:19.632304784 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8972024-02-01T23:04:20.874Z2024-02-01 23:04:20.848285195 UTC: attempting to log into API
8982024-02-01T23:04:36.110Z2024-02-01 23:04:35.998664417 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8992024-02-01T23:04:37.038Z2024-02-01 23:04:37.000232126 UTC: attempting to log into API
9002024-02-01T23:04:52.072Z2024-02-01 23:04:52.046663350 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9012024-02-01T23:04:52.942ZFeb 1 23:04:52 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9022024-02-01T23:04:53.068Z2024-02-01 23:04:53.047440731 UTC: attempting to log into API
 
9032024-02-01T23:04:58.693ZFeb 1 23:04:58 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9042024-02-01T23:05:04.339ZFeb 1 23:05:03 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9052024-02-01T23:05:08.104Z2024-02-01 23:05:08.081142297 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9062024-02-01T23:05:09.101Z2024-02-01 23:05:09.082120692 UTC: attempting to log into API
 
9072024-02-01T23:05:11.622ZFeb 1 23:05:11 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9082024-02-01T23:05:24.132Z2024-02-01 23:05:24.115100564 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9092024-02-01T23:05:25.136Z2024-02-01 23:05:25.116097600 UTC: attempting to log into API
9102024-02-01T23:05:35.935Z2024-02-01 23:05:35.906488657 UTC: login succeeded
9112024-02-01T23:05:35.970Zwaiting for nexus to come up...
9122024-02-01T23:05:36.043Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9132024-02-01T23:05:36.238Zsetting up quotas...
9142024-02-01T23:05:36.285Zensuring datasets are ready...
9152024-02-01T23:05:38.733Zlet's roll.
9162024-02-01T23:05:38.768Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
9172024-02-01T23:05:38.768ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9182024-02-01T23:05:38.768ZOXIDE_TOKEN=oxide-token-b51d4e4fbedcdc906e0e979d59137cb0aae0f5b9'
9192024-02-01T23:05:38.768Z++ OXIDE_HOST=https://recovery.sys.oxide.test
9202024-02-01T23:05:38.768Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9212024-02-01T23:05:38.768Z++ OXIDE_TOKEN=oxide-token-b51d4e4fbedcdc906e0e979d59137cb0aae0f5b9
9222024-02-01T23:05:38.768Z+ export OXIDE_HOST OXIDE_TOKEN
9232024-02-01T23:05:38.768Z+ retry=0
9242024-02-01T23:05:38.768Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9252024-02-01T23:05:38.768Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9262024-02-01T23:05:38.768Z+ [[ 0 -gt 60 ]]
9272024-02-01T23:05:38.768Z+ sleep 1
9282024-02-01T23:05:39.777Z+ retry=1
9292024-02-01T23:05:39.777Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9302024-02-01T23:05:39.815Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9312024-02-01T23:05:39.815Z+ [[ 1 -gt 60 ]]
9322024-02-01T23:05:39.815Z+ sleep 1
9332024-02-01T23:05:40.797Z+ retry=2
9342024-02-01T23:05:40.797Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9352024-02-01T23:05:40.837Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9362024-02-01T23:05:40.837Z+ [[ 2 -gt 60 ]]
9372024-02-01T23:05:40.837Z+ sleep 1
9382024-02-01T23:05:41.823Z+ retry=3
9392024-02-01T23:05:41.824Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9402024-02-01T23:05:41.863Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9412024-02-01T23:05:41.863Z+ [[ 3 -gt 60 ]]
9422024-02-01T23:05:41.863Z+ sleep 1
9432024-02-01T23:05:42.820Z+ retry=4
9442024-02-01T23:05:42.820Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9452024-02-01T23:05:42.862Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9462024-02-01T23:05:42.862Z+ [[ 4 -gt 60 ]]
9472024-02-01T23:05:42.862Z+ sleep 1
9482024-02-01T23:05:43.834Z+ retry=5
9492024-02-01T23:05:43.834Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9502024-02-01T23:05:43.867Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9512024-02-01T23:05:43.867Z+ [[ 5 -gt 60 ]]
9522024-02-01T23:05:43.867Z+ sleep 1
9532024-02-01T23:05:44.852Z+ retry=6
9542024-02-01T23:05:44.853Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9552024-02-01T23:05:44.964Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9562024-02-01T23:05:44.964Z+ [[ 6 -gt 60 ]]
9572024-02-01T23:05:44.964Z+ sleep 1
9582024-02-01T23:05:45.862Z+ retry=7
9592024-02-01T23:05:45.862Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9602024-02-01T23:05:45.894Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9612024-02-01T23:05:45.894Z+ [[ 7 -gt 60 ]]
9622024-02-01T23:05:45.894Z+ sleep 1
9632024-02-01T23:05:46.879Z+ retry=8
9642024-02-01T23:05:46.880Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9652024-02-01T23:05:46.922Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9662024-02-01T23:05:46.922Z+ [[ 8 -gt 60 ]]
9672024-02-01T23:05:46.922Z+ sleep 1
9682024-02-01T23:05:47.924Z+ retry=9
9692024-02-01T23:05:47.924Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9702024-02-01T23:05:47.964Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9712024-02-01T23:05:47.964Z+ [[ 9 -gt 60 ]]
9722024-02-01T23:05:47.964Z+ sleep 1
9732024-02-01T23:05:48.910Z+ retry=10
9742024-02-01T23:05:48.910Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9752024-02-01T23:05:48.950Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9762024-02-01T23:05:48.950Z+ [[ 10 -gt 60 ]]
9772024-02-01T23:05:48.950Z+ sleep 1
9782024-02-01T23:05:49.927Z+ retry=11
9792024-02-01T23:05:49.927Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9802024-02-01T23:05:49.969Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9812024-02-01T23:05:49.969Z+ [[ 11 -gt 60 ]]
9822024-02-01T23:05:49.969Z+ sleep 1
9832024-02-01T23:05:50.950Z+ retry=12
9842024-02-01T23:05:50.951Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9852024-02-01T23:05:50.993Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9862024-02-01T23:05:50.993Z+ [[ 12 -gt 60 ]]
9872024-02-01T23:05:50.994Z+ sleep 1
9882024-02-01T23:05:51.954Z+ retry=13
9892024-02-01T23:05:51.955Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9902024-02-01T23:05:51.986Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9912024-02-01T23:05:51.986Z+ [[ 13 -gt 60 ]]
9922024-02-01T23:05:51.986Z+ sleep 1
9932024-02-01T23:05:52.970Z+ retry=14
9942024-02-01T23:05:52.970Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9952024-02-01T23:05:53.002Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9962024-02-01T23:05:53.002Z+ [[ 14 -gt 60 ]]
9972024-02-01T23:05:53.002Z+ sleep 1
9982024-02-01T23:05:53.990Z+ retry=15
9992024-02-01T23:05:53.990Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10002024-02-01T23:05:54.024Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10012024-02-01T23:05:54.024Z+ [[ 15 -gt 60 ]]
10022024-02-01T23:05:54.025Z+ sleep 1
10032024-02-01T23:05:55.005Z+ retry=16
10042024-02-01T23:05:55.005Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10052024-02-01T23:05:55.059Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10062024-02-01T23:05:55.059Z+ [[ 16 -gt 60 ]]
10072024-02-01T23:05:55.059Z+ sleep 1
10082024-02-01T23:05:56.065Z+ retry=17
10092024-02-01T23:05:56.065Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10102024-02-01T23:05:56.066Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10112024-02-01T23:05:56.066Z+ [[ 17 -gt 60 ]]
10122024-02-01T23:05:56.066Z+ sleep 1
10132024-02-01T23:05:57.041Z+ retry=18
10142024-02-01T23:05:57.041Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10152024-02-01T23:05:57.486Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10162024-02-01T23:05:57.487Z+ [[ 18 -gt 60 ]]
10172024-02-01T23:05:57.487Z+ sleep 1
10182024-02-01T23:05:58.053Z+ retry=19
10192024-02-01T23:05:58.053Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10202024-02-01T23:05:58.097Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10212024-02-01T23:05:58.097Z+ [[ 19 -gt 60 ]]
10222024-02-01T23:05:58.098Z+ sleep 1
10232024-02-01T23:05:59.066Z+ retry=20
10242024-02-01T23:05:59.066Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10252024-02-01T23:05:59.101Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10262024-02-01T23:05:59.102Z+ [[ 20 -gt 60 ]]
10272024-02-01T23:05:59.102Z+ sleep 1
10282024-02-01T23:06:00.078Z+ retry=21
10292024-02-01T23:06:00.079Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10302024-02-01T23:06:00.122Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10312024-02-01T23:06:00.122Z+ [[ 21 -gt 60 ]]
10322024-02-01T23:06:00.122Z+ sleep 1
10332024-02-01T23:06:01.098Z+ retry=22
10342024-02-01T23:06:01.098Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10352024-02-01T23:06:01.138Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10362024-02-01T23:06:01.138Z+ [[ 22 -gt 60 ]]
10372024-02-01T23:06:01.138Z+ sleep 1
10382024-02-01T23:06:02.119Z+ retry=23
10392024-02-01T23:06:02.119Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10402024-02-01T23:06:02.164Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10412024-02-01T23:06:02.165Z+ [[ 23 -gt 60 ]]
10422024-02-01T23:06:02.165Z+ sleep 1
10432024-02-01T23:06:03.127Z+ retry=24
10442024-02-01T23:06:03.127Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10452024-02-01T23:06:03.162Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10462024-02-01T23:06:03.162Z+ [[ 24 -gt 60 ]]
10472024-02-01T23:06:03.162Z+ sleep 1
10482024-02-01T23:06:04.143Z+ retry=25
10492024-02-01T23:06:04.143Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10502024-02-01T23:06:04.179Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10512024-02-01T23:06:04.180Z+ [[ 25 -gt 60 ]]
10522024-02-01T23:06:04.180Z+ sleep 1
10532024-02-01T23:06:05.156Z+ retry=26
10542024-02-01T23:06:05.156Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10552024-02-01T23:06:05.676Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10562024-02-01T23:06:05.676Z+ [[ 26 -gt 60 ]]
10572024-02-01T23:06:05.676Z+ sleep 1
10582024-02-01T23:06:06.211Z+ retry=27
10592024-02-01T23:06:06.211Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10602024-02-01T23:06:06.249Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10612024-02-01T23:06:06.250Z+ [[ 27 -gt 60 ]]
10622024-02-01T23:06:06.250Z+ sleep 1
10632024-02-01T23:06:07.190Z+ retry=28
10642024-02-01T23:06:07.190Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10652024-02-01T23:06:07.249Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10662024-02-01T23:06:07.249Z+ [[ 28 -gt 60 ]]
10672024-02-01T23:06:07.249Z+ sleep 1
10682024-02-01T23:06:08.216Z+ retry=29
10692024-02-01T23:06:08.217Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10702024-02-01T23:06:08.250Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10712024-02-01T23:06:08.250Z+ [[ 29 -gt 60 ]]
10722024-02-01T23:06:08.250Z+ sleep 1
10732024-02-01T23:06:09.217Z+ retry=30
10742024-02-01T23:06:09.218Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10752024-02-01T23:06:09.251Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10762024-02-01T23:06:09.251Z+ [[ 30 -gt 60 ]]
10772024-02-01T23:06:09.251Z+ sleep 1
10782024-02-01T23:06:10.237Z+ retry=31
10792024-02-01T23:06:10.238Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10802024-02-01T23:06:10.435Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10812024-02-01T23:06:10.436Z+ [[ 31 -gt 60 ]]
10822024-02-01T23:06:10.436Z+ sleep 1
10832024-02-01T23:06:11.253Z+ retry=32
10842024-02-01T23:06:11.253Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10852024-02-01T23:06:11.291Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10862024-02-01T23:06:11.291Z+ [[ 32 -gt 60 ]]
10872024-02-01T23:06:11.291Z+ sleep 1
10882024-02-01T23:06:12.271Z+ retry=33
10892024-02-01T23:06:12.271Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10902024-02-01T23:06:12.314Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10912024-02-01T23:06:12.314Z+ [[ 33 -gt 60 ]]
10922024-02-01T23:06:12.314Z+ sleep 1
10932024-02-01T23:06:13.281Z+ retry=34
10942024-02-01T23:06:13.281Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10952024-02-01T23:06:13.317Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10962024-02-01T23:06:13.317Z+ [[ 34 -gt 60 ]]
10972024-02-01T23:06:13.317Z+ sleep 1
10982024-02-01T23:06:14.295Z+ retry=35
10992024-02-01T23:06:14.296Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11002024-02-01T23:06:14.328Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11012024-02-01T23:06:14.328Z+ [[ 35 -gt 60 ]]
11022024-02-01T23:06:14.328Z+ sleep 1
11032024-02-01T23:06:15.318Z+ retry=36
11042024-02-01T23:06:15.318Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11052024-02-01T23:06:15.354Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11062024-02-01T23:06:15.355Z+ [[ 36 -gt 60 ]]
11072024-02-01T23:06:15.355Z+ sleep 1
11082024-02-01T23:06:16.329Z+ retry=37
11092024-02-01T23:06:16.329Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11102024-02-01T23:06:16.375Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11112024-02-01T23:06:16.376Z+ [[ 37 -gt 60 ]]
11122024-02-01T23:06:16.376Z+ sleep 1
11132024-02-01T23:06:17.345Z+ retry=38
11142024-02-01T23:06:17.346Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11152024-02-01T23:06:17.386Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11162024-02-01T23:06:17.386Z+ [[ 38 -gt 60 ]]
11172024-02-01T23:06:17.386Z+ sleep 1
11182024-02-01T23:06:18.358Z+ retry=39
11192024-02-01T23:06:18.359Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11202024-02-01T23:06:18.404Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11212024-02-01T23:06:18.404Z+ [[ 39 -gt 60 ]]
11222024-02-01T23:06:18.405Z+ sleep 1
11232024-02-01T23:06:19.377Z+ retry=40
11242024-02-01T23:06:19.377Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11252024-02-01T23:06:19.454Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
11262024-02-01T23:06:19.791Z{"status":"ok"}success
11272024-02-01T23:06:19.791ZProject {
11282024-02-01T23:06:19.791Z description: "some images",
11292024-02-01T23:06:19.791Z id: 527ebbee-41dd-4abe-a1b8-6a0ec0c23982,
11302024-02-01T23:06:19.791Z name: Name(
11312024-02-01T23:06:19.791Z "images",
11322024-02-01T23:06:19.791Z ),
11332024-02-01T23:06:19.826Z time_created: 2024-02-01T23:06:19.526558Z,
11342024-02-01T23:06:19.826Z time_modified: 2024-02-01T23:06:19.526558Z,
11352024-02-01T23:06:19.826Z}
11362024-02-01T23:06:19.827Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
 
11372024-02-01T23:06:32.612ZError: No response to keepalive - Terminating session
11382024-02-01T23:06:42.824ZError: No response de-activating SOL payload
11392024-02-01T23:06:44.361Z[SOL Session operational. Use ~? for help]
11402024-02-01T23:09:32.847ZError: No response to keepalive - Terminating session
11412024-02-01T23:09:42.759ZError: No response de-activating SOL payload
11422024-02-01T23:09:44.140Z[SOL Session operational. Use ~? for help]
 
11432024-02-01T23:11:47.533Zdone!
11442024-02-01T23:11:47.638Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
11452024-02-01T23:11:47.753Zsuccess
11462024-02-01T23:11:47.753ZImage {
11472024-02-01T23:11:47.753Z block_size: ByteCount(
11482024-02-01T23:11:47.753Z 512,
11492024-02-01T23:11:47.753Z ),
11502024-02-01T23:11:47.753Z description: "debian 11 original base image",
11512024-02-01T23:11:47.809Z digest: None,
11522024-02-01T23:11:47.809Z id: 84ea5bfc-2897-4ada-99e7-ab1449d49663,
11532024-02-01T23:11:47.809Z name: Name(
11542024-02-01T23:11:47.809Z "debian11",
11552024-02-01T23:11:47.809Z ),
11562024-02-01T23:11:47.810Z os: "debian",
11572024-02-01T23:11:47.810Z project_id: None,
11582024-02-01T23:11:47.810Z size: ByteCount(
11592024-02-01T23:11:47.810Z 2147483648,
11602024-02-01T23:11:47.810Z ),
11612024-02-01T23:11:47.810Z time_created: 2024-02-01T23:11:47.492509Z,
11622024-02-01T23:11:47.810Z time_modified: 2024-02-01T23:11:47.722616Z,
11632024-02-01T23:11:47.810Z url: None,
11642024-02-01T23:11:47.810Z version: "11",
11652024-02-01T23:11:47.810Z}
11662024-02-01T23:11:47.810Z+ rm ./tests/bootstrap
11672024-02-01T23:11:47.846Z+ for test_bin in tests/*
11682024-02-01T23:11:47.846Z+ ./tests/bootstrap-d85755d230534e96
11692024-02-01T23:11:47.880Z
11702024-02-01T23:11:47.880Zrunning 0 tests
11712024-02-01T23:11:47.880Z
11722024-02-01T23:11:47.880Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
11732024-02-01T23:11:47.880Z
11742024-02-01T23:11:47.880Z+ for test_bin in tests/*
11752024-02-01T23:11:47.880Z+ ./tests/end_to_end_tests-2bc5efa67eecf70d
11762024-02-01T23:11:48.048Z
11772024-02-01T23:11:48.048Zrunning 2 tests
11782024-02-01T23:11:48.335Ztest no_spoof::no_spoof_here ... ok
11792024-02-01T23:12:20.813Ztest instance_launch::instance_launch ... ok
11802024-02-01T23:12:20.814Z
11812024-02-01T23:12:20.814Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 32.76s
11822024-02-01T23:12:20.814Z
11832024-02-01T23:12:20.847Z+ _exit_trap
11842024-02-01T23:12:20.847Z+ local status=0
11852024-02-01T23:12:20.848Z+ [[ 0 -eq 0 ]]
11862024-02-01T23:12:20.848Z+ exit 0
11872024-02-01T23:12:20.848Zprocess exited: duration 814071 ms, exit code 0
 
11882024-02-01T23:12:20.993Zfound 72 output files
11892024-02-01T23:12:20.993Zuploading: /var/svc/log/oxide-sled-agent:default.log (1782624 bytes)
11902024-02-01T23:12:24.314Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 1782624 -> 1783035
11912024-02-01T23:12:25.419Zuploaded: /var/svc/log/oxide-sled-agent:default.log
11922024-02-01T23:12:25.419Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log (2830542 bytes)
11932024-02-01T23:12:29.262Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log" changed size mid upload: 2830542 -> 2863300
11942024-02-01T23:12:30.378Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log
11952024-02-01T23:12:30.378Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log (105252 bytes)
11962024-02-01T23:12:30.737Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log" changed size mid upload: 105252 -> 106502
11972024-02-01T23:12:31.844Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log
11982024-02-01T23:12:31.844Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log (123287 bytes)
11992024-02-01T23:12:32.261Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log" changed size mid upload: 123287 -> 124925
12002024-02-01T23:12:33.773Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log
12012024-02-01T23:12:33.773Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log (1416749 bytes)
12022024-02-01T23:12:36.305Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log" changed size mid upload: 1416749 -> 1450302
12032024-02-01T23:12:37.411Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log
12042024-02-01T23:12:37.412Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log (869695 bytes)
12052024-02-01T23:12:38.409Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log" changed size mid upload: 869695 -> 895309
12062024-02-01T23:12:39.508Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log
12072024-02-01T23:12:39.508Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log (1388 bytes)
12082024-02-01T23:12:40.703Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log
12092024-02-01T23:12:40.703Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log (326407 bytes)
12102024-02-01T23:12:41.290Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log" changed size mid upload: 326407 -> 337512
12112024-02-01T23:12:42.384Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log
12122024-02-01T23:12:42.385Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log (617 bytes)
12132024-02-01T23:12:43.518Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log
12142024-02-01T23:12:43.551Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log (22679 bytes)
12152024-02-01T23:12:43.658Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log" changed size mid upload: 22679 -> 23465
12162024-02-01T23:12:44.753Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log
12172024-02-01T23:12:44.753Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_d7884317-3d5e-4e8a-9f22-032f22a0a1d4/root/var/svc/log/oxide-clickhouse:default.log (1239745 bytes)
12182024-02-01T23:12:46.130Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_d7884317-3d5e-4e8a-9f22-032f22a0a1d4/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 1239745 -> 1327476
12192024-02-01T23:12:47.253Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_d7884317-3d5e-4e8a-9f22-032f22a0a1d4/root/var/svc/log/oxide-clickhouse:default.log
12202024-02-01T23:12:47.253Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_d7884317-3d5e-4e8a-9f22-032f22a0a1d4/root/var/svc/log/oxide-zone-network-setup:default.log (1708 bytes)
12212024-02-01T23:12:48.393Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_d7884317-3d5e-4e8a-9f22-032f22a0a1d4/root/var/svc/log/oxide-zone-network-setup:default.log
12222024-02-01T23:12:48.393Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_ef036f5d-8b3b-4644-943a-03fb21676282/root/var/svc/log/oxide-cockroachdb:default.log (4406 bytes)
12232024-02-01T23:12:48.770ZError: No response to keepalive - Terminating session
12242024-02-01T23:12:49.536Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_ef036f5d-8b3b-4644-943a-03fb21676282/root/var/svc/log/oxide-cockroachdb:default.log
12252024-02-01T23:12:49.537Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_ef036f5d-8b3b-4644-943a-03fb21676282/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
12262024-02-01T23:12:50.686Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_ef036f5d-8b3b-4644-943a-03fb21676282/root/var/svc/log/oxide-zone-network-setup:default.log
12272024-02-01T23:12:50.719Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df09189e-e0ab-49c4-9a16-1a507165b1f5/root/var/svc/log/oxide-crucible-agent:default.log (25247 bytes)
12282024-02-01T23:12:51.935Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df09189e-e0ab-49c4-9a16-1a507165b1f5/root/var/svc/log/oxide-crucible-agent:default.log
12292024-02-01T23:12:51.935Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df09189e-e0ab-49c4-9a16-1a507165b1f5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12302024-02-01T23:12:53.071Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df09189e-e0ab-49c4-9a16-1a507165b1f5/root/var/svc/log/oxide-crucible-downstairs:default.log
12312024-02-01T23:12:53.071Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df09189e-e0ab-49c4-9a16-1a507165b1f5/root/var/svc/log/oxide-crucible-downstairs:downstairs-ac20cb35-1e4a-4bca-a17b-ffcd92e6b193.log (2944 bytes)
12322024-02-01T23:12:54.204Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df09189e-e0ab-49c4-9a16-1a507165b1f5/root/var/svc/log/oxide-crucible-downstairs:downstairs-ac20cb35-1e4a-4bca-a17b-ffcd92e6b193.log
12332024-02-01T23:12:54.267Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df09189e-e0ab-49c4-9a16-1a507165b1f5/root/var/svc/log/oxide-crucible-downstairs:downstairs-f95b8766-f192-49cd-87b0-65601044e1c8.log (5536 bytes)
12342024-02-01T23:12:55.427Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df09189e-e0ab-49c4-9a16-1a507165b1f5/root/var/svc/log/oxide-crucible-downstairs:downstairs-f95b8766-f192-49cd-87b0-65601044e1c8.log
12352024-02-01T23:12:55.471Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_51987a43-d1b3-4ff7-b9ed-346a37844968/root/var/svc/log/oxide-external_dns:default.log (30717 bytes)
12362024-02-01T23:12:56.789Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_51987a43-d1b3-4ff7-b9ed-346a37844968/root/var/svc/log/oxide-external_dns:default.log
12372024-02-01T23:12:56.789Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_df60bed6-1c04-45f3-94dc-120f3ef44efe/root/var/svc/log/oxide-internal_dns:default.log (114852 bytes)
12382024-02-01T23:12:58.371Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_df60bed6-1c04-45f3-94dc-120f3ef44efe/root/var/svc/log/oxide-internal_dns:default.log
12392024-02-01T23:12:58.372Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3f5380b4-3ef3-49b8-919d-5a11b7ff8cf8/root/var/svc/log/oxide-oximeter:default.log (338076 bytes)
12402024-02-01T23:12:58.727ZError: No response de-activating SOL payload
12412024-02-01T23:12:59.298Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3f5380b4-3ef3-49b8-919d-5a11b7ff8cf8/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 338076 -> 367030
12422024-02-01T23:13:00.100Z[SOL Session operational. Use ~? for help]
12432024-02-01T23:13:00.398Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3f5380b4-3ef3-49b8-919d-5a11b7ff8cf8/root/var/svc/log/oxide-oximeter:default.log
12442024-02-01T23:13:00.398Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3f5380b4-3ef3-49b8-919d-5a11b7ff8cf8/root/var/svc/log/oxide-zone-network-setup:default.log (1697 bytes)
12452024-02-01T23:13:01.530Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_3f5380b4-3ef3-49b8-919d-5a11b7ff8cf8/root/var/svc/log/oxide-zone-network-setup:default.log
12462024-02-01T23:13:01.530Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a3765b91-f3cd-4e04-8d11-4be4cb142850/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
12472024-02-01T23:13:02.737Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a3765b91-f3cd-4e04-8d11-4be4cb142850/root/var/svc/log/oxide-cockroachdb:default.log
12482024-02-01T23:13:02.737Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a3765b91-f3cd-4e04-8d11-4be4cb142850/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
12492024-02-01T23:13:03.910Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a3765b91-f3cd-4e04-8d11-4be4cb142850/root/var/svc/log/oxide-zone-network-setup:default.log
12502024-02-01T23:13:03.911Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bb658c5-cd67-45df-be53-726b1e038112/root/var/svc/log/oxide-crucible-agent:default.log (33719 bytes)
12512024-02-01T23:13:05.472Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bb658c5-cd67-45df-be53-726b1e038112/root/var/svc/log/oxide-crucible-agent:default.log
12522024-02-01T23:13:05.472Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bb658c5-cd67-45df-be53-726b1e038112/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12532024-02-01T23:13:06.611Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bb658c5-cd67-45df-be53-726b1e038112/root/var/svc/log/oxide-crucible-downstairs:default.log
12542024-02-01T23:13:06.612Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bb658c5-cd67-45df-be53-726b1e038112/root/var/svc/log/oxide-crucible-downstairs:downstairs-28bc1d52-ccb4-4fcb-8755-c6b481e3ab11.log (2942 bytes)
12552024-02-01T23:13:07.740Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bb658c5-cd67-45df-be53-726b1e038112/root/var/svc/log/oxide-crucible-downstairs:downstairs-28bc1d52-ccb4-4fcb-8755-c6b481e3ab11.log
12562024-02-01T23:13:07.740Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bb658c5-cd67-45df-be53-726b1e038112/root/var/svc/log/oxide-crucible-downstairs:downstairs-68270274-3c15-4bad-a0b3-d14e647daea3.log (5620 bytes)
12572024-02-01T23:13:08.903Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bb658c5-cd67-45df-be53-726b1e038112/root/var/svc/log/oxide-crucible-downstairs:downstairs-68270274-3c15-4bad-a0b3-d14e647daea3.log
12582024-02-01T23:13:08.904Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bb658c5-cd67-45df-be53-726b1e038112/root/var/svc/log/oxide-crucible-downstairs:downstairs-bed36ab7-b152-4c54-b074-edb681429b6a.log (3050 bytes)
12592024-02-01T23:13:10.036Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bb658c5-cd67-45df-be53-726b1e038112/root/var/svc/log/oxide-crucible-downstairs:downstairs-bed36ab7-b152-4c54-b074-edb681429b6a.log
12602024-02-01T23:13:10.068Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bb658c5-cd67-45df-be53-726b1e038112/root/var/svc/log/oxide-crucible-downstairs:snapshot-68270274-3c15-4bad-a0b3-d14e647daea3-02bf3710-1c87-41ec-aee5-3f0193287fee.log (5180 bytes)
12612024-02-01T23:13:11.163Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6bb658c5-cd67-45df-be53-726b1e038112/root/var/svc/log/oxide-crucible-downstairs:snapshot-68270274-3c15-4bad-a0b3-d14e647daea3-02bf3710-1c87-41ec-aee5-3f0193287fee.log
12622024-02-01T23:13:11.163Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_4cb72df6-970b-4a4c-b034-d8331ee69620/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
12632024-02-01T23:13:12.318Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_4cb72df6-970b-4a4c-b034-d8331ee69620/root/var/svc/log/oxide-crucible-pantry:default.log
12642024-02-01T23:13:12.318Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4899ecb1-2238-4aea-ab22-0cb405d35de3/root/var/svc/log/oxide-external_dns:default.log (30725 bytes)
12652024-02-01T23:13:13.651Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4899ecb1-2238-4aea-ab22-0cb405d35de3/root/var/svc/log/oxide-external_dns:default.log
12662024-02-01T23:13:13.651Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_15fc6fc9-f319-40b0-a8e3-83e7739520a6/root/var/svc/log/oxide-internal_dns:default.log (118105 bytes)
12672024-02-01T23:13:15.202Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_15fc6fc9-f319-40b0-a8e3-83e7739520a6/root/var/svc/log/oxide-internal_dns:default.log
12682024-02-01T23:13:15.244Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_217fe4ec-e832-469e-8920-f05d70ca9d28/root/var/svc/log/oxide-cockroachdb:default.log (4380 bytes)
12692024-02-01T23:13:16.331Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_217fe4ec-e832-469e-8920-f05d70ca9d28/root/var/svc/log/oxide-cockroachdb:default.log
12702024-02-01T23:13:16.331Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_217fe4ec-e832-469e-8920-f05d70ca9d28/root/var/svc/log/oxide-zone-network-setup:default.log (1705 bytes)
12712024-02-01T23:13:17.464Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_217fe4ec-e832-469e-8920-f05d70ca9d28/root/var/svc/log/oxide-zone-network-setup:default.log
12722024-02-01T23:13:17.464Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1d32b1ea-a762-4d23-8f5a-e88f5df6758d/root/var/svc/log/oxide-crucible-agent:default.log (27797 bytes)
12732024-02-01T23:13:18.794Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1d32b1ea-a762-4d23-8f5a-e88f5df6758d/root/var/svc/log/oxide-crucible-agent:default.log
12742024-02-01T23:13:18.794Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1d32b1ea-a762-4d23-8f5a-e88f5df6758d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12752024-02-01T23:13:19.926Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1d32b1ea-a762-4d23-8f5a-e88f5df6758d/root/var/svc/log/oxide-crucible-downstairs:default.log
12762024-02-01T23:13:19.960Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1d32b1ea-a762-4d23-8f5a-e88f5df6758d/root/var/svc/log/oxide-crucible-downstairs:downstairs-0277934e-a002-4846-8492-d9b0e46438d4.log (5621 bytes)
12772024-02-01T23:13:21.093Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1d32b1ea-a762-4d23-8f5a-e88f5df6758d/root/var/svc/log/oxide-crucible-downstairs:downstairs-0277934e-a002-4846-8492-d9b0e46438d4.log
12782024-02-01T23:13:21.093Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1d32b1ea-a762-4d23-8f5a-e88f5df6758d/root/var/svc/log/oxide-crucible-downstairs:downstairs-cd8bfa46-8372-403d-992a-0383d7b4ddc1.log (5535 bytes)
12792024-02-01T23:13:22.259Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1d32b1ea-a762-4d23-8f5a-e88f5df6758d/root/var/svc/log/oxide-crucible-downstairs:downstairs-cd8bfa46-8372-403d-992a-0383d7b4ddc1.log
12802024-02-01T23:13:22.259Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1d32b1ea-a762-4d23-8f5a-e88f5df6758d/root/var/svc/log/oxide-crucible-downstairs:snapshot-0277934e-a002-4846-8492-d9b0e46438d4-02bf3710-1c87-41ec-aee5-3f0193287fee.log (5175 bytes)
12812024-02-01T23:13:23.384Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_1d32b1ea-a762-4d23-8f5a-e88f5df6758d/root/var/svc/log/oxide-crucible-downstairs:snapshot-0277934e-a002-4846-8492-d9b0e46438d4-02bf3710-1c87-41ec-aee5-3f0193287fee.log
12822024-02-01T23:13:23.385Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_fe280561-71ce-4c58-bbb7-ef63f86462c6/root/var/svc/log/oxide-internal_dns:default.log (112909 bytes)
12832024-02-01T23:13:23.844Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_fe280561-71ce-4c58-bbb7-ef63f86462c6/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 112909 -> 116665
12842024-02-01T23:13:24.942Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_fe280561-71ce-4c58-bbb7-ef63f86462c6/root/var/svc/log/oxide-internal_dns:default.log
12852024-02-01T23:13:24.942Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_20f012c7-967f-436d-bdcb-075712c0f246/root/var/svc/log/oxide-nexus:default.log (2831586 bytes)
12862024-02-01T23:13:28.548Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_20f012c7-967f-436d-bdcb-075712c0f246/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 2831586 -> 2953071
12872024-02-01T23:13:29.653Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_20f012c7-967f-436d-bdcb-075712c0f246/root/var/svc/log/oxide-nexus:default.log
12882024-02-01T23:13:29.653Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7/root/var/svc/log/oxide-ntp:default.log (1324 bytes)
12892024-02-01T23:13:30.790Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7/root/var/svc/log/oxide-ntp:default.log
12902024-02-01T23:13:30.826Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7deb005c-e7b7-44fa-aca7-aa86bc6441cf/root/var/svc/log/oxide-cockroachdb:default.log (4406 bytes)
12912024-02-01T23:13:32.328Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7deb005c-e7b7-44fa-aca7-aa86bc6441cf/root/var/svc/log/oxide-cockroachdb:default.log
12922024-02-01T23:13:32.328Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7deb005c-e7b7-44fa-aca7-aa86bc6441cf/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
12932024-02-01T23:13:33.467Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7deb005c-e7b7-44fa-aca7-aa86bc6441cf/root/var/svc/log/oxide-zone-network-setup:default.log
12942024-02-01T23:13:33.467Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8d27bd79-80eb-4e7c-8425-27fd831064de/root/var/svc/log/oxide-crucible-agent:default.log (14117 bytes)
12952024-02-01T23:13:34.662Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8d27bd79-80eb-4e7c-8425-27fd831064de/root/var/svc/log/oxide-crucible-agent:default.log
12962024-02-01T23:13:34.662Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8d27bd79-80eb-4e7c-8425-27fd831064de/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12972024-02-01T23:13:35.813Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8d27bd79-80eb-4e7c-8425-27fd831064de/root/var/svc/log/oxide-crucible-downstairs:default.log
12982024-02-01T23:13:35.814Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8d27bd79-80eb-4e7c-8425-27fd831064de/root/var/svc/log/oxide-crucible-downstairs:downstairs-ef5e2fa8-0d7a-43d4-8828-8263e9a75ba9.log (3050 bytes)
12992024-02-01T23:13:36.954Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8d27bd79-80eb-4e7c-8425-27fd831064de/root/var/svc/log/oxide-crucible-downstairs:downstairs-ef5e2fa8-0d7a-43d4-8828-8263e9a75ba9.log
13002024-02-01T23:13:36.954Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_fd197456-37a2-4fbd-ba2c-f498b850a012/root/var/svc/log/oxide-cockroachdb:default.log (4405 bytes)
13012024-02-01T23:13:38.108Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_fd197456-37a2-4fbd-ba2c-f498b850a012/root/var/svc/log/oxide-cockroachdb:default.log
13022024-02-01T23:13:38.108Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_fd197456-37a2-4fbd-ba2c-f498b850a012/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
13032024-02-01T23:13:39.238Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_fd197456-37a2-4fbd-ba2c-f498b850a012/root/var/svc/log/oxide-zone-network-setup:default.log
13042024-02-01T23:13:39.238Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_54dc9f93-e68d-44c5-b046-daa6b5b79f03/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
13052024-02-01T23:13:40.383Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_54dc9f93-e68d-44c5-b046-daa6b5b79f03/root/var/svc/log/oxide-crucible-agent:default.log
13062024-02-01T23:13:40.383Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_54dc9f93-e68d-44c5-b046-daa6b5b79f03/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
13072024-02-01T23:13:41.533Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_54dc9f93-e68d-44c5-b046-daa6b5b79f03/root/var/svc/log/oxide-crucible-downstairs:default.log
13082024-02-01T23:13:41.533Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_495f8858-1cdc-4408-b66b-01d84de61eb9/root/var/svc/log/oxide-nexus:default.log (34531704 bytes)
13092024-02-01T23:13:58.289Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_495f8858-1cdc-4408-b66b-01d84de61eb9/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 34531704 -> 34673001
13102024-02-01T23:13:59.796Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_495f8858-1cdc-4408-b66b-01d84de61eb9/root/var/svc/log/oxide-nexus:default.log
13112024-02-01T23:13:59.796Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_8620539b-445d-4347-8624-07c8901c62f7/root/var/svc/log/oxide-nexus:default.log (1281961 bytes)
13122024-02-01T23:14:00.950Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_8620539b-445d-4347-8624-07c8901c62f7/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1281961 -> 1427178
13132024-02-01T23:14:02.478Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_8620539b-445d-4347-8624-07c8901c62f7/root/var/svc/log/oxide-nexus:default.log
13142024-02-01T23:14:02.479Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_79d7668a-d5d1-4d8f-81d4-c9d0aacc9020/root/var/svc/log/oxide-crucible-agent:default.log (14112 bytes)
13152024-02-01T23:14:03.674Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_79d7668a-d5d1-4d8f-81d4-c9d0aacc9020/root/var/svc/log/oxide-crucible-agent:default.log
13162024-02-01T23:14:03.674Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_79d7668a-d5d1-4d8f-81d4-c9d0aacc9020/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
13172024-02-01T23:14:04.811Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_79d7668a-d5d1-4d8f-81d4-c9d0aacc9020/root/var/svc/log/oxide-crucible-downstairs:default.log
13182024-02-01T23:14:04.811Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_79d7668a-d5d1-4d8f-81d4-c9d0aacc9020/root/var/svc/log/oxide-crucible-downstairs:downstairs-80acaf23-d50f-4d8a-a36b-28e0fe3562e4.log (3050 bytes)
13192024-02-01T23:14:05.948Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_79d7668a-d5d1-4d8f-81d4-c9d0aacc9020/root/var/svc/log/oxide-crucible-downstairs:downstairs-80acaf23-d50f-4d8a-a36b-28e0fe3562e4.log
13202024-02-01T23:14:05.949Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_c7d992b4-eab3-4b94-9455-42b7f29d0b0b/root/var/svc/log/oxide-crucible-agent:default.log (14112 bytes)
13212024-02-01T23:14:07.167Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_c7d992b4-eab3-4b94-9455-42b7f29d0b0b/root/var/svc/log/oxide-crucible-agent:default.log
13222024-02-01T23:14:07.168Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_c7d992b4-eab3-4b94-9455-42b7f29d0b0b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
13232024-02-01T23:14:08.296Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_c7d992b4-eab3-4b94-9455-42b7f29d0b0b/root/var/svc/log/oxide-crucible-downstairs:default.log
13242024-02-01T23:14:08.296Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_c7d992b4-eab3-4b94-9455-42b7f29d0b0b/root/var/svc/log/oxide-crucible-downstairs:downstairs-f2ec404c-e94c-4fe7-a228-f36a3f2139c1.log (5538 bytes)
13252024-02-01T23:14:09.453Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_c7d992b4-eab3-4b94-9455-42b7f29d0b0b/root/var/svc/log/oxide-crucible-downstairs:downstairs-f2ec404c-e94c-4fe7-a228-f36a3f2139c1.log
13262024-02-01T23:14:09.487Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_c4e80f8a-a06d-4d36-bd39-522aeefba954/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
13272024-02-01T23:14:10.581Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_c4e80f8a-a06d-4d36-bd39-522aeefba954/root/var/svc/log/oxide-crucible-pantry:default.log
13282024-02-01T23:14:10.632Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b2ae224c-ab7f-491f-ae44-9207af202690/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
13292024-02-01T23:14:11.727Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b2ae224c-ab7f-491f-ae44-9207af202690/root/var/svc/log/oxide-crucible-agent:default.log
13302024-02-01T23:14:11.728Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b2ae224c-ab7f-491f-ae44-9207af202690/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
13312024-02-01T23:14:12.887Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b2ae224c-ab7f-491f-ae44-9207af202690/root/var/svc/log/oxide-crucible-downstairs:default.log
13322024-02-01T23:14:12.887Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_1b7d6188-23e9-4164-a862-410ec842a095/root/var/svc/log/oxide-crucible-pantry:default.log (1218869 bytes)
13332024-02-01T23:14:22.887Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_1b7d6188-23e9-4164-a862-410ec842a095/root/var/svc/log/oxide-crucible-pantry:default.log
13342024-02-01T23:14:22.887Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c1d64f1b-34e4-4856-83eb-121d7977f99d/root/var/svc/log/oxide-crucible-agent:default.log (23475 bytes)
13352024-02-01T23:14:24.546Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c1d64f1b-34e4-4856-83eb-121d7977f99d/root/var/svc/log/oxide-crucible-agent:default.log
13362024-02-01T23:14:24.579Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c1d64f1b-34e4-4856-83eb-121d7977f99d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
13372024-02-01T23:14:25.285Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c1d64f1b-34e4-4856-83eb-121d7977f99d/root/var/svc/log/oxide-crucible-downstairs:default.log
13382024-02-01T23:14:25.318Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c1d64f1b-34e4-4856-83eb-121d7977f99d/root/var/svc/log/oxide-crucible-downstairs:downstairs-1f85a585-02fb-4084-99b5-fee154a2eda3.log (2943 bytes)
13392024-02-01T23:14:26.833Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c1d64f1b-34e4-4856-83eb-121d7977f99d/root/var/svc/log/oxide-crucible-downstairs:downstairs-1f85a585-02fb-4084-99b5-fee154a2eda3.log
13402024-02-01T23:14:26.833Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c1d64f1b-34e4-4856-83eb-121d7977f99d/root/var/svc/log/oxide-crucible-downstairs:downstairs-bd445e26-5f44-4334-aa79-823e96700855.log (5620 bytes)
13412024-02-01T23:14:27.965Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c1d64f1b-34e4-4856-83eb-121d7977f99d/root/var/svc/log/oxide-crucible-downstairs:downstairs-bd445e26-5f44-4334-aa79-823e96700855.log
13422024-02-01T23:14:27.965Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c1d64f1b-34e4-4856-83eb-121d7977f99d/root/var/svc/log/oxide-crucible-downstairs:snapshot-bd445e26-5f44-4334-aa79-823e96700855-02bf3710-1c87-41ec-aee5-3f0193287fee.log (5180 bytes)
13432024-02-01T23:14:29.093Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c1d64f1b-34e4-4856-83eb-121d7977f99d/root/var/svc/log/oxide-crucible-downstairs:snapshot-bd445e26-5f44-4334-aa79-823e96700855-02bf3710-1c87-41ec-aee5-3f0193287fee.log
13442024-02-01T23:14:29.093Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7/root/var/log/chrony/measurements.log (22468 bytes)
13452024-02-01T23:14:29.219Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7/root/var/log/chrony/measurements.log" changed size mid upload: 22468 -> 26578
13462024-02-01T23:14:30.316Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7/root/var/log/chrony/measurements.log
13472024-02-01T23:14:30.316Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7/root/var/log/chrony/statistics.log (13572 bytes)
13482024-02-01T23:14:30.437Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7/root/var/log/chrony/statistics.log" changed size mid upload: 13572 -> 16848
13492024-02-01T23:14:31.534Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7/root/var/log/chrony/statistics.log
13502024-02-01T23:14:31.535Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7/root/var/log/chrony/tracking.log (2119 bytes)
13512024-02-01T23:14:31.596Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7/root/var/log/chrony/tracking.log" changed size mid upload: 2119 -> 2383
13522024-02-01T23:14:32.963Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_45712398-535f-4d1b-8254-ffa03ea38ca7/root/var/log/chrony/tracking.log