01HJ21EYSP1VJZ5AK3M2PJFY9N: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.27",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HJ21FEKWZFM32HF8QSC4VYY8

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-19T22:47:02.974Zjob dependencies complete; ready to run (waiting for 47 m 6 s)
22023-12-19T22:50:36.263Zjob assigned to worker 01HJ245WP34ZAKM18E2GGWABJW (queued for 3 m 33 s)
32023-12-19T22:50:36.562Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-12-19T22:50:36.615Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-12-19T22:50:36.659Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-12-19T22:50:36.707Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-19T22:50:36.752Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-12-19T22:50:36.802ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-19T22:50:36.842Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-19T22:50:36.889ZOxide Helios Version helios-2.0.22333 64-bit
112023-12-19T22:50:36.936ZConfiguring devices.
122023-12-19T22:50:36.985Z##################################################
132023-12-19T22:50:37.034Z#### ############# ##
142023-12-19T22:50:37.081Z### ### ############ ##
152023-12-19T22:50:37.125Z## ### # ## ### ## Oxide Computer Company ##
162023-12-19T22:50:37.173Z## ## ## ### # ### ##
172023-12-19T22:50:37.218Z## # ### #### #### This Station Under ##
182023-12-19T22:50:37.268Z### ### #### # ### Computer Control ##
192023-12-19T22:50:37.313Z#### #### ### ## ##
202023-12-19T22:50:37.359Z##################################################
212023-12-19T22:50:37.402ZHostname: sock
222023-12-19T22:50:37.464ZLoading smf(7) service descriptions:
232023-12-19T22:50:37.809Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-8db5a4cac1990b57.gz
242023-12-19T22:50:39.628Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-8db5a4cac1990b57.gz
252023-12-19T22:50:39.628Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-19T22:51:03.959Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-19T22:51:04.001Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e4a60fbd4bd87582.gz
282023-12-19T22:51:29.089Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e4a60fbd4bd87582.gz
292023-12-19T22:51:29.125Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-19T22:51:30.281Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-19T22:51:30.281Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-19T22:51:34.166Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-19T22:51:34.203Zdownloading input: /input/package/work/version.txt
342023-12-19T22:51:34.235Zdownloaded input: /input/package/work/version.txt
352023-12-19T22:51:34.235Zdownloading input: /input/package/work/package.tar.gz
362023-12-19T22:51:54.793Zdownloaded input: /input/package/work/package.tar.gz
372023-12-19T22:51:54.826Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-19T22:52:25.880Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-19T22:52:25.914Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-19T22:52:35.196Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-19T22:52:35.235Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-19T22:53:20.481Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-19T22:53:20.483Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-19T22:54:06.612Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-19T22:54:06.612Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-19T22:54:37.117Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-19T22:54:37.152Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-19T22:54:42.853Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-19T22:54:42.895Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-19T22:54:53.658Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-19T22:54:53.731Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-19T22:54:59.971Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-19T22:55:00.004Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-19T22:55:06.339Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-19T22:55:06.339Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-19T22:55:06.391Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-19T22:55:06.427Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-19T22:55:10.976Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-19T22:55:10.977Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-19T22:55:15.521Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-19T22:55:15.523Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-19T22:55:34.104Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-19T22:55:34.104Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-19T22:55:52.899Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-19T22:55:52.937Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-19T22:55:52.992Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-19T22:55:53.026Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-19T22:56:02.037Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-19T22:56:02.037Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-19T22:56:18.717Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-19T22:56:18.717Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-19T22:57:20.017Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-19T22:57:20.017Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-19T22:58:02.029Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-19T22:58:02.063Zstarting task 0: "setup"
762023-12-19T22:58:02.063Z++ uname -s
772023-12-19T22:58:02.063Z+ kern=SunOS
782023-12-19T22:58:02.063Z+ case "$kern" in
792023-12-19T22:58:02.063Z+ groupadd -g 12345 build
802023-12-19T22:58:02.063Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-19T22:58:04.040Z+ zfs create -o mountpoint=/work rpool/work
822023-12-19T22:58:04.076Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-19T22:58:04.108Z+ home_fs=zfs
842023-12-19T22:58:04.108Z+ [[ zfs == autofs ]]
852023-12-19T22:58:04.108Z+ mkdir -p /home/build
862023-12-19T22:58:04.109Z+ chown build:build /home/build /work
872023-12-19T22:58:06.080Z+ chmod 0700 /home/build /work
882023-12-19T22:58:06.113Zprocess exited: duration 4059 ms, exit code 0
 
892023-12-19T22:58:06.180Zstarting task 1: "authentication"
902023-12-19T22:58:06.243Zprocess exited: duration 61 ms, exit code 0
 
912023-12-19T22:58:06.310Zstarting task 2: "build"
922023-12-19T22:58:06.345Z+ trap _exit_trap EXIT
932023-12-19T22:58:06.345Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-19T22:58:06.520Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-19T22:58:06.520Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-19T22:58:06.520Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-19T22:58:06.520Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-19T22:58:06.520Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-19T22:58:06.520Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-19T22:58:06.520Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-19T22:58:06.520Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-19T22:58:06.557Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-19T22:58:06.557Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-19T22:58:06.557Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-19T22:58:06.557Z+ [[ -d /opt/oxide ]]
1062023-12-19T22:58:06.557Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-19T22:58:06.557Z+ pfexec mkdir /opt/oxide
1082023-12-19T22:58:06.557Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-19T22:58:06.557Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-19T22:58:06.557Z+ cd /opt/oxide-underneath
1112023-12-19T22:58:06.557Z+ pfexec tar ceEp@/f - .
1122023-12-19T22:58:06.558Z+ cd /opt/oxide
1132023-12-19T22:58:06.558Z+ pfexec tar xveEp@/f -
1142023-12-19T22:58:06.677Zx ., 0 bytes, 0 tape blocks
1152023-12-19T22:58:06.677Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-19T22:58:06.677Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-19T22:58:06.678Zx ./opte/bin/opteadm, 52567720 bytes, 102672 tape blocks
1182023-12-19T22:58:06.716Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-19T22:58:06.716Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-19T22:58:06.716Z+ pfexec mkdir /opt/oxide/work
1212023-12-19T22:58:06.716Z+ pfexec chown build:build /opt/oxide/work
1222023-12-19T22:58:06.716Z+ cd /opt/oxide/work
1232023-12-19T22:58:06.716Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-19T22:58:08.225ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-12-19T22:58:08.225Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-12-19T22:58:08.225Zx out/omicron-sled-agent.tar, 111899648 bytes, 218554 tape blocks
1272023-12-19T22:58:08.225Zx out/target/test, 75 bytes, 1 tape blocks
1282023-12-19T22:58:08.225Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1292023-12-19T22:58:08.225Zx package-manifest.toml, 22445 bytes, 44 tape blocks
1302023-12-19T22:58:08.225Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1312023-12-19T22:58:08.265Zx target/release/omicron-package, 42522064 bytes, 83051 tape blocks
1322023-12-19T22:58:08.265Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-12-19T22:58:08.265Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-12-19T22:58:08.265Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-12-19T22:58:08.265Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-12-19T22:58:08.265Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-12-19T22:58:08.265Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-12-19T22:58:08.265Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-12-19T22:58:08.265Z
1402023-12-19T22:58:08.266Zreal 1.543092239
1412023-12-19T22:58:08.266Zuser 1.385876395
1422023-12-19T22:58:08.266Zsys 0.272658426
1432023-12-19T22:58:08.266Ztrap 0.000058069
1442023-12-19T22:58:08.266Ztflt 0.000000000
1452023-12-19T22:58:08.266Zdflt 0.000000000
1462023-12-19T22:58:08.266Zkflt 0.000000000
1472023-12-19T22:58:08.266Zlock 0.000000000
1482023-12-19T22:58:08.266Zslp 1.388340588
1492023-12-19T22:58:08.266Zlat 0.037454560
1502023-12-19T22:58:08.266Zstop 0.000116972
1512023-12-19T22:58:08.266Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-12-19T22:58:09.315Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-12-19T22:58:09.350Z+ mkdir tests
1542023-12-19T22:58:09.350Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-12-19T22:58:09.350Z+ ptime -m gunzip
1562023-12-19T22:58:09.350Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8db5a4cac1990b57
1572023-12-19T22:58:09.477Z
1582023-12-19T22:58:09.517Zreal 0.141882013
1592023-12-19T22:58:09.517Zuser 0.128622015
1602023-12-19T22:58:09.518Zsys 0.013024794
1612023-12-19T22:58:09.518Ztrap 0.000008720
1622023-12-19T22:58:09.519Ztflt 0.000000000
1632023-12-19T22:58:09.519Zdflt 0.000000000
1642023-12-19T22:58:09.519Zkflt 0.000000000
1652023-12-19T22:58:09.519Zlock 0.000000000
1662023-12-19T22:58:09.519Zslp 0.000017700
1672023-12-19T22:58:09.519Zlat 0.000020060
1682023-12-19T22:58:09.519Zstop 0.000120381
1692023-12-19T22:58:09.519Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8db5a4cac1990b57
1702023-12-19T22:58:09.519Z+ chmod a+x tests/bootstrap-8db5a4cac1990b57
1712023-12-19T22:58:09.519Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-12-19T22:58:09.519Z+ ptime -m gunzip
1732023-12-19T22:58:09.519Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-12-19T22:58:11.755Z
1752023-12-19T22:58:11.797Zreal 2.269624681
1762023-12-19T22:58:11.797Zuser 2.125757215
1772023-12-19T22:58:11.797Zsys 0.143465708
1782023-12-19T22:58:11.797Ztrap 0.000176664
1792023-12-19T22:58:11.797Ztflt 0.000000000
1802023-12-19T22:58:11.797Zdflt 0.000000000
1812023-12-19T22:58:11.797Zkflt 0.000000000
1822023-12-19T22:58:11.797Zlock 0.000000000
1832023-12-19T22:58:11.797Zslp 0.000023210
1842023-12-19T22:58:11.798Zlat 0.000022319
1852023-12-19T22:58:11.798Zstop 0.000129551
1862023-12-19T22:58:11.798Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-12-19T22:58:11.798Z+ chmod a+x tests/bootstrap
1882023-12-19T22:58:11.798Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-12-19T22:58:11.798Z+ ptime -m gunzip
1902023-12-19T22:58:11.798Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e4a60fbd4bd87582
1912023-12-19T22:58:14.068Z
1922023-12-19T22:58:14.108Zreal 2.300215124
1932023-12-19T22:58:14.108Zuser 2.161636119
1942023-12-19T22:58:14.108Zsys 0.138205108
1952023-12-19T22:58:14.108Ztrap 0.000161063
1962023-12-19T22:58:14.109Ztflt 0.000000000
1972023-12-19T22:58:14.109Zdflt 0.000000000
1982023-12-19T22:58:14.109Zkflt 0.000000000
1992023-12-19T22:58:14.109Zlock 0.000000000
2002023-12-19T22:58:14.109Zslp 0.000020410
2012023-12-19T22:58:14.109Zlat 0.000016909
2022023-12-19T22:58:14.109Zstop 0.000112342
2032023-12-19T22:58:14.109Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e4a60fbd4bd87582
2042023-12-19T22:58:14.111Z+ chmod a+x tests/end_to_end_tests-e4a60fbd4bd87582
2052023-12-19T22:58:14.111Z+ bmat address ls
2062023-12-19T22:58:14.111ZNAME CIDR FIRST LAST COUNT
2072023-12-19T22:58:14.111Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-12-19T22:58:14.111Z++ bmat address ls -f extra -Ho first,last,count
2092023-12-19T22:58:14.111Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-12-19T22:58:14.111Z+ EXTRA_IP_START=10.151.2.164
2112023-12-19T22:58:14.111Z+ EXTRA_IP_END=10.151.2.227
2122023-12-19T22:58:14.111Z+ EXTRA_IP_COUNT=64
2132023-12-19T22:58:14.111Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-12-19T22:58:14.111Z+ EXTRA_IP_BASE=10.151.2
2152023-12-19T22:58:14.111Z+ EXTRA_IP_FOCTET=164
2162023-12-19T22:58:14.111Z+ EXTRA_IP_LOCTET=227
2172023-12-19T22:58:14.111Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-12-19T22:58:14.111Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-12-19T22:58:14.111Z+ DNS_IP1=10.151.2.164
2202023-12-19T22:58:14.111Z+ DNS_IP2=10.151.2.165
2212023-12-19T22:58:14.111Z+ UPLINK_IP=10.151.2.174
2222023-12-19T22:58:14.111Z+ IPPOOL_START=10.151.2.175
2232023-12-19T22:58:14.111Z+ IPPOOL_END=10.151.2.227
2242023-12-19T22:58:14.112Z++ ipadm show-addr -po type,addr
2252023-12-19T22:58:14.112Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-12-19T22:58:14.112Z+ GATEWAY_IP=10.151.2.100
2272023-12-19T22:58:14.112Z+ [[ -n 10.151.2.100 ]]
2282023-12-19T22:58:14.112Z+ ping -s 10.151.2.100 56 1
2292023-12-19T22:58:14.149ZPING 10.151.2.100: 56 data bytes
2302023-12-19T22:58:14.149Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2312023-12-19T22:58:14.150Z
2322023-12-19T22:58:14.150Z----10.151.2.100 PING Statistics----
2332023-12-19T22:58:14.150Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-12-19T22:58:14.150Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-12-19T22:58:14.150Z++ arp -an
2362023-12-19T22:58:14.150Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-12-19T22:58:14.150Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-12-19T22:58:14.150Z+ routeadm -e ipv4-forwarding -u
2392023-12-19T22:58:14.253Z+ PXA_START=10.151.2.164
2402023-12-19T22:58:14.254Z+ PXA_END=10.151.2.227
2412023-12-19T22:58:14.290Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-12-19T22:58:14.290Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-12-19T22:58:14.386Z+ ZPOOL_VDEV_DIR=/scratch
2442023-12-19T22:58:14.386Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-12-19T22:58:14.421Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-12-19T22:58:14.421Z++ cd ./tools
2472023-12-19T22:58:14.423Z++ pwd
2482023-12-19T22:58:14.423Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-12-19T22:58:14.423Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-12-19T22:58:14.423Z+ SOFTNPU_MODE=zone
2512023-12-19T22:58:14.423Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-12-19T22:58:14.423Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-12-19T22:58:14.423Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-12-19T22:58:14.423Z++ dladm show-phys -p -o LINK
2552023-12-19T22:58:14.423Z++ head -1
2562023-12-19T22:58:14.467Z+ PHYSICAL_LINK=igb0
2572023-12-19T22:58:14.536Z+ echo 'Using igb0 as physical link'
2582023-12-19T22:58:14.537ZUsing igb0 as physical link
2592023-12-19T22:58:14.538Z+ ensure_run_as_root
2602023-12-19T22:58:14.538Z++ id -u
2612023-12-19T22:58:14.538Z+ [[ 0 -ne 0 ]]
2622023-12-19T22:58:14.538Z+ ensure_zpools
2632023-12-19T22:58:14.538Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-12-19T22:58:14.538Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-12-19T22:58:14.538Z+ readarray -t ZPOOLS
2662023-12-19T22:58:14.538Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-12-19T22:58:14.538Z++ sed 's/[ ",]//g'
2682023-12-19T22:58:14.538Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-12-19T22:58:14.538Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-12-19T22:58:14.538ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-12-19T22:58:14.538Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-12-19T22:58:14.538Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-12-19T22:58:14.538Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-12-19T22:58:14.538Z0+0 records in
2752023-12-19T22:58:14.539Z0+0 records out
2762023-12-19T22:58:14.539Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2772023-12-19T22:58:14.539Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-12-19T22:58:14.539Z+ set +x
2792023-12-19T22:58:14.539ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-12-19T22:58:14.667Z++ zpool list -o name
2812023-12-19T22:58:14.667Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-12-19T22:58:14.667Z+ [[ -z '' ]]
2832023-12-19T22:58:14.667Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-12-19T22:58:14.667Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-12-19T22:58:14.669Z+ set +x
2862023-12-19T22:58:14.669ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-12-19T22:58:14.669Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-12-19T22:58:14.669Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-12-19T22:58:14.669ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-12-19T22:58:14.669Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-12-19T22:58:14.669Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-12-19T22:58:14.669Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-12-19T22:58:14.669Z0+0 records in
2942023-12-19T22:58:14.669Z0+0 records out
2952023-12-19T22:58:14.669Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-12-19T22:58:14.669Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-12-19T22:58:14.669Z+ set +x
2982023-12-19T22:58:14.669ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-12-19T22:58:14.669Z++ zpool list -o name
3002023-12-19T22:58:14.669Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-12-19T22:58:14.669Z+ [[ -z '' ]]
3022023-12-19T22:58:14.669Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-12-19T22:58:14.669Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-12-19T22:58:14.669Z+ set +x
3052023-12-19T22:58:14.669ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-12-19T22:58:14.669Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-12-19T22:58:14.669Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-12-19T22:58:14.670ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3092023-12-19T22:58:14.670Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-12-19T22:58:14.670Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-12-19T22:58:14.670Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-12-19T22:58:14.670Z0+0 records in
3132023-12-19T22:58:14.670Z0+0 records out
3142023-12-19T22:58:14.670Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3152023-12-19T22:58:14.670Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-12-19T22:58:14.670Z+ set +x
3172023-12-19T22:58:14.670ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-12-19T22:58:14.670Z++ zpool list -o name
3192023-12-19T22:58:14.670Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-12-19T22:58:14.735Z+ [[ -z '' ]]
3212023-12-19T22:58:14.735Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-12-19T22:58:14.735Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-12-19T22:58:14.735Z+ set +x
3242023-12-19T22:58:14.735ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-12-19T22:58:14.735Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-12-19T22:58:14.735Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-12-19T22:58:14.735ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-12-19T22:58:14.735Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-12-19T22:58:14.735Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-12-19T22:58:14.735Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-12-19T22:58:14.735Z0+0 records in
3322023-12-19T22:58:14.735Z0+0 records out
3332023-12-19T22:58:14.735Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-12-19T22:58:14.735Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-12-19T22:58:14.735Z+ set +x
3362023-12-19T22:58:14.736ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-12-19T22:58:14.736Z++ zpool list -o name
3382023-12-19T22:58:14.736Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-12-19T22:58:14.736Z+ [[ -z '' ]]
3402023-12-19T22:58:14.736Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-12-19T22:58:14.736Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-12-19T22:58:14.736Z+ set +x
3432023-12-19T22:58:14.736ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-12-19T22:58:14.736Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-12-19T22:58:14.736Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-12-19T22:58:14.736ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-12-19T22:58:14.736Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-12-19T22:58:14.736Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-12-19T22:58:14.736Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-12-19T22:58:14.736Z0+0 records in
3512023-12-19T22:58:14.736Z0+0 records out
3522023-12-19T22:58:14.736Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3532023-12-19T22:58:14.736Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-12-19T22:58:14.736Z+ set +x
3552023-12-19T22:58:14.736ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-12-19T22:58:14.736Z++ zpool list -o name
3572023-12-19T22:58:14.736Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-12-19T22:58:14.737Z+ [[ -z '' ]]
3592023-12-19T22:58:14.738Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-12-19T22:58:14.738Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-12-19T22:58:14.738Z+ set +x
3622023-12-19T22:58:14.738ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-12-19T22:58:14.738Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-12-19T22:58:14.738Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-12-19T22:58:14.738ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-12-19T22:58:14.738Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-12-19T22:58:14.738Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-12-19T22:58:14.738Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-12-19T22:58:14.738Z0+0 records in
3702023-12-19T22:58:14.738Z0+0 records out
3712023-12-19T22:58:14.738Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3722023-12-19T22:58:14.738Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-12-19T22:58:14.738Z+ set +x
3742023-12-19T22:58:14.738ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-12-19T22:58:14.738Z++ zpool list -o name
3762023-12-19T22:58:14.738Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-12-19T22:58:14.776Z+ [[ -z '' ]]
3782023-12-19T22:58:14.776Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-12-19T22:58:14.776Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-12-19T22:58:14.776Z+ set +x
3812023-12-19T22:58:14.776ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-12-19T22:58:14.776Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-12-19T22:58:14.776Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-12-19T22:58:14.776ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-12-19T22:58:14.776Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-12-19T22:58:14.776Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-12-19T22:58:14.776Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-12-19T22:58:14.776Z0+0 records in
3892023-12-19T22:58:14.777Z0+0 records out
3902023-12-19T22:58:14.777Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-12-19T22:58:14.777Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-12-19T22:58:14.777Z+ set +x
3932023-12-19T22:58:14.777ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-12-19T22:58:14.777Z++ zpool list -o name
3952023-12-19T22:58:14.777Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-12-19T22:58:14.777Z+ [[ -z '' ]]
3972023-12-19T22:58:14.777Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-12-19T22:58:14.815ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3992023-12-19T22:58:14.815Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4002023-12-19T22:58:14.815ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4012023-12-19T22:58:14.815Z+ set +x
4022023-12-19T22:58:14.815Z+ for ZPOOL in "${ZPOOLS[@]}"
4032023-12-19T22:58:14.815Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4042023-12-19T22:58:14.815Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-12-19T22:58:14.815Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-12-19T22:58:14.815Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-12-19T22:58:14.815Z0+0 records in
4082023-12-19T22:58:14.815Z0+0 records out
4092023-12-19T22:58:14.815Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-12-19T22:58:14.815Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-12-19T22:58:14.815Z+ set +x
4122023-12-19T22:58:14.815ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-12-19T22:58:14.815Z++ zpool list -o name
4142023-12-19T22:58:14.815Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-12-19T22:58:14.852Z+ [[ -z '' ]]
4162023-12-19T22:58:14.852Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-12-19T22:58:14.853Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-12-19T22:58:14.853Z+ set +x
4192023-12-19T22:58:14.853ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-12-19T22:58:14.853ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4212023-12-19T22:58:14.853Z+ for ZPOOL in "${ZPOOLS[@]}"
4222023-12-19T22:58:14.853Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4232023-12-19T22:58:14.853Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-12-19T22:58:14.853Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-12-19T22:58:14.853Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-12-19T22:58:14.853Z0+0 records in
4272023-12-19T22:58:14.853Z0+0 records out
4282023-12-19T22:58:14.853Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4292023-12-19T22:58:14.853Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-12-19T22:58:14.853Z+ set +x
4312023-12-19T22:58:14.853ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-12-19T22:58:14.889Z++ zpool list -o name
4332023-12-19T22:58:14.891Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-12-19T22:58:14.891Z+ [[ -z '' ]]
4352023-12-19T22:58:14.891Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-12-19T22:58:14.955Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-12-19T22:58:14.956Z+ set +x
4382023-12-19T22:58:14.956ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-12-19T22:58:14.956Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-12-19T22:58:14.956Z+ readarray -t ZPOOLS
4412023-12-19T22:58:14.956Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-12-19T22:58:14.956Z++ sed 's/[ ",]//g'
4432023-12-19T22:58:14.956Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-12-19T22:58:14.956Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-12-19T22:58:14.956ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-12-19T22:58:14.956Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-12-19T22:58:14.956Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-12-19T22:58:14.956Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-12-19T22:58:14.956Z0+0 records in
4502023-12-19T22:58:14.956Z0+0 records out
4512023-12-19T22:58:14.956Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4522023-12-19T22:58:14.956Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-12-19T22:58:14.956Z+ set +x
4542023-12-19T22:58:14.956ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-12-19T22:58:14.956Z++ zpool list -o name
4562023-12-19T22:58:14.956Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-12-19T22:58:14.956Z+ [[ -z '' ]]
4582023-12-19T22:58:14.956Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-12-19T22:58:14.999Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-12-19T22:58:14.999Z+ set +x
4612023-12-19T22:58:14.999ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-12-19T22:58:14.999Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-12-19T22:58:14.999Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-12-19T22:58:14.999ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-12-19T22:58:14.999Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-12-19T22:58:14.999Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-12-19T22:58:14.999Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-12-19T22:58:14.999Z0+0 records in
4692023-12-19T22:58:14.999Z0+0 records out
4702023-12-19T22:58:14.999Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4712023-12-19T22:58:14.999Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-12-19T22:58:14.999Z+ set +x
4732023-12-19T22:58:14.999ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-12-19T22:58:14.999Z++ zpool list -o name
4752023-12-19T22:58:14.999Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-12-19T22:58:14.999Z+ [[ -z '' ]]
4772023-12-19T22:58:14.999Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-12-19T22:58:14.999Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-12-19T22:58:15.000Z+ set +x
4802023-12-19T22:58:15.000ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-12-19T22:58:15.000Z+ [[ zone == \z\o\n\e ]]
4822023-12-19T22:58:15.001Z+ ensure_simulated_links igb0
4832023-12-19T22:58:15.001Z+ local PHYSICAL_LINK=igb0
4842023-12-19T22:58:15.001Z+ INDICES=("0" "1")
4852023-12-19T22:58:15.001Z+ for I in "${INDICES[@]}"
4862023-12-19T22:58:15.001Z++ get_simnet_name_if_exists net0
4872023-12-19T22:58:15.001Z++ dladm show-simnet -p -o LINK net0
4882023-12-19T22:58:15.037Z++ echo ''
4892023-12-19T22:58:15.037Z+ [[ -z '' ]]
4902023-12-19T22:58:15.037Z+ dladm create-simnet -t net0
4912023-12-19T22:58:15.037Z+ dladm create-simnet -t sc0_0
4922023-12-19T22:58:15.037Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-12-19T22:58:15.037Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-12-19T22:58:15.037Z+ success 'Simnet net0/sc0_0 exists'
4952023-12-19T22:58:15.037Z+ set +x
4962023-12-19T22:58:15.037Z+ for I in "${INDICES[@]}"
4972023-12-19T22:58:15.037ZSimnet net0/sc0_0 exists
4982023-12-19T22:58:15.037Z++ get_simnet_name_if_exists net1
4992023-12-19T22:58:15.037Z++ dladm show-simnet -p -o LINK net1
5002023-12-19T22:58:15.037Z++ echo ''
5012023-12-19T22:58:15.037Z+ [[ -z '' ]]
5022023-12-19T22:58:15.037Z+ dladm create-simnet -t net1
5032023-12-19T22:58:15.037Z+ dladm create-simnet -t sc1_0
5042023-12-19T22:58:15.037Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-12-19T22:58:15.037Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-12-19T22:58:15.037Z+ success 'Simnet net1/sc1_0 exists'
5072023-12-19T22:58:15.037Z+ set +x
5082023-12-19T22:58:15.038ZSimnet net1/sc1_0 exists
5092023-12-19T22:58:15.038Z++ get_vnic_name_if_exists sc0_1
5102023-12-19T22:58:15.038Z++ dladm show-vnic -p -o LINK sc0_1
5112023-12-19T22:58:15.075Z++ echo ''
5122023-12-19T22:58:15.075Z+ [[ -z '' ]]
5132023-12-19T22:58:15.075Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-12-19T22:58:15.075Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-12-19T22:58:15.075Z+ success 'Vnic sc0_1 exists'
5162023-12-19T22:58:15.075Z+ set +x
5172023-12-19T22:58:15.076ZVnic sc0_1 exists
5182023-12-19T22:58:15.076Z+ warn_if_no_proxy_arp
5192023-12-19T22:58:15.076Z+ [[ -v PXA_START ]]
5202023-12-19T22:58:15.076Z+ [[ -v PXA_END ]]
5212023-12-19T22:58:15.076Z+ ensure_softnpu_zone
5222023-12-19T22:58:15.076Z+ zoneadm list
5232023-12-19T22:58:15.076Z+ grep -q sidecar_softnpu
5242023-12-19T22:58:15.076Z+ [[ -f out/npuzone/npuzone ]]
5252023-12-19T22:58:15.076Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit 45ed98fea5824feb4d42f45bbf218e597dc9fc58 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5262023-12-19T22:58:15.076Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5272023-12-19T22:58:23.630Zdownloading softnpu ...done
5282023-12-19T22:58:23.668Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/libsidecar_lite.so.sha256.txt
5292023-12-19T22:58:29.729Zdownloading asic_program.so ...done
5302023-12-19T22:58:29.767Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/scadm.sha256.txt
5312023-12-19T22:58:37.221Zdownloading scadm ...done
5322023-12-19T22:58:37.264Zcreating port tfportrear0_0 for sc0_0
5332023-12-19T22:58:37.264Zcreating port tfportqsfp0_0 for sc0_1
5342023-12-19T22:58:37.264Zsoftnpu zone setup
5352023-12-19T22:58:37.264Zconfigure zone
5362023-12-19T22:58:37.298Zinstall zone
5372023-12-19T22:58:38.346Zboot zone
5382023-12-19T22:58:41.457Z[sidecar_softnpu] svcs milestone/network
5392023-12-19T22:58:41.490ZSTATE STIME FMRI
5402023-12-19T22:58:41.491Zonline 22:58:39 svc:/milestone/network:default
5412023-12-19T22:58:41.491Z[sidecar_softnpu] /softnpu/npu start
5422023-12-19T22:58:41.533Z
5432023-12-19T22:58:41.533Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-12-19T22:58:41.533Z+ GATEWAY_IP=10.151.2.100
5452023-12-19T22:58:41.533Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-12-19T22:58:41.534ZUsing 10.151.2.100 as gateway ip
5472023-12-19T22:58:41.534Z+ [[ ! -v GATEWAY_MAC ]]
5482023-12-19T22:58:41.535Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-12-19T22:58:41.535Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-12-19T22:58:41.535Z+++ echo 18:c0:4d:0d:a0:2a
5512023-12-19T22:58:41.535Z+ COUNT=1
5522023-12-19T22:58:41.535Z+ [[ 1 -ne 1 ]]
5532023-12-19T22:58:41.535Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-12-19T22:58:41.535ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-12-19T22:58:41.535Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-12-19T22:58:41.535Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-12-19T22:58:41.574Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-12-19T22:58:41.574Z+ [[ -v PXA_START ]]
5592023-12-19T22:58:41.574Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-12-19T22:58:41.574Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-12-19T22:58:41.616Z+ z_scadm dump-state
5622023-12-19T22:58:41.616Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-12-19T22:58:41.616Zlocal v6:
5642023-12-19T22:58:41.616Zlocal v4:
5652023-12-19T22:58:41.616Zrouter v6:
5662023-12-19T22:58:41.616Zrouter v4_idx:
5672023-12-19T22:58:41.616Zrouter v4_routes:
5682023-12-19T22:58:41.616Zresolver v4:
5692023-12-19T22:58:41.616Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-12-19T22:58:41.616Zresolver v6:
5712023-12-19T22:58:41.616Znat v4:
5722023-12-19T22:58:41.616Znat_v6:
5732023-12-19T22:58:41.617Zport_mac:
5742023-12-19T22:58:41.617Zproxy arp:
5752023-12-19T22:58:41.617Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-12-19T22:58:41.617Z+ success 'softnpu zone exists'
5772023-12-19T22:58:41.617Z+ set +x
5782023-12-19T22:58:41.617Zsoftnpu zone exists
5792023-12-19T22:58:41.618Z
5802023-12-19T22:58:41.618Zreal 27.220566027
5812023-12-19T22:58:41.618Zuser 0.712210010
5822023-12-19T22:58:41.618Zsys 1.635342937
5832023-12-19T22:58:41.618Ztrap 0.000093536
5842023-12-19T22:58:41.618Ztflt 0.000197107
5852023-12-19T22:58:41.618Zdflt 0.002219610
5862023-12-19T22:58:41.618Zkflt 0.000000000
5872023-12-19T22:58:41.618Zlock 21:05.789610002
5882023-12-19T22:58:41.618Zslp 1:21.693304396
5892023-12-19T22:58:41.618Zlat 0.064514847
5902023-12-19T22:58:41.618Zstop 0.340426373
5912023-12-19T22:58:41.618Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-12-19T22:58:41.661Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-12-19T22:58:41.661Z+ SILO_NAME=recovery
5942023-12-19T22:58:41.661Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-19T22:58:41.661Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-12-19T22:58:41.661Z+ sed -i~ '
5972023-12-19T22:58:41.661Z /^external_dns_ips/c\
5982023-12-19T22:58:41.661Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-12-19T22:58:41.661Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-12-19T22:58:41.661Z /^first/c\
6012023-12-19T22:58:41.661Zfirst = "10.151.2.164"
6022023-12-19T22:58:41.662Z /^last/c\
6032023-12-19T22:58:41.662Zlast = "10.151.2.173"
6042023-12-19T22:58:41.662Z }
6052023-12-19T22:58:41.662Z /^\[rack_network_config/,/^$/ {
6062023-12-19T22:58:41.662Z /^infra_ip_first/c\
6072023-12-19T22:58:41.662Zinfra_ip_first = "10.151.2.174"
6082023-12-19T22:58:41.662Z /^infra_ip_last/c\
6092023-12-19T22:58:41.662Zinfra_ip_last = "10.151.2.174"
6102023-12-19T22:58:41.662Z }
6112023-12-19T22:58:41.662Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-12-19T22:58:41.662Z /^routes/c\
6132023-12-19T22:58:41.662Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6142023-12-19T22:58:41.663Z /^addresses/c\
6152023-12-19T22:58:41.663Zaddresses = \["10.151.2.174/32"\]
6162023-12-19T22:58:41.663Z }
6172023-12-19T22:58:41.663Z' pkg/config-rss.toml
6182023-12-19T22:58:41.663Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-12-19T22:58:41.664Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-12-19T22:58:41.664Z+++ pkg/config-rss.toml Tue Dec 19 22:58:41 2023
6212023-12-19T22:58:41.664Z@@ -41,7 +41,7 @@
6222023-12-19T22:58:41.664Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-12-19T22:58:41.664Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-12-19T22:58:41.664Z # below.
6252023-12-19T22:58:41.664Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-12-19T22:58:41.664Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6272023-12-19T22:58:41.664Z
6282023-12-19T22:58:41.664Z # Initial TLS certificates for the external API
6292023-12-19T22:58:41.664Z #
6302023-12-19T22:58:41.665Z@@ -77,8 +77,8 @@
6312023-12-19T22:58:41.665Z #
6322023-12-19T22:58:41.665Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-12-19T22:58:41.665Z [[internal_services_ip_pool_ranges]]
6342023-12-19T22:58:41.665Z-first = "192.168.1.20"
6352023-12-19T22:58:41.665Z-last = "192.168.1.29"
6362023-12-19T22:58:41.665Z+first = "10.151.2.164"
6372023-12-19T22:58:41.665Z+last = "10.151.2.173"
6382023-12-19T22:58:41.665Z
6392023-12-19T22:58:41.665Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-12-19T22:58:41.665Z # parameters should be provided to the control plane via wicket, but we need to
6412023-12-19T22:58:41.666Z@@ -92,8 +92,8 @@
6422023-12-19T22:58:41.666Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-12-19T22:58:41.666Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-12-19T22:58:41.666Z # With softnpu, only one address is used.
6452023-12-19T22:58:41.666Z-infra_ip_first = "192.168.1.30"
6462023-12-19T22:58:41.666Z-infra_ip_last = "192.168.1.30"
6472023-12-19T22:58:41.666Z+infra_ip_first = "10.151.2.174"
6482023-12-19T22:58:41.666Z+infra_ip_last = "10.151.2.174"
6492023-12-19T22:58:41.666Z
6502023-12-19T22:58:41.666Z # Configurations for BGP routers to run on the scrimlets.
6512023-12-19T22:58:41.666Z bgp = []
6522023-12-19T22:58:41.666Z@@ -101,9 +101,9 @@
6532023-12-19T22:58:41.666Z # You can configure multiple uplinks by repeating the following stanza
6542023-12-19T22:58:41.666Z [[rack_network_config.ports]]
6552023-12-19T22:58:41.666Z # Routes associated with this port.
6562023-12-19T22:58:41.666Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-12-19T22:58:41.667Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6582023-12-19T22:58:41.667Z # Addresses associated with this port.
6592023-12-19T22:58:41.667Z-addresses = ["192.168.1.30/32"]
6602023-12-19T22:58:41.667Z+addresses = ["10.151.2.174/32"]
6612023-12-19T22:58:41.667Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-12-19T22:58:41.667Z port = "qsfp0"
6632023-12-19T22:58:41.667Z # The speed of this port.
6642023-12-19T22:58:41.667Z+ true
6652023-12-19T22:58:41.667Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-12-19T22:58:41.706Za pkg/config-rss.toml 7K
6672023-12-19T22:58:41.706Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-12-19T22:58:41.706Z+ TLS_NAME=recovery.sys.oxide.test
6692023-12-19T22:58:41.706Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-12-19T22:58:42.881Z.......+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..................+.........+...+..........+..+.+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+.....+................+.....+.......+..+.............+...+...........+...+.+......+...+..+......+....+...+..+.............+........+....+........+.+..+..........+.....+...................+........+.+....................+....+..................+........................+...+......+..+....+.....+...+.+.....+.+......+..+...+....+..............+.+.........+............+.........+........+......+....+............+......+..+.+.....+.........+..........+.....+...+....+...........+...+......+.+..+.+.........+..................+...+...........+...+.+.........................................+.............+.................+................+..+..........+..+.........+.+...........+....+...............+..................+.....+...+.......+..+....+..+...+.+......+..+.+............+...+.....+.+.....+......+...+.+....................+.............+..+...+...+......................+.....+......+.......+....................+...+.......+........+......+.+...+.........+............+........+.............+...+..............+...............+......+......+................+.....+.......+.................+......+..........+.....+...+....+.....+.............+...+............+..+............+......+...............+....+.........+.....+.........+.+......+...+.....+......+.+........+.+............+..+.......................................................+.....+.+...........+...+.............+.....+....+.....+...+...+.......+..+.+.....+...+.......+.....+....+...........+....+...+......+......+.....+....+.....+.........+...+.......+..+.........+....+...........+..........+.........+.....+.+..+...+..........+..+..........+...............+.....+...+...............+.+...+..+.........+....+......+..+....+...........+............+.............+..+...............+......................+.........+.....+....+...+...........................+........+.......+..+.........................+...+...+.....+.......+..+.........+....+..+....+..+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-12-19T22:58:43.045Z...+...+...+....+..+.....................+.........+....+.....+.......+...+......+...+..+.+..+......+......+.............+...+..+.......+.....+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+..+....+......+...+.........+..+...............+.+........+.+......+..............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.....+..........+..+.............+............+..+.............+..+.........+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-19T22:58:43.082Z-----
6732023-12-19T22:58:43.082Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-12-19T22:58:43.082Za pkg/initial-tls-cert.pem 2K
6752023-12-19T22:58:43.082Za pkg/initial-tls-key.pem 4K
6762023-12-19T22:58:43.082Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-12-19T22:58:43.082Z+ rmdir pkg
6782023-12-19T22:58:43.083Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-12-19T22:58:43.083Z+ pfexec mkdir -p /usr/oxide
6802023-12-19T22:58:43.084Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6812023-12-19T22:58:43.845Z+ pfexec chmod +x /usr/oxide/oxide
6822023-12-19T22:58:43.913Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6832023-12-19T22:59:02.395Z+ OMICRON_NO_UNINSTALL=1
6842023-12-19T22:59:02.397Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6852023-12-19T22:59:02.397ZDec 19 22:59:02.389 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6862023-12-19T22:59:02.397ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6872023-12-19T22:59:02.397ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6882023-12-19T22:59:02.397ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6892023-12-19T22:59:02.397ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6902023-12-19T22:59:02.397ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6912023-12-19T22:59:02.397ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6922023-12-19T22:59:02.397ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6932023-12-19T22:59:02.397ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6942023-12-19T22:59:02.397ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6952023-12-19T22:59:02.397ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6962023-12-19T22:59:02.397ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6972023-12-19T22:59:02.397ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6982023-12-19T22:59:02.398ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6992023-12-19T22:59:02.398ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7002023-12-19T22:59:02.398ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7012023-12-19T22:59:02.398ZDec 19 22:59:02.392 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7022023-12-19T22:59:02.537ZDec 19 22:59:02.620 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7032023-12-19T22:59:02.575ZDec 19 22:59:02.654 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7042023-12-19T22:59:02.635ZDec 19 22:59:02.711 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7052023-12-19T22:59:02.760Z
7062023-12-19T22:59:02.798Zreal 0.461206551
7072023-12-19T22:59:02.799Zuser 0.098962280
7082023-12-19T22:59:02.799Zsys 1.913680200
7092023-12-19T22:59:02.799Ztrap 0.000007638
7102023-12-19T22:59:02.799Ztflt 0.000039819
7112023-12-19T22:59:02.799Zdflt 0.000335597
7122023-12-19T22:59:02.800Zkflt 0.000000000
7132023-12-19T22:59:02.800Zlock 41.295267164
7142023-12-19T22:59:02.800Zslp 0.667472657
7152023-12-19T22:59:02.801Zlat 0.003805017
7162023-12-19T22:59:02.801Zstop 0.206783649
7172023-12-19T22:59:02.801Z+ retry=0
7182023-12-19T22:59:02.801Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7192023-12-19T22:59:02.801Z+ [[ 0 -gt 30 ]]
7202023-12-19T22:59:02.801Z+ sleep 1
7212023-12-19T22:59:03.773Z+ retry=1
7222023-12-19T22:59:03.773Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232023-12-19T22:59:03.806Z+ [[ 1 -gt 30 ]]
7242023-12-19T22:59:03.806Z+ sleep 1
7252023-12-19T22:59:04.788Z+ retry=2
7262023-12-19T22:59:04.789Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272023-12-19T22:59:04.824Z+ [[ 2 -gt 30 ]]
7282023-12-19T22:59:04.824Z+ sleep 1
7292023-12-19T22:59:05.802Z+ retry=3
7302023-12-19T22:59:05.803Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312023-12-19T22:59:05.836Z+ [[ 3 -gt 30 ]]
7322023-12-19T22:59:05.836Z+ sleep 1
7332023-12-19T22:59:06.816Z+ retry=4
7342023-12-19T22:59:06.816Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352023-12-19T22:59:26.736Z+ [[ 4 -gt 30 ]]
7362023-12-19T22:59:26.736Z+ sleep 1
7372023-12-19T22:59:27.742Z+ retry=5
7382023-12-19T22:59:27.777Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7392023-12-19T22:59:27.777Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7402023-12-19T22:59:27.777Zlocal v6:
7412023-12-19T22:59:27.777Zfd00:99::1
7422023-12-19T22:59:27.777Zfe80::f827:20ff:fec0:57d0
7432023-12-19T22:59:27.777Zlocal v4:
7442023-12-19T22:59:27.777Z10.151.2.174
7452023-12-19T22:59:27.777Zrouter v6:
7462023-12-19T22:59:27.777Zrouter v4_idx:
7472023-12-19T22:59:27.777Z0.0.0.0/0 -> 0
7482023-12-19T22:59:27.777Zrouter v4_routes:
7492023-12-19T22:59:27.777Z0 -> 10.151.2.100 (2)
7502023-12-19T22:59:27.777Zresolver v4:
7512023-12-19T22:59:27.777Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7522023-12-19T22:59:27.777Zresolver v6:
7532023-12-19T22:59:27.777Zfe80::f827:20ff:fec0:57d0 -> fa:27:20:c0:57:d0
7542023-12-19T22:59:27.778Zfe80::a8e7:50ff:fe8b:993 -> aa:e7:50:8b:09:93
7552023-12-19T22:59:27.778Znat v4:
7562023-12-19T22:59:27.778Znat_v6:
7572023-12-19T22:59:27.778Zport_mac:
7582023-12-19T22:59:27.778Z1000: a8:40:25:34:cd:2b
7592023-12-19T22:59:27.778Z1: a8:40:25:34:cd:2c
7602023-12-19T22:59:27.778Zproxy arp:
7612023-12-19T22:59:27.778Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7622023-12-19T22:59:27.815Z+ retry=0
7632023-12-19T22:59:27.815Z+++ zoneadm list -n
7642023-12-19T22:59:27.815Z+++ grep oxz_ntp
7652023-12-19T22:59:27.815Z++ pfexec svcs -z -Hostate oxide/ntp
7662023-12-19T22:59:27.815Zsvcs: zone '-Hostate': entity not found
7672023-12-19T22:59:27.815Z++ true
7682023-12-19T22:59:27.815Z+ [[ '' != online ]]
7692023-12-19T22:59:27.815Z+ [[ 0 -gt 60 ]]
7702023-12-19T22:59:27.815Z+ sleep 1
7712023-12-19T22:59:28.798Z+ retry=1
7722023-12-19T22:59:28.831Z+++ zoneadm list -n
7732023-12-19T22:59:28.831Z+++ grep oxz_ntp
7742023-12-19T22:59:28.831Z++ pfexec svcs -z -Hostate oxide/ntp
7752023-12-19T22:59:28.831Zsvcs: zone '-Hostate': entity not found
7762023-12-19T22:59:28.831Z++ true
7772023-12-19T22:59:28.832Z+ [[ '' != online ]]
7782023-12-19T22:59:28.832Z+ [[ 1 -gt 60 ]]
7792023-12-19T22:59:28.832Z+ sleep 1
7802023-12-19T22:59:29.817Z+ retry=2
7812023-12-19T22:59:29.852Z+++ zoneadm list -n
7822023-12-19T22:59:29.852Z+++ grep oxz_ntp
7832023-12-19T22:59:29.852Z++ pfexec svcs -z -Hostate oxide/ntp
7842023-12-19T22:59:29.852Zsvcs: zone '-Hostate': entity not found
7852023-12-19T22:59:29.852Z++ true
7862023-12-19T22:59:29.852Z+ [[ '' != online ]]
7872023-12-19T22:59:29.852Z+ [[ 2 -gt 60 ]]
7882023-12-19T22:59:29.852Z+ sleep 1
7892023-12-19T22:59:30.841Z+ retry=3
7902023-12-19T22:59:30.876Z+++ zoneadm list -n
7912023-12-19T22:59:30.876Z+++ grep oxz_ntp
7922023-12-19T22:59:30.876Z++ pfexec svcs -z -Hostate oxide/ntp
7932023-12-19T22:59:30.876Zsvcs: zone '-Hostate': entity not found
7942023-12-19T22:59:30.876Z++ true
7952023-12-19T22:59:30.876Z+ [[ '' != online ]]
7962023-12-19T22:59:30.876Z+ [[ 3 -gt 60 ]]
7972023-12-19T22:59:30.877Z+ sleep 1
7982023-12-19T22:59:31.865Z+ retry=4
7992023-12-19T22:59:31.899Z+++ zoneadm list -n
8002023-12-19T22:59:31.899Z+++ grep oxz_ntp
8012023-12-19T22:59:31.899Z++ pfexec svcs -z -Hostate oxide/ntp
8022023-12-19T22:59:31.899Zsvcs: zone '-Hostate': entity not found
8032023-12-19T22:59:31.899Z++ true
8042023-12-19T22:59:31.899Z+ [[ '' != online ]]
8052023-12-19T22:59:31.900Z+ [[ 4 -gt 60 ]]
8062023-12-19T22:59:31.900Z+ sleep 1
8072023-12-19T22:59:32.885Z+ retry=5
8082023-12-19T22:59:32.923Z+++ zoneadm list -n
8092023-12-19T22:59:32.923Z+++ grep oxz_ntp
8102023-12-19T22:59:32.923Z++ pfexec svcs -z oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce -Hostate oxide/ntp
8112023-12-19T22:59:32.923Zsvcs: Could not bind to repository server for zone oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce: repository server unavailable
8122023-12-19T22:59:32.923Z++ true
8132023-12-19T22:59:32.923Z+ [[ '' != online ]]
8142023-12-19T22:59:32.923Z+ [[ 5 -gt 60 ]]
8152023-12-19T22:59:32.923Z+ sleep 1
8162023-12-19T22:59:33.906Z+ retry=6
8172023-12-19T22:59:33.939Z+++ zoneadm list -n
8182023-12-19T22:59:33.939Z+++ grep oxz_ntp
8192023-12-19T22:59:33.939Z++ pfexec svcs -z oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce -Hostate oxide/ntp
8202023-12-19T22:59:33.939Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8212023-12-19T22:59:33.939Z++ true
8222023-12-19T22:59:33.939Z+ [[ '' != online ]]
8232023-12-19T22:59:33.939Z+ [[ 6 -gt 60 ]]
8242023-12-19T22:59:33.939Z+ sleep 1
8252023-12-19T22:59:34.942Z+ retry=7
8262023-12-19T22:59:34.980Z+++ zoneadm list -n
8272023-12-19T22:59:34.980Z+++ grep oxz_ntp
8282023-12-19T22:59:34.981Z++ pfexec svcs -z oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce -Hostate oxide/ntp
8292023-12-19T22:59:34.981Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8302023-12-19T22:59:34.981Z++ true
8312023-12-19T22:59:34.981Z+ [[ '' != online ]]
8322023-12-19T22:59:34.982Z+ [[ 7 -gt 60 ]]
8332023-12-19T22:59:34.982Z+ sleep 1
8342023-12-19T22:59:35.974Z+ retry=8
8352023-12-19T22:59:36.007Z+++ zoneadm list -n
8362023-12-19T22:59:36.007Z+++ grep oxz_ntp
8372023-12-19T22:59:36.007Z++ pfexec svcs -z oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce -Hostate oxide/ntp
8382023-12-19T22:59:36.007Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8392023-12-19T22:59:36.007Z++ true
8402023-12-19T22:59:36.007Z+ [[ '' != online ]]
8412023-12-19T22:59:36.008Z+ [[ 8 -gt 60 ]]
8422023-12-19T22:59:36.008Z+ sleep 1
8432023-12-19T22:59:37.002Z+ retry=9
8442023-12-19T22:59:37.035Z+++ zoneadm list -n
8452023-12-19T22:59:37.035Z+++ grep oxz_ntp
8462023-12-19T22:59:37.035Z++ pfexec svcs -z oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce -Hostate oxide/ntp
8472023-12-19T22:59:37.035Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8482023-12-19T22:59:37.035Z++ true
8492023-12-19T22:59:37.035Z+ [[ '' != online ]]
8502023-12-19T22:59:37.035Z+ [[ 9 -gt 60 ]]
8512023-12-19T22:59:37.035Z+ sleep 1
8522023-12-19T22:59:38.028Z+ retry=10
8532023-12-19T22:59:38.066Z+++ zoneadm list -n
8542023-12-19T22:59:38.066Z+++ grep oxz_ntp
8552023-12-19T22:59:38.066Z++ pfexec svcs -z oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce -Hostate oxide/ntp
8562023-12-19T22:59:38.066Z+ [[ disabled != online ]]
8572023-12-19T22:59:38.066Z+ [[ 10 -gt 60 ]]
8582023-12-19T22:59:38.066Z+ sleep 1
8592023-12-19T22:59:39.055Z+ retry=11
8602023-12-19T22:59:39.088Z+++ zoneadm list -n
8612023-12-19T22:59:39.088Z+++ grep oxz_ntp
8622023-12-19T22:59:39.088Z++ pfexec svcs -z oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce -Hostate oxide/ntp
8632023-12-19T22:59:39.088Z+ [[ disabled != online ]]
8642023-12-19T22:59:39.088Z+ [[ 11 -gt 60 ]]
8652023-12-19T22:59:39.088Z+ sleep 1
8662023-12-19T22:59:40.075Z+ retry=12
8672023-12-19T22:59:40.107Z+++ zoneadm list -n
8682023-12-19T22:59:40.107Z+++ grep oxz_ntp
8692023-12-19T22:59:40.107Z++ pfexec svcs -z oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce -Hostate oxide/ntp
8702023-12-19T22:59:40.107Z+ [[ disabled != online ]]
8712023-12-19T22:59:40.107Z+ [[ 12 -gt 60 ]]
8722023-12-19T22:59:40.107Z+ sleep 1
8732023-12-19T22:59:41.098Z+ retry=13
8742023-12-19T22:59:41.132Z+++ zoneadm list -n
8752023-12-19T22:59:41.132Z+++ grep oxz_ntp
8762023-12-19T22:59:41.132Z++ pfexec svcs -z oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce -Hostate oxide/ntp
8772023-12-19T22:59:41.132Z+ [[ disabled != online ]]
8782023-12-19T22:59:41.132Z+ [[ 13 -gt 60 ]]
8792023-12-19T22:59:41.132Z+ sleep 1
8802023-12-19T22:59:42.118Z+ retry=14
8812023-12-19T22:59:42.151Z+++ zoneadm list -n
8822023-12-19T22:59:42.151Z+++ grep oxz_ntp
8832023-12-19T22:59:42.151Z++ pfexec svcs -z oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce -Hostate oxide/ntp
8842023-12-19T22:59:42.151Z+ [[ disabled != online ]]
8852023-12-19T22:59:42.151Z+ [[ 14 -gt 60 ]]
8862023-12-19T22:59:42.151Z+ sleep 1
8872023-12-19T22:59:43.142Z+ retry=15
8882023-12-19T22:59:43.179Z+++ zoneadm list -n
8892023-12-19T22:59:43.179Z+++ grep oxz_ntp
8902023-12-19T22:59:43.179Z++ pfexec svcs -z oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce -Hostate oxide/ntp
8912023-12-19T22:59:43.179Z+ [[ disabled != online ]]
8922023-12-19T22:59:43.179Z+ [[ 15 -gt 60 ]]
8932023-12-19T22:59:43.179Z+ sleep 1
8942023-12-19T22:59:44.163Z+ retry=16
8952023-12-19T22:59:44.206Z+++ zoneadm list -n
8962023-12-19T22:59:44.206Z+++ grep oxz_ntp
8972023-12-19T22:59:44.206Z++ pfexec svcs -z oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce -Hostate oxide/ntp
8982023-12-19T22:59:44.208Z+ [[ disabled != online ]]
8992023-12-19T22:59:44.208Z+ [[ 16 -gt 60 ]]
9002023-12-19T22:59:44.208Z+ sleep 1
9012023-12-19T22:59:45.188Z+ retry=17
9022023-12-19T22:59:45.221Z+++ zoneadm list -n
9032023-12-19T22:59:45.221Z+++ grep oxz_ntp
9042023-12-19T22:59:45.221Z++ pfexec svcs -z oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce -Hostate oxide/ntp
9052023-12-19T22:59:45.221Z+ [[ disabled != online ]]
9062023-12-19T22:59:45.221Z+ [[ 17 -gt 60 ]]
9072023-12-19T22:59:45.221Z+ sleep 1
 
9082023-12-19T22:59:45.501Zsock console login: Dec 19 22:59:45 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9092023-12-19T22:59:46.208Z+ retry=18
9102023-12-19T22:59:46.250Z+++ zoneadm list -n
9112023-12-19T22:59:46.250Z+++ grep oxz_ntp
9122023-12-19T22:59:46.250Z++ pfexec svcs -z oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce -Hostate oxide/ntp
9132023-12-19T22:59:46.250Z+ [[ online != online ]]
9142023-12-19T22:59:46.250Z+ echo 'Waited for chrony: 18s'
9152023-12-19T22:59:46.250Z+ export RUST_BACKTRACE=1
9162023-12-19T22:59:46.250ZWaited for chrony: 18s
9172023-12-19T22:59:46.250Z+ RUST_BACKTRACE=1
9182023-12-19T22:59:46.250Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9192023-12-19T22:59:46.250Z++ ./tests/bootstrap
9202023-12-19T22:59:46.285ZUsing external DNS server at 10.151.2.164:53
9212023-12-19T22:59:46.285Z2023-12-19 22:59:46.342266166 UTC: attempting to log into API
9222023-12-19T23:00:01.302Z2023-12-19 23:00:01.352776037 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9232023-12-19T23:00:02.333Z2023-12-19 23:00:02.351072556 UTC: attempting to log into API
9242023-12-19T23:00:17.334Z2023-12-19 23:00:17.342752800 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9252023-12-19T23:00:18.335Z2023-12-19 23:00:18.340942517 UTC: attempting to log into API
9262023-12-19T23:00:33.364Z2023-12-19 23:00:33.348354178 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9272023-12-19T23:00:34.366Z2023-12-19 23:00:34.350143717 UTC: attempting to log into API
9282023-12-19T23:00:49.403Z2023-12-19 23:00:49.382493950 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9292023-12-19T23:00:50.399Z2023-12-19 23:00:50.383329246 UTC: attempting to log into API
9302023-12-19T23:01:05.432Z2023-12-19 23:01:05.416207394 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9312023-12-19T23:01:06.438Z2023-12-19 23:01:06.416955729 UTC: attempting to log into API
9322023-12-19T23:01:21.467Z2023-12-19 23:01:21.450423253 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9332023-12-19T23:01:22.469Z2023-12-19 23:01:22.451528523 UTC: attempting to log into API
9342023-12-19T23:01:37.509Z2023-12-19 23:01:37.491650339 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9352023-12-19T23:01:38.510Z2023-12-19 23:01:38.492851921 UTC: attempting to log into API
9362023-12-19T23:01:53.562Z2023-12-19 23:01:53.546589745 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9372023-12-19T23:01:54.568Z2023-12-19 23:01:54.547899598 UTC: attempting to log into API
9382023-12-19T23:02:09.618Z2023-12-19 23:02:09.598211276 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9392023-12-19T23:02:10.614Z2023-12-19 23:02:10.599291036 UTC: attempting to log into API
 
9402023-12-19T23:02:20.676ZDec 19 23:02:20 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9412023-12-19T23:02:25.656Z2023-12-19 23:02:25.633513512 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9422023-12-19T23:02:26.649Z2023-12-19 23:02:26.634563041 UTC: attempting to log into API
 
9432023-12-19T23:02:27.102ZDec 19 23:02:26 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9442023-12-19T23:02:34.437ZDec 19 23:02:33 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9452023-12-19T23:02:40.573ZDec 19 23:02:40 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9462023-12-19T23:02:41.681Z2023-12-19 23:02:41.666986212 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9472023-12-19T23:02:42.684Z2023-12-19 23:02:42.668719451 UTC: attempting to log into API
 
9482023-12-19T23:02:46.647ZDec 19 23:02:45 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9492023-12-19T23:02:57.738Z2023-12-19 23:02:57.701745291 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9502023-12-19T23:02:58.722Z2023-12-19 23:02:58.702731810 UTC: attempting to log into API
9512023-12-19T23:02:58.764Z2023-12-19 23:02:58.749555917 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: tcp connect error: Connection refused (os error 146)
9522023-12-19T23:02:59.768Z2023-12-19 23:02:59.750911101 UTC: attempting to log into API
9532023-12-19T23:03:00.547Z2023-12-19 23:03:00.532718417 UTC: login succeeded
9542023-12-19T23:03:00.586Zwaiting for nexus to come up...
9552023-12-19T23:03:00.650Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9562023-12-19T23:03:00.697Zsetting up quotas...
9572023-12-19T23:03:00.731Zensuring datasets are ready...
9582023-12-19T23:03:03.185Zlet's roll.
9592023-12-19T23:03:03.226Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
9602023-12-19T23:03:03.227ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9612023-12-19T23:03:03.227ZOXIDE_TOKEN=oxide-token-3f8e00710ac52125419f88171be4060f2cc512ef'
9622023-12-19T23:03:03.227Z++ OXIDE_HOST=https://recovery.sys.oxide.test
9632023-12-19T23:03:03.227Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9642023-12-19T23:03:03.227Z++ OXIDE_TOKEN=oxide-token-3f8e00710ac52125419f88171be4060f2cc512ef
9652023-12-19T23:03:03.227Z+ export OXIDE_HOST OXIDE_TOKEN
9662023-12-19T23:03:03.227Z+ retry=0
9672023-12-19T23:03:03.227Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9682023-12-19T23:03:03.227Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9692023-12-19T23:03:03.227Z+ [[ 0 -gt 60 ]]
9702023-12-19T23:03:03.227Z+ sleep 1
9712023-12-19T23:03:04.221Z+ retry=1
9722023-12-19T23:03:04.221Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9732023-12-19T23:03:04.256Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9742023-12-19T23:03:04.256Z+ [[ 1 -gt 60 ]]
9752023-12-19T23:03:04.256Z+ sleep 1
9762023-12-19T23:03:05.239Z+ retry=2
9772023-12-19T23:03:05.239Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9782023-12-19T23:03:05.273Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9792023-12-19T23:03:05.274Z+ [[ 2 -gt 60 ]]
9802023-12-19T23:03:05.274Z+ sleep 1
9812023-12-19T23:03:06.252Z+ retry=3
9822023-12-19T23:03:06.252Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9832023-12-19T23:03:06.291Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9842023-12-19T23:03:06.291Z+ [[ 3 -gt 60 ]]
9852023-12-19T23:03:06.291Z+ sleep 1
9862023-12-19T23:03:07.266Z+ retry=4
9872023-12-19T23:03:07.267Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9882023-12-19T23:03:07.302Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9892023-12-19T23:03:07.302Z+ [[ 4 -gt 60 ]]
9902023-12-19T23:03:07.303Z+ sleep 1
9912023-12-19T23:03:08.288Z+ retry=5
9922023-12-19T23:03:08.289Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9932023-12-19T23:03:08.322Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9942023-12-19T23:03:08.323Z+ [[ 5 -gt 60 ]]
9952023-12-19T23:03:08.323Z+ sleep 1
9962023-12-19T23:03:09.307Z+ retry=6
9972023-12-19T23:03:09.307Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9982023-12-19T23:03:09.347Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9992023-12-19T23:03:09.348Z+ [[ 6 -gt 60 ]]
10002023-12-19T23:03:09.348Z+ sleep 1
10012023-12-19T23:03:10.323Z+ retry=7
10022023-12-19T23:03:10.323Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10032023-12-19T23:03:10.357Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10042023-12-19T23:03:10.357Z+ [[ 7 -gt 60 ]]
10052023-12-19T23:03:10.357Z+ sleep 1
10062023-12-19T23:03:11.338Z+ retry=8
10072023-12-19T23:03:11.338Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10082023-12-19T23:03:11.382Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10092023-12-19T23:03:11.383Z+ [[ 8 -gt 60 ]]
10102023-12-19T23:03:11.383Z+ sleep 1
10112023-12-19T23:03:12.348Z+ retry=9
10122023-12-19T23:03:12.348Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10132023-12-19T23:03:12.387Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10142023-12-19T23:03:12.387Z+ [[ 9 -gt 60 ]]
10152023-12-19T23:03:12.387Z+ sleep 1
10162023-12-19T23:03:13.424Z+ retry=10
10172023-12-19T23:03:13.424Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10182023-12-19T23:03:13.460Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10192023-12-19T23:03:13.460Z+ [[ 10 -gt 60 ]]
10202023-12-19T23:03:13.460Z+ sleep 1
10212023-12-19T23:03:14.380Z+ retry=11
10222023-12-19T23:03:14.380Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10232023-12-19T23:03:14.414Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10242023-12-19T23:03:14.415Z+ [[ 11 -gt 60 ]]
10252023-12-19T23:03:14.415Z+ sleep 1
10262023-12-19T23:03:15.398Z+ retry=12
10272023-12-19T23:03:15.398Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10282023-12-19T23:03:15.437Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10292023-12-19T23:03:15.438Z+ [[ 12 -gt 60 ]]
10302023-12-19T23:03:15.439Z+ sleep 1
10312023-12-19T23:03:16.412Z+ retry=13
10322023-12-19T23:03:16.412Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10332023-12-19T23:03:16.446Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10342023-12-19T23:03:16.446Z+ [[ 13 -gt 60 ]]
10352023-12-19T23:03:16.447Z+ sleep 1
10362023-12-19T23:03:17.429Z+ retry=14
10372023-12-19T23:03:17.429Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10382023-12-19T23:03:17.465Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10392023-12-19T23:03:17.465Z+ [[ 14 -gt 60 ]]
10402023-12-19T23:03:17.465Z+ sleep 1
10412023-12-19T23:03:18.441Z+ retry=15
10422023-12-19T23:03:18.441Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10432023-12-19T23:03:18.476Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10442023-12-19T23:03:18.476Z+ [[ 15 -gt 60 ]]
10452023-12-19T23:03:18.476Z+ sleep 1
10462023-12-19T23:03:19.466Z+ retry=16
10472023-12-19T23:03:19.466Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10482023-12-19T23:03:19.505Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10492023-12-19T23:03:19.505Z+ [[ 16 -gt 60 ]]
10502023-12-19T23:03:19.505Z+ sleep 1
10512023-12-19T23:03:20.498Z+ retry=17
10522023-12-19T23:03:20.498Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10532023-12-19T23:03:20.536Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10542023-12-19T23:03:20.536Z+ [[ 17 -gt 60 ]]
10552023-12-19T23:03:20.536Z+ sleep 1
10562023-12-19T23:03:21.491Z+ retry=18
10572023-12-19T23:03:21.491Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10582023-12-19T23:03:21.530Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10592023-12-19T23:03:21.530Z+ [[ 18 -gt 60 ]]
10602023-12-19T23:03:21.530Z+ sleep 1
10612023-12-19T23:03:22.501Z+ retry=19
10622023-12-19T23:03:22.501Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10632023-12-19T23:03:22.562Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10642023-12-19T23:03:22.562Z+ [[ 19 -gt 60 ]]
10652023-12-19T23:03:22.562Z+ sleep 1
10662023-12-19T23:03:23.516Z+ retry=20
10672023-12-19T23:03:23.556Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10682023-12-19T23:03:23.556Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10692023-12-19T23:03:23.556Z+ [[ 20 -gt 60 ]]
10702023-12-19T23:03:23.556Z+ sleep 1
10712023-12-19T23:03:24.530Z+ retry=21
10722023-12-19T23:03:24.530Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10732023-12-19T23:03:24.570Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10742023-12-19T23:03:24.570Z+ [[ 21 -gt 60 ]]
10752023-12-19T23:03:24.570Z+ sleep 1
10762023-12-19T23:03:25.543Z+ retry=22
10772023-12-19T23:03:25.543Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10782023-12-19T23:03:25.582Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10792023-12-19T23:03:25.582Z+ [[ 22 -gt 60 ]]
10802023-12-19T23:03:25.582Z+ sleep 1
10812023-12-19T23:03:26.563Z+ retry=23
10822023-12-19T23:03:26.563Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10832023-12-19T23:03:26.597Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10842023-12-19T23:03:26.597Z+ [[ 23 -gt 60 ]]
10852023-12-19T23:03:26.597Z+ sleep 1
10862023-12-19T23:03:27.581Z+ retry=24
10872023-12-19T23:03:27.581Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10882023-12-19T23:03:27.615Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10892023-12-19T23:03:27.615Z+ [[ 24 -gt 60 ]]
10902023-12-19T23:03:27.615Z+ sleep 1
10912023-12-19T23:03:28.592Z+ retry=25
10922023-12-19T23:03:28.592Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10932023-12-19T23:03:28.627Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10942023-12-19T23:03:28.627Z+ [[ 25 -gt 60 ]]
10952023-12-19T23:03:28.627Z+ sleep 1
10962023-12-19T23:03:29.607Z+ retry=26
10972023-12-19T23:03:29.607Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10982023-12-19T23:03:29.642Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10992023-12-19T23:03:29.642Z+ [[ 26 -gt 60 ]]
11002023-12-19T23:03:29.642Z+ sleep 1
11012023-12-19T23:03:30.623Z+ retry=27
11022023-12-19T23:03:30.623Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11032023-12-19T23:03:30.659Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11042023-12-19T23:03:30.659Z+ [[ 27 -gt 60 ]]
11052023-12-19T23:03:30.659Z+ sleep 1
11062023-12-19T23:03:31.640Z+ retry=28
11072023-12-19T23:03:31.640Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11082023-12-19T23:03:31.685Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11092023-12-19T23:03:31.685Z+ [[ 28 -gt 60 ]]
11102023-12-19T23:03:31.685Z+ sleep 1
11112023-12-19T23:03:32.652Z+ retry=29
11122023-12-19T23:03:32.652Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11132023-12-19T23:03:32.686Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11142023-12-19T23:03:32.686Z+ [[ 29 -gt 60 ]]
11152023-12-19T23:03:32.686Z+ sleep 1
11162023-12-19T23:03:33.671Z+ retry=30
11172023-12-19T23:03:33.671Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11182023-12-19T23:03:33.705Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11192023-12-19T23:03:33.705Z+ [[ 30 -gt 60 ]]
11202023-12-19T23:03:33.706Z+ sleep 1
11212023-12-19T23:03:34.681Z+ retry=31
11222023-12-19T23:03:34.681Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11232023-12-19T23:03:34.722Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11242023-12-19T23:03:34.722Z+ [[ 31 -gt 60 ]]
11252023-12-19T23:03:34.722Z+ sleep 1
11262023-12-19T23:03:35.695Z+ retry=32
11272023-12-19T23:03:35.731Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11282023-12-19T23:03:35.731Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11292023-12-19T23:03:35.732Z+ [[ 32 -gt 60 ]]
11302023-12-19T23:03:35.732Z+ sleep 1
11312023-12-19T23:03:36.713Z+ retry=33
11322023-12-19T23:03:36.713Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11332023-12-19T23:03:36.746Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11342023-12-19T23:03:36.746Z+ [[ 33 -gt 60 ]]
11352023-12-19T23:03:36.746Z+ sleep 1
11362023-12-19T23:03:37.741Z+ retry=34
11372023-12-19T23:03:37.742Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11382023-12-19T23:03:37.778Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11392023-12-19T23:03:37.779Z+ [[ 34 -gt 60 ]]
11402023-12-19T23:03:37.779Z+ sleep 1
11412023-12-19T23:03:38.742Z+ retry=35
11422023-12-19T23:03:38.742Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11432023-12-19T23:03:38.775Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11442023-12-19T23:03:38.775Z+ [[ 35 -gt 60 ]]
11452023-12-19T23:03:38.775Z+ sleep 1
11462023-12-19T23:03:39.758Z+ retry=36
11472023-12-19T23:03:39.758Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11482023-12-19T23:03:39.794Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11492023-12-19T23:03:39.794Z+ [[ 36 -gt 60 ]]
11502023-12-19T23:03:39.794Z+ sleep 1
11512023-12-19T23:03:40.776Z+ retry=37
11522023-12-19T23:03:40.776Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11532023-12-19T23:03:40.812Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11542023-12-19T23:03:40.812Z+ [[ 37 -gt 60 ]]
11552023-12-19T23:03:40.812Z+ sleep 1
11562023-12-19T23:03:41.790Z+ retry=38
11572023-12-19T23:03:41.790Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11582023-12-19T23:03:41.826Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11592023-12-19T23:03:41.826Z+ [[ 38 -gt 60 ]]
11602023-12-19T23:03:41.826Z+ sleep 1
11612023-12-19T23:03:42.810Z+ retry=39
11622023-12-19T23:03:42.845Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11632023-12-19T23:03:42.845Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11642023-12-19T23:03:42.845Z+ [[ 39 -gt 60 ]]
11652023-12-19T23:03:42.845Z+ sleep 1
11662023-12-19T23:03:43.824Z+ retry=40
11672023-12-19T23:03:43.824Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11682023-12-19T23:03:43.859Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11692023-12-19T23:03:43.859Z+ [[ 40 -gt 60 ]]
11702023-12-19T23:03:43.859Z+ sleep 1
11712023-12-19T23:03:44.835Z+ retry=41
11722023-12-19T23:03:44.876Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11732023-12-19T23:03:44.877Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11742023-12-19T23:03:44.877Z+ [[ 41 -gt 60 ]]
11752023-12-19T23:03:44.877Z+ sleep 1
11762023-12-19T23:03:45.856Z+ retry=42
11772023-12-19T23:03:45.856Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11782023-12-19T23:03:45.892Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11792023-12-19T23:03:45.892Z+ [[ 42 -gt 60 ]]
11802023-12-19T23:03:45.892Z+ sleep 1
11812023-12-19T23:03:46.868Z+ retry=43
11822023-12-19T23:03:46.868Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11832023-12-19T23:03:46.901Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11842023-12-19T23:03:46.901Z+ [[ 43 -gt 60 ]]
11852023-12-19T23:03:46.901Z+ sleep 1
11862023-12-19T23:03:47.887Z+ retry=44
11872023-12-19T23:03:47.887Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11882023-12-19T23:03:47.960Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
11892023-12-19T23:03:48.444Z{"status":"ok"}success
11902023-12-19T23:03:48.444ZProject {
11912023-12-19T23:03:48.444Z description: "some images",
11922023-12-19T23:03:48.478Z id: 8ca6e0a6-b0ba-4b3b-b0d7-31115ca4e55e,
11932023-12-19T23:03:48.478Z name: Name(
11942023-12-19T23:03:48.478Z "images",
11952023-12-19T23:03:48.478Z ),
11962023-12-19T23:03:48.478Z time_created: 2023-12-19T23:03:48.067481Z,
11972023-12-19T23:03:48.478Z time_modified: 2023-12-19T23:03:48.067481Z,
11982023-12-19T23:03:48.478Z}
11992023-12-19T23:03:48.478Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
12002023-12-19T23:09:16.606Zdone!
12012023-12-19T23:09:16.705Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
12022023-12-19T23:09:16.800Zsuccess
12032023-12-19T23:09:16.800ZImage {
12042023-12-19T23:09:16.839Z block_size: ByteCount(
12052023-12-19T23:09:16.839Z 512,
12062023-12-19T23:09:16.839Z ),
12072023-12-19T23:09:16.839Z description: "debian 11 original base image",
12082023-12-19T23:09:16.839Z digest: None,
12092023-12-19T23:09:16.839Z id: 88e8f2c6-26e8-455a-a0fe-19d50c0a214a,
12102023-12-19T23:09:16.839Z name: Name(
12112023-12-19T23:09:16.839Z "debian11",
12122023-12-19T23:09:16.839Z ),
12132023-12-19T23:09:16.839Z os: "debian",
12142023-12-19T23:09:16.839Z project_id: None,
12152023-12-19T23:09:16.839Z size: ByteCount(
12162023-12-19T23:09:16.840Z 2147483648,
12172023-12-19T23:09:16.840Z ),
12182023-12-19T23:09:16.840Z time_created: 2023-12-19T23:09:16.541087Z,
12192023-12-19T23:09:16.840Z time_modified: 2023-12-19T23:09:16.776999Z,
12202023-12-19T23:09:16.840Z url: None,
12212023-12-19T23:09:16.840Z version: "11",
12222023-12-19T23:09:16.840Z}
12232023-12-19T23:09:16.840Z+ rm ./tests/bootstrap
12242023-12-19T23:09:16.890Z+ for test_bin in tests/*
12252023-12-19T23:09:16.890Z+ ./tests/bootstrap-8db5a4cac1990b57
12262023-12-19T23:09:16.925Z
12272023-12-19T23:09:16.926Zrunning 0 tests
12282023-12-19T23:09:16.926Z
12292023-12-19T23:09:16.927Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
12302023-12-19T23:09:16.927Z
12312023-12-19T23:09:16.927Z+ for test_bin in tests/*
12322023-12-19T23:09:16.927Z+ ./tests/end_to_end_tests-e4a60fbd4bd87582
12332023-12-19T23:09:16.990Z
12342023-12-19T23:09:16.990Zrunning 2 tests
12352023-12-19T23:09:17.214Ztest no_spoof::no_spoof_here ... ok
12362023-12-19T23:09:51.116Ztest instance_launch::instance_launch ... ok
12372023-12-19T23:09:51.117Z
12382023-12-19T23:09:51.117Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 34.13s
12392023-12-19T23:09:51.117Z
12402023-12-19T23:09:51.150Z+ _exit_trap
12412023-12-19T23:09:51.150Z+ local status=0
12422023-12-19T23:09:51.151Z+ [[ 0 -eq 0 ]]
12432023-12-19T23:09:51.151Z+ exit 0
12442023-12-19T23:09:51.151Zprocess exited: duration 704812 ms, exit code 0
 
12452023-12-19T23:09:51.295Zfound 60 output files
12462023-12-19T23:09:51.295Zuploading: /var/svc/log/oxide-sled-agent:default.log (757041 bytes)
12472023-12-19T23:09:54.345Zuploaded: /var/svc/log/oxide-sled-agent:default.log
12482023-12-19T23:09:54.345Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_b7ac2af6-17fd-4d90-8ddd-8a5571b8ab81/root/var/svc/log/oxide-clickhouse:default.log (883481 bytes)
12492023-12-19T23:09:56.549Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_b7ac2af6-17fd-4d90-8ddd-8a5571b8ab81/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 883481 -> 905381
12502023-12-19T23:09:57.657Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_b7ac2af6-17fd-4d90-8ddd-8a5571b8ab81/root/var/svc/log/oxide-clickhouse:default.log
12512023-12-19T23:09:57.657Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_d1bf2dad-cfa5-403e-a7fc-29d6ef199701/root/var/svc/log/oxide-cockroachdb:default.log (5413 bytes)
12522023-12-19T23:09:58.795Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_d1bf2dad-cfa5-403e-a7fc-29d6ef199701/root/var/svc/log/oxide-cockroachdb:default.log
12532023-12-19T23:09:58.795Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c274f29a-0e2f-47cb-aeb9-f4e6528f99fe/root/var/svc/log/oxide-crucible-agent:default.log (13684 bytes)
12542023-12-19T23:10:00.016Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c274f29a-0e2f-47cb-aeb9-f4e6528f99fe/root/var/svc/log/oxide-crucible-agent:default.log
12552023-12-19T23:10:00.016Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c274f29a-0e2f-47cb-aeb9-f4e6528f99fe/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12562023-12-19T23:10:01.168Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c274f29a-0e2f-47cb-aeb9-f4e6528f99fe/root/var/svc/log/oxide-crucible-downstairs:default.log
12572023-12-19T23:10:01.168Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c274f29a-0e2f-47cb-aeb9-f4e6528f99fe/root/var/svc/log/oxide-crucible-downstairs:downstairs-fc1fcd2a-241d-48a9-b8b7-b7a7928aa3ce.log (3051 bytes)
12582023-12-19T23:10:02.313Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c274f29a-0e2f-47cb-aeb9-f4e6528f99fe/root/var/svc/log/oxide-crucible-downstairs:downstairs-fc1fcd2a-241d-48a9-b8b7-b7a7928aa3ce.log
12592023-12-19T23:10:02.320Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d6e6ced2-9b95-4e22-89ed-a3457ea988e6/root/var/svc/log/oxide-crucible-agent:default.log (20043 bytes)
12602023-12-19T23:10:03.576Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d6e6ced2-9b95-4e22-89ed-a3457ea988e6/root/var/svc/log/oxide-crucible-agent:default.log
12612023-12-19T23:10:03.576Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d6e6ced2-9b95-4e22-89ed-a3457ea988e6/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12622023-12-19T23:10:04.738Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d6e6ced2-9b95-4e22-89ed-a3457ea988e6/root/var/svc/log/oxide-crucible-downstairs:default.log
12632023-12-19T23:10:04.738Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d6e6ced2-9b95-4e22-89ed-a3457ea988e6/root/var/svc/log/oxide-crucible-downstairs:downstairs-28455600-bcd6-4002-9a1d-774f0779b2d8.log (2945 bytes)
12642023-12-19T23:10:05.895Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d6e6ced2-9b95-4e22-89ed-a3457ea988e6/root/var/svc/log/oxide-crucible-downstairs:downstairs-28455600-bcd6-4002-9a1d-774f0779b2d8.log
12652023-12-19T23:10:05.899Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d6e6ced2-9b95-4e22-89ed-a3457ea988e6/root/var/svc/log/oxide-crucible-downstairs:downstairs-5df8d535-eab6-4472-9339-bf9b521d821e.log (5535 bytes)
12662023-12-19T23:10:07.073Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d6e6ced2-9b95-4e22-89ed-a3457ea988e6/root/var/svc/log/oxide-crucible-downstairs:downstairs-5df8d535-eab6-4472-9339-bf9b521d821e.log
12672023-12-19T23:10:07.073Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d98fe4cb-8603-4152-90ae-dfd37d4299f6/root/var/svc/log/oxide-crucible-agent:default.log (13685 bytes)
12682023-12-19T23:10:08.296Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d98fe4cb-8603-4152-90ae-dfd37d4299f6/root/var/svc/log/oxide-crucible-agent:default.log
12692023-12-19T23:10:08.297Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d98fe4cb-8603-4152-90ae-dfd37d4299f6/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12702023-12-19T23:10:09.448Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d98fe4cb-8603-4152-90ae-dfd37d4299f6/root/var/svc/log/oxide-crucible-downstairs:default.log
12712023-12-19T23:10:09.448Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d98fe4cb-8603-4152-90ae-dfd37d4299f6/root/var/svc/log/oxide-crucible-downstairs:downstairs-2a04a947-28df-4ca4-b51f-ee3e7aeae7f4.log (5531 bytes)
12722023-12-19T23:10:10.620Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d98fe4cb-8603-4152-90ae-dfd37d4299f6/root/var/svc/log/oxide-crucible-downstairs:downstairs-2a04a947-28df-4ca4-b51f-ee3e7aeae7f4.log
12732023-12-19T23:10:10.621Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1836c519-ef76-43dd-bdf3-9b7695cddf39/root/var/svc/log/oxide-crucible-agent:default.log (23474 bytes)
12742023-12-19T23:10:13.120Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1836c519-ef76-43dd-bdf3-9b7695cddf39/root/var/svc/log/oxide-crucible-agent:default.log
12752023-12-19T23:10:13.120Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1836c519-ef76-43dd-bdf3-9b7695cddf39/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12762023-12-19T23:10:14.256Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1836c519-ef76-43dd-bdf3-9b7695cddf39/root/var/svc/log/oxide-crucible-downstairs:default.log
12772023-12-19T23:10:14.257Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1836c519-ef76-43dd-bdf3-9b7695cddf39/root/var/svc/log/oxide-crucible-downstairs:downstairs-31477ee9-9334-4d4b-93a2-253b4be4f7a7.log (6436 bytes)
12782023-12-19T23:10:15.427Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1836c519-ef76-43dd-bdf3-9b7695cddf39/root/var/svc/log/oxide-crucible-downstairs:downstairs-31477ee9-9334-4d4b-93a2-253b4be4f7a7.log
12792023-12-19T23:10:15.427Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1836c519-ef76-43dd-bdf3-9b7695cddf39/root/var/svc/log/oxide-crucible-downstairs:downstairs-b51309bc-3a84-40bc-b5f1-d5574c439e94.log (2942 bytes)
12802023-12-19T23:10:16.570Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1836c519-ef76-43dd-bdf3-9b7695cddf39/root/var/svc/log/oxide-crucible-downstairs:downstairs-b51309bc-3a84-40bc-b5f1-d5574c439e94.log
12812023-12-19T23:10:16.571Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1836c519-ef76-43dd-bdf3-9b7695cddf39/root/var/svc/log/oxide-crucible-downstairs:snapshot-31477ee9-9334-4d4b-93a2-253b4be4f7a7-f52bac84-3c6e-4589-a377-3af5b7e2cbc8.log (5174 bytes)
12822023-12-19T23:10:17.698Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1836c519-ef76-43dd-bdf3-9b7695cddf39/root/var/svc/log/oxide-crucible-downstairs:snapshot-31477ee9-9334-4d4b-93a2-253b4be4f7a7-f52bac84-3c6e-4589-a377-3af5b7e2cbc8.log
12832023-12-19T23:10:17.698Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_520e66f4-c3b2-45a3-bc56-a4b8a15908f9/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
12842023-12-19T23:10:18.841Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_520e66f4-c3b2-45a3-bc56-a4b8a15908f9/root/var/svc/log/oxide-crucible-agent:default.log
12852023-12-19T23:10:18.841Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_520e66f4-c3b2-45a3-bc56-a4b8a15908f9/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12862023-12-19T23:10:19.988Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_520e66f4-c3b2-45a3-bc56-a4b8a15908f9/root/var/svc/log/oxide-crucible-downstairs:default.log
12872023-12-19T23:10:19.988Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_52cdc140-3588-4e5a-9e40-c66f3fc84e54/root/var/svc/log/oxide-crucible-agent:default.log (23908 bytes)
12882023-12-19T23:10:21.228Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_52cdc140-3588-4e5a-9e40-c66f3fc84e54/root/var/svc/log/oxide-crucible-agent:default.log
12892023-12-19T23:10:21.229Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_52cdc140-3588-4e5a-9e40-c66f3fc84e54/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12902023-12-19T23:10:22.448Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_52cdc140-3588-4e5a-9e40-c66f3fc84e54/root/var/svc/log/oxide-crucible-downstairs:default.log
12912023-12-19T23:10:22.448Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_52cdc140-3588-4e5a-9e40-c66f3fc84e54/root/var/svc/log/oxide-crucible-downstairs:downstairs-02b9389f-7b0d-45bf-a009-fc8230b907d7.log (6437 bytes)
12922023-12-19T23:10:24.145Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_52cdc140-3588-4e5a-9e40-c66f3fc84e54/root/var/svc/log/oxide-crucible-downstairs:downstairs-02b9389f-7b0d-45bf-a009-fc8230b907d7.log
12932023-12-19T23:10:24.145Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_52cdc140-3588-4e5a-9e40-c66f3fc84e54/root/var/svc/log/oxide-crucible-downstairs:downstairs-06af3dcc-80bd-4d17-9b52-6d9564a8c505.log (2944 bytes)
12942023-12-19T23:10:25.289Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_52cdc140-3588-4e5a-9e40-c66f3fc84e54/root/var/svc/log/oxide-crucible-downstairs:downstairs-06af3dcc-80bd-4d17-9b52-6d9564a8c505.log
12952023-12-19T23:10:25.289Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_52cdc140-3588-4e5a-9e40-c66f3fc84e54/root/var/svc/log/oxide-crucible-downstairs:snapshot-02b9389f-7b0d-45bf-a009-fc8230b907d7-f52bac84-3c6e-4589-a377-3af5b7e2cbc8.log (5174 bytes)
12962023-12-19T23:10:26.426Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_52cdc140-3588-4e5a-9e40-c66f3fc84e54/root/var/svc/log/oxide-crucible-downstairs:snapshot-02b9389f-7b0d-45bf-a009-fc8230b907d7-f52bac84-3c6e-4589-a377-3af5b7e2cbc8.log
12972023-12-19T23:10:26.426Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8ac3950d-a97f-436c-a7c4-6ec86cd2cdef/root/var/svc/log/oxide-crucible-agent:default.log (13686 bytes)
12982023-12-19T23:10:27.665Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8ac3950d-a97f-436c-a7c4-6ec86cd2cdef/root/var/svc/log/oxide-crucible-agent:default.log
12992023-12-19T23:10:27.703Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8ac3950d-a97f-436c-a7c4-6ec86cd2cdef/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
13002023-12-19T23:10:28.804Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8ac3950d-a97f-436c-a7c4-6ec86cd2cdef/root/var/svc/log/oxide-crucible-downstairs:default.log
13012023-12-19T23:10:28.804Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8ac3950d-a97f-436c-a7c4-6ec86cd2cdef/root/var/svc/log/oxide-crucible-downstairs:downstairs-d5d9c10f-9cc9-4157-9e6e-36c9cad95e40.log (3049 bytes)
13022023-12-19T23:10:29.947Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8ac3950d-a97f-436c-a7c4-6ec86cd2cdef/root/var/svc/log/oxide-crucible-downstairs:downstairs-d5d9c10f-9cc9-4157-9e6e-36c9cad95e40.log
13032023-12-19T23:10:29.947Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_18eb7385-4ed5-41f8-9b37-97eb0466d487/root/var/svc/log/oxide-internal_dns:default.log (108568 bytes)
13042023-12-19T23:10:31.512Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_18eb7385-4ed5-41f8-9b37-97eb0466d487/root/var/svc/log/oxide-internal_dns:default.log
13052023-12-19T23:10:31.513Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_18eb7385-4ed5-41f8-9b37-97eb0466d487/root/var/svc/log/oxide-internal_dns:default.log.0 (9157 bytes)
13062023-12-19T23:10:32.686Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_18eb7385-4ed5-41f8-9b37-97eb0466d487/root/var/svc/log/oxide-internal_dns:default.log.0
13072023-12-19T23:10:32.686Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_38b01531-4202-4bd1-aa60-b1a0d98599e3/root/var/svc/log/oxide-nexus:default.log (1167449 bytes)
13082023-12-19T23:10:34.666Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_38b01531-4202-4bd1-aa60-b1a0d98599e3/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1167449 -> 1176728
13092023-12-19T23:10:35.766Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_38b01531-4202-4bd1-aa60-b1a0d98599e3/root/var/svc/log/oxide-nexus:default.log
13102023-12-19T23:10:35.766Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_715f5168-1120-43d3-afff-21f89d0bbaa1/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
13112023-12-19T23:10:36.910Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_715f5168-1120-43d3-afff-21f89d0bbaa1/root/var/svc/log/oxide-crucible-pantry:default.log
13122023-12-19T23:10:36.910Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_84fda2b6-423e-4cf4-9244-a7408c62a861/root/var/svc/log/oxide-nexus:default.log (34110083 bytes)
13132023-12-19T23:10:59.600Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_84fda2b6-423e-4cf4-9244-a7408c62a861/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 34110083 -> 34182642
13142023-12-19T23:11:00.714Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_84fda2b6-423e-4cf4-9244-a7408c62a861/root/var/svc/log/oxide-nexus:default.log
13152023-12-19T23:11:00.714Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_8661716b-9deb-496e-a80f-9e7183d7d40c/root/var/svc/log/oxide-cockroachdb:default.log (5414 bytes)
13162023-12-19T23:11:02.010Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_8661716b-9deb-496e-a80f-9e7183d7d40c/root/var/svc/log/oxide-cockroachdb:default.log
13172023-12-19T23:11:02.010Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_b3aefdf0-ff4a-4bda-9cf3-a9252914fcd4/root/var/svc/log/oxide-cockroachdb:default.log (5414 bytes)
13182023-12-19T23:11:03.156Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_b3aefdf0-ff4a-4bda-9cf3-a9252914fcd4/root/var/svc/log/oxide-cockroachdb:default.log
13192023-12-19T23:11:03.156Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d47ed049-99da-499f-add0-332af0fbd8d5/root/var/svc/log/oxide-crucible-agent:default.log (14118 bytes)
13202023-12-19T23:11:04.327Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d47ed049-99da-499f-add0-332af0fbd8d5/root/var/svc/log/oxide-crucible-agent:default.log
13212023-12-19T23:11:04.327Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d47ed049-99da-499f-add0-332af0fbd8d5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
13222023-12-19T23:11:05.464Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d47ed049-99da-499f-add0-332af0fbd8d5/root/var/svc/log/oxide-crucible-downstairs:default.log
13232023-12-19T23:11:05.464Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d47ed049-99da-499f-add0-332af0fbd8d5/root/var/svc/log/oxide-crucible-downstairs:downstairs-a841cde7-bd57-4e3b-b423-2734f1855af1.log (3050 bytes)
13242023-12-19T23:11:06.602Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d47ed049-99da-499f-add0-332af0fbd8d5/root/var/svc/log/oxide-crucible-downstairs:downstairs-a841cde7-bd57-4e3b-b423-2734f1855af1.log
13252023-12-19T23:11:06.602Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_eb324745-26bc-4e5e-95d6-0812a97a4706/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
13262023-12-19T23:11:07.766Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_eb324745-26bc-4e5e-95d6-0812a97a4706/root/var/svc/log/oxide-crucible-pantry:default.log
13272023-12-19T23:11:07.766Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_27b27166-2734-4409-b6d8-9557b3956118/root/var/svc/log/oxide-cockroachdb:default.log (5387 bytes)
13282023-12-19T23:11:08.890Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_27b27166-2734-4409-b6d8-9557b3956118/root/var/svc/log/oxide-cockroachdb:default.log
13292023-12-19T23:11:08.891Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_7fbbf0b3-3503-4308-b3c8-b7f429219325/root/var/svc/log/oxide-crucible-agent:default.log (27803 bytes)
13302023-12-19T23:11:10.106Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_7fbbf0b3-3503-4308-b3c8-b7f429219325/root/var/svc/log/oxide-crucible-agent:default.log
13312023-12-19T23:11:10.106Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_7fbbf0b3-3503-4308-b3c8-b7f429219325/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
13322023-12-19T23:11:11.261Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_7fbbf0b3-3503-4308-b3c8-b7f429219325/root/var/svc/log/oxide-crucible-downstairs:default.log
13332023-12-19T23:11:11.261Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_7fbbf0b3-3503-4308-b3c8-b7f429219325/root/var/svc/log/oxide-crucible-downstairs:downstairs-6d063c65-4c9c-46ab-bb56-111af27dd7b1.log (6433 bytes)
13342023-12-19T23:11:12.408Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_7fbbf0b3-3503-4308-b3c8-b7f429219325/root/var/svc/log/oxide-crucible-downstairs:downstairs-6d063c65-4c9c-46ab-bb56-111af27dd7b1.log
13352023-12-19T23:11:12.408Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_7fbbf0b3-3503-4308-b3c8-b7f429219325/root/var/svc/log/oxide-crucible-downstairs:downstairs-854ce8d1-3d8d-42fb-b37f-50ffe42a9688.log (5535 bytes)
13362023-12-19T23:11:13.548Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_7fbbf0b3-3503-4308-b3c8-b7f429219325/root/var/svc/log/oxide-crucible-downstairs:downstairs-854ce8d1-3d8d-42fb-b37f-50ffe42a9688.log
13372023-12-19T23:11:13.548Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_7fbbf0b3-3503-4308-b3c8-b7f429219325/root/var/svc/log/oxide-crucible-downstairs:snapshot-6d063c65-4c9c-46ab-bb56-111af27dd7b1-f52bac84-3c6e-4589-a377-3af5b7e2cbc8.log (5175 bytes)
13382023-12-19T23:11:14.694Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_7fbbf0b3-3503-4308-b3c8-b7f429219325/root/var/svc/log/oxide-crucible-downstairs:snapshot-6d063c65-4c9c-46ab-bb56-111af27dd7b1-f52bac84-3c6e-4589-a377-3af5b7e2cbc8.log
13392023-12-19T23:11:14.694Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_22feef1b-3641-4f5b-812a-6e074694eb5c/root/var/svc/log/oxide-nexus:default.log (2362024 bytes)
13402023-12-19T23:11:15.961Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_22feef1b-3641-4f5b-812a-6e074694eb5c/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 2362024 -> 2435208
13412023-12-19T23:11:17.072Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_22feef1b-3641-4f5b-812a-6e074694eb5c/root/var/svc/log/oxide-nexus:default.log
13422023-12-19T23:11:17.072Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_93d69613-245d-4c9d-bfae-666f10c736c7/root/var/svc/log/oxide-external_dns:default.log (30717 bytes)
13432023-12-19T23:11:17.247Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_93d69613-245d-4c9d-bfae-666f10c736c7/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 30717 -> 34492
13442023-12-19T23:11:18.358Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_93d69613-245d-4c9d-bfae-666f10c736c7/root/var/svc/log/oxide-external_dns:default.log
13452023-12-19T23:11:18.358Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_d7bbc3bb-0141-4d4b-a0e2-3d6705e0483e/root/var/svc/log/oxide-external_dns:default.log (30717 bytes)
13462023-12-19T23:11:18.537Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_d7bbc3bb-0141-4d4b-a0e2-3d6705e0483e/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 30717 -> 34489
13472023-12-19T23:11:19.651Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_d7bbc3bb-0141-4d4b-a0e2-3d6705e0483e/root/var/svc/log/oxide-external_dns:default.log
13482023-12-19T23:11:19.652Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_1bd57612-2d1b-4a6a-8ec9-eb1ee2bf2ed4/root/var/svc/log/oxide-internal_dns:default.log (108580 bytes)
13492023-12-19T23:11:19.906Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_1bd57612-2d1b-4a6a-8ec9-eb1ee2bf2ed4/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 108580 -> 112334
13502023-12-19T23:11:21.012Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_1bd57612-2d1b-4a6a-8ec9-eb1ee2bf2ed4/root/var/svc/log/oxide-internal_dns:default.log
13512023-12-19T23:11:21.012Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_1bd57612-2d1b-4a6a-8ec9-eb1ee2bf2ed4/root/var/svc/log/oxide-internal_dns:default.log.0 (15660 bytes)
13522023-12-19T23:11:22.230Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_1bd57612-2d1b-4a6a-8ec9-eb1ee2bf2ed4/root/var/svc/log/oxide-internal_dns:default.log.0
13532023-12-19T23:11:22.230Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a2b50530-cea8-44ca-9039-1c21722dae22/root/var/svc/log/oxide-cockroachdb:default.log (5413 bytes)
13542023-12-19T23:11:23.368Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a2b50530-cea8-44ca-9039-1c21722dae22/root/var/svc/log/oxide-cockroachdb:default.log
13552023-12-19T23:11:23.368Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_082f20c1-e817-474e-960c-34ee887924b4/root/var/svc/log/oxide-crucible-pantry:default.log (1228102 bytes)
13562023-12-19T23:11:25.497Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_082f20c1-e817-474e-960c-34ee887924b4/root/var/svc/log/oxide-crucible-pantry:default.log
13572023-12-19T23:11:25.497Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_23f90f02-1967-4f0c-905f-8557297e61a6/root/var/svc/log/oxide-internal_dns:default.log (108579 bytes)
13582023-12-19T23:11:25.759Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_23f90f02-1967-4f0c-905f-8557297e61a6/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 108579 -> 112334
13592023-12-19T23:11:26.860Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_23f90f02-1967-4f0c-905f-8557297e61a6/root/var/svc/log/oxide-internal_dns:default.log
13602023-12-19T23:11:26.860Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_23f90f02-1967-4f0c-905f-8557297e61a6/root/var/svc/log/oxide-internal_dns:default.log.0 (11608 bytes)
13612023-12-19T23:11:28.039Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_23f90f02-1967-4f0c-905f-8557297e61a6/root/var/svc/log/oxide-internal_dns:default.log.0
13622023-12-19T23:11:28.039Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce/root/var/svc/log/oxide-ntp:default.log (70 bytes)
13632023-12-19T23:11:29.180Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce/root/var/svc/log/oxide-ntp:default.log
13642023-12-19T23:11:29.180Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce/root/var/svc/log/oxide-ntp:default.log.0 (783 bytes)
13652023-12-19T23:11:30.323Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce/root/var/svc/log/oxide-ntp:default.log.0
13662023-12-19T23:11:30.324Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_5c0cc372-d135-4cad-85c1-9d822b358bba/root/var/svc/log/oxide-oximeter:default.log (273591 bytes)
13672023-12-19T23:11:30.675Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_5c0cc372-d135-4cad-85c1-9d822b358bba/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 273591 -> 334655
13682023-12-19T23:11:31.778Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_5c0cc372-d135-4cad-85c1-9d822b358bba/root/var/svc/log/oxide-oximeter:default.log
13692023-12-19T23:11:31.817Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce/root/var/log/chrony/measurements.log (18495 bytes)
13702023-12-19T23:11:31.948Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce/root/var/log/chrony/measurements.log" changed size mid upload: 18495 -> 21509
13712023-12-19T23:11:33.048Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce/root/var/log/chrony/measurements.log
13722023-12-19T23:11:33.048Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce/root/var/log/chrony/statistics.log (11700 bytes)
13732023-12-19T23:11:33.163Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce/root/var/log/chrony/statistics.log" changed size mid upload: 11700 -> 13923
13742023-12-19T23:11:34.265Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce/root/var/log/chrony/statistics.log
13752023-12-19T23:11:34.265Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce/root/var/log/chrony/tracking.log (1983 bytes)
13762023-12-19T23:11:34.337Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce/root/var/log/chrony/tracking.log" changed size mid upload: 1983 -> 2115
13772023-12-19T23:11:35.430Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5e413d38-1439-408f-bda4-4b126836d8ce/root/var/log/chrony/tracking.log