01HPR522FF8NGT83N8MJ5624R9: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.28",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HPR55Z7WYA52NKG7BNSHVJQ2

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12024-02-16T05:51:14.313Zjob dependencies complete; ready to run (waiting for 40 m 29 s)
22024-02-16T06:46:13.760Zjob assigned to worker 01HPRABMPYBR5CGTCK4NA36CWH (queued for 54 m 59 s)
32024-02-16T06:46:14.150Zhttp://172.20.0.5:9969/ipxe/sock... ok
42024-02-16T06:46:14.619Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52024-02-16T06:46:15.081Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... 21%Error: No response to keepalive - Terminating session
62024-02-16T06:46:15.127ZError: No response de-activating SOL payload
72024-02-16T06:46:15.162Z[SOL Session operational. Use ~? for help]
82024-02-16T06:46:15.204Z ok
92024-02-16T06:46:15.247Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
102024-02-16T06:46:15.290Zhttp://172.20.0.5:9969/postboot/sock... ok
112024-02-16T06:46:15.339ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
122024-02-16T06:46:15.895Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
132024-02-16T06:46:15.943ZOxide Helios Version helios-2.0.22451 64-bit
142024-02-16T06:46:16.953Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-22e122cc756fa22c.gz
152024-02-16T06:46:26.141Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-22e122cc756fa22c.gz
162024-02-16T06:46:26.141Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
172024-02-16T06:46:47.513ZError: No response to keepalive - Terminating session
182024-02-16T06:47:03.352ZError: No response de-activating SOL payload
192024-02-16T06:47:03.471Z[SOL Session operational. Use ~? for help]
202024-02-16T06:47:04.554Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
212024-02-16T06:47:04.604Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-4eb1211856b7c67f.gz
222024-02-16T06:48:04.278Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-4eb1211856b7c67f.gz
232024-02-16T06:48:04.278Zdownloading input: /input/ci-tools/work/caboose-util.gz
242024-02-16T06:48:07.321Zdownloaded input: /input/ci-tools/work/caboose-util.gz
252024-02-16T06:48:07.805Zdownloading input: /input/ci-tools/work/tufaceous.gz
262024-02-16T06:48:17.744Zdownloaded input: /input/ci-tools/work/tufaceous.gz
272024-02-16T06:48:17.772Zdownloading input: /input/package/work/version.txt
282024-02-16T06:48:17.869Zdownloaded input: /input/package/work/version.txt
292024-02-16T06:48:17.898Zdownloading input: /input/package/work/package.tar.gz
302024-02-16T06:49:09.858Zdownloaded input: /input/package/work/package.tar.gz
312024-02-16T06:49:09.858Zdownloading input: /input/package/work/global-zone-packages.tar.gz
322024-02-16T06:49:57.646ZError: No response to keepalive - Terminating session
332024-02-16T06:50:07.821ZError: No response de-activating SOL payload
342024-02-16T06:50:09.204Z[SOL Session operational. Use ~? for help]
352024-02-16T06:50:13.244Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
362024-02-16T06:50:13.244Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
372024-02-16T06:50:34.509Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
382024-02-16T06:50:34.565Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
392024-02-16T06:52:42.844Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
402024-02-16T06:52:42.870Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
412024-02-16T06:52:57.904ZError: No response to keepalive - Terminating session
422024-02-16T06:53:07.834ZError: No response de-activating SOL payload
432024-02-16T06:53:09.063Z[SOL Session operational. Use ~? for help]
442024-02-16T06:54:13.495Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452024-02-16T06:54:13.565Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462024-02-16T06:55:28.018Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472024-02-16T06:55:28.019Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482024-02-16T06:55:33.791Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492024-02-16T06:55:33.791Zdownloading input: /input/package/work/zones/crucible.tar.gz
502024-02-16T06:55:43.303Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512024-02-16T06:55:43.304Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522024-02-16T06:55:57.729ZError: No response to keepalive - Terminating session
532024-02-16T06:56:02.641Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
542024-02-16T06:56:02.682Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
552024-02-16T06:56:07.695ZError: No response de-activating SOL payload
562024-02-16T06:56:09.493Z[SOL Session operational. Use ~? for help]
572024-02-16T06:56:11.532Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
582024-02-16T06:56:11.532Zdownloading input: /input/package/work/zones/ntp.tar.gz
592024-02-16T06:56:11.582Zdownloaded input: /input/package/work/zones/ntp.tar.gz
602024-02-16T06:56:11.636Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
612024-02-16T06:56:15.526Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
622024-02-16T06:56:15.526Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
632024-02-16T06:56:24.546Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
642024-02-16T06:56:24.546Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
652024-02-16T06:56:41.707Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
662024-02-16T06:56:41.707Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
672024-02-16T06:57:08.503Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
682024-02-16T06:57:08.503Zdownloading input: /input/package/work/zones/overlay.tar.gz
692024-02-16T06:57:08.592Zdownloaded input: /input/package/work/zones/overlay.tar.gz
702024-02-16T06:57:08.592Zdownloading input: /input/package/work/zones/oximeter.tar.gz
712024-02-16T06:57:25.818Zdownloaded input: /input/package/work/zones/oximeter.tar.gz
722024-02-16T06:57:25.818Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
732024-02-16T06:57:44.593Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
742024-02-16T06:57:44.593Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
752024-02-16T06:58:40.724Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
762024-02-16T06:58:40.724Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
772024-02-16T06:59:13.783ZError: No response to keepalive - Terminating session
782024-02-16T06:59:23.749ZError: No response de-activating SOL payload
792024-02-16T06:59:25.145Z[SOL Session operational. Use ~? for help]
802024-02-16T06:59:30.278Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
812024-02-16T06:59:30.304Zstarting task 0: "setup"
822024-02-16T06:59:30.304Z++ uname -s
832024-02-16T06:59:30.304Z+ kern=SunOS
842024-02-16T06:59:30.305Z+ case "$kern" in
852024-02-16T06:59:30.305Z+ groupadd -g 12345 build
862024-02-16T06:59:30.305Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
872024-02-16T06:59:32.297Z+ zfs create -o mountpoint=/work rpool/work
882024-02-16T06:59:32.333Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
892024-02-16T06:59:32.359Z+ home_fs=zfs
902024-02-16T06:59:32.359Z+ [[ zfs == autofs ]]
912024-02-16T06:59:32.359Z+ mkdir -p /home/build
922024-02-16T06:59:32.359Z+ chown build:build /home/build /work
932024-02-16T06:59:34.341Z+ chmod 0700 /home/build /work
942024-02-16T06:59:34.367Zprocess exited: duration 4065 ms, exit code 0
 
952024-02-16T06:59:34.420Zstarting task 1: "authentication"
962024-02-16T06:59:34.465Zprocess exited: duration 44 ms, exit code 0
 
972024-02-16T06:59:34.516Zstarting task 2: "build"
982024-02-16T06:59:34.541Z+ trap _exit_trap EXIT
992024-02-16T06:59:34.541Z+ set +x
1002024-02-16T06:59:59.685Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
1012024-02-16T06:59:59.937Zsecpolicy_fs_linkdir: 0x55 = 0x55
1022024-02-16T06:59:59.938Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
1032024-02-16T06:59:59.938Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
1042024-02-16T06:59:59.938Zsecpolicy_fs_linkdir+3: 0 = 0xe5
1052024-02-16T06:59:59.938Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1062024-02-16T06:59:59.938Zsecpolicy_fs_linkdir+5: 0 = 0x1
1072024-02-16T06:59:59.938Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1082024-02-16T06:59:59.938Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1092024-02-16T06:59:59.964Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1102024-02-16T06:59:59.964Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1112024-02-16T06:59:59.965Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1122024-02-16T06:59:59.965Z+ [[ -d /opt/oxide ]]
1132024-02-16T06:59:59.965Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1142024-02-16T06:59:59.965Z+ pfexec mkdir /opt/oxide
1152024-02-16T06:59:59.965Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1162024-02-16T06:59:59.965Z+ [[ -d /opt/oxide-underneath ]]
1172024-02-16T06:59:59.965Z+ cd /opt/oxide-underneath
1182024-02-16T06:59:59.965Z+ pfexec tar ceEp@/f - .
1192024-02-16T06:59:59.965Z+ cd /opt/oxide
1202024-02-16T06:59:59.966Z+ pfexec tar xveEp@/f -
1212024-02-16T06:59:59.996Zx ., 0 bytes, 0 tape blocks
1222024-02-16T06:59:59.996Zx ./opte, 0 bytes, 0 tape blocks
1232024-02-16T06:59:59.997Zx ./opte/bin, 0 bytes, 0 tape blocks
1242024-02-16T06:59:59.997Zx ./opte/bin/opteadm, 51799224 bytes, 101171 tape blocks
1252024-02-16T07:00:00.023Z+ pfexec rm -rf /opt/oxide-underneath
1262024-02-16T07:00:00.023Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1272024-02-16T07:00:00.049Z+ pfexec mkdir /opt/oxide/work
1282024-02-16T07:00:00.049Z+ pfexec chown build:build /opt/oxide/work
1292024-02-16T07:00:00.049Z+ cd /opt/oxide/work
1302024-02-16T07:00:00.049Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1312024-02-16T07:00:01.752ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1322024-02-16T07:00:01.752Zx out/mg-ddm-gz.tar, 69308416 bytes, 135368 tape blocks
1332024-02-16T07:00:01.752Zx out/omicron-sled-agent.tar, 122944512 bytes, 240126 tape blocks
1342024-02-16T07:00:01.752Zx out/oxlog.tar, 8382464 bytes, 16372 tape blocks
1352024-02-16T07:00:01.753Zx out/target/test, 75 bytes, 1 tape blocks
1362024-02-16T07:00:01.753Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1372024-02-16T07:00:01.753Zx package-manifest.toml, 23732 bytes, 47 tape blocks
1382024-02-16T07:00:01.753Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1392024-02-16T07:00:01.753Zx target/release/omicron-package, 45826752 bytes, 89506 tape blocks
1402024-02-16T07:00:01.753Zx tools/create_virtual_hardware.sh, 3184 bytes, 7 tape blocks
1412024-02-16T07:00:01.753Zx tools/virtual_hardware.sh, 3416 bytes, 7 tape blocks
1422024-02-16T07:00:01.754Zx tools/scrimlet/create-softnpu-zone.sh, 627 bytes, 2 tape blocks
1432024-02-16T07:00:01.754Zx tools/scrimlet/destroy-softnpu-zone.sh, 203 bytes, 1 tape blocks
1442024-02-16T07:00:01.754Zx tools/scrimlet/softnpu-init.sh, 2157 bytes, 5 tape blocks
1452024-02-16T07:00:01.754Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1462024-02-16T07:00:01.754Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1472024-02-16T07:00:01.781Z
1482024-02-16T07:00:01.781Zreal 1.710531143
1492024-02-16T07:00:01.782Zuser 1.540470645
1502024-02-16T07:00:01.782Zsys 0.301438704
1512024-02-16T07:00:01.782Ztrap 0.000066162
1522024-02-16T07:00:01.782Ztflt 0.000000000
1532024-02-16T07:00:01.782Zdflt 0.000000000
1542024-02-16T07:00:01.782Zkflt 0.000000000
1552024-02-16T07:00:01.782Zlock 0.000000000
1562024-02-16T07:00:01.783Zslp 1.540717664
1572024-02-16T07:00:01.783Zlat 0.036463608
1582024-02-16T07:00:01.783Zstop 0.000120983
1592024-02-16T07:00:01.783Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1602024-02-16T07:00:02.974Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1612024-02-16T07:00:03.013Z+ mkdir tests
1622024-02-16T07:00:03.013Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1632024-02-16T07:00:03.013Z+ ptime -m gunzip
1642024-02-16T07:00:03.014Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-22e122cc756fa22c
1652024-02-16T07:00:03.233Z
1662024-02-16T07:00:03.259Zreal 0.231944448
1672024-02-16T07:00:03.260Zuser 0.213737862
1682024-02-16T07:00:03.260Zsys 0.017942699
1692024-02-16T07:00:03.260Ztrap 0.000019369
1702024-02-16T07:00:03.260Ztflt 0.000000000
1712024-02-16T07:00:03.261Zdflt 0.000000000
1722024-02-16T07:00:03.261Zkflt 0.000000000
1732024-02-16T07:00:03.261Zlock 0.000000000
1742024-02-16T07:00:03.261Zslp 0.000020020
1752024-02-16T07:00:03.261Zlat 0.000021270
1762024-02-16T07:00:03.261Zstop 0.000129113
1772024-02-16T07:00:03.261Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-22e122cc756fa22c
1782024-02-16T07:00:03.262Z+ chmod a+x tests/bootstrap-22e122cc756fa22c
1792024-02-16T07:00:03.262Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1802024-02-16T07:00:03.262Z+ ptime -m gunzip
1812024-02-16T07:00:03.262Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1822024-02-16T07:00:05.737Z
1832024-02-16T07:00:05.764Zreal 2.492645817
1842024-02-16T07:00:05.764Zuser 2.347784649
1852024-02-16T07:00:05.764Zsys 0.144467027
1862024-02-16T07:00:05.765Ztrap 0.000171625
1872024-02-16T07:00:05.765Ztflt 0.000000000
1882024-02-16T07:00:05.765Zdflt 0.000000000
1892024-02-16T07:00:05.765Zkflt 0.000000000
1902024-02-16T07:00:05.765Zlock 0.000000000
1912024-02-16T07:00:05.765Zslp 0.000020450
1922024-02-16T07:00:05.765Zlat 0.000015950
1932024-02-16T07:00:05.766Zstop 0.000120992
1942024-02-16T07:00:05.766Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1952024-02-16T07:00:05.766Z+ chmod a+x tests/bootstrap
1962024-02-16T07:00:05.766Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1972024-02-16T07:00:05.766Z+ ptime -m gunzip
1982024-02-16T07:00:05.766Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-4eb1211856b7c67f
1992024-02-16T07:00:08.304Z
2002024-02-16T07:00:08.331Zreal 2.554396008
2012024-02-16T07:00:08.331Zuser 2.409642040
2022024-02-16T07:00:08.331Zsys 0.144350035
2032024-02-16T07:00:08.331Ztrap 0.000170607
2042024-02-16T07:00:08.332Ztflt 0.000000000
2052024-02-16T07:00:08.332Zdflt 0.000000000
2062024-02-16T07:00:08.332Zkflt 0.000000000
2072024-02-16T07:00:08.332Zlock 0.000000000
2082024-02-16T07:00:08.332Zslp 0.000023860
2092024-02-16T07:00:08.332Zlat 0.000021500
2102024-02-16T07:00:08.333Zstop 0.000122472
2112024-02-16T07:00:08.333Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-4eb1211856b7c67f
2122024-02-16T07:00:08.333Z+ chmod a+x tests/end_to_end_tests-4eb1211856b7c67f
2132024-02-16T07:00:08.333Z+ bmat address ls
2142024-02-16T07:00:08.333ZNAME CIDR FIRST LAST COUNT
2152024-02-16T07:00:08.333Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2162024-02-16T07:00:08.333Z++ bmat address ls -f extra -Ho first,last,count
2172024-02-16T07:00:08.333Z+ set -- 10.151.2.164 10.151.2.227 64
2182024-02-16T07:00:08.333Z+ EXTRA_IP_START=10.151.2.164
2192024-02-16T07:00:08.334Z+ EXTRA_IP_END=10.151.2.227
2202024-02-16T07:00:08.334Z+ EXTRA_IP_COUNT=64
2212024-02-16T07:00:08.334Z+ (( EXTRA_IP_COUNT >= 32 ))
2222024-02-16T07:00:08.334Z+ EXTRA_IP_BASE=10.151.2
2232024-02-16T07:00:08.334Z+ EXTRA_IP_FOCTET=164
2242024-02-16T07:00:08.334Z+ EXTRA_IP_LOCTET=227
2252024-02-16T07:00:08.334Z+ SERVICE_IP_POOL_START=10.151.2.164
2262024-02-16T07:00:08.335Z+ SERVICE_IP_POOL_END=10.151.2.173
2272024-02-16T07:00:08.335Z+ DNS_IP1=10.151.2.164
2282024-02-16T07:00:08.335Z+ DNS_IP2=10.151.2.165
2292024-02-16T07:00:08.335Z+ UPLINK_IP=10.151.2.174
2302024-02-16T07:00:08.335Z+ IPPOOL_START=10.151.2.175
2312024-02-16T07:00:08.335Z+ IPPOOL_END=10.151.2.227
2322024-02-16T07:00:08.362Z++ ipadm show-addr -po type,addr
2332024-02-16T07:00:08.362Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2342024-02-16T07:00:08.362Z+ GATEWAY_IP=10.151.2.100
2352024-02-16T07:00:08.362Z+ [[ -n 10.151.2.100 ]]
2362024-02-16T07:00:08.362Z+ ping -s 10.151.2.100 56 1
2372024-02-16T07:00:08.363ZPING 10.151.2.100: 56 data bytes
2382024-02-16T07:00:08.363Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2392024-02-16T07:00:08.363Z
2402024-02-16T07:00:08.363Z----10.151.2.100 PING Statistics----
2412024-02-16T07:00:08.363Z1 packets transmitted, 1 packets received, 0% packet loss
2422024-02-16T07:00:08.363Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2432024-02-16T07:00:08.363Z++ arp -an
2442024-02-16T07:00:08.364Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2452024-02-16T07:00:08.364Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2462024-02-16T07:00:08.364Z+ routeadm -e ipv4-forwarding -u
2472024-02-16T07:00:08.466Z+ PXA_START=10.151.2.164
2482024-02-16T07:00:08.466Z+ PXA_END=10.151.2.227
2492024-02-16T07:00:08.493Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2502024-02-16T07:00:08.493Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2512024-02-16T07:00:08.673Z+ ZPOOL_VDEV_DIR=/scratch
2522024-02-16T07:00:08.673Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2532024-02-16T07:00:08.700Z+++ dirname ./tools/create_virtual_hardware.sh
2542024-02-16T07:00:08.701Z++ cd ./tools
2552024-02-16T07:00:08.701Z++ pwd
2562024-02-16T07:00:08.701Z+ SOURCE_DIR=/opt/oxide/work/tools
2572024-02-16T07:00:08.701Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2582024-02-16T07:00:08.701Z+ SOFTNPU_MODE=zone
2592024-02-16T07:00:08.701Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2602024-02-16T07:00:08.701Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2612024-02-16T07:00:08.702Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2622024-02-16T07:00:08.702Z++ dladm show-phys -p -o LINK
2632024-02-16T07:00:08.702Z++ head -1
2642024-02-16T07:00:08.702Z+ PHYSICAL_LINK=igb0
2652024-02-16T07:00:08.702Z+ echo 'Using igb0 as physical link'
2662024-02-16T07:00:08.702ZUsing igb0 as physical link
2672024-02-16T07:00:08.702Z+ ensure_run_as_root
2682024-02-16T07:00:08.730Z++ id -u
2692024-02-16T07:00:08.730Z+ [[ 0 -ne 0 ]]
2702024-02-16T07:00:08.730Z+ ensure_zpools
2712024-02-16T07:00:08.730Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2722024-02-16T07:00:08.731Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2732024-02-16T07:00:08.731Z+ readarray -t ZPOOLS
2742024-02-16T07:00:08.731Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2752024-02-16T07:00:08.731Z++ sed 's/[ ",]//g'
2762024-02-16T07:00:08.731Z+ for ZPOOL in "${ZPOOLS[@]}"
2772024-02-16T07:00:08.731Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2782024-02-16T07:00:08.732ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2792024-02-16T07:00:08.732Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2802024-02-16T07:00:08.732Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2812024-02-16T07:00:08.732Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2822024-02-16T07:00:08.732Z0+0 records in
2832024-02-16T07:00:08.732Z0+0 records out
2842024-02-16T07:00:08.732Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2852024-02-16T07:00:08.732Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2862024-02-16T07:00:08.733Z+ set +x
2872024-02-16T07:00:08.733ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2882024-02-16T07:00:08.733Z++ zpool list -o name
2892024-02-16T07:00:08.733Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2902024-02-16T07:00:08.733Z+ [[ -z '' ]]
2912024-02-16T07:00:08.733Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2922024-02-16T07:00:08.765Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2932024-02-16T07:00:08.792Z+ set +x
2942024-02-16T07:00:08.792ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2952024-02-16T07:00:08.792Z+ for ZPOOL in "${ZPOOLS[@]}"
2962024-02-16T07:00:08.792Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2972024-02-16T07:00:08.792Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2982024-02-16T07:00:08.792ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2992024-02-16T07:00:08.792Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3002024-02-16T07:00:08.792Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3012024-02-16T07:00:08.793Z0+0 records in
3022024-02-16T07:00:08.793Z0+0 records out
3032024-02-16T07:00:08.793Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3042024-02-16T07:00:08.793Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3052024-02-16T07:00:08.793Z+ set +x
3062024-02-16T07:00:08.793ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3072024-02-16T07:00:08.793Z++ zpool list -o name
3082024-02-16T07:00:08.793Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3092024-02-16T07:00:08.793Z+ [[ -z '' ]]
3102024-02-16T07:00:08.793Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3112024-02-16T07:00:08.818Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3122024-02-16T07:00:08.819Z+ set +x
3132024-02-16T07:00:08.819ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3142024-02-16T07:00:08.819Z+ for ZPOOL in "${ZPOOLS[@]}"
3152024-02-16T07:00:08.819Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3162024-02-16T07:00:08.819ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3172024-02-16T07:00:08.819Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3182024-02-16T07:00:08.819Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3192024-02-16T07:00:08.819Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3202024-02-16T07:00:08.819Z0+0 records in
3212024-02-16T07:00:08.819Z0+0 records out
3222024-02-16T07:00:08.819Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3232024-02-16T07:00:08.819Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3242024-02-16T07:00:08.819Z+ set +x
3252024-02-16T07:00:08.820ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3262024-02-16T07:00:08.820Z++ zpool list -o name
3272024-02-16T07:00:08.820Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3282024-02-16T07:00:08.845Z+ [[ -z '' ]]
3292024-02-16T07:00:08.845Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3302024-02-16T07:00:08.872Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3312024-02-16T07:00:08.872Z+ set +x
3322024-02-16T07:00:08.872ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3332024-02-16T07:00:08.872Z+ for ZPOOL in "${ZPOOLS[@]}"
3342024-02-16T07:00:08.872Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3352024-02-16T07:00:08.872ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3362024-02-16T07:00:08.873Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3372024-02-16T07:00:08.873Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3382024-02-16T07:00:08.873Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3392024-02-16T07:00:08.873Z0+0 records in
3402024-02-16T07:00:08.873Z0+0 records out
3412024-02-16T07:00:08.873Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3422024-02-16T07:00:08.873Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3432024-02-16T07:00:08.873Z+ set +x
3442024-02-16T07:00:08.874ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3452024-02-16T07:00:08.874Z++ zpool list -o name
3462024-02-16T07:00:08.874Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3472024-02-16T07:00:08.874Z+ [[ -z '' ]]
3482024-02-16T07:00:08.874Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3492024-02-16T07:00:08.911Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3502024-02-16T07:00:08.911Z+ set +x
3512024-02-16T07:00:08.937ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3522024-02-16T07:00:08.938Z+ for ZPOOL in "${ZPOOLS[@]}"
3532024-02-16T07:00:08.938Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542024-02-16T07:00:08.938ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3552024-02-16T07:00:08.938Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562024-02-16T07:00:08.938Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572024-02-16T07:00:08.938Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582024-02-16T07:00:08.938Z0+0 records in
3592024-02-16T07:00:08.939Z0+0 records out
3602024-02-16T07:00:08.939Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3612024-02-16T07:00:08.939Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622024-02-16T07:00:08.939Z+ set +x
3632024-02-16T07:00:08.939ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3642024-02-16T07:00:08.939Z++ zpool list -o name
3652024-02-16T07:00:08.940Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3662024-02-16T07:00:08.940Z+ [[ -z '' ]]
3672024-02-16T07:00:08.940Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3682024-02-16T07:00:08.965Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3692024-02-16T07:00:08.966Z+ set +x
3702024-02-16T07:00:08.966ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3712024-02-16T07:00:08.966Z+ for ZPOOL in "${ZPOOLS[@]}"
3722024-02-16T07:00:08.966Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3732024-02-16T07:00:08.966ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3742024-02-16T07:00:08.966Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3752024-02-16T07:00:08.966Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3762024-02-16T07:00:08.966Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3772024-02-16T07:00:08.966Z0+0 records in
3782024-02-16T07:00:08.966Z0+0 records out
3792024-02-16T07:00:08.967Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3802024-02-16T07:00:08.967Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3812024-02-16T07:00:08.967Z+ set +x
3822024-02-16T07:00:08.967ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3832024-02-16T07:00:08.967Z++ zpool list -o name
3842024-02-16T07:00:08.967Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3852024-02-16T07:00:08.993Z+ [[ -z '' ]]
3862024-02-16T07:00:08.993Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872024-02-16T07:00:09.019Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3882024-02-16T07:00:09.019Z+ set +x
3892024-02-16T07:00:09.019ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3902024-02-16T07:00:09.020Z+ for ZPOOL in "${ZPOOLS[@]}"
3912024-02-16T07:00:09.020Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3922024-02-16T07:00:09.020ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3932024-02-16T07:00:09.020Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3942024-02-16T07:00:09.020Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3952024-02-16T07:00:09.020Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3962024-02-16T07:00:09.020Z0+0 records in
3972024-02-16T07:00:09.020Z0+0 records out
3982024-02-16T07:00:09.021Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3992024-02-16T07:00:09.021Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4002024-02-16T07:00:09.021Z+ set +x
4012024-02-16T07:00:09.021ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
4022024-02-16T07:00:09.021Z++ zpool list -o name
4032024-02-16T07:00:09.021Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4042024-02-16T07:00:09.047Z+ [[ -z '' ]]
4052024-02-16T07:00:09.047Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4062024-02-16T07:00:09.072Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4072024-02-16T07:00:09.073Z+ set +x
4082024-02-16T07:00:09.073ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4092024-02-16T07:00:09.073Z+ for ZPOOL in "${ZPOOLS[@]}"
4102024-02-16T07:00:09.073ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4112024-02-16T07:00:09.073Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4122024-02-16T07:00:09.074Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4132024-02-16T07:00:09.074Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4142024-02-16T07:00:09.074Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4152024-02-16T07:00:09.074Z0+0 records in
4162024-02-16T07:00:09.074Z0+0 records out
4172024-02-16T07:00:09.074Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
4182024-02-16T07:00:09.074Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4192024-02-16T07:00:09.075Z+ set +x
4202024-02-16T07:00:09.075ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4212024-02-16T07:00:09.075Z++ zpool list -o name
4222024-02-16T07:00:09.075Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4232024-02-16T07:00:09.075Z+ [[ -z '' ]]
4242024-02-16T07:00:09.075Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4252024-02-16T07:00:09.101Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4262024-02-16T07:00:09.101Z+ set +x
4272024-02-16T07:00:09.127ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4282024-02-16T07:00:09.127Z+ for ZPOOL in "${ZPOOLS[@]}"
4292024-02-16T07:00:09.127Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4302024-02-16T07:00:09.127ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4312024-02-16T07:00:09.127Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4322024-02-16T07:00:09.127Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4332024-02-16T07:00:09.128Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4342024-02-16T07:00:09.128Z0+0 records in
4352024-02-16T07:00:09.128Z0+0 records out
4362024-02-16T07:00:09.128Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4372024-02-16T07:00:09.128Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4382024-02-16T07:00:09.128Z+ set +x
4392024-02-16T07:00:09.128ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4402024-02-16T07:00:09.128Z++ zpool list -o name
4412024-02-16T07:00:09.128Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4422024-02-16T07:00:09.129Z+ [[ -z '' ]]
4432024-02-16T07:00:09.129Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4442024-02-16T07:00:09.154Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4452024-02-16T07:00:09.154Z+ set +x
4462024-02-16T07:00:09.155ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4472024-02-16T07:00:09.155Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4482024-02-16T07:00:09.155Z+ readarray -t ZPOOLS
4492024-02-16T07:00:09.155Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4502024-02-16T07:00:09.180Z++ sed 's/[ ",]//g'
4512024-02-16T07:00:09.181Z+ for ZPOOL in "${ZPOOLS[@]}"
4522024-02-16T07:00:09.181Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4532024-02-16T07:00:09.181ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4542024-02-16T07:00:09.181Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4552024-02-16T07:00:09.181Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4562024-02-16T07:00:09.181Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4572024-02-16T07:00:09.181Z0+0 records in
4582024-02-16T07:00:09.182Z0+0 records out
4592024-02-16T07:00:09.182Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4602024-02-16T07:00:09.182Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4612024-02-16T07:00:09.182Z+ set +x
4622024-02-16T07:00:09.182ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4632024-02-16T07:00:09.182Z++ zpool list -o name
4642024-02-16T07:00:09.183Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4652024-02-16T07:00:09.183Z+ [[ -z '' ]]
4662024-02-16T07:00:09.183Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4672024-02-16T07:00:09.208Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4682024-02-16T07:00:09.209Z+ set +x
4692024-02-16T07:00:09.209ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4702024-02-16T07:00:09.209Z+ for ZPOOL in "${ZPOOLS[@]}"
4712024-02-16T07:00:09.209Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4722024-02-16T07:00:09.209ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4732024-02-16T07:00:09.209Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4742024-02-16T07:00:09.209Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4752024-02-16T07:00:09.210Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4762024-02-16T07:00:09.210Z0+0 records in
4772024-02-16T07:00:09.210Z0+0 records out
4782024-02-16T07:00:09.210Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4792024-02-16T07:00:09.210Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4802024-02-16T07:00:09.210Z+ set +x
4812024-02-16T07:00:09.210ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4822024-02-16T07:00:09.265Z++ zpool list -o name
4832024-02-16T07:00:09.266Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4842024-02-16T07:00:09.266Z+ [[ -z '' ]]
4852024-02-16T07:00:09.266Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4862024-02-16T07:00:09.266Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4872024-02-16T07:00:09.266Z+ set +x
4882024-02-16T07:00:09.266ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4892024-02-16T07:00:09.267Z+ [[ zone == \z\o\n\e ]]
4902024-02-16T07:00:09.267Z+ ensure_simulated_links igb0
4912024-02-16T07:00:09.267Z+ local PHYSICAL_LINK=igb0
4922024-02-16T07:00:09.267Z+ INDICES=("0" "1")
4932024-02-16T07:00:09.267Z+ for I in "${INDICES[@]}"
4942024-02-16T07:00:09.268Z++ get_simnet_name_if_exists net0
4952024-02-16T07:00:09.268Z++ dladm show-simnet -p -o LINK net0
4962024-02-16T07:00:09.268Z++ echo ''
4972024-02-16T07:00:09.268Z+ [[ -z '' ]]
4982024-02-16T07:00:09.268Z+ dladm create-simnet -t net0
4992024-02-16T07:00:09.296Z+ dladm create-simnet -t sc0_0
5002024-02-16T07:00:09.296Z+ dladm modify-simnet -t -p net0 sc0_0
5012024-02-16T07:00:09.296Z+ dladm set-linkprop -p mtu=9000 sc0_0
5022024-02-16T07:00:09.296Z+ success 'Simnet net0/sc0_0 exists'
5032024-02-16T07:00:09.296Z+ set +x
5042024-02-16T07:00:09.296ZSimnet net0/sc0_0 exists
5052024-02-16T07:00:09.296Z+ for I in "${INDICES[@]}"
5062024-02-16T07:00:09.297Z++ get_simnet_name_if_exists net1
5072024-02-16T07:00:09.297Z++ dladm show-simnet -p -o LINK net1
5082024-02-16T07:00:09.297Z++ echo ''
5092024-02-16T07:00:09.297Z+ [[ -z '' ]]
5102024-02-16T07:00:09.297Z+ dladm create-simnet -t net1
5112024-02-16T07:00:09.297Z+ dladm create-simnet -t sc1_0
5122024-02-16T07:00:09.297Z+ dladm modify-simnet -t -p net1 sc1_0
5132024-02-16T07:00:09.324Z+ dladm set-linkprop -p mtu=9000 sc1_0
5142024-02-16T07:00:09.324Z+ success 'Simnet net1/sc1_0 exists'
5152024-02-16T07:00:09.324Z+ set +x
5162024-02-16T07:00:09.324ZSimnet net1/sc1_0 exists
5172024-02-16T07:00:09.324Z++ get_vnic_name_if_exists sc0_1
5182024-02-16T07:00:09.325Z++ dladm show-vnic -p -o LINK sc0_1
5192024-02-16T07:00:09.325Z++ echo ''
5202024-02-16T07:00:09.325Z+ [[ -z '' ]]
5212024-02-16T07:00:09.325Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5222024-02-16T07:00:09.325Z+ [[ -v PROMISC_FILT_OFF ]]
5232024-02-16T07:00:09.325Z+ success 'Vnic sc0_1 exists'
5242024-02-16T07:00:09.326Z+ set +x
5252024-02-16T07:00:09.326ZVnic sc0_1 exists
5262024-02-16T07:00:09.326Z+ warn_if_no_proxy_arp
5272024-02-16T07:00:09.326Z+ [[ -v PXA_START ]]
5282024-02-16T07:00:09.326Z+ [[ -v PXA_END ]]
5292024-02-16T07:00:09.326Z+ ensure_softnpu_zone
5302024-02-16T07:00:09.326Z+ zoneadm list
5312024-02-16T07:00:09.326Z+ grep -q sidecar_softnpu
5322024-02-16T07:00:09.326Z+ [[ -f out/npuzone/npuzone ]]
5332024-02-16T07:00:09.327Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit e3ea4b495ba0a71801ded0776ae4bbd31df57e26 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5342024-02-16T07:00:09.353Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5352024-02-16T07:00:15.878Zdownloading softnpu ...done
5362024-02-16T07:00:15.904Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/libsidecar_lite.so.sha256.txt
5372024-02-16T07:00:20.850Zdownloading asic_program.so ...done
5382024-02-16T07:00:20.876Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/scadm.sha256.txt
5392024-02-16T07:00:27.186Zdownloading scadm ...done
5402024-02-16T07:00:27.211Zcreating port tfportrear0_0 for sc0_0
5412024-02-16T07:00:27.211Zcreating port tfportqsfp0_0 for sc0_1
5422024-02-16T07:00:27.236Zsoftnpu zone setup
5432024-02-16T07:00:27.236Zconfigure zone
5442024-02-16T07:00:27.236Zinstall zone
5452024-02-16T07:00:28.278Zboot zone
5462024-02-16T07:00:31.398Z[sidecar_softnpu] svcs milestone/network
5472024-02-16T07:00:31.424ZSTATE STIME FMRI
5482024-02-16T07:00:31.424Zonline 7:00:29 svc:/milestone/network:default
5492024-02-16T07:00:31.424Z[sidecar_softnpu] /softnpu/npu start
5502024-02-16T07:00:31.449Z
5512024-02-16T07:00:31.474Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5522024-02-16T07:00:31.474Z+ GATEWAY_IP=10.151.2.100
5532024-02-16T07:00:31.474Z+ echo 'Using 10.151.2.100 as gateway ip'
5542024-02-16T07:00:31.474ZUsing 10.151.2.100 as gateway ip
5552024-02-16T07:00:31.474Z+ [[ ! -v GATEWAY_MAC ]]
5562024-02-16T07:00:31.474Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5572024-02-16T07:00:31.474Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5582024-02-16T07:00:31.474Z+++ echo 18:c0:4d:0d:a0:2a
5592024-02-16T07:00:31.474Z+ COUNT=1
5602024-02-16T07:00:31.475Z+ [[ 1 -ne 1 ]]
5612024-02-16T07:00:31.475Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5622024-02-16T07:00:31.475ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5632024-02-16T07:00:31.475Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5642024-02-16T07:00:31.475Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5652024-02-16T07:00:31.499Z+ PXA_MAC=a8:e1:de:01:70:1d
5662024-02-16T07:00:31.499Z+ [[ -v PXA_START ]]
5672024-02-16T07:00:31.500Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5682024-02-16T07:00:31.500Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5692024-02-16T07:00:31.524Z+ z_scadm dump-state
5702024-02-16T07:00:31.524Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5712024-02-16T07:00:31.549Zlocal v6:
5722024-02-16T07:00:31.549Zlocal v4:
5732024-02-16T07:00:31.549Zrouter v6:
5742024-02-16T07:00:31.549Zrouter v4_idx:
5752024-02-16T07:00:31.549Zrouter v4_routes:
5762024-02-16T07:00:31.549Zresolver v4:
5772024-02-16T07:00:31.549Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5782024-02-16T07:00:31.549Zresolver v6:
5792024-02-16T07:00:31.549Znat v4:
5802024-02-16T07:00:31.549Znat_v6:
5812024-02-16T07:00:31.549Zport_mac:
5822024-02-16T07:00:31.549Zproxy arp:
5832024-02-16T07:00:31.549Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5842024-02-16T07:00:31.549Z+ success 'softnpu zone exists'
5852024-02-16T07:00:31.549Z+ set +x
5862024-02-16T07:00:31.549Zsoftnpu zone exists
5872024-02-16T07:00:31.550Z
5882024-02-16T07:00:31.550Zreal 22.872712037
5892024-02-16T07:00:31.550Zuser 0.710730136
5902024-02-16T07:00:31.550Zsys 1.602080393
5912024-02-16T07:00:31.550Ztrap 0.000072129
5922024-02-16T07:00:31.550Ztflt 0.000214968
5932024-02-16T07:00:31.550Zdflt 0.000973054
5942024-02-16T07:00:31.550Zkflt 0.000000000
5952024-02-16T07:00:31.550Zlock 17:38.379648781
5962024-02-16T07:00:31.550Zslp 1:08.191340168
5972024-02-16T07:00:31.550Zlat 0.082326515
5982024-02-16T07:00:31.550Zstop 0.341505820
5992024-02-16T07:00:31.575Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
6002024-02-16T07:00:31.600Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
6012024-02-16T07:00:31.600Z+ SILO_NAME=recovery
6022024-02-16T07:00:31.600Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
6032024-02-16T07:00:31.600Z+ EXTERNAL_DNS_DOMAIN=oxide.test
6042024-02-16T07:00:31.600Z+ sed -i~ '
6052024-02-16T07:00:31.600Z /^external_dns_ips/c\
6062024-02-16T07:00:31.600Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6072024-02-16T07:00:31.600Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6082024-02-16T07:00:31.600Z /^first/c\
6092024-02-16T07:00:31.600Zfirst = "10.151.2.164"
6102024-02-16T07:00:31.600Z /^last/c\
6112024-02-16T07:00:31.600Zlast = "10.151.2.173"
6122024-02-16T07:00:31.600Z }
6132024-02-16T07:00:31.600Z /^\[rack_network_config/,/^$/ {
6142024-02-16T07:00:31.600Z /^infra_ip_first/c\
6152024-02-16T07:00:31.601Zinfra_ip_first = "10.151.2.174"
6162024-02-16T07:00:31.601Z /^infra_ip_last/c\
6172024-02-16T07:00:31.601Zinfra_ip_last = "10.151.2.174"
6182024-02-16T07:00:31.601Z }
6192024-02-16T07:00:31.601Z /^\[\[rack_network_config.ports/,/^$/ {
6202024-02-16T07:00:31.601Z /^routes/c\
6212024-02-16T07:00:31.601Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6222024-02-16T07:00:31.601Z /^addresses/c\
6232024-02-16T07:00:31.601Zaddresses = \["10.151.2.174/24"\]
6242024-02-16T07:00:31.601Z }
6252024-02-16T07:00:31.601Z' pkg/config-rss.toml
6262024-02-16T07:00:31.601Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6272024-02-16T07:00:31.601Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6282024-02-16T07:00:31.601Z+++ pkg/config-rss.toml Fri Feb 16 07:00:31 2024
6292024-02-16T07:00:31.601Z@@ -33,7 +33,7 @@
6302024-02-16T07:00:31.601Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6312024-02-16T07:00:31.601Z # must be contained in one of the "internal services" IP Pool ranges listed
6322024-02-16T07:00:31.601Z # below.
6332024-02-16T07:00:31.601Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6342024-02-16T07:00:31.601Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6352024-02-16T07:00:31.601Z
6362024-02-16T07:00:31.602Z # Initial TLS certificates for the external API
6372024-02-16T07:00:31.602Z #
6382024-02-16T07:00:31.602Z@@ -69,8 +69,8 @@
6392024-02-16T07:00:31.602Z #
6402024-02-16T07:00:31.602Z # For more on this and what to put here, see docs/how-to-run.adoc.
6412024-02-16T07:00:31.602Z [[internal_services_ip_pool_ranges]]
6422024-02-16T07:00:31.602Z-first = "192.168.1.20"
6432024-02-16T07:00:31.602Z-last = "192.168.1.29"
6442024-02-16T07:00:31.602Z+first = "10.151.2.164"
6452024-02-16T07:00:31.602Z+last = "10.151.2.173"
6462024-02-16T07:00:31.602Z
6472024-02-16T07:00:31.602Z # TODO - this configuration is subject to change going forward. Ultimately these
6482024-02-16T07:00:31.602Z # parameters should be provided to the control plane via wicket, but we need to
6492024-02-16T07:00:31.602Z@@ -100,9 +100,9 @@
6502024-02-16T07:00:31.602Z # You can configure multiple uplinks by repeating the following stanza
6512024-02-16T07:00:31.602Z [[rack_network_config.ports]]
6522024-02-16T07:00:31.602Z # Routes associated with this port.
6532024-02-16T07:00:31.602Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6542024-02-16T07:00:31.602Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6552024-02-16T07:00:31.602Z # Addresses associated with this port.
6562024-02-16T07:00:31.602Z-addresses = ["192.168.1.30/24"]
6572024-02-16T07:00:31.602Z+addresses = ["10.151.2.174/24"]
6582024-02-16T07:00:31.602Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6592024-02-16T07:00:31.603Z port = "qsfp0"
6602024-02-16T07:00:31.603Z # The speed of this port.
6612024-02-16T07:00:31.603Z+ true
6622024-02-16T07:00:31.603Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6632024-02-16T07:00:31.603Za pkg/config-rss.toml 7K
6642024-02-16T07:00:31.603Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6652024-02-16T07:00:31.628Z+ TLS_NAME=recovery.sys.oxide.test
6662024-02-16T07:00:31.628Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6672024-02-16T07:00:31.997Z...+.......+..+.+.........+...........+...+.+.........+.....+.+......+..+.+.....+...............+.......+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...+...+......+.........+......+...+...+....+...+.....+.......+......+..+.+..+...+.........+.......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.+...+.....+....+..+...+.......+...........+....+...+...+..+...+...+.+...............+..............+.+...+..+........................+.......+...+...........+..........+...+........+...+.........+................+.....+..................................+............+...+.......................+.............+............+...+............+............+......+.....+....+..+.+..............+...+.+............+.....+.......+..+..................+.+..+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6682024-02-16T07:00:32.496Z...+.....+.......+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+...+...+....+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+............+......+.....................+.+........+.......+........+....+.....+...+..................+....+...+............+..+....+...........+................+..............+....+.....................+.....+.............+.....+.+.....+.+.....+................+.....+.......+...+........+....+...+.................+...............+.+......+...........+...+.......+..+............+....+..+.......+........+....+...+..+......+.........+...............................+..+......+.+.....+.+........+.+.....+.+..+.............+........+.......+.....+..........+..+.+..+...................+...+......+.........+...+........+...............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6692024-02-16T07:00:32.522Z-----
6702024-02-16T07:00:32.522Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6712024-02-16T07:00:32.522Za pkg/initial-tls-cert.pem 2K
6722024-02-16T07:00:32.522Za pkg/initial-tls-key.pem 4K
6732024-02-16T07:00:32.522Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742024-02-16T07:00:32.522Z+ rmdir pkg
6752024-02-16T07:00:32.523Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6762024-02-16T07:00:32.523Z+ pfexec mkdir -p /usr/oxide
6772024-02-16T07:00:32.523Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6782024-02-16T07:00:33.228Z+ pfexec chmod +x /usr/oxide/oxide
6792024-02-16T07:00:33.255Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6802024-02-16T07:00:51.485Z+ OMICRON_NO_UNINSTALL=1
6812024-02-16T07:00:51.485Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6822024-02-16T07:00:51.510ZLogging to: /opt/oxide/work/out/LOG
6832024-02-16T07:00:51.989Z
6842024-02-16T07:00:52.016Zreal 0.499541619
6852024-02-16T07:00:52.016Zuser 0.109518711
6862024-02-16T07:00:52.016Zsys 1.991043609
6872024-02-16T07:00:52.016Ztrap 0.000000320
6882024-02-16T07:00:52.016Ztflt 0.000095869
6892024-02-16T07:00:52.016Zdflt 0.000386070
6902024-02-16T07:00:52.017Zkflt 0.000000000
6912024-02-16T07:00:52.017Zlock 45.311181989
6922024-02-16T07:00:52.017Zslp 0.721710067
6932024-02-16T07:00:52.017Zlat 0.002950140
6942024-02-16T07:00:52.017Zstop 0.211367642
6952024-02-16T07:00:52.017Z+ retry=0
6962024-02-16T07:00:52.017Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
6972024-02-16T07:00:52.017Z+ [[ 0 -gt 30 ]]
6982024-02-16T07:00:52.017Z+ sleep 1
6992024-02-16T07:00:53.004Z+ retry=1
7002024-02-16T07:00:53.005Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7012024-02-16T07:00:53.031Z+ [[ 1 -gt 30 ]]
7022024-02-16T07:00:53.031Z+ sleep 1
7032024-02-16T07:00:54.021Z+ retry=2
7042024-02-16T07:00:54.021Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7052024-02-16T07:00:54.047Z+ [[ 2 -gt 30 ]]
7062024-02-16T07:00:54.047Z+ sleep 1
7072024-02-16T07:00:55.033Z+ retry=3
7082024-02-16T07:00:55.033Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7092024-02-16T07:00:55.061Z+ [[ 3 -gt 30 ]]
7102024-02-16T07:00:55.061Z+ sleep 1
7112024-02-16T07:00:56.049Z+ retry=4
7122024-02-16T07:00:56.049Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7132024-02-16T07:01:21.226Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7142024-02-16T07:01:21.258Zlocal v6:
7152024-02-16T07:01:21.258Zfd0e:61b4:e2c8:25fe::1
7162024-02-16T07:01:21.258Zfe80::ac41:9ff:fe14:d8cb
7172024-02-16T07:01:21.258Zlocal v4:
7182024-02-16T07:01:21.258Z10.151.2.174
7192024-02-16T07:01:21.258Zrouter v6:
7202024-02-16T07:01:21.258Zfd00:1122:3344:2::/64 -> fe80::8c38:94ff:fefa:6031 (1)
7212024-02-16T07:01:21.259Zfd00:1122:3344:1::/64 -> fe80::8c38:94ff:fefa:6031 (1)
7222024-02-16T07:01:21.259Zfd00:1122:3344:3::/64 -> fe80::8c38:94ff:fefa:6031 (1)
7232024-02-16T07:01:21.259Zfd00:1122:3344:101::/64 -> fe80::8c38:94ff:fefa:6031 (1)
7242024-02-16T07:01:21.259Zfdb0:18c0:4d0d:a02a::/64 -> fe80::8c38:94ff:fefa:6031 (1)
7252024-02-16T07:01:21.259Zrouter v4_idx:
7262024-02-16T07:01:21.259Z0.0.0.0/0 -> 0
7272024-02-16T07:01:21.259Zrouter v4_routes:
7282024-02-16T07:01:21.284Z0 -> 10.151.2.100 (2)
7292024-02-16T07:01:21.284Zresolver v4:
7302024-02-16T07:01:21.284Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7312024-02-16T07:01:21.284Z10.151.2.174 -> 46:91:56:8f:29:ac
7322024-02-16T07:01:21.284Zresolver v6:
7332024-02-16T07:01:21.284Zfe80::8c38:94ff:fefa:6031 -> 8e:38:94:fa:60:31
7342024-02-16T07:01:21.284Zfe80::ac41:9ff:fe14:d8cb -> ae:41:09:14:d8:cb
7352024-02-16T07:01:21.284Znat v4:
7362024-02-16T07:01:21.284Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:db:f5
7372024-02-16T07:01:21.284Znat_v6:
7382024-02-16T07:01:21.284Zport_mac:
7392024-02-16T07:01:21.284Z1: a8:40:25:53:ba:6f
7402024-02-16T07:01:21.284Z2: a8:40:25:53:ba:8f
7412024-02-16T07:01:21.284Z1000: a8:40:25:53:ba:6e
7422024-02-16T07:01:21.284Zproxy arp:
7432024-02-16T07:01:21.284Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7442024-02-16T07:01:21.284Z+ retry=0
7452024-02-16T07:01:21.284Z+++ zoneadm list -n
7462024-02-16T07:01:21.284Z+++ grep oxz_ntp
7472024-02-16T07:01:21.284Z++ pfexec svcs -z oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092 -Hostate oxide/ntp
7482024-02-16T07:01:21.310Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7492024-02-16T07:01:21.310Z++ true
7502024-02-16T07:01:21.310Z+ [[ '' != online ]]
7512024-02-16T07:01:21.310Z+ [[ 0 -gt 60 ]]
7522024-02-16T07:01:21.310Z+ sleep 1
7532024-02-16T07:01:22.291Z+ retry=1
7542024-02-16T07:01:22.316Z+++ zoneadm list -n
7552024-02-16T07:01:22.316Z+++ grep oxz_ntp
7562024-02-16T07:01:22.316Z++ pfexec svcs -z oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092 -Hostate oxide/ntp
7572024-02-16T07:01:22.342Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7582024-02-16T07:01:22.342Z++ true
7592024-02-16T07:01:22.342Z+ [[ '' != online ]]
7602024-02-16T07:01:22.342Z+ [[ 1 -gt 60 ]]
7612024-02-16T07:01:22.343Z+ sleep 1
7622024-02-16T07:01:23.320Z+ retry=2
7632024-02-16T07:01:23.346Z+++ zoneadm list -n
7642024-02-16T07:01:23.346Z+++ grep oxz_ntp
7652024-02-16T07:01:23.346Z++ pfexec svcs -z oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092 -Hostate oxide/ntp
7662024-02-16T07:01:23.346Z+ [[ disabled != online ]]
7672024-02-16T07:01:23.346Z+ [[ 2 -gt 60 ]]
7682024-02-16T07:01:23.346Z+ sleep 1
7692024-02-16T07:01:24.344Z+ retry=3
7702024-02-16T07:01:24.370Z+++ zoneadm list -n
7712024-02-16T07:01:24.370Z+++ grep oxz_ntp
7722024-02-16T07:01:24.371Z++ pfexec svcs -z oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092 -Hostate oxide/ntp
7732024-02-16T07:01:24.371Z+ [[ disabled != online ]]
7742024-02-16T07:01:24.371Z+ [[ 3 -gt 60 ]]
7752024-02-16T07:01:24.371Z+ sleep 1
7762024-02-16T07:01:25.367Z+ retry=4
7772024-02-16T07:01:25.393Z+++ zoneadm list -n
7782024-02-16T07:01:25.393Z+++ grep oxz_ntp
7792024-02-16T07:01:25.393Z++ pfexec svcs -z oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092 -Hostate oxide/ntp
7802024-02-16T07:01:25.393Z+ [[ disabled != online ]]
7812024-02-16T07:01:25.393Z+ [[ 4 -gt 60 ]]
7822024-02-16T07:01:25.393Z+ sleep 1
7832024-02-16T07:01:26.392Z+ retry=5
7842024-02-16T07:01:26.422Z+++ zoneadm list -n
7852024-02-16T07:01:26.422Z+++ grep oxz_ntp
7862024-02-16T07:01:26.423Z++ pfexec svcs -z oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092 -Hostate oxide/ntp
7872024-02-16T07:01:26.423Z+ [[ disabled != online ]]
7882024-02-16T07:01:26.423Z+ [[ 5 -gt 60 ]]
7892024-02-16T07:01:26.423Z+ sleep 1
7902024-02-16T07:01:27.413Z+ retry=6
7912024-02-16T07:01:27.440Z+++ zoneadm list -n
7922024-02-16T07:01:27.440Z+++ grep oxz_ntp
7932024-02-16T07:01:27.440Z++ pfexec svcs -z oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092 -Hostate oxide/ntp
7942024-02-16T07:01:27.440Z+ [[ disabled != online ]]
7952024-02-16T07:01:27.440Z+ [[ 6 -gt 60 ]]
7962024-02-16T07:01:27.441Z+ sleep 1
7972024-02-16T07:01:28.435Z+ retry=7
7982024-02-16T07:01:28.461Z+++ zoneadm list -n
7992024-02-16T07:01:28.461Z+++ grep oxz_ntp
8002024-02-16T07:01:28.461Z++ pfexec svcs -z oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092 -Hostate oxide/ntp
8012024-02-16T07:01:28.461Z+ [[ disabled != online ]]
8022024-02-16T07:01:28.461Z+ [[ 7 -gt 60 ]]
8032024-02-16T07:01:28.461Z+ sleep 1
8042024-02-16T07:01:29.457Z+ retry=8
8052024-02-16T07:01:29.483Z+++ zoneadm list -n
8062024-02-16T07:01:29.483Z+++ grep oxz_ntp
8072024-02-16T07:01:29.483Z++ pfexec svcs -z oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092 -Hostate oxide/ntp
8082024-02-16T07:01:29.483Z+ [[ disabled != online ]]
8092024-02-16T07:01:29.483Z+ [[ 8 -gt 60 ]]
8102024-02-16T07:01:29.484Z+ sleep 1
8112024-02-16T07:01:30.479Z+ retry=9
8122024-02-16T07:01:30.505Z+++ zoneadm list -n
8132024-02-16T07:01:30.505Z+++ grep oxz_ntp
8142024-02-16T07:01:30.505Z++ pfexec svcs -z oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092 -Hostate oxide/ntp
8152024-02-16T07:01:30.505Z+ [[ disabled != online ]]
8162024-02-16T07:01:30.505Z+ [[ 9 -gt 60 ]]
8172024-02-16T07:01:30.505Z+ sleep 1
8182024-02-16T07:01:31.501Z+ retry=10
8192024-02-16T07:01:31.933Z+++ zoneadm list -n
8202024-02-16T07:01:31.933Z+++ grep oxz_ntp
8212024-02-16T07:01:31.933Z++ pfexec svcs -z oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092 -Hostate oxide/ntp
8222024-02-16T07:01:31.933Z+ [[ offline* != online ]]
8232024-02-16T07:01:31.933Z+ [[ 10 -gt 60 ]]
8242024-02-16T07:01:31.934Z+ sleep 1
8252024-02-16T07:01:32.525Z+ retry=11
8262024-02-16T07:01:32.551Z+++ zoneadm list -n
8272024-02-16T07:01:32.552Z+++ grep oxz_ntp
8282024-02-16T07:01:32.552Z++ pfexec svcs -z oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092 -Hostate oxide/ntp
8292024-02-16T07:01:32.552Z+ [[ online != online ]]
8302024-02-16T07:01:32.552Z+ echo 'Waited for chrony: 11s'
8312024-02-16T07:01:32.552Z+ export RUST_BACKTRACE=1
8322024-02-16T07:01:32.552ZWaited for chrony: 11s
8332024-02-16T07:01:32.552Z+ RUST_BACKTRACE=1
8342024-02-16T07:01:32.552Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8352024-02-16T07:01:32.552Z++ ./tests/bootstrap
8362024-02-16T07:01:32.580ZUsing external DNS server at 10.151.2.164:53
8372024-02-16T07:01:32.580Z2024-02-16 07:01:32.604076830 UTC: attempting to log into API
8382024-02-16T07:01:47.611Z2024-02-16 07:01:47.610841064 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8392024-02-16T07:01:48.611Z2024-02-16 07:01:48.609208564 UTC: attempting to log into API
8402024-02-16T07:02:03.642Z2024-02-16 07:02:03.628262920 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8412024-02-16T07:02:04.646Z2024-02-16 07:02:04.629418162 UTC: attempting to log into API
 
8422024-02-16T07:02:13.848ZError: No response to keepalive - Terminating session
 
8432024-02-16T07:02:19.693Z2024-02-16 07:02:19.664339870 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8442024-02-16T07:02:20.681Z2024-02-16 07:02:20.666690892 UTC: attempting to log into API
 
8452024-02-16T07:02:23.717ZError: No response de-activating SOL payload
8462024-02-16T07:02:25.080Z[SOL Session operational. Use ~? for help]
 
8472024-02-16T07:02:36.426Z2024-02-16 07:02:36.023810558 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8482024-02-16T07:02:37.284Z2024-02-16 07:02:37.270020794 UTC: attempting to log into API
8492024-02-16T07:02:52.575Z2024-02-16 07:02:52.535356544 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8502024-02-16T07:02:53.552Z2024-02-16 07:02:53.536664290 UTC: attempting to log into API
 
8512024-02-16T07:03:07.274ZFeb 16 07:03:06 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
8522024-02-16T07:03:08.583Z2024-02-16 07:03:08.568418511 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8532024-02-16T07:03:09.584Z2024-02-16 07:03:09.569134742 UTC: attempting to log into API
 
8542024-02-16T07:03:11.321ZSUNW-MSG-ID: ZFS-8000-FD, TYPE: Fault, VER: 1, SEVERITY: Major
8552024-02-16T07:03:11.374ZEVENT-TIME: Fri Feb 16 07:03:10 UTC 2024
8562024-02-16T07:03:11.415ZPLATFORM: R152-Z32-00, CSN: GKG8N7212A0003, HOSTNAME: sock
8572024-02-16T07:03:11.452ZSOURCE: zfs-diagnosis, REV: 1.0
8582024-02-16T07:03:11.485ZEVENT-ID: ba329890-533f-42ea-a445-52cb673b52c5
8592024-02-16T07:03:11.518ZDESC: The number of I/O errors associated with a ZFS device exceeded
8602024-02-16T07:03:11.552Zacceptable levels. Refer to http://illumos.org/msg/ZFS-8000-FD for more information.
8612024-02-16T07:03:11.585ZAUTO-RESPONSE: The device has been offlined and marked as faulted. An attempt
8622024-02-16T07:03:11.619Zwill be made to activate a hot spare if available.
8632024-02-16T07:03:11.652ZIMPACT: Fault tolerance of the pool may be compromised.
8642024-02-16T07:03:11.686ZREC-ACTION: Run 'zpool status -x' and replace the bad device.
8652024-02-16T07:03:14.897Z^@Feb 16 07:03:14 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
8662024-02-16T07:03:21.032ZFeb 16 07:03:20 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
8672024-02-16T07:03:24.615Z2024-02-16 07:03:24.600366630 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8682024-02-16T07:03:25.617Z2024-02-16 07:03:25.601459790 UTC: attempting to log into API
 
8692024-02-16T07:03:27.704ZFeb 16 07:03:27 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
8702024-02-16T07:03:34.380ZFeb 16 07:03:33 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8712024-02-16T07:03:40.650Z2024-02-16 07:03:40.634835963 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8722024-02-16T07:03:41.651Z2024-02-16 07:03:41.635941534 UTC: attempting to log into API
8732024-02-16T07:03:52.445Z2024-02-16 07:03:52.428863209 UTC: login succeeded
8742024-02-16T07:03:52.480Zwaiting for nexus to come up...
8752024-02-16T07:03:52.546Zcreating IP pool... 10.151.2.175 - 10.151.2.227
8762024-02-16T07:03:52.710Zsetting up quotas...
8772024-02-16T07:03:52.753Zensuring datasets are ready...
8782024-02-16T07:03:54.645Zlet's roll.
8792024-02-16T07:03:54.670Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
8802024-02-16T07:03:54.671ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
8812024-02-16T07:03:54.671ZOXIDE_TOKEN=oxide-token-70310831eecb0dad3a8b7be5c2af557584891292'
8822024-02-16T07:03:54.671Z++ OXIDE_HOST=https://recovery.sys.oxide.test
8832024-02-16T07:03:54.671Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
8842024-02-16T07:03:54.671Z++ OXIDE_TOKEN=oxide-token-70310831eecb0dad3a8b7be5c2af557584891292
8852024-02-16T07:03:54.671Z+ export OXIDE_HOST OXIDE_TOKEN
8862024-02-16T07:03:54.671Z+ retry=0
8872024-02-16T07:03:54.672Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8882024-02-16T07:03:54.755Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
8892024-02-16T07:03:54.991Z{"status":"ok"}success
8902024-02-16T07:03:54.992ZProject {
8912024-02-16T07:03:54.992Z description: "some images",
8922024-02-16T07:03:54.992Z id: 19dbbbf0-f86a-4831-b865-dbe837563cc0,
8932024-02-16T07:03:54.992Z name: Name(
8942024-02-16T07:03:54.992Z "images",
8952024-02-16T07:03:54.993Z ),
8962024-02-16T07:03:54.993Z time_created: 2024-02-16T07:03:54.805314Z,
8972024-02-16T07:03:54.993Z time_modified: 2024-02-16T07:03:54.805314Z,
8982024-02-16T07:03:54.993Z}
8992024-02-16T07:03:55.018Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
 
9002024-02-16T07:05:13.507ZError: No response to keepalive - Terminating session
9012024-02-16T07:05:23.665ZError: No response de-activating SOL payload
9022024-02-16T07:05:24.765Z[SOL Session operational. Use ~? for help]
9032024-02-16T07:08:29.424ZError: No response to keepalive - Terminating session
9042024-02-16T07:08:39.619ZError: No response de-activating SOL payload
9052024-02-16T07:08:40.734Z[SOL Session operational. Use ~? for help]
 
9062024-02-16T07:09:38.222Zdone!
9072024-02-16T07:09:38.334Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
9082024-02-16T07:09:38.424Zsuccess
9092024-02-16T07:09:38.425ZImage {
9102024-02-16T07:09:38.425Z block_size: ByteCount(
9112024-02-16T07:09:38.425Z 512,
9122024-02-16T07:09:38.425Z ),
9132024-02-16T07:09:38.425Z description: "debian 11 original base image",
9142024-02-16T07:09:38.426Z digest: None,
9152024-02-16T07:09:38.426Z id: 77c4f281-0452-44a8-95fc-989865a46dd9,
9162024-02-16T07:09:38.426Z name: Name(
9172024-02-16T07:09:38.456Z "debian11",
9182024-02-16T07:09:38.456Z ),
9192024-02-16T07:09:38.456Z os: "debian",
9202024-02-16T07:09:38.456Z project_id: None,
9212024-02-16T07:09:38.457Z size: ByteCount(
9222024-02-16T07:09:38.457Z 2147483648,
9232024-02-16T07:09:38.457Z ),
9242024-02-16T07:09:38.457Z time_created: 2024-02-16T07:09:38.175333Z,
9252024-02-16T07:09:38.457Z time_modified: 2024-02-16T07:09:38.396547Z,
9262024-02-16T07:09:38.457Z url: None,
9272024-02-16T07:09:38.457Z version: "11",
9282024-02-16T07:09:38.457Z}
9292024-02-16T07:09:38.457Z+ rm ./tests/bootstrap
9302024-02-16T07:09:38.499Z+ for test_bin in tests/*
9312024-02-16T07:09:38.500Z+ ./tests/bootstrap-22e122cc756fa22c
9322024-02-16T07:09:38.549Z
9332024-02-16T07:09:38.549Zrunning 0 tests
9342024-02-16T07:09:38.549Z
9352024-02-16T07:09:38.583Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9362024-02-16T07:09:38.583Z
9372024-02-16T07:09:38.584Z+ for test_bin in tests/*
9382024-02-16T07:09:38.584Z+ ./tests/end_to_end_tests-4eb1211856b7c67f
9392024-02-16T07:09:38.753Z
9402024-02-16T07:09:38.754Zrunning 2 tests
9412024-02-16T07:09:39.011Ztest no_spoof::no_spoof_here ... ok
9422024-02-16T07:10:22.998Ztest instance_launch::instance_launch ... ok
9432024-02-16T07:10:22.998Z
9442024-02-16T07:10:22.998Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 44.24s
9452024-02-16T07:10:22.998Z
9462024-02-16T07:10:23.023Z+ _exit_trap
9472024-02-16T07:10:23.063Z+ local status=0
9482024-02-16T07:10:23.098Z+ [[ 0 -eq 0 ]]
9492024-02-16T07:10:23.099Z+ exit 0
9502024-02-16T07:10:23.099Zprocess exited: duration 648553 ms, exit code 0
 
9512024-02-16T07:10:23.473Zfound 72 output files
9522024-02-16T07:10:23.524Zuploading: /var/svc/log/oxide-sled-agent:default.log (906327 bytes)
9532024-02-16T07:10:23.841Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 906327 -> 908550
9542024-02-16T07:10:24.921Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9552024-02-16T07:10:24.945Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log (2360808 bytes)
9562024-02-16T07:10:25.846Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log" changed size mid upload: 2360808 -> 2371267
9572024-02-16T07:10:26.947Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log
9582024-02-16T07:10:26.947Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log (8459 bytes)
9592024-02-16T07:10:28.057Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log
9602024-02-16T07:10:28.058Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log (24447 bytes)
9612024-02-16T07:10:29.304Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log
9622024-02-16T07:10:29.337Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log (1520798 bytes)
9632024-02-16T07:10:30.047Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log" changed size mid upload: 1520798 -> 1548129
9642024-02-16T07:10:31.130Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log
9652024-02-16T07:10:31.130Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log (876135 bytes)
9662024-02-16T07:10:31.905Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log" changed size mid upload: 876135 -> 889999
9672024-02-16T07:10:33.365Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log
9682024-02-16T07:10:33.366Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log (1388 bytes)
9692024-02-16T07:10:34.460Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log
9702024-02-16T07:10:34.460Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log (267856 bytes)
9712024-02-16T07:10:34.662Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log" changed size mid upload: 267856 -> 273900
9722024-02-16T07:10:35.745Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log
9732024-02-16T07:10:35.782Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log (617 bytes)
9742024-02-16T07:10:37.258Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log
9752024-02-16T07:10:37.259Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log (5392 bytes)
9762024-02-16T07:10:39.226Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log
9772024-02-16T07:10:39.226Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_64ded72d-bc8a-4810-b289-b5210197921e/root/var/svc/log/oxide-clickhouse:default.log (1191279 bytes)
9782024-02-16T07:10:39.665Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_64ded72d-bc8a-4810-b289-b5210197921e/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 1191279 -> 1242649
9792024-02-16T07:10:40.605Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_64ded72d-bc8a-4810-b289-b5210197921e/root/var/svc/log/oxide-clickhouse:default.log
9802024-02-16T07:10:40.606Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_64ded72d-bc8a-4810-b289-b5210197921e/root/var/svc/log/oxide-zone-network-setup:default.log (1708 bytes)
9812024-02-16T07:10:41.711Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_64ded72d-bc8a-4810-b289-b5210197921e/root/var/svc/log/oxide-zone-network-setup:default.log
9822024-02-16T07:10:41.740Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_10503827-57aa-4e38-8591-34b9522f64c1/root/var/svc/log/oxide-cockroachdb:default.log (4456 bytes)
9832024-02-16T07:10:42.818Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_10503827-57aa-4e38-8591-34b9522f64c1/root/var/svc/log/oxide-cockroachdb:default.log
9842024-02-16T07:10:42.818Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_10503827-57aa-4e38-8591-34b9522f64c1/root/var/svc/log/oxide-zone-network-setup:default.log (1705 bytes)
9852024-02-16T07:10:43.955Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_10503827-57aa-4e38-8591-34b9522f64c1/root/var/svc/log/oxide-zone-network-setup:default.log
9862024-02-16T07:10:43.955Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_08d5efd9-1a86-4b11-a0d0-0d93b60d80dc/root/var/svc/log/oxide-crucible-agent:default.log (27836 bytes)
9872024-02-16T07:10:45.116Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_08d5efd9-1a86-4b11-a0d0-0d93b60d80dc/root/var/svc/log/oxide-crucible-agent:default.log
9882024-02-16T07:10:45.117Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_08d5efd9-1a86-4b11-a0d0-0d93b60d80dc/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9892024-02-16T07:10:46.241Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_08d5efd9-1a86-4b11-a0d0-0d93b60d80dc/root/var/svc/log/oxide-crucible-downstairs:default.log
9902024-02-16T07:10:46.242Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_08d5efd9-1a86-4b11-a0d0-0d93b60d80dc/root/var/svc/log/oxide-crucible-downstairs:downstairs-30ef8df5-4489-421e-9a5e-98893b355ad4.log (5611 bytes)
9912024-02-16T07:10:47.335Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_08d5efd9-1a86-4b11-a0d0-0d93b60d80dc/root/var/svc/log/oxide-crucible-downstairs:downstairs-30ef8df5-4489-421e-9a5e-98893b355ad4.log
9922024-02-16T07:10:47.335Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_08d5efd9-1a86-4b11-a0d0-0d93b60d80dc/root/var/svc/log/oxide-crucible-downstairs:downstairs-c709deda-1414-4f58-a3d6-38ba03679d53.log (5692 bytes)
9932024-02-16T07:10:48.444Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_08d5efd9-1a86-4b11-a0d0-0d93b60d80dc/root/var/svc/log/oxide-crucible-downstairs:downstairs-c709deda-1414-4f58-a3d6-38ba03679d53.log
9942024-02-16T07:10:48.476Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_08d5efd9-1a86-4b11-a0d0-0d93b60d80dc/root/var/svc/log/oxide-crucible-downstairs:snapshot-c709deda-1414-4f58-a3d6-38ba03679d53-c5ff36fd-fda4-4d5b-8854-5b065543fc40.log (5305 bytes)
9952024-02-16T07:10:49.540Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_08d5efd9-1a86-4b11-a0d0-0d93b60d80dc/root/var/svc/log/oxide-crucible-downstairs:snapshot-c709deda-1414-4f58-a3d6-38ba03679d53-c5ff36fd-fda4-4d5b-8854-5b065543fc40.log
9962024-02-16T07:10:49.540Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_2fe79fc8-476d-4676-9095-5a487c5270f1/root/var/svc/log/oxide-external_dns:default.log (26936 bytes)
9972024-02-16T07:10:49.656Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_2fe79fc8-476d-4676-9095-5a487c5270f1/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 26936 -> 30712
9982024-02-16T07:10:50.730Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_2fe79fc8-476d-4676-9095-5a487c5270f1/root/var/svc/log/oxide-external_dns:default.log
9992024-02-16T07:10:50.730Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b30c21bb-8bc3-4926-89ff-16fa9d865c11/root/var/svc/log/oxide-internal_dns:default.log (108253 bytes)
10002024-02-16T07:10:50.892Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b30c21bb-8bc3-4926-89ff-16fa9d865c11/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 108253 -> 112009
10012024-02-16T07:10:51.966Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b30c21bb-8bc3-4926-89ff-16fa9d865c11/root/var/svc/log/oxide-internal_dns:default.log
10022024-02-16T07:10:51.966Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8bd8801a-18ce-48ba-9ef2-3cec813ea995/root/var/svc/log/oxide-nexus:default.log (1230103 bytes)
10032024-02-16T07:10:52.788Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8bd8801a-18ce-48ba-9ef2-3cec813ea995/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1230103 -> 1315757
10042024-02-16T07:10:53.860Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8bd8801a-18ce-48ba-9ef2-3cec813ea995/root/var/svc/log/oxide-nexus:default.log
10052024-02-16T07:10:53.860Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_88637054-2fdd-4a3c-b3a5-058e6e0fc436/root/var/svc/log/oxide-oximeter:default.log (322988 bytes)
10062024-02-16T07:10:54.100Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_88637054-2fdd-4a3c-b3a5-058e6e0fc436/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 322988 -> 347899
10072024-02-16T07:10:55.194Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_88637054-2fdd-4a3c-b3a5-058e6e0fc436/root/var/svc/log/oxide-oximeter:default.log
10082024-02-16T07:10:55.194Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_88637054-2fdd-4a3c-b3a5-058e6e0fc436/root/var/svc/log/oxide-zone-network-setup:default.log (1696 bytes)
10092024-02-16T07:10:56.291Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_88637054-2fdd-4a3c-b3a5-058e6e0fc436/root/var/svc/log/oxide-zone-network-setup:default.log
10102024-02-16T07:10:56.291Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_21d4fd23-096c-49f7-82bd-9d4d51c58401/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
10112024-02-16T07:10:57.393Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_21d4fd23-096c-49f7-82bd-9d4d51c58401/root/var/svc/log/oxide-cockroachdb:default.log
10122024-02-16T07:10:57.393Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_21d4fd23-096c-49f7-82bd-9d4d51c58401/root/var/svc/log/oxide-zone-network-setup:default.log (1704 bytes)
10132024-02-16T07:10:58.492Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_21d4fd23-096c-49f7-82bd-9d4d51c58401/root/var/svc/log/oxide-zone-network-setup:default.log
10142024-02-16T07:10:58.493Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ed4ad331-7705-4b9b-9dde-52e410173374/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10152024-02-16T07:10:59.591Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ed4ad331-7705-4b9b-9dde-52e410173374/root/var/svc/log/oxide-crucible-agent:default.log
10162024-02-16T07:10:59.591Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ed4ad331-7705-4b9b-9dde-52e410173374/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10172024-02-16T07:11:00.710Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ed4ad331-7705-4b9b-9dde-52e410173374/root/var/svc/log/oxide-crucible-downstairs:default.log
10182024-02-16T07:11:00.710Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_b5c12663-b663-4eae-957e-f7d9a47ec7d4/root/var/svc/log/oxide-external_dns:default.log (26936 bytes)
10192024-02-16T07:11:00.848Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_b5c12663-b663-4eae-957e-f7d9a47ec7d4/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 26936 -> 30713
10202024-02-16T07:11:01.934Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_b5c12663-b663-4eae-957e-f7d9a47ec7d4/root/var/svc/log/oxide-external_dns:default.log
10212024-02-16T07:11:01.934Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_62b2d7eb-013c-4138-be10-2e4de2464146/root/var/svc/log/oxide-internal_dns:default.log (113530 bytes)
10222024-02-16T07:11:02.127Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_62b2d7eb-013c-4138-be10-2e4de2464146/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 113530 -> 117284
10232024-02-16T07:11:03.201Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_62b2d7eb-013c-4138-be10-2e4de2464146/root/var/svc/log/oxide-internal_dns:default.log
10242024-02-16T07:11:03.201Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9391ae64-1acb-4daa-8a34-f254cb193cb6/root/var/svc/log/oxide-nexus:default.log (1228527 bytes)
10252024-02-16T07:11:03.611Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9391ae64-1acb-4daa-8a34-f254cb193cb6/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1228527 -> 1318090
10262024-02-16T07:11:04.681Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9391ae64-1acb-4daa-8a34-f254cb193cb6/root/var/svc/log/oxide-nexus:default.log
10272024-02-16T07:11:04.705Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_2cc36482-64d9-49f9-8342-269af02f542a/root/var/svc/log/oxide-cockroachdb:default.log (4482 bytes)
10282024-02-16T07:11:05.799Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_2cc36482-64d9-49f9-8342-269af02f542a/root/var/svc/log/oxide-cockroachdb:default.log
10292024-02-16T07:11:05.799Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_2cc36482-64d9-49f9-8342-269af02f542a/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
10302024-02-16T07:11:06.911Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_2cc36482-64d9-49f9-8342-269af02f542a/root/var/svc/log/oxide-zone-network-setup:default.log
10312024-02-16T07:11:06.912Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ab16e63a-5348-4997-80d6-1afa7667a236/root/var/svc/log/oxide-crucible-agent:default.log (23475 bytes)
10322024-02-16T07:11:08.048Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ab16e63a-5348-4997-80d6-1afa7667a236/root/var/svc/log/oxide-crucible-agent:default.log
10332024-02-16T07:11:08.048Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ab16e63a-5348-4997-80d6-1afa7667a236/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10342024-02-16T07:11:09.140Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ab16e63a-5348-4997-80d6-1afa7667a236/root/var/svc/log/oxide-crucible-downstairs:default.log
10352024-02-16T07:11:09.193Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ab16e63a-5348-4997-80d6-1afa7667a236/root/var/svc/log/oxide-crucible-downstairs:downstairs-9ae5bc01-4b77-4ad7-8518-ee45f44bd4d7.log (5694 bytes)
10362024-02-16T07:11:10.257Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ab16e63a-5348-4997-80d6-1afa7667a236/root/var/svc/log/oxide-crucible-downstairs:downstairs-9ae5bc01-4b77-4ad7-8518-ee45f44bd4d7.log
10372024-02-16T07:11:10.282Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ab16e63a-5348-4997-80d6-1afa7667a236/root/var/svc/log/oxide-crucible-downstairs:downstairs-ac5fc8c3-a73e-479a-b44a-b3c0116b30f2.log (3019 bytes)
10382024-02-16T07:11:11.351Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ab16e63a-5348-4997-80d6-1afa7667a236/root/var/svc/log/oxide-crucible-downstairs:downstairs-ac5fc8c3-a73e-479a-b44a-b3c0116b30f2.log
10392024-02-16T07:11:11.351Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ab16e63a-5348-4997-80d6-1afa7667a236/root/var/svc/log/oxide-crucible-downstairs:snapshot-9ae5bc01-4b77-4ad7-8518-ee45f44bd4d7-c5ff36fd-fda4-4d5b-8854-5b065543fc40.log (5305 bytes)
10402024-02-16T07:11:12.503Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ab16e63a-5348-4997-80d6-1afa7667a236/root/var/svc/log/oxide-crucible-downstairs:snapshot-9ae5bc01-4b77-4ad7-8518-ee45f44bd4d7-c5ff36fd-fda4-4d5b-8854-5b065543fc40.log
10412024-02-16T07:11:12.503Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_faec7c14-a96a-4e55-a346-ab5b27f1dd3c/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10422024-02-16T07:11:13.596Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_faec7c14-a96a-4e55-a346-ab5b27f1dd3c/root/var/svc/log/oxide-crucible-pantry:default.log
10432024-02-16T07:11:13.596Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_81afef0c-bfcd-4ddf-a253-fbf74130bdc6/root/var/svc/log/oxide-internal_dns:default.log (110280 bytes)
10442024-02-16T07:11:13.760Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_81afef0c-bfcd-4ddf-a253-fbf74130bdc6/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 110280 -> 114036
10452024-02-16T07:11:14.831Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_81afef0c-bfcd-4ddf-a253-fbf74130bdc6/root/var/svc/log/oxide-internal_dns:default.log
10462024-02-16T07:11:14.831Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_34f0afd4-b6d4-468b-bac9-5a05bf460779/root/var/svc/log/oxide-nexus:default.log (39391182 bytes)
10472024-02-16T07:11:23.794Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_34f0afd4-b6d4-468b-bac9-5a05bf460779/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 39391182 -> 39692626
10482024-02-16T07:11:24.865Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_34f0afd4-b6d4-468b-bac9-5a05bf460779/root/var/svc/log/oxide-nexus:default.log
10492024-02-16T07:11:24.890Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092/root/var/svc/log/oxide-ntp:default.log (847 bytes)
10502024-02-16T07:11:25.961Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092/root/var/svc/log/oxide-ntp:default.log
10512024-02-16T07:11:25.961Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_d2dd05a0-7998-4b4e-b483-b23c52dfdb2c/root/var/svc/log/oxide-cockroachdb:default.log (4406 bytes)
10522024-02-16T07:11:27.055Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_d2dd05a0-7998-4b4e-b483-b23c52dfdb2c/root/var/svc/log/oxide-cockroachdb:default.log
10532024-02-16T07:11:27.055Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_d2dd05a0-7998-4b4e-b483-b23c52dfdb2c/root/var/svc/log/oxide-zone-network-setup:default.log (1705 bytes)
10542024-02-16T07:11:28.146Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_d2dd05a0-7998-4b4e-b483-b23c52dfdb2c/root/var/svc/log/oxide-zone-network-setup:default.log
10552024-02-16T07:11:28.171Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_17fb421d-08d1-4991-aedd-9d8dda406f71/root/var/svc/log/oxide-crucible-agent:default.log (14583 bytes)
10562024-02-16T07:11:29.265Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_17fb421d-08d1-4991-aedd-9d8dda406f71/root/var/svc/log/oxide-crucible-agent:default.log
10572024-02-16T07:11:29.289Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_17fb421d-08d1-4991-aedd-9d8dda406f71/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10582024-02-16T07:11:29.518ZError: No response to keepalive - Terminating session
10592024-02-16T07:11:30.357Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_17fb421d-08d1-4991-aedd-9d8dda406f71/root/var/svc/log/oxide-crucible-downstairs:default.log
10602024-02-16T07:11:30.358Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_17fb421d-08d1-4991-aedd-9d8dda406f71/root/var/svc/log/oxide-crucible-downstairs:downstairs-e5553cc4-5ab5-4967-8f91-fc964a8c5747.log (5612 bytes)
10612024-02-16T07:11:31.451Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_17fb421d-08d1-4991-aedd-9d8dda406f71/root/var/svc/log/oxide-crucible-downstairs:downstairs-e5553cc4-5ab5-4967-8f91-fc964a8c5747.log
10622024-02-16T07:11:31.475Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_5be34856-ef50-4e5c-ad6a-349bf498d312/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
10632024-02-16T07:11:32.546Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_5be34856-ef50-4e5c-ad6a-349bf498d312/root/var/svc/log/oxide-cockroachdb:default.log
10642024-02-16T07:11:32.546Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_5be34856-ef50-4e5c-ad6a-349bf498d312/root/var/svc/log/oxide-zone-network-setup:default.log (1705 bytes)
10652024-02-16T07:11:33.640Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_5be34856-ef50-4e5c-ad6a-349bf498d312/root/var/svc/log/oxide-zone-network-setup:default.log
10662024-02-16T07:11:33.664Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a244a68e-123d-4e69-a61e-467a08d75e02/root/var/svc/log/oxide-crucible-agent:default.log (19612 bytes)
10672024-02-16T07:11:34.776Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a244a68e-123d-4e69-a61e-467a08d75e02/root/var/svc/log/oxide-crucible-agent:default.log
10682024-02-16T07:11:34.776Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a244a68e-123d-4e69-a61e-467a08d75e02/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10692024-02-16T07:11:35.869Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a244a68e-123d-4e69-a61e-467a08d75e02/root/var/svc/log/oxide-crucible-downstairs:default.log
10702024-02-16T07:11:35.869Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a244a68e-123d-4e69-a61e-467a08d75e02/root/var/svc/log/oxide-crucible-downstairs:downstairs-4846bc02-2e5b-42be-8178-48bbe87df59d.log (3125 bytes)
10712024-02-16T07:11:36.978Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a244a68e-123d-4e69-a61e-467a08d75e02/root/var/svc/log/oxide-crucible-downstairs:downstairs-4846bc02-2e5b-42be-8178-48bbe87df59d.log
10722024-02-16T07:11:36.978Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a244a68e-123d-4e69-a61e-467a08d75e02/root/var/svc/log/oxide-crucible-downstairs:downstairs-8b06923f-9a49-4986-9172-6964a218a12c.log (3018 bytes)
10732024-02-16T07:11:38.108Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a244a68e-123d-4e69-a61e-467a08d75e02/root/var/svc/log/oxide-crucible-downstairs:downstairs-8b06923f-9a49-4986-9172-6964a218a12c.log
10742024-02-16T07:11:38.142Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_10968697-a7e8-4d79-9cf9-2ac2c5ee91b0/root/var/svc/log/oxide-crucible-pantry:default.log (1218788 bytes)
10752024-02-16T07:11:39.541Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_10968697-a7e8-4d79-9cf9-2ac2c5ee91b0/root/var/svc/log/oxide-crucible-pantry:default.log
10762024-02-16T07:11:39.541Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a98b71d2-3e2f-4fb6-90c5-2e50de4d705b/root/var/svc/log/oxide-crucible-agent:default.log (14579 bytes)
10772024-02-16T07:11:39.611ZError: No response de-activating SOL payload
10782024-02-16T07:11:40.668Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a98b71d2-3e2f-4fb6-90c5-2e50de4d705b/root/var/svc/log/oxide-crucible-agent:default.log
10792024-02-16T07:11:40.668Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a98b71d2-3e2f-4fb6-90c5-2e50de4d705b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10802024-02-16T07:11:40.737Z[SOL Session operational. Use ~? for help]
10812024-02-16T07:11:41.775Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a98b71d2-3e2f-4fb6-90c5-2e50de4d705b/root/var/svc/log/oxide-crucible-downstairs:default.log
10822024-02-16T07:11:41.775Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a98b71d2-3e2f-4fb6-90c5-2e50de4d705b/root/var/svc/log/oxide-crucible-downstairs:downstairs-14898c1d-6568-4472-80a0-bb67e5306dca.log (5611 bytes)
10832024-02-16T07:11:42.867Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a98b71d2-3e2f-4fb6-90c5-2e50de4d705b/root/var/svc/log/oxide-crucible-downstairs:downstairs-14898c1d-6568-4472-80a0-bb67e5306dca.log
10842024-02-16T07:11:42.867Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2c8f8f6b-7426-4579-b888-98c5d4f92e15/root/var/svc/log/oxide-crucible-agent:default.log (13686 bytes)
10852024-02-16T07:11:43.984Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2c8f8f6b-7426-4579-b888-98c5d4f92e15/root/var/svc/log/oxide-crucible-agent:default.log
10862024-02-16T07:11:43.984Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2c8f8f6b-7426-4579-b888-98c5d4f92e15/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10872024-02-16T07:11:45.079Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2c8f8f6b-7426-4579-b888-98c5d4f92e15/root/var/svc/log/oxide-crucible-downstairs:default.log
10882024-02-16T07:11:45.079Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2c8f8f6b-7426-4579-b888-98c5d4f92e15/root/var/svc/log/oxide-crucible-downstairs:downstairs-93e97db7-8bfd-4738-a28c-a0c558469bea.log (3126 bytes)
10892024-02-16T07:11:46.172Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2c8f8f6b-7426-4579-b888-98c5d4f92e15/root/var/svc/log/oxide-crucible-downstairs:downstairs-93e97db7-8bfd-4738-a28c-a0c558469bea.log
10902024-02-16T07:11:46.172Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_643e9df1-7d24-4551-813d-7c58f246ccbc/root/var/svc/log/oxide-crucible-agent:default.log (20514 bytes)
10912024-02-16T07:11:47.313Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_643e9df1-7d24-4551-813d-7c58f246ccbc/root/var/svc/log/oxide-crucible-agent:default.log
10922024-02-16T07:11:47.314Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_643e9df1-7d24-4551-813d-7c58f246ccbc/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10932024-02-16T07:11:48.409Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_643e9df1-7d24-4551-813d-7c58f246ccbc/root/var/svc/log/oxide-crucible-downstairs:default.log
10942024-02-16T07:11:48.409Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_643e9df1-7d24-4551-813d-7c58f246ccbc/root/var/svc/log/oxide-crucible-downstairs:downstairs-123be6e7-fe64-4260-87a8-72eae8e9f906.log (3018 bytes)
10952024-02-16T07:11:49.501Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_643e9df1-7d24-4551-813d-7c58f246ccbc/root/var/svc/log/oxide-crucible-downstairs:downstairs-123be6e7-fe64-4260-87a8-72eae8e9f906.log
10962024-02-16T07:11:49.525Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_643e9df1-7d24-4551-813d-7c58f246ccbc/root/var/svc/log/oxide-crucible-downstairs:downstairs-cb2654c1-0214-46e1-aaf0-733fbb9be756.log (3126 bytes)
10972024-02-16T07:11:50.610Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_643e9df1-7d24-4551-813d-7c58f246ccbc/root/var/svc/log/oxide-crucible-downstairs:downstairs-cb2654c1-0214-46e1-aaf0-733fbb9be756.log
10982024-02-16T07:11:50.648Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_d3f2ecd3-c994-4a91-b7a8-ca8b1c02475c/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10992024-02-16T07:11:51.752Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_d3f2ecd3-c994-4a91-b7a8-ca8b1c02475c/root/var/svc/log/oxide-crucible-pantry:default.log
11002024-02-16T07:11:51.787Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c093a4e8-46ea-4e77-a429-3cf96c6a0a25/root/var/svc/log/oxide-crucible-agent:default.log (17118 bytes)
11012024-02-16T07:11:52.922Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c093a4e8-46ea-4e77-a429-3cf96c6a0a25/root/var/svc/log/oxide-crucible-agent:default.log
11022024-02-16T07:11:52.922Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c093a4e8-46ea-4e77-a429-3cf96c6a0a25/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11032024-02-16T07:11:54.070Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c093a4e8-46ea-4e77-a429-3cf96c6a0a25/root/var/svc/log/oxide-crucible-downstairs:default.log
11042024-02-16T07:11:54.070Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c093a4e8-46ea-4e77-a429-3cf96c6a0a25/root/var/svc/log/oxide-crucible-downstairs:downstairs-dc4c94a1-8fb5-4950-b6de-e83fac607f24.log (5690 bytes)
11052024-02-16T07:11:55.244Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c093a4e8-46ea-4e77-a429-3cf96c6a0a25/root/var/svc/log/oxide-crucible-downstairs:downstairs-dc4c94a1-8fb5-4950-b6de-e83fac607f24.log
11062024-02-16T07:11:55.244Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c093a4e8-46ea-4e77-a429-3cf96c6a0a25/root/var/svc/log/oxide-crucible-downstairs:snapshot-dc4c94a1-8fb5-4950-b6de-e83fac607f24-c5ff36fd-fda4-4d5b-8854-5b065543fc40.log (5302 bytes)
11072024-02-16T07:11:56.366Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c093a4e8-46ea-4e77-a429-3cf96c6a0a25/root/var/svc/log/oxide-crucible-downstairs:snapshot-dc4c94a1-8fb5-4950-b6de-e83fac607f24-c5ff36fd-fda4-4d5b-8854-5b065543fc40.log
11082024-02-16T07:11:56.366Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092/root/var/log/chrony/measurements.log (20413 bytes)
11092024-02-16T07:11:56.488Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092/root/var/log/chrony/measurements.log" changed size mid upload: 20413 -> 23564
11102024-02-16T07:11:57.561Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092/root/var/log/chrony/measurements.log
11112024-02-16T07:11:57.561Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092/root/var/log/chrony/statistics.log (13338 bytes)
11122024-02-16T07:11:57.676Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092/root/var/log/chrony/statistics.log" changed size mid upload: 13338 -> 15912
11132024-02-16T07:11:58.773Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092/root/var/log/chrony/statistics.log
11142024-02-16T07:11:58.774Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092/root/var/log/chrony/tracking.log (1849 bytes)
11152024-02-16T07:11:58.823Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092/root/var/log/chrony/tracking.log" changed size mid upload: 1849 -> 1981
11162024-02-16T07:11:59.902Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_c327c44a-8044-4bad-bedf-47ac60177092/root/var/log/chrony/tracking.log