01HN6T8VY5W2WBATYV3SFAHAJV: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.28",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HN6T9A7ZTFHB4J7G2PP4WAQM

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12024-01-28T05:04:25.527Zjob dependencies complete; ready to run (waiting for 3 h 46 m 52 s)
22024-01-28T05:09:34.124Zjob assigned to worker 01HN778RTXVM8YDJ1GHJVKSVVC (queued for 5 m 8 s)
32024-01-28T05:09:34.298Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42024-01-28T05:09:34.348Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52024-01-28T05:09:34.381Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62024-01-28T05:09:34.415Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72024-01-28T05:09:34.448Zhttp://172.20.0.5:9969/postboot/buskin... ok
82024-01-28T05:09:34.482ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92024-01-28T05:09:34.515Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102024-01-28T05:09:34.549ZOxide Helios Version helios-2.0.22418 64-bit
112024-01-28T05:09:34.582ZConfiguring devices.
122024-01-28T05:09:34.616Z##################################################
132024-01-28T05:09:34.649Z#### ############# ##
142024-01-28T05:09:34.684Z### ### ############ ##
152024-01-28T05:09:34.717Z## ### # ## ### ## Oxide Computer Company ##
162024-01-28T05:09:34.752Z## ## ## ### # ### ##
172024-01-28T05:09:34.789Z## # ### #### #### This Station Under ##
182024-01-28T05:09:34.822Z### ### #### # ### Computer Control ##
192024-01-28T05:09:34.855Z#### #### ### ## ##
202024-01-28T05:09:34.889Z##################################################
212024-01-28T05:09:34.922ZHostname: buskin
222024-01-28T05:09:34.956ZLoading smf(7) service descriptions:
232024-01-28T05:09:35.652Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-3358a8ad2ce34473.gz
242024-01-28T05:09:37.554Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-3358a8ad2ce34473.gz
252024-01-28T05:09:37.554Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262024-01-28T05:09:55.884Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272024-01-28T05:09:55.884Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-71427024e0379c76.gz
282024-01-28T05:10:14.569Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-71427024e0379c76.gz
292024-01-28T05:10:14.569Zdownloading input: /input/ci-tools/work/caboose-util.gz
302024-01-28T05:10:15.362Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312024-01-28T05:10:15.362Zdownloading input: /input/ci-tools/work/tufaceous.gz
322024-01-28T05:10:18.092Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332024-01-28T05:10:18.115Zdownloading input: /input/package/work/version.txt
342024-01-28T05:10:18.138Zdownloaded input: /input/package/work/version.txt
352024-01-28T05:10:18.139Zdownloading input: /input/package/work/package.tar.gz
362024-01-28T05:10:32.746Zdownloaded input: /input/package/work/package.tar.gz
372024-01-28T05:10:32.746Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382024-01-28T05:10:54.179Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392024-01-28T05:10:54.180Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402024-01-28T05:11:00.895Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412024-01-28T05:11:00.924Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422024-01-28T05:11:39.565Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432024-01-28T05:11:39.565Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442024-01-28T05:12:13.944Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452024-01-28T05:12:13.944Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462024-01-28T05:12:37.567Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472024-01-28T05:12:37.593Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482024-01-28T05:12:41.267Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492024-01-28T05:12:41.267Zdownloading input: /input/package/work/zones/crucible.tar.gz
502024-01-28T05:12:46.774Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512024-01-28T05:12:46.774Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522024-01-28T05:12:51.097Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532024-01-28T05:12:51.097Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542024-01-28T05:12:55.224Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552024-01-28T05:12:55.225Zdownloading input: /input/package/work/zones/ntp.tar.gz
562024-01-28T05:12:55.267Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572024-01-28T05:12:55.268Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582024-01-28T05:12:58.692Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592024-01-28T05:12:58.692Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602024-01-28T05:13:02.199Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612024-01-28T05:13:02.199Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622024-01-28T05:13:17.572Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632024-01-28T05:13:17.596Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642024-01-28T05:13:36.686Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652024-01-28T05:13:36.686Zdownloading input: /input/package/work/zones/overlay.tar.gz
662024-01-28T05:13:36.786Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672024-01-28T05:13:36.786Zdownloading input: /input/package/work/zones/oximeter.tar.gz
682024-01-28T05:13:46.727Zdownloaded input: /input/package/work/zones/oximeter.tar.gz
692024-01-28T05:13:47.203Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702024-01-28T05:13:58.132Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712024-01-28T05:13:58.132Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722024-01-28T05:14:47.796Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732024-01-28T05:14:47.825Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742024-01-28T05:15:24.092Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752024-01-28T05:15:24.092Zstarting task 0: "setup"
762024-01-28T05:15:24.116Z++ uname -s
772024-01-28T05:15:24.116Z+ kern=SunOS
782024-01-28T05:15:24.116Z+ case "$kern" in
792024-01-28T05:15:24.116Z+ groupadd -g 12345 build
802024-01-28T05:15:24.116Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812024-01-28T05:15:26.109Z+ zfs create -o mountpoint=/work rpool/work
822024-01-28T05:15:26.134Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832024-01-28T05:15:26.157Z+ home_fs=zfs
842024-01-28T05:15:26.157Z+ [[ zfs == autofs ]]
852024-01-28T05:15:26.157Z+ mkdir -p /home/build
862024-01-28T05:15:26.157Z+ chown build:build /home/build /work
872024-01-28T05:15:28.142Z+ chmod 0700 /home/build /work
882024-01-28T05:15:28.165Zprocess exited: duration 4054 ms, exit code 0
 
892024-01-28T05:15:28.212Zstarting task 1: "authentication"
902024-01-28T05:15:28.256Zprocess exited: duration 43 ms, exit code 0
 
912024-01-28T05:15:28.304Zstarting task 2: "build"
922024-01-28T05:15:28.327Z+ trap _exit_trap EXIT
932024-01-28T05:15:28.327Z+ set +x
942024-01-28T05:15:52.397Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
952024-01-28T05:15:52.738Zsecpolicy_fs_linkdir: 0x55 = 0x55
962024-01-28T05:15:52.739Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
972024-01-28T05:15:52.739Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
982024-01-28T05:15:52.739Zsecpolicy_fs_linkdir+3: 0 = 0xe5
992024-01-28T05:15:52.739Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1002024-01-28T05:15:52.739Zsecpolicy_fs_linkdir+5: 0 = 0x1
1012024-01-28T05:15:52.739Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1022024-01-28T05:15:52.739Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1032024-01-28T05:15:52.763Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1042024-01-28T05:15:52.763Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1052024-01-28T05:15:52.763Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1062024-01-28T05:15:52.763Z+ [[ -d /opt/oxide ]]
1072024-01-28T05:15:52.763Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1082024-01-28T05:15:52.763Z+ pfexec mkdir /opt/oxide
1092024-01-28T05:15:52.763Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1102024-01-28T05:15:52.764Z+ [[ -d /opt/oxide-underneath ]]
1112024-01-28T05:15:52.764Z+ cd /opt/oxide-underneath
1122024-01-28T05:15:52.764Z+ pfexec tar ceEp@/f - .
1132024-01-28T05:15:52.764Z+ cd /opt/oxide
1142024-01-28T05:15:52.764Z+ pfexec tar xveEp@/f -
1152024-01-28T05:15:52.764Zx ., 0 bytes, 0 tape blocks
1162024-01-28T05:15:52.764Zx ./opte, 0 bytes, 0 tape blocks
1172024-01-28T05:15:52.764Zx ./opte/bin, 0 bytes, 0 tape blocks
1182024-01-28T05:15:52.764Zx ./opte/bin/opteadm, 51799224 bytes, 101171 tape blocks
1192024-01-28T05:15:52.764Z+ pfexec rm -rf /opt/oxide-underneath
1202024-01-28T05:15:52.764Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1212024-01-28T05:15:52.764Z+ pfexec mkdir /opt/oxide/work
1222024-01-28T05:15:52.764Z+ pfexec chown build:build /opt/oxide/work
1232024-01-28T05:15:52.764Z+ cd /opt/oxide/work
1242024-01-28T05:15:52.764Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1252024-01-28T05:15:54.413ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1262024-01-28T05:15:54.414Zx out/mg-ddm-gz.tar, 63063552 bytes, 123171 tape blocks
1272024-01-28T05:15:54.414Zx out/omicron-sled-agent.tar, 122220032 bytes, 238711 tape blocks
1282024-01-28T05:15:54.414Zx out/oxlog.tar, 8382464 bytes, 16372 tape blocks
1292024-01-28T05:15:54.414Zx out/target/test, 75 bytes, 1 tape blocks
1302024-01-28T05:15:54.414Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1312024-01-28T05:15:54.414Zx package-manifest.toml, 23732 bytes, 47 tape blocks
1322024-01-28T05:15:54.414Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1332024-01-28T05:15:54.414Zx target/release/omicron-package, 44584112 bytes, 87079 tape blocks
1342024-01-28T05:15:54.414Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1352024-01-28T05:15:54.414Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1362024-01-28T05:15:54.414Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1372024-01-28T05:15:54.414Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1382024-01-28T05:15:54.414Zx tools/scrimlet/softnpu-init.sh, 2149 bytes, 5 tape blocks
1392024-01-28T05:15:54.414Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1402024-01-28T05:15:54.414Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1412024-01-28T05:15:54.439Z
1422024-01-28T05:15:54.439Zreal 1.667331549
1432024-01-28T05:15:54.439Zuser 1.501522235
1442024-01-28T05:15:54.439Zsys 0.296034439
1452024-01-28T05:15:54.440Ztrap 0.000076911
1462024-01-28T05:15:54.440Ztflt 0.000000000
1472024-01-28T05:15:54.440Zdflt 0.000000000
1482024-01-28T05:15:54.440Zkflt 0.000000000
1492024-01-28T05:15:54.440Zlock 0.000000000
1502024-01-28T05:15:54.440Zslp 1.495983473
1512024-01-28T05:15:54.440Zlat 0.039012981
1522024-01-28T05:15:54.440Zstop 0.000125819
1532024-01-28T05:15:54.440Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1542024-01-28T05:15:55.634Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1552024-01-28T05:15:55.658Z+ mkdir tests
1562024-01-28T05:15:55.658Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1572024-01-28T05:15:55.658Z+ ptime -m gunzip
1582024-01-28T05:15:55.659Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-3358a8ad2ce34473
1592024-01-28T05:15:55.891Z
1602024-01-28T05:15:55.915Zreal 0.238192549
1612024-01-28T05:15:55.915Zuser 0.220994308
1622024-01-28T05:15:55.915Zsys 0.016935838
1632024-01-28T05:15:55.915Ztrap 0.000018339
1642024-01-28T05:15:55.915Ztflt 0.000000000
1652024-01-28T05:15:55.915Zdflt 0.000000000
1662024-01-28T05:15:55.915Zkflt 0.000000000
1672024-01-28T05:15:55.915Zlock 0.000000000
1682024-01-28T05:15:55.915Zslp 0.000022050
1692024-01-28T05:15:55.915Zlat 0.000024640
1702024-01-28T05:15:55.915Zstop 0.000127941
1712024-01-28T05:15:55.915Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-3358a8ad2ce34473
1722024-01-28T05:15:55.915Z+ chmod a+x tests/bootstrap-3358a8ad2ce34473
1732024-01-28T05:15:55.915Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1742024-01-28T05:15:55.915Z+ ptime -m gunzip
1752024-01-28T05:15:55.916Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1762024-01-28T05:15:58.453Z
1772024-01-28T05:15:58.480Zreal 2.550572365
1782024-01-28T05:15:58.480Zuser 2.404039395
1792024-01-28T05:15:58.481Zsys 0.146119804
1802024-01-28T05:15:58.481Ztrap 0.000181341
1812024-01-28T05:15:58.481Ztflt 0.000000000
1822024-01-28T05:15:58.481Zdflt 0.000000000
1832024-01-28T05:15:58.481Zkflt 0.000000000
1842024-01-28T05:15:58.481Zlock 0.000000000
1852024-01-28T05:15:58.484Zslp 0.000012550
1862024-01-28T05:15:58.484Zlat 0.000023570
1872024-01-28T05:15:58.484Zstop 0.000127461
1882024-01-28T05:15:58.484Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1892024-01-28T05:15:58.484Z+ chmod a+x tests/bootstrap
1902024-01-28T05:15:58.484Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1912024-01-28T05:15:58.484Z+ ptime -m gunzip
1922024-01-28T05:15:58.484Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-71427024e0379c76
1932024-01-28T05:16:01.026Z
1942024-01-28T05:16:01.050Zreal 2.561091646
1952024-01-28T05:16:01.050Zuser 2.404997079
1962024-01-28T05:16:01.050Zsys 0.155674585
1972024-01-28T05:16:01.050Ztrap 0.000190249
1982024-01-28T05:16:01.051Ztflt 0.000000000
1992024-01-28T05:16:01.051Zdflt 0.000000000
2002024-01-28T05:16:01.051Zkflt 0.000000000
2012024-01-28T05:16:01.051Zlock 0.000000000
2022024-01-28T05:16:01.051Zslp 0.000022350
2032024-01-28T05:16:01.051Zlat 0.000016180
2042024-01-28T05:16:01.051Zstop 0.000120611
2052024-01-28T05:16:01.051Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-71427024e0379c76
2062024-01-28T05:16:01.051Z+ chmod a+x tests/end_to_end_tests-71427024e0379c76
2072024-01-28T05:16:01.051Z+ bmat address ls
2082024-01-28T05:16:01.051ZNAME CIDR FIRST LAST COUNT
2092024-01-28T05:16:01.051Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2102024-01-28T05:16:01.054Z++ bmat address ls -f extra -Ho first,last,count
2112024-01-28T05:16:01.080Z+ set -- 10.151.1.164 10.151.1.227 64
2122024-01-28T05:16:01.080Z+ EXTRA_IP_START=10.151.1.164
2132024-01-28T05:16:01.080Z+ EXTRA_IP_END=10.151.1.227
2142024-01-28T05:16:01.080Z+ EXTRA_IP_COUNT=64
2152024-01-28T05:16:01.080Z+ (( EXTRA_IP_COUNT >= 32 ))
2162024-01-28T05:16:01.080Z+ EXTRA_IP_BASE=10.151.1
2172024-01-28T05:16:01.080Z+ EXTRA_IP_FOCTET=164
2182024-01-28T05:16:01.080Z+ EXTRA_IP_LOCTET=227
2192024-01-28T05:16:01.080Z+ SERVICE_IP_POOL_START=10.151.1.164
2202024-01-28T05:16:01.080Z+ SERVICE_IP_POOL_END=10.151.1.173
2212024-01-28T05:16:01.080Z+ DNS_IP1=10.151.1.164
2222024-01-28T05:16:01.080Z+ DNS_IP2=10.151.1.165
2232024-01-28T05:16:01.080Z+ UPLINK_IP=10.151.1.174
2242024-01-28T05:16:01.081Z+ IPPOOL_START=10.151.1.175
2252024-01-28T05:16:01.081Z+ IPPOOL_END=10.151.1.227
2262024-01-28T05:16:01.081Z++ ipadm show-addr -po type,addr
2272024-01-28T05:16:01.081Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2282024-01-28T05:16:01.081Z+ GATEWAY_IP=10.151.1.100
2292024-01-28T05:16:01.081Z+ [[ -n 10.151.1.100 ]]
2302024-01-28T05:16:01.081Z+ ping -s 10.151.1.100 56 1
2312024-01-28T05:16:01.082ZPING 10.151.1.100: 56 data bytes
2322024-01-28T05:16:01.082Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
2332024-01-28T05:16:01.082Z
2342024-01-28T05:16:01.083Z----10.151.1.100 PING Statistics----
2352024-01-28T05:16:01.083Z1 packets transmitted, 1 packets received, 0% packet loss
2362024-01-28T05:16:01.083Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2372024-01-28T05:16:01.083Z++ arp -an
2382024-01-28T05:16:01.083Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2392024-01-28T05:16:01.083Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2402024-01-28T05:16:01.083Z+ routeadm -e ipv4-forwarding -u
2412024-01-28T05:16:01.191Z+ PXA_START=10.151.1.164
2422024-01-28T05:16:01.191Z+ PXA_END=10.151.1.227
2432024-01-28T05:16:01.191Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2442024-01-28T05:16:01.216Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2452024-01-28T05:16:01.389Z+ ZPOOL_VDEV_DIR=/scratch
2462024-01-28T05:16:01.389Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2472024-01-28T05:16:01.413Z+++ dirname ./tools/create_virtual_hardware.sh
2482024-01-28T05:16:01.413Z++ cd ./tools
2492024-01-28T05:16:01.413Z++ pwd
2502024-01-28T05:16:01.413Z+ SOURCE_DIR=/opt/oxide/work/tools
2512024-01-28T05:16:01.413Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2522024-01-28T05:16:01.413Z+ SOFTNPU_MODE=zone
2532024-01-28T05:16:01.413Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2542024-01-28T05:16:01.413Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2552024-01-28T05:16:01.413Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2562024-01-28T05:16:01.413Z++ dladm show-phys -p -o LINK
2572024-01-28T05:16:01.413Z++ head -1
2582024-01-28T05:16:01.437Z+ PHYSICAL_LINK=igb0
2592024-01-28T05:16:01.437Z+ echo 'Using igb0 as physical link'
2602024-01-28T05:16:01.437ZUsing igb0 as physical link
2612024-01-28T05:16:01.437Z+ ensure_run_as_root
2622024-01-28T05:16:01.437Z++ id -u
2632024-01-28T05:16:01.437Z+ [[ 0 -ne 0 ]]
2642024-01-28T05:16:01.437Z+ ensure_zpools
2652024-01-28T05:16:01.437Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2662024-01-28T05:16:01.438Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2672024-01-28T05:16:01.438Z+ readarray -t ZPOOLS
2682024-01-28T05:16:01.438Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2692024-01-28T05:16:01.439Z++ sed 's/[ ",]//g'
2702024-01-28T05:16:01.439Z+ for ZPOOL in "${ZPOOLS[@]}"
2712024-01-28T05:16:01.439Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2722024-01-28T05:16:01.439ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2732024-01-28T05:16:01.439Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2742024-01-28T05:16:01.439Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2752024-01-28T05:16:01.439Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2762024-01-28T05:16:01.439Z0+0 records in
2772024-01-28T05:16:01.439Z0+0 records out
2782024-01-28T05:16:01.439Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2792024-01-28T05:16:01.439Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2802024-01-28T05:16:01.439Z+ set +x
2812024-01-28T05:16:01.439ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2822024-01-28T05:16:01.439Z++ zpool list -o name
2832024-01-28T05:16:01.439Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2842024-01-28T05:16:01.439Z+ [[ -z '' ]]
2852024-01-28T05:16:01.439Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2862024-01-28T05:16:01.478Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2872024-01-28T05:16:01.502Z+ set +x
2882024-01-28T05:16:01.502ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2892024-01-28T05:16:01.503Z+ for ZPOOL in "${ZPOOLS[@]}"
2902024-01-28T05:16:01.503Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2912024-01-28T05:16:01.503ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2922024-01-28T05:16:01.503Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2932024-01-28T05:16:01.503Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2942024-01-28T05:16:01.503Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2952024-01-28T05:16:01.503Z0+0 records in
2962024-01-28T05:16:01.504Z0+0 records out
2972024-01-28T05:16:01.504Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2982024-01-28T05:16:01.504Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2992024-01-28T05:16:01.504Z+ set +x
3002024-01-28T05:16:01.504ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3012024-01-28T05:16:01.504Z++ zpool list -o name
3022024-01-28T05:16:01.504Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3032024-01-28T05:16:01.504Z+ [[ -z '' ]]
3042024-01-28T05:16:01.504Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3052024-01-28T05:16:01.528Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3062024-01-28T05:16:01.528Z+ set +x
3072024-01-28T05:16:01.529ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3082024-01-28T05:16:01.529Z+ for ZPOOL in "${ZPOOLS[@]}"
3092024-01-28T05:16:01.529Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3102024-01-28T05:16:01.529ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3112024-01-28T05:16:01.529Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3122024-01-28T05:16:01.529Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3132024-01-28T05:16:01.529Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3142024-01-28T05:16:01.529Z0+0 records in
3152024-01-28T05:16:01.529Z0+0 records out
3162024-01-28T05:16:01.529Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3172024-01-28T05:16:01.529Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3182024-01-28T05:16:01.529Z+ set +x
3192024-01-28T05:16:01.529ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3202024-01-28T05:16:01.529Z++ zpool list -o name
3212024-01-28T05:16:01.529Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3222024-01-28T05:16:01.553Z+ [[ -z '' ]]
3232024-01-28T05:16:01.553Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3242024-01-28T05:16:01.579Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3252024-01-28T05:16:01.579Z+ set +x
3262024-01-28T05:16:01.579ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3272024-01-28T05:16:01.579Z+ for ZPOOL in "${ZPOOLS[@]}"
3282024-01-28T05:16:01.580ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3292024-01-28T05:16:01.580Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3302024-01-28T05:16:01.580Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3312024-01-28T05:16:01.580Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3322024-01-28T05:16:01.580Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3332024-01-28T05:16:01.580Z0+0 records in
3342024-01-28T05:16:01.580Z0+0 records out
3352024-01-28T05:16:01.580Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3362024-01-28T05:16:01.580Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3372024-01-28T05:16:01.580Z+ set +x
3382024-01-28T05:16:01.580ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3392024-01-28T05:16:01.581Z++ zpool list -o name
3402024-01-28T05:16:01.581Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3412024-01-28T05:16:01.581Z+ [[ -z '' ]]
3422024-01-28T05:16:01.581Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3432024-01-28T05:16:01.606Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3442024-01-28T05:16:01.606Z+ set +x
3452024-01-28T05:16:01.606ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3462024-01-28T05:16:01.606Z+ for ZPOOL in "${ZPOOLS[@]}"
3472024-01-28T05:16:01.606Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3482024-01-28T05:16:01.606ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3492024-01-28T05:16:01.606Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502024-01-28T05:16:01.606Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3512024-01-28T05:16:01.606Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3522024-01-28T05:16:01.606Z0+0 records in
3532024-01-28T05:16:01.606Z0+0 records out
3542024-01-28T05:16:01.606Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3552024-01-28T05:16:01.607Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3562024-01-28T05:16:01.607Z+ set +x
3572024-01-28T05:16:01.607ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3582024-01-28T05:16:01.607Z++ zpool list -o name
3592024-01-28T05:16:01.631Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3602024-01-28T05:16:01.631Z+ [[ -z '' ]]
3612024-01-28T05:16:01.631Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3622024-01-28T05:16:01.655Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3632024-01-28T05:16:01.656Z+ set +x
3642024-01-28T05:16:01.656ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3652024-01-28T05:16:01.656Z+ for ZPOOL in "${ZPOOLS[@]}"
3662024-01-28T05:16:01.656Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3672024-01-28T05:16:01.656ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3682024-01-28T05:16:01.656Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3692024-01-28T05:16:01.656Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3702024-01-28T05:16:01.657Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3712024-01-28T05:16:01.657Z0+0 records in
3722024-01-28T05:16:01.657Z0+0 records out
3732024-01-28T05:16:01.657Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3742024-01-28T05:16:01.657Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3752024-01-28T05:16:01.657Z+ set +x
3762024-01-28T05:16:01.657ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3772024-01-28T05:16:01.657Z++ zpool list -o name
3782024-01-28T05:16:01.657Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3792024-01-28T05:16:01.682Z+ [[ -z '' ]]
3802024-01-28T05:16:01.682Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3812024-01-28T05:16:01.708Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3822024-01-28T05:16:01.708Z+ set +x
3832024-01-28T05:16:01.708Z+ for ZPOOL in "${ZPOOLS[@]}"
3842024-01-28T05:16:01.708ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3852024-01-28T05:16:01.708Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3862024-01-28T05:16:01.708ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3872024-01-28T05:16:01.708Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3882024-01-28T05:16:01.709Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3892024-01-28T05:16:01.709Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3902024-01-28T05:16:01.709Z0+0 records in
3912024-01-28T05:16:01.709Z0+0 records out
3922024-01-28T05:16:01.709Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3932024-01-28T05:16:01.709Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3942024-01-28T05:16:01.709Z+ set +x
3952024-01-28T05:16:01.709ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3962024-01-28T05:16:01.709Z++ zpool list -o name
3972024-01-28T05:16:01.709Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3982024-01-28T05:16:01.709Z+ [[ -z '' ]]
3992024-01-28T05:16:01.709Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4002024-01-28T05:16:01.733Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4012024-01-28T05:16:01.733Z+ set +x
4022024-01-28T05:16:01.733ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4032024-01-28T05:16:01.734Z+ for ZPOOL in "${ZPOOLS[@]}"
4042024-01-28T05:16:01.734Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4052024-01-28T05:16:01.734ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4062024-01-28T05:16:01.734Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4072024-01-28T05:16:01.734Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4082024-01-28T05:16:01.734Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4092024-01-28T05:16:01.734Z0+0 records in
4102024-01-28T05:16:01.734Z0+0 records out
4112024-01-28T05:16:01.734Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4122024-01-28T05:16:01.734Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4132024-01-28T05:16:01.734Z+ set +x
4142024-01-28T05:16:01.734ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4152024-01-28T05:16:01.734Z++ zpool list -o name
4162024-01-28T05:16:01.734Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4172024-01-28T05:16:01.758Z+ [[ -z '' ]]
4182024-01-28T05:16:01.758Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192024-01-28T05:16:01.782Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4202024-01-28T05:16:01.782Z+ set +x
4212024-01-28T05:16:01.782ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4222024-01-28T05:16:01.782Z+ for ZPOOL in "${ZPOOLS[@]}"
4232024-01-28T05:16:01.782Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4242024-01-28T05:16:01.782ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4252024-01-28T05:16:01.782Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4262024-01-28T05:16:01.783Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4272024-01-28T05:16:01.783Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4282024-01-28T05:16:01.783Z0+0 records in
4292024-01-28T05:16:01.783Z0+0 records out
4302024-01-28T05:16:01.783Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4312024-01-28T05:16:01.783Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4322024-01-28T05:16:01.783Z+ set +x
4332024-01-28T05:16:01.783ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4342024-01-28T05:16:01.783Z++ zpool list -o name
4352024-01-28T05:16:01.783Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4362024-01-28T05:16:01.783Z+ [[ -z '' ]]
4372024-01-28T05:16:01.783Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4382024-01-28T05:16:01.809Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4392024-01-28T05:16:01.809Z+ set +x
4402024-01-28T05:16:01.834ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4412024-01-28T05:16:01.834Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4422024-01-28T05:16:01.834Z+ readarray -t ZPOOLS
4432024-01-28T05:16:01.834Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4442024-01-28T05:16:01.834Z++ sed 's/[ ",]//g'
4452024-01-28T05:16:01.834Z+ for ZPOOL in "${ZPOOLS[@]}"
4462024-01-28T05:16:01.834Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4472024-01-28T05:16:01.834ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4482024-01-28T05:16:01.834Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4492024-01-28T05:16:01.834Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4502024-01-28T05:16:01.834Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4512024-01-28T05:16:01.835Z0+0 records in
4522024-01-28T05:16:01.835Z0+0 records out
4532024-01-28T05:16:01.835Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4542024-01-28T05:16:01.835Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4552024-01-28T05:16:01.835Z+ set +x
4562024-01-28T05:16:01.835ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4572024-01-28T05:16:01.835Z++ zpool list -o name
4582024-01-28T05:16:01.835Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4592024-01-28T05:16:01.836Z+ [[ -z '' ]]
4602024-01-28T05:16:01.836Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4612024-01-28T05:16:01.860Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4622024-01-28T05:16:01.860Z+ set +x
4632024-01-28T05:16:01.860ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4642024-01-28T05:16:01.860Z+ for ZPOOL in "${ZPOOLS[@]}"
4652024-01-28T05:16:01.860Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4662024-01-28T05:16:01.860ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4672024-01-28T05:16:01.860Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4682024-01-28T05:16:01.860Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4692024-01-28T05:16:01.860Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4702024-01-28T05:16:01.884Z0+0 records in
4712024-01-28T05:16:01.884Z0+0 records out
4722024-01-28T05:16:01.884Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4732024-01-28T05:16:01.884Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4742024-01-28T05:16:01.885Z+ set +x
4752024-01-28T05:16:01.885ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4762024-01-28T05:16:01.885Z++ zpool list -o name
4772024-01-28T05:16:01.885Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4782024-01-28T05:16:01.885Z+ [[ -z '' ]]
4792024-01-28T05:16:01.885Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4802024-01-28T05:16:01.909Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4812024-01-28T05:16:01.909Z+ set +x
4822024-01-28T05:16:01.909ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4832024-01-28T05:16:01.909Z+ [[ zone == \z\o\n\e ]]
4842024-01-28T05:16:01.909Z+ ensure_simulated_links igb0
4852024-01-28T05:16:01.909Z+ local PHYSICAL_LINK=igb0
4862024-01-28T05:16:01.909Z+ INDICES=("0" "1")
4872024-01-28T05:16:01.909Z+ for I in "${INDICES[@]}"
4882024-01-28T05:16:01.909Z++ get_simnet_name_if_exists net0
4892024-01-28T05:16:01.909Z++ dladm show-simnet -p -o LINK net0
4902024-01-28T05:16:01.909Z++ echo ''
4912024-01-28T05:16:01.909Z+ [[ -z '' ]]
4922024-01-28T05:16:01.909Z+ dladm create-simnet -t net0
4932024-01-28T05:16:01.933Z+ dladm create-simnet -t sc0_0
4942024-01-28T05:16:01.933Z+ dladm modify-simnet -t -p net0 sc0_0
4952024-01-28T05:16:01.934Z+ dladm set-linkprop -p mtu=9000 sc0_0
4962024-01-28T05:16:01.934Z+ success 'Simnet net0/sc0_0 exists'
4972024-01-28T05:16:01.934Z+ set +x
4982024-01-28T05:16:01.934ZSimnet net0/sc0_0 exists
4992024-01-28T05:16:01.934Z+ for I in "${INDICES[@]}"
5002024-01-28T05:16:01.934Z++ get_simnet_name_if_exists net1
5012024-01-28T05:16:01.935Z++ dladm show-simnet -p -o LINK net1
5022024-01-28T05:16:01.935Z++ echo ''
5032024-01-28T05:16:01.935Z+ [[ -z '' ]]
5042024-01-28T05:16:01.935Z+ dladm create-simnet -t net1
5052024-01-28T05:16:01.959Z+ dladm create-simnet -t sc1_0
5062024-01-28T05:16:01.959Z+ dladm modify-simnet -t -p net1 sc1_0
5072024-01-28T05:16:01.959Z+ dladm set-linkprop -p mtu=9000 sc1_0
5082024-01-28T05:16:01.959Z+ success 'Simnet net1/sc1_0 exists'
5092024-01-28T05:16:01.959Z+ set +x
5102024-01-28T05:16:01.959ZSimnet net1/sc1_0 exists
5112024-01-28T05:16:01.959Z++ get_vnic_name_if_exists sc0_1
5122024-01-28T05:16:01.959Z++ dladm show-vnic -p -o LINK sc0_1
5132024-01-28T05:16:01.959Z++ echo ''
5142024-01-28T05:16:01.959Z+ [[ -z '' ]]
5152024-01-28T05:16:01.960Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162024-01-28T05:16:01.960Z+ [[ -v PROMISC_FILT_OFF ]]
5172024-01-28T05:16:01.960Z+ success 'Vnic sc0_1 exists'
5182024-01-28T05:16:01.960Z+ set +x
5192024-01-28T05:16:01.960ZVnic sc0_1 exists
5202024-01-28T05:16:01.960Z+ warn_if_no_proxy_arp
5212024-01-28T05:16:01.960Z+ [[ -v PXA_START ]]
5222024-01-28T05:16:01.960Z+ [[ -v PXA_END ]]
5232024-01-28T05:16:01.961Z+ ensure_softnpu_zone
5242024-01-28T05:16:01.961Z+ zoneadm list
5252024-01-28T05:16:01.984Z+ grep -q sidecar_softnpu
5262024-01-28T05:16:01.984Z+ [[ -f out/npuzone/npuzone ]]
5272024-01-28T05:16:01.984Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit e3ea4b495ba0a71801ded0776ae4bbd31df57e26 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5282024-01-28T05:16:01.984Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5292024-01-28T05:16:06.711Zdownloading softnpu ...done
5302024-01-28T05:16:06.735Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/libsidecar_lite.so.sha256.txt
5312024-01-28T05:16:11.379Zdownloading asic_program.so ...done
5322024-01-28T05:16:11.403Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/scadm.sha256.txt
5332024-01-28T05:16:17.335Zdownloading scadm ...done
5342024-01-28T05:16:17.359Zcreating port tfportrear0_0 for sc0_0
5352024-01-28T05:16:17.359Zcreating port tfportqsfp0_0 for sc0_1
5362024-01-28T05:16:17.383Zsoftnpu zone setup
5372024-01-28T05:16:17.383Zconfigure zone
5382024-01-28T05:16:17.383Zinstall zone
5392024-01-28T05:16:18.439Zboot zone
5402024-01-28T05:16:21.549Z[sidecar_softnpu] svcs milestone/network
5412024-01-28T05:16:21.576ZSTATE STIME FMRI
5422024-01-28T05:16:21.576Zonline 5:16:19 svc:/milestone/network:default
5432024-01-28T05:16:21.576Z[sidecar_softnpu] /softnpu/npu start
5442024-01-28T05:16:21.600Z
5452024-01-28T05:16:21.626Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5462024-01-28T05:16:21.626Z+ GATEWAY_IP=10.151.1.100
5472024-01-28T05:16:21.626Z+ echo 'Using 10.151.1.100 as gateway ip'
5482024-01-28T05:16:21.626ZUsing 10.151.1.100 as gateway ip
5492024-01-28T05:16:21.626Z+ [[ ! -v GATEWAY_MAC ]]
5502024-01-28T05:16:21.626Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5512024-01-28T05:16:21.626Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5522024-01-28T05:16:21.626Z+++ echo 18:c0:4d:0d:9f:b2
5532024-01-28T05:16:21.626Z+ COUNT=1
5542024-01-28T05:16:21.626Z+ [[ 1 -ne 1 ]]
5552024-01-28T05:16:21.626Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5562024-01-28T05:16:21.626ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5572024-01-28T05:16:21.627Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5582024-01-28T05:16:21.627Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5592024-01-28T05:16:21.650Z+ PXA_MAC=a8:e1:de:01:70:1d
5602024-01-28T05:16:21.651Z+ [[ -v PXA_START ]]
5612024-01-28T05:16:21.651Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5622024-01-28T05:16:21.651Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5632024-01-28T05:16:21.676Z+ z_scadm dump-state
5642024-01-28T05:16:21.676Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5652024-01-28T05:16:21.703Zlocal v6:
5662024-01-28T05:16:21.703Zlocal v4:
5672024-01-28T05:16:21.703Zrouter v6:
5682024-01-28T05:16:21.703Zrouter v4_idx:
5692024-01-28T05:16:21.703Zrouter v4_routes:
5702024-01-28T05:16:21.703Zresolver v4:
5712024-01-28T05:16:21.703Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5722024-01-28T05:16:21.703Zresolver v6:
5732024-01-28T05:16:21.704Znat v4:
5742024-01-28T05:16:21.704Znat_v6:
5752024-01-28T05:16:21.704Zport_mac:
5762024-01-28T05:16:21.704Zproxy arp:
5772024-01-28T05:16:21.704Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5782024-01-28T05:16:21.704Z+ success 'softnpu zone exists'
5792024-01-28T05:16:21.704Z+ set +x
5802024-01-28T05:16:21.704Zsoftnpu zone exists
5812024-01-28T05:16:21.731Z
5822024-01-28T05:16:21.731Zreal 20.309881390
5832024-01-28T05:16:21.731Zuser 0.705031069
5842024-01-28T05:16:21.731Zsys 1.509568621
5852024-01-28T05:16:21.731Ztrap 0.000087225
5862024-01-28T05:16:21.731Ztflt 0.000399484
5872024-01-28T05:16:21.731Zdflt 0.001420489
5882024-01-28T05:16:21.731Zkflt 0.000000000
5892024-01-28T05:16:21.731Zlock 15:32.619735034
5902024-01-28T05:16:21.732Zslp 1:00.733609616
5912024-01-28T05:16:21.732Zlat 0.057151820
5922024-01-28T05:16:21.733Zstop 0.361765685
5932024-01-28T05:16:21.733Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5942024-01-28T05:16:21.757Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952024-01-28T05:16:21.758Z+ SILO_NAME=recovery
5962024-01-28T05:16:21.758Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5972024-01-28T05:16:21.758Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5982024-01-28T05:16:21.758Z+ sed -i~ '
5992024-01-28T05:16:21.758Z /^external_dns_ips/c\
6002024-01-28T05:16:21.758Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6012024-01-28T05:16:21.758Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6022024-01-28T05:16:21.758Z /^first/c\
6032024-01-28T05:16:21.758Zfirst = "10.151.1.164"
6042024-01-28T05:16:21.758Z /^last/c\
6052024-01-28T05:16:21.758Zlast = "10.151.1.173"
6062024-01-28T05:16:21.758Z }
6072024-01-28T05:16:21.758Z /^\[rack_network_config/,/^$/ {
6082024-01-28T05:16:21.758Z /^infra_ip_first/c\
6092024-01-28T05:16:21.758Zinfra_ip_first = "10.151.1.174"
6102024-01-28T05:16:21.758Z /^infra_ip_last/c\
6112024-01-28T05:16:21.758Zinfra_ip_last = "10.151.1.174"
6122024-01-28T05:16:21.758Z }
6132024-01-28T05:16:21.758Z /^\[\[rack_network_config.ports/,/^$/ {
6142024-01-28T05:16:21.759Z /^routes/c\
6152024-01-28T05:16:21.759Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6162024-01-28T05:16:21.759Z /^addresses/c\
6172024-01-28T05:16:21.759Zaddresses = \["10.151.1.174/24"\]
6182024-01-28T05:16:21.759Z }
6192024-01-28T05:16:21.759Z' pkg/config-rss.toml
6202024-01-28T05:16:21.759Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6212024-01-28T05:16:21.759Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6222024-01-28T05:16:21.759Z+++ pkg/config-rss.toml Sun Jan 28 05:16:21 2024
6232024-01-28T05:16:21.759Z@@ -41,7 +41,7 @@
6242024-01-28T05:16:21.759Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6252024-01-28T05:16:21.759Z # must be contained in one of the "internal services" IP Pool ranges listed
6262024-01-28T05:16:21.759Z # below.
6272024-01-28T05:16:21.759Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6282024-01-28T05:16:21.759Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6292024-01-28T05:16:21.759Z
6302024-01-28T05:16:21.759Z # Initial TLS certificates for the external API
6312024-01-28T05:16:21.759Z #
6322024-01-28T05:16:21.759Z@@ -77,8 +77,8 @@
6332024-01-28T05:16:21.759Z #
6342024-01-28T05:16:21.759Z # For more on this and what to put here, see docs/how-to-run.adoc.
6352024-01-28T05:16:21.759Z [[internal_services_ip_pool_ranges]]
6362024-01-28T05:16:21.760Z-first = "192.168.1.20"
6372024-01-28T05:16:21.760Z-last = "192.168.1.29"
6382024-01-28T05:16:21.760Z+first = "10.151.1.164"
6392024-01-28T05:16:21.760Z+last = "10.151.1.173"
6402024-01-28T05:16:21.760Z
6412024-01-28T05:16:21.760Z # TODO - this configuration is subject to change going forward. Ultimately these
6422024-01-28T05:16:21.760Z # parameters should be provided to the control plane via wicket, but we need to
6432024-01-28T05:16:21.760Z@@ -92,8 +92,8 @@
6442024-01-28T05:16:21.760Z # A range of IP addresses used by Boundary Services on the external network. In
6452024-01-28T05:16:21.760Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6462024-01-28T05:16:21.760Z # With softnpu, only one address is used.
6472024-01-28T05:16:21.760Z-infra_ip_first = "192.168.1.30"
6482024-01-28T05:16:21.760Z-infra_ip_last = "192.168.1.30"
6492024-01-28T05:16:21.760Z+infra_ip_first = "10.151.1.174"
6502024-01-28T05:16:21.760Z+infra_ip_last = "10.151.1.174"
6512024-01-28T05:16:21.760Z
6522024-01-28T05:16:21.760Z # Configurations for BGP routers to run on the scrimlets.
6532024-01-28T05:16:21.760Z bgp = []
6542024-01-28T05:16:21.760Z@@ -101,9 +101,9 @@
6552024-01-28T05:16:21.760Z # You can configure multiple uplinks by repeating the following stanza
6562024-01-28T05:16:21.760Z [[rack_network_config.ports]]
6572024-01-28T05:16:21.760Z # Routes associated with this port.
6582024-01-28T05:16:21.760Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6592024-01-28T05:16:21.760Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6602024-01-28T05:16:21.760Z # Addresses associated with this port.
6612024-01-28T05:16:21.761Z-addresses = ["192.168.1.30/24"]
6622024-01-28T05:16:21.761Z+addresses = ["10.151.1.174/24"]
6632024-01-28T05:16:21.761Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6642024-01-28T05:16:21.761Z port = "qsfp0"
6652024-01-28T05:16:21.761Z # The speed of this port.
6662024-01-28T05:16:21.761Z+ true
6672024-01-28T05:16:21.761Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6682024-01-28T05:16:21.761Za pkg/config-rss.toml 7K
6692024-01-28T05:16:21.761Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6702024-01-28T05:16:21.761Z+ TLS_NAME=recovery.sys.oxide.test
6712024-01-28T05:16:21.761Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6722024-01-28T05:16:22.478Z...+......+.....+...+...+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+.+.........+..+....+...............+......+...+.....+.+.....+.....................+.+.....+...+......+.+...+.........+..+.+..+...+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+..........+..+..........+.........+......+...+...+..+.+......+........+.............+........+.+...+...+........................+...+......+.....+.......+...+.....+.........+......+............+............+......+.+.........+.................+.........+...+...+....+......+.....+.......+..+.+..+......+............+...+...................+...+............+........+............+.......+........+......+................+..............+......+..........+...+....................+...+.......+..+....+..+...+............+...+..........+...............+.................+....+......+........+....+.....+.+...........+....+.........+.....+..........+.....+.+......+.....+...+...+....+.....+.+............+..+.........+....+..+....+...........+.........+.......+.....+....+...+...+..+.......+..............+.+.....+......+.......+....................+..........+......+..+....+...+...+.....+....+......+..............+.........+....+........+...+.+..................+...+.....+.........+.............+.................+.........+...................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6732024-01-28T05:16:23.005Z.....+...+............+..+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+..................+.....+....+...........+.+.....+.+...+......+............................................+.......+.....+..................+.......+..+.+..+....+.........+......+.....+......+..........+..+............+.............+..+.+.....+.........+...+...+..........+........+.+.....+...+............+...+.+.....+....+..+.+........+.......+.....+...+......+....+..+.............+..............+..........+...+...+............+...+............+..+...+.........+....+..+.......+........+....+.........+..............+...+............+.+............+...+......+.....+.........+......+....+..+...+....+......+...........+.........+...............................+.....+......+....+......+.....+...............+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6742024-01-28T05:16:23.030Z-----
6752024-01-28T05:16:23.030Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6762024-01-28T05:16:23.054Za pkg/initial-tls-cert.pem 2K
6772024-01-28T05:16:23.054Za pkg/initial-tls-key.pem 4K
6782024-01-28T05:16:23.054Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6792024-01-28T05:16:23.055Z+ rmdir pkg
6802024-01-28T05:16:23.055Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6812024-01-28T05:16:23.055Z+ pfexec mkdir -p /usr/oxide
6822024-01-28T05:16:23.055Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6832024-01-28T05:16:23.751Z+ pfexec chmod +x /usr/oxide/oxide
6842024-01-28T05:16:23.778Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6852024-01-28T05:16:42.052Z+ OMICRON_NO_UNINSTALL=1
6862024-01-28T05:16:42.052Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6872024-01-28T05:16:42.076ZJan 28 05:16:41.324 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6882024-01-28T05:16:42.076ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6892024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6902024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6912024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6922024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6932024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/oxlog.tar, src: out/oxlog.tar
6942024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6952024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6962024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6972024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6982024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6992024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7002024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7012024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter.tar.gz
7022024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7032024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7042024-01-28T05:16:42.077ZJan 28 05:16:41.326 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7052024-01-28T05:16:42.370ZJan 28 05:16:41.627 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062024-01-28T05:16:42.405ZJan 28 05:16:41.662 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072024-01-28T05:16:42.469ZJan 28 05:16:41.726 INFO Unpacking service tarball, service_path: /opt/oxide/oxlog, tar_path: /opt/oxide/oxlog.tar
7082024-01-28T05:16:42.492ZJan 28 05:16:41.730 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7092024-01-28T05:16:42.605Z
7102024-01-28T05:16:42.629Zreal 0.548332823
7112024-01-28T05:16:42.629Zuser 0.108164112
7122024-01-28T05:16:42.629Zsys 2.393533613
7132024-01-28T05:16:42.629Ztrap 0.000007280
7142024-01-28T05:16:42.630Ztflt 0.000009569
7152024-01-28T05:16:42.630Zdflt 0.001017663
7162024-01-28T05:16:42.630Zkflt 0.000000000
7172024-01-28T05:16:42.630Zlock 49.148316804
7182024-01-28T05:16:42.630Zslp 0.789587033
7192024-01-28T05:16:42.630Zlat 0.003042405
7202024-01-28T05:16:42.630Zstop 0.217483772
7212024-01-28T05:16:42.630Z+ retry=0
7222024-01-28T05:16:42.630Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232024-01-28T05:16:42.630Z+ [[ 0 -gt 30 ]]
7242024-01-28T05:16:42.630Z+ sleep 1
7252024-01-28T05:16:43.620Z+ retry=1
7262024-01-28T05:16:43.621Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272024-01-28T05:16:43.644Z+ [[ 1 -gt 30 ]]
7282024-01-28T05:16:43.644Z+ sleep 1
7292024-01-28T05:16:44.635Z+ retry=2
7302024-01-28T05:16:44.636Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312024-01-28T05:16:44.659Z+ [[ 2 -gt 30 ]]
7322024-01-28T05:16:44.659Z+ sleep 1
7332024-01-28T05:16:45.651Z+ retry=3
7342024-01-28T05:16:45.651Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352024-01-28T05:16:45.675Z+ [[ 3 -gt 30 ]]
7362024-01-28T05:16:45.675Z+ sleep 1
7372024-01-28T05:16:46.666Z+ retry=4
7382024-01-28T05:16:46.667Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7392024-01-28T05:17:11.844Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7402024-01-28T05:17:11.874Zlocal v6:
7412024-01-28T05:17:11.874Zfe80::ecaa:eff:fede:e124
7422024-01-28T05:17:11.874Zfd46:b4cd:1092:737::1
7432024-01-28T05:17:11.874Zlocal v4:
7442024-01-28T05:17:11.874Z10.151.1.174
7452024-01-28T05:17:11.874Zrouter v6:
7462024-01-28T05:17:11.874Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::9029:d1ff:fedf:d40c (1)
7472024-01-28T05:17:11.874Zfd00:1122:3344:101::/64 -> fe80::9029:d1ff:fedf:d40c (1)
7482024-01-28T05:17:11.874Zfd00:1122:3344:1::/64 -> fe80::9029:d1ff:fedf:d40c (1)
7492024-01-28T05:17:11.874Zfd00:1122:3344:3::/64 -> fe80::9029:d1ff:fedf:d40c (1)
7502024-01-28T05:17:11.874Zfd00:1122:3344:2::/64 -> fe80::9029:d1ff:fedf:d40c (1)
7512024-01-28T05:17:11.874Zrouter v4_idx:
7522024-01-28T05:17:11.874Z0.0.0.0/0 -> 0
7532024-01-28T05:17:11.874Zrouter v4_routes:
7542024-01-28T05:17:11.874Z0 -> 10.151.1.100 (2)
7552024-01-28T05:17:11.874Zresolver v4:
7562024-01-28T05:17:11.898Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7572024-01-28T05:17:11.898Z10.151.1.174 -> 22:46:65:5d:d8:50
7582024-01-28T05:17:11.898Zresolver v6:
7592024-01-28T05:17:11.900Zfe80::ecaa:eff:fede:e124 -> ee:aa:0e:de:e1:24
7602024-01-28T05:17:11.900Zfe80::9029:d1ff:fedf:d40c -> 92:29:d1:df:d4:0c
7612024-01-28T05:17:11.900Znat v4:
7622024-01-28T05:17:11.900Z10.151.1.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:d9:2b
7632024-01-28T05:17:11.900Znat_v6:
7642024-01-28T05:17:11.900Zport_mac:
7652024-01-28T05:17:11.900Z1: a8:40:25:5f:7d:36
7662024-01-28T05:17:11.900Z1000: a8:40:25:5f:7d:35
7672024-01-28T05:17:11.900Zproxy arp:
7682024-01-28T05:17:11.901Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7692024-01-28T05:17:11.901Z+ retry=0
7702024-01-28T05:17:11.901Z+++ zoneadm list -n
7712024-01-28T05:17:11.901Z+++ grep oxz_ntp
7722024-01-28T05:17:11.901Z++ pfexec svcs -z oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c -Hostate oxide/ntp
7732024-01-28T05:17:11.924Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7742024-01-28T05:17:11.924Z++ true
7752024-01-28T05:17:11.924Z+ [[ '' != online ]]
7762024-01-28T05:17:11.924Z+ [[ 0 -gt 60 ]]
7772024-01-28T05:17:11.924Z+ sleep 1
7782024-01-28T05:17:12.911Z+ retry=1
7792024-01-28T05:17:12.934Z+++ zoneadm list -n
7802024-01-28T05:17:12.935Z+++ grep oxz_ntp
7812024-01-28T05:17:12.935Z++ pfexec svcs -z oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c -Hostate oxide/ntp
7822024-01-28T05:17:12.959Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7832024-01-28T05:17:12.959Z++ true
7842024-01-28T05:17:12.959Z+ [[ '' != online ]]
7852024-01-28T05:17:12.959Z+ [[ 1 -gt 60 ]]
7862024-01-28T05:17:12.959Z+ sleep 1
7872024-01-28T05:17:13.941Z+ retry=2
7882024-01-28T05:17:13.965Z+++ zoneadm list -n
7892024-01-28T05:17:13.965Z+++ grep oxz_ntp
7902024-01-28T05:17:13.965Z++ pfexec svcs -z oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c -Hostate oxide/ntp
7912024-01-28T05:17:13.988Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7922024-01-28T05:17:13.988Z++ true
7932024-01-28T05:17:13.988Z+ [[ '' != online ]]
7942024-01-28T05:17:13.988Z+ [[ 2 -gt 60 ]]
7952024-01-28T05:17:13.988Z+ sleep 1
7962024-01-28T05:17:14.973Z+ retry=3
7972024-01-28T05:17:14.997Z+++ zoneadm list -n
7982024-01-28T05:17:14.997Z+++ grep oxz_ntp
7992024-01-28T05:17:14.997Z++ pfexec svcs -z oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c -Hostate oxide/ntp
8002024-01-28T05:17:14.997Z+ [[ disabled != online ]]
8012024-01-28T05:17:14.997Z+ [[ 3 -gt 60 ]]
8022024-01-28T05:17:14.997Z+ sleep 1
8032024-01-28T05:17:15.999Z+ retry=4
8042024-01-28T05:17:16.023Z+++ zoneadm list -n
8052024-01-28T05:17:16.023Z+++ grep oxz_ntp
8062024-01-28T05:17:16.023Z++ pfexec svcs -z oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c -Hostate oxide/ntp
8072024-01-28T05:17:16.023Z+ [[ disabled != online ]]
8082024-01-28T05:17:16.024Z+ [[ 4 -gt 60 ]]
8092024-01-28T05:17:16.024Z+ sleep 1
8102024-01-28T05:17:17.017Z+ retry=5
8112024-01-28T05:17:17.040Z+++ zoneadm list -n
8122024-01-28T05:17:17.040Z+++ grep oxz_ntp
8132024-01-28T05:17:17.040Z++ pfexec svcs -z oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c -Hostate oxide/ntp
8142024-01-28T05:17:17.042Z+ [[ disabled != online ]]
8152024-01-28T05:17:17.042Z+ [[ 5 -gt 60 ]]
8162024-01-28T05:17:17.042Z+ sleep 1
8172024-01-28T05:17:18.039Z+ retry=6
8182024-01-28T05:17:18.063Z+++ zoneadm list -n
8192024-01-28T05:17:18.063Z+++ grep oxz_ntp
8202024-01-28T05:17:18.063Z++ pfexec svcs -z oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c -Hostate oxide/ntp
8212024-01-28T05:17:18.063Z+ [[ disabled != online ]]
8222024-01-28T05:17:18.063Z+ [[ 6 -gt 60 ]]
8232024-01-28T05:17:18.063Z+ sleep 1
8242024-01-28T05:17:19.062Z+ retry=7
8252024-01-28T05:17:19.086Z+++ zoneadm list -n
8262024-01-28T05:17:19.086Z+++ grep oxz_ntp
8272024-01-28T05:17:19.086Z++ pfexec svcs -z oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c -Hostate oxide/ntp
8282024-01-28T05:17:19.086Z+ [[ disabled != online ]]
8292024-01-28T05:17:19.086Z+ [[ 7 -gt 60 ]]
8302024-01-28T05:17:19.086Z+ sleep 1
8312024-01-28T05:17:20.494Z+ retry=8
8322024-01-28T05:17:20.518Z+++ zoneadm list -n
8332024-01-28T05:17:20.518Z+++ grep oxz_ntp
8342024-01-28T05:17:20.518Z++ pfexec svcs -z oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c -Hostate oxide/ntp
8352024-01-28T05:17:20.518Z+ [[ disabled != online ]]
8362024-01-28T05:17:20.518Z+ [[ 8 -gt 60 ]]
8372024-01-28T05:17:20.518Z+ sleep 1
 
8382024-01-28T05:17:21.029Zbuskin console login: Jan 28 05:17:19 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8392024-01-28T05:17:21.106Z+ retry=9
8402024-01-28T05:17:21.130Z+++ zoneadm list -n
8412024-01-28T05:17:21.130Z+++ grep oxz_ntp
8422024-01-28T05:17:21.131Z++ pfexec svcs -z oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c -Hostate oxide/ntp
8432024-01-28T05:17:21.131Z+ [[ disabled != online ]]
8442024-01-28T05:17:21.131Z+ [[ 9 -gt 60 ]]
8452024-01-28T05:17:21.131Z+ sleep 1
8462024-01-28T05:17:22.128Z+ retry=10
8472024-01-28T05:17:22.152Z+++ zoneadm list -n
8482024-01-28T05:17:22.152Z+++ grep oxz_ntp
8492024-01-28T05:17:22.152Z++ pfexec svcs -z oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c -Hostate oxide/ntp
8502024-01-28T05:17:22.154Z+ [[ online != online ]]
8512024-01-28T05:17:22.154Z+ echo 'Waited for chrony: 10s'
8522024-01-28T05:17:22.155Z+ export RUST_BACKTRACE=1
8532024-01-28T05:17:22.155ZWaited for chrony: 10s
8542024-01-28T05:17:22.155Z+ RUST_BACKTRACE=1
8552024-01-28T05:17:22.155Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8562024-01-28T05:17:22.155Z++ ./tests/bootstrap
8572024-01-28T05:17:22.178ZUsing external DNS server at 10.151.1.164:53
8582024-01-28T05:17:22.178Z2024-01-28 05:17:21.434210779 UTC: attempting to log into API
8592024-01-28T05:17:37.215Z2024-01-28 05:17:37.080953164 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8602024-01-28T05:17:38.216Z2024-01-28 05:17:38.082728648 UTC: attempting to log into API
8612024-01-28T05:17:53.251Z2024-01-28 05:17:53.117225301 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8622024-01-28T05:17:54.253Z2024-01-28 05:17:54.118638121 UTC: attempting to log into API
8632024-01-28T05:18:09.298Z2024-01-28 05:18:09.163435963 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8642024-01-28T05:18:10.299Z2024-01-28 05:18:10.164472828 UTC: attempting to log into API
8652024-01-28T05:18:25.333Z2024-01-28 05:18:25.197920848 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8662024-01-28T05:18:26.333Z2024-01-28 05:18:26.199148465 UTC: attempting to log into API
8672024-01-28T05:18:41.367Z2024-01-28 05:18:41.256793801 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8682024-01-28T05:18:42.368Z2024-01-28 05:18:42.261436072 UTC: attempting to log into API
8692024-01-28T05:18:57.401Z2024-01-28 05:18:57.335062756 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8702024-01-28T05:18:58.402Z2024-01-28 05:18:58.338842431 UTC: attempting to log into API
8712024-01-28T05:19:13.437Z2024-01-28 05:19:13.414046863 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8722024-01-28T05:19:14.438Z2024-01-28 05:19:14.417741990 UTC: attempting to log into API
8732024-01-28T05:19:33.548Z2024-01-28 05:19:29.482988650 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8742024-01-28T05:19:33.710Z2024-01-28 05:19:30.484599002 UTC: attempting to log into API
 
8752024-01-28T05:19:39.678ZJan 28 05:19:39 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
8762024-01-28T05:19:45.545Z2024-01-28 05:19:45.530769823 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
8772024-01-28T05:19:45.788ZJan 28 05:19:45 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8782024-01-28T05:19:46.543Z2024-01-28 05:19:46.531743247 UTC: attempting to log into API
 
8792024-01-28T05:19:51.629ZJan 28 05:19:51 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
8802024-01-28T05:19:58.262ZJan 28 05:19:57 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
8812024-01-28T05:20:01.586Z2024-01-28 05:20:01.565872934 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8822024-01-28T05:20:02.579Z2024-01-28 05:20:02.566544264 UTC: attempting to log into API
 
8832024-01-28T05:20:04.923ZJan 28 05:20:04 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
8842024-01-28T05:20:17.615Z2024-01-28 05:20:17.599828280 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8852024-01-28T05:20:18.612Z2024-01-28 05:20:18.600897795 UTC: attempting to log into API
8862024-01-28T05:20:33.645Z2024-01-28 05:20:33.633146848 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8872024-01-28T05:20:34.646Z2024-01-28 05:20:34.634243804 UTC: attempting to log into API
8882024-01-28T05:20:35.445Z2024-01-28 05:20:35.432077318 UTC: login succeeded
8892024-01-28T05:20:35.478Zwaiting for nexus to come up...
8902024-01-28T05:20:35.548Zcreating IP pool... 10.151.1.175 - 10.151.1.227
8912024-01-28T05:20:35.761Zsetting up quotas...
8922024-01-28T05:20:35.786Zensuring datasets are ready...
8932024-01-28T05:20:38.756Zlet's roll.
8942024-01-28T05:20:38.780Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
8952024-01-28T05:20:38.780ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
8962024-01-28T05:20:38.780ZOXIDE_TOKEN=oxide-token-ef7ecab87f86f1fb149c1ba3a01db5d9775ef11b'
8972024-01-28T05:20:38.780Z++ OXIDE_HOST=https://recovery.sys.oxide.test
8982024-01-28T05:20:38.780Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
8992024-01-28T05:20:38.780Z++ OXIDE_TOKEN=oxide-token-ef7ecab87f86f1fb149c1ba3a01db5d9775ef11b
9002024-01-28T05:20:38.780Z+ export OXIDE_HOST OXIDE_TOKEN
9012024-01-28T05:20:38.781Z+ retry=0
9022024-01-28T05:20:38.781Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9032024-01-28T05:20:38.805Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9042024-01-28T05:20:38.805Z+ [[ 0 -gt 60 ]]
9052024-01-28T05:20:38.805Z+ sleep 1
9062024-01-28T05:20:39.786Z+ retry=1
9072024-01-28T05:20:39.787Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9082024-01-28T05:20:39.810Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9092024-01-28T05:20:39.810Z+ [[ 1 -gt 60 ]]
9102024-01-28T05:20:39.810Z+ sleep 1
9112024-01-28T05:20:40.804Z+ retry=2
9122024-01-28T05:20:40.805Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9132024-01-28T05:20:40.833Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9142024-01-28T05:20:40.834Z+ [[ 2 -gt 60 ]]
9152024-01-28T05:20:40.834Z+ sleep 1
9162024-01-28T05:20:41.826Z+ retry=3
9172024-01-28T05:20:41.827Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9182024-01-28T05:20:41.862Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9192024-01-28T05:20:41.862Z+ [[ 3 -gt 60 ]]
9202024-01-28T05:20:41.862Z+ sleep 1
9212024-01-28T05:20:42.835Z+ retry=4
9222024-01-28T05:20:42.835Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9232024-01-28T05:20:42.861Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9242024-01-28T05:20:42.861Z+ [[ 4 -gt 60 ]]
9252024-01-28T05:20:42.862Z+ sleep 1
9262024-01-28T05:20:43.855Z+ retry=5
9272024-01-28T05:20:43.856Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9282024-01-28T05:20:43.880Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9292024-01-28T05:20:43.880Z+ [[ 5 -gt 60 ]]
9302024-01-28T05:20:43.880Z+ sleep 1
9312024-01-28T05:20:44.874Z+ retry=6
9322024-01-28T05:20:44.875Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9332024-01-28T05:20:44.900Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9342024-01-28T05:20:44.900Z+ [[ 6 -gt 60 ]]
9352024-01-28T05:20:44.900Z+ sleep 1
9362024-01-28T05:20:45.891Z+ retry=7
9372024-01-28T05:20:45.892Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9382024-01-28T05:20:45.920Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9392024-01-28T05:20:45.920Z+ [[ 7 -gt 60 ]]
9402024-01-28T05:20:45.920Z+ sleep 1
9412024-01-28T05:20:46.905Z+ retry=8
9422024-01-28T05:20:46.905Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9432024-01-28T05:20:46.960Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9442024-01-28T05:20:46.960Z+ [[ 8 -gt 60 ]]
9452024-01-28T05:20:46.960Z+ sleep 1
9462024-01-28T05:20:47.911Z+ retry=9
9472024-01-28T05:20:47.911Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9482024-01-28T05:20:47.935Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9492024-01-28T05:20:47.935Z+ [[ 9 -gt 60 ]]
9502024-01-28T05:20:47.935Z+ sleep 1
9512024-01-28T05:20:48.927Z+ retry=10
9522024-01-28T05:20:48.927Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9532024-01-28T05:20:48.951Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9542024-01-28T05:20:48.951Z+ [[ 10 -gt 60 ]]
9552024-01-28T05:20:48.953Z+ sleep 1
9562024-01-28T05:20:49.943Z+ retry=11
9572024-01-28T05:20:49.943Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9582024-01-28T05:20:50.077Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9592024-01-28T05:20:50.077Z+ [[ 11 -gt 60 ]]
9602024-01-28T05:20:50.077Z+ sleep 1
9612024-01-28T05:20:50.959Z+ retry=12
9622024-01-28T05:20:50.959Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9632024-01-28T05:20:50.982Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9642024-01-28T05:20:50.983Z+ [[ 12 -gt 60 ]]
9652024-01-28T05:20:50.983Z+ sleep 1
9662024-01-28T05:20:51.975Z+ retry=13
9672024-01-28T05:20:51.975Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9682024-01-28T05:20:51.999Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9692024-01-28T05:20:51.999Z+ [[ 13 -gt 60 ]]
9702024-01-28T05:20:51.999Z+ sleep 1
9712024-01-28T05:20:52.990Z+ retry=14
9722024-01-28T05:20:52.990Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9732024-01-28T05:20:53.014Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9742024-01-28T05:20:53.014Z+ [[ 14 -gt 60 ]]
9752024-01-28T05:20:53.014Z+ sleep 1
9762024-01-28T05:20:54.006Z+ retry=15
9772024-01-28T05:20:54.006Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9782024-01-28T05:20:54.030Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9792024-01-28T05:20:54.030Z+ [[ 15 -gt 60 ]]
9802024-01-28T05:20:54.030Z+ sleep 1
9812024-01-28T05:20:55.022Z+ retry=16
9822024-01-28T05:20:55.022Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9832024-01-28T05:20:55.050Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9842024-01-28T05:20:55.050Z+ [[ 16 -gt 60 ]]
9852024-01-28T05:20:55.050Z+ sleep 1
9862024-01-28T05:20:56.038Z+ retry=17
9872024-01-28T05:20:56.039Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9882024-01-28T05:20:56.062Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9892024-01-28T05:20:56.062Z+ [[ 17 -gt 60 ]]
9902024-01-28T05:20:56.063Z+ sleep 1
9912024-01-28T05:20:57.055Z+ retry=18
9922024-01-28T05:20:57.055Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9932024-01-28T05:20:57.082Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9942024-01-28T05:20:57.083Z+ [[ 18 -gt 60 ]]
9952024-01-28T05:20:57.083Z+ sleep 1
9962024-01-28T05:20:58.074Z+ retry=19
9972024-01-28T05:20:58.074Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9982024-01-28T05:20:58.098Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9992024-01-28T05:20:58.098Z+ [[ 19 -gt 60 ]]
10002024-01-28T05:20:58.098Z+ sleep 1
10012024-01-28T05:20:59.088Z+ retry=20
10022024-01-28T05:20:59.088Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10032024-01-28T05:20:59.112Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10042024-01-28T05:20:59.112Z+ [[ 20 -gt 60 ]]
10052024-01-28T05:20:59.113Z+ sleep 1
10062024-01-28T05:21:00.102Z+ retry=21
10072024-01-28T05:21:00.102Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10082024-01-28T05:21:00.126Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10092024-01-28T05:21:00.126Z+ [[ 21 -gt 60 ]]
10102024-01-28T05:21:00.126Z+ sleep 1
10112024-01-28T05:21:01.118Z+ retry=22
10122024-01-28T05:21:01.118Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10132024-01-28T05:21:01.142Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10142024-01-28T05:21:01.142Z+ [[ 22 -gt 60 ]]
10152024-01-28T05:21:01.142Z+ sleep 1
10162024-01-28T05:21:02.136Z+ retry=23
10172024-01-28T05:21:02.136Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10182024-01-28T05:21:02.161Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10192024-01-28T05:21:02.162Z+ [[ 23 -gt 60 ]]
10202024-01-28T05:21:02.162Z+ sleep 1
10212024-01-28T05:21:03.152Z+ retry=24
10222024-01-28T05:21:03.153Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10232024-01-28T05:21:03.194Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10242024-01-28T05:21:03.194Z+ [[ 24 -gt 60 ]]
10252024-01-28T05:21:03.194Z+ sleep 1
10262024-01-28T05:21:04.165Z+ retry=25
10272024-01-28T05:21:04.166Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10282024-01-28T05:21:04.195Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10292024-01-28T05:21:04.195Z+ [[ 25 -gt 60 ]]
10302024-01-28T05:21:04.195Z+ sleep 1
10312024-01-28T05:21:05.182Z+ retry=26
10322024-01-28T05:21:05.182Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10332024-01-28T05:21:05.207Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10342024-01-28T05:21:05.207Z+ [[ 26 -gt 60 ]]
10352024-01-28T05:21:05.207Z+ sleep 1
10362024-01-28T05:21:06.195Z+ retry=27
10372024-01-28T05:21:06.196Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10382024-01-28T05:21:06.278Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
10392024-01-28T05:21:06.611Z{"status":"ok"}success
10402024-01-28T05:21:06.611ZProject {
10412024-01-28T05:21:06.611Z description: "some images",
10422024-01-28T05:21:06.611Z id: 8d8d69e4-7399-41e2-843d-558287257df9,
10432024-01-28T05:21:06.611Z name: Name(
10442024-01-28T05:21:06.611Z "images",
10452024-01-28T05:21:06.611Z ),
10462024-01-28T05:21:06.611Z time_created: 2024-01-28T05:21:06.359097Z,
10472024-01-28T05:21:06.611Z time_modified: 2024-01-28T05:21:06.359097Z,
10482024-01-28T05:21:06.611Z}
10492024-01-28T05:21:06.635Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
10502024-01-28T05:26:34.538Zdone!
10512024-01-28T05:26:34.633Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
10522024-01-28T05:26:34.738Zsuccess
10532024-01-28T05:26:34.738ZImage {
10542024-01-28T05:26:34.738Z block_size: ByteCount(
10552024-01-28T05:26:34.738Z 512,
10562024-01-28T05:26:34.738Z ),
10572024-01-28T05:26:34.761Z description: "debian 11 original base image",
10582024-01-28T05:26:34.761Z digest: None,
10592024-01-28T05:26:34.762Z id: c374abf3-bd63-487c-b1c1-63158ec12769,
10602024-01-28T05:26:34.762Z name: Name(
10612024-01-28T05:26:34.762Z "debian11",
10622024-01-28T05:26:34.762Z ),
10632024-01-28T05:26:34.762Z os: "debian",
10642024-01-28T05:26:34.762Z project_id: None,
10652024-01-28T05:26:34.762Z size: ByteCount(
10662024-01-28T05:26:34.762Z 2147483648,
10672024-01-28T05:26:34.762Z ),
10682024-01-28T05:26:34.762Z time_created: 2024-01-28T05:26:34.500362Z,
10692024-01-28T05:26:34.762Z time_modified: 2024-01-28T05:26:34.721614Z,
10702024-01-28T05:26:34.762Z url: None,
10712024-01-28T05:26:34.762Z version: "11",
10722024-01-28T05:26:34.762Z}
10732024-01-28T05:26:34.762Z+ rm ./tests/bootstrap
10742024-01-28T05:26:34.825Z+ for test_bin in tests/*
10752024-01-28T05:26:34.825Z+ ./tests/bootstrap-3358a8ad2ce34473
10762024-01-28T05:26:34.849Z
10772024-01-28T05:26:34.849Zrunning 0 tests
10782024-01-28T05:26:34.849Z
10792024-01-28T05:26:34.849Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10802024-01-28T05:26:34.849Z
10812024-01-28T05:26:34.849Z+ for test_bin in tests/*
10822024-01-28T05:26:34.849Z+ ./tests/end_to_end_tests-71427024e0379c76
10832024-01-28T05:26:34.919Z
10842024-01-28T05:26:34.919Zrunning 2 tests
10852024-01-28T05:26:35.163Ztest no_spoof::no_spoof_here ... ok
10862024-01-28T05:27:11.474Ztest instance_launch::instance_launch ... ok
10872024-01-28T05:27:11.475Z
10882024-01-28T05:27:11.475Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 36.55s
10892024-01-28T05:27:11.475Z
10902024-01-28T05:27:11.501Z+ _exit_trap
10912024-01-28T05:27:11.501Z+ local status=0
10922024-01-28T05:27:11.501Z+ [[ 0 -eq 0 ]]
10932024-01-28T05:27:11.501Z+ exit 0
10942024-01-28T05:27:11.501Zprocess exited: duration 703164 ms, exit code 0
 
10952024-01-28T05:27:11.615Zfound 63 output files
10962024-01-28T05:27:11.615Zuploading: /var/svc/log/oxide-sled-agent:default.log (1603052 bytes)
10972024-01-28T05:27:12.385Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 1603052 -> 1603910
10982024-01-28T05:27:13.469Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10992024-01-28T05:27:13.469Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ac408b7a-78e8-4679-8821-01dc016e9221/root/var/svc/log/oxide-clickhouse:default.log (802328 bytes)
11002024-01-28T05:27:15.320Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ac408b7a-78e8-4679-8821-01dc016e9221/root/var/svc/log/oxide-clickhouse:default.log
11012024-01-28T05:27:15.320Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ac408b7a-78e8-4679-8821-01dc016e9221/root/var/svc/log/oxide-zone-network-setup:default.log (1709 bytes)
11022024-01-28T05:27:16.446Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ac408b7a-78e8-4679-8821-01dc016e9221/root/var/svc/log/oxide-zone-network-setup:default.log
11032024-01-28T05:27:16.447Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_bfa44899-e0ad-4fb2-a494-0d25513411df/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
11042024-01-28T05:27:17.535Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_bfa44899-e0ad-4fb2-a494-0d25513411df/root/var/svc/log/oxide-cockroachdb:default.log
11052024-01-28T05:27:17.535Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_bfa44899-e0ad-4fb2-a494-0d25513411df/root/var/svc/log/oxide-zone-network-setup:default.log (1704 bytes)
11062024-01-28T05:27:18.629Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_bfa44899-e0ad-4fb2-a494-0d25513411df/root/var/svc/log/oxide-zone-network-setup:default.log
11072024-01-28T05:27:18.629Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_183731e5-f016-4454-96f8-b6d087779655/root/var/svc/log/oxide-crucible-agent:default.log (44405 bytes)
11082024-01-28T05:27:19.784Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_183731e5-f016-4454-96f8-b6d087779655/root/var/svc/log/oxide-crucible-agent:default.log
11092024-01-28T05:27:19.784Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_183731e5-f016-4454-96f8-b6d087779655/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11102024-01-28T05:27:21.333Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_183731e5-f016-4454-96f8-b6d087779655/root/var/svc/log/oxide-crucible-downstairs:default.log
11112024-01-28T05:27:21.333Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_183731e5-f016-4454-96f8-b6d087779655/root/var/svc/log/oxide-crucible-downstairs:downstairs-509c38fc-d74e-4f41-9043-909d239c4a65.log (2944 bytes)
11122024-01-28T05:27:22.430Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_183731e5-f016-4454-96f8-b6d087779655/root/var/svc/log/oxide-crucible-downstairs:downstairs-509c38fc-d74e-4f41-9043-909d239c4a65.log
11132024-01-28T05:27:22.430Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_183731e5-f016-4454-96f8-b6d087779655/root/var/svc/log/oxide-crucible-downstairs:downstairs-b559726e-caa1-498d-ab77-a597e101ed62.log (5534 bytes)
11142024-01-28T05:27:23.542Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_183731e5-f016-4454-96f8-b6d087779655/root/var/svc/log/oxide-crucible-downstairs:downstairs-b559726e-caa1-498d-ab77-a597e101ed62.log
11152024-01-28T05:27:23.542Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_183731e5-f016-4454-96f8-b6d087779655/root/var/svc/log/oxide-crucible-downstairs:downstairs-b59ca6df-d979-440c-9888-5bc11f22387d.log (5615 bytes)
11162024-01-28T05:27:24.655Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_183731e5-f016-4454-96f8-b6d087779655/root/var/svc/log/oxide-crucible-downstairs:downstairs-b59ca6df-d979-440c-9888-5bc11f22387d.log
11172024-01-28T05:27:24.655Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_183731e5-f016-4454-96f8-b6d087779655/root/var/svc/log/oxide-crucible-downstairs:downstairs-f95ee247-1c34-48bb-804a-5058ac21d1e3.log (3050 bytes)
11182024-01-28T05:27:25.747Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_183731e5-f016-4454-96f8-b6d087779655/root/var/svc/log/oxide-crucible-downstairs:downstairs-f95ee247-1c34-48bb-804a-5058ac21d1e3.log
11192024-01-28T05:27:25.747Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_183731e5-f016-4454-96f8-b6d087779655/root/var/svc/log/oxide-crucible-downstairs:snapshot-b59ca6df-d979-440c-9888-5bc11f22387d-092dcc3b-9cd1-4510-af0a-d0fb02162747.log (5177 bytes)
11202024-01-28T05:27:26.842Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_183731e5-f016-4454-96f8-b6d087779655/root/var/svc/log/oxide-crucible-downstairs:snapshot-b59ca6df-d979-440c-9888-5bc11f22387d-092dcc3b-9cd1-4510-af0a-d0fb02162747.log
11212024-01-28T05:27:26.842Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_f4fca792-af58-4b0a-b85e-64c537ccc7d2/root/var/svc/log/oxide-external_dns:default.log (30720 bytes)
11222024-01-28T05:27:28.044Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_f4fca792-af58-4b0a-b85e-64c537ccc7d2/root/var/svc/log/oxide-external_dns:default.log
11232024-01-28T05:27:28.044Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_89c3e461-161d-4464-817e-7020702d3f2a/root/var/svc/log/oxide-internal_dns:default.log (113232 bytes)
11242024-01-28T05:27:29.413Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_89c3e461-161d-4464-817e-7020702d3f2a/root/var/svc/log/oxide-internal_dns:default.log
11252024-01-28T05:27:29.437Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_913e2519-5d95-471c-8b96-d513c05ffe93/root/var/svc/log/oxide-cockroachdb:default.log (4406 bytes)
11262024-01-28T05:27:30.506Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_913e2519-5d95-471c-8b96-d513c05ffe93/root/var/svc/log/oxide-cockroachdb:default.log
11272024-01-28T05:27:30.506Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_913e2519-5d95-471c-8b96-d513c05ffe93/root/var/svc/log/oxide-zone-network-setup:default.log (1705 bytes)
11282024-01-28T05:27:31.602Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_913e2519-5d95-471c-8b96-d513c05ffe93/root/var/svc/log/oxide-zone-network-setup:default.log
11292024-01-28T05:27:31.602Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_32554e42-09be-4108-89e8-0a6adda2ff25/root/var/svc/log/oxide-crucible-agent:default.log (14580 bytes)
11302024-01-28T05:27:32.738Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_32554e42-09be-4108-89e8-0a6adda2ff25/root/var/svc/log/oxide-crucible-agent:default.log
11312024-01-28T05:27:32.738Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_32554e42-09be-4108-89e8-0a6adda2ff25/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11322024-01-28T05:27:33.827Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_32554e42-09be-4108-89e8-0a6adda2ff25/root/var/svc/log/oxide-crucible-downstairs:default.log
11332024-01-28T05:27:33.827Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_32554e42-09be-4108-89e8-0a6adda2ff25/root/var/svc/log/oxide-crucible-downstairs:downstairs-9cff7765-5eac-40d0-b968-21feaafcef8d.log (5538 bytes)
11342024-01-28T05:27:34.939Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_32554e42-09be-4108-89e8-0a6adda2ff25/root/var/svc/log/oxide-crucible-downstairs:downstairs-9cff7765-5eac-40d0-b968-21feaafcef8d.log
11352024-01-28T05:27:34.939Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4809cb5d-bc59-4ce4-a008-7e0dc8d54653/root/var/svc/log/oxide-external_dns:default.log (30715 bytes)
11362024-01-28T05:27:36.136Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4809cb5d-bc59-4ce4-a008-7e0dc8d54653/root/var/svc/log/oxide-external_dns:default.log
11372024-01-28T05:27:36.136Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_2b269bcd-7e44-4765-8e20-9b98b1bd17a8/root/var/svc/log/oxide-internal_dns:default.log (120220 bytes)
11382024-01-28T05:27:37.539Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_2b269bcd-7e44-4765-8e20-9b98b1bd17a8/root/var/svc/log/oxide-internal_dns:default.log
11392024-01-28T05:27:37.539Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_84446163-c3b3-41d7-bb02-0bccc571fc09/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
11402024-01-28T05:27:38.631Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_84446163-c3b3-41d7-bb02-0bccc571fc09/root/var/svc/log/oxide-cockroachdb:default.log
11412024-01-28T05:27:38.631Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_84446163-c3b3-41d7-bb02-0bccc571fc09/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
11422024-01-28T05:27:39.726Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_84446163-c3b3-41d7-bb02-0bccc571fc09/root/var/svc/log/oxide-zone-network-setup:default.log
11432024-01-28T05:27:39.726Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ef6d3802-0a32-4840-a7eb-9b007664504a/root/var/svc/log/oxide-crucible-agent:default.log (28253 bytes)
11442024-01-28T05:27:40.904Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ef6d3802-0a32-4840-a7eb-9b007664504a/root/var/svc/log/oxide-crucible-agent:default.log
11452024-01-28T05:27:40.904Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ef6d3802-0a32-4840-a7eb-9b007664504a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11462024-01-28T05:27:41.996Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ef6d3802-0a32-4840-a7eb-9b007664504a/root/var/svc/log/oxide-crucible-downstairs:default.log
11472024-01-28T05:27:41.996Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ef6d3802-0a32-4840-a7eb-9b007664504a/root/var/svc/log/oxide-crucible-downstairs:downstairs-4fdd0683-4006-4792-9730-6938548e65c6.log (2942 bytes)
11482024-01-28T05:27:43.087Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ef6d3802-0a32-4840-a7eb-9b007664504a/root/var/svc/log/oxide-crucible-downstairs:downstairs-4fdd0683-4006-4792-9730-6938548e65c6.log
11492024-01-28T05:27:43.088Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ef6d3802-0a32-4840-a7eb-9b007664504a/root/var/svc/log/oxide-crucible-downstairs:downstairs-fefcb7ae-fee2-4644-9bc9-959f3cc1b532.log (5619 bytes)
11502024-01-28T05:27:44.202Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ef6d3802-0a32-4840-a7eb-9b007664504a/root/var/svc/log/oxide-crucible-downstairs:downstairs-fefcb7ae-fee2-4644-9bc9-959f3cc1b532.log
11512024-01-28T05:27:44.203Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ef6d3802-0a32-4840-a7eb-9b007664504a/root/var/svc/log/oxide-crucible-downstairs:snapshot-fefcb7ae-fee2-4644-9bc9-959f3cc1b532-092dcc3b-9cd1-4510-af0a-d0fb02162747.log (5179 bytes)
11522024-01-28T05:27:45.294Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_ef6d3802-0a32-4840-a7eb-9b007664504a/root/var/svc/log/oxide-crucible-downstairs:snapshot-fefcb7ae-fee2-4644-9bc9-959f3cc1b532-092dcc3b-9cd1-4510-af0a-d0fb02162747.log
11532024-01-28T05:27:45.294Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_4b908392-fb69-4dd9-89eb-f23718f3b1ea/root/var/svc/log/oxide-internal_dns:default.log (112925 bytes)
11542024-01-28T05:27:46.668Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_4b908392-fb69-4dd9-89eb-f23718f3b1ea/root/var/svc/log/oxide-internal_dns:default.log
11552024-01-28T05:27:46.668Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_b29fc594-cc78-4f7b-87c6-35e2f8fb166c/root/var/svc/log/oxide-nexus:default.log (2368315 bytes)
11562024-01-28T05:27:47.423Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_b29fc594-cc78-4f7b-87c6-35e2f8fb166c/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 2368315 -> 2383619
11572024-01-28T05:27:48.495Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_b29fc594-cc78-4f7b-87c6-35e2f8fb166c/root/var/svc/log/oxide-nexus:default.log
11582024-01-28T05:27:48.495Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c/root/var/svc/log/oxide-ntp:default.log (1327 bytes)
11592024-01-28T05:27:49.590Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c/root/var/svc/log/oxide-ntp:default.log
11602024-01-28T05:27:49.590Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_b0435d6a-ae53-4afb-875e-2f21d9c9244d/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
11612024-01-28T05:27:50.685Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_b0435d6a-ae53-4afb-875e-2f21d9c9244d/root/var/svc/log/oxide-cockroachdb:default.log
11622024-01-28T05:27:50.685Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_b0435d6a-ae53-4afb-875e-2f21d9c9244d/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
11632024-01-28T05:27:51.781Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_b0435d6a-ae53-4afb-875e-2f21d9c9244d/root/var/svc/log/oxide-zone-network-setup:default.log
11642024-01-28T05:27:51.781Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d27c5f76-9b91-4ebc-9a8e-15c73ac3a250/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
11652024-01-28T05:27:52.878Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d27c5f76-9b91-4ebc-9a8e-15c73ac3a250/root/var/svc/log/oxide-crucible-agent:default.log
11662024-01-28T05:27:52.878Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d27c5f76-9b91-4ebc-9a8e-15c73ac3a250/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11672024-01-28T05:27:53.972Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_d27c5f76-9b91-4ebc-9a8e-15c73ac3a250/root/var/svc/log/oxide-crucible-downstairs:default.log
11682024-01-28T05:27:53.972Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_ee91d550-5eb0-49da-84e9-66f26cda38fd/root/var/svc/log/oxide-nexus:default.log (1383107 bytes)
11692024-01-28T05:27:54.289Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_ee91d550-5eb0-49da-84e9-66f26cda38fd/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1383107 -> 1400208
11702024-01-28T05:27:55.361Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_ee91d550-5eb0-49da-84e9-66f26cda38fd/root/var/svc/log/oxide-nexus:default.log
11712024-01-28T05:27:55.361Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_8dd5bd3f-2073-4fd5-945b-484addb0212a/root/var/svc/log/oxide-oximeter:default.log (253265 bytes)
11722024-01-28T05:27:55.571Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_8dd5bd3f-2073-4fd5-945b-484addb0212a/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 253265 -> 276571
11732024-01-28T05:27:56.644Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_8dd5bd3f-2073-4fd5-945b-484addb0212a/root/var/svc/log/oxide-oximeter:default.log
11742024-01-28T05:27:56.644Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_8dd5bd3f-2073-4fd5-945b-484addb0212a/root/var/svc/log/oxide-zone-network-setup:default.log (1697 bytes)
11752024-01-28T05:27:57.737Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_8dd5bd3f-2073-4fd5-945b-484addb0212a/root/var/svc/log/oxide-zone-network-setup:default.log
11762024-01-28T05:27:57.737Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_33114384-3877-49da-9cb7-900c959154f9/root/var/svc/log/oxide-cockroachdb:default.log (4381 bytes)
11772024-01-28T05:27:58.829Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_33114384-3877-49da-9cb7-900c959154f9/root/var/svc/log/oxide-cockroachdb:default.log
11782024-01-28T05:27:58.830Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_33114384-3877-49da-9cb7-900c959154f9/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
11792024-01-28T05:27:59.922Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_33114384-3877-49da-9cb7-900c959154f9/root/var/svc/log/oxide-zone-network-setup:default.log
11802024-01-28T05:27:59.922Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d22f5caa-8944-4963-a01c-feb63f085d83/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
11812024-01-28T05:28:01.058Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d22f5caa-8944-4963-a01c-feb63f085d83/root/var/svc/log/oxide-crucible-agent:default.log
11822024-01-28T05:28:01.058Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d22f5caa-8944-4963-a01c-feb63f085d83/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11832024-01-28T05:28:02.149Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d22f5caa-8944-4963-a01c-feb63f085d83/root/var/svc/log/oxide-crucible-downstairs:default.log
11842024-01-28T05:28:02.149Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d22f5caa-8944-4963-a01c-feb63f085d83/root/var/svc/log/oxide-crucible-downstairs:downstairs-0b34cd15-981f-4da1-9f68-17e6b81baf8e.log (3048 bytes)
11852024-01-28T05:28:03.274Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d22f5caa-8944-4963-a01c-feb63f085d83/root/var/svc/log/oxide-crucible-downstairs:downstairs-0b34cd15-981f-4da1-9f68-17e6b81baf8e.log
11862024-01-28T05:28:03.275Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9bdf639a-8f3d-4066-8f68-f6b5feb0a4ec/root/var/svc/log/oxide-crucible-agent:default.log (17546 bytes)
11872024-01-28T05:28:04.412Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9bdf639a-8f3d-4066-8f68-f6b5feb0a4ec/root/var/svc/log/oxide-crucible-agent:default.log
11882024-01-28T05:28:04.436Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9bdf639a-8f3d-4066-8f68-f6b5feb0a4ec/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11892024-01-28T05:28:05.506Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9bdf639a-8f3d-4066-8f68-f6b5feb0a4ec/root/var/svc/log/oxide-crucible-downstairs:default.log
11902024-01-28T05:28:05.506Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9bdf639a-8f3d-4066-8f68-f6b5feb0a4ec/root/var/svc/log/oxide-crucible-downstairs:downstairs-4c34c6d7-ef33-46e7-9223-21d0351402e0.log (5617 bytes)
11912024-01-28T05:28:06.619Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9bdf639a-8f3d-4066-8f68-f6b5feb0a4ec/root/var/svc/log/oxide-crucible-downstairs:downstairs-4c34c6d7-ef33-46e7-9223-21d0351402e0.log
11922024-01-28T05:28:06.620Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9bdf639a-8f3d-4066-8f68-f6b5feb0a4ec/root/var/svc/log/oxide-crucible-downstairs:snapshot-4c34c6d7-ef33-46e7-9223-21d0351402e0-092dcc3b-9cd1-4510-af0a-d0fb02162747.log (5177 bytes)
11932024-01-28T05:28:07.710Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9bdf639a-8f3d-4066-8f68-f6b5feb0a4ec/root/var/svc/log/oxide-crucible-downstairs:snapshot-4c34c6d7-ef33-46e7-9223-21d0351402e0-092dcc3b-9cd1-4510-af0a-d0fb02162747.log
11942024-01-28T05:28:07.710Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_b50e12e5-3015-4878-8f00-a0c2f22b19cf/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
11952024-01-28T05:28:08.815Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_b50e12e5-3015-4878-8f00-a0c2f22b19cf/root/var/svc/log/oxide-crucible-pantry:default.log
11962024-01-28T05:28:08.904Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_fef623af-2da4-4813-a6aa-89e07054beaa/root/var/svc/log/oxide-nexus:default.log (34319482 bytes)
11972024-01-28T05:28:23.229Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_fef623af-2da4-4813-a6aa-89e07054beaa/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 34319482 -> 34407671
11982024-01-28T05:28:24.409Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_fef623af-2da4-4813-a6aa-89e07054beaa/root/var/svc/log/oxide-nexus:default.log
11992024-01-28T05:28:24.409Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2fc7f368-9958-4717-841e-85ca24716f82/root/var/svc/log/oxide-crucible-agent:default.log (14120 bytes)
12002024-01-28T05:28:25.522Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2fc7f368-9958-4717-841e-85ca24716f82/root/var/svc/log/oxide-crucible-agent:default.log
12012024-01-28T05:28:25.522Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2fc7f368-9958-4717-841e-85ca24716f82/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12022024-01-28T05:28:26.613Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2fc7f368-9958-4717-841e-85ca24716f82/root/var/svc/log/oxide-crucible-downstairs:default.log
12032024-01-28T05:28:26.613Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2fc7f368-9958-4717-841e-85ca24716f82/root/var/svc/log/oxide-crucible-downstairs:downstairs-60dc632e-356f-449c-afd7-45165d25d106.log (5539 bytes)
12042024-01-28T05:28:27.708Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2fc7f368-9958-4717-841e-85ca24716f82/root/var/svc/log/oxide-crucible-downstairs:downstairs-60dc632e-356f-449c-afd7-45165d25d106.log
12052024-01-28T05:28:27.708Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3d0d6ae8-14ff-488b-a0db-77c3440d2125/root/var/svc/log/oxide-crucible-agent:default.log (14148 bytes)
12062024-01-28T05:28:28.825Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3d0d6ae8-14ff-488b-a0db-77c3440d2125/root/var/svc/log/oxide-crucible-agent:default.log
12072024-01-28T05:28:28.825Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3d0d6ae8-14ff-488b-a0db-77c3440d2125/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12082024-01-28T05:28:29.915Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3d0d6ae8-14ff-488b-a0db-77c3440d2125/root/var/svc/log/oxide-crucible-downstairs:default.log
12092024-01-28T05:28:29.915Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3d0d6ae8-14ff-488b-a0db-77c3440d2125/root/var/svc/log/oxide-crucible-downstairs:downstairs-6dfc8c18-9575-4529-90fb-0dfd3ff5f52d.log (3049 bytes)
12102024-01-28T05:28:31.006Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3d0d6ae8-14ff-488b-a0db-77c3440d2125/root/var/svc/log/oxide-crucible-downstairs:downstairs-6dfc8c18-9575-4529-90fb-0dfd3ff5f52d.log
12112024-01-28T05:28:31.006Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_1a1fa78a-b127-49a9-9ff9-c08d5683295d/root/var/svc/log/oxide-crucible-pantry:default.log (1224182 bytes)
12122024-01-28T05:28:32.964Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_1a1fa78a-b127-49a9-9ff9-c08d5683295d/root/var/svc/log/oxide-crucible-pantry:default.log
12132024-01-28T05:28:32.988Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b0fd8fe9-afd3-4eea-9747-826617c654bd/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
12142024-01-28T05:28:34.055Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b0fd8fe9-afd3-4eea-9747-826617c654bd/root/var/svc/log/oxide-crucible-pantry:default.log
12152024-01-28T05:28:34.055Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ea3fb73-ebca-4fd5-9f5c-d2db67361b12/root/var/svc/log/oxide-crucible-agent:default.log (9794 bytes)
12162024-01-28T05:28:35.167Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ea3fb73-ebca-4fd5-9f5c-d2db67361b12/root/var/svc/log/oxide-crucible-agent:default.log
12172024-01-28T05:28:35.167Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ea3fb73-ebca-4fd5-9f5c-d2db67361b12/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12182024-01-28T05:28:36.257Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ea3fb73-ebca-4fd5-9f5c-d2db67361b12/root/var/svc/log/oxide-crucible-downstairs:default.log
12192024-01-28T05:28:36.257Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ea3fb73-ebca-4fd5-9f5c-d2db67361b12/root/var/svc/log/oxide-crucible-downstairs:downstairs-8a1400c4-aa6f-4451-9b07-5cd5dd413da7.log (2944 bytes)
12202024-01-28T05:28:37.349Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ea3fb73-ebca-4fd5-9f5c-d2db67361b12/root/var/svc/log/oxide-crucible-downstairs:downstairs-8a1400c4-aa6f-4451-9b07-5cd5dd413da7.log
12212024-01-28T05:28:37.349Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c/root/var/log/chrony/measurements.log (20961 bytes)
12222024-01-28T05:28:37.437Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c/root/var/log/chrony/measurements.log" changed size mid upload: 20961 -> 23427
12232024-01-28T05:28:38.506Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c/root/var/log/chrony/measurements.log
12242024-01-28T05:28:38.506Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c/root/var/log/chrony/statistics.log (12168 bytes)
12252024-01-28T05:28:38.574Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c/root/var/log/chrony/statistics.log" changed size mid upload: 12168 -> 14391
12262024-01-28T05:28:39.646Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c/root/var/log/chrony/statistics.log
12272024-01-28T05:28:39.646Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c/root/var/log/chrony/tracking.log (2117 bytes)
12282024-01-28T05:28:39.691Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c/root/var/log/chrony/tracking.log" changed size mid upload: 2117 -> 2249
12292024-01-28T05:28:40.757Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_54ac74f3-e544-4ce5-94cc-b8fa5f815c5c/root/var/log/chrony/tracking.log