01HQ2B793SYZ9ANMJ7VN198Y64: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.28",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HQ2B7SKV6C123MQ1KD57RJ2Z

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12024-02-20T04:47:52.644Zjob dependencies complete; ready to run (waiting for 38 m 51 s)
22024-02-20T04:53:07.892Zjob assigned to worker 01HQ2DEPJBE2X9WCXV72ZRVD2Y (queued for 5 m 15 s)
32024-02-20T04:53:07.900Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42024-02-20T04:53:07.942Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52024-02-20T04:53:07.982Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62024-02-20T04:53:08.020Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72024-02-20T04:53:08.053Zhttp://172.20.0.5:9969/postboot/buskin... ok
82024-02-20T04:53:08.089ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92024-02-20T04:53:08.129Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102024-02-20T04:53:08.165ZOxide Helios Version helios-2.0.22451 64-bit
112024-02-20T04:53:08.197ZConfiguring devices.
122024-02-20T04:53:08.232Z##################################################
132024-02-20T04:53:08.265Z#### ############# ##
142024-02-20T04:53:08.298Z### ### ############ ##
152024-02-20T04:53:08.331Z## ### # ## ### ## Oxide Computer Company ##
162024-02-20T04:53:08.365Z## ## ## ### # ### ##
172024-02-20T04:53:08.401Z## # ### #### #### This Station Under ##
182024-02-20T04:53:08.450Z### ### #### # ### Computer Control ##
192024-02-20T04:53:08.488Z#### #### ### ## ##
202024-02-20T04:53:08.528Z##################################################
212024-02-20T04:53:08.565ZHostname: buskin
222024-02-20T04:53:08.599ZLoading smf(7) service descriptions:
232024-02-20T04:53:08.813Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-c43b64681519df89.gz
242024-02-20T04:53:10.787Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-c43b64681519df89.gz
252024-02-20T04:53:10.787Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262024-02-20T04:53:31.233Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272024-02-20T04:53:31.233Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8b0279dffce37f90.gz
282024-02-20T04:53:51.248Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8b0279dffce37f90.gz
292024-02-20T04:53:51.248Zdownloading input: /input/ci-tools/work/caboose-util.gz
302024-02-20T04:53:52.059Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312024-02-20T04:53:52.059Zdownloading input: /input/ci-tools/work/tufaceous.gz
322024-02-20T04:53:54.887Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332024-02-20T04:53:54.913Zdownloading input: /input/package/work/version.txt
342024-02-20T04:53:54.938Zdownloaded input: /input/package/work/version.txt
352024-02-20T04:53:54.938Zdownloading input: /input/package/work/package.tar.gz
362024-02-20T04:54:10.768Zdownloaded input: /input/package/work/package.tar.gz
372024-02-20T04:54:10.769Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382024-02-20T04:54:34.501Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392024-02-20T04:54:34.501Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402024-02-20T04:54:41.570Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412024-02-20T04:54:41.601Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422024-02-20T04:55:15.219Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432024-02-20T04:55:15.219Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442024-02-20T04:55:48.771Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452024-02-20T04:55:48.771Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462024-02-20T04:56:13.291Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472024-02-20T04:56:13.291Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482024-02-20T04:56:16.553Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492024-02-20T04:56:16.553Zdownloading input: /input/package/work/zones/crucible.tar.gz
502024-02-20T04:56:21.836Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512024-02-20T04:56:21.836Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522024-02-20T04:56:25.949Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532024-02-20T04:56:25.949Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542024-02-20T04:56:30.046Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552024-02-20T04:56:30.046Zdownloading input: /input/package/work/zones/ntp.tar.gz
562024-02-20T04:56:30.117Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572024-02-20T04:56:30.144Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582024-02-20T04:56:33.612Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592024-02-20T04:56:33.612Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602024-02-20T04:56:37.140Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612024-02-20T04:56:37.141Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622024-02-20T04:56:56.933Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632024-02-20T04:56:56.933Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642024-02-20T04:57:15.933Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652024-02-20T04:57:15.933Zdownloading input: /input/package/work/zones/overlay.tar.gz
662024-02-20T04:57:16.004Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672024-02-20T04:57:16.004Zdownloading input: /input/package/work/zones/oximeter.tar.gz
682024-02-20T04:57:27.902Zdownloaded input: /input/package/work/zones/oximeter.tar.gz
692024-02-20T04:57:27.902Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702024-02-20T04:57:42.713Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712024-02-20T04:57:42.713Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722024-02-20T04:58:32.673Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732024-02-20T04:58:32.673Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742024-02-20T04:59:09.549Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752024-02-20T04:59:09.574Zstarting task 0: "setup"
762024-02-20T04:59:09.574Z++ uname -s
772024-02-20T04:59:09.574Z+ kern=SunOS
782024-02-20T04:59:09.575Z+ case "$kern" in
792024-02-20T04:59:09.575Z+ groupadd -g 12345 build
802024-02-20T04:59:09.575Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812024-02-20T04:59:11.566Z+ zfs create -o mountpoint=/work rpool/work
822024-02-20T04:59:11.607Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832024-02-20T04:59:11.633Z+ home_fs=zfs
842024-02-20T04:59:11.633Z+ [[ zfs == autofs ]]
852024-02-20T04:59:11.633Z+ mkdir -p /home/build
862024-02-20T04:59:11.633Z+ chown build:build /home/build /work
872024-02-20T04:59:13.616Z+ chmod 0700 /home/build /work
882024-02-20T04:59:13.642Zprocess exited: duration 4071 ms, exit code 0
 
892024-02-20T04:59:13.693Zstarting task 1: "authentication"
902024-02-20T04:59:13.738Zprocess exited: duration 45 ms, exit code 0
 
912024-02-20T04:59:13.789Zstarting task 2: "build"
922024-02-20T04:59:13.820Z+ trap _exit_trap EXIT
932024-02-20T04:59:13.820Z+ set +x
942024-02-20T04:59:42.307Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
952024-02-20T04:59:42.571Zsecpolicy_fs_linkdir: 0x55 = 0x55
962024-02-20T04:59:42.571Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
972024-02-20T04:59:42.571Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
982024-02-20T04:59:42.571Zsecpolicy_fs_linkdir+3: 0 = 0xe5
992024-02-20T04:59:42.571Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1002024-02-20T04:59:42.571Zsecpolicy_fs_linkdir+5: 0 = 0x1
1012024-02-20T04:59:42.571Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1022024-02-20T04:59:42.597Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1032024-02-20T04:59:42.597Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1042024-02-20T04:59:42.597Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1052024-02-20T04:59:42.597Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1062024-02-20T04:59:42.597Z+ [[ -d /opt/oxide ]]
1072024-02-20T04:59:42.597Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1082024-02-20T04:59:42.597Z+ pfexec mkdir /opt/oxide
1092024-02-20T04:59:42.597Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1102024-02-20T04:59:42.597Z+ [[ -d /opt/oxide-underneath ]]
1112024-02-20T04:59:42.597Z+ cd /opt/oxide-underneath
1122024-02-20T04:59:42.597Z+ pfexec tar ceEp@/f - .
1132024-02-20T04:59:42.598Z+ cd /opt/oxide
1142024-02-20T04:59:42.598Z+ pfexec tar xveEp@/f -
1152024-02-20T04:59:42.635Zx ., 0 bytes, 0 tape blocks
1162024-02-20T04:59:42.635Zx ./opte, 0 bytes, 0 tape blocks
1172024-02-20T04:59:42.635Zx ./opte/bin, 0 bytes, 0 tape blocks
1182024-02-20T04:59:42.635Zx ./opte/bin/opteadm, 51799224 bytes, 101171 tape blocks
1192024-02-20T04:59:42.660Z+ pfexec rm -rf /opt/oxide-underneath
1202024-02-20T04:59:42.660Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1212024-02-20T04:59:42.684Z+ pfexec mkdir /opt/oxide/work
1222024-02-20T04:59:42.685Z+ pfexec chown build:build /opt/oxide/work
1232024-02-20T04:59:42.685Z+ cd /opt/oxide/work
1242024-02-20T04:59:42.685Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1252024-02-20T04:59:44.398ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1262024-02-20T04:59:44.398Zx out/mg-ddm-gz.tar, 69308416 bytes, 135368 tape blocks
1272024-02-20T04:59:44.398Zx out/omicron-sled-agent.tar, 123007488 bytes, 240249 tape blocks
1282024-02-20T04:59:44.398Zx out/oxlog.tar, 8557568 bytes, 16714 tape blocks
1292024-02-20T04:59:44.398Zx out/target/test, 75 bytes, 1 tape blocks
1302024-02-20T04:59:44.398Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1312024-02-20T04:59:44.398Zx package-manifest.toml, 23732 bytes, 47 tape blocks
1322024-02-20T04:59:44.398Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1332024-02-20T04:59:44.398Zx target/release/omicron-package, 45889576 bytes, 89629 tape blocks
1342024-02-20T04:59:44.398Zx tools/create_virtual_hardware.sh, 3184 bytes, 7 tape blocks
1352024-02-20T04:59:44.398Zx tools/virtual_hardware.sh, 3416 bytes, 7 tape blocks
1362024-02-20T04:59:44.398Zx tools/scrimlet/create-softnpu-zone.sh, 627 bytes, 2 tape blocks
1372024-02-20T04:59:44.398Zx tools/scrimlet/destroy-softnpu-zone.sh, 203 bytes, 1 tape blocks
1382024-02-20T04:59:44.398Zx tools/scrimlet/softnpu-init.sh, 2157 bytes, 5 tape blocks
1392024-02-20T04:59:44.398Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1402024-02-20T04:59:44.398Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1412024-02-20T04:59:44.424Z
1422024-02-20T04:59:44.424Zreal 1.719184209
1432024-02-20T04:59:44.424Zuser 1.543138366
1442024-02-20T04:59:44.424Zsys 0.303635696
1452024-02-20T04:59:44.424Ztrap 0.000079850
1462024-02-20T04:59:44.424Ztflt 0.000000000
1472024-02-20T04:59:44.424Zdflt 0.000000000
1482024-02-20T04:59:44.424Zkflt 0.000000000
1492024-02-20T04:59:44.424Zlock 0.000000000
1502024-02-20T04:59:44.425Zslp 1.547510616
1512024-02-20T04:59:44.425Zlat 0.041794870
1522024-02-20T04:59:44.425Zstop 0.000129572
1532024-02-20T04:59:44.425Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1542024-02-20T04:59:45.567Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1552024-02-20T04:59:45.592Z+ mkdir tests
1562024-02-20T04:59:45.592Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1572024-02-20T04:59:45.592Z+ ptime -m gunzip
1582024-02-20T04:59:45.592Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-c43b64681519df89
1592024-02-20T04:59:45.821Z
1602024-02-20T04:59:45.847Zreal 0.232667704
1612024-02-20T04:59:45.847Zuser 0.213810234
1622024-02-20T04:59:45.847Zsys 0.018610276
1632024-02-20T04:59:45.847Ztrap 0.000020049
1642024-02-20T04:59:45.847Ztflt 0.000000000
1652024-02-20T04:59:45.847Zdflt 0.000000000
1662024-02-20T04:59:45.848Zkflt 0.000000000
1672024-02-20T04:59:45.848Zlock 0.000000000
1682024-02-20T04:59:45.848Zslp 0.000020100
1692024-02-20T04:59:45.848Zlat 0.000024180
1702024-02-20T04:59:45.848Zstop 0.000125702
1712024-02-20T04:59:45.848Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-c43b64681519df89
1722024-02-20T04:59:45.848Z+ chmod a+x tests/bootstrap-c43b64681519df89
1732024-02-20T04:59:45.848Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1742024-02-20T04:59:45.848Z+ ptime -m gunzip
1752024-02-20T04:59:45.848Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1762024-02-20T04:59:48.329Z
1772024-02-20T04:59:48.356Zreal 2.497550996
1782024-02-20T04:59:48.356Zuser 2.346857156
1792024-02-20T04:59:48.356Zsys 0.150288353
1802024-02-20T04:59:48.356Ztrap 0.000160012
1812024-02-20T04:59:48.356Ztflt 0.000000000
1822024-02-20T04:59:48.356Zdflt 0.000000000
1832024-02-20T04:59:48.356Zkflt 0.000000000
1842024-02-20T04:59:48.356Zlock 0.000000000
1852024-02-20T04:59:48.356Zslp 0.000025460
1862024-02-20T04:59:48.356Zlat 0.000018140
1872024-02-20T04:59:48.356Zstop 0.000126302
1882024-02-20T04:59:48.356Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1892024-02-20T04:59:48.356Z+ chmod a+x tests/bootstrap
1902024-02-20T04:59:48.356Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1912024-02-20T04:59:48.356Z+ ptime -m gunzip
1922024-02-20T04:59:48.356Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8b0279dffce37f90
1932024-02-20T04:59:50.931Z
1942024-02-20T04:59:50.956Zreal 2.589345315
1952024-02-20T04:59:50.956Zuser 2.423921154
1962024-02-20T04:59:50.956Zsys 0.164975103
1972024-02-20T04:59:50.956Ztrap 0.000196164
1982024-02-20T04:59:50.956Ztflt 0.000000000
1992024-02-20T04:59:50.956Zdflt 0.000000000
2002024-02-20T04:59:50.956Zkflt 0.000000000
2012024-02-20T04:59:50.956Zlock 0.000000000
2022024-02-20T04:59:50.956Zslp 0.000023240
2032024-02-20T04:59:50.956Zlat 0.000025040
2042024-02-20T04:59:50.956Zstop 0.000127902
2052024-02-20T04:59:50.956Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8b0279dffce37f90
2062024-02-20T04:59:50.956Z+ chmod a+x tests/end_to_end_tests-8b0279dffce37f90
2072024-02-20T04:59:50.956Z+ bmat address ls
2082024-02-20T04:59:50.956ZNAME CIDR FIRST LAST COUNT
2092024-02-20T04:59:50.956Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2102024-02-20T04:59:50.956Z++ bmat address ls -f extra -Ho first,last,count
2112024-02-20T04:59:50.986Z+ set -- 10.151.1.164 10.151.1.227 64
2122024-02-20T04:59:50.986Z+ EXTRA_IP_START=10.151.1.164
2132024-02-20T04:59:50.986Z+ EXTRA_IP_END=10.151.1.227
2142024-02-20T04:59:50.986Z+ EXTRA_IP_COUNT=64
2152024-02-20T04:59:50.986Z+ (( EXTRA_IP_COUNT >= 32 ))
2162024-02-20T04:59:50.986Z+ EXTRA_IP_BASE=10.151.1
2172024-02-20T04:59:50.986Z+ EXTRA_IP_FOCTET=164
2182024-02-20T04:59:50.986Z+ EXTRA_IP_LOCTET=227
2192024-02-20T04:59:50.986Z+ SERVICE_IP_POOL_START=10.151.1.164
2202024-02-20T04:59:50.986Z+ SERVICE_IP_POOL_END=10.151.1.173
2212024-02-20T04:59:50.986Z+ DNS_IP1=10.151.1.164
2222024-02-20T04:59:50.986Z+ DNS_IP2=10.151.1.165
2232024-02-20T04:59:50.986Z+ UPLINK_IP=10.151.1.174
2242024-02-20T04:59:50.986Z+ IPPOOL_START=10.151.1.175
2252024-02-20T04:59:50.986Z+ IPPOOL_END=10.151.1.227
2262024-02-20T04:59:50.986Z++ ipadm show-addr -po type,addr
2272024-02-20T04:59:50.986Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2282024-02-20T04:59:50.986Z+ GATEWAY_IP=10.151.1.100
2292024-02-20T04:59:50.986Z+ [[ -n 10.151.1.100 ]]
2302024-02-20T04:59:50.986Z+ ping -s 10.151.1.100 56 1
2312024-02-20T04:59:50.986ZPING 10.151.1.100: 56 data bytes
2322024-02-20T04:59:50.987Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.010 ms
2332024-02-20T04:59:50.987Z
2342024-02-20T04:59:50.987Z----10.151.1.100 PING Statistics----
2352024-02-20T04:59:50.987Z1 packets transmitted, 1 packets received, 0% packet loss
2362024-02-20T04:59:50.987Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2372024-02-20T04:59:50.987Z++ arp -an
2382024-02-20T04:59:50.987Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2392024-02-20T04:59:51.012Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2402024-02-20T04:59:51.012Z+ routeadm -e ipv4-forwarding -u
2412024-02-20T04:59:51.096Z+ PXA_START=10.151.1.164
2422024-02-20T04:59:51.096Z+ PXA_END=10.151.1.227
2432024-02-20T04:59:51.121Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2442024-02-20T04:59:51.121Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2452024-02-20T04:59:51.294Z+ ZPOOL_VDEV_DIR=/scratch
2462024-02-20T04:59:51.294Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2472024-02-20T04:59:51.319Z+++ dirname ./tools/create_virtual_hardware.sh
2482024-02-20T04:59:51.319Z++ cd ./tools
2492024-02-20T04:59:51.319Z++ pwd
2502024-02-20T04:59:51.320Z+ SOURCE_DIR=/opt/oxide/work/tools
2512024-02-20T04:59:51.320Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2522024-02-20T04:59:51.320Z+ SOFTNPU_MODE=zone
2532024-02-20T04:59:51.320Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2542024-02-20T04:59:51.320Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2552024-02-20T04:59:51.320Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2562024-02-20T04:59:51.320Z++ dladm show-phys -p -o LINK
2572024-02-20T04:59:51.320Z++ head -1
2582024-02-20T04:59:51.346Z+ PHYSICAL_LINK=igb0
2592024-02-20T04:59:51.346Z+ echo 'Using igb0 as physical link'
2602024-02-20T04:59:51.346ZUsing igb0 as physical link
2612024-02-20T04:59:51.346Z+ ensure_run_as_root
2622024-02-20T04:59:51.346Z++ id -u
2632024-02-20T04:59:51.346Z+ [[ 0 -ne 0 ]]
2642024-02-20T04:59:51.346Z+ ensure_zpools
2652024-02-20T04:59:51.346Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2662024-02-20T04:59:51.346Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2672024-02-20T04:59:51.346Z+ readarray -t ZPOOLS
2682024-02-20T04:59:51.346Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2692024-02-20T04:59:51.346Z++ sed 's/[ ",]//g'
2702024-02-20T04:59:51.346Z+ for ZPOOL in "${ZPOOLS[@]}"
2712024-02-20T04:59:51.346Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2722024-02-20T04:59:51.346ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2732024-02-20T04:59:51.346Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2742024-02-20T04:59:51.347Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2752024-02-20T04:59:51.347Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2762024-02-20T04:59:51.347Z0+0 records in
2772024-02-20T04:59:51.347Z0+0 records out
2782024-02-20T04:59:51.347Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2792024-02-20T04:59:51.347Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2802024-02-20T04:59:51.347Z+ set +x
2812024-02-20T04:59:51.347ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2822024-02-20T04:59:51.347Z++ zpool list -o name
2832024-02-20T04:59:51.347Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2842024-02-20T04:59:51.347Z+ [[ -z '' ]]
2852024-02-20T04:59:51.347Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2862024-02-20T04:59:51.375Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2872024-02-20T04:59:51.400Z+ set +x
2882024-02-20T04:59:51.400ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2892024-02-20T04:59:51.400Z+ for ZPOOL in "${ZPOOLS[@]}"
2902024-02-20T04:59:51.400Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2912024-02-20T04:59:51.400ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2922024-02-20T04:59:51.400Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2932024-02-20T04:59:51.400Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2942024-02-20T04:59:51.400Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2952024-02-20T04:59:51.400Z0+0 records in
2962024-02-20T04:59:51.400Z0+0 records out
2972024-02-20T04:59:51.400Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2982024-02-20T04:59:51.400Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2992024-02-20T04:59:51.400Z+ set +x
3002024-02-20T04:59:51.400ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3012024-02-20T04:59:51.400Z++ zpool list -o name
3022024-02-20T04:59:51.400Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3032024-02-20T04:59:51.400Z+ [[ -z '' ]]
3042024-02-20T04:59:51.400Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3052024-02-20T04:59:51.426Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3062024-02-20T04:59:51.426Z+ set +x
3072024-02-20T04:59:51.426ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3082024-02-20T04:59:51.426Z+ for ZPOOL in "${ZPOOLS[@]}"
3092024-02-20T04:59:51.426Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3102024-02-20T04:59:51.426Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3112024-02-20T04:59:51.426ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3122024-02-20T04:59:51.426Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3132024-02-20T04:59:51.426Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3142024-02-20T04:59:51.426Z0+0 records in
3152024-02-20T04:59:51.426Z0+0 records out
3162024-02-20T04:59:51.426Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3172024-02-20T04:59:51.426Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3182024-02-20T04:59:51.426Z+ set +x
3192024-02-20T04:59:51.426ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3202024-02-20T04:59:51.427Z++ zpool list -o name
3212024-02-20T04:59:51.427Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3222024-02-20T04:59:51.452Z+ [[ -z '' ]]
3232024-02-20T04:59:51.452Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3242024-02-20T04:59:51.481Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3252024-02-20T04:59:51.481Z+ set +x
3262024-02-20T04:59:51.481ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3272024-02-20T04:59:51.481Z+ for ZPOOL in "${ZPOOLS[@]}"
3282024-02-20T04:59:51.481Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3292024-02-20T04:59:51.481ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3302024-02-20T04:59:51.481Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3312024-02-20T04:59:51.481Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3322024-02-20T04:59:51.481Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3332024-02-20T04:59:51.481Z0+0 records in
3342024-02-20T04:59:51.481Z0+0 records out
3352024-02-20T04:59:51.481Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3362024-02-20T04:59:51.481Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3372024-02-20T04:59:51.481Z+ set +x
3382024-02-20T04:59:51.481ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3392024-02-20T04:59:51.481Z++ zpool list -o name
3402024-02-20T04:59:51.481Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3412024-02-20T04:59:51.482Z+ [[ -z '' ]]
3422024-02-20T04:59:51.482Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3432024-02-20T04:59:51.509Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3442024-02-20T04:59:51.509Z+ set +x
3452024-02-20T04:59:51.509ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3462024-02-20T04:59:51.509Z+ for ZPOOL in "${ZPOOLS[@]}"
3472024-02-20T04:59:51.509Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3482024-02-20T04:59:51.509ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3492024-02-20T04:59:51.509Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502024-02-20T04:59:51.509Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3512024-02-20T04:59:51.509Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3522024-02-20T04:59:51.509Z0+0 records in
3532024-02-20T04:59:51.509Z0+0 records out
3542024-02-20T04:59:51.509Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3552024-02-20T04:59:51.509Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3562024-02-20T04:59:51.509Z+ set +x
3572024-02-20T04:59:51.509ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3582024-02-20T04:59:51.509Z++ zpool list -o name
3592024-02-20T04:59:51.534Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3602024-02-20T04:59:51.535Z+ [[ -z '' ]]
3612024-02-20T04:59:51.535Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3622024-02-20T04:59:51.560Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3632024-02-20T04:59:51.560Z+ set +x
3642024-02-20T04:59:51.560ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3652024-02-20T04:59:51.560Z+ for ZPOOL in "${ZPOOLS[@]}"
3662024-02-20T04:59:51.560Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3672024-02-20T04:59:51.560ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3682024-02-20T04:59:51.560Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3692024-02-20T04:59:51.560Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3702024-02-20T04:59:51.560Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3712024-02-20T04:59:51.560Z0+0 records in
3722024-02-20T04:59:51.560Z0+0 records out
3732024-02-20T04:59:51.560Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3742024-02-20T04:59:51.560Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3752024-02-20T04:59:51.560Z+ set +x
3762024-02-20T04:59:51.560ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3772024-02-20T04:59:51.560Z++ zpool list -o name
3782024-02-20T04:59:51.560Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3792024-02-20T04:59:51.560Z+ [[ -z '' ]]
3802024-02-20T04:59:51.560Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3812024-02-20T04:59:51.587Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3822024-02-20T04:59:51.587Z+ set +x
3832024-02-20T04:59:51.587ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3842024-02-20T04:59:51.587Z+ for ZPOOL in "${ZPOOLS[@]}"
3852024-02-20T04:59:51.587Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3862024-02-20T04:59:51.587ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3872024-02-20T04:59:51.587Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3882024-02-20T04:59:51.587Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3892024-02-20T04:59:51.587Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3902024-02-20T04:59:51.613Z0+0 records in
3912024-02-20T04:59:51.614Z0+0 records out
3922024-02-20T04:59:51.614Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3932024-02-20T04:59:51.614Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3942024-02-20T04:59:51.614Z+ set +x
3952024-02-20T04:59:51.614ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3962024-02-20T04:59:51.614Z++ zpool list -o name
3972024-02-20T04:59:51.614Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3982024-02-20T04:59:51.614Z+ [[ -z '' ]]
3992024-02-20T04:59:51.614Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4002024-02-20T04:59:51.640Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4012024-02-20T04:59:51.640Z+ set +x
4022024-02-20T04:59:51.640ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4032024-02-20T04:59:51.640Z+ for ZPOOL in "${ZPOOLS[@]}"
4042024-02-20T04:59:51.640Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4052024-02-20T04:59:51.640ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4062024-02-20T04:59:51.640Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4072024-02-20T04:59:51.640Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4082024-02-20T04:59:51.640Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4092024-02-20T04:59:51.640Z0+0 records in
4102024-02-20T04:59:51.640Z0+0 records out
4112024-02-20T04:59:51.640Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4122024-02-20T04:59:51.640Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4132024-02-20T04:59:51.640Z+ set +x
4142024-02-20T04:59:51.641ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4152024-02-20T04:59:51.641Z++ zpool list -o name
4162024-02-20T04:59:51.641Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4172024-02-20T04:59:51.666Z+ [[ -z '' ]]
4182024-02-20T04:59:51.666Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192024-02-20T04:59:51.691Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4202024-02-20T04:59:51.691Z+ set +x
4212024-02-20T04:59:51.691ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4222024-02-20T04:59:51.691Z+ for ZPOOL in "${ZPOOLS[@]}"
4232024-02-20T04:59:51.691Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4242024-02-20T04:59:51.691ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4252024-02-20T04:59:51.692Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4262024-02-20T04:59:51.692Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4272024-02-20T04:59:51.692Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4282024-02-20T04:59:51.692Z0+0 records in
4292024-02-20T04:59:51.692Z0+0 records out
4302024-02-20T04:59:51.692Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4312024-02-20T04:59:51.692Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4322024-02-20T04:59:51.692Z+ set +x
4332024-02-20T04:59:51.692ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4342024-02-20T04:59:51.692Z++ zpool list -o name
4352024-02-20T04:59:51.692Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4362024-02-20T04:59:51.692Z+ [[ -z '' ]]
4372024-02-20T04:59:51.692Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4382024-02-20T04:59:51.717Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4392024-02-20T04:59:51.717Z+ set +x
4402024-02-20T04:59:51.717ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4412024-02-20T04:59:51.717Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4422024-02-20T04:59:51.717Z+ readarray -t ZPOOLS
4432024-02-20T04:59:51.742Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4442024-02-20T04:59:51.742Z++ sed 's/[ ",]//g'
4452024-02-20T04:59:51.742Z+ for ZPOOL in "${ZPOOLS[@]}"
4462024-02-20T04:59:51.742Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4472024-02-20T04:59:51.742ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4482024-02-20T04:59:51.742Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4492024-02-20T04:59:51.742Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4502024-02-20T04:59:51.742Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4512024-02-20T04:59:51.742Z0+0 records in
4522024-02-20T04:59:51.742Z0+0 records out
4532024-02-20T04:59:51.742Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4542024-02-20T04:59:51.742Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4552024-02-20T04:59:51.742Z+ set +x
4562024-02-20T04:59:51.742ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4572024-02-20T04:59:51.742Z++ zpool list -o name
4582024-02-20T04:59:51.742Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4592024-02-20T04:59:51.742Z+ [[ -z '' ]]
4602024-02-20T04:59:51.742Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4612024-02-20T04:59:51.768Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4622024-02-20T04:59:51.768Z+ set +x
4632024-02-20T04:59:51.768ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4642024-02-20T04:59:51.768Z+ for ZPOOL in "${ZPOOLS[@]}"
4652024-02-20T04:59:51.768Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4662024-02-20T04:59:51.768ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4672024-02-20T04:59:51.768Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4682024-02-20T04:59:51.768Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4692024-02-20T04:59:51.768Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4702024-02-20T04:59:51.768Z0+0 records in
4712024-02-20T04:59:51.768Z0+0 records out
4722024-02-20T04:59:51.768Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4732024-02-20T04:59:51.768Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4742024-02-20T04:59:51.768Z+ set +x
4752024-02-20T04:59:51.768ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4762024-02-20T04:59:51.793Z++ zpool list -o name
4772024-02-20T04:59:51.793Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4782024-02-20T04:59:51.793Z+ [[ -z '' ]]
4792024-02-20T04:59:51.794Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4802024-02-20T04:59:51.819Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4812024-02-20T04:59:51.819Z+ set +x
4822024-02-20T04:59:51.819ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4832024-02-20T04:59:51.819Z+ [[ zone == \z\o\n\e ]]
4842024-02-20T04:59:51.819Z+ ensure_simulated_links igb0
4852024-02-20T04:59:51.819Z+ local PHYSICAL_LINK=igb0
4862024-02-20T04:59:51.819Z+ INDICES=("0" "1")
4872024-02-20T04:59:51.819Z+ for I in "${INDICES[@]}"
4882024-02-20T04:59:51.819Z++ get_simnet_name_if_exists net0
4892024-02-20T04:59:51.819Z++ dladm show-simnet -p -o LINK net0
4902024-02-20T04:59:51.819Z++ echo ''
4912024-02-20T04:59:51.819Z+ [[ -z '' ]]
4922024-02-20T04:59:51.819Z+ dladm create-simnet -t net0
4932024-02-20T04:59:51.845Z+ dladm create-simnet -t sc0_0
4942024-02-20T04:59:51.845Z+ dladm modify-simnet -t -p net0 sc0_0
4952024-02-20T04:59:51.845Z+ dladm set-linkprop -p mtu=9000 sc0_0
4962024-02-20T04:59:51.845Z+ success 'Simnet net0/sc0_0 exists'
4972024-02-20T04:59:51.845Z+ set +x
4982024-02-20T04:59:51.845ZSimnet net0/sc0_0 exists
4992024-02-20T04:59:51.845Z+ for I in "${INDICES[@]}"
5002024-02-20T04:59:51.845Z++ get_simnet_name_if_exists net1
5012024-02-20T04:59:51.845Z++ dladm show-simnet -p -o LINK net1
5022024-02-20T04:59:51.845Z++ echo ''
5032024-02-20T04:59:51.846Z+ [[ -z '' ]]
5042024-02-20T04:59:51.846Z+ dladm create-simnet -t net1
5052024-02-20T04:59:51.846Z+ dladm create-simnet -t sc1_0
5062024-02-20T04:59:51.872Z+ dladm modify-simnet -t -p net1 sc1_0
5072024-02-20T04:59:51.872Z+ dladm set-linkprop -p mtu=9000 sc1_0
5082024-02-20T04:59:51.872Z+ success 'Simnet net1/sc1_0 exists'
5092024-02-20T04:59:51.872Z+ set +x
5102024-02-20T04:59:51.872ZSimnet net1/sc1_0 exists
5112024-02-20T04:59:51.872Z++ get_vnic_name_if_exists sc0_1
5122024-02-20T04:59:51.872Z++ dladm show-vnic -p -o LINK sc0_1
5132024-02-20T04:59:51.872Z++ echo ''
5142024-02-20T04:59:51.872Z+ [[ -z '' ]]
5152024-02-20T04:59:51.872Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162024-02-20T04:59:51.872Z+ [[ -v PROMISC_FILT_OFF ]]
5172024-02-20T04:59:51.873Z+ success 'Vnic sc0_1 exists'
5182024-02-20T04:59:51.873Z+ set +x
5192024-02-20T04:59:51.873ZVnic sc0_1 exists
5202024-02-20T04:59:51.873Z+ warn_if_no_proxy_arp
5212024-02-20T04:59:51.873Z+ [[ -v PXA_START ]]
5222024-02-20T04:59:51.873Z+ [[ -v PXA_END ]]
5232024-02-20T04:59:51.873Z+ ensure_softnpu_zone
5242024-02-20T04:59:51.901Z+ zoneadm list
5252024-02-20T04:59:51.901Z+ grep -q sidecar_softnpu
5262024-02-20T04:59:51.901Z+ [[ -f out/npuzone/npuzone ]]
5272024-02-20T04:59:51.901Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit e3ea4b495ba0a71801ded0776ae4bbd31df57e26 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5282024-02-20T04:59:51.901Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5292024-02-20T04:59:56.946Zdownloading softnpu ...done
5302024-02-20T04:59:56.971Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/libsidecar_lite.so.sha256.txt
5312024-02-20T05:00:02.416Zdownloading asic_program.so ...done
5322024-02-20T05:00:02.441Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/scadm.sha256.txt
5332024-02-20T05:00:10.919Zdownloading scadm ...done
5342024-02-20T05:00:10.945Zcreating port tfportrear0_0 for sc0_0
5352024-02-20T05:00:10.945Zcreating port tfportqsfp0_0 for sc0_1
5362024-02-20T05:00:10.945Zsoftnpu zone setup
5372024-02-20T05:00:10.945Zconfigure zone
5382024-02-20T05:00:10.976Zinstall zone
5392024-02-20T05:00:12.012Zboot zone
5402024-02-20T05:00:15.116Z[sidecar_softnpu] svcs milestone/network
5412024-02-20T05:00:15.151ZSTATE STIME FMRI
5422024-02-20T05:00:15.151Zonline 5:00:13 svc:/milestone/network:default
5432024-02-20T05:00:15.151Z[sidecar_softnpu] /softnpu/npu start
5442024-02-20T05:00:15.176Z
5452024-02-20T05:00:15.176Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5462024-02-20T05:00:15.203Z+ GATEWAY_IP=10.151.1.100
5472024-02-20T05:00:15.203Z+ echo 'Using 10.151.1.100 as gateway ip'
5482024-02-20T05:00:15.204ZUsing 10.151.1.100 as gateway ip
5492024-02-20T05:00:15.204Z+ [[ ! -v GATEWAY_MAC ]]
5502024-02-20T05:00:15.204Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5512024-02-20T05:00:15.204Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5522024-02-20T05:00:15.204Z+++ echo 18:c0:4d:0d:9f:b2
5532024-02-20T05:00:15.204Z+ COUNT=1
5542024-02-20T05:00:15.204Z+ [[ 1 -ne 1 ]]
5552024-02-20T05:00:15.204Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5562024-02-20T05:00:15.204ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5572024-02-20T05:00:15.204Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5582024-02-20T05:00:15.204Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5592024-02-20T05:00:15.231Z+ PXA_MAC=a8:e1:de:01:70:1d
5602024-02-20T05:00:15.231Z+ [[ -v PXA_START ]]
5612024-02-20T05:00:15.231Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5622024-02-20T05:00:15.231Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5632024-02-20T05:00:15.257Z+ z_scadm dump-state
5642024-02-20T05:00:15.257Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5652024-02-20T05:00:15.283Zlocal v6:
5662024-02-20T05:00:15.283Zlocal v4:
5672024-02-20T05:00:15.283Zrouter v6:
5682024-02-20T05:00:15.283Zrouter v4_idx:
5692024-02-20T05:00:15.284Zrouter v4_routes:
5702024-02-20T05:00:15.284Zresolver v4:
5712024-02-20T05:00:15.284Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5722024-02-20T05:00:15.284Zresolver v6:
5732024-02-20T05:00:15.284Znat v4:
5742024-02-20T05:00:15.284Znat_v6:
5752024-02-20T05:00:15.284Zport_mac:
5762024-02-20T05:00:15.284Zproxy arp:
5772024-02-20T05:00:15.284Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5782024-02-20T05:00:15.284Z+ success 'softnpu zone exists'
5792024-02-20T05:00:15.284Z+ set +x
5802024-02-20T05:00:15.284Zsoftnpu zone exists
5812024-02-20T05:00:15.284Z
5822024-02-20T05:00:15.284Zreal 23.967087777
5832024-02-20T05:00:15.284Zuser 0.724822976
5842024-02-20T05:00:15.284Zsys 1.579542606
5852024-02-20T05:00:15.284Ztrap 0.000082847
5862024-02-20T05:00:15.284Ztflt 0.000165028
5872024-02-20T05:00:15.284Zdflt 0.001320887
5882024-02-20T05:00:15.285Zkflt 0.000000000
5892024-02-20T05:00:15.285Zlock 18:35.542867803
5902024-02-20T05:00:15.285Zslp 1:11.529700526
5912024-02-20T05:00:15.285Zlat 0.092014339
5922024-02-20T05:00:15.285Zstop 0.363672889
5932024-02-20T05:00:15.285Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5942024-02-20T05:00:15.310Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952024-02-20T05:00:15.311Z+ SILO_NAME=recovery
5962024-02-20T05:00:15.311Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5972024-02-20T05:00:15.311Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5982024-02-20T05:00:15.311Z+ sed -i~ '
5992024-02-20T05:00:15.311Z /^external_dns_ips/c\
6002024-02-20T05:00:15.311Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6012024-02-20T05:00:15.311Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6022024-02-20T05:00:15.311Z /^first/c\
6032024-02-20T05:00:15.311Zfirst = "10.151.1.164"
6042024-02-20T05:00:15.311Z /^last/c\
6052024-02-20T05:00:15.311Zlast = "10.151.1.173"
6062024-02-20T05:00:15.312Z }
6072024-02-20T05:00:15.312Z /^\[rack_network_config/,/^$/ {
6082024-02-20T05:00:15.312Z /^infra_ip_first/c\
6092024-02-20T05:00:15.312Zinfra_ip_first = "10.151.1.174"
6102024-02-20T05:00:15.312Z /^infra_ip_last/c\
6112024-02-20T05:00:15.312Zinfra_ip_last = "10.151.1.174"
6122024-02-20T05:00:15.312Z }
6132024-02-20T05:00:15.312Z /^\[\[rack_network_config.ports/,/^$/ {
6142024-02-20T05:00:15.312Z /^routes/c\
6152024-02-20T05:00:15.312Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6162024-02-20T05:00:15.312Z /^addresses/c\
6172024-02-20T05:00:15.312Zaddresses = \["10.151.1.174/24"\]
6182024-02-20T05:00:15.313Z }
6192024-02-20T05:00:15.313Z' pkg/config-rss.toml
6202024-02-20T05:00:15.313Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6212024-02-20T05:00:15.339Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6222024-02-20T05:00:15.339Z+++ pkg/config-rss.toml Tue Feb 20 05:00:14 2024
6232024-02-20T05:00:15.339Z@@ -33,7 +33,7 @@
6242024-02-20T05:00:15.339Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6252024-02-20T05:00:15.339Z # must be contained in one of the "internal services" IP Pool ranges listed
6262024-02-20T05:00:15.339Z # below.
6272024-02-20T05:00:15.339Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6282024-02-20T05:00:15.339Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6292024-02-20T05:00:15.339Z
6302024-02-20T05:00:15.339Z # Initial TLS certificates for the external API
6312024-02-20T05:00:15.339Z #
6322024-02-20T05:00:15.339Z@@ -69,8 +69,8 @@
6332024-02-20T05:00:15.339Z #
6342024-02-20T05:00:15.340Z # For more on this and what to put here, see docs/how-to-run.adoc.
6352024-02-20T05:00:15.340Z [[internal_services_ip_pool_ranges]]
6362024-02-20T05:00:15.340Z-first = "192.168.1.20"
6372024-02-20T05:00:15.340Z-last = "192.168.1.29"
6382024-02-20T05:00:15.340Z+first = "10.151.1.164"
6392024-02-20T05:00:15.340Z+last = "10.151.1.173"
6402024-02-20T05:00:15.340Z
6412024-02-20T05:00:15.340Z # TODO - this configuration is subject to change going forward. Ultimately these
6422024-02-20T05:00:15.340Z # parameters should be provided to the control plane via wicket, but we need to
6432024-02-20T05:00:15.340Z@@ -100,9 +100,9 @@
6442024-02-20T05:00:15.340Z # You can configure multiple uplinks by repeating the following stanza
6452024-02-20T05:00:15.340Z [[rack_network_config.ports]]
6462024-02-20T05:00:15.340Z # Routes associated with this port.
6472024-02-20T05:00:15.340Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6482024-02-20T05:00:15.340Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6492024-02-20T05:00:15.340Z # Addresses associated with this port.
6502024-02-20T05:00:15.340Z-addresses = ["192.168.1.30/24"]
6512024-02-20T05:00:15.340Z+addresses = ["10.151.1.174/24"]
6522024-02-20T05:00:15.340Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6532024-02-20T05:00:15.340Z port = "qsfp0"
6542024-02-20T05:00:15.340Z # The speed of this port.
6552024-02-20T05:00:15.340Z+ true
6562024-02-20T05:00:15.340Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6572024-02-20T05:00:15.340Za pkg/config-rss.toml 7K
6582024-02-20T05:00:15.340Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6592024-02-20T05:00:15.341Z+ TLS_NAME=recovery.sys.oxide.test
6602024-02-20T05:00:15.341Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6612024-02-20T05:00:15.708Z...+....+..+.+...+...........+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+....+.....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+..+......+............+...+.+......+...+.........+..+.+...+......+...+..+......+.......+......+.........+.....+.............+..+...+..........+.....+.......+..+....+...........+...+.+.........+..+................+...............+..+.+...........+......+...............+.......+..+.+........+......+..........+.........+..............+............+...+......+............+...+......+.........+.+...+......+.....+.......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6622024-02-20T05:00:16.255Z....+....+...+..+......+...+....+...........+......+.+..+......+......+.+.....+.......+..+.+.....+.......+......+.....+....+............+.....+....+..+....+......+...+......+.....+...+............+.+..+......+.+...+...+.........+...+...+..............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+............+........+...+....+...+.........+...+..+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+....+..+.......+................................+...+......+.......+.....+..........+............+............+.....+....+.....+.........+.+..+.........+.+..+.......+.....+.............+...............+........+...+............+..................+....+........+..........+..+.+..............+......+.+...+..+..........+........................+..+....+.....+..........+......+..+.............+..+.............+...+..+...+......................+............+..+.+..+...+.......+.....+.......+......+...+............+..............+...+.+......+...+.....+.......+..................+............+.........+......+...+......+......+...+..+..........+.....................+..+......+......+....+...........+.+.....+....+........+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6632024-02-20T05:00:16.283Z-----
6642024-02-20T05:00:16.283Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6652024-02-20T05:00:16.308Za pkg/initial-tls-cert.pem 2K
6662024-02-20T05:00:16.308Za pkg/initial-tls-key.pem 4K
6672024-02-20T05:00:16.308Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6682024-02-20T05:00:16.308Z+ rmdir pkg
6692024-02-20T05:00:16.308Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6702024-02-20T05:00:16.308Z+ pfexec mkdir -p /usr/oxide
6712024-02-20T05:00:16.308Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6722024-02-20T05:00:16.997Z+ pfexec chmod +x /usr/oxide/oxide
6732024-02-20T05:00:17.024Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6742024-02-20T05:00:35.317Z+ OMICRON_NO_UNINSTALL=1
6752024-02-20T05:00:35.317Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6762024-02-20T05:00:35.343ZLogging to: /opt/oxide/work/out/LOG
6772024-02-20T05:00:35.852Z
6782024-02-20T05:00:35.878Zreal 0.527458828
6792024-02-20T05:00:35.878Zuser 0.109394006
6802024-02-20T05:00:35.879Zsys 2.180955048
6812024-02-20T05:00:35.879Ztrap 0.000017570
6822024-02-20T05:00:35.879Ztflt 0.000791830
6832024-02-20T05:00:35.879Zdflt 0.000498124
6842024-02-20T05:00:35.879Zkflt 0.000009109
6852024-02-20T05:00:35.879Zlock 47.857876978
6862024-02-20T05:00:35.879Zslp 0.741966221
6872024-02-20T05:00:35.879Zlat 0.003025289
6882024-02-20T05:00:35.879Zstop 0.224866934
6892024-02-20T05:00:35.879Z+ retry=0
6902024-02-20T05:00:35.879Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
6912024-02-20T05:00:35.879Z+ [[ 0 -gt 30 ]]
6922024-02-20T05:00:35.879Z+ sleep 1
6932024-02-20T05:00:36.864Z+ retry=1
6942024-02-20T05:00:36.864Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
6952024-02-20T05:00:36.890Z+ [[ 1 -gt 30 ]]
6962024-02-20T05:00:36.890Z+ sleep 1
6972024-02-20T05:00:37.880Z+ retry=2
6982024-02-20T05:00:37.880Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
6992024-02-20T05:00:37.909Z+ [[ 2 -gt 30 ]]
7002024-02-20T05:00:37.909Z+ sleep 1
7012024-02-20T05:00:38.894Z+ retry=3
7022024-02-20T05:00:38.894Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7032024-02-20T05:00:38.920Z+ [[ 3 -gt 30 ]]
7042024-02-20T05:00:38.920Z+ sleep 1
7052024-02-20T05:00:39.910Z+ retry=4
7062024-02-20T05:00:39.910Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7072024-02-20T05:01:05.079Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7082024-02-20T05:01:05.107Zlocal v6:
7092024-02-20T05:01:05.107Zfd12:63fd:117:7283::1
7102024-02-20T05:01:05.107Zfe80::f0:4ff:feca:6871
7112024-02-20T05:01:05.107Zlocal v4:
7122024-02-20T05:01:05.107Z10.151.1.174
7132024-02-20T05:01:05.107Zrouter v6:
7142024-02-20T05:01:05.107Zfd00:1122:3344:3::/64 -> fe80::e422:e6ff:fea5:cbdf (1)
7152024-02-20T05:01:05.107Zfd00:1122:3344:101::/64 -> fe80::e422:e6ff:fea5:cbdf (1)
7162024-02-20T05:01:05.107Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::e422:e6ff:fea5:cbdf (1)
7172024-02-20T05:01:05.107Zfd00:1122:3344:1::/64 -> fe80::e422:e6ff:fea5:cbdf (1)
7182024-02-20T05:01:05.107Zfd00:1122:3344:2::/64 -> fe80::e422:e6ff:fea5:cbdf (1)
7192024-02-20T05:01:05.107Zrouter v4_idx:
7202024-02-20T05:01:05.133Z0.0.0.0/0 -> 0
7212024-02-20T05:01:05.133Zrouter v4_routes:
7222024-02-20T05:01:05.133Z0 -> 10.151.1.100 (2)
7232024-02-20T05:01:05.133Zresolver v4:
7242024-02-20T05:01:05.133Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7252024-02-20T05:01:05.133Z10.151.1.174 -> 62:20:eb:54:e0:ea
7262024-02-20T05:01:05.133Zresolver v6:
7272024-02-20T05:01:05.133Zfe80::f0:4ff:feca:6871 -> 02:f0:04:ca:68:71
7282024-02-20T05:01:05.133Zfe80::e422:e6ff:fea5:cbdf -> e6:22:e6:a5:cb:df
7292024-02-20T05:01:05.133Znat v4:
7302024-02-20T05:01:05.133Z10.151.1.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:fc:1a
7312024-02-20T05:01:05.133Znat_v6:
7322024-02-20T05:01:05.134Zport_mac:
7332024-02-20T05:01:05.134Z2: a8:40:25:09:f0:23
7342024-02-20T05:01:05.134Z1000: a8:40:25:09:f0:02
7352024-02-20T05:01:05.134Z1: a8:40:25:09:f0:03
7362024-02-20T05:01:05.134Zproxy arp:
7372024-02-20T05:01:05.134Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7382024-02-20T05:01:05.134Z+ retry=0
7392024-02-20T05:01:05.134Z+++ zoneadm list -n
7402024-02-20T05:01:05.134Z+++ grep oxz_ntp
7412024-02-20T05:01:05.134Z++ pfexec svcs -z oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc -Hostate oxide/ntp
7422024-02-20T05:01:05.159Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7432024-02-20T05:01:05.159Z++ true
7442024-02-20T05:01:05.159Z+ [[ '' != online ]]
7452024-02-20T05:01:05.159Z+ [[ 0 -gt 60 ]]
7462024-02-20T05:01:05.159Z+ sleep 1
7472024-02-20T05:01:06.139Z+ retry=1
7482024-02-20T05:01:06.164Z+++ zoneadm list -n
7492024-02-20T05:01:06.165Z+++ grep oxz_ntp
7502024-02-20T05:01:06.165Z++ pfexec svcs -z oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc -Hostate oxide/ntp
7512024-02-20T05:01:06.190Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7522024-02-20T05:01:06.190Z++ true
7532024-02-20T05:01:06.190Z+ [[ '' != online ]]
7542024-02-20T05:01:06.190Z+ [[ 1 -gt 60 ]]
7552024-02-20T05:01:06.190Z+ sleep 1
7562024-02-20T05:01:07.170Z+ retry=2
7572024-02-20T05:01:07.195Z+++ zoneadm list -n
7582024-02-20T05:01:07.195Z+++ grep oxz_ntp
7592024-02-20T05:01:07.195Z++ pfexec svcs -z oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc -Hostate oxide/ntp
7602024-02-20T05:01:07.195Z+ [[ disabled != online ]]
7612024-02-20T05:01:07.195Z+ [[ 2 -gt 60 ]]
7622024-02-20T05:01:07.196Z+ sleep 1
7632024-02-20T05:01:08.193Z+ retry=3
7642024-02-20T05:01:08.219Z+++ zoneadm list -n
7652024-02-20T05:01:08.219Z+++ grep oxz_ntp
7662024-02-20T05:01:08.219Z++ pfexec svcs -z oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc -Hostate oxide/ntp
7672024-02-20T05:01:08.219Z+ [[ disabled != online ]]
7682024-02-20T05:01:08.219Z+ [[ 3 -gt 60 ]]
7692024-02-20T05:01:08.219Z+ sleep 1
7702024-02-20T05:01:09.217Z+ retry=4
7712024-02-20T05:01:09.243Z+++ zoneadm list -n
7722024-02-20T05:01:09.243Z+++ grep oxz_ntp
7732024-02-20T05:01:09.243Z++ pfexec svcs -z oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc -Hostate oxide/ntp
7742024-02-20T05:01:09.243Z+ [[ disabled != online ]]
7752024-02-20T05:01:09.243Z+ [[ 4 -gt 60 ]]
7762024-02-20T05:01:09.243Z+ sleep 1
7772024-02-20T05:01:10.240Z+ retry=5
7782024-02-20T05:01:10.266Z+++ zoneadm list -n
7792024-02-20T05:01:10.266Z+++ grep oxz_ntp
7802024-02-20T05:01:10.266Z++ pfexec svcs -z oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc -Hostate oxide/ntp
7812024-02-20T05:01:10.266Z+ [[ disabled != online ]]
7822024-02-20T05:01:10.266Z+ [[ 5 -gt 60 ]]
7832024-02-20T05:01:10.266Z+ sleep 1
7842024-02-20T05:01:11.262Z+ retry=6
7852024-02-20T05:01:11.287Z+++ zoneadm list -n
7862024-02-20T05:01:11.287Z+++ grep oxz_ntp
7872024-02-20T05:01:11.287Z++ pfexec svcs -z oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc -Hostate oxide/ntp
7882024-02-20T05:01:11.287Z+ [[ disabled != online ]]
7892024-02-20T05:01:11.287Z+ [[ 6 -gt 60 ]]
7902024-02-20T05:01:11.287Z+ sleep 1
7912024-02-20T05:01:12.284Z+ retry=7
7922024-02-20T05:01:12.309Z+++ zoneadm list -n
7932024-02-20T05:01:12.309Z+++ grep oxz_ntp
7942024-02-20T05:01:12.309Z++ pfexec svcs -z oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc -Hostate oxide/ntp
7952024-02-20T05:01:12.309Z+ [[ disabled != online ]]
7962024-02-20T05:01:12.309Z+ [[ 7 -gt 60 ]]
7972024-02-20T05:01:12.309Z+ sleep 1
7982024-02-20T05:01:13.307Z+ retry=8
7992024-02-20T05:01:13.335Z+++ zoneadm list -n
8002024-02-20T05:01:13.335Z+++ grep oxz_ntp
8012024-02-20T05:01:13.335Z++ pfexec svcs -z oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc -Hostate oxide/ntp
8022024-02-20T05:01:13.336Z+ [[ offline* != online ]]
8032024-02-20T05:01:13.336Z+ [[ 8 -gt 60 ]]
8042024-02-20T05:01:13.336Z+ sleep 1
 
8052024-02-20T05:01:13.338Zbuskin console login: Feb 20 05:01:12 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8062024-02-20T05:01:14.330Z+ retry=9
8072024-02-20T05:01:14.362Z+++ zoneadm list -n
8082024-02-20T05:01:14.362Z+++ grep oxz_ntp
8092024-02-20T05:01:14.362Z++ pfexec svcs -z oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc -Hostate oxide/ntp
8102024-02-20T05:01:14.362Z+ [[ online != online ]]
8112024-02-20T05:01:14.362Z+ echo 'Waited for chrony: 9s'
8122024-02-20T05:01:14.362ZWaited for chrony: 9s
8132024-02-20T05:01:14.362Z+ export RUST_BACKTRACE=1
8142024-02-20T05:01:14.362Z+ RUST_BACKTRACE=1
8152024-02-20T05:01:14.362Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8162024-02-20T05:01:14.362Z++ ./tests/bootstrap
8172024-02-20T05:01:14.406ZUsing external DNS server at 10.151.1.164:53
8182024-02-20T05:01:14.406Z2024-02-20 05:01:13.912957319 UTC: attempting to log into API
8192024-02-20T05:01:29.417Z2024-02-20 05:01:29.111281254 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8202024-02-20T05:01:30.418Z2024-02-20 05:01:30.112056568 UTC: attempting to log into API
8212024-02-20T05:01:45.450Z2024-02-20 05:01:45.143870330 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8222024-02-20T05:01:46.452Z2024-02-20 05:01:46.145762254 UTC: attempting to log into API
8232024-02-20T05:02:01.485Z2024-02-20 05:02:01.177933922 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8242024-02-20T05:02:02.486Z2024-02-20 05:02:02.179030822 UTC: attempting to log into API
8252024-02-20T05:02:17.516Z2024-02-20 05:02:17.210129883 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8262024-02-20T05:02:18.519Z2024-02-20 05:02:18.211272333 UTC: attempting to log into API
8272024-02-20T05:02:34.120Z2024-02-20 05:02:33.782035183 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8282024-02-20T05:02:35.092Z2024-02-20 05:02:34.786967117 UTC: attempting to log into API
8292024-02-20T05:02:50.150Z2024-02-20 05:02:49.876566225 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8302024-02-20T05:02:51.143Z2024-02-20 05:02:50.880808094 UTC: attempting to log into API
8312024-02-20T05:03:06.178Z2024-02-20 05:03:05.953131246 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8322024-02-20T05:03:07.175Z2024-02-20 05:03:06.957058641 UTC: attempting to log into API
 
8332024-02-20T05:03:09.945ZFeb 20 05:03:09 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
8342024-02-20T05:03:16.593ZFeb 20 05:03:15 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8352024-02-20T05:03:22.208Z2024-02-20 05:03:22.030333787 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
8362024-02-20T05:03:22.740ZFeb 20 05:03:22 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
8372024-02-20T05:03:23.214Z2024-02-20 05:03:23.034073830 UTC: attempting to log into API
 
8382024-02-20T05:03:29.940ZFeb 20 05:03:29 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
8392024-02-20T05:03:37.657ZFeb 20 05:03:36 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
8402024-02-20T05:03:38.243Z2024-02-20 05:03:38.106088176 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8412024-02-20T05:03:39.242Z2024-02-20 05:03:39.109890853 UTC: attempting to log into API
8422024-02-20T05:03:54.274Z2024-02-20 05:03:54.181971164 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8432024-02-20T05:03:55.275Z2024-02-20 05:03:55.185545160 UTC: attempting to log into API
8442024-02-20T05:04:06.054Z2024-02-20 05:04:05.995178632 UTC: login succeeded
8452024-02-20T05:04:06.087Zwaiting for nexus to come up...
8462024-02-20T05:04:06.152Zcreating IP pool... 10.151.1.175 - 10.151.1.227
8472024-02-20T05:04:06.334Zsetting up quotas...
8482024-02-20T05:04:06.376Zensuring datasets are ready...
8492024-02-20T05:04:08.686Zlet's roll.
8502024-02-20T05:04:08.712Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
8512024-02-20T05:04:08.712ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
8522024-02-20T05:04:08.712ZOXIDE_TOKEN=oxide-token-5191dce6e8b74c6c0051162d36b67719704835b1'
8532024-02-20T05:04:08.712Z++ OXIDE_HOST=https://recovery.sys.oxide.test
8542024-02-20T05:04:08.712Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
8552024-02-20T05:04:08.712Z++ OXIDE_TOKEN=oxide-token-5191dce6e8b74c6c0051162d36b67719704835b1
8562024-02-20T05:04:08.713Z+ export OXIDE_HOST OXIDE_TOKEN
8572024-02-20T05:04:08.713Z+ retry=0
8582024-02-20T05:04:08.713Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8592024-02-20T05:04:08.780Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
8602024-02-20T05:04:09.108Z{"status":"ok"}success
8612024-02-20T05:04:09.108ZProject {
8622024-02-20T05:04:09.108Z description: "some images",
8632024-02-20T05:04:09.108Z id: bcfd3cc9-a1dd-4b5b-a77b-762e7ad2f96e,
8642024-02-20T05:04:09.108Z name: Name(
8652024-02-20T05:04:09.108Z "images",
8662024-02-20T05:04:09.108Z ),
8672024-02-20T05:04:09.108Z time_created: 2024-02-20T05:04:08.808017Z,
8682024-02-20T05:04:09.108Z time_modified: 2024-02-20T05:04:08.808017Z,
8692024-02-20T05:04:09.108Z}
8702024-02-20T05:04:09.137Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
8712024-02-20T05:09:21.436Zdone!
8722024-02-20T05:09:21.537Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
8732024-02-20T05:09:21.642Zsuccess
8742024-02-20T05:09:21.642ZImage {
8752024-02-20T05:09:21.642Z block_size: ByteCount(
8762024-02-20T05:09:21.642Z 512,
8772024-02-20T05:09:21.642Z ),
8782024-02-20T05:09:21.642Z description: "debian 11 original base image",
8792024-02-20T05:09:21.668Z digest: None,
8802024-02-20T05:09:21.668Z id: 1de1c06a-25ed-4710-9ae0-2baf074543ae,
8812024-02-20T05:09:21.668Z name: Name(
8822024-02-20T05:09:21.668Z "debian11",
8832024-02-20T05:09:21.668Z ),
8842024-02-20T05:09:21.668Z os: "debian",
8852024-02-20T05:09:21.668Z project_id: None,
8862024-02-20T05:09:21.668Z size: ByteCount(
8872024-02-20T05:09:21.668Z 2147483648,
8882024-02-20T05:09:21.668Z ),
8892024-02-20T05:09:21.668Z time_created: 2024-02-20T05:09:21.374464Z,
8902024-02-20T05:09:21.668Z time_modified: 2024-02-20T05:09:21.612403Z,
8912024-02-20T05:09:21.668Z url: None,
8922024-02-20T05:09:21.668Z version: "11",
8932024-02-20T05:09:21.668Z}
8942024-02-20T05:09:21.668Z+ rm ./tests/bootstrap
8952024-02-20T05:09:21.713Z+ for test_bin in tests/*
8962024-02-20T05:09:21.713Z+ ./tests/bootstrap-c43b64681519df89
8972024-02-20T05:09:21.799Z
8982024-02-20T05:09:21.799Zrunning 0 tests
8992024-02-20T05:09:21.799Z
9002024-02-20T05:09:21.834Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9012024-02-20T05:09:21.835Z
9022024-02-20T05:09:21.835Z+ for test_bin in tests/*
9032024-02-20T05:09:21.835Z+ ./tests/end_to_end_tests-8b0279dffce37f90
9042024-02-20T05:09:22.013Z
9052024-02-20T05:09:22.014Zrunning 2 tests
9062024-02-20T05:09:22.248Ztest no_spoof::no_spoof_here ... ok
9072024-02-20T05:09:58.014Ztest instance_launch::instance_launch ... ok
9082024-02-20T05:09:58.014Z
9092024-02-20T05:09:58.041Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 36.00s
9102024-02-20T05:09:58.041Z
9112024-02-20T05:09:58.041Z+ _exit_trap
9122024-02-20T05:09:58.041Z+ local status=0
9132024-02-20T05:09:58.041Z+ [[ 0 -eq 0 ]]
9142024-02-20T05:09:58.041Z+ exit 0
9152024-02-20T05:09:58.041Zprocess exited: duration 644230 ms, exit code 0
 
9162024-02-20T05:09:58.163Zfound 72 output files
9172024-02-20T05:09:58.163Zuploading: /var/svc/log/oxide-sled-agent:default.log (1582149 bytes)
9182024-02-20T05:09:59.932Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9192024-02-20T05:09:59.933Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log (2319356 bytes)
9202024-02-20T05:10:01.609Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log" changed size mid upload: 2319356 -> 2332398
9212024-02-20T05:10:02.687Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log
9222024-02-20T05:10:02.687Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log (5045 bytes)
9232024-02-20T05:10:03.828Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log
9242024-02-20T05:10:03.828Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log (23408 bytes)
9252024-02-20T05:10:04.940Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log
9262024-02-20T05:10:04.940Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log (1400602 bytes)
9272024-02-20T05:10:05.681Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log" changed size mid upload: 1400602 -> 1422160
9282024-02-20T05:10:06.758Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log
9292024-02-20T05:10:06.758Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log (815861 bytes)
9302024-02-20T05:10:07.050Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log" changed size mid upload: 815861 -> 829017
9312024-02-20T05:10:08.126Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log
9322024-02-20T05:10:08.126Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log (1388 bytes)
9332024-02-20T05:10:09.231Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log
9342024-02-20T05:10:09.231Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log (262253 bytes)
9352024-02-20T05:10:09.466Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log" changed size mid upload: 262253 -> 268273
9362024-02-20T05:10:10.541Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log
9372024-02-20T05:10:10.541Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log (617 bytes)
9382024-02-20T05:10:11.648Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log
9392024-02-20T05:10:11.648Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log (5390 bytes)
9402024-02-20T05:10:12.751Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log
9412024-02-20T05:10:12.751Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_f211c8bf-f983-427b-8ccb-94c843449e11/root/var/svc/log/oxide-clickhouse:default.log (1100259 bytes)
9422024-02-20T05:10:13.509Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_f211c8bf-f983-427b-8ccb-94c843449e11/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 1100259 -> 1144975
9432024-02-20T05:10:14.586Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_f211c8bf-f983-427b-8ccb-94c843449e11/root/var/svc/log/oxide-clickhouse:default.log
9442024-02-20T05:10:14.586Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_f211c8bf-f983-427b-8ccb-94c843449e11/root/var/svc/log/oxide-zone-network-setup:default.log (1709 bytes)
9452024-02-20T05:10:15.686Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_f211c8bf-f983-427b-8ccb-94c843449e11/root/var/svc/log/oxide-zone-network-setup:default.log
9462024-02-20T05:10:15.686Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b0c2b5bb-f23f-4e0c-9996-9ca8408554d4/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
9472024-02-20T05:10:16.790Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b0c2b5bb-f23f-4e0c-9996-9ca8408554d4/root/var/svc/log/oxide-cockroachdb:default.log
9482024-02-20T05:10:16.790Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b0c2b5bb-f23f-4e0c-9996-9ca8408554d4/root/var/svc/log/oxide-zone-network-setup:default.log (1703 bytes)
9492024-02-20T05:10:17.892Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b0c2b5bb-f23f-4e0c-9996-9ca8408554d4/root/var/svc/log/oxide-zone-network-setup:default.log
9502024-02-20T05:10:17.892Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2bd71b24-a406-4f18-9c29-cc1e57ffdf08/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
9512024-02-20T05:10:18.992Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2bd71b24-a406-4f18-9c29-cc1e57ffdf08/root/var/svc/log/oxide-crucible-agent:default.log
9522024-02-20T05:10:18.993Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2bd71b24-a406-4f18-9c29-cc1e57ffdf08/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9532024-02-20T05:10:20.093Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2bd71b24-a406-4f18-9c29-cc1e57ffdf08/root/var/svc/log/oxide-crucible-downstairs:default.log
9542024-02-20T05:10:20.093Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_fb06b7f5-4b4b-4a23-9ac3-4966e07d3b6a/root/var/svc/log/oxide-external_dns:default.log (26945 bytes)
9552024-02-20T05:10:21.241Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_fb06b7f5-4b4b-4a23-9ac3-4966e07d3b6a/root/var/svc/log/oxide-external_dns:default.log
9562024-02-20T05:10:21.241Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_843e29e0-3cb1-4919-9ca9-f314c10d78a4/root/var/svc/log/oxide-internal_dns:default.log (107456 bytes)
9572024-02-20T05:10:22.463Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_843e29e0-3cb1-4919-9ca9-f314c10d78a4/root/var/svc/log/oxide-internal_dns:default.log
9582024-02-20T05:10:22.463Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_989ffb73-f706-4331-966e-f805d176a8c6/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
9592024-02-20T05:10:23.577Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_989ffb73-f706-4331-966e-f805d176a8c6/root/var/svc/log/oxide-cockroachdb:default.log
9602024-02-20T05:10:23.577Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_989ffb73-f706-4331-966e-f805d176a8c6/root/var/svc/log/oxide-zone-network-setup:default.log (1705 bytes)
9612024-02-20T05:10:24.675Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_989ffb73-f706-4331-966e-f805d176a8c6/root/var/svc/log/oxide-zone-network-setup:default.log
9622024-02-20T05:10:24.675Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8c58d2be-e43f-49cc-80e1-202439da3e23/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
9632024-02-20T05:10:25.781Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8c58d2be-e43f-49cc-80e1-202439da3e23/root/var/svc/log/oxide-crucible-agent:default.log
9642024-02-20T05:10:25.781Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8c58d2be-e43f-49cc-80e1-202439da3e23/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9652024-02-20T05:10:26.886Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8c58d2be-e43f-49cc-80e1-202439da3e23/root/var/svc/log/oxide-crucible-downstairs:default.log
9662024-02-20T05:10:26.886Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_e5ef7e0d-410f-4021-8d3d-999b09a1f64c/root/var/svc/log/oxide-crucible-pantry:default.log (2332 bytes)
9672024-02-20T05:10:27.988Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_e5ef7e0d-410f-4021-8d3d-999b09a1f64c/root/var/svc/log/oxide-crucible-pantry:default.log
9682024-02-20T05:10:27.988Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e1e5da67-42e0-4f44-8cee-236628c1e0b9/root/var/svc/log/oxide-external_dns:default.log (26947 bytes)
9692024-02-20T05:10:29.143Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e1e5da67-42e0-4f44-8cee-236628c1e0b9/root/var/svc/log/oxide-external_dns:default.log
9702024-02-20T05:10:29.143Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_3031bdd9-5369-4280-9ac5-aa178bbdc2af/root/var/svc/log/oxide-internal_dns:default.log (109473 bytes)
9712024-02-20T05:10:30.362Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_3031bdd9-5369-4280-9ac5-aa178bbdc2af/root/var/svc/log/oxide-internal_dns:default.log
9722024-02-20T05:10:30.362Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_2ccddf5f-2bef-4053-9ebe-9dd4e6224a52/root/var/svc/log/oxide-cockroachdb:default.log (4406 bytes)
9732024-02-20T05:10:31.468Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_2ccddf5f-2bef-4053-9ebe-9dd4e6224a52/root/var/svc/log/oxide-cockroachdb:default.log
9742024-02-20T05:10:31.468Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_2ccddf5f-2bef-4053-9ebe-9dd4e6224a52/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
9752024-02-20T05:10:33.000Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_2ccddf5f-2bef-4053-9ebe-9dd4e6224a52/root/var/svc/log/oxide-zone-network-setup:default.log
9762024-02-20T05:10:33.000Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6b9bfb56-6920-479a-8b12-f3c01f854dd9/root/var/svc/log/oxide-crucible-agent:default.log (17118 bytes)
9772024-02-20T05:10:34.150Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6b9bfb56-6920-479a-8b12-f3c01f854dd9/root/var/svc/log/oxide-crucible-agent:default.log
9782024-02-20T05:10:34.150Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6b9bfb56-6920-479a-8b12-f3c01f854dd9/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9792024-02-20T05:10:35.253Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6b9bfb56-6920-479a-8b12-f3c01f854dd9/root/var/svc/log/oxide-crucible-downstairs:default.log
9802024-02-20T05:10:35.278Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6b9bfb56-6920-479a-8b12-f3c01f854dd9/root/var/svc/log/oxide-crucible-downstairs:downstairs-cd5a7bf0-4ecb-40eb-8a34-060f742cea1a.log (5693 bytes)
9812024-02-20T05:10:36.355Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6b9bfb56-6920-479a-8b12-f3c01f854dd9/root/var/svc/log/oxide-crucible-downstairs:downstairs-cd5a7bf0-4ecb-40eb-8a34-060f742cea1a.log
9822024-02-20T05:10:36.355Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6b9bfb56-6920-479a-8b12-f3c01f854dd9/root/var/svc/log/oxide-crucible-downstairs:snapshot-cd5a7bf0-4ecb-40eb-8a34-060f742cea1a-7e5a1d92-9326-4e45-8c88-b98f38716c0d.log (5304 bytes)
9832024-02-20T05:10:37.460Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6b9bfb56-6920-479a-8b12-f3c01f854dd9/root/var/svc/log/oxide-crucible-downstairs:snapshot-cd5a7bf0-4ecb-40eb-8a34-060f742cea1a-7e5a1d92-9326-4e45-8c88-b98f38716c0d.log
9842024-02-20T05:10:37.460Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_8f10c271-0742-42bb-b6c1-dd30ee297879/root/var/svc/log/oxide-internal_dns:default.log (112309 bytes)
9852024-02-20T05:10:38.683Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_8f10c271-0742-42bb-b6c1-dd30ee297879/root/var/svc/log/oxide-internal_dns:default.log
9862024-02-20T05:10:38.710Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc/root/var/svc/log/oxide-ntp:default.log (1252 bytes)
9872024-02-20T05:10:39.790Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc/root/var/svc/log/oxide-ntp:default.log
9882024-02-20T05:10:39.791Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_6b4c13c7-e86b-448f-ab0b-f4178b30a9f7/root/var/svc/log/oxide-oximeter:default.log (297847 bytes)
9892024-02-20T05:10:40.049Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_6b4c13c7-e86b-448f-ab0b-f4178b30a9f7/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 297847 -> 331942
9902024-02-20T05:10:41.142Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_6b4c13c7-e86b-448f-ab0b-f4178b30a9f7/root/var/svc/log/oxide-oximeter:default.log
9912024-02-20T05:10:41.171Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_6b4c13c7-e86b-448f-ab0b-f4178b30a9f7/root/var/svc/log/oxide-zone-network-setup:default.log (1696 bytes)
9922024-02-20T05:10:42.245Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_6b4c13c7-e86b-448f-ab0b-f4178b30a9f7/root/var/svc/log/oxide-zone-network-setup:default.log
9932024-02-20T05:10:42.246Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_01b90361-aabc-4782-9fc5-744436b2f6b1/root/var/svc/log/oxide-cockroachdb:default.log (4379 bytes)
9942024-02-20T05:10:43.345Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_01b90361-aabc-4782-9fc5-744436b2f6b1/root/var/svc/log/oxide-cockroachdb:default.log
9952024-02-20T05:10:43.371Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_01b90361-aabc-4782-9fc5-744436b2f6b1/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
9962024-02-20T05:10:44.479Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_01b90361-aabc-4782-9fc5-744436b2f6b1/root/var/svc/log/oxide-zone-network-setup:default.log
9972024-02-20T05:10:44.479Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_df4a9b97-d93e-4264-8726-a397f078ae9e/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9982024-02-20T05:10:45.580Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_df4a9b97-d93e-4264-8726-a397f078ae9e/root/var/svc/log/oxide-crucible-agent:default.log
9992024-02-20T05:10:45.580Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_df4a9b97-d93e-4264-8726-a397f078ae9e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10002024-02-20T05:10:46.679Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_df4a9b97-d93e-4264-8726-a397f078ae9e/root/var/svc/log/oxide-crucible-downstairs:default.log
10012024-02-20T05:10:46.679Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_0c91d572-cb0e-4bac-a114-cb3a716c9a9d/root/var/svc/log/oxide-crucible-pantry:default.log (1218587 bytes)
10022024-02-20T05:10:48.466Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_0c91d572-cb0e-4bac-a114-cb3a716c9a9d/root/var/svc/log/oxide-crucible-pantry:default.log
10032024-02-20T05:10:48.492Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_887c687d-6a03-43d9-a8f0-cd6b46851ed8/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
10042024-02-20T05:10:49.567Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_887c687d-6a03-43d9-a8f0-cd6b46851ed8/root/var/svc/log/oxide-cockroachdb:default.log
10052024-02-20T05:10:49.567Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_887c687d-6a03-43d9-a8f0-cd6b46851ed8/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
10062024-02-20T05:10:50.678Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_887c687d-6a03-43d9-a8f0-cd6b46851ed8/root/var/svc/log/oxide-zone-network-setup:default.log
10072024-02-20T05:10:50.678Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_21c2cc69-6705-4917-a919-465347ec76d0/root/var/svc/log/oxide-crucible-agent:default.log (33723 bytes)
10082024-02-20T05:10:51.968Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_21c2cc69-6705-4917-a919-465347ec76d0/root/var/svc/log/oxide-crucible-agent:default.log
10092024-02-20T05:10:51.968Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_21c2cc69-6705-4917-a919-465347ec76d0/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10102024-02-20T05:10:53.064Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_21c2cc69-6705-4917-a919-465347ec76d0/root/var/svc/log/oxide-crucible-downstairs:default.log
10112024-02-20T05:10:53.064Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_21c2cc69-6705-4917-a919-465347ec76d0/root/var/svc/log/oxide-crucible-downstairs:downstairs-22ade54e-665f-4fa8-8f02-3729ca944ad9.log (5697 bytes)
10122024-02-20T05:10:54.168Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_21c2cc69-6705-4917-a919-465347ec76d0/root/var/svc/log/oxide-crucible-downstairs:downstairs-22ade54e-665f-4fa8-8f02-3729ca944ad9.log
10132024-02-20T05:10:54.168Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_21c2cc69-6705-4917-a919-465347ec76d0/root/var/svc/log/oxide-crucible-downstairs:downstairs-3161d12d-548a-40c4-95ab-da8d7b93b543.log (5614 bytes)
10142024-02-20T05:10:55.267Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_21c2cc69-6705-4917-a919-465347ec76d0/root/var/svc/log/oxide-crucible-downstairs:downstairs-3161d12d-548a-40c4-95ab-da8d7b93b543.log
10152024-02-20T05:10:55.267Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_21c2cc69-6705-4917-a919-465347ec76d0/root/var/svc/log/oxide-crucible-downstairs:downstairs-72c7b535-2019-41e5-b661-76d322685a71.log (3017 bytes)
10162024-02-20T05:10:56.373Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_21c2cc69-6705-4917-a919-465347ec76d0/root/var/svc/log/oxide-crucible-downstairs:downstairs-72c7b535-2019-41e5-b661-76d322685a71.log
10172024-02-20T05:10:56.373Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_21c2cc69-6705-4917-a919-465347ec76d0/root/var/svc/log/oxide-crucible-downstairs:snapshot-22ade54e-665f-4fa8-8f02-3729ca944ad9-7e5a1d92-9326-4e45-8c88-b98f38716c0d.log (5305 bytes)
10182024-02-20T05:10:57.926Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_21c2cc69-6705-4917-a919-465347ec76d0/root/var/svc/log/oxide-crucible-downstairs:snapshot-22ade54e-665f-4fa8-8f02-3729ca944ad9-7e5a1d92-9326-4e45-8c88-b98f38716c0d.log
10192024-02-20T05:10:57.926Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_48c70ae1-e08b-4130-a2de-f8ec8f9c96da/root/var/svc/log/oxide-crucible-pantry:default.log (2337 bytes)
10202024-02-20T05:10:59.054Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_48c70ae1-e08b-4130-a2de-f8ec8f9c96da/root/var/svc/log/oxide-crucible-pantry:default.log
10212024-02-20T05:10:59.054Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_2cdefb3b-1938-4295-be53-1bf6cf708aee/root/var/svc/log/oxide-crucible-agent:default.log (20039 bytes)
10222024-02-20T05:11:00.180Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_2cdefb3b-1938-4295-be53-1bf6cf708aee/root/var/svc/log/oxide-crucible-agent:default.log
10232024-02-20T05:11:00.180Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_2cdefb3b-1938-4295-be53-1bf6cf708aee/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10242024-02-20T05:11:01.733Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_2cdefb3b-1938-4295-be53-1bf6cf708aee/root/var/svc/log/oxide-crucible-downstairs:default.log
10252024-02-20T05:11:01.733Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_2cdefb3b-1938-4295-be53-1bf6cf708aee/root/var/svc/log/oxide-crucible-downstairs:downstairs-4749000f-db16-4bc3-827c-9951d67d99aa.log (3017 bytes)
10262024-02-20T05:11:02.835Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_2cdefb3b-1938-4295-be53-1bf6cf708aee/root/var/svc/log/oxide-crucible-downstairs:downstairs-4749000f-db16-4bc3-827c-9951d67d99aa.log
10272024-02-20T05:11:02.835Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_2cdefb3b-1938-4295-be53-1bf6cf708aee/root/var/svc/log/oxide-crucible-downstairs:downstairs-fc10bca3-3d45-4a6d-ba7c-e6ed82365753.log (3125 bytes)
10282024-02-20T05:11:03.951Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_2cdefb3b-1938-4295-be53-1bf6cf708aee/root/var/svc/log/oxide-crucible-downstairs:downstairs-fc10bca3-3d45-4a6d-ba7c-e6ed82365753.log
10292024-02-20T05:11:03.951Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_0786932d-3f01-41a7-876c-3391d644ab62/root/var/svc/log/oxide-nexus:default.log (38515970 bytes)
10302024-02-20T05:11:20.705Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_0786932d-3f01-41a7-876c-3391d644ab62/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 38515970 -> 38837393
10312024-02-20T05:11:21.807Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_0786932d-3f01-41a7-876c-3391d644ab62/root/var/svc/log/oxide-nexus:default.log
10322024-02-20T05:11:21.807Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_29367a2c-4eaa-4b32-bab9-5ee9033587d3/root/var/svc/log/oxide-crucible-agent:default.log (27794 bytes)
10332024-02-20T05:11:22.941Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_29367a2c-4eaa-4b32-bab9-5ee9033587d3/root/var/svc/log/oxide-crucible-agent:default.log
10342024-02-20T05:11:22.941Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_29367a2c-4eaa-4b32-bab9-5ee9033587d3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10352024-02-20T05:11:24.068Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_29367a2c-4eaa-4b32-bab9-5ee9033587d3/root/var/svc/log/oxide-crucible-downstairs:default.log
10362024-02-20T05:11:24.068Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_29367a2c-4eaa-4b32-bab9-5ee9033587d3/root/var/svc/log/oxide-crucible-downstairs:downstairs-5d7c79c6-4971-48de-9009-77c865a51b81.log (5614 bytes)
10372024-02-20T05:11:25.194Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_29367a2c-4eaa-4b32-bab9-5ee9033587d3/root/var/svc/log/oxide-crucible-downstairs:downstairs-5d7c79c6-4971-48de-9009-77c865a51b81.log
10382024-02-20T05:11:25.235Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_29367a2c-4eaa-4b32-bab9-5ee9033587d3/root/var/svc/log/oxide-crucible-downstairs:downstairs-d65c564b-669b-495b-9b55-6b0c8dd04dd5.log (5693 bytes)
10392024-02-20T05:11:26.317Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_29367a2c-4eaa-4b32-bab9-5ee9033587d3/root/var/svc/log/oxide-crucible-downstairs:downstairs-d65c564b-669b-495b-9b55-6b0c8dd04dd5.log
10402024-02-20T05:11:26.317Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_29367a2c-4eaa-4b32-bab9-5ee9033587d3/root/var/svc/log/oxide-crucible-downstairs:snapshot-d65c564b-669b-495b-9b55-6b0c8dd04dd5-7e5a1d92-9326-4e45-8c88-b98f38716c0d.log (5299 bytes)
10412024-02-20T05:11:27.866Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_29367a2c-4eaa-4b32-bab9-5ee9033587d3/root/var/svc/log/oxide-crucible-downstairs:snapshot-d65c564b-669b-495b-9b55-6b0c8dd04dd5-7e5a1d92-9326-4e45-8c88-b98f38716c0d.log
10422024-02-20T05:11:27.866Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_848b8e68-af88-4937-8826-8ee5665322bb/root/var/svc/log/oxide-nexus:default.log (1188640 bytes)
10432024-02-20T05:11:28.500Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_848b8e68-af88-4937-8826-8ee5665322bb/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1188640 -> 1332559
10442024-02-20T05:11:29.576Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_848b8e68-af88-4937-8826-8ee5665322bb/root/var/svc/log/oxide-nexus:default.log
10452024-02-20T05:11:29.576Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c37ac688-f579-49df-8b7b-46844c345a85/root/var/svc/log/oxide-crucible-agent:default.log (24396 bytes)
10462024-02-20T05:11:30.724Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c37ac688-f579-49df-8b7b-46844c345a85/root/var/svc/log/oxide-crucible-agent:default.log
10472024-02-20T05:11:30.724Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c37ac688-f579-49df-8b7b-46844c345a85/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10482024-02-20T05:11:31.823Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c37ac688-f579-49df-8b7b-46844c345a85/root/var/svc/log/oxide-crucible-downstairs:default.log
10492024-02-20T05:11:31.823Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c37ac688-f579-49df-8b7b-46844c345a85/root/var/svc/log/oxide-crucible-downstairs:downstairs-01217165-eb4d-4162-a853-ffdf074d66f0.log (5611 bytes)
10502024-02-20T05:11:32.926Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c37ac688-f579-49df-8b7b-46844c345a85/root/var/svc/log/oxide-crucible-downstairs:downstairs-01217165-eb4d-4162-a853-ffdf074d66f0.log
10512024-02-20T05:11:32.926Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c37ac688-f579-49df-8b7b-46844c345a85/root/var/svc/log/oxide-crucible-downstairs:downstairs-b7ad007b-84ee-4af6-8dc0-9c9d3f117ae9.log (3125 bytes)
10522024-02-20T05:11:34.507Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c37ac688-f579-49df-8b7b-46844c345a85/root/var/svc/log/oxide-crucible-downstairs:downstairs-b7ad007b-84ee-4af6-8dc0-9c9d3f117ae9.log
10532024-02-20T05:11:34.507Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_87120d0e-9c2b-40f6-90ee-01e3523ff08f/root/var/svc/log/oxide-nexus:default.log (1189708 bytes)
10542024-02-20T05:11:36.525Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_87120d0e-9c2b-40f6-90ee-01e3523ff08f/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1189708 -> 1333627
10552024-02-20T05:11:37.596Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_87120d0e-9c2b-40f6-90ee-01e3523ff08f/root/var/svc/log/oxide-nexus:default.log
10562024-02-20T05:11:37.596Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17374765-318e-4b82-8fff-41ba202df949/root/var/svc/log/oxide-crucible-agent:default.log (20468 bytes)
10572024-02-20T05:11:38.750Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17374765-318e-4b82-8fff-41ba202df949/root/var/svc/log/oxide-crucible-agent:default.log
10582024-02-20T05:11:38.805Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17374765-318e-4b82-8fff-41ba202df949/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10592024-02-20T05:11:39.902Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17374765-318e-4b82-8fff-41ba202df949/root/var/svc/log/oxide-crucible-downstairs:default.log
10602024-02-20T05:11:39.902Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17374765-318e-4b82-8fff-41ba202df949/root/var/svc/log/oxide-crucible-downstairs:downstairs-2775a9de-02c9-49cd-903c-d85b67298f06.log (3126 bytes)
10612024-02-20T05:11:41.017Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17374765-318e-4b82-8fff-41ba202df949/root/var/svc/log/oxide-crucible-downstairs:downstairs-2775a9de-02c9-49cd-903c-d85b67298f06.log
10622024-02-20T05:11:41.017Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17374765-318e-4b82-8fff-41ba202df949/root/var/svc/log/oxide-crucible-downstairs:downstairs-2b222c6e-4614-4275-ac46-65365616ab88.log (3020 bytes)
10632024-02-20T05:11:42.118Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17374765-318e-4b82-8fff-41ba202df949/root/var/svc/log/oxide-crucible-downstairs:downstairs-2b222c6e-4614-4275-ac46-65365616ab88.log
10642024-02-20T05:11:42.118Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc/root/var/log/chrony/measurements.log (18495 bytes)
10652024-02-20T05:11:42.221Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc/root/var/log/chrony/measurements.log" changed size mid upload: 18495 -> 22605
10662024-02-20T05:11:43.299Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc/root/var/log/chrony/measurements.log
10672024-02-20T05:11:43.299Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc/root/var/log/chrony/statistics.log (11700 bytes)
10682024-02-20T05:11:43.884Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc/root/var/log/chrony/statistics.log" changed size mid upload: 11700 -> 15210
10692024-02-20T05:11:44.905Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc/root/var/log/chrony/statistics.log
10702024-02-20T05:11:44.905Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc/root/var/log/chrony/tracking.log (1986 bytes)
10712024-02-20T05:11:44.959Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc/root/var/log/chrony/tracking.log" changed size mid upload: 1986 -> 2250
10722024-02-20T05:11:46.062Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_d6841929-b8f0-423a-88e0-8e06a6cc69cc/root/var/log/chrony/tracking.log