01HMZKSBEQPHYGYV6SJJ28SP40: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.28",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HMZKV9NQGTJ4Q28FXK20TF8H

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12024-01-25T07:51:13.032Zjob dependencies complete; ready to run (waiting for 1 h 40 m 51 s)
22024-01-25T07:52:00.317Zjob assigned to worker 01HMZSM0GCX2BR0XXRW9J45CHY (queued for 47 s)
32024-01-25T07:52:00.473Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42024-01-25T07:52:00.530Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52024-01-25T07:52:00.581Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62024-01-25T07:52:00.629Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72024-01-25T07:52:00.681Zhttp://172.20.0.5:9969/postboot/buskin... ok
82024-01-25T07:52:00.729ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92024-01-25T07:52:00.782Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102024-01-25T07:52:00.830ZOxide Helios Version helios-2.0.22418 64-bit
112024-01-25T07:52:00.907ZConfiguring devices.
122024-01-25T07:52:00.959Z##################################################
132024-01-25T07:52:01.230Z#### ############# ##
142024-01-25T07:52:01.308Z### ### ############ ##
152024-01-25T07:52:01.361Z## ### # ## ### ## Oxide Computer Company ##
162024-01-25T07:52:01.409Z## ## ## ### # ### ##
172024-01-25T07:52:01.461Z## # ### #### #### This Station Under ##
182024-01-25T07:52:01.508Z### ### #### # ### Computer Control ##
192024-01-25T07:52:01.561Z#### #### ### ## ##
202024-01-25T07:52:01.609Z##################################################
212024-01-25T07:52:01.665ZHostname: buskin
222024-01-25T07:52:01.717ZLoading smf(7) service descriptions:
232024-01-25T07:52:02.886Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-bb8a8abc67437a2f.gz
242024-01-25T07:52:04.757Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-bb8a8abc67437a2f.gz
252024-01-25T07:52:04.797Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262024-01-25T07:52:34.507Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272024-01-25T07:52:34.507Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-c4605ef041dbbb6b.gz
282024-01-25T07:53:05.316Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-c4605ef041dbbb6b.gz
292024-01-25T07:53:05.318Zdownloading input: /input/ci-tools/work/caboose-util.gz
302024-01-25T07:53:06.678Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312024-01-25T07:53:06.678Zdownloading input: /input/ci-tools/work/tufaceous.gz
322024-01-25T07:53:11.252Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332024-01-25T07:53:11.290Zdownloading input: /input/package/work/version.txt
342024-01-25T07:53:11.329Zdownloaded input: /input/package/work/version.txt
352024-01-25T07:53:11.329Zdownloading input: /input/package/work/package.tar.gz
362024-01-25T07:53:36.146Zdownloaded input: /input/package/work/package.tar.gz
372024-01-25T07:53:36.146Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382024-01-25T07:54:12.013Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392024-01-25T07:54:12.013Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402024-01-25T07:54:22.906Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412024-01-25T07:54:22.946Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422024-01-25T07:55:17.616Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432024-01-25T07:55:17.616Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442024-01-25T07:56:12.362Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452024-01-25T07:56:12.362Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462024-01-25T07:56:49.404Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472024-01-25T07:56:49.404Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482024-01-25T07:56:53.985Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492024-01-25T07:56:53.985Zdownloading input: /input/package/work/zones/crucible.tar.gz
502024-01-25T07:57:02.510Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512024-01-25T07:57:02.510Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522024-01-25T07:57:09.293Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532024-01-25T07:57:09.293Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542024-01-25T07:57:16.127Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552024-01-25T07:57:16.127Zdownloading input: /input/package/work/zones/ntp.tar.gz
562024-01-25T07:57:16.252Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572024-01-25T07:57:16.252Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582024-01-25T07:57:21.356Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592024-01-25T07:57:21.357Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602024-01-25T07:57:26.453Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612024-01-25T07:57:26.453Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622024-01-25T07:57:50.012Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632024-01-25T07:57:50.012Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642024-01-25T07:58:13.560Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652024-01-25T07:58:13.560Zdownloading input: /input/package/work/zones/overlay.tar.gz
662024-01-25T07:58:13.639Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672024-01-25T07:58:13.639Zdownloading input: /input/package/work/zones/oximeter.tar.gz
682024-01-25T07:58:28.437Zdownloaded input: /input/package/work/zones/oximeter.tar.gz
692024-01-25T07:58:28.437Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702024-01-25T07:58:46.604Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712024-01-25T07:58:46.604Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722024-01-25T08:00:05.397Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732024-01-25T08:00:05.397Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742024-01-25T08:01:02.391Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752024-01-25T08:01:02.431Zstarting task 0: "setup"
762024-01-25T08:01:02.431Z++ uname -s
772024-01-25T08:01:02.431Z+ kern=SunOS
782024-01-25T08:01:02.431Z+ case "$kern" in
792024-01-25T08:01:02.431Z+ groupadd -g 12345 build
802024-01-25T08:01:02.431Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812024-01-25T08:01:04.410Z+ zfs create -o mountpoint=/work rpool/work
822024-01-25T08:01:04.449Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832024-01-25T08:01:04.449Z+ home_fs=zfs
842024-01-25T08:01:04.450Z+ [[ zfs == autofs ]]
852024-01-25T08:01:04.450Z+ mkdir -p /home/build
862024-01-25T08:01:04.450Z+ chown build:build /home/build /work
872024-01-25T08:01:06.440Z+ chmod 0700 /home/build /work
882024-01-25T08:01:06.479Zprocess exited: duration 4051 ms, exit code 0
 
892024-01-25T08:01:06.557Zstarting task 1: "authentication"
902024-01-25T08:01:06.621Zprocess exited: duration 63 ms, exit code 0
 
912024-01-25T08:01:06.700Zstarting task 2: "build"
922024-01-25T08:01:06.739Z+ trap _exit_trap EXIT
932024-01-25T08:01:06.740Z+ set +x
942024-01-25T08:01:43.687Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
952024-01-25T08:01:43.937Zsecpolicy_fs_linkdir: 0x55 = 0x55
962024-01-25T08:01:43.937Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
972024-01-25T08:01:43.937Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
982024-01-25T08:01:43.937Zsecpolicy_fs_linkdir+3: 0 = 0xe5
992024-01-25T08:01:43.937Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1002024-01-25T08:01:43.938Zsecpolicy_fs_linkdir+5: 0 = 0x1
1012024-01-25T08:01:43.938Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1022024-01-25T08:01:43.980Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1032024-01-25T08:01:43.980Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1042024-01-25T08:01:43.981Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1052024-01-25T08:01:43.981Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1062024-01-25T08:01:43.981Z+ [[ -d /opt/oxide ]]
1072024-01-25T08:01:43.981Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1082024-01-25T08:01:43.981Z+ pfexec mkdir /opt/oxide
1092024-01-25T08:01:43.981Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1102024-01-25T08:01:43.982Z+ [[ -d /opt/oxide-underneath ]]
1112024-01-25T08:01:43.982Z+ cd /opt/oxide-underneath
1122024-01-25T08:01:43.982Z+ pfexec tar ceEp@/f - .
1132024-01-25T08:01:43.982Z+ cd /opt/oxide
1142024-01-25T08:01:43.982Z+ pfexec tar xveEp@/f -
1152024-01-25T08:01:44.022Zx ., 0 bytes, 0 tape blocks
1162024-01-25T08:01:44.022Zx ./opte, 0 bytes, 0 tape blocks
1172024-01-25T08:01:44.022Zx ./opte/bin, 0 bytes, 0 tape blocks
1182024-01-25T08:01:44.022Zx ./opte/bin/opteadm, 51799224 bytes, 101171 tape blocks
1192024-01-25T08:01:44.023Z+ pfexec rm -rf /opt/oxide-underneath
1202024-01-25T08:01:44.023Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1212024-01-25T08:01:44.063Z+ pfexec mkdir /opt/oxide/work
1222024-01-25T08:01:44.063Z+ pfexec chown build:build /opt/oxide/work
1232024-01-25T08:01:44.064Z+ cd /opt/oxide/work
1242024-01-25T08:01:44.064Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1252024-01-25T08:01:45.706ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1262024-01-25T08:01:45.706Zx out/mg-ddm-gz.tar, 62812160 bytes, 122680 tape blocks
1272024-01-25T08:01:45.706Zx out/omicron-sled-agent.tar, 121277952 bytes, 236871 tape blocks
1282024-01-25T08:01:45.706Zx out/oxlog.tar, 8382464 bytes, 16372 tape blocks
1292024-01-25T08:01:45.706Zx out/target/test, 75 bytes, 1 tape blocks
1302024-01-25T08:01:45.707Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1312024-01-25T08:01:45.707Zx package-manifest.toml, 23732 bytes, 47 tape blocks
1322024-01-25T08:01:45.707Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1332024-01-25T08:01:45.707Zx target/release/omicron-package, 44605632 bytes, 87121 tape blocks
1342024-01-25T08:01:45.707Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1352024-01-25T08:01:45.707Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1362024-01-25T08:01:45.707Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1372024-01-25T08:01:45.707Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1382024-01-25T08:01:45.708Zx tools/scrimlet/softnpu-init.sh, 2149 bytes, 5 tape blocks
1392024-01-25T08:01:45.708Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1402024-01-25T08:01:45.709Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1412024-01-25T08:01:45.750Z
1422024-01-25T08:01:45.750Zreal 1.661435049
1432024-01-25T08:01:45.750Zuser 1.494932534
1442024-01-25T08:01:45.750Zsys 0.288582373
1452024-01-25T08:01:45.750Ztrap 0.000073890
1462024-01-25T08:01:45.750Ztflt 0.000000000
1472024-01-25T08:01:45.750Zdflt 0.000000000
1482024-01-25T08:01:45.750Zkflt 0.000000000
1492024-01-25T08:01:45.750Zlock 0.000000000
1502024-01-25T08:01:45.750Zslp 1.497734912
1512024-01-25T08:01:45.750Zlat 0.039271091
1522024-01-25T08:01:45.750Zstop 0.000136161
1532024-01-25T08:01:45.750Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1542024-01-25T08:01:46.880Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1552024-01-25T08:01:46.920Z+ mkdir tests
1562024-01-25T08:01:46.920Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1572024-01-25T08:01:46.920Z+ ptime -m gunzip
1582024-01-25T08:01:46.920Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-bb8a8abc67437a2f
1592024-01-25T08:01:47.027Z
1602024-01-25T08:01:47.070Zreal 0.139648191
1612024-01-25T08:01:47.070Zuser 0.128325074
1622024-01-25T08:01:47.070Zsys 0.011082571
1632024-01-25T08:01:47.071Ztrap 0.000012170
1642024-01-25T08:01:47.071Ztflt 0.000000000
1652024-01-25T08:01:47.071Zdflt 0.000000000
1662024-01-25T08:01:47.071Zkflt 0.000000000
1672024-01-25T08:01:47.071Zlock 0.000000000
1682024-01-25T08:01:47.071Zslp 0.000024270
1692024-01-25T08:01:47.071Zlat 0.000022950
1702024-01-25T08:01:47.072Zstop 0.000123512
1712024-01-25T08:01:47.072Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-bb8a8abc67437a2f
1722024-01-25T08:01:47.072Z+ chmod a+x tests/bootstrap-bb8a8abc67437a2f
1732024-01-25T08:01:47.072Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1742024-01-25T08:01:47.072Z+ ptime -m gunzip
1752024-01-25T08:01:47.072Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1762024-01-25T08:01:49.439Z
1772024-01-25T08:01:49.479Zreal 2.403369179
1782024-01-25T08:01:49.479Zuser 2.270802938
1792024-01-25T08:01:49.479Zsys 0.132170325
1802024-01-25T08:01:49.479Ztrap 0.000163531
1812024-01-25T08:01:49.479Ztflt 0.000000000
1822024-01-25T08:01:49.479Zdflt 0.000000000
1832024-01-25T08:01:49.479Zkflt 0.000000000
1842024-01-25T08:01:49.479Zlock 0.000000000
1852024-01-25T08:01:49.479Zslp 0.000020500
1862024-01-25T08:01:49.480Zlat 0.000029321
1872024-01-25T08:01:49.480Zstop 0.000119601
1882024-01-25T08:01:49.480Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1892024-01-25T08:01:49.480Z+ chmod a+x tests/bootstrap
1902024-01-25T08:01:49.480Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1912024-01-25T08:01:49.480Z+ ptime -m gunzip
1922024-01-25T08:01:49.480Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-c4605ef041dbbb6b
1932024-01-25T08:01:51.929Z
1942024-01-25T08:01:51.969Zreal 2.477181292
1952024-01-25T08:01:51.969Zuser 2.327972616
1962024-01-25T08:01:51.969Zsys 0.148800598
1972024-01-25T08:01:51.969Ztrap 0.000181754
1982024-01-25T08:01:51.969Ztflt 0.000000000
1992024-01-25T08:01:51.969Zdflt 0.000000000
2002024-01-25T08:01:51.969Zkflt 0.000000000
2012024-01-25T08:01:51.969Zlock 0.000000000
2022024-01-25T08:01:51.969Zslp 0.000020900
2032024-01-25T08:01:51.969Zlat 0.000020579
2042024-01-25T08:01:51.969Zstop 0.000121572
2052024-01-25T08:01:51.969Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-c4605ef041dbbb6b
2062024-01-25T08:01:51.969Z+ chmod a+x tests/end_to_end_tests-c4605ef041dbbb6b
2072024-01-25T08:01:51.969Z+ bmat address ls
2082024-01-25T08:01:51.969ZNAME CIDR FIRST LAST COUNT
2092024-01-25T08:01:51.969Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2102024-01-25T08:01:51.969Z++ bmat address ls -f extra -Ho first,last,count
2112024-01-25T08:01:51.969Z+ set -- 10.151.1.164 10.151.1.227 64
2122024-01-25T08:01:51.970Z+ EXTRA_IP_START=10.151.1.164
2132024-01-25T08:01:51.970Z+ EXTRA_IP_END=10.151.1.227
2142024-01-25T08:01:51.972Z+ EXTRA_IP_COUNT=64
2152024-01-25T08:01:51.972Z+ (( EXTRA_IP_COUNT >= 32 ))
2162024-01-25T08:01:51.972Z+ EXTRA_IP_BASE=10.151.1
2172024-01-25T08:01:51.972Z+ EXTRA_IP_FOCTET=164
2182024-01-25T08:01:51.972Z+ EXTRA_IP_LOCTET=227
2192024-01-25T08:01:51.972Z+ SERVICE_IP_POOL_START=10.151.1.164
2202024-01-25T08:01:51.972Z+ SERVICE_IP_POOL_END=10.151.1.173
2212024-01-25T08:01:51.972Z+ DNS_IP1=10.151.1.164
2222024-01-25T08:01:51.972Z+ DNS_IP2=10.151.1.165
2232024-01-25T08:01:51.972Z+ UPLINK_IP=10.151.1.174
2242024-01-25T08:01:51.972Z+ IPPOOL_START=10.151.1.175
2252024-01-25T08:01:51.972Z+ IPPOOL_END=10.151.1.227
2262024-01-25T08:01:51.972Z++ ipadm show-addr -po type,addr
2272024-01-25T08:01:51.972Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2282024-01-25T08:01:51.972Z+ GATEWAY_IP=10.151.1.100
2292024-01-25T08:01:51.972Z+ [[ -n 10.151.1.100 ]]
2302024-01-25T08:01:51.972Z+ ping -s 10.151.1.100 56 1
2312024-01-25T08:01:51.973ZPING 10.151.1.100: 56 data bytes
2322024-01-25T08:01:52.014Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.012 ms
2332024-01-25T08:01:52.014Z
2342024-01-25T08:01:52.014Z----10.151.1.100 PING Statistics----
2352024-01-25T08:01:52.014Z1 packets transmitted, 1 packets received, 0% packet loss
2362024-01-25T08:01:52.014Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2372024-01-25T08:01:52.014Z++ arp -an
2382024-01-25T08:01:52.014Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2392024-01-25T08:01:52.014Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2402024-01-25T08:01:52.014Z+ routeadm -e ipv4-forwarding -u
2412024-01-25T08:01:52.094Z+ PXA_START=10.151.1.164
2422024-01-25T08:01:52.094Z+ PXA_END=10.151.1.227
2432024-01-25T08:01:52.133Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2442024-01-25T08:01:52.133Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2452024-01-25T08:01:52.305Z+ ZPOOL_VDEV_DIR=/scratch
2462024-01-25T08:01:52.305Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2472024-01-25T08:01:52.353Z+++ dirname ./tools/create_virtual_hardware.sh
2482024-01-25T08:01:52.353Z++ cd ./tools
2492024-01-25T08:01:52.353Z++ pwd
2502024-01-25T08:01:52.353Z+ SOURCE_DIR=/opt/oxide/work/tools
2512024-01-25T08:01:52.353Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2522024-01-25T08:01:52.354Z+ SOFTNPU_MODE=zone
2532024-01-25T08:01:52.354Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2542024-01-25T08:01:52.356Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2552024-01-25T08:01:52.356Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2562024-01-25T08:01:52.357Z++ dladm show-phys -p -o LINK
2572024-01-25T08:01:52.357Z++ head -1
2582024-01-25T08:01:52.357Z+ PHYSICAL_LINK=igb0
2592024-01-25T08:01:52.357Z+ echo 'Using igb0 as physical link'
2602024-01-25T08:01:52.357ZUsing igb0 as physical link
2612024-01-25T08:01:52.357Z+ ensure_run_as_root
2622024-01-25T08:01:52.357Z++ id -u
2632024-01-25T08:01:52.358Z+ [[ 0 -ne 0 ]]
2642024-01-25T08:01:52.358Z+ ensure_zpools
2652024-01-25T08:01:52.358Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2662024-01-25T08:01:52.358Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2672024-01-25T08:01:52.358Z+ readarray -t ZPOOLS
2682024-01-25T08:01:52.358Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2692024-01-25T08:01:52.358Z++ sed 's/[ ",]//g'
2702024-01-25T08:01:52.358Z+ for ZPOOL in "${ZPOOLS[@]}"
2712024-01-25T08:01:52.359Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2722024-01-25T08:01:52.359ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2732024-01-25T08:01:52.359Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2742024-01-25T08:01:52.359Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2752024-01-25T08:01:52.359Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2762024-01-25T08:01:52.359Z0+0 records in
2772024-01-25T08:01:52.359Z0+0 records out
2782024-01-25T08:01:52.360Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2792024-01-25T08:01:52.360Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2802024-01-25T08:01:52.360Z+ set +x
2812024-01-25T08:01:52.361ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2822024-01-25T08:01:52.361Z++ zpool list -o name
2832024-01-25T08:01:52.361Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2842024-01-25T08:01:52.411Z+ [[ -z '' ]]
2852024-01-25T08:01:52.411Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2862024-01-25T08:01:52.411Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2872024-01-25T08:01:52.411Z+ set +x
2882024-01-25T08:01:52.412ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2892024-01-25T08:01:52.412Z+ for ZPOOL in "${ZPOOLS[@]}"
2902024-01-25T08:01:52.412Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2912024-01-25T08:01:52.412ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2922024-01-25T08:01:52.412Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2932024-01-25T08:01:52.412Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2942024-01-25T08:01:52.413Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2952024-01-25T08:01:52.413Z0+0 records in
2962024-01-25T08:01:52.413Z0+0 records out
2972024-01-25T08:01:52.413Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2982024-01-25T08:01:52.413Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2992024-01-25T08:01:52.413Z+ set +x
3002024-01-25T08:01:52.413ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3012024-01-25T08:01:52.413Z++ zpool list -o name
3022024-01-25T08:01:52.414Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3032024-01-25T08:01:52.458Z+ [[ -z '' ]]
3042024-01-25T08:01:52.458Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3052024-01-25T08:01:52.458Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3062024-01-25T08:01:52.459Z+ set +x
3072024-01-25T08:01:52.459Z+ for ZPOOL in "${ZPOOLS[@]}"
3082024-01-25T08:01:52.459Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3092024-01-25T08:01:52.459ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3102024-01-25T08:01:52.459Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3112024-01-25T08:01:52.459ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3122024-01-25T08:01:52.459Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3132024-01-25T08:01:52.459Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3142024-01-25T08:01:52.459Z0+0 records in
3152024-01-25T08:01:52.460Z0+0 records out
3162024-01-25T08:01:52.460Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3172024-01-25T08:01:52.460Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3182024-01-25T08:01:52.460Z+ set +x
3192024-01-25T08:01:52.460ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3202024-01-25T08:01:52.460Z++ zpool list -o name
3212024-01-25T08:01:52.460Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3222024-01-25T08:01:52.512Z+ [[ -z '' ]]
3232024-01-25T08:01:52.512Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3242024-01-25T08:01:52.512Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3252024-01-25T08:01:52.512Z+ set +x
3262024-01-25T08:01:52.512ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3272024-01-25T08:01:52.514Z+ for ZPOOL in "${ZPOOLS[@]}"
3282024-01-25T08:01:52.515Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3292024-01-25T08:01:52.515ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3302024-01-25T08:01:52.515Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3312024-01-25T08:01:52.515Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3322024-01-25T08:01:52.515Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3332024-01-25T08:01:52.515Z0+0 records in
3342024-01-25T08:01:52.515Z0+0 records out
3352024-01-25T08:01:52.516Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3362024-01-25T08:01:52.516Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3372024-01-25T08:01:52.516Z+ set +x
3382024-01-25T08:01:52.516ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3392024-01-25T08:01:52.516Z++ zpool list -o name
3402024-01-25T08:01:52.516Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3412024-01-25T08:01:52.566Z+ [[ -z '' ]]
3422024-01-25T08:01:52.566Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3432024-01-25T08:01:52.567Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3442024-01-25T08:01:52.567Z+ set +x
3452024-01-25T08:01:52.567ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3462024-01-25T08:01:52.567Z+ for ZPOOL in "${ZPOOLS[@]}"
3472024-01-25T08:01:52.568Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3482024-01-25T08:01:52.568ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3492024-01-25T08:01:52.568Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502024-01-25T08:01:52.568Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3512024-01-25T08:01:52.568Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3522024-01-25T08:01:52.568Z0+0 records in
3532024-01-25T08:01:52.569Z0+0 records out
3542024-01-25T08:01:52.569Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3552024-01-25T08:01:52.569Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3562024-01-25T08:01:52.569Z+ set +x
3572024-01-25T08:01:52.569ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3582024-01-25T08:01:52.570Z++ zpool list -o name
3592024-01-25T08:01:52.570Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3602024-01-25T08:01:52.570Z+ [[ -z '' ]]
3612024-01-25T08:01:52.570Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3622024-01-25T08:01:52.617Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3632024-01-25T08:01:52.618Z+ set +x
3642024-01-25T08:01:52.618ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3652024-01-25T08:01:52.618Z+ for ZPOOL in "${ZPOOLS[@]}"
3662024-01-25T08:01:52.618Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3672024-01-25T08:01:52.618ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3682024-01-25T08:01:52.619Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3692024-01-25T08:01:52.619Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3702024-01-25T08:01:52.619Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3712024-01-25T08:01:52.619Z0+0 records in
3722024-01-25T08:01:52.619Z0+0 records out
3732024-01-25T08:01:52.619Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3742024-01-25T08:01:52.620Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3752024-01-25T08:01:52.620Z+ set +x
3762024-01-25T08:01:52.620ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3772024-01-25T08:01:52.620Z++ zpool list -o name
3782024-01-25T08:01:52.621Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3792024-01-25T08:01:52.621Z+ [[ -z '' ]]
3802024-01-25T08:01:52.621Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3812024-01-25T08:01:52.664Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3822024-01-25T08:01:52.664Z+ set +x
3832024-01-25T08:01:52.664ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3842024-01-25T08:01:52.664Z+ for ZPOOL in "${ZPOOLS[@]}"
3852024-01-25T08:01:52.664Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3862024-01-25T08:01:52.665ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3872024-01-25T08:01:52.665Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3882024-01-25T08:01:52.665Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3892024-01-25T08:01:52.665Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3902024-01-25T08:01:52.665Z0+0 records in
3912024-01-25T08:01:52.665Z0+0 records out
3922024-01-25T08:01:52.665Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3932024-01-25T08:01:52.666Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3942024-01-25T08:01:52.666Z+ set +x
3952024-01-25T08:01:52.666ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3962024-01-25T08:01:52.668Z++ zpool list -o name
3972024-01-25T08:01:52.668Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3982024-01-25T08:01:52.668Z+ [[ -z '' ]]
3992024-01-25T08:01:52.668Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4002024-01-25T08:01:52.714Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4012024-01-25T08:01:52.714Z+ set +x
4022024-01-25T08:01:52.715ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4032024-01-25T08:01:52.715Z+ for ZPOOL in "${ZPOOLS[@]}"
4042024-01-25T08:01:52.715Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4052024-01-25T08:01:52.715ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4062024-01-25T08:01:52.715Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4072024-01-25T08:01:52.715Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4082024-01-25T08:01:52.715Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4092024-01-25T08:01:52.715Z0+0 records in
4102024-01-25T08:01:52.715Z0+0 records out
4112024-01-25T08:01:52.715Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4122024-01-25T08:01:52.715Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4132024-01-25T08:01:52.715Z+ set +x
4142024-01-25T08:01:52.715ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4152024-01-25T08:01:52.715Z++ zpool list -o name
4162024-01-25T08:01:52.715Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4172024-01-25T08:01:52.715Z+ [[ -z '' ]]
4182024-01-25T08:01:52.715Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192024-01-25T08:01:52.755Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4202024-01-25T08:01:52.756Z+ set +x
4212024-01-25T08:01:52.756ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4222024-01-25T08:01:52.756Z+ for ZPOOL in "${ZPOOLS[@]}"
4232024-01-25T08:01:52.756Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4242024-01-25T08:01:52.757Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4252024-01-25T08:01:52.757ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4262024-01-25T08:01:52.757Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4272024-01-25T08:01:52.757Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4282024-01-25T08:01:52.758Z0+0 records in
4292024-01-25T08:01:52.758Z0+0 records out
4302024-01-25T08:01:52.758Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4312024-01-25T08:01:52.758Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4322024-01-25T08:01:52.758Z+ set +x
4332024-01-25T08:01:52.758ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4342024-01-25T08:01:52.759Z++ zpool list -o name
4352024-01-25T08:01:52.759Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4362024-01-25T08:01:52.759Z+ [[ -z '' ]]
4372024-01-25T08:01:52.759Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4382024-01-25T08:01:52.817Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4392024-01-25T08:01:52.817Z+ set +x
4402024-01-25T08:01:52.817ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4412024-01-25T08:01:52.817Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4422024-01-25T08:01:52.818Z+ readarray -t ZPOOLS
4432024-01-25T08:01:52.818Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4442024-01-25T08:01:52.818Z++ sed 's/[ ",]//g'
4452024-01-25T08:01:52.818Z+ for ZPOOL in "${ZPOOLS[@]}"
4462024-01-25T08:01:52.818Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4472024-01-25T08:01:52.818ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4482024-01-25T08:01:52.818Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4492024-01-25T08:01:52.819Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4502024-01-25T08:01:52.819Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4512024-01-25T08:01:52.819Z0+0 records in
4522024-01-25T08:01:52.819Z0+0 records out
4532024-01-25T08:01:52.819Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4542024-01-25T08:01:52.819Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4552024-01-25T08:01:52.819Z+ set +x
4562024-01-25T08:01:52.819ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4572024-01-25T08:01:52.819Z++ zpool list -o name
4582024-01-25T08:01:52.819Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4592024-01-25T08:01:52.819Z+ [[ -z '' ]]
4602024-01-25T08:01:52.819Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4612024-01-25T08:01:52.858Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4622024-01-25T08:01:52.858Z+ set +x
4632024-01-25T08:01:52.858ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4642024-01-25T08:01:52.859Z+ for ZPOOL in "${ZPOOLS[@]}"
4652024-01-25T08:01:52.859Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4662024-01-25T08:01:52.859ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4672024-01-25T08:01:52.859Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4682024-01-25T08:01:52.859Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4692024-01-25T08:01:52.859Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4702024-01-25T08:01:52.859Z0+0 records in
4712024-01-25T08:01:52.859Z0+0 records out
4722024-01-25T08:01:52.859Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4732024-01-25T08:01:52.859Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4742024-01-25T08:01:52.859Z+ set +x
4752024-01-25T08:01:52.859ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4762024-01-25T08:01:52.859Z++ zpool list -o name
4772024-01-25T08:01:52.859Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4782024-01-25T08:01:52.860Z+ [[ -z '' ]]
4792024-01-25T08:01:52.860Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4802024-01-25T08:01:52.899Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4812024-01-25T08:01:52.899Z+ set +x
4822024-01-25T08:01:52.899ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4832024-01-25T08:01:52.900Z+ [[ zone == \z\o\n\e ]]
4842024-01-25T08:01:52.900Z+ ensure_simulated_links igb0
4852024-01-25T08:01:52.900Z+ local PHYSICAL_LINK=igb0
4862024-01-25T08:01:52.900Z+ INDICES=("0" "1")
4872024-01-25T08:01:52.900Z+ for I in "${INDICES[@]}"
4882024-01-25T08:01:52.900Z++ get_simnet_name_if_exists net0
4892024-01-25T08:01:52.902Z++ dladm show-simnet -p -o LINK net0
4902024-01-25T08:01:52.902Z++ echo ''
4912024-01-25T08:01:52.902Z+ [[ -z '' ]]
4922024-01-25T08:01:52.902Z+ dladm create-simnet -t net0
4932024-01-25T08:01:52.902Z+ dladm create-simnet -t sc0_0
4942024-01-25T08:01:52.946Z+ dladm modify-simnet -t -p net0 sc0_0
4952024-01-25T08:01:52.946Z+ dladm set-linkprop -p mtu=9000 sc0_0
4962024-01-25T08:01:52.946Z+ success 'Simnet net0/sc0_0 exists'
4972024-01-25T08:01:52.946Z+ set +x
4982024-01-25T08:01:52.946ZSimnet net0/sc0_0 exists
4992024-01-25T08:01:52.947Z+ for I in "${INDICES[@]}"
5002024-01-25T08:01:52.947Z++ get_simnet_name_if_exists net1
5012024-01-25T08:01:52.949Z++ dladm show-simnet -p -o LINK net1
5022024-01-25T08:01:52.949Z++ echo ''
5032024-01-25T08:01:52.949Z+ [[ -z '' ]]
5042024-01-25T08:01:52.949Z+ dladm create-simnet -t net1
5052024-01-25T08:01:52.949Z+ dladm create-simnet -t sc1_0
5062024-01-25T08:01:52.949Z+ dladm modify-simnet -t -p net1 sc1_0
5072024-01-25T08:01:52.949Z+ dladm set-linkprop -p mtu=9000 sc1_0
5082024-01-25T08:01:52.949Z+ success 'Simnet net1/sc1_0 exists'
5092024-01-25T08:01:52.949Z+ set +x
5102024-01-25T08:01:52.949ZSimnet net1/sc1_0 exists
5112024-01-25T08:01:52.949Z++ get_vnic_name_if_exists sc0_1
5122024-01-25T08:01:52.950Z++ dladm show-vnic -p -o LINK sc0_1
5132024-01-25T08:01:52.950Z++ echo ''
5142024-01-25T08:01:52.950Z+ [[ -z '' ]]
5152024-01-25T08:01:52.950Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162024-01-25T08:01:52.989Z+ [[ -v PROMISC_FILT_OFF ]]
5172024-01-25T08:01:52.989Z+ success 'Vnic sc0_1 exists'
5182024-01-25T08:01:52.989Z+ set +x
5192024-01-25T08:01:52.989ZVnic sc0_1 exists
5202024-01-25T08:01:52.990Z+ warn_if_no_proxy_arp
5212024-01-25T08:01:52.990Z+ [[ -v PXA_START ]]
5222024-01-25T08:01:52.990Z+ [[ -v PXA_END ]]
5232024-01-25T08:01:52.990Z+ ensure_softnpu_zone
5242024-01-25T08:01:52.990Z+ zoneadm list
5252024-01-25T08:01:52.990Z+ grep -q sidecar_softnpu
5262024-01-25T08:01:52.990Z+ [[ -f out/npuzone/npuzone ]]
5272024-01-25T08:01:52.990Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit e3ea4b495ba0a71801ded0776ae4bbd31df57e26 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5282024-01-25T08:01:52.990Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5292024-01-25T08:02:00.604Zdownloading softnpu ...done
5302024-01-25T08:02:00.643Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/libsidecar_lite.so.sha256.txt
5312024-01-25T08:02:07.817Zdownloading asic_program.so ...done
5322024-01-25T08:02:07.818Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/scadm.sha256.txt
5332024-01-25T08:02:16.906Zdownloading scadm ...done
5342024-01-25T08:02:16.970Zcreating port tfportrear0_0 for sc0_0
5352024-01-25T08:02:16.970Zcreating port tfportqsfp0_0 for sc0_1
5362024-01-25T08:02:16.970Zsoftnpu zone setup
5372024-01-25T08:02:16.970Zconfigure zone
5382024-01-25T08:02:16.971Zinstall zone
5392024-01-25T08:02:18.006Zboot zone
5402024-01-25T08:02:21.110Z[sidecar_softnpu] svcs milestone/network
5412024-01-25T08:02:21.156ZSTATE STIME FMRI
5422024-01-25T08:02:21.156Zonline 8:02:19 svc:/milestone/network:default
5432024-01-25T08:02:21.156Z[sidecar_softnpu] /softnpu/npu start
5442024-01-25T08:02:21.213Z
5452024-01-25T08:02:21.213Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5462024-01-25T08:02:21.213Z+ GATEWAY_IP=10.151.1.100
5472024-01-25T08:02:21.213Z+ echo 'Using 10.151.1.100 as gateway ip'
5482024-01-25T08:02:21.213ZUsing 10.151.1.100 as gateway ip
5492024-01-25T08:02:21.213Z+ [[ ! -v GATEWAY_MAC ]]
5502024-01-25T08:02:21.213Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5512024-01-25T08:02:21.214Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5522024-01-25T08:02:21.214Z+++ echo 18:c0:4d:0d:9f:b2
5532024-01-25T08:02:21.214Z+ COUNT=1
5542024-01-25T08:02:21.214Z+ [[ 1 -ne 1 ]]
5552024-01-25T08:02:21.214Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5562024-01-25T08:02:21.214ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5572024-01-25T08:02:21.214Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5582024-01-25T08:02:21.214Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5592024-01-25T08:02:21.301Z+ PXA_MAC=a8:e1:de:01:70:1d
5602024-01-25T08:02:21.301Z+ [[ -v PXA_START ]]
5612024-01-25T08:02:21.301Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5622024-01-25T08:02:21.301Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5632024-01-25T08:02:21.301Z+ z_scadm dump-state
5642024-01-25T08:02:21.301Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5652024-01-25T08:02:21.301Zlocal v6:
5662024-01-25T08:02:21.302Zlocal v4:
5672024-01-25T08:02:21.302Zrouter v6:
5682024-01-25T08:02:21.302Zrouter v4_idx:
5692024-01-25T08:02:21.302Zrouter v4_routes:
5702024-01-25T08:02:21.302Zresolver v4:
5712024-01-25T08:02:21.302Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5722024-01-25T08:02:21.302Zresolver v6:
5732024-01-25T08:02:21.302Znat v4:
5742024-01-25T08:02:21.302Znat_v6:
5752024-01-25T08:02:21.302Zport_mac:
5762024-01-25T08:02:21.302Zproxy arp:
5772024-01-25T08:02:21.302Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5782024-01-25T08:02:21.382Z+ success 'softnpu zone exists'
5792024-01-25T08:02:21.382Z+ set +x
5802024-01-25T08:02:21.382Zsoftnpu zone exists
5812024-01-25T08:02:21.382Z
5822024-01-25T08:02:21.382Zreal 28.952422710
5832024-01-25T08:02:21.382Zuser 0.714127707
5842024-01-25T08:02:21.382Zsys 1.570135447
5852024-01-25T08:02:21.382Ztrap 0.000091033
5862024-01-25T08:02:21.382Ztflt 0.000092088
5872024-01-25T08:02:21.382Zdflt 0.001169787
5882024-01-25T08:02:21.382Zkflt 0.000000000
5892024-01-25T08:02:21.382Zlock 22:31.769635749
5902024-01-25T08:02:21.382Zslp 1:26.482717434
5912024-01-25T08:02:21.383Zlat 0.077533718
5922024-01-25T08:02:21.383Zstop 0.356673938
5932024-01-25T08:02:21.383Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5942024-01-25T08:02:21.383Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952024-01-25T08:02:21.383Z+ SILO_NAME=recovery
5962024-01-25T08:02:21.383Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5972024-01-25T08:02:21.383Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5982024-01-25T08:02:21.383Z+ sed -i~ '
5992024-01-25T08:02:21.383Z /^external_dns_ips/c\
6002024-01-25T08:02:21.383Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6012024-01-25T08:02:21.383Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6022024-01-25T08:02:21.383Z /^first/c\
6032024-01-25T08:02:21.383Zfirst = "10.151.1.164"
6042024-01-25T08:02:21.383Z /^last/c\
6052024-01-25T08:02:21.383Zlast = "10.151.1.173"
6062024-01-25T08:02:21.383Z }
6072024-01-25T08:02:21.383Z /^\[rack_network_config/,/^$/ {
6082024-01-25T08:02:21.383Z /^infra_ip_first/c\
6092024-01-25T08:02:21.383Zinfra_ip_first = "10.151.1.174"
6102024-01-25T08:02:21.383Z /^infra_ip_last/c\
6112024-01-25T08:02:21.383Zinfra_ip_last = "10.151.1.174"
6122024-01-25T08:02:21.383Z }
6132024-01-25T08:02:21.383Z /^\[\[rack_network_config.ports/,/^$/ {
6142024-01-25T08:02:21.384Z /^routes/c\
6152024-01-25T08:02:21.384Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6162024-01-25T08:02:21.384Z /^addresses/c\
6172024-01-25T08:02:21.384Zaddresses = \["10.151.1.174/24"\]
6182024-01-25T08:02:21.384Z }
6192024-01-25T08:02:21.384Z' pkg/config-rss.toml
6202024-01-25T08:02:21.384Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6212024-01-25T08:02:21.384Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6222024-01-25T08:02:21.384Z+++ pkg/config-rss.toml Thu Jan 25 08:02:21 2024
6232024-01-25T08:02:21.384Z@@ -41,7 +41,7 @@
6242024-01-25T08:02:21.384Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6252024-01-25T08:02:21.384Z # must be contained in one of the "internal services" IP Pool ranges listed
6262024-01-25T08:02:21.384Z # below.
6272024-01-25T08:02:21.384Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6282024-01-25T08:02:21.384Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6292024-01-25T08:02:21.384Z
6302024-01-25T08:02:21.384Z # Initial TLS certificates for the external API
6312024-01-25T08:02:21.384Z #
6322024-01-25T08:02:21.384Z@@ -77,8 +77,8 @@
6332024-01-25T08:02:21.384Z #
6342024-01-25T08:02:21.384Z # For more on this and what to put here, see docs/how-to-run.adoc.
6352024-01-25T08:02:21.384Z [[internal_services_ip_pool_ranges]]
6362024-01-25T08:02:21.385Z-first = "192.168.1.20"
6372024-01-25T08:02:21.385Z-last = "192.168.1.29"
6382024-01-25T08:02:21.385Z+first = "10.151.1.164"
6392024-01-25T08:02:21.385Z+last = "10.151.1.173"
6402024-01-25T08:02:21.385Z
6412024-01-25T08:02:21.385Z # TODO - this configuration is subject to change going forward. Ultimately these
6422024-01-25T08:02:21.385Z # parameters should be provided to the control plane via wicket, but we need to
6432024-01-25T08:02:21.385Z@@ -92,8 +92,8 @@
6442024-01-25T08:02:21.386Z # A range of IP addresses used by Boundary Services on the external network. In
6452024-01-25T08:02:21.386Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6462024-01-25T08:02:21.386Z # With softnpu, only one address is used.
6472024-01-25T08:02:21.386Z-infra_ip_first = "192.168.1.30"
6482024-01-25T08:02:21.386Z-infra_ip_last = "192.168.1.30"
6492024-01-25T08:02:21.386Z+infra_ip_first = "10.151.1.174"
6502024-01-25T08:02:21.386Z+infra_ip_last = "10.151.1.174"
6512024-01-25T08:02:21.386Z
6522024-01-25T08:02:21.386Z # Configurations for BGP routers to run on the scrimlets.
6532024-01-25T08:02:21.386Z bgp = []
6542024-01-25T08:02:21.386Z@@ -101,9 +101,9 @@
6552024-01-25T08:02:21.386Z # You can configure multiple uplinks by repeating the following stanza
6562024-01-25T08:02:21.386Z [[rack_network_config.ports]]
6572024-01-25T08:02:21.386Z # Routes associated with this port.
6582024-01-25T08:02:21.386Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6592024-01-25T08:02:21.386Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6602024-01-25T08:02:21.386Z # Addresses associated with this port.
6612024-01-25T08:02:21.386Z-addresses = ["192.168.1.30/24"]
6622024-01-25T08:02:21.386Z+addresses = ["10.151.1.174/24"]
6632024-01-25T08:02:21.386Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6642024-01-25T08:02:21.386Z port = "qsfp0"
6652024-01-25T08:02:21.386Z # The speed of this port.
6662024-01-25T08:02:21.386Z+ true
6672024-01-25T08:02:21.387Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6682024-01-25T08:02:21.387Za pkg/config-rss.toml 7K
6692024-01-25T08:02:21.387Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6702024-01-25T08:02:21.387Z+ TLS_NAME=recovery.sys.oxide.test
6712024-01-25T08:02:21.389Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6722024-01-25T08:02:22.021Z...+...+.+.....+.............+...+...+..+.......+..+...+...+..........+..+.+.....+....+..+.+...............+..+...+....+..................+...........+......+...+.......+...+........+...+.......+...+......+.....+.+..+.+...........+....+..+............+...+......................+..+....+..............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...+..+...+.+.....+.+........+.+........+..........+..+......+....+......+.....+.......+..+.+...+...............+......+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+...+..........+.....+.......+.....+.......+......+.....+..........+..+.........+..................+.......+......+..+.............+...+.................+....+...+.....+.......+...............+..............+.+.........+.....+....+........+..........+...+............+............+..+...+.+......+.........+......+..............+................+...+.....+...+...............+.......+........+....+........+.......+...........+...+.......+........+..............................+.+.....+...............+.+..+...+.........+...+.+...................................+...............+.........+...+.......+..+................+...+...........+....+......+...........+............+.......+........+...+....+........+.............+........+........................+.......+...+...+..............................+...+..+...................+...+...+.................+..........+........+....+..............+....+...+...+.....+......+...+......+.......+...............+........+......+.................................+.......+......+...+......+...........+....+.....+.............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6732024-01-25T08:02:22.291Z...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+..........+..+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.+......+..+.......+......+...+...........+......+.........+....+.....+.......+............+.....+......+............+.......+...+.....+........................+....+......+.....................+...+......+......+..+......+...........................+.............+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6742024-01-25T08:02:22.331Z-----
6752024-01-25T08:02:22.331Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6762024-01-25T08:02:22.331Za pkg/initial-tls-cert.pem 2K
6772024-01-25T08:02:22.331Za pkg/initial-tls-key.pem 4K
6782024-01-25T08:02:22.331Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6792024-01-25T08:02:22.331Z+ rmdir pkg
6802024-01-25T08:02:22.331Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6812024-01-25T08:02:22.331Z+ pfexec mkdir -p /usr/oxide
6822024-01-25T08:02:22.331Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6832024-01-25T08:02:23.107Z+ pfexec chmod +x /usr/oxide/oxide
6842024-01-25T08:02:23.151Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6852024-01-25T08:02:41.420Z+ OMICRON_NO_UNINSTALL=1
6862024-01-25T08:02:41.420Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6872024-01-25T08:02:41.458ZJan 25 08:02:41.184 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6882024-01-25T08:02:41.458ZJan 25 08:02:41.186 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6892024-01-25T08:02:41.458ZJan 25 08:02:41.186 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6902024-01-25T08:02:41.459ZJan 25 08:02:41.186 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6912024-01-25T08:02:41.459ZJan 25 08:02:41.186 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6922024-01-25T08:02:41.459ZJan 25 08:02:41.186 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6932024-01-25T08:02:41.459ZJan 25 08:02:41.186 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6942024-01-25T08:02:41.459ZJan 25 08:02:41.186 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6952024-01-25T08:02:41.461ZJan 25 08:02:41.186 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6962024-01-25T08:02:41.461ZJan 25 08:02:41.186 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6972024-01-25T08:02:41.461ZJan 25 08:02:41.186 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter.tar.gz
6982024-01-25T08:02:41.461ZJan 25 08:02:41.186 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6992024-01-25T08:02:41.461ZJan 25 08:02:41.186 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7002024-01-25T08:02:41.461ZJan 25 08:02:41.186 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7012024-01-25T08:02:41.461ZJan 25 08:02:41.186 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7022024-01-25T08:02:41.461ZJan 25 08:02:41.187 INFO Installing service, dst: /opt/oxide/oxlog.tar, src: out/oxlog.tar
7032024-01-25T08:02:41.461ZJan 25 08:02:41.187 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7042024-01-25T08:02:41.461ZJan 25 08:02:41.187 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7052024-01-25T08:02:41.679ZJan 25 08:02:41.428 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062024-01-25T08:02:41.717ZJan 25 08:02:41.466 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072024-01-25T08:02:41.784ZJan 25 08:02:41.532 INFO Unpacking service tarball, service_path: /opt/oxide/oxlog, tar_path: /opt/oxide/oxlog.tar
7082024-01-25T08:02:41.825ZJan 25 08:02:41.537 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7092024-01-25T08:02:41.918Z
7102024-01-25T08:02:41.961Zreal 0.491562396
7112024-01-25T08:02:41.962Zuser 0.107353283
7122024-01-25T08:02:41.962Zsys 1.959056332
7132024-01-25T08:02:41.962Ztrap 0.000002740
7142024-01-25T08:02:41.962Ztflt 0.000324662
7152024-01-25T08:02:41.962Zdflt 0.000347624
7162024-01-25T08:02:41.962Zkflt 0.000000000
7172024-01-25T08:02:41.962Zlock 44.144909129
7182024-01-25T08:02:41.962Zslp 0.708991209
7192024-01-25T08:02:41.962Zlat 0.002946995
7202024-01-25T08:02:41.962Zstop 0.196456254
7212024-01-25T08:02:41.963Z+ retry=0
7222024-01-25T08:02:41.963Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232024-01-25T08:02:41.963Z+ [[ 0 -gt 30 ]]
7242024-01-25T08:02:41.963Z+ sleep 1
7252024-01-25T08:02:42.933Z+ retry=1
7262024-01-25T08:02:42.933Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272024-01-25T08:02:42.981Z+ [[ 1 -gt 30 ]]
7282024-01-25T08:02:42.981Z+ sleep 1
7292024-01-25T08:02:43.946Z+ retry=2
7302024-01-25T08:02:43.946Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312024-01-25T08:02:43.985Z+ [[ 2 -gt 30 ]]
7322024-01-25T08:02:43.985Z+ sleep 1
7332024-01-25T08:02:44.961Z+ retry=3
7342024-01-25T08:02:44.961Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352024-01-25T08:02:45.004Z+ [[ 3 -gt 30 ]]
7362024-01-25T08:02:45.004Z+ sleep 1
7372024-01-25T08:02:45.980Z+ retry=4
7382024-01-25T08:02:45.981Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7392024-01-25T08:03:11.151Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7402024-01-25T08:03:11.191Zlocal v6:
7412024-01-25T08:03:11.191Zfe80::dc75:a9ff:fec3:2802
7422024-01-25T08:03:11.191Zfd87:2463:405c:11f8::1
7432024-01-25T08:03:11.191Zlocal v4:
7442024-01-25T08:03:11.191Z10.151.1.174
7452024-01-25T08:03:11.191Zrouter v6:
7462024-01-25T08:03:11.191Zfd00:1122:3344:2::/64 -> fe80::98d9:bcff:fe61:e59c (1)
7472024-01-25T08:03:11.191Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::98d9:bcff:fe61:e59c (1)
7482024-01-25T08:03:11.192Zfd00:1122:3344:101::/64 -> fe80::98d9:bcff:fe61:e59c (1)
7492024-01-25T08:03:11.192Zfd00:1122:3344:1::/64 -> fe80::98d9:bcff:fe61:e59c (1)
7502024-01-25T08:03:11.192Zfd00:1122:3344:3::/64 -> fe80::98d9:bcff:fe61:e59c (1)
7512024-01-25T08:03:11.192Zrouter v4_idx:
7522024-01-25T08:03:11.192Z0.0.0.0/0 -> 0
7532024-01-25T08:03:11.192Zrouter v4_routes:
7542024-01-25T08:03:11.192Z0 -> 10.151.1.100 (2)
7552024-01-25T08:03:11.192Zresolver v4:
7562024-01-25T08:03:11.192Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7572024-01-25T08:03:11.192Z10.151.1.174 -> aa:c0:88:50:60:d9
7582024-01-25T08:03:11.192Zresolver v6:
7592024-01-25T08:03:11.192Zfe80::dc75:a9ff:fec3:2802 -> de:75:a9:c3:28:02
7602024-01-25T08:03:11.192Zfe80::98d9:bcff:fe61:e59c -> 9a:d9:bc:61:e5:9c
7612024-01-25T08:03:11.192Znat v4:
7622024-01-25T08:03:11.192Z10.151.1.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:f0:e8
7632024-01-25T08:03:11.192Znat_v6:
7642024-01-25T08:03:11.192Zport_mac:
7652024-01-25T08:03:11.192Z1000: a8:40:25:2b:15:9f
7662024-01-25T08:03:11.192Z1: a8:40:25:2b:15:a0
7672024-01-25T08:03:11.193Zproxy arp:
7682024-01-25T08:03:11.193Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7692024-01-25T08:03:11.193Z+ retry=0
7702024-01-25T08:03:11.193Z+++ zoneadm list -n
7712024-01-25T08:03:11.193Z+++ grep oxz_ntp
7722024-01-25T08:03:11.232Z++ pfexec svcs -z oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba -Hostate oxide/ntp
7732024-01-25T08:03:11.270Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7742024-01-25T08:03:11.270Z++ true
7752024-01-25T08:03:11.270Z+ [[ '' != online ]]
7762024-01-25T08:03:11.270Z+ [[ 0 -gt 60 ]]
7772024-01-25T08:03:11.270Z+ sleep 1
7782024-01-25T08:03:12.243Z+ retry=1
7792024-01-25T08:03:12.284Z+++ zoneadm list -n
7802024-01-25T08:03:12.284Z+++ grep oxz_ntp
7812024-01-25T08:03:12.284Z++ pfexec svcs -z oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba -Hostate oxide/ntp
7822024-01-25T08:03:12.285Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7832024-01-25T08:03:12.285Z++ true
7842024-01-25T08:03:12.285Z+ [[ '' != online ]]
7852024-01-25T08:03:12.285Z+ [[ 1 -gt 60 ]]
7862024-01-25T08:03:12.285Z+ sleep 1
7872024-01-25T08:03:13.280Z+ retry=2
7882024-01-25T08:03:13.333Z+++ zoneadm list -n
7892024-01-25T08:03:13.334Z+++ grep oxz_ntp
7902024-01-25T08:03:13.334Z++ pfexec svcs -z oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba -Hostate oxide/ntp
7912024-01-25T08:03:13.334Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7922024-01-25T08:03:13.334Z++ true
7932024-01-25T08:03:13.334Z+ [[ '' != online ]]
7942024-01-25T08:03:13.334Z+ [[ 2 -gt 60 ]]
7952024-01-25T08:03:13.334Z+ sleep 1
7962024-01-25T08:03:14.303Z+ retry=3
7972024-01-25T08:03:14.351Z+++ zoneadm list -n
7982024-01-25T08:03:14.352Z+++ grep oxz_ntp
7992024-01-25T08:03:14.352Z++ pfexec svcs -z oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba -Hostate oxide/ntp
8002024-01-25T08:03:14.352Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8012024-01-25T08:03:14.352Z++ true
8022024-01-25T08:03:14.352Z+ [[ '' != online ]]
8032024-01-25T08:03:14.352Z+ [[ 3 -gt 60 ]]
8042024-01-25T08:03:14.352Z+ sleep 1
8052024-01-25T08:03:15.333Z+ retry=4
8062024-01-25T08:03:15.380Z+++ zoneadm list -n
8072024-01-25T08:03:15.381Z+++ grep oxz_ntp
8082024-01-25T08:03:15.381Z++ pfexec svcs -z oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba -Hostate oxide/ntp
8092024-01-25T08:03:15.381Z+ [[ disabled != online ]]
8102024-01-25T08:03:15.382Z+ [[ 4 -gt 60 ]]
8112024-01-25T08:03:15.382Z+ sleep 1
8122024-01-25T08:03:16.357Z+ retry=5
8132024-01-25T08:03:16.396Z+++ zoneadm list -n
8142024-01-25T08:03:16.396Z+++ grep oxz_ntp
8152024-01-25T08:03:16.396Z++ pfexec svcs -z oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba -Hostate oxide/ntp
8162024-01-25T08:03:16.396Z+ [[ disabled != online ]]
8172024-01-25T08:03:16.396Z+ [[ 5 -gt 60 ]]
8182024-01-25T08:03:16.396Z+ sleep 1
8192024-01-25T08:03:17.377Z+ retry=6
8202024-01-25T08:03:17.418Z+++ zoneadm list -n
8212024-01-25T08:03:17.418Z+++ grep oxz_ntp
8222024-01-25T08:03:17.418Z++ pfexec svcs -z oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba -Hostate oxide/ntp
8232024-01-25T08:03:17.418Z+ [[ disabled != online ]]
8242024-01-25T08:03:17.418Z+ [[ 6 -gt 60 ]]
8252024-01-25T08:03:17.421Z+ sleep 1
8262024-01-25T08:03:18.402Z+ retry=7
8272024-01-25T08:03:18.442Z+++ zoneadm list -n
8282024-01-25T08:03:18.442Z+++ grep oxz_ntp
8292024-01-25T08:03:18.442Z++ pfexec svcs -z oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba -Hostate oxide/ntp
8302024-01-25T08:03:18.442Z+ [[ disabled != online ]]
8312024-01-25T08:03:18.442Z+ [[ 7 -gt 60 ]]
8322024-01-25T08:03:18.443Z+ sleep 1
8332024-01-25T08:03:19.420Z+ retry=8
8342024-01-25T08:03:19.459Z+++ zoneadm list -n
8352024-01-25T08:03:19.459Z+++ grep oxz_ntp
8362024-01-25T08:03:19.459Z++ pfexec svcs -z oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba -Hostate oxide/ntp
8372024-01-25T08:03:19.459Z+ [[ disabled != online ]]
8382024-01-25T08:03:19.459Z+ [[ 8 -gt 60 ]]
8392024-01-25T08:03:19.460Z+ sleep 1
8402024-01-25T08:03:20.444Z+ retry=9
8412024-01-25T08:03:20.486Z+++ zoneadm list -n
8422024-01-25T08:03:20.486Z+++ grep oxz_ntp
8432024-01-25T08:03:20.486Z++ pfexec svcs -z oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba -Hostate oxide/ntp
8442024-01-25T08:03:20.487Z+ [[ disabled != online ]]
8452024-01-25T08:03:20.487Z+ [[ 9 -gt 60 ]]
8462024-01-25T08:03:20.487Z+ sleep 1
 
8472024-01-25T08:03:21.256Zbuskin console login: Jan 25 08:03:20 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8482024-01-25T08:03:21.465Z+ retry=10
8492024-01-25T08:03:21.504Z+++ zoneadm list -n
8502024-01-25T08:03:21.504Z+++ grep oxz_ntp
8512024-01-25T08:03:21.504Z++ pfexec svcs -z oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba -Hostate oxide/ntp
8522024-01-25T08:03:21.504Z+ [[ offline* != online ]]
8532024-01-25T08:03:21.505Z+ [[ 10 -gt 60 ]]
8542024-01-25T08:03:21.505Z+ sleep 1
8552024-01-25T08:03:22.489Z+ retry=11
8562024-01-25T08:03:22.531Z+++ zoneadm list -n
8572024-01-25T08:03:22.531Z+++ grep oxz_ntp
8582024-01-25T08:03:22.531Z++ pfexec svcs -z oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba -Hostate oxide/ntp
8592024-01-25T08:03:22.531Z+ [[ online != online ]]
8602024-01-25T08:03:22.532Z+ echo 'Waited for chrony: 11s'
8612024-01-25T08:03:22.532Z+ export RUST_BACKTRACE=1
8622024-01-25T08:03:22.532ZWaited for chrony: 11s
8632024-01-25T08:03:22.532Z+ RUST_BACKTRACE=1
8642024-01-25T08:03:22.532Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8652024-01-25T08:03:22.532Z++ ./tests/bootstrap
8662024-01-25T08:03:22.533ZUsing external DNS server at 10.151.1.164:53
8672024-01-25T08:03:22.571Z2024-01-25 08:03:22.289086045 UTC: attempting to log into API
8682024-01-25T08:03:37.578Z2024-01-25 08:03:37.926279878 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8692024-01-25T08:03:38.579Z2024-01-25 08:03:38.924562157 UTC: attempting to log into API
8702024-01-25T08:03:53.614Z2024-01-25 08:03:53.919006311 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8712024-01-25T08:03:54.615Z2024-01-25 08:03:54.917049097 UTC: attempting to log into API
8722024-01-25T08:04:09.649Z2024-01-25 08:04:09.909926117 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8732024-01-25T08:04:10.681Z2024-01-25 08:04:10.907888122 UTC: attempting to log into API
8742024-01-25T08:04:25.683Z2024-01-25 08:04:25.900869213 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8752024-01-25T08:04:26.688Z2024-01-25 08:04:26.899782072 UTC: attempting to log into API
8762024-01-25T08:04:41.742Z2024-01-25 08:04:41.893650538 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8772024-01-25T08:04:42.721Z2024-01-25 08:04:42.892511116 UTC: attempting to log into API
8782024-01-25T08:04:57.755Z2024-01-25 08:04:57.886010585 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8792024-01-25T08:04:58.758Z2024-01-25 08:04:58.884986745 UTC: attempting to log into API
8802024-01-25T08:05:13.825Z2024-01-25 08:05:13.877639250 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8812024-01-25T08:05:14.793Z2024-01-25 08:05:14.876588270 UTC: attempting to log into API
8822024-01-25T08:05:29.827Z2024-01-25 08:05:29.870229681 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8832024-01-25T08:05:30.832Z2024-01-25 08:05:30.869519795 UTC: attempting to log into API
8842024-01-25T08:05:45.919Z2024-01-25 08:05:45.890701523 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8852024-01-25T08:05:46.892Z2024-01-25 08:05:46.889231196 UTC: attempting to log into API
8862024-01-25T08:06:01.936Z2024-01-25 08:06:01.916508244 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8872024-01-25T08:06:02.939Z2024-01-25 08:06:02.918591540 UTC: attempting to log into API
8882024-01-25T08:06:17.990Z2024-01-25 08:06:17.955484357 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8892024-01-25T08:06:18.978Z2024-01-25 08:06:18.956335804 UTC: attempting to log into API
8902024-01-25T08:06:34.618Z2024-01-25 08:06:34.596254931 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8912024-01-25T08:06:35.618Z2024-01-25 08:06:35.597780009 UTC: attempting to log into API
8922024-01-25T08:06:50.658Z2024-01-25 08:06:50.637808934 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8932024-01-25T08:06:51.658Z2024-01-25 08:06:51.638612430 UTC: attempting to log into API
 
8942024-01-25T08:07:02.981ZJan 25 08:07:02 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
8952024-01-25T08:07:06.700Z2024-01-25 08:07:06.679791395 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8962024-01-25T08:07:07.701Z2024-01-25 08:07:07.680801375 UTC: attempting to log into API
 
8972024-01-25T08:07:10.704ZJan 25 08:07:09 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
8982024-01-25T08:07:17.382ZJan 25 08:07:17 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
8992024-01-25T08:07:22.734Z2024-01-25 08:07:22.715342732 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9002024-01-25T08:07:23.736Z2024-01-25 08:07:23.716272772 UTC: attempting to log into API
 
9012024-01-25T08:07:24.587ZJan 25 08:07:23 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9022024-01-25T08:07:32.075ZJan 25 08:07:31 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9032024-01-25T08:07:38.768Z2024-01-25 08:07:38.749399815 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9042024-01-25T08:07:42.854Z2024-01-25 08:07:39.750275513 UTC: attempting to log into API
9052024-01-25T08:07:54.808Z2024-01-25 08:07:54.783957361 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9062024-01-25T08:07:55.805Z2024-01-25 08:07:55.785808933 UTC: attempting to log into API
9072024-01-25T08:08:01.588Z2024-01-25 08:08:01.568604595 UTC: login succeeded
9082024-01-25T08:08:01.628Zwaiting for nexus to come up...
9092024-01-25T08:08:01.694Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9102024-01-25T08:08:01.854Zsetting up quotas...
9112024-01-25T08:08:01.894Zensuring datasets are ready...
9122024-01-25T08:08:04.405Zlet's roll.
9132024-01-25T08:08:04.445Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
9142024-01-25T08:08:04.448ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
9152024-01-25T08:08:04.448ZOXIDE_TOKEN=oxide-token-0013458be67234e9ac1c8b9ecd561b76af805880'
9162024-01-25T08:08:04.448Z++ OXIDE_HOST=https://recovery.sys.oxide.test
9172024-01-25T08:08:04.448Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
9182024-01-25T08:08:04.450Z++ OXIDE_TOKEN=oxide-token-0013458be67234e9ac1c8b9ecd561b76af805880
9192024-01-25T08:08:04.450Z+ export OXIDE_HOST OXIDE_TOKEN
9202024-01-25T08:08:04.450Z+ retry=0
9212024-01-25T08:08:04.451Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9222024-01-25T08:08:04.451Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9232024-01-25T08:08:04.451Z+ [[ 0 -gt 60 ]]
9242024-01-25T08:08:04.451Z+ sleep 1
9252024-01-25T08:08:05.440Z+ retry=1
9262024-01-25T08:08:05.440Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9272024-01-25T08:08:05.480Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9282024-01-25T08:08:05.480Z+ [[ 1 -gt 60 ]]
9292024-01-25T08:08:05.480Z+ sleep 1
9302024-01-25T08:08:06.456Z+ retry=2
9312024-01-25T08:08:06.456Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9322024-01-25T08:08:06.496Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9332024-01-25T08:08:06.498Z+ [[ 2 -gt 60 ]]
9342024-01-25T08:08:06.498Z+ sleep 1
9352024-01-25T08:08:07.473Z+ retry=3
9362024-01-25T08:08:07.473Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9372024-01-25T08:08:07.513Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9382024-01-25T08:08:07.513Z+ [[ 3 -gt 60 ]]
9392024-01-25T08:08:07.513Z+ sleep 1
9402024-01-25T08:08:08.484Z+ retry=4
9412024-01-25T08:08:08.484Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9422024-01-25T08:08:08.525Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9432024-01-25T08:08:08.525Z+ [[ 4 -gt 60 ]]
9442024-01-25T08:08:08.525Z+ sleep 1
9452024-01-25T08:08:09.500Z+ retry=5
9462024-01-25T08:08:09.500Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9472024-01-25T08:08:09.539Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9482024-01-25T08:08:09.540Z+ [[ 5 -gt 60 ]]
9492024-01-25T08:08:09.540Z+ sleep 1
9502024-01-25T08:08:10.516Z+ retry=6
9512024-01-25T08:08:10.516Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9522024-01-25T08:08:10.556Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9532024-01-25T08:08:10.556Z+ [[ 6 -gt 60 ]]
9542024-01-25T08:08:10.556Z+ sleep 1
9552024-01-25T08:08:11.531Z+ retry=7
9562024-01-25T08:08:11.531Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9572024-01-25T08:08:11.571Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9582024-01-25T08:08:11.571Z+ [[ 7 -gt 60 ]]
9592024-01-25T08:08:11.571Z+ sleep 1
9602024-01-25T08:08:12.545Z+ retry=8
9612024-01-25T08:08:12.546Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9622024-01-25T08:08:12.585Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9632024-01-25T08:08:12.585Z+ [[ 8 -gt 60 ]]
9642024-01-25T08:08:12.585Z+ sleep 1
9652024-01-25T08:08:13.561Z+ retry=9
9662024-01-25T08:08:13.561Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9672024-01-25T08:08:13.600Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9682024-01-25T08:08:13.601Z+ [[ 9 -gt 60 ]]
9692024-01-25T08:08:13.601Z+ sleep 1
9702024-01-25T08:08:14.577Z+ retry=10
9712024-01-25T08:08:14.580Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9722024-01-25T08:08:14.620Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9732024-01-25T08:08:14.620Z+ [[ 10 -gt 60 ]]
9742024-01-25T08:08:14.620Z+ sleep 1
9752024-01-25T08:08:15.594Z+ retry=11
9762024-01-25T08:08:15.594Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9772024-01-25T08:08:15.634Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9782024-01-25T08:08:15.634Z+ [[ 11 -gt 60 ]]
9792024-01-25T08:08:15.634Z+ sleep 1
9802024-01-25T08:08:16.611Z+ retry=12
9812024-01-25T08:08:16.611Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9822024-01-25T08:08:16.656Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9832024-01-25T08:08:16.657Z+ [[ 12 -gt 60 ]]
9842024-01-25T08:08:16.657Z+ sleep 1
9852024-01-25T08:08:17.626Z+ retry=13
9862024-01-25T08:08:17.626Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9872024-01-25T08:08:17.666Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9882024-01-25T08:08:17.666Z+ [[ 13 -gt 60 ]]
9892024-01-25T08:08:17.666Z+ sleep 1
9902024-01-25T08:08:18.642Z+ retry=14
9912024-01-25T08:08:18.642Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9922024-01-25T08:08:18.686Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9932024-01-25T08:08:18.686Z+ [[ 14 -gt 60 ]]
9942024-01-25T08:08:18.686Z+ sleep 1
9952024-01-25T08:08:19.658Z+ retry=15
9962024-01-25T08:08:19.661Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9972024-01-25T08:08:19.701Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9982024-01-25T08:08:19.702Z+ [[ 15 -gt 60 ]]
9992024-01-25T08:08:19.702Z+ sleep 1
10002024-01-25T08:08:20.675Z+ retry=16
10012024-01-25T08:08:20.675Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10022024-01-25T08:08:20.715Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10032024-01-25T08:08:20.715Z+ [[ 16 -gt 60 ]]
10042024-01-25T08:08:20.715Z+ sleep 1
10052024-01-25T08:08:21.690Z+ retry=17
10062024-01-25T08:08:21.690Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10072024-01-25T08:08:21.730Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10082024-01-25T08:08:21.730Z+ [[ 17 -gt 60 ]]
10092024-01-25T08:08:21.730Z+ sleep 1
10102024-01-25T08:08:22.710Z+ retry=18
10112024-01-25T08:08:22.710Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10122024-01-25T08:08:22.750Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10132024-01-25T08:08:22.750Z+ [[ 18 -gt 60 ]]
10142024-01-25T08:08:22.750Z+ sleep 1
10152024-01-25T08:08:26.699Z+ retry=19
10162024-01-25T08:08:26.699Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10172024-01-25T08:08:26.754Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10182024-01-25T08:08:26.754Z+ [[ 19 -gt 60 ]]
10192024-01-25T08:08:26.755Z+ sleep 1
10202024-01-25T08:08:26.755Z+ retry=20
10212024-01-25T08:08:26.755Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10222024-01-25T08:08:26.755Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10232024-01-25T08:08:26.755Z+ [[ 20 -gt 60 ]]
10242024-01-25T08:08:26.755Z+ sleep 1
10252024-01-25T08:08:26.756Z+ retry=21
10262024-01-25T08:08:26.756Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10272024-01-25T08:08:26.757Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10282024-01-25T08:08:26.757Z+ [[ 21 -gt 60 ]]
10292024-01-25T08:08:26.757Z+ sleep 1
10302024-01-25T08:08:26.816Z+ retry=22
10312024-01-25T08:08:26.817Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10322024-01-25T08:08:26.817Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10332024-01-25T08:08:26.817Z+ [[ 22 -gt 60 ]]
10342024-01-25T08:08:26.817Z+ sleep 1
10352024-01-25T08:08:27.896Z+ retry=23
10362024-01-25T08:08:27.896Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10372024-01-25T08:08:27.896Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10382024-01-25T08:08:27.896Z+ [[ 23 -gt 60 ]]
10392024-01-25T08:08:27.896Z+ sleep 1
10402024-01-25T08:08:28.798Z+ retry=24
10412024-01-25T08:08:28.798Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10422024-01-25T08:08:28.837Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10432024-01-25T08:08:28.837Z+ [[ 24 -gt 60 ]]
10442024-01-25T08:08:28.837Z+ sleep 1
10452024-01-25T08:08:29.813Z+ retry=25
10462024-01-25T08:08:29.813Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10472024-01-25T08:08:29.856Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10482024-01-25T08:08:29.856Z+ [[ 25 -gt 60 ]]
10492024-01-25T08:08:29.856Z+ sleep 1
10502024-01-25T08:08:30.828Z+ retry=26
10512024-01-25T08:08:30.828Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10522024-01-25T08:08:30.867Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10532024-01-25T08:08:30.867Z+ [[ 26 -gt 60 ]]
10542024-01-25T08:08:30.867Z+ sleep 1
10552024-01-25T08:08:31.852Z+ retry=27
10562024-01-25T08:08:31.852Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10572024-01-25T08:08:31.892Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10582024-01-25T08:08:31.892Z+ [[ 27 -gt 60 ]]
10592024-01-25T08:08:31.892Z+ sleep 1
10602024-01-25T08:08:32.887Z+ retry=28
10612024-01-25T08:08:32.887Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10622024-01-25T08:08:32.934Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
10632024-01-25T08:08:33.282Z{"status":"ok"}success
10642024-01-25T08:08:33.284ZProject {
10652024-01-25T08:08:33.284Z description: "some images",
10662024-01-25T08:08:33.284Z id: b520b840-6d15-450e-9618-f412f00f87b0,
10672024-01-25T08:08:33.284Z name: Name(
10682024-01-25T08:08:33.284Z "images",
10692024-01-25T08:08:33.284Z ),
10702024-01-25T08:08:33.284Z time_created: 2024-01-25T08:08:32.997888Z,
10712024-01-25T08:08:33.284Z time_modified: 2024-01-25T08:08:32.997888Z,
10722024-01-25T08:08:33.284Z}
10732024-01-25T08:08:33.326Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
10742024-01-25T08:13:57.642Zdone!
10752024-01-25T08:13:57.738Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
10762024-01-25T08:13:57.840Zsuccess
10772024-01-25T08:13:57.840ZImage {
10782024-01-25T08:13:57.841Z block_size: ByteCount(
10792024-01-25T08:13:57.841Z 512,
10802024-01-25T08:13:57.841Z ),
10812024-01-25T08:13:57.841Z description: "debian 11 original base image",
10822024-01-25T08:13:57.880Z digest: None,
10832024-01-25T08:13:57.880Z id: a5885ca4-cdcb-4faa-a637-6203a36f9e9a,
10842024-01-25T08:13:57.880Z name: Name(
10852024-01-25T08:13:57.880Z "debian11",
10862024-01-25T08:13:57.881Z ),
10872024-01-25T08:13:57.883Z os: "debian",
10882024-01-25T08:13:57.883Z project_id: None,
10892024-01-25T08:13:57.883Z size: ByteCount(
10902024-01-25T08:13:57.883Z 2147483648,
10912024-01-25T08:13:57.883Z ),
10922024-01-25T08:13:57.883Z time_created: 2024-01-25T08:13:57.593649Z,
10932024-01-25T08:13:57.883Z time_modified: 2024-01-25T08:13:57.813647Z,
10942024-01-25T08:13:57.883Z url: None,
10952024-01-25T08:13:57.883Z version: "11",
10962024-01-25T08:13:57.883Z}
10972024-01-25T08:13:57.883Z+ rm ./tests/bootstrap
10982024-01-25T08:13:57.923Z+ for test_bin in tests/*
10992024-01-25T08:13:57.923Z+ ./tests/bootstrap-bb8a8abc67437a2f
11002024-01-25T08:13:57.962Z
11012024-01-25T08:13:57.962Zrunning 0 tests
11022024-01-25T08:13:57.962Z
11032024-01-25T08:13:57.963Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
11042024-01-25T08:13:57.963Z
11052024-01-25T08:13:57.963Z+ for test_bin in tests/*
11062024-01-25T08:13:57.963Z+ ./tests/end_to_end_tests-c4605ef041dbbb6b
11072024-01-25T08:13:58.045Z
11082024-01-25T08:13:58.045Zrunning 2 tests
11092024-01-25T08:13:58.269Ztest no_spoof::no_spoof_here ... ok
11102024-01-25T08:14:32.390Ztest instance_launch::instance_launch ... ok
11112024-01-25T08:14:32.390Z
11122024-01-25T08:14:32.390Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 34.34s
11132024-01-25T08:14:32.390Z
11142024-01-25T08:14:32.429Z+ _exit_trap
11152024-01-25T08:14:32.429Z+ local status=0
11162024-01-25T08:14:32.429Z+ [[ 0 -eq 0 ]]
11172024-01-25T08:14:32.429Z+ exit 0
11182024-01-25T08:14:32.429Zprocess exited: duration 805693 ms, exit code 0
 
11192024-01-25T08:14:32.623Zfound 63 output files
11202024-01-25T08:14:32.623Zuploading: /var/svc/log/oxide-sled-agent:default.log (1230442 bytes)
11212024-01-25T08:14:34.139Zuploaded: /var/svc/log/oxide-sled-agent:default.log
11222024-01-25T08:14:34.139Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_53e94480-4f5e-4a02-9ee5-30668eebc462/root/var/svc/log/oxide-clickhouse:default.log (1191567 bytes)
11232024-01-25T08:14:34.677Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_53e94480-4f5e-4a02-9ee5-30668eebc462/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 1191567 -> 1203283
11242024-01-25T08:14:35.792Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_53e94480-4f5e-4a02-9ee5-30668eebc462/root/var/svc/log/oxide-clickhouse:default.log
11252024-01-25T08:14:35.792Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_53e94480-4f5e-4a02-9ee5-30668eebc462/root/var/svc/log/oxide-zone-network-setup:default.log (1709 bytes)
11262024-01-25T08:14:36.950Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_53e94480-4f5e-4a02-9ee5-30668eebc462/root/var/svc/log/oxide-zone-network-setup:default.log
11272024-01-25T08:14:36.983Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_543c2378-6c00-417a-a3b4-6e46a9286fd6/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
11282024-01-25T08:14:38.107Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_543c2378-6c00-417a-a3b4-6e46a9286fd6/root/var/svc/log/oxide-cockroachdb:default.log
11292024-01-25T08:14:38.107Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_543c2378-6c00-417a-a3b4-6e46a9286fd6/root/var/svc/log/oxide-zone-network-setup:default.log (1708 bytes)
11302024-01-25T08:14:39.267Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_543c2378-6c00-417a-a3b4-6e46a9286fd6/root/var/svc/log/oxide-zone-network-setup:default.log
11312024-01-25T08:14:39.267Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ad03f235-64c3-437c-adf6-3d9849463ec3/root/var/svc/log/oxide-crucible-agent:default.log (14147 bytes)
11322024-01-25T08:14:40.459Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ad03f235-64c3-437c-adf6-3d9849463ec3/root/var/svc/log/oxide-crucible-agent:default.log
11332024-01-25T08:14:40.493Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ad03f235-64c3-437c-adf6-3d9849463ec3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11342024-01-25T08:14:41.613Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ad03f235-64c3-437c-adf6-3d9849463ec3/root/var/svc/log/oxide-crucible-downstairs:default.log
11352024-01-25T08:14:41.614Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ad03f235-64c3-437c-adf6-3d9849463ec3/root/var/svc/log/oxide-crucible-downstairs:downstairs-06ba3bf6-a2a3-47ac-8763-b4d33ceaca5b.log (3047 bytes)
11362024-01-25T08:14:42.772Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ad03f235-64c3-437c-adf6-3d9849463ec3/root/var/svc/log/oxide-crucible-downstairs:downstairs-06ba3bf6-a2a3-47ac-8763-b4d33ceaca5b.log
11372024-01-25T08:14:42.772Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_06481c0c-b530-4118-a59f-ad1ebc3c929c/root/var/svc/log/oxide-external_dns:default.log (26949 bytes)
11382024-01-25T08:14:42.970Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_06481c0c-b530-4118-a59f-ad1ebc3c929c/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 26949 -> 30728
11392024-01-25T08:14:44.084Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_06481c0c-b530-4118-a59f-ad1ebc3c929c/root/var/svc/log/oxide-external_dns:default.log
11402024-01-25T08:14:44.123Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_1b26b9f3-d39a-4015-8504-45023e0379d9/root/var/svc/log/oxide-internal_dns:default.log (108663 bytes)
11412024-01-25T08:14:44.347Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_1b26b9f3-d39a-4015-8504-45023e0379d9/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 108663 -> 112420
11422024-01-25T08:14:45.463Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_1b26b9f3-d39a-4015-8504-45023e0379d9/root/var/svc/log/oxide-internal_dns:default.log
11432024-01-25T08:14:45.463Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_5980c1df-2ff4-42eb-ac5c-2b3fac9f4ff8/root/var/svc/log/oxide-cockroachdb:default.log (4482 bytes)
11442024-01-25T08:14:46.621Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_5980c1df-2ff4-42eb-ac5c-2b3fac9f4ff8/root/var/svc/log/oxide-cockroachdb:default.log
11452024-01-25T08:14:46.661Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_5980c1df-2ff4-42eb-ac5c-2b3fac9f4ff8/root/var/svc/log/oxide-zone-network-setup:default.log (1711 bytes)
11462024-01-25T08:14:47.776Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_5980c1df-2ff4-42eb-ac5c-2b3fac9f4ff8/root/var/svc/log/oxide-zone-network-setup:default.log
11472024-01-25T08:14:47.816Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2566e12-74b4-4a18-95bf-52a975aaaeb1/root/var/svc/log/oxide-crucible-agent:default.log (14582 bytes)
11482024-01-25T08:14:48.987Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2566e12-74b4-4a18-95bf-52a975aaaeb1/root/var/svc/log/oxide-crucible-agent:default.log
11492024-01-25T08:14:48.987Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2566e12-74b4-4a18-95bf-52a975aaaeb1/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11502024-01-25T08:14:50.163Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2566e12-74b4-4a18-95bf-52a975aaaeb1/root/var/svc/log/oxide-crucible-downstairs:default.log
11512024-01-25T08:14:50.163Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2566e12-74b4-4a18-95bf-52a975aaaeb1/root/var/svc/log/oxide-crucible-downstairs:downstairs-848568c5-9ff2-44d7-b5db-fd4208ae339e.log (5536 bytes)
11522024-01-25T08:14:51.317Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2566e12-74b4-4a18-95bf-52a975aaaeb1/root/var/svc/log/oxide-crucible-downstairs:downstairs-848568c5-9ff2-44d7-b5db-fd4208ae339e.log
11532024-01-25T08:14:51.318Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_2da4bb81-8a26-4ae3-9e0d-d60628f38f1f/root/var/svc/log/oxide-crucible-pantry:default.log (1221999 bytes)
11542024-01-25T08:14:51.824Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_2da4bb81-8a26-4ae3-9e0d-d60628f38f1f/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 1221999 -> 1237847
11552024-01-25T08:14:52.943Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_2da4bb81-8a26-4ae3-9e0d-d60628f38f1f/root/var/svc/log/oxide-crucible-pantry:default.log
11562024-01-25T08:14:52.943Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_9682db84-c869-4da7-8df7-0c77a67d82db/root/var/svc/log/oxide-external_dns:default.log (26944 bytes)
11572024-01-25T08:14:53.095Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_9682db84-c869-4da7-8df7-0c77a67d82db/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 26944 -> 30722
11582024-01-25T08:14:54.210Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_9682db84-c869-4da7-8df7-0c77a67d82db/root/var/svc/log/oxide-external_dns:default.log
11592024-01-25T08:14:54.250Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_60aff06d-d865-4b56-8708-f8c52777c51b/root/var/svc/log/oxide-internal_dns:default.log (111509 bytes)
11602024-01-25T08:14:54.473Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_60aff06d-d865-4b56-8708-f8c52777c51b/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 111509 -> 115265
11612024-01-25T08:14:55.593Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_60aff06d-d865-4b56-8708-f8c52777c51b/root/var/svc/log/oxide-internal_dns:default.log
11622024-01-25T08:14:55.593Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5750c34e-1ad4-4c03-a58f-25d5480aafc3/root/var/svc/log/oxide-nexus:default.log (34436891 bytes)
11632024-01-25T08:15:03.232Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5750c34e-1ad4-4c03-a58f-25d5480aafc3/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 34436891 -> 26214400
11642024-01-25T08:15:04.349Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5750c34e-1ad4-4c03-a58f-25d5480aafc3/root/var/svc/log/oxide-nexus:default.log
11652024-01-25T08:15:04.389Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6195251f-cdc0-446c-adcb-9f0e0ea1a7bf/root/var/svc/log/oxide-cockroachdb:default.log (4406 bytes)
11662024-01-25T08:15:04.389Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6195251f-cdc0-446c-adcb-9f0e0ea1a7bf/root/var/svc/log/oxide-cockroachdb:default.log" changed size mid upload: 4406 -> 0
11672024-01-25T08:15:05.507Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6195251f-cdc0-446c-adcb-9f0e0ea1a7bf/root/var/svc/log/oxide-cockroachdb:default.log
11682024-01-25T08:15:05.507Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6195251f-cdc0-446c-adcb-9f0e0ea1a7bf/root/var/svc/log/oxide-zone-network-setup:default.log (1711 bytes)
11692024-01-25T08:15:05.547Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6195251f-cdc0-446c-adcb-9f0e0ea1a7bf/root/var/svc/log/oxide-zone-network-setup:default.log" changed size mid upload: 1711 -> 0
11702024-01-25T08:15:06.665Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_6195251f-cdc0-446c-adcb-9f0e0ea1a7bf/root/var/svc/log/oxide-zone-network-setup:default.log
11712024-01-25T08:15:06.665Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f44c7ffe-ac31-43e7-aefa-83fb4ee782e8/root/var/svc/log/oxide-crucible-agent:default.log (36497 bytes)
11722024-01-25T08:15:06.738Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f44c7ffe-ac31-43e7-aefa-83fb4ee782e8/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 36497 -> 0
11732024-01-25T08:15:07.863Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f44c7ffe-ac31-43e7-aefa-83fb4ee782e8/root/var/svc/log/oxide-crucible-agent:default.log
11742024-01-25T08:15:07.863Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f44c7ffe-ac31-43e7-aefa-83fb4ee782e8/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11752024-01-25T08:15:07.903Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f44c7ffe-ac31-43e7-aefa-83fb4ee782e8/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
11762024-01-25T08:15:09.011Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f44c7ffe-ac31-43e7-aefa-83fb4ee782e8/root/var/svc/log/oxide-crucible-downstairs:default.log
11772024-01-25T08:15:09.011Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f44c7ffe-ac31-43e7-aefa-83fb4ee782e8/root/var/svc/log/oxide-crucible-downstairs:downstairs-627678dd-c062-4267-85e8-a21726089427.log (5621 bytes)
11782024-01-25T08:15:09.053Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f44c7ffe-ac31-43e7-aefa-83fb4ee782e8/root/var/svc/log/oxide-crucible-downstairs:downstairs-627678dd-c062-4267-85e8-a21726089427.log" changed size mid upload: 5621 -> 0
11792024-01-25T08:15:10.164Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f44c7ffe-ac31-43e7-aefa-83fb4ee782e8/root/var/svc/log/oxide-crucible-downstairs:downstairs-627678dd-c062-4267-85e8-a21726089427.log
11802024-01-25T08:15:10.204Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f44c7ffe-ac31-43e7-aefa-83fb4ee782e8/root/var/svc/log/oxide-crucible-downstairs:downstairs-cc506288-3a8d-4b15-8a54-f50c308a06dc.log (5534 bytes)
11812024-01-25T08:15:10.204Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f44c7ffe-ac31-43e7-aefa-83fb4ee782e8/root/var/svc/log/oxide-crucible-downstairs:downstairs-cc506288-3a8d-4b15-8a54-f50c308a06dc.log" changed size mid upload: 5534 -> 0
11822024-01-25T08:15:11.317Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f44c7ffe-ac31-43e7-aefa-83fb4ee782e8/root/var/svc/log/oxide-crucible-downstairs:downstairs-cc506288-3a8d-4b15-8a54-f50c308a06dc.log
11832024-01-25T08:15:11.318Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f44c7ffe-ac31-43e7-aefa-83fb4ee782e8/root/var/svc/log/oxide-crucible-downstairs:snapshot-627678dd-c062-4267-85e8-a21726089427-816840a7-c666-4126-b4f4-54ad7c49ba03.log (5178 bytes)
11842024-01-25T08:15:11.358Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f44c7ffe-ac31-43e7-aefa-83fb4ee782e8/root/var/svc/log/oxide-crucible-downstairs:snapshot-627678dd-c062-4267-85e8-a21726089427-816840a7-c666-4126-b4f4-54ad7c49ba03.log" changed size mid upload: 5178 -> 0
11852024-01-25T08:15:12.470Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f44c7ffe-ac31-43e7-aefa-83fb4ee782e8/root/var/svc/log/oxide-crucible-downstairs:snapshot-627678dd-c062-4267-85e8-a21726089427-816840a7-c666-4126-b4f4-54ad7c49ba03.log
11862024-01-25T08:15:12.470Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_b2e21cdb-47f0-4226-92e5-fcccfa7faa5f/root/var/svc/log/oxide-internal_dns:default.log (115155 bytes)
11872024-01-25T08:15:12.510Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_b2e21cdb-47f0-4226-92e5-fcccfa7faa5f/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 115155 -> 0
11882024-01-25T08:15:13.624Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_b2e21cdb-47f0-4226-92e5-fcccfa7faa5f/root/var/svc/log/oxide-internal_dns:default.log
11892024-01-25T08:15:13.663Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_ac76a12e-fc97-4b5a-b826-45dce44dfa30/root/var/svc/log/oxide-oximeter:default.log (326852 bytes)
11902024-01-25T08:15:13.738Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_ac76a12e-fc97-4b5a-b826-45dce44dfa30/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 326852 -> 11500
11912024-01-25T08:15:14.854Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_ac76a12e-fc97-4b5a-b826-45dce44dfa30/root/var/svc/log/oxide-oximeter:default.log
11922024-01-25T08:15:14.854Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_ac76a12e-fc97-4b5a-b826-45dce44dfa30/root/var/svc/log/oxide-zone-network-setup:default.log (1696 bytes)
11932024-01-25T08:15:14.893Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_ac76a12e-fc97-4b5a-b826-45dce44dfa30/root/var/svc/log/oxide-zone-network-setup:default.log" changed size mid upload: 1696 -> 0
11942024-01-25T08:15:16.008Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_ac76a12e-fc97-4b5a-b826-45dce44dfa30/root/var/svc/log/oxide-zone-network-setup:default.log
11952024-01-25T08:15:16.008Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_329660ec-3331-4eaf-a8a6-040ce0708c88/root/var/svc/log/oxide-cockroachdb:default.log (4380 bytes)
11962024-01-25T08:15:16.047Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_329660ec-3331-4eaf-a8a6-040ce0708c88/root/var/svc/log/oxide-cockroachdb:default.log" changed size mid upload: 4380 -> 0
11972024-01-25T08:15:17.193Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_329660ec-3331-4eaf-a8a6-040ce0708c88/root/var/svc/log/oxide-cockroachdb:default.log
11982024-01-25T08:15:17.233Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_329660ec-3331-4eaf-a8a6-040ce0708c88/root/var/svc/log/oxide-zone-network-setup:default.log (1711 bytes)
11992024-01-25T08:15:17.233Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_329660ec-3331-4eaf-a8a6-040ce0708c88/root/var/svc/log/oxide-zone-network-setup:default.log" changed size mid upload: 1711 -> 0
12002024-01-25T08:15:18.346Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_329660ec-3331-4eaf-a8a6-040ce0708c88/root/var/svc/log/oxide-zone-network-setup:default.log
12012024-01-25T08:15:18.346Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ed702e51-e784-4b3b-bdd8-48ebced0c6f5/root/var/svc/log/oxide-crucible-agent:default.log (19720 bytes)
12022024-01-25T08:15:18.385Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ed702e51-e784-4b3b-bdd8-48ebced0c6f5/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 19720 -> 0
12032024-01-25T08:15:19.499Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ed702e51-e784-4b3b-bdd8-48ebced0c6f5/root/var/svc/log/oxide-crucible-agent:default.log
12042024-01-25T08:15:19.499Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ed702e51-e784-4b3b-bdd8-48ebced0c6f5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12052024-01-25T08:15:19.539Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ed702e51-e784-4b3b-bdd8-48ebced0c6f5/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
12062024-01-25T08:15:20.652Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ed702e51-e784-4b3b-bdd8-48ebced0c6f5/root/var/svc/log/oxide-crucible-downstairs:default.log
12072024-01-25T08:15:20.652Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ed702e51-e784-4b3b-bdd8-48ebced0c6f5/root/var/svc/log/oxide-crucible-downstairs:downstairs-4ef31bb9-77ab-408e-807c-238bdeb98506.log (5623 bytes)
12082024-01-25T08:15:20.691Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ed702e51-e784-4b3b-bdd8-48ebced0c6f5/root/var/svc/log/oxide-crucible-downstairs:downstairs-4ef31bb9-77ab-408e-807c-238bdeb98506.log" changed size mid upload: 5623 -> 0
12092024-01-25T08:15:21.805Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ed702e51-e784-4b3b-bdd8-48ebced0c6f5/root/var/svc/log/oxide-crucible-downstairs:downstairs-4ef31bb9-77ab-408e-807c-238bdeb98506.log
12102024-01-25T08:15:21.805Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ed702e51-e784-4b3b-bdd8-48ebced0c6f5/root/var/svc/log/oxide-crucible-downstairs:snapshot-4ef31bb9-77ab-408e-807c-238bdeb98506-816840a7-c666-4126-b4f4-54ad7c49ba03.log (5178 bytes)
12112024-01-25T08:15:21.844Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ed702e51-e784-4b3b-bdd8-48ebced0c6f5/root/var/svc/log/oxide-crucible-downstairs:snapshot-4ef31bb9-77ab-408e-807c-238bdeb98506-816840a7-c666-4126-b4f4-54ad7c49ba03.log" changed size mid upload: 5178 -> 0
12122024-01-25T08:15:22.957Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ed702e51-e784-4b3b-bdd8-48ebced0c6f5/root/var/svc/log/oxide-crucible-downstairs:snapshot-4ef31bb9-77ab-408e-807c-238bdeb98506-816840a7-c666-4126-b4f4-54ad7c49ba03.log
12132024-01-25T08:15:22.957Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_46483470-f83e-44d8-9514-07bf154a4a27/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
12142024-01-25T08:15:22.997Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_46483470-f83e-44d8-9514-07bf154a4a27/root/var/svc/log/oxide-cockroachdb:default.log" changed size mid upload: 4407 -> 0
12152024-01-25T08:15:24.116Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_46483470-f83e-44d8-9514-07bf154a4a27/root/var/svc/log/oxide-cockroachdb:default.log
12162024-01-25T08:15:24.116Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_46483470-f83e-44d8-9514-07bf154a4a27/root/var/svc/log/oxide-zone-network-setup:default.log (1711 bytes)
12172024-01-25T08:15:24.160Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_46483470-f83e-44d8-9514-07bf154a4a27/root/var/svc/log/oxide-zone-network-setup:default.log" changed size mid upload: 1711 -> 0
12182024-01-25T08:15:25.277Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_46483470-f83e-44d8-9514-07bf154a4a27/root/var/svc/log/oxide-zone-network-setup:default.log
12192024-01-25T08:15:25.316Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_76ed6a84-f386-4c6b-b90e-c09a1d36ccc7/root/var/svc/log/oxide-crucible-agent:default.log (14154 bytes)
12202024-01-25T08:15:25.316Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_76ed6a84-f386-4c6b-b90e-c09a1d36ccc7/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 14154 -> 0
12212024-01-25T08:15:26.431Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_76ed6a84-f386-4c6b-b90e-c09a1d36ccc7/root/var/svc/log/oxide-crucible-agent:default.log
12222024-01-25T08:15:26.504Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_76ed6a84-f386-4c6b-b90e-c09a1d36ccc7/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12232024-01-25T08:15:26.505Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_76ed6a84-f386-4c6b-b90e-c09a1d36ccc7/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
12242024-01-25T08:15:27.589Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_76ed6a84-f386-4c6b-b90e-c09a1d36ccc7/root/var/svc/log/oxide-crucible-downstairs:default.log
12252024-01-25T08:15:27.589Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_76ed6a84-f386-4c6b-b90e-c09a1d36ccc7/root/var/svc/log/oxide-crucible-downstairs:downstairs-ceb5c0e8-a0d1-4cc1-a0fc-43275d98bc92.log (3049 bytes)
12262024-01-25T08:15:27.629Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_76ed6a84-f386-4c6b-b90e-c09a1d36ccc7/root/var/svc/log/oxide-crucible-downstairs:downstairs-ceb5c0e8-a0d1-4cc1-a0fc-43275d98bc92.log" changed size mid upload: 3049 -> 0
12272024-01-25T08:15:28.743Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_76ed6a84-f386-4c6b-b90e-c09a1d36ccc7/root/var/svc/log/oxide-crucible-downstairs:downstairs-ceb5c0e8-a0d1-4cc1-a0fc-43275d98bc92.log
12282024-01-25T08:15:28.745Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_dffbbc27-ff43-4d5b-8e22-eb78d03bac12/root/var/svc/log/oxide-nexus:default.log (949173 bytes)
12292024-01-25T08:15:28.821Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_dffbbc27-ff43-4d5b-8e22-eb78d03bac12/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 949173 -> 6885
12302024-01-25T08:15:29.935Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_dffbbc27-ff43-4d5b-8e22-eb78d03bac12/root/var/svc/log/oxide-nexus:default.log
12312024-01-25T08:15:29.935Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-agent:default.log (34156 bytes)
12322024-01-25T08:15:29.978Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 34156 -> 0
12332024-01-25T08:15:31.098Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-agent:default.log
12342024-01-25T08:15:31.098Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12352024-01-25T08:15:31.138Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
12362024-01-25T08:15:32.251Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-downstairs:default.log
12372024-01-25T08:15:32.253Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-downstairs:downstairs-1feb9491-b441-414a-a06a-c746d2a0777c.log (5622 bytes)
12382024-01-25T08:15:32.293Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-downstairs:downstairs-1feb9491-b441-414a-a06a-c746d2a0777c.log" changed size mid upload: 5622 -> 0
12392024-01-25T08:15:33.412Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-downstairs:downstairs-1feb9491-b441-414a-a06a-c746d2a0777c.log
12402024-01-25T08:15:33.412Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-downstairs:downstairs-4278e2ac-2cf2-4b2c-9e05-6c3df7d90bb0.log (3051 bytes)
12412024-01-25T08:15:33.454Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-downstairs:downstairs-4278e2ac-2cf2-4b2c-9e05-6c3df7d90bb0.log" changed size mid upload: 3051 -> 0
12422024-01-25T08:15:34.565Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-downstairs:downstairs-4278e2ac-2cf2-4b2c-9e05-6c3df7d90bb0.log
12432024-01-25T08:15:34.605Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-downstairs:downstairs-6a9a1279-0780-4403-a976-446114a3be3c.log (2943 bytes)
12442024-01-25T08:15:34.605Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-downstairs:downstairs-6a9a1279-0780-4403-a976-446114a3be3c.log" changed size mid upload: 2943 -> 0
12452024-01-25T08:15:35.719Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-downstairs:downstairs-6a9a1279-0780-4403-a976-446114a3be3c.log
12462024-01-25T08:15:35.758Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-downstairs:snapshot-1feb9491-b441-414a-a06a-c746d2a0777c-816840a7-c666-4126-b4f4-54ad7c49ba03.log (5179 bytes)
12472024-01-25T08:15:35.759Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-downstairs:snapshot-1feb9491-b441-414a-a06a-c746d2a0777c-816840a7-c666-4126-b4f4-54ad7c49ba03.log" changed size mid upload: 5179 -> 0
12482024-01-25T08:15:36.893Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dd7141b-24ca-4063-8041-aa7da0922f09/root/var/svc/log/oxide-crucible-downstairs:snapshot-1feb9491-b441-414a-a06a-c746d2a0777c-816840a7-c666-4126-b4f4-54ad7c49ba03.log
12492024-01-25T08:15:36.894Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8a1e0053-7907-471b-8224-553d2a15b041/root/var/svc/log/oxide-crucible-agent:default.log (9796 bytes)
12502024-01-25T08:15:36.935Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8a1e0053-7907-471b-8224-553d2a15b041/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 9796 -> 0
12512024-01-25T08:15:38.051Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8a1e0053-7907-471b-8224-553d2a15b041/root/var/svc/log/oxide-crucible-agent:default.log
12522024-01-25T08:15:38.051Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8a1e0053-7907-471b-8224-553d2a15b041/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12532024-01-25T08:15:38.091Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8a1e0053-7907-471b-8224-553d2a15b041/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
12542024-01-25T08:15:39.205Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8a1e0053-7907-471b-8224-553d2a15b041/root/var/svc/log/oxide-crucible-downstairs:default.log
12552024-01-25T08:15:39.205Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8a1e0053-7907-471b-8224-553d2a15b041/root/var/svc/log/oxide-crucible-downstairs:downstairs-5bd077de-e59d-43a6-8326-54139002be1c.log (2945 bytes)
12562024-01-25T08:15:39.244Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8a1e0053-7907-471b-8224-553d2a15b041/root/var/svc/log/oxide-crucible-downstairs:downstairs-5bd077de-e59d-43a6-8326-54139002be1c.log" changed size mid upload: 2945 -> 0
12572024-01-25T08:15:40.361Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8a1e0053-7907-471b-8224-553d2a15b041/root/var/svc/log/oxide-crucible-downstairs:downstairs-5bd077de-e59d-43a6-8326-54139002be1c.log
12582024-01-25T08:15:40.361Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_8c38251c-f429-4c75-ba91-3ff6a42ac576/root/var/svc/log/oxide-crucible-pantry:default.log (2337 bytes)
12592024-01-25T08:15:40.401Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_8c38251c-f429-4c75-ba91-3ff6a42ac576/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 2337 -> 0
12602024-01-25T08:15:41.518Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_8c38251c-f429-4c75-ba91-3ff6a42ac576/root/var/svc/log/oxide-crucible-pantry:default.log
12612024-01-25T08:15:41.518Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_49c4cfa4-c1b1-45ca-8dad-8f333a87a66e/root/var/svc/log/oxide-nexus:default.log (2197206 bytes)
12622024-01-25T08:15:41.743Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_49c4cfa4-c1b1-45ca-8dad-8f333a87a66e/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 2197206 -> 67476
12632024-01-25T08:15:42.859Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_49c4cfa4-c1b1-45ca-8dad-8f333a87a66e/root/var/svc/log/oxide-nexus:default.log
12642024-01-25T08:15:42.859Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba/root/var/svc/log/oxide-ntp:default.log (1252 bytes)
12652024-01-25T08:15:42.898Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba/root/var/svc/log/oxide-ntp:default.log" changed size mid upload: 1252 -> 0
12662024-01-25T08:15:44.015Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba/root/var/svc/log/oxide-ntp:default.log
12672024-01-25T08:15:44.055Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dd8b1f21-45c4-429c-b346-2f3e54a8893d/root/var/svc/log/oxide-crucible-agent:default.log (3433 bytes)
12682024-01-25T08:15:44.055Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dd8b1f21-45c4-429c-b346-2f3e54a8893d/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 3433 -> 0
12692024-01-25T08:15:45.169Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dd8b1f21-45c4-429c-b346-2f3e54a8893d/root/var/svc/log/oxide-crucible-agent:default.log
12702024-01-25T08:15:45.169Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dd8b1f21-45c4-429c-b346-2f3e54a8893d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12712024-01-25T08:15:45.242Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dd8b1f21-45c4-429c-b346-2f3e54a8893d/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
12722024-01-25T08:15:46.324Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dd8b1f21-45c4-429c-b346-2f3e54a8893d/root/var/svc/log/oxide-crucible-downstairs:default.log
12732024-01-25T08:15:46.324Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_440b7589-6b62-419d-9282-a6ce8f28c191/root/var/svc/log/oxide-crucible-agent:default.log (25715 bytes)
12742024-01-25T08:15:46.364Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_440b7589-6b62-419d-9282-a6ce8f28c191/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 25715 -> 0
12752024-01-25T08:15:47.488Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_440b7589-6b62-419d-9282-a6ce8f28c191/root/var/svc/log/oxide-crucible-agent:default.log
12762024-01-25T08:15:47.488Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_440b7589-6b62-419d-9282-a6ce8f28c191/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12772024-01-25T08:15:47.554Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_440b7589-6b62-419d-9282-a6ce8f28c191/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
12782024-01-25T08:15:48.642Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_440b7589-6b62-419d-9282-a6ce8f28c191/root/var/svc/log/oxide-crucible-downstairs:default.log
12792024-01-25T08:15:48.642Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_440b7589-6b62-419d-9282-a6ce8f28c191/root/var/svc/log/oxide-crucible-downstairs:downstairs-71e96550-1030-40d1-a80d-a0df59d85df0.log (5536 bytes)
12802024-01-25T08:15:48.682Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_440b7589-6b62-419d-9282-a6ce8f28c191/root/var/svc/log/oxide-crucible-downstairs:downstairs-71e96550-1030-40d1-a80d-a0df59d85df0.log" changed size mid upload: 5536 -> 0
12812024-01-25T08:15:49.795Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_440b7589-6b62-419d-9282-a6ce8f28c191/root/var/svc/log/oxide-crucible-downstairs:downstairs-71e96550-1030-40d1-a80d-a0df59d85df0.log
12822024-01-25T08:15:49.797Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_440b7589-6b62-419d-9282-a6ce8f28c191/root/var/svc/log/oxide-crucible-downstairs:downstairs-c1519a13-ca02-4bfd-a716-18f6dbcca70a.log (2945 bytes)
12832024-01-25T08:15:49.836Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_440b7589-6b62-419d-9282-a6ce8f28c191/root/var/svc/log/oxide-crucible-downstairs:downstairs-c1519a13-ca02-4bfd-a716-18f6dbcca70a.log" changed size mid upload: 2945 -> 0
12842024-01-25T08:15:50.951Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_440b7589-6b62-419d-9282-a6ce8f28c191/root/var/svc/log/oxide-crucible-downstairs:downstairs-c1519a13-ca02-4bfd-a716-18f6dbcca70a.log
12852024-01-25T08:15:50.952Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_86cc8388-c7f7-4ed8-bac7-8bf4a50f9f43/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
12862024-01-25T08:15:50.992Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_86cc8388-c7f7-4ed8-bac7-8bf4a50f9f43/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 2333 -> 0
12872024-01-25T08:15:52.110Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_86cc8388-c7f7-4ed8-bac7-8bf4a50f9f43/root/var/svc/log/oxide-crucible-pantry:default.log
12882024-01-25T08:15:52.110Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba/root/var/log/chrony/measurements.log (25208 bytes)
12892024-01-25T08:15:52.261Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba/root/var/log/chrony/measurements.log" changed size mid upload: 25208 -> 27811
12902024-01-25T08:15:53.376Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba/root/var/log/chrony/measurements.log
12912024-01-25T08:15:53.377Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba/root/var/log/chrony/statistics.log (15444 bytes)
12922024-01-25T08:15:53.564Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba/root/var/log/chrony/statistics.log" changed size mid upload: 15444 -> 17784
12932024-01-25T08:15:54.679Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba/root/var/log/chrony/statistics.log
12942024-01-25T08:15:54.679Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba/root/var/log/chrony/tracking.log (2514 bytes)
12952024-01-25T08:15:54.759Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba/root/var/log/chrony/tracking.log" changed size mid upload: 2514 -> 2778
12962024-01-25T08:15:55.876Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_9a912a55-b86e-42cb-abaa-e5ba339f4dba/root/var/log/chrony/tracking.log