01HMY97NJRNMRM0NK7G5KHV887: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.28",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HMY98FZDVJ27J9DYC6QG6YJ5

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12024-01-24T19:15:58.185Zjob dependencies complete; ready to run (waiting for 1 h 29 m 53 s)
22024-01-24T19:19:14.116Zjob assigned to worker 01HMYED9214MZXCX7MDQTDPNTV (queued for 3 m 15 s)
32024-01-24T19:19:14.412Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42024-01-24T19:19:14.480Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52024-01-24T19:19:14.533Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62024-01-24T19:19:14.588Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72024-01-24T19:19:14.637Zhttp://172.20.0.5:9969/postboot/buskin... ok
82024-01-24T19:19:14.689ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92024-01-24T19:19:14.737Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102024-01-24T19:19:14.790ZOxide Helios Version helios-2.0.22418 64-bit
112024-01-24T19:19:14.838ZConfiguring devices.
122024-01-24T19:19:14.890Z##################################################
132024-01-24T19:19:14.939Z#### ############# ##
142024-01-24T19:19:14.992Z### ### ############ ##
152024-01-24T19:19:15.038Z## ### # ## ### ## Oxide Computer Company ##
162024-01-24T19:19:15.091Z## ## ## ### # ### ##
172024-01-24T19:19:15.139Z## # ### #### #### This Station Under ##
182024-01-24T19:19:15.191Z### ### #### # ### Computer Control ##
192024-01-24T19:19:15.263Z#### #### ### ## ##
202024-01-24T19:19:15.321Z##################################################
212024-01-24T19:19:15.372ZHostname: buskin
222024-01-24T19:19:15.429ZLoading smf(7) service descriptions:
232024-01-24T19:19:16.651Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-03ff62cdf334f689.gz
242024-01-24T19:19:19.398Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-03ff62cdf334f689.gz
252024-01-24T19:19:19.398Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262024-01-24T19:19:53.079Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272024-01-24T19:19:53.079Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d8fb2167cd3fcce8.gz
282024-01-24T19:20:26.978Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d8fb2167cd3fcce8.gz
292024-01-24T19:20:26.978Zdownloading input: /input/ci-tools/work/caboose-util.gz
302024-01-24T19:20:28.362Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312024-01-24T19:20:28.362Zdownloading input: /input/ci-tools/work/tufaceous.gz
322024-01-24T19:20:33.306Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332024-01-24T19:20:33.346Zdownloading input: /input/package/work/version.txt
342024-01-24T19:20:33.387Zdownloaded input: /input/package/work/version.txt
352024-01-24T19:20:33.388Zdownloading input: /input/package/work/package.tar.gz
362024-01-24T19:20:58.035Zdownloaded input: /input/package/work/package.tar.gz
372024-01-24T19:20:58.075Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382024-01-24T19:21:33.173Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392024-01-24T19:21:33.216Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402024-01-24T19:21:45.496Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412024-01-24T19:21:45.536Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422024-01-24T19:22:43.685Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432024-01-24T19:22:43.685Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442024-01-24T19:23:43.757Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452024-01-24T19:23:43.757Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462024-01-24T19:24:22.029Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472024-01-24T19:24:22.029Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482024-01-24T19:24:26.739Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492024-01-24T19:24:26.798Zdownloading input: /input/package/work/zones/crucible.tar.gz
502024-01-24T19:24:35.610Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512024-01-24T19:24:35.610Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522024-01-24T19:24:42.724Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532024-01-24T19:24:42.766Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542024-01-24T19:24:50.015Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552024-01-24T19:24:50.015Zdownloading input: /input/package/work/zones/ntp.tar.gz
562024-01-24T19:24:50.123Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572024-01-24T19:24:50.163Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582024-01-24T19:24:55.320Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592024-01-24T19:24:55.320Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602024-01-24T19:25:00.556Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612024-01-24T19:25:00.556Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622024-01-24T19:25:24.253Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632024-01-24T19:25:24.293Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642024-01-24T19:25:47.288Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652024-01-24T19:25:47.328Zdownloading input: /input/package/work/zones/overlay.tar.gz
662024-01-24T19:25:47.386Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672024-01-24T19:25:47.425Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682024-01-24T19:25:59.124Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692024-01-24T19:25:59.124Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702024-01-24T19:26:18.539Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712024-01-24T19:26:18.539Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722024-01-24T19:27:41.657Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732024-01-24T19:27:41.657Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742024-01-24T19:28:46.976Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752024-01-24T19:28:46.976Zstarting task 0: "setup"
762024-01-24T19:28:47.025Z++ uname -s
772024-01-24T19:28:47.025Z+ kern=SunOS
782024-01-24T19:28:47.025Z+ case "$kern" in
792024-01-24T19:28:47.025Z+ groupadd -g 12345 build
802024-01-24T19:28:47.025Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812024-01-24T19:28:48.991Z+ zfs create -o mountpoint=/work rpool/work
822024-01-24T19:28:49.036Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832024-01-24T19:28:49.036Z+ home_fs=zfs
842024-01-24T19:28:49.036Z+ [[ zfs == autofs ]]
852024-01-24T19:28:49.036Z+ mkdir -p /home/build
862024-01-24T19:28:49.036Z+ chown build:build /home/build /work
872024-01-24T19:28:51.018Z+ chmod 0700 /home/build /work
882024-01-24T19:28:51.061Zprocess exited: duration 4049 ms, exit code 0
 
892024-01-24T19:28:51.149Zstarting task 1: "authentication"
902024-01-24T19:28:51.212Zprocess exited: duration 62 ms, exit code 0
 
912024-01-24T19:28:51.299Zstarting task 2: "build"
922024-01-24T19:28:51.343Z+ trap _exit_trap EXIT
932024-01-24T19:28:51.343Z+ set +x
942024-01-24T19:29:27.856Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
952024-01-24T19:29:28.099Zsecpolicy_fs_linkdir: 0x55 = 0x55
962024-01-24T19:29:28.099Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
972024-01-24T19:29:28.099Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
982024-01-24T19:29:28.099Zsecpolicy_fs_linkdir+3: 0 = 0xe5
992024-01-24T19:29:28.099Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1002024-01-24T19:29:28.099Zsecpolicy_fs_linkdir+5: 0 = 0x1
1012024-01-24T19:29:28.099Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1022024-01-24T19:29:28.099Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1032024-01-24T19:29:28.099Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1042024-01-24T19:29:28.099Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1052024-01-24T19:29:28.139Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1062024-01-24T19:29:28.139Z+ [[ -d /opt/oxide ]]
1072024-01-24T19:29:28.139Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1082024-01-24T19:29:28.140Z+ pfexec mkdir /opt/oxide
1092024-01-24T19:29:28.140Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1102024-01-24T19:29:28.140Z+ [[ -d /opt/oxide-underneath ]]
1112024-01-24T19:29:28.140Z+ cd /opt/oxide-underneath
1122024-01-24T19:29:28.141Z+ pfexec tar ceEp@/f - .
1132024-01-24T19:29:28.141Z+ cd /opt/oxide
1142024-01-24T19:29:28.141Z+ pfexec tar xveEp@/f -
1152024-01-24T19:29:28.182Zx ., 0 bytes, 0 tape blocks
1162024-01-24T19:29:28.182Zx ./opte, 0 bytes, 0 tape blocks
1172024-01-24T19:29:28.182Zx ./opte/bin, 0 bytes, 0 tape blocks
1182024-01-24T19:29:28.183Zx ./opte/bin/opteadm, 51799224 bytes, 101171 tape blocks
1192024-01-24T19:29:28.183Z+ pfexec rm -rf /opt/oxide-underneath
1202024-01-24T19:29:28.186Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1212024-01-24T19:29:28.229Z+ pfexec mkdir /opt/oxide/work
1222024-01-24T19:29:28.229Z+ pfexec chown build:build /opt/oxide/work
1232024-01-24T19:29:28.229Z+ cd /opt/oxide/work
1242024-01-24T19:29:28.229Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1252024-01-24T19:29:29.803ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1262024-01-24T19:29:29.803Zx out/mg-ddm-gz.tar, 62812160 bytes, 122680 tape blocks
1272024-01-24T19:29:29.803Zx out/omicron-sled-agent.tar, 120909824 bytes, 236152 tape blocks
1282024-01-24T19:29:29.803Zx out/target/test, 75 bytes, 1 tape blocks
1292024-01-24T19:29:29.803Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1302024-01-24T19:29:29.803Zx package-manifest.toml, 23296 bytes, 46 tape blocks
1312024-01-24T19:29:29.803Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1322024-01-24T19:29:29.803Zx target/release/omicron-package, 44520432 bytes, 86954 tape blocks
1332024-01-24T19:29:29.803Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1342024-01-24T19:29:29.803Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1352024-01-24T19:29:29.803Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1362024-01-24T19:29:29.803Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1372024-01-24T19:29:29.804Zx tools/scrimlet/softnpu-init.sh, 2149 bytes, 5 tape blocks
1382024-01-24T19:29:29.804Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1392024-01-24T19:29:29.804Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1402024-01-24T19:29:29.843Z
1412024-01-24T19:29:29.843Zreal 1.602582405
1422024-01-24T19:29:29.843Zuser 1.442330598
1432024-01-24T19:29:29.843Zsys 0.280547759
1442024-01-24T19:29:29.843Ztrap 0.000064702
1452024-01-24T19:29:29.843Ztflt 0.000000000
1462024-01-24T19:29:29.843Zdflt 0.000000000
1472024-01-24T19:29:29.843Zkflt 0.000000000
1482024-01-24T19:29:29.843Zlock 0.000000000
1492024-01-24T19:29:29.843Zslp 1.444320857
1502024-01-24T19:29:29.843Zlat 0.036040372
1512024-01-24T19:29:29.843Zstop 0.000117061
1522024-01-24T19:29:29.851Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1532024-01-24T19:29:30.966Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1542024-01-24T19:29:31.005Z+ mkdir tests
1552024-01-24T19:29:31.005Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1562024-01-24T19:29:31.005Z+ ptime -m gunzip
1572024-01-24T19:29:31.005Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-03ff62cdf334f689
1582024-01-24T19:29:31.122Z
1592024-01-24T19:29:31.163Zreal 0.138653269
1602024-01-24T19:29:31.163Zuser 0.126779693
1612024-01-24T19:29:31.163Zsys 0.011626159
1622024-01-24T19:29:31.163Ztrap 0.000012990
1632024-01-24T19:29:31.163Ztflt 0.000000000
1642024-01-24T19:29:31.163Zdflt 0.000000000
1652024-01-24T19:29:31.163Zkflt 0.000000000
1662024-01-24T19:29:31.163Zlock 0.000000000
1672024-01-24T19:29:31.163Zslp 0.000022830
1682024-01-24T19:29:31.163Zlat 0.000019740
1692024-01-24T19:29:31.163Zstop 0.000126922
1702024-01-24T19:29:31.163Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-03ff62cdf334f689
1712024-01-24T19:29:31.163Z+ chmod a+x tests/bootstrap-03ff62cdf334f689
1722024-01-24T19:29:31.163Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1732024-01-24T19:29:31.163Z+ ptime -m gunzip
1742024-01-24T19:29:31.163Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1752024-01-24T19:29:33.544Z
1762024-01-24T19:29:33.544Zreal 2.406065429
1772024-01-24T19:29:33.544Zuser 2.258700152
1782024-01-24T19:29:33.585Zsys 0.147004709
1792024-01-24T19:29:33.585Ztrap 0.000173754
1802024-01-24T19:29:33.585Ztflt 0.000000000
1812024-01-24T19:29:33.585Zdflt 0.000000000
1822024-01-24T19:29:33.585Zkflt 0.000000000
1832024-01-24T19:29:33.586Zlock 0.000000000
1842024-01-24T19:29:33.586Zslp 0.000022470
1852024-01-24T19:29:33.586Zlat 0.000017750
1862024-01-24T19:29:33.586Zstop 0.000094671
1872024-01-24T19:29:33.586Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1882024-01-24T19:29:33.586Z+ chmod a+x tests/bootstrap
1892024-01-24T19:29:33.586Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1902024-01-24T19:29:33.586Z+ ptime -m gunzip
1912024-01-24T19:29:33.586Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d8fb2167cd3fcce8
1922024-01-24T19:29:36.032Z
1932024-01-24T19:29:36.075Zreal 2.475574063
1942024-01-24T19:29:36.075Zuser 2.324138600
1952024-01-24T19:29:36.075Zsys 0.150992063
1962024-01-24T19:29:36.075Ztrap 0.000204614
1972024-01-24T19:29:36.075Ztflt 0.000000000
1982024-01-24T19:29:36.075Zdflt 0.000000000
1992024-01-24T19:29:36.075Zkflt 0.000000000
2002024-01-24T19:29:36.075Zlock 0.000000000
2012024-01-24T19:29:36.075Zslp 0.000023600
2022024-01-24T19:29:36.075Zlat 0.000023500
2032024-01-24T19:29:36.075Zstop 0.000126683
2042024-01-24T19:29:36.075Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d8fb2167cd3fcce8
2052024-01-24T19:29:36.075Z+ chmod a+x tests/end_to_end_tests-d8fb2167cd3fcce8
2062024-01-24T19:29:36.075Z+ bmat address ls
2072024-01-24T19:29:36.075ZNAME CIDR FIRST LAST COUNT
2082024-01-24T19:29:36.075Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2092024-01-24T19:29:36.075Z++ bmat address ls -f extra -Ho first,last,count
2102024-01-24T19:29:36.075Z+ set -- 10.151.1.164 10.151.1.227 64
2112024-01-24T19:29:36.075Z+ EXTRA_IP_START=10.151.1.164
2122024-01-24T19:29:36.077Z+ EXTRA_IP_END=10.151.1.227
2132024-01-24T19:29:36.078Z+ EXTRA_IP_COUNT=64
2142024-01-24T19:29:36.078Z+ (( EXTRA_IP_COUNT >= 32 ))
2152024-01-24T19:29:36.078Z+ EXTRA_IP_BASE=10.151.1
2162024-01-24T19:29:36.078Z+ EXTRA_IP_FOCTET=164
2172024-01-24T19:29:36.078Z+ EXTRA_IP_LOCTET=227
2182024-01-24T19:29:36.078Z+ SERVICE_IP_POOL_START=10.151.1.164
2192024-01-24T19:29:36.078Z+ SERVICE_IP_POOL_END=10.151.1.173
2202024-01-24T19:29:36.078Z+ DNS_IP1=10.151.1.164
2212024-01-24T19:29:36.078Z+ DNS_IP2=10.151.1.165
2222024-01-24T19:29:36.078Z+ UPLINK_IP=10.151.1.174
2232024-01-24T19:29:36.078Z+ IPPOOL_START=10.151.1.175
2242024-01-24T19:29:36.078Z+ IPPOOL_END=10.151.1.227
2252024-01-24T19:29:36.078Z++ ipadm show-addr -po type,addr
2262024-01-24T19:29:36.078Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2272024-01-24T19:29:36.078Z+ GATEWAY_IP=10.151.1.100
2282024-01-24T19:29:36.079Z+ [[ -n 10.151.1.100 ]]
2292024-01-24T19:29:36.079Z+ ping -s 10.151.1.100 56 1
2302024-01-24T19:29:36.079ZPING 10.151.1.100: 56 data bytes
2312024-01-24T19:29:36.119Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
2322024-01-24T19:29:36.119Z
2332024-01-24T19:29:36.119Z----10.151.1.100 PING Statistics----
2342024-01-24T19:29:36.119Z1 packets transmitted, 1 packets received, 0% packet loss
2352024-01-24T19:29:36.120Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2362024-01-24T19:29:36.120Z++ arp -an
2372024-01-24T19:29:36.120Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2382024-01-24T19:29:36.120Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2392024-01-24T19:29:36.121Z+ routeadm -e ipv4-forwarding -u
2402024-01-24T19:29:36.207Z+ PXA_START=10.151.1.164
2412024-01-24T19:29:36.208Z+ PXA_END=10.151.1.227
2422024-01-24T19:29:36.246Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2432024-01-24T19:29:36.247Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2442024-01-24T19:29:36.402Z+ ZPOOL_VDEV_DIR=/scratch
2452024-01-24T19:29:36.402Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2462024-01-24T19:29:36.443Z+++ dirname ./tools/create_virtual_hardware.sh
2472024-01-24T19:29:36.444Z++ cd ./tools
2482024-01-24T19:29:36.444Z++ pwd
2492024-01-24T19:29:36.445Z+ SOURCE_DIR=/opt/oxide/work/tools
2502024-01-24T19:29:36.445Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2512024-01-24T19:29:36.445Z+ SOFTNPU_MODE=zone
2522024-01-24T19:29:36.445Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2532024-01-24T19:29:36.445Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2542024-01-24T19:29:36.445Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2552024-01-24T19:29:36.445Z++ dladm show-phys -p -o LINK
2562024-01-24T19:29:36.445Z++ head -1
2572024-01-24T19:29:36.445Z+ PHYSICAL_LINK=igb0
2582024-01-24T19:29:36.445Z+ echo 'Using igb0 as physical link'
2592024-01-24T19:29:36.445ZUsing igb0 as physical link
2602024-01-24T19:29:36.445Z+ ensure_run_as_root
2612024-01-24T19:29:36.445Z++ id -u
2622024-01-24T19:29:36.445Z+ [[ 0 -ne 0 ]]
2632024-01-24T19:29:36.445Z+ ensure_zpools
2642024-01-24T19:29:36.445Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2652024-01-24T19:29:36.445Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2662024-01-24T19:29:36.445Z+ readarray -t ZPOOLS
2672024-01-24T19:29:36.445Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2682024-01-24T19:29:36.445Z++ sed 's/[ ",]//g'
2692024-01-24T19:29:36.445Z+ for ZPOOL in "${ZPOOLS[@]}"
2702024-01-24T19:29:36.445Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2712024-01-24T19:29:36.446ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2722024-01-24T19:29:36.446Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2732024-01-24T19:29:36.446Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2742024-01-24T19:29:36.446Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2752024-01-24T19:29:36.447Z0+0 records in
2762024-01-24T19:29:36.447Z0+0 records out
2772024-01-24T19:29:36.447Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2782024-01-24T19:29:36.447Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2792024-01-24T19:29:36.447Z+ set +x
2802024-01-24T19:29:36.447ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2812024-01-24T19:29:36.447Z++ zpool list -o name
2822024-01-24T19:29:36.447Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2832024-01-24T19:29:36.488Z+ [[ -z '' ]]
2842024-01-24T19:29:36.488Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2852024-01-24T19:29:36.488Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2862024-01-24T19:29:36.488Z+ set +x
2872024-01-24T19:29:36.488ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2882024-01-24T19:29:36.488Z+ for ZPOOL in "${ZPOOLS[@]}"
2892024-01-24T19:29:36.489Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2902024-01-24T19:29:36.489ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2912024-01-24T19:29:36.489Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2922024-01-24T19:29:36.489Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2932024-01-24T19:29:36.489Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2942024-01-24T19:29:36.489Z0+0 records in
2952024-01-24T19:29:36.489Z0+0 records out
2962024-01-24T19:29:36.489Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2972024-01-24T19:29:36.489Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2982024-01-24T19:29:36.489Z+ set +x
2992024-01-24T19:29:36.489ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3002024-01-24T19:29:36.529Z++ zpool list -o name
3012024-01-24T19:29:36.529Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3022024-01-24T19:29:36.529Z+ [[ -z '' ]]
3032024-01-24T19:29:36.529Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3042024-01-24T19:29:36.529Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3052024-01-24T19:29:36.530Z+ set +x
3062024-01-24T19:29:36.531ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3072024-01-24T19:29:36.531Z+ for ZPOOL in "${ZPOOLS[@]}"
3082024-01-24T19:29:36.531Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3092024-01-24T19:29:36.531ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3102024-01-24T19:29:36.531Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3112024-01-24T19:29:36.531Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3122024-01-24T19:29:36.531Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3132024-01-24T19:29:36.531Z0+0 records in
3142024-01-24T19:29:36.531Z0+0 records out
3152024-01-24T19:29:36.532Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3162024-01-24T19:29:36.532Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3172024-01-24T19:29:36.532Z+ set +x
3182024-01-24T19:29:36.532ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3192024-01-24T19:29:36.581Z++ zpool list -o name
3202024-01-24T19:29:36.581Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3212024-01-24T19:29:36.581Z+ [[ -z '' ]]
3222024-01-24T19:29:36.581Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3232024-01-24T19:29:36.624Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3242024-01-24T19:29:36.624Z+ set +x
3252024-01-24T19:29:36.624ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3262024-01-24T19:29:36.624Z+ for ZPOOL in "${ZPOOLS[@]}"
3272024-01-24T19:29:36.624Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3282024-01-24T19:29:36.624ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3292024-01-24T19:29:36.624Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3302024-01-24T19:29:36.624Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3312024-01-24T19:29:36.624Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3322024-01-24T19:29:36.624Z0+0 records in
3332024-01-24T19:29:36.624Z0+0 records out
3342024-01-24T19:29:36.624Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3352024-01-24T19:29:36.624Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3362024-01-24T19:29:36.624Z+ set +x
3372024-01-24T19:29:36.624ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3382024-01-24T19:29:36.624Z++ zpool list -o name
3392024-01-24T19:29:36.625Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3402024-01-24T19:29:36.625Z+ [[ -z '' ]]
3412024-01-24T19:29:36.625Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3422024-01-24T19:29:36.625Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3432024-01-24T19:29:36.625Z+ set +x
3442024-01-24T19:29:36.625ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3452024-01-24T19:29:36.625Z+ for ZPOOL in "${ZPOOLS[@]}"
3462024-01-24T19:29:36.625Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3472024-01-24T19:29:36.626ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3482024-01-24T19:29:36.626Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3492024-01-24T19:29:36.626Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3502024-01-24T19:29:36.626Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3512024-01-24T19:29:36.626Z0+0 records in
3522024-01-24T19:29:36.626Z0+0 records out
3532024-01-24T19:29:36.626Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3542024-01-24T19:29:36.626Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3552024-01-24T19:29:36.626Z+ set +x
3562024-01-24T19:29:36.626ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3572024-01-24T19:29:36.626Z++ zpool list -o name
3582024-01-24T19:29:36.626Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3592024-01-24T19:29:36.667Z+ [[ -z '' ]]
3602024-01-24T19:29:36.667Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3612024-01-24T19:29:36.667Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3622024-01-24T19:29:36.667Z+ set +x
3632024-01-24T19:29:36.667ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3642024-01-24T19:29:36.667Z+ for ZPOOL in "${ZPOOLS[@]}"
3652024-01-24T19:29:36.667Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3662024-01-24T19:29:36.667ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3672024-01-24T19:29:36.667Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3682024-01-24T19:29:36.667Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3692024-01-24T19:29:36.667Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3702024-01-24T19:29:36.667Z0+0 records in
3712024-01-24T19:29:36.667Z0+0 records out
3722024-01-24T19:29:36.667Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3732024-01-24T19:29:36.667Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3742024-01-24T19:29:36.667Z+ set +x
3752024-01-24T19:29:36.667ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3762024-01-24T19:29:36.667Z++ zpool list -o name
3772024-01-24T19:29:36.668Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3782024-01-24T19:29:36.668Z+ [[ -z '' ]]
3792024-01-24T19:29:36.668Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3802024-01-24T19:29:36.709Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3812024-01-24T19:29:36.709Z+ set +x
3822024-01-24T19:29:36.710ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3832024-01-24T19:29:36.710Z+ for ZPOOL in "${ZPOOLS[@]}"
3842024-01-24T19:29:36.710Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3852024-01-24T19:29:36.710ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3862024-01-24T19:29:36.710Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3872024-01-24T19:29:36.710Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3882024-01-24T19:29:36.710Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3892024-01-24T19:29:36.710Z0+0 records in
3902024-01-24T19:29:36.710Z0+0 records out
3912024-01-24T19:29:36.710Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922024-01-24T19:29:36.710Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3932024-01-24T19:29:36.710Z+ set +x
3942024-01-24T19:29:36.710ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3952024-01-24T19:29:36.710Z++ zpool list -o name
3962024-01-24T19:29:36.710Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3972024-01-24T19:29:36.751Z+ [[ -z '' ]]
3982024-01-24T19:29:36.752Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3992024-01-24T19:29:36.755Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4002024-01-24T19:29:36.755Z+ set +x
4012024-01-24T19:29:36.755ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4022024-01-24T19:29:36.755Z+ for ZPOOL in "${ZPOOLS[@]}"
4032024-01-24T19:29:36.755Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4042024-01-24T19:29:36.755Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052024-01-24T19:29:36.755ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4062024-01-24T19:29:36.755Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4072024-01-24T19:29:36.755Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4082024-01-24T19:29:36.755Z0+0 records in
4092024-01-24T19:29:36.755Z0+0 records out
4102024-01-24T19:29:36.756Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4112024-01-24T19:29:36.756Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4122024-01-24T19:29:36.757Z+ set +x
4132024-01-24T19:29:36.757ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4142024-01-24T19:29:36.757Z++ zpool list -o name
4152024-01-24T19:29:36.757Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4162024-01-24T19:29:36.798Z+ [[ -z '' ]]
4172024-01-24T19:29:36.798Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4182024-01-24T19:29:36.798Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4192024-01-24T19:29:36.798Z+ set +x
4202024-01-24T19:29:36.798ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4212024-01-24T19:29:36.798Z+ for ZPOOL in "${ZPOOLS[@]}"
4222024-01-24T19:29:36.798Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4232024-01-24T19:29:36.798ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4242024-01-24T19:29:36.798Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4252024-01-24T19:29:36.798Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4262024-01-24T19:29:36.798Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4272024-01-24T19:29:36.798Z0+0 records in
4282024-01-24T19:29:36.798Z0+0 records out
4292024-01-24T19:29:36.798Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4302024-01-24T19:29:36.798Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4312024-01-24T19:29:36.798Z+ set +x
4322024-01-24T19:29:36.799ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4332024-01-24T19:29:36.799Z++ zpool list -o name
4342024-01-24T19:29:36.799Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4352024-01-24T19:29:36.799Z+ [[ -z '' ]]
4362024-01-24T19:29:36.799Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4372024-01-24T19:29:36.840Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4382024-01-24T19:29:36.840Z+ set +x
4392024-01-24T19:29:36.840ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4402024-01-24T19:29:36.840Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4412024-01-24T19:29:36.840Z+ readarray -t ZPOOLS
4422024-01-24T19:29:36.840Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4432024-01-24T19:29:36.840Z++ sed 's/[ ",]//g'
4442024-01-24T19:29:36.840Z+ for ZPOOL in "${ZPOOLS[@]}"
4452024-01-24T19:29:36.840Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4462024-01-24T19:29:36.840ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4472024-01-24T19:29:36.840Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4482024-01-24T19:29:36.840Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4492024-01-24T19:29:36.841Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4502024-01-24T19:29:36.841Z0+0 records in
4512024-01-24T19:29:36.841Z0+0 records out
4522024-01-24T19:29:36.841Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4532024-01-24T19:29:36.841Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4542024-01-24T19:29:36.841Z+ set +x
4552024-01-24T19:29:36.841ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4562024-01-24T19:29:36.841Z++ zpool list -o name
4572024-01-24T19:29:36.841Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4582024-01-24T19:29:36.886Z+ [[ -z '' ]]
4592024-01-24T19:29:36.886Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4602024-01-24T19:29:36.886Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4612024-01-24T19:29:36.887Z+ set +x
4622024-01-24T19:29:36.887ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4632024-01-24T19:29:36.887Z+ for ZPOOL in "${ZPOOLS[@]}"
4642024-01-24T19:29:36.887Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4652024-01-24T19:29:36.887ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4662024-01-24T19:29:36.887Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4672024-01-24T19:29:36.887Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4682024-01-24T19:29:36.887Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4692024-01-24T19:29:36.887Z0+0 records in
4702024-01-24T19:29:36.887Z0+0 records out
4712024-01-24T19:29:36.887Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4722024-01-24T19:29:36.887Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4732024-01-24T19:29:36.887Z+ set +x
4742024-01-24T19:29:36.887ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4752024-01-24T19:29:36.887Z++ zpool list -o name
4762024-01-24T19:29:36.887Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4772024-01-24T19:29:36.931Z+ [[ -z '' ]]
4782024-01-24T19:29:36.931Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4792024-01-24T19:29:36.931Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4802024-01-24T19:29:36.931Z+ set +x
4812024-01-24T19:29:36.931ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4822024-01-24T19:29:36.931Z+ [[ zone == \z\o\n\e ]]
4832024-01-24T19:29:36.931Z+ ensure_simulated_links igb0
4842024-01-24T19:29:36.931Z+ local PHYSICAL_LINK=igb0
4852024-01-24T19:29:36.931Z+ INDICES=("0" "1")
4862024-01-24T19:29:36.932Z+ for I in "${INDICES[@]}"
4872024-01-24T19:29:36.932Z++ get_simnet_name_if_exists net0
4882024-01-24T19:29:36.932Z++ dladm show-simnet -p -o LINK net0
4892024-01-24T19:29:36.932Z++ echo ''
4902024-01-24T19:29:36.932Z+ [[ -z '' ]]
4912024-01-24T19:29:36.932Z+ dladm create-simnet -t net0
4922024-01-24T19:29:36.973Z+ dladm create-simnet -t sc0_0
4932024-01-24T19:29:36.973Z+ dladm modify-simnet -t -p net0 sc0_0
4942024-01-24T19:29:36.973Z+ dladm set-linkprop -p mtu=9000 sc0_0
4952024-01-24T19:29:36.973Z+ success 'Simnet net0/sc0_0 exists'
4962024-01-24T19:29:36.973Z+ set +x
4972024-01-24T19:29:36.973ZSimnet net0/sc0_0 exists
4982024-01-24T19:29:36.973Z+ for I in "${INDICES[@]}"
4992024-01-24T19:29:36.973Z++ get_simnet_name_if_exists net1
5002024-01-24T19:29:36.973Z++ dladm show-simnet -p -o LINK net1
5012024-01-24T19:29:36.973Z++ echo ''
5022024-01-24T19:29:36.974Z+ [[ -z '' ]]
5032024-01-24T19:29:36.974Z+ dladm create-simnet -t net1
5042024-01-24T19:29:36.974Z+ dladm create-simnet -t sc1_0
5052024-01-24T19:29:36.974Z+ dladm modify-simnet -t -p net1 sc1_0
5062024-01-24T19:29:36.974Z+ dladm set-linkprop -p mtu=9000 sc1_0
5072024-01-24T19:29:36.974Z+ success 'Simnet net1/sc1_0 exists'
5082024-01-24T19:29:36.974Z+ set +x
5092024-01-24T19:29:36.974ZSimnet net1/sc1_0 exists
5102024-01-24T19:29:36.974Z++ get_vnic_name_if_exists sc0_1
5112024-01-24T19:29:36.974Z++ dladm show-vnic -p -o LINK sc0_1
5122024-01-24T19:29:36.974Z++ echo ''
5132024-01-24T19:29:37.013Z+ [[ -z '' ]]
5142024-01-24T19:29:37.013Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5152024-01-24T19:29:37.013Z+ [[ -v PROMISC_FILT_OFF ]]
5162024-01-24T19:29:37.013Z+ success 'Vnic sc0_1 exists'
5172024-01-24T19:29:37.013Z+ set +x
5182024-01-24T19:29:37.013ZVnic sc0_1 exists
5192024-01-24T19:29:37.013Z+ warn_if_no_proxy_arp
5202024-01-24T19:29:37.013Z+ [[ -v PXA_START ]]
5212024-01-24T19:29:37.013Z+ [[ -v PXA_END ]]
5222024-01-24T19:29:37.013Z+ ensure_softnpu_zone
5232024-01-24T19:29:37.013Z+ zoneadm list
5242024-01-24T19:29:37.014Z+ grep -q sidecar_softnpu
5252024-01-24T19:29:37.014Z+ [[ -f out/npuzone/npuzone ]]
5262024-01-24T19:29:37.014Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit e3ea4b495ba0a71801ded0776ae4bbd31df57e26 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5272024-01-24T19:29:37.014Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5282024-01-24T19:29:45.262Zdownloading softnpu ...done
5292024-01-24T19:29:45.304Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/libsidecar_lite.so.sha256.txt
5302024-01-24T19:29:52.689Zdownloading asic_program.so ...done
5312024-01-24T19:29:52.736Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/scadm.sha256.txt
5322024-01-24T19:30:00.839Zdownloading scadm ...done
5332024-01-24T19:30:00.880Zcreating port tfportrear0_0 for sc0_0
5342024-01-24T19:30:00.880Zcreating port tfportqsfp0_0 for sc0_1
5352024-01-24T19:30:00.880Zsoftnpu zone setup
5362024-01-24T19:30:00.880Zconfigure zone
5372024-01-24T19:30:00.922Zinstall zone
5382024-01-24T19:30:01.918Zboot zone
5392024-01-24T19:30:05.027Z[sidecar_softnpu] svcs milestone/network
5402024-01-24T19:30:05.069ZSTATE STIME FMRI
5412024-01-24T19:30:05.069Zonline 19:30:03 svc:/milestone/network:default
5422024-01-24T19:30:05.069Z[sidecar_softnpu] /softnpu/npu start
5432024-01-24T19:30:05.112Z
5442024-01-24T19:30:05.112Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5452024-01-24T19:30:05.113Z+ GATEWAY_IP=10.151.1.100
5462024-01-24T19:30:05.113Z+ echo 'Using 10.151.1.100 as gateway ip'
5472024-01-24T19:30:05.113ZUsing 10.151.1.100 as gateway ip
5482024-01-24T19:30:05.113Z+ [[ ! -v GATEWAY_MAC ]]
5492024-01-24T19:30:05.113Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5502024-01-24T19:30:05.113Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5512024-01-24T19:30:05.113Z+++ echo 18:c0:4d:0d:9f:b2
5522024-01-24T19:30:05.113Z+ COUNT=1
5532024-01-24T19:30:05.113Z+ [[ 1 -ne 1 ]]
5542024-01-24T19:30:05.113Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5552024-01-24T19:30:05.113ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5562024-01-24T19:30:05.113Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572024-01-24T19:30:05.113Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5582024-01-24T19:30:05.156Z+ PXA_MAC=a8:e1:de:01:70:1d
5592024-01-24T19:30:05.156Z+ [[ -v PXA_START ]]
5602024-01-24T19:30:05.156Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612024-01-24T19:30:05.156Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5622024-01-24T19:30:05.156Z+ z_scadm dump-state
5632024-01-24T19:30:05.156Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5642024-01-24T19:30:05.197Zlocal v6:
5652024-01-24T19:30:05.197Zlocal v4:
5662024-01-24T19:30:05.197Zrouter v6:
5672024-01-24T19:30:05.197Zrouter v4_idx:
5682024-01-24T19:30:05.197Zrouter v4_routes:
5692024-01-24T19:30:05.197Zresolver v4:
5702024-01-24T19:30:05.197Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5712024-01-24T19:30:05.198Zresolver v6:
5722024-01-24T19:30:05.198Znat v4:
5732024-01-24T19:30:05.198Znat_v6:
5742024-01-24T19:30:05.198Zport_mac:
5752024-01-24T19:30:05.198Zproxy arp:
5762024-01-24T19:30:05.198Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5772024-01-24T19:30:05.198Z+ success 'softnpu zone exists'
5782024-01-24T19:30:05.198Z+ set +x
5792024-01-24T19:30:05.198Zsoftnpu zone exists
5802024-01-24T19:30:05.198Z
5812024-01-24T19:30:05.198Zreal 28.771744301
5822024-01-24T19:30:05.198Zuser 0.708584090
5832024-01-24T19:30:05.198Zsys 1.572059520
5842024-01-24T19:30:05.198Ztrap 0.000080296
5852024-01-24T19:30:05.198Ztflt 0.000150858
5862024-01-24T19:30:05.198Zdflt 0.001176271
5872024-01-24T19:30:05.198Zkflt 0.000000000
5882024-01-24T19:30:05.198Zlock 22:24.460289948
5892024-01-24T19:30:05.198Zslp 1:25.967550871
5902024-01-24T19:30:05.198Zlat 0.081390786
5912024-01-24T19:30:05.198Zstop 0.374106075
5922024-01-24T19:30:05.199Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5932024-01-24T19:30:05.240Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5942024-01-24T19:30:05.240Z+ SILO_NAME=recovery
5952024-01-24T19:30:05.240Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5962024-01-24T19:30:05.240Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5972024-01-24T19:30:05.240Z+ sed -i~ '
5982024-01-24T19:30:05.241Z /^external_dns_ips/c\
5992024-01-24T19:30:05.241Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6002024-01-24T19:30:05.241Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6012024-01-24T19:30:05.241Z /^first/c\
6022024-01-24T19:30:05.241Zfirst = "10.151.1.164"
6032024-01-24T19:30:05.241Z /^last/c\
6042024-01-24T19:30:05.241Zlast = "10.151.1.173"
6052024-01-24T19:30:05.241Z }
6062024-01-24T19:30:05.241Z /^\[rack_network_config/,/^$/ {
6072024-01-24T19:30:05.241Z /^infra_ip_first/c\
6082024-01-24T19:30:05.242Zinfra_ip_first = "10.151.1.174"
6092024-01-24T19:30:05.242Z /^infra_ip_last/c\
6102024-01-24T19:30:05.242Zinfra_ip_last = "10.151.1.174"
6112024-01-24T19:30:05.242Z }
6122024-01-24T19:30:05.242Z /^\[\[rack_network_config.ports/,/^$/ {
6132024-01-24T19:30:05.242Z /^routes/c\
6142024-01-24T19:30:05.243Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6152024-01-24T19:30:05.243Z /^addresses/c\
6162024-01-24T19:30:05.243Zaddresses = \["10.151.1.174/24"\]
6172024-01-24T19:30:05.243Z }
6182024-01-24T19:30:05.243Z' pkg/config-rss.toml
6192024-01-24T19:30:05.243Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6202024-01-24T19:30:05.243Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6212024-01-24T19:30:05.243Z+++ pkg/config-rss.toml Wed Jan 24 19:30:05 2024
6222024-01-24T19:30:05.243Z@@ -41,7 +41,7 @@
6232024-01-24T19:30:05.243Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6242024-01-24T19:30:05.243Z # must be contained in one of the "internal services" IP Pool ranges listed
6252024-01-24T19:30:05.243Z # below.
6262024-01-24T19:30:05.243Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6272024-01-24T19:30:05.243Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6282024-01-24T19:30:05.243Z
6292024-01-24T19:30:05.243Z # Initial TLS certificates for the external API
6302024-01-24T19:30:05.243Z #
6312024-01-24T19:30:05.243Z@@ -77,8 +77,8 @@
6322024-01-24T19:30:05.243Z #
6332024-01-24T19:30:05.243Z # For more on this and what to put here, see docs/how-to-run.adoc.
6342024-01-24T19:30:05.243Z [[internal_services_ip_pool_ranges]]
6352024-01-24T19:30:05.243Z-first = "192.168.1.20"
6362024-01-24T19:30:05.244Z-last = "192.168.1.29"
6372024-01-24T19:30:05.244Z+first = "10.151.1.164"
6382024-01-24T19:30:05.244Z+last = "10.151.1.173"
6392024-01-24T19:30:05.244Z
6402024-01-24T19:30:05.244Z # TODO - this configuration is subject to change going forward. Ultimately these
6412024-01-24T19:30:05.244Z # parameters should be provided to the control plane via wicket, but we need to
6422024-01-24T19:30:05.244Z@@ -92,8 +92,8 @@
6432024-01-24T19:30:05.244Z # A range of IP addresses used by Boundary Services on the external network. In
6442024-01-24T19:30:05.244Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6452024-01-24T19:30:05.244Z # With softnpu, only one address is used.
6462024-01-24T19:30:05.244Z-infra_ip_first = "192.168.1.30"
6472024-01-24T19:30:05.244Z-infra_ip_last = "192.168.1.30"
6482024-01-24T19:30:05.244Z+infra_ip_first = "10.151.1.174"
6492024-01-24T19:30:05.244Z+infra_ip_last = "10.151.1.174"
6502024-01-24T19:30:05.244Z
6512024-01-24T19:30:05.244Z # Configurations for BGP routers to run on the scrimlets.
6522024-01-24T19:30:05.244Z bgp = []
6532024-01-24T19:30:05.244Z@@ -101,9 +101,9 @@
6542024-01-24T19:30:05.244Z # You can configure multiple uplinks by repeating the following stanza
6552024-01-24T19:30:05.244Z [[rack_network_config.ports]]
6562024-01-24T19:30:05.244Z # Routes associated with this port.
6572024-01-24T19:30:05.244Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6582024-01-24T19:30:05.244Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6592024-01-24T19:30:05.245Z # Addresses associated with this port.
6602024-01-24T19:30:05.245Z-addresses = ["192.168.1.30/24"]
6612024-01-24T19:30:05.245Z+addresses = ["10.151.1.174/24"]
6622024-01-24T19:30:05.245Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6632024-01-24T19:30:05.245Z port = "qsfp0"
6642024-01-24T19:30:05.245Z # The speed of this port.
6652024-01-24T19:30:05.245Z+ true
6662024-01-24T19:30:05.245Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6672024-01-24T19:30:05.245Za pkg/config-rss.toml 7K
6682024-01-24T19:30:05.245Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6692024-01-24T19:30:05.245Z+ TLS_NAME=recovery.sys.oxide.test
6702024-01-24T19:30:05.245Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6712024-01-24T19:30:05.928Z.....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+......+...............+.....+.+..+..........+.....+......+...............+.......+..+...+....+.........+..+....+.........+.....+................+.....+....+..+....+........+.............+...+.........+..+....+......+.......................+...+...+.......+..+.+.....+.+......+.........+.....+.+................................+..........+.....+..................+.......+..+...............+...+...+...............+...+....+..+................+...+.........+.....+..........+....................+.+............+...+.........+...+...............+...+............+...+.........+...............+........+...+.......+........+............+.......+.....+..........+...+........+....+...+.....+.......+..+..........+...........+.........+.+...+.................+....+........+......+......+....+.....................+........+.+.....+................+......+..............+....+.....+......+.+.........+......+..............+..........+........+...+...+.........+.+.........+..+...+..................+...+.........+...+....+..............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722024-01-24T19:30:07.662Z....+...+.....+..........+...+..............+.......+.....+...+.......+...+......+..+.........+...+...+.........+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.........+..+...+......+......+.+.....+.......+..+....+..+...................+......+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+...+..........+..+................+...............+..+.+...........+...+.+......+...+..+.+.....+...+.......+.....+...+..........+..+..................+....+............+.....+...............+...+.+......+......+........+.+....................+....+.........+......+..+....+........+.......+........+..................+...+.+.................................+......+...............+.....+.......+...+..+.........+....+......+......+..+......+.........+...+...+.......+...............+.........+..............+....+...+..+.+..+.......+...........+.......+...+.........+..+.............+......+.....+.......+...+..+.........+..................+......+....+......+......+.....+.+............+........+...+..........+..+..........+..............+..................+.......+.........+......+...........+.......+.....+...+.......+........+..........+..+.+..+............+...+...+.+...+......+......+...+.................+....+..+.......+........+........................+.+..................+......+........+.+....................+......+....+.....+.+........+......+....+......+.....+......+....+...............+.....+....+.....+..........+...+..+.......+...........+...+.+..................+..+.+...+..+...................+...+..+......+....+...........+............+.............+........+.+..............+....+......+..+...+....+.....+.......+.......................+.......+......+..+.......+.........+..+...+......+..........+...+..+................+...+..+....+.........+......+...........+....+...+..+.+..+.............+..+............+.+..+............+.........................+.....+.............+.....+......+.+.....+..........+......+...............+........+..........+............+...+..+.......+..+...+......+............+............+.+.....................+...+...............+...........+......+.+..+...............+.............+...+......+.....+...................+..+...+.......+..+......+.......+...+...........+......+............+.......+........+...............+.+............+........+.......+.......................+................+.........+.....+...................+..+...+.+.....+.+...........+.......+..+......+...+............+...............+....+...........+............+..........+.....+......+......+...............+.+.....+...+.......+...+.....+......+...+.......+..................+.....+...+...+.........+............+......+...+.+...+..+................+...........................+........+.......+...+...+.....................+........+.+..+...+.+......+.....+.........+.+........+...................+......+..+.......+...............+...+...+.....+.+...........+...+....+............+..............+....+..+.+...+...........+.........+.......+..+...+.......+.....................+..+.........+...+.........................+........+.......+..+.............+........+...+.+...+..+...+................+...........................+...............+...+......+...........+...+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6732024-01-24T19:30:07.702Z-----
6742024-01-24T19:30:07.702Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6752024-01-24T19:30:07.702Za pkg/initial-tls-cert.pem 2K
6762024-01-24T19:30:07.702Za pkg/initial-tls-key.pem 4K
6772024-01-24T19:30:07.703Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6782024-01-24T19:30:07.703Z+ rmdir pkg
6792024-01-24T19:30:07.703Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6802024-01-24T19:30:07.703Z+ pfexec mkdir -p /usr/oxide
6812024-01-24T19:30:07.704Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6822024-01-24T19:30:08.496Z+ pfexec chmod +x /usr/oxide/oxide
6832024-01-24T19:30:08.538Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6842024-01-24T19:30:26.783Z+ OMICRON_NO_UNINSTALL=1
6852024-01-24T19:30:26.783Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6862024-01-24T19:30:26.828ZJan 24 19:30:26.853 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6872024-01-24T19:30:26.828ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6882024-01-24T19:30:26.828ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6892024-01-24T19:30:26.828ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6902024-01-24T19:30:26.828ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6912024-01-24T19:30:26.829ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6922024-01-24T19:30:26.829ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6932024-01-24T19:30:26.829ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6942024-01-24T19:30:26.829ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6952024-01-24T19:30:26.831ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6962024-01-24T19:30:26.831ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6972024-01-24T19:30:26.831ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6982024-01-24T19:30:26.831ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6992024-01-24T19:30:26.831ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7002024-01-24T19:30:26.831ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7012024-01-24T19:30:26.831ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7022024-01-24T19:30:26.831ZJan 24 19:30:26.856 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7032024-01-24T19:30:27.049ZJan 24 19:30:27.105 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7042024-01-24T19:30:27.090ZJan 24 19:30:27.141 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7052024-01-24T19:30:27.234ZJan 24 19:30:27.205 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7062024-01-24T19:30:27.278Z
7072024-01-24T19:30:27.320Zreal 0.490655919
7082024-01-24T19:30:27.320Zuser 0.105401905
7092024-01-24T19:30:27.320Zsys 1.977897939
7102024-01-24T19:30:27.320Ztrap 0.000008189
7112024-01-24T19:30:27.320Ztflt 0.000018029
7122024-01-24T19:30:27.320Zdflt 0.000356238
7132024-01-24T19:30:27.320Zkflt 0.000000000
7142024-01-24T19:30:27.320Zlock 44.046572678
7152024-01-24T19:30:27.320Zslp 0.704449774
7162024-01-24T19:30:27.320Zlat 0.003151210
7172024-01-24T19:30:27.320Zstop 0.196505952
7182024-01-24T19:30:27.320Z+ retry=0
7192024-01-24T19:30:27.320Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7202024-01-24T19:30:27.320Z+ [[ 0 -gt 30 ]]
7212024-01-24T19:30:27.320Z+ sleep 1
7222024-01-24T19:30:28.297Z+ retry=1
7232024-01-24T19:30:28.297Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7242024-01-24T19:30:28.338Z+ [[ 1 -gt 30 ]]
7252024-01-24T19:30:28.338Z+ sleep 1
7262024-01-24T19:30:29.306Z+ retry=2
7272024-01-24T19:30:29.306Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7282024-01-24T19:30:29.346Z+ [[ 2 -gt 30 ]]
7292024-01-24T19:30:29.346Z+ sleep 1
7302024-01-24T19:30:30.320Z+ retry=3
7312024-01-24T19:30:30.320Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7322024-01-24T19:30:30.360Z+ [[ 3 -gt 30 ]]
7332024-01-24T19:30:30.360Z+ sleep 1
7342024-01-24T19:30:31.334Z+ retry=4
7352024-01-24T19:30:31.334Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7362024-01-24T19:30:56.507Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7372024-01-24T19:30:56.549Zlocal v6:
7382024-01-24T19:30:56.549Zfde5:fef1:60a9:bd0c::1
7392024-01-24T19:30:56.549Zfe80::ac9d:4ff:fe90:29c6
7402024-01-24T19:30:56.549Zlocal v4:
7412024-01-24T19:30:56.549Z10.151.1.174
7422024-01-24T19:30:56.550Zrouter v6:
7432024-01-24T19:30:56.550Zfd00:1122:3344:101::/64 -> fe80::a08e:79ff:fe3a:bc0a (1)
7442024-01-24T19:30:56.550Zfd00:1122:3344:1::/64 -> fe80::a08e:79ff:fe3a:bc0a (1)
7452024-01-24T19:30:56.550Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::a08e:79ff:fe3a:bc0a (1)
7462024-01-24T19:30:56.551Zfd00:1122:3344:3::/64 -> fe80::a08e:79ff:fe3a:bc0a (1)
7472024-01-24T19:30:56.551Zfd00:1122:3344:2::/64 -> fe80::a08e:79ff:fe3a:bc0a (1)
7482024-01-24T19:30:56.551Zrouter v4_idx:
7492024-01-24T19:30:56.551Z0.0.0.0/0 -> 0
7502024-01-24T19:30:56.551Zrouter v4_routes:
7512024-01-24T19:30:56.551Z0 -> 10.151.1.100 (2)
7522024-01-24T19:30:56.551Zresolver v4:
7532024-01-24T19:30:56.551Z10.151.1.174 -> 66:6c:a7:a9:a2:04
7542024-01-24T19:30:56.551Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7552024-01-24T19:30:56.551Zresolver v6:
7562024-01-24T19:30:56.551Zfe80::a08e:79ff:fe3a:bc0a -> a2:8e:79:3a:bc:0a
7572024-01-24T19:30:56.551Zfe80::ac9d:4ff:fe90:29c6 -> ae:9d:04:90:29:c6
7582024-01-24T19:30:56.551Znat v4:
7592024-01-24T19:30:56.551Z10.151.1.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:ce:6b
7602024-01-24T19:30:56.551Znat_v6:
7612024-01-24T19:30:56.551Zport_mac:
7622024-01-24T19:30:56.551Z1000: a8:40:25:1d:7c:f0
7632024-01-24T19:30:56.551Z1: a8:40:25:1d:7c:f1
7642024-01-24T19:30:56.551Zproxy arp:
7652024-01-24T19:30:56.551Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7662024-01-24T19:30:56.551Z+ retry=0
7672024-01-24T19:30:56.591Z+++ zoneadm list -n
7682024-01-24T19:30:56.591Z+++ grep oxz_ntp
7692024-01-24T19:30:56.591Z++ pfexec svcs -z oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78 -Hostate oxide/ntp
7702024-01-24T19:30:56.591Zsvcs: Could not bind to repository server for zone oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78: repository server unavailable
7712024-01-24T19:30:56.593Z++ true
7722024-01-24T19:30:56.593Z+ [[ '' != online ]]
7732024-01-24T19:30:56.593Z+ [[ 0 -gt 60 ]]
7742024-01-24T19:30:56.593Z+ sleep 1
7752024-01-24T19:30:57.561Z+ retry=1
7762024-01-24T19:30:57.600Z+++ zoneadm list -n
7772024-01-24T19:30:57.600Z+++ grep oxz_ntp
7782024-01-24T19:30:57.600Z++ pfexec svcs -z oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78 -Hostate oxide/ntp
7792024-01-24T19:30:57.600Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7802024-01-24T19:30:57.600Z++ true
7812024-01-24T19:30:57.600Z+ [[ '' != online ]]
7822024-01-24T19:30:57.600Z+ [[ 1 -gt 60 ]]
7832024-01-24T19:30:57.600Z+ sleep 1
7842024-01-24T19:30:58.597Z+ retry=2
7852024-01-24T19:30:58.638Z+++ zoneadm list -n
7862024-01-24T19:30:58.638Z+++ grep oxz_ntp
7872024-01-24T19:30:58.639Z++ pfexec svcs -z oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78 -Hostate oxide/ntp
7882024-01-24T19:30:58.639Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7892024-01-24T19:30:58.639Z++ true
7902024-01-24T19:30:58.639Z+ [[ '' != online ]]
7912024-01-24T19:30:58.639Z+ [[ 2 -gt 60 ]]
7922024-01-24T19:30:58.639Z+ sleep 1
7932024-01-24T19:30:59.626Z+ retry=3
7942024-01-24T19:30:59.665Z+++ zoneadm list -n
7952024-01-24T19:30:59.665Z+++ grep oxz_ntp
7962024-01-24T19:30:59.665Z++ pfexec svcs -z oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78 -Hostate oxide/ntp
7972024-01-24T19:30:59.665Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7982024-01-24T19:30:59.666Z++ true
7992024-01-24T19:30:59.666Z+ [[ '' != online ]]
8002024-01-24T19:30:59.666Z+ [[ 3 -gt 60 ]]
8012024-01-24T19:30:59.667Z+ sleep 1
8022024-01-24T19:31:00.657Z+ retry=4
8032024-01-24T19:31:00.712Z+++ zoneadm list -n
8042024-01-24T19:31:00.712Z+++ grep oxz_ntp
8052024-01-24T19:31:00.712Z++ pfexec svcs -z oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78 -Hostate oxide/ntp
8062024-01-24T19:31:00.712Z+ [[ disabled != online ]]
8072024-01-24T19:31:00.712Z+ [[ 4 -gt 60 ]]
8082024-01-24T19:31:00.712Z+ sleep 1
8092024-01-24T19:31:01.680Z+ retry=5
8102024-01-24T19:31:01.721Z+++ zoneadm list -n
8112024-01-24T19:31:01.722Z+++ grep oxz_ntp
8122024-01-24T19:31:01.722Z++ pfexec svcs -z oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78 -Hostate oxide/ntp
8132024-01-24T19:31:01.722Z+ [[ disabled != online ]]
8142024-01-24T19:31:01.722Z+ [[ 5 -gt 60 ]]
8152024-01-24T19:31:01.722Z+ sleep 1
8162024-01-24T19:31:02.702Z+ retry=6
8172024-01-24T19:31:02.741Z+++ zoneadm list -n
8182024-01-24T19:31:02.741Z+++ grep oxz_ntp
8192024-01-24T19:31:02.741Z++ pfexec svcs -z oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78 -Hostate oxide/ntp
8202024-01-24T19:31:02.741Z+ [[ disabled != online ]]
8212024-01-24T19:31:02.741Z+ [[ 6 -gt 60 ]]
8222024-01-24T19:31:02.742Z+ sleep 1
8232024-01-24T19:31:03.724Z+ retry=7
8242024-01-24T19:31:03.766Z+++ zoneadm list -n
8252024-01-24T19:31:03.766Z+++ grep oxz_ntp
8262024-01-24T19:31:03.766Z++ pfexec svcs -z oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78 -Hostate oxide/ntp
8272024-01-24T19:31:03.766Z+ [[ disabled != online ]]
8282024-01-24T19:31:03.766Z+ [[ 7 -gt 60 ]]
8292024-01-24T19:31:03.766Z+ sleep 1
8302024-01-24T19:31:04.750Z+ retry=8
8312024-01-24T19:31:04.791Z+++ zoneadm list -n
8322024-01-24T19:31:04.792Z+++ grep oxz_ntp
8332024-01-24T19:31:04.792Z++ pfexec svcs -z oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78 -Hostate oxide/ntp
8342024-01-24T19:31:04.792Z+ [[ disabled != online ]]
8352024-01-24T19:31:04.792Z+ [[ 8 -gt 60 ]]
8362024-01-24T19:31:04.792Z+ sleep 1
8372024-01-24T19:31:05.767Z+ retry=9
8382024-01-24T19:31:05.808Z+++ zoneadm list -n
8392024-01-24T19:31:05.808Z+++ grep oxz_ntp
8402024-01-24T19:31:05.808Z++ pfexec svcs -z oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78 -Hostate oxide/ntp
8412024-01-24T19:31:05.808Z+ [[ disabled != online ]]
8422024-01-24T19:31:05.808Z+ [[ 9 -gt 60 ]]
8432024-01-24T19:31:05.808Z+ sleep 1
8442024-01-24T19:31:06.795Z+ retry=10
8452024-01-24T19:31:06.837Z+++ zoneadm list -n
8462024-01-24T19:31:06.837Z+++ grep oxz_ntp
8472024-01-24T19:31:06.837Z++ pfexec svcs -z oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78 -Hostate oxide/ntp
8482024-01-24T19:31:06.837Z+ [[ disabled != online ]]
8492024-01-24T19:31:06.837Z+ [[ 10 -gt 60 ]]
8502024-01-24T19:31:06.838Z+ sleep 1
 
8512024-01-24T19:31:07.798Zbuskin console login: Jan 24 19:31:07 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8522024-01-24T19:31:07.813Z+ retry=11
8532024-01-24T19:31:07.853Z+++ zoneadm list -n
8542024-01-24T19:31:07.853Z+++ grep oxz_ntp
8552024-01-24T19:31:07.853Z++ pfexec svcs -z oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78 -Hostate oxide/ntp
8562024-01-24T19:31:07.853Z+ [[ disabled != online ]]
8572024-01-24T19:31:07.853Z+ [[ 11 -gt 60 ]]
8582024-01-24T19:31:07.853Z+ sleep 1
8592024-01-24T19:31:08.836Z+ retry=12
8602024-01-24T19:31:08.877Z+++ zoneadm list -n
8612024-01-24T19:31:08.877Z+++ grep oxz_ntp
8622024-01-24T19:31:08.877Z++ pfexec svcs -z oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78 -Hostate oxide/ntp
8632024-01-24T19:31:08.877Z+ [[ online != online ]]
8642024-01-24T19:31:08.877Z+ echo 'Waited for chrony: 12s'
8652024-01-24T19:31:08.877ZWaited for chrony: 12s
8662024-01-24T19:31:08.877Z+ export RUST_BACKTRACE=1
8672024-01-24T19:31:08.877Z+ RUST_BACKTRACE=1
8682024-01-24T19:31:08.877Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8692024-01-24T19:31:08.878Z++ ./tests/bootstrap
8702024-01-24T19:31:08.878ZUsing external DNS server at 10.151.1.164:53
8712024-01-24T19:31:08.918Z2024-01-24 19:31:08.941571803 UTC: attempting to log into API
8722024-01-24T19:31:23.921Z2024-01-24 19:31:23.951854513 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8732024-01-24T19:31:24.922Z2024-01-24 19:31:24.949946089 UTC: attempting to log into API
8742024-01-24T19:31:39.956Z2024-01-24 19:31:39.942791891 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8752024-01-24T19:31:40.960Z2024-01-24 19:31:40.942016861 UTC: attempting to log into API
8762024-01-24T19:31:55.992Z2024-01-24 19:31:55.970156439 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8772024-01-24T19:31:56.994Z2024-01-24 19:31:56.971096117 UTC: attempting to log into API
8782024-01-24T19:32:12.026Z2024-01-24 19:32:12.005556918 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8792024-01-24T19:32:13.029Z2024-01-24 19:32:13.007498637 UTC: attempting to log into API
8802024-01-24T19:32:28.063Z2024-01-24 19:32:28.040880632 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8812024-01-24T19:32:29.063Z2024-01-24 19:32:29.041945023 UTC: attempting to log into API
8822024-01-24T19:32:44.097Z2024-01-24 19:32:44.075458212 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8832024-01-24T19:32:45.140Z2024-01-24 19:32:45.078908312 UTC: attempting to log into API
8842024-01-24T19:33:00.211Z2024-01-24 19:33:00.190205372 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8852024-01-24T19:33:01.213Z2024-01-24 19:33:01.191137519 UTC: attempting to log into API
8862024-01-24T19:33:16.247Z2024-01-24 19:33:16.225203357 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8872024-01-24T19:33:17.247Z2024-01-24 19:33:17.225921040 UTC: attempting to log into API
8882024-01-24T19:33:32.416Z2024-01-24 19:33:32.392328902 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8892024-01-24T19:33:33.415Z2024-01-24 19:33:33.393472504 UTC: attempting to log into API
 
8902024-01-24T19:33:44.559ZJan 24 19:33:43 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8912024-01-24T19:33:48.448Z2024-01-24 19:33:48.426787762 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8922024-01-24T19:33:49.449Z2024-01-24 19:33:49.427957264 UTC: attempting to log into API
 
8932024-01-24T19:33:53.796ZJan 24 19:33:53 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
8942024-01-24T19:34:00.853ZJan 24 19:34:00 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
8952024-01-24T19:34:04.482Z2024-01-24 19:34:04.461051040 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8962024-01-24T19:34:05.483Z2024-01-24 19:34:05.462227413 UTC: attempting to log into API
 
8972024-01-24T19:34:07.907ZJan 24 19:34:07 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
8982024-01-24T19:34:14.689ZJan 24 19:34:13 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
8992024-01-24T19:34:20.517Z2024-01-24 19:34:20.496656686 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9002024-01-24T19:34:21.519Z2024-01-24 19:34:21.497764568 UTC: attempting to log into API
9012024-01-24T19:34:26.622Z2024-01-24 19:34:26.541943967 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: tcp connect error: Connection refused (os error 146)
9022024-01-24T19:34:27.564Z2024-01-24 19:34:27.543299054 UTC: attempting to log into API
9032024-01-24T19:34:28.343Z2024-01-24 19:34:28.321801004 UTC: login succeeded
9042024-01-24T19:34:28.382Zwaiting for nexus to come up...
9052024-01-24T19:34:28.447Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9062024-01-24T19:34:28.645Zsetting up quotas...
9072024-01-24T19:34:28.688Zensuring datasets are ready...
9082024-01-24T19:34:30.986Zlet's roll.
9092024-01-24T19:34:31.027Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
9102024-01-24T19:34:31.027ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
9112024-01-24T19:34:31.027ZOXIDE_TOKEN=oxide-token-878d11d8453e1a67d5db91f10b38b9d93a4f5161'
9122024-01-24T19:34:31.027Z++ OXIDE_HOST=https://recovery.sys.oxide.test
9132024-01-24T19:34:31.027Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
9142024-01-24T19:34:31.028Z++ OXIDE_TOKEN=oxide-token-878d11d8453e1a67d5db91f10b38b9d93a4f5161
9152024-01-24T19:34:31.028Z+ export OXIDE_HOST OXIDE_TOKEN
9162024-01-24T19:34:31.028Z+ retry=0
9172024-01-24T19:34:31.028Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9182024-01-24T19:34:31.077Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
9192024-01-24T19:34:31.427Z{"status":"ok"}success
9202024-01-24T19:34:31.427ZProject {
9212024-01-24T19:34:31.466Z description: "some images",
9222024-01-24T19:34:31.466Z id: 43b6c5b6-4c6d-478a-bdd4-e53d0f95fc99,
9232024-01-24T19:34:31.467Z name: Name(
9242024-01-24T19:34:31.467Z "images",
9252024-01-24T19:34:31.467Z ),
9262024-01-24T19:34:31.467Z time_created: 2024-01-24T19:34:31.151457Z,
9272024-01-24T19:34:31.467Z time_modified: 2024-01-24T19:34:31.151457Z,
9282024-01-24T19:34:31.467Z}
9292024-01-24T19:34:31.467Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
9302024-01-24T19:39:52.839Zdone!
9312024-01-24T19:39:52.943Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
9322024-01-24T19:39:53.059Zsuccess
9332024-01-24T19:39:53.060ZImage {
9342024-01-24T19:39:53.113Z block_size: ByteCount(
9352024-01-24T19:39:53.113Z 512,
9362024-01-24T19:39:53.113Z ),
9372024-01-24T19:39:53.113Z description: "debian 11 original base image",
9382024-01-24T19:39:53.113Z digest: None,
9392024-01-24T19:39:53.113Z id: f6e42403-48be-4123-b291-a987024df63c,
9402024-01-24T19:39:53.114Z name: Name(
9412024-01-24T19:39:53.114Z "debian11",
9422024-01-24T19:39:53.114Z ),
9432024-01-24T19:39:53.114Z os: "debian",
9442024-01-24T19:39:53.114Z project_id: None,
9452024-01-24T19:39:53.114Z size: ByteCount(
9462024-01-24T19:39:53.114Z 2147483648,
9472024-01-24T19:39:53.114Z ),
9482024-01-24T19:39:53.114Z time_created: 2024-01-24T19:39:52.787311Z,
9492024-01-24T19:39:53.115Z time_modified: 2024-01-24T19:39:53.016414Z,
9502024-01-24T19:39:53.115Z url: None,
9512024-01-24T19:39:53.115Z version: "11",
9522024-01-24T19:39:53.115Z}
9532024-01-24T19:39:53.115Z+ rm ./tests/bootstrap
9542024-01-24T19:39:53.157Z+ for test_bin in tests/*
9552024-01-24T19:39:53.157Z+ ./tests/bootstrap-03ff62cdf334f689
9562024-01-24T19:39:53.157Z
9572024-01-24T19:39:53.158Zrunning 0 tests
9582024-01-24T19:39:53.158Z
9592024-01-24T19:39:53.158Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9602024-01-24T19:39:53.158Z
9612024-01-24T19:39:53.158Z+ for test_bin in tests/*
9622024-01-24T19:39:53.158Z+ ./tests/end_to_end_tests-d8fb2167cd3fcce8
9632024-01-24T19:39:53.159Z
9642024-01-24T19:39:53.159Zrunning 2 tests
9652024-01-24T19:39:53.361Ztest no_spoof::no_spoof_here ... ok
9662024-01-24T19:40:28.602Ztest instance_launch::instance_launch ... ok
9672024-01-24T19:40:28.603Z
9682024-01-24T19:40:28.603Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 35.45s
9692024-01-24T19:40:28.603Z
9702024-01-24T19:40:28.642Z+ _exit_trap
9712024-01-24T19:40:28.642Z+ local status=0
9722024-01-24T19:40:28.642Z+ [[ 0 -eq 0 ]]
9732024-01-24T19:40:28.642Z+ exit 0
9742024-01-24T19:40:28.642Zprocess exited: duration 697292 ms, exit code 0
 
9752024-01-24T19:40:28.816Zfound 62 output files
9762024-01-24T19:40:28.816Zuploading: /var/svc/log/oxide-sled-agent:default.log (765777 bytes)
9772024-01-24T19:40:30.775Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 765777 -> 766189
9782024-01-24T19:40:32.920Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9792024-01-24T19:40:32.920Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_29f98148-db89-4aa0-b80a-1c90f804f702/root/var/svc/log/oxide-clickhouse:default.log (1097092 bytes)
9802024-01-24T19:40:35.903Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_29f98148-db89-4aa0-b80a-1c90f804f702/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 1097092 -> 1118443
9812024-01-24T19:40:37.024Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_29f98148-db89-4aa0-b80a-1c90f804f702/root/var/svc/log/oxide-clickhouse:default.log
9822024-01-24T19:40:37.024Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_29f98148-db89-4aa0-b80a-1c90f804f702/root/var/svc/log/oxide-zone-network-setup:default.log (1709 bytes)
9832024-01-24T19:40:38.198Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_29f98148-db89-4aa0-b80a-1c90f804f702/root/var/svc/log/oxide-zone-network-setup:default.log
9842024-01-24T19:40:38.199Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1a5c7a97-d91c-4187-81eb-a381878cdd6b/root/var/svc/log/oxide-cockroachdb:default.log (4380 bytes)
9852024-01-24T19:40:39.365Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1a5c7a97-d91c-4187-81eb-a381878cdd6b/root/var/svc/log/oxide-cockroachdb:default.log
9862024-01-24T19:40:39.365Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1a5c7a97-d91c-4187-81eb-a381878cdd6b/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
9872024-01-24T19:40:40.540Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1a5c7a97-d91c-4187-81eb-a381878cdd6b/root/var/svc/log/oxide-zone-network-setup:default.log
9882024-01-24T19:40:40.540Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_44d125b5-0a90-4090-ba7d-cb28bfc5f7b6/root/var/svc/log/oxide-crucible-agent:default.log (25246 bytes)
9892024-01-24T19:40:42.420Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_44d125b5-0a90-4090-ba7d-cb28bfc5f7b6/root/var/svc/log/oxide-crucible-agent:default.log
9902024-01-24T19:40:42.420Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_44d125b5-0a90-4090-ba7d-cb28bfc5f7b6/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9912024-01-24T19:40:43.580Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_44d125b5-0a90-4090-ba7d-cb28bfc5f7b6/root/var/svc/log/oxide-crucible-downstairs:default.log
9922024-01-24T19:40:43.580Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_44d125b5-0a90-4090-ba7d-cb28bfc5f7b6/root/var/svc/log/oxide-crucible-downstairs:downstairs-2f3610fb-106b-4622-977a-ee2d54670bab.log (2945 bytes)
9932024-01-24T19:40:45.149Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_44d125b5-0a90-4090-ba7d-cb28bfc5f7b6/root/var/svc/log/oxide-crucible-downstairs:downstairs-2f3610fb-106b-4622-977a-ee2d54670bab.log
9942024-01-24T19:40:45.149Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_44d125b5-0a90-4090-ba7d-cb28bfc5f7b6/root/var/svc/log/oxide-crucible-downstairs:downstairs-9bc04a9d-7880-4965-9c9e-9dde35d01a88.log (5539 bytes)
9952024-01-24T19:40:46.350Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_44d125b5-0a90-4090-ba7d-cb28bfc5f7b6/root/var/svc/log/oxide-crucible-downstairs:downstairs-9bc04a9d-7880-4965-9c9e-9dde35d01a88.log
9962024-01-24T19:40:46.352Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_97499add-995a-46b3-948a-17f6f75f5d00/root/var/svc/log/oxide-external_dns:default.log (26941 bytes)
9972024-01-24T19:40:47.669Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_97499add-995a-46b3-948a-17f6f75f5d00/root/var/svc/log/oxide-external_dns:default.log
9982024-01-24T19:40:47.670Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_5f34887c-5cb3-4f55-a5f6-16f8046273b2/root/var/svc/log/oxide-internal_dns:default.log (110281 bytes)
9992024-01-24T19:40:49.049Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_5f34887c-5cb3-4f55-a5f6-16f8046273b2/root/var/svc/log/oxide-internal_dns:default.log
10002024-01-24T19:40:49.050Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_ddefbd3d-3e5c-46a0-8cab-901c84a00bb0/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
10012024-01-24T19:40:50.208Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_ddefbd3d-3e5c-46a0-8cab-901c84a00bb0/root/var/svc/log/oxide-cockroachdb:default.log
10022024-01-24T19:40:50.208Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_ddefbd3d-3e5c-46a0-8cab-901c84a00bb0/root/var/svc/log/oxide-zone-network-setup:default.log (1709 bytes)
10032024-01-24T19:40:51.381Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_ddefbd3d-3e5c-46a0-8cab-901c84a00bb0/root/var/svc/log/oxide-zone-network-setup:default.log
10042024-01-24T19:40:51.381Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_83dfe8e3-6b3c-49a7-8e20-12388c7613f3/root/var/svc/log/oxide-crucible-agent:default.log (14149 bytes)
10052024-01-24T19:40:52.578Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_83dfe8e3-6b3c-49a7-8e20-12388c7613f3/root/var/svc/log/oxide-crucible-agent:default.log
10062024-01-24T19:40:52.578Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_83dfe8e3-6b3c-49a7-8e20-12388c7613f3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10072024-01-24T19:40:53.741Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_83dfe8e3-6b3c-49a7-8e20-12388c7613f3/root/var/svc/log/oxide-crucible-downstairs:default.log
10082024-01-24T19:40:53.741Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_83dfe8e3-6b3c-49a7-8e20-12388c7613f3/root/var/svc/log/oxide-crucible-downstairs:downstairs-94e826b1-ae34-4f5c-bf8a-4377c3f1ea81.log (3051 bytes)
10092024-01-24T19:40:55.364Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_83dfe8e3-6b3c-49a7-8e20-12388c7613f3/root/var/svc/log/oxide-crucible-downstairs:downstairs-94e826b1-ae34-4f5c-bf8a-4377c3f1ea81.log
10102024-01-24T19:40:55.364Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_46619276-62f2-4827-9d15-c0c6d25acfd5/root/var/svc/log/oxide-crucible-pantry:default.log (1211656 bytes)
10112024-01-24T19:40:57.382Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_46619276-62f2-4827-9d15-c0c6d25acfd5/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 1211656 -> 1227500
10122024-01-24T19:40:58.924Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_46619276-62f2-4827-9d15-c0c6d25acfd5/root/var/svc/log/oxide-crucible-pantry:default.log
10132024-01-24T19:40:58.925Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_f0de3ab8-bfa3-4f4b-8778-4dcd69407508/root/var/svc/log/oxide-external_dns:default.log (26946 bytes)
10142024-01-24T19:40:59.787Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_f0de3ab8-bfa3-4f4b-8778-4dcd69407508/root/var/svc/log/oxide-external_dns:default.log
10152024-01-24T19:40:59.787Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_81b4be4b-24cd-4c40-9545-3c2bbb4857bf/root/var/svc/log/oxide-internal_dns:default.log (113024 bytes)
10162024-01-24T19:41:01.457Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_81b4be4b-24cd-4c40-9545-3c2bbb4857bf/root/var/svc/log/oxide-internal_dns:default.log
10172024-01-24T19:41:01.457Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f82e086f-2689-4588-94e6-4efd5197c86c/root/var/svc/log/oxide-cockroachdb:default.log (4406 bytes)
10182024-01-24T19:41:03.175Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f82e086f-2689-4588-94e6-4efd5197c86c/root/var/svc/log/oxide-cockroachdb:default.log
10192024-01-24T19:41:03.175Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f82e086f-2689-4588-94e6-4efd5197c86c/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
10202024-01-24T19:41:04.338Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f82e086f-2689-4588-94e6-4efd5197c86c/root/var/svc/log/oxide-zone-network-setup:default.log
10212024-01-24T19:41:04.338Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_09103344-7702-4bb1-8783-0c042143bde8/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10222024-01-24T19:41:05.590Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_09103344-7702-4bb1-8783-0c042143bde8/root/var/svc/log/oxide-crucible-agent:default.log
10232024-01-24T19:41:05.590Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_09103344-7702-4bb1-8783-0c042143bde8/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10242024-01-24T19:41:06.775Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_09103344-7702-4bb1-8783-0c042143bde8/root/var/svc/log/oxide-crucible-downstairs:default.log
10252024-01-24T19:41:06.776Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_8439feb9-da55-40b1-8824-fb3f12533c55/root/var/svc/log/oxide-internal_dns:default.log (123008 bytes)
10262024-01-24T19:41:08.484Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_8439feb9-da55-40b1-8824-fb3f12533c55/root/var/svc/log/oxide-internal_dns:default.log
10272024-01-24T19:41:08.484Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_79a8cff9-ddc9-4874-9ae9-63947142a748/root/var/svc/log/oxide-nexus:default.log (939589 bytes)
10282024-01-24T19:41:10.940Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_79a8cff9-ddc9-4874-9ae9-63947142a748/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 939589 -> 948872
10292024-01-24T19:41:12.066Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_79a8cff9-ddc9-4874-9ae9-63947142a748/root/var/svc/log/oxide-nexus:default.log
10302024-01-24T19:41:12.066Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7f13f9e8-bb98-448c-be34-3bc6313af543/root/var/svc/log/oxide-cockroachdb:default.log (4406 bytes)
10312024-01-24T19:41:13.248Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7f13f9e8-bb98-448c-be34-3bc6313af543/root/var/svc/log/oxide-cockroachdb:default.log
10322024-01-24T19:41:13.248Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7f13f9e8-bb98-448c-be34-3bc6313af543/root/var/svc/log/oxide-zone-network-setup:default.log (1705 bytes)
10332024-01-24T19:41:14.481Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7f13f9e8-bb98-448c-be34-3bc6313af543/root/var/svc/log/oxide-zone-network-setup:default.log
10342024-01-24T19:41:14.482Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_bd4c1a47-ea43-4b1c-b6b3-5d343cbbaa75/root/var/svc/log/oxide-crucible-agent:default.log (17120 bytes)
10352024-01-24T19:41:15.737Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_bd4c1a47-ea43-4b1c-b6b3-5d343cbbaa75/root/var/svc/log/oxide-crucible-agent:default.log
10362024-01-24T19:41:15.738Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_bd4c1a47-ea43-4b1c-b6b3-5d343cbbaa75/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10372024-01-24T19:41:16.904Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_bd4c1a47-ea43-4b1c-b6b3-5d343cbbaa75/root/var/svc/log/oxide-crucible-downstairs:default.log
10382024-01-24T19:41:16.905Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_bd4c1a47-ea43-4b1c-b6b3-5d343cbbaa75/root/var/svc/log/oxide-crucible-downstairs:downstairs-14b073ce-b40d-40a8-8e90-1fe658fc22cb.log (5620 bytes)
10392024-01-24T19:41:18.141Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_bd4c1a47-ea43-4b1c-b6b3-5d343cbbaa75/root/var/svc/log/oxide-crucible-downstairs:downstairs-14b073ce-b40d-40a8-8e90-1fe658fc22cb.log
10402024-01-24T19:41:18.141Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_bd4c1a47-ea43-4b1c-b6b3-5d343cbbaa75/root/var/svc/log/oxide-crucible-downstairs:snapshot-14b073ce-b40d-40a8-8e90-1fe658fc22cb-a3b29d54-acfd-4978-bda5-b1397605fab9.log (5178 bytes)
10412024-01-24T19:41:19.300Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_bd4c1a47-ea43-4b1c-b6b3-5d343cbbaa75/root/var/svc/log/oxide-crucible-downstairs:snapshot-14b073ce-b40d-40a8-8e90-1fe658fc22cb-a3b29d54-acfd-4978-bda5-b1397605fab9.log
10422024-01-24T19:41:19.300Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9f162514-a317-408e-8ce7-eb06bfdf9224/root/var/svc/log/oxide-nexus:default.log (885606 bytes)
10432024-01-24T19:41:21.220Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9f162514-a317-408e-8ce7-eb06bfdf9224/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 885606 -> 956955
10442024-01-24T19:41:22.471Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9f162514-a317-408e-8ce7-eb06bfdf9224/root/var/svc/log/oxide-nexus:default.log
10452024-01-24T19:41:22.471Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78/root/var/svc/log/oxide-ntp:default.log (781 bytes)
10462024-01-24T19:41:22.558Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78/root/var/svc/log/oxide-ntp:default.log" changed size mid upload: 781 -> 847
10472024-01-24T19:41:23.682Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78/root/var/svc/log/oxide-ntp:default.log
10482024-01-24T19:41:23.682Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_d6165799-3104-4bbe-8405-30103745db59/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
10492024-01-24T19:41:24.849Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_d6165799-3104-4bbe-8405-30103745db59/root/var/svc/log/oxide-cockroachdb:default.log
10502024-01-24T19:41:24.850Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_d6165799-3104-4bbe-8405-30103745db59/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
10512024-01-24T19:41:26.037Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_d6165799-3104-4bbe-8405-30103745db59/root/var/svc/log/oxide-zone-network-setup:default.log
10522024-01-24T19:41:26.038Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4d5607cc-e42d-4254-8313-4eb78e3db7b3/root/var/svc/log/oxide-crucible-agent:default.log (14114 bytes)
10532024-01-24T19:41:27.268Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4d5607cc-e42d-4254-8313-4eb78e3db7b3/root/var/svc/log/oxide-crucible-agent:default.log
10542024-01-24T19:41:27.269Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4d5607cc-e42d-4254-8313-4eb78e3db7b3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10552024-01-24T19:41:28.451Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4d5607cc-e42d-4254-8313-4eb78e3db7b3/root/var/svc/log/oxide-crucible-downstairs:default.log
10562024-01-24T19:41:28.451Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4d5607cc-e42d-4254-8313-4eb78e3db7b3/root/var/svc/log/oxide-crucible-downstairs:downstairs-532678e5-2aa5-436d-b310-518d13afd571.log (5534 bytes)
10572024-01-24T19:41:29.651Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4d5607cc-e42d-4254-8313-4eb78e3db7b3/root/var/svc/log/oxide-crucible-downstairs:downstairs-532678e5-2aa5-436d-b310-518d13afd571.log
10582024-01-24T19:41:29.651Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_70a117e9-1a5f-484f-b2b1-32f5056916f5/root/var/svc/log/oxide-oximeter:default.log (294341 bytes)
10592024-01-24T19:41:30.581Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_70a117e9-1a5f-484f-b2b1-32f5056916f5/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 294341 -> 341858
10602024-01-24T19:41:31.708Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_70a117e9-1a5f-484f-b2b1-32f5056916f5/root/var/svc/log/oxide-oximeter:default.log
10612024-01-24T19:41:31.708Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6c671d53-24e2-4e27-9da3-f34ff6cdd06e/root/var/svc/log/oxide-crucible-agent:default.log (34152 bytes)
10622024-01-24T19:41:33.016Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6c671d53-24e2-4e27-9da3-f34ff6cdd06e/root/var/svc/log/oxide-crucible-agent:default.log
10632024-01-24T19:41:33.016Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6c671d53-24e2-4e27-9da3-f34ff6cdd06e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10642024-01-24T19:41:34.217Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6c671d53-24e2-4e27-9da3-f34ff6cdd06e/root/var/svc/log/oxide-crucible-downstairs:default.log
10652024-01-24T19:41:34.217Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6c671d53-24e2-4e27-9da3-f34ff6cdd06e/root/var/svc/log/oxide-crucible-downstairs:downstairs-04311c2b-9975-4f31-89b2-39ed4a6bbeae.log (3050 bytes)
10662024-01-24T19:41:35.459Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6c671d53-24e2-4e27-9da3-f34ff6cdd06e/root/var/svc/log/oxide-crucible-downstairs:downstairs-04311c2b-9975-4f31-89b2-39ed4a6bbeae.log
10672024-01-24T19:41:35.459Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6c671d53-24e2-4e27-9da3-f34ff6cdd06e/root/var/svc/log/oxide-crucible-downstairs:downstairs-1d06710b-3550-4c4e-8c41-eb2e8dbc20cd.log (2941 bytes)
10682024-01-24T19:41:37.047Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6c671d53-24e2-4e27-9da3-f34ff6cdd06e/root/var/svc/log/oxide-crucible-downstairs:downstairs-1d06710b-3550-4c4e-8c41-eb2e8dbc20cd.log
10692024-01-24T19:41:37.048Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6c671d53-24e2-4e27-9da3-f34ff6cdd06e/root/var/svc/log/oxide-crucible-downstairs:downstairs-f21ca891-3ef6-4fa0-ab10-6b831a573f45.log (5621 bytes)
10702024-01-24T19:41:38.734Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6c671d53-24e2-4e27-9da3-f34ff6cdd06e/root/var/svc/log/oxide-crucible-downstairs:downstairs-f21ca891-3ef6-4fa0-ab10-6b831a573f45.log
10712024-01-24T19:41:38.793Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6c671d53-24e2-4e27-9da3-f34ff6cdd06e/root/var/svc/log/oxide-crucible-downstairs:snapshot-f21ca891-3ef6-4fa0-ab10-6b831a573f45-a3b29d54-acfd-4978-bda5-b1397605fab9.log (5176 bytes)
10722024-01-24T19:41:39.941Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6c671d53-24e2-4e27-9da3-f34ff6cdd06e/root/var/svc/log/oxide-crucible-downstairs:snapshot-f21ca891-3ef6-4fa0-ab10-6b831a573f45-a3b29d54-acfd-4978-bda5-b1397605fab9.log
10732024-01-24T19:41:39.942Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_eda708b6-5594-4a00-a9bf-811fa09c623e/root/var/svc/log/oxide-crucible-agent:default.log (38048 bytes)
10742024-01-24T19:41:41.340Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_eda708b6-5594-4a00-a9bf-811fa09c623e/root/var/svc/log/oxide-crucible-agent:default.log
10752024-01-24T19:41:41.340Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_eda708b6-5594-4a00-a9bf-811fa09c623e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10762024-01-24T19:41:42.517Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_eda708b6-5594-4a00-a9bf-811fa09c623e/root/var/svc/log/oxide-crucible-downstairs:default.log
10772024-01-24T19:41:42.517Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_eda708b6-5594-4a00-a9bf-811fa09c623e/root/var/svc/log/oxide-crucible-downstairs:downstairs-2ec09e51-4f39-4d1d-b61c-7f09f8ee861d.log (5537 bytes)
10782024-01-24T19:41:43.718Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_eda708b6-5594-4a00-a9bf-811fa09c623e/root/var/svc/log/oxide-crucible-downstairs:downstairs-2ec09e51-4f39-4d1d-b61c-7f09f8ee861d.log
10792024-01-24T19:41:43.718Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_eda708b6-5594-4a00-a9bf-811fa09c623e/root/var/svc/log/oxide-crucible-downstairs:downstairs-dea68826-1ead-4a2a-9dfe-21aea3633ff6.log (3049 bytes)
10802024-01-24T19:41:44.935Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_eda708b6-5594-4a00-a9bf-811fa09c623e/root/var/svc/log/oxide-crucible-downstairs:downstairs-dea68826-1ead-4a2a-9dfe-21aea3633ff6.log
10812024-01-24T19:41:44.935Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_eda708b6-5594-4a00-a9bf-811fa09c623e/root/var/svc/log/oxide-crucible-downstairs:downstairs-e4ba53a7-f338-47ac-bc53-7f1df94d58e7.log (5620 bytes)
10822024-01-24T19:41:46.129Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_eda708b6-5594-4a00-a9bf-811fa09c623e/root/var/svc/log/oxide-crucible-downstairs:downstairs-e4ba53a7-f338-47ac-bc53-7f1df94d58e7.log
10832024-01-24T19:41:46.129Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_eda708b6-5594-4a00-a9bf-811fa09c623e/root/var/svc/log/oxide-crucible-downstairs:snapshot-e4ba53a7-f338-47ac-bc53-7f1df94d58e7-a3b29d54-acfd-4978-bda5-b1397605fab9.log (5178 bytes)
10842024-01-24T19:41:47.289Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_eda708b6-5594-4a00-a9bf-811fa09c623e/root/var/svc/log/oxide-crucible-downstairs:snapshot-e4ba53a7-f338-47ac-bc53-7f1df94d58e7-a3b29d54-acfd-4978-bda5-b1397605fab9.log
10852024-01-24T19:41:47.289Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_89a4ea2a-de3f-41fc-80b5-33daf003f2d9/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10862024-01-24T19:41:48.446Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_89a4ea2a-de3f-41fc-80b5-33daf003f2d9/root/var/svc/log/oxide-crucible-pantry:default.log
10872024-01-24T19:41:48.447Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d04fa8-8754-4ae2-8a1e-9c1d01352d7d/root/var/svc/log/oxide-crucible-agent:default.log (9795 bytes)
10882024-01-24T19:41:49.641Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d04fa8-8754-4ae2-8a1e-9c1d01352d7d/root/var/svc/log/oxide-crucible-agent:default.log
10892024-01-24T19:41:49.641Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d04fa8-8754-4ae2-8a1e-9c1d01352d7d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10902024-01-24T19:41:50.797Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d04fa8-8754-4ae2-8a1e-9c1d01352d7d/root/var/svc/log/oxide-crucible-downstairs:default.log
10912024-01-24T19:41:50.797Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d04fa8-8754-4ae2-8a1e-9c1d01352d7d/root/var/svc/log/oxide-crucible-downstairs:downstairs-e5e2ad59-0f68-4a2f-802f-df6812916ec6.log (2945 bytes)
10922024-01-24T19:41:51.953Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d04fa8-8754-4ae2-8a1e-9c1d01352d7d/root/var/svc/log/oxide-crucible-downstairs:downstairs-e5e2ad59-0f68-4a2f-802f-df6812916ec6.log
10932024-01-24T19:41:51.953Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_022e1381-887b-4622-a90f-cd4c19a584dd/root/var/svc/log/oxide-nexus:default.log (35942098 bytes)
10942024-01-24T19:42:07.915Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_022e1381-887b-4622-a90f-cd4c19a584dd/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 35942098 -> 36021557
10952024-01-24T19:42:09.047Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_022e1381-887b-4622-a90f-cd4c19a584dd/root/var/svc/log/oxide-nexus:default.log
10962024-01-24T19:42:09.047Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_4d47b41e-3603-43a9-896e-32334571e69f/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10972024-01-24T19:42:10.205Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_4d47b41e-3603-43a9-896e-32334571e69f/root/var/svc/log/oxide-crucible-agent:default.log
10982024-01-24T19:42:10.206Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_4d47b41e-3603-43a9-896e-32334571e69f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10992024-01-24T19:42:11.364Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_4d47b41e-3603-43a9-896e-32334571e69f/root/var/svc/log/oxide-crucible-downstairs:default.log
11002024-01-24T19:42:11.364Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_98758bf2-e4b8-4f1a-989c-368cbfd7b7e5/root/var/svc/log/oxide-crucible-pantry:default.log (2337 bytes)
11012024-01-24T19:42:12.523Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_98758bf2-e4b8-4f1a-989c-368cbfd7b7e5/root/var/svc/log/oxide-crucible-pantry:default.log
11022024-01-24T19:42:12.524Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78/root/var/log/chrony/measurements.log (15618 bytes)
11032024-01-24T19:42:12.676Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78/root/var/log/chrony/measurements.log" changed size mid upload: 15618 -> 17673
11042024-01-24T19:42:13.796Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78/root/var/log/chrony/measurements.log
11052024-01-24T19:42:13.796Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78/root/var/log/chrony/statistics.log (10413 bytes)
11062024-01-24T19:42:13.921Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78/root/var/log/chrony/statistics.log" changed size mid upload: 10413 -> 12168
11072024-01-24T19:42:15.047Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78/root/var/log/chrony/statistics.log
11082024-01-24T19:42:15.047Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78/root/var/log/chrony/tracking.log (1719 bytes)
11092024-01-24T19:42:15.152Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78/root/var/log/chrony/tracking.log" changed size mid upload: 1719 -> 1983
11102024-01-24T19:42:16.272Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_36cbece9-6e43-4715-91bf-3c89c5339f78/root/var/log/chrony/tracking.log