01HM7Q8JB11WD4G141DGDSK0BJ: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.27",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HM7Q9VB5SQ3WDKBBRVVZ2AAC

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12024-01-16T00:22:53.734Zjob dependencies complete; ready to run (waiting for 53 m 56 s)
22024-01-16T00:37:54.443Zjob assigned to worker 01HM7TY6R6WMGR4VNSAK5D642H (queued for 15 m 0 s)
32024-01-16T00:37:54.728Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42024-01-16T00:37:54.786Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52024-01-16T00:37:55.010Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62024-01-16T00:37:55.089Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72024-01-16T00:37:55.134Zhttp://172.20.0.5:9969/postboot/buskin... ok
82024-01-16T00:37:55.179ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92024-01-16T00:37:55.245Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102024-01-16T00:37:55.304ZOxide Helios Version helios-2.0.22333 64-bit
112024-01-16T00:37:55.349ZConfiguring devices.
122024-01-16T00:37:55.393Z##################################################
132024-01-16T00:37:55.438Z#### ############# ##
142024-01-16T00:37:55.482Z### ### ############ ##
152024-01-16T00:37:55.526Z## ### # ## ### ## Oxide Computer Company ##
162024-01-16T00:37:55.571Z## ## ## ### # ### ##
172024-01-16T00:37:55.616Z## # ### #### #### This Station Under ##
182024-01-16T00:37:55.660Z### ### #### # ### Computer Control ##
192024-01-16T00:37:55.705Z#### #### ### ## ##
202024-01-16T00:37:55.749Z##################################################
212024-01-16T00:37:55.794ZHostname: buskin
222024-01-16T00:37:55.838ZLoading smf(7) service descriptions:
232024-01-16T00:37:56.222Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-29cdd1ea464df3b1.gz
242024-01-16T00:37:57.434Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-29cdd1ea464df3b1.gz
252024-01-16T00:37:57.434Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262024-01-16T00:38:14.773Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272024-01-16T00:38:14.773Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-bae04a45dbbda4ab.gz
282024-01-16T00:38:33.105Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-bae04a45dbbda4ab.gz
292024-01-16T00:38:33.105Zdownloading input: /input/ci-tools/work/caboose-util.gz
302024-01-16T00:38:33.932Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312024-01-16T00:38:33.932Zdownloading input: /input/ci-tools/work/tufaceous.gz
322024-01-16T00:38:38.877Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332024-01-16T00:38:38.901Zdownloading input: /input/package/work/version.txt
342024-01-16T00:38:38.926Zdownloaded input: /input/package/work/version.txt
352024-01-16T00:38:38.926Zdownloading input: /input/package/work/package.tar.gz
362024-01-16T00:38:52.537Zdownloaded input: /input/package/work/package.tar.gz
372024-01-16T00:38:52.537Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382024-01-16T00:39:12.622Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392024-01-16T00:39:12.622Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402024-01-16T00:39:18.908Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412024-01-16T00:39:18.932Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422024-01-16T00:39:52.916Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432024-01-16T00:39:52.917Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442024-01-16T00:40:26.860Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452024-01-16T00:40:26.884Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462024-01-16T00:40:49.889Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472024-01-16T00:40:49.889Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482024-01-16T00:40:52.758Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492024-01-16T00:40:52.758Zdownloading input: /input/package/work/zones/crucible.tar.gz
502024-01-16T00:40:58.024Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512024-01-16T00:40:58.024Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522024-01-16T00:41:02.382Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532024-01-16T00:41:02.407Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542024-01-16T00:41:06.699Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552024-01-16T00:41:06.699Zdownloading input: /input/package/work/zones/ntp.tar.gz
562024-01-16T00:41:06.757Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572024-01-16T00:41:06.782Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582024-01-16T00:41:10.044Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592024-01-16T00:41:10.044Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602024-01-16T00:41:13.359Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612024-01-16T00:41:13.360Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622024-01-16T00:41:26.783Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632024-01-16T00:41:26.783Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642024-01-16T00:41:40.307Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652024-01-16T00:41:40.307Zdownloading input: /input/package/work/zones/overlay.tar.gz
662024-01-16T00:41:40.389Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672024-01-16T00:41:40.413Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682024-01-16T00:41:47.136Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692024-01-16T00:41:47.163Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702024-01-16T00:41:58.228Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712024-01-16T00:41:58.228Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722024-01-16T00:42:42.152Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732024-01-16T00:42:42.152Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742024-01-16T00:43:13.974Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752024-01-16T00:43:13.974Zstarting task 0: "setup"
762024-01-16T00:43:14.000Z++ uname -s
772024-01-16T00:43:14.000Z+ kern=SunOS
782024-01-16T00:43:14.000Z+ case "$kern" in
792024-01-16T00:43:14.000Z+ groupadd -g 12345 build
802024-01-16T00:43:14.000Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812024-01-16T00:43:16.400Z+ zfs create -o mountpoint=/work rpool/work
822024-01-16T00:43:16.427Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832024-01-16T00:43:16.427Z+ home_fs=zfs
842024-01-16T00:43:16.427Z+ [[ zfs == autofs ]]
852024-01-16T00:43:16.427Z+ mkdir -p /home/build
862024-01-16T00:43:16.427Z+ chown build:build /home/build /work
872024-01-16T00:43:18.018Z+ chmod 0700 /home/build /work
882024-01-16T00:43:18.043Zprocess exited: duration 4049 ms, exit code 0
 
892024-01-16T00:43:18.096Zstarting task 1: "authentication"
902024-01-16T00:43:18.137Zprocess exited: duration 44 ms, exit code 0
 
912024-01-16T00:43:18.185Zstarting task 2: "build"
922024-01-16T00:43:18.209Z+ trap _exit_trap EXIT
932024-01-16T00:43:18.209Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942024-01-16T00:43:18.389Zsecpolicy_fs_linkdir: 0x55 = 0x55
952024-01-16T00:43:18.389Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962024-01-16T00:43:18.389Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972024-01-16T00:43:18.389Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982024-01-16T00:43:18.389Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992024-01-16T00:43:18.389Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002024-01-16T00:43:18.390Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012024-01-16T00:43:18.390Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022024-01-16T00:43:18.390Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032024-01-16T00:43:18.390Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042024-01-16T00:43:18.414Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052024-01-16T00:43:18.414Z+ [[ -d /opt/oxide ]]
1062024-01-16T00:43:18.414Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072024-01-16T00:43:18.414Z+ pfexec mkdir /opt/oxide
1082024-01-16T00:43:18.414Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092024-01-16T00:43:18.414Z+ [[ -d /opt/oxide-underneath ]]
1102024-01-16T00:43:18.414Z+ cd /opt/oxide-underneath
1112024-01-16T00:43:18.414Z+ pfexec tar ceEp@/f - .
1122024-01-16T00:43:18.415Z+ cd /opt/oxide
1132024-01-16T00:43:18.415Z+ pfexec tar xveEp@/f -
1142024-01-16T00:43:18.518Zx ., 0 bytes, 0 tape blocks
1152024-01-16T00:43:18.518Zx ./opte, 0 bytes, 0 tape blocks
1162024-01-16T00:43:18.518Zx ./opte/bin, 0 bytes, 0 tape blocks
1172024-01-16T00:43:18.518Zx ./opte/bin/opteadm, 52567720 bytes, 102672 tape blocks
1182024-01-16T00:43:18.542Z+ pfexec rm -rf /opt/oxide-underneath
1192024-01-16T00:43:18.542Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202024-01-16T00:43:18.566Z+ pfexec mkdir /opt/oxide/work
1212024-01-16T00:43:18.566Z+ pfexec chown build:build /opt/oxide/work
1222024-01-16T00:43:18.566Z+ cd /opt/oxide/work
1232024-01-16T00:43:18.566Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242024-01-16T00:43:20.114ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252024-01-16T00:43:20.114Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262024-01-16T00:43:20.114Zx out/omicron-sled-agent.tar, 114390016 bytes, 223418 tape blocks
1272024-01-16T00:43:20.115Zx out/target/test, 75 bytes, 1 tape blocks
1282024-01-16T00:43:20.115Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1292024-01-16T00:43:20.115Zx package-manifest.toml, 22919 bytes, 45 tape blocks
1302024-01-16T00:43:20.115Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1312024-01-16T00:43:20.115Zx target/release/omicron-package, 43063472 bytes, 84109 tape blocks
1322024-01-16T00:43:20.115Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332024-01-16T00:43:20.115Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342024-01-16T00:43:20.115Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352024-01-16T00:43:20.115Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362024-01-16T00:43:20.115Zx tools/scrimlet/softnpu-init.sh, 2149 bytes, 5 tape blocks
1372024-01-16T00:43:20.116Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382024-01-16T00:43:20.116Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392024-01-16T00:43:20.141Z
1402024-01-16T00:43:20.141Zreal 1.554203922
1412024-01-16T00:43:20.141Zuser 1.398737879
1422024-01-16T00:43:20.141Zsys 0.273874504
1432024-01-16T00:43:20.141Ztrap 0.000058150
1442024-01-16T00:43:20.141Ztflt 0.000000000
1452024-01-16T00:43:20.141Zdflt 0.000000000
1462024-01-16T00:43:20.141Zkflt 0.000000000
1472024-01-16T00:43:20.141Zlock 0.000000000
1482024-01-16T00:43:20.141Zslp 1.396372496
1492024-01-16T00:43:20.141Zlat 0.037291000
1502024-01-16T00:43:20.141Zstop 0.000130839
1512024-01-16T00:43:20.141Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522024-01-16T00:43:21.209Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532024-01-16T00:43:21.233Z+ mkdir tests
1542024-01-16T00:43:21.233Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552024-01-16T00:43:21.233Z+ ptime -m gunzip
1562024-01-16T00:43:21.233Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-29cdd1ea464df3b1
1572024-01-16T00:43:21.369Z
1582024-01-16T00:43:21.398Zreal 0.139316222
1592024-01-16T00:43:21.398Zuser 0.126216449
1602024-01-16T00:43:21.398Zsys 0.012859780
1612024-01-16T00:43:21.398Ztrap 0.000015489
1622024-01-16T00:43:21.399Ztflt 0.000000000
1632024-01-16T00:43:21.399Zdflt 0.000000000
1642024-01-16T00:43:21.399Zkflt 0.000000000
1652024-01-16T00:43:21.399Zlock 0.000000000
1662024-01-16T00:43:21.399Zslp 0.000017929
1672024-01-16T00:43:21.399Zlat 0.000027210
1682024-01-16T00:43:21.399Zstop 0.000120151
1692024-01-16T00:43:21.399Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-29cdd1ea464df3b1
1702024-01-16T00:43:21.399Z+ chmod a+x tests/bootstrap-29cdd1ea464df3b1
1712024-01-16T00:43:21.399Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722024-01-16T00:43:21.399Z+ ptime -m gunzip
1732024-01-16T00:43:21.399Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742024-01-16T00:43:23.657Z
1752024-01-16T00:43:23.682Zreal 2.273452909
1762024-01-16T00:43:23.682Zuser 2.134183939
1772024-01-16T00:43:23.682Zsys 0.138882159
1782024-01-16T00:43:23.682Ztrap 0.000158670
1792024-01-16T00:43:23.682Ztflt 0.000000000
1802024-01-16T00:43:23.682Zdflt 0.000000000
1812024-01-16T00:43:23.682Zkflt 0.000000000
1822024-01-16T00:43:23.682Zlock 0.000000000
1832024-01-16T00:43:23.682Zslp 0.000024700
1842024-01-16T00:43:23.682Zlat 0.000023239
1852024-01-16T00:43:23.682Zstop 0.000126511
1862024-01-16T00:43:23.682Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872024-01-16T00:43:23.683Z+ chmod a+x tests/bootstrap
1882024-01-16T00:43:23.683Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892024-01-16T00:43:23.683Z+ ptime -m gunzip
1902024-01-16T00:43:23.683Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-bae04a45dbbda4ab
1912024-01-16T00:43:25.995Z
1922024-01-16T00:43:26.019Zreal 2.329773577
1932024-01-16T00:43:26.019Zuser 2.191448102
1942024-01-16T00:43:26.019Zsys 0.137934622
1952024-01-16T00:43:26.019Ztrap 0.000166120
1962024-01-16T00:43:26.019Ztflt 0.000000000
1972024-01-16T00:43:26.020Zdflt 0.000000000
1982024-01-16T00:43:26.020Zkflt 0.000000000
1992024-01-16T00:43:26.020Zlock 0.000000000
2002024-01-16T00:43:26.020Zslp 0.000019700
2012024-01-16T00:43:26.020Zlat 0.000021470
2022024-01-16T00:43:26.020Zstop 0.000117730
2032024-01-16T00:43:26.020Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-bae04a45dbbda4ab
2042024-01-16T00:43:26.020Z+ chmod a+x tests/end_to_end_tests-bae04a45dbbda4ab
2052024-01-16T00:43:26.020Z+ bmat address ls
2062024-01-16T00:43:26.020ZNAME CIDR FIRST LAST COUNT
2072024-01-16T00:43:26.020Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082024-01-16T00:43:26.020Z++ bmat address ls -f extra -Ho first,last,count
2092024-01-16T00:43:26.048Z+ set -- 10.151.1.164 10.151.1.227 64
2102024-01-16T00:43:26.049Z+ EXTRA_IP_START=10.151.1.164
2112024-01-16T00:43:26.049Z+ EXTRA_IP_END=10.151.1.227
2122024-01-16T00:43:26.049Z+ EXTRA_IP_COUNT=64
2132024-01-16T00:43:26.049Z+ (( EXTRA_IP_COUNT >= 32 ))
2142024-01-16T00:43:26.049Z+ EXTRA_IP_BASE=10.151.1
2152024-01-16T00:43:26.050Z+ EXTRA_IP_FOCTET=164
2162024-01-16T00:43:26.050Z+ EXTRA_IP_LOCTET=227
2172024-01-16T00:43:26.050Z+ SERVICE_IP_POOL_START=10.151.1.164
2182024-01-16T00:43:26.050Z+ SERVICE_IP_POOL_END=10.151.1.173
2192024-01-16T00:43:26.050Z+ DNS_IP1=10.151.1.164
2202024-01-16T00:43:26.050Z+ DNS_IP2=10.151.1.165
2212024-01-16T00:43:26.050Z+ UPLINK_IP=10.151.1.174
2222024-01-16T00:43:26.050Z+ IPPOOL_START=10.151.1.175
2232024-01-16T00:43:26.050Z+ IPPOOL_END=10.151.1.227
2242024-01-16T00:43:26.050Z++ ipadm show-addr -po type,addr
2252024-01-16T00:43:26.050Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262024-01-16T00:43:26.050Z+ GATEWAY_IP=10.151.1.100
2272024-01-16T00:43:26.050Z+ [[ -n 10.151.1.100 ]]
2282024-01-16T00:43:26.050Z+ ping -s 10.151.1.100 56 1
2292024-01-16T00:43:26.050ZPING 10.151.1.100: 56 data bytes
2302024-01-16T00:43:26.050Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
2312024-01-16T00:43:26.050Z
2322024-01-16T00:43:26.050Z----10.151.1.100 PING Statistics----
2332024-01-16T00:43:26.051Z1 packets transmitted, 1 packets received, 0% packet loss
2342024-01-16T00:43:26.051Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352024-01-16T00:43:26.051Z++ arp -an
2362024-01-16T00:43:26.051Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372024-01-16T00:43:26.075Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382024-01-16T00:43:26.075Z+ routeadm -e ipv4-forwarding -u
2392024-01-16T00:43:26.220Z+ PXA_START=10.151.1.164
2402024-01-16T00:43:26.221Z+ PXA_END=10.151.1.227
2412024-01-16T00:43:26.221Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422024-01-16T00:43:26.221Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432024-01-16T00:43:26.298Z+ ZPOOL_VDEV_DIR=/scratch
2442024-01-16T00:43:26.298Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452024-01-16T00:43:26.323Z+++ dirname ./tools/create_virtual_hardware.sh
2462024-01-16T00:43:26.323Z++ cd ./tools
2472024-01-16T00:43:26.323Z++ pwd
2482024-01-16T00:43:26.323Z+ SOURCE_DIR=/opt/oxide/work/tools
2492024-01-16T00:43:26.323Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502024-01-16T00:43:26.323Z+ SOFTNPU_MODE=zone
2512024-01-16T00:43:26.323Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522024-01-16T00:43:26.324Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532024-01-16T00:43:26.324Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542024-01-16T00:43:26.324Z++ dladm show-phys -p -o LINK
2552024-01-16T00:43:26.324Z++ head -1
2562024-01-16T00:43:26.382Z+ PHYSICAL_LINK=igb0
2572024-01-16T00:43:26.407Z+ echo 'Using igb0 as physical link'
2582024-01-16T00:43:26.407ZUsing igb0 as physical link
2592024-01-16T00:43:26.407Z+ ensure_run_as_root
2602024-01-16T00:43:26.407Z++ id -u
2612024-01-16T00:43:26.407Z+ [[ 0 -ne 0 ]]
2622024-01-16T00:43:26.407Z+ ensure_zpools
2632024-01-16T00:43:26.407Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642024-01-16T00:43:26.407Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652024-01-16T00:43:26.407Z+ readarray -t ZPOOLS
2662024-01-16T00:43:26.407Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672024-01-16T00:43:26.407Z++ sed 's/[ ",]//g'
2682024-01-16T00:43:26.407Z+ for ZPOOL in "${ZPOOLS[@]}"
2692024-01-16T00:43:26.407Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702024-01-16T00:43:26.407ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712024-01-16T00:43:26.407Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722024-01-16T00:43:26.407Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732024-01-16T00:43:26.408Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742024-01-16T00:43:26.432Z0+0 records in
2752024-01-16T00:43:26.432Z0+0 records out
2762024-01-16T00:43:26.432Z0 bytes (0 iB) transferred in 0.000008 secs (0 iB/sec)
2772024-01-16T00:43:26.432Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782024-01-16T00:43:26.432Z+ set +x
2792024-01-16T00:43:26.432ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802024-01-16T00:43:26.432Z++ zpool list -o name
2812024-01-16T00:43:26.432Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822024-01-16T00:43:26.432Z+ [[ -z '' ]]
2832024-01-16T00:43:26.432Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842024-01-16T00:43:26.477Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852024-01-16T00:43:26.504Z+ set +x
2862024-01-16T00:43:26.504ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872024-01-16T00:43:26.504Z+ for ZPOOL in "${ZPOOLS[@]}"
2882024-01-16T00:43:26.504Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892024-01-16T00:43:26.504ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902024-01-16T00:43:26.504Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912024-01-16T00:43:26.505Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922024-01-16T00:43:26.505Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932024-01-16T00:43:26.505Z0+0 records in
2942024-01-16T00:43:26.505Z0+0 records out
2952024-01-16T00:43:26.505Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2962024-01-16T00:43:26.505Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972024-01-16T00:43:26.505Z+ set +x
2982024-01-16T00:43:26.506ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992024-01-16T00:43:26.506Z++ zpool list -o name
3002024-01-16T00:43:26.506Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012024-01-16T00:43:26.506Z+ [[ -z '' ]]
3022024-01-16T00:43:26.506Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032024-01-16T00:43:26.530Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042024-01-16T00:43:26.530Z+ set +x
3052024-01-16T00:43:26.530ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062024-01-16T00:43:26.530Z+ for ZPOOL in "${ZPOOLS[@]}"
3072024-01-16T00:43:26.530Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082024-01-16T00:43:26.530ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3092024-01-16T00:43:26.530Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102024-01-16T00:43:26.530Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112024-01-16T00:43:26.530Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122024-01-16T00:43:26.530Z0+0 records in
3132024-01-16T00:43:26.531Z0+0 records out
3142024-01-16T00:43:26.531Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3152024-01-16T00:43:26.531Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162024-01-16T00:43:26.531Z+ set +x
3172024-01-16T00:43:26.531ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182024-01-16T00:43:26.531Z++ zpool list -o name
3192024-01-16T00:43:26.531Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202024-01-16T00:43:26.531Z+ [[ -z '' ]]
3212024-01-16T00:43:26.531Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222024-01-16T00:43:26.555Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232024-01-16T00:43:26.555Z+ set +x
3242024-01-16T00:43:26.555ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252024-01-16T00:43:26.555Z+ for ZPOOL in "${ZPOOLS[@]}"
3262024-01-16T00:43:26.556Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272024-01-16T00:43:26.556ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282024-01-16T00:43:26.557Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292024-01-16T00:43:26.557Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302024-01-16T00:43:26.557Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312024-01-16T00:43:26.581Z0+0 records in
3322024-01-16T00:43:26.581Z0+0 records out
3332024-01-16T00:43:26.581Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342024-01-16T00:43:26.581Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352024-01-16T00:43:26.581Z+ set +x
3362024-01-16T00:43:26.581ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372024-01-16T00:43:26.582Z++ zpool list -o name
3382024-01-16T00:43:26.582Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392024-01-16T00:43:26.582Z+ [[ -z '' ]]
3402024-01-16T00:43:26.582Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412024-01-16T00:43:26.606Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422024-01-16T00:43:26.606Z+ set +x
3432024-01-16T00:43:26.606ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442024-01-16T00:43:26.606Z+ for ZPOOL in "${ZPOOLS[@]}"
3452024-01-16T00:43:26.606Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462024-01-16T00:43:26.606ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472024-01-16T00:43:26.607Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482024-01-16T00:43:26.607Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492024-01-16T00:43:26.607Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502024-01-16T00:43:26.607Z0+0 records in
3512024-01-16T00:43:26.607Z0+0 records out
3522024-01-16T00:43:26.607Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3532024-01-16T00:43:26.607Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542024-01-16T00:43:26.607Z+ set +x
3552024-01-16T00:43:26.607ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562024-01-16T00:43:26.607Z++ zpool list -o name
3572024-01-16T00:43:26.607Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582024-01-16T00:43:26.631Z+ [[ -z '' ]]
3592024-01-16T00:43:26.631Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602024-01-16T00:43:26.655Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612024-01-16T00:43:26.655Z+ set +x
3622024-01-16T00:43:26.655ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632024-01-16T00:43:26.656Z+ for ZPOOL in "${ZPOOLS[@]}"
3642024-01-16T00:43:26.656Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652024-01-16T00:43:26.656Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3662024-01-16T00:43:26.656Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3672024-01-16T00:43:26.656Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3682024-01-16T00:43:26.656ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3692024-01-16T00:43:26.656Z0+0 records in
3702024-01-16T00:43:26.656Z0+0 records out
3712024-01-16T00:43:26.656Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3722024-01-16T00:43:26.656Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732024-01-16T00:43:26.656Z+ set +x
3742024-01-16T00:43:26.656ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752024-01-16T00:43:26.656Z++ zpool list -o name
3762024-01-16T00:43:26.656Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772024-01-16T00:43:26.682Z+ [[ -z '' ]]
3782024-01-16T00:43:26.682Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792024-01-16T00:43:26.707Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802024-01-16T00:43:26.707Z+ set +x
3812024-01-16T00:43:26.707ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822024-01-16T00:43:26.707Z+ for ZPOOL in "${ZPOOLS[@]}"
3832024-01-16T00:43:26.707Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842024-01-16T00:43:26.707ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852024-01-16T00:43:26.707Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862024-01-16T00:43:26.707Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872024-01-16T00:43:26.707Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882024-01-16T00:43:26.708Z0+0 records in
3892024-01-16T00:43:26.708Z0+0 records out
3902024-01-16T00:43:26.708Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912024-01-16T00:43:26.708Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922024-01-16T00:43:26.708Z+ set +x
3932024-01-16T00:43:26.708ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942024-01-16T00:43:26.708Z++ zpool list -o name
3952024-01-16T00:43:26.709Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962024-01-16T00:43:26.735Z+ [[ -z '' ]]
3972024-01-16T00:43:26.735Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982024-01-16T00:43:26.761Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992024-01-16T00:43:26.761Z+ set +x
4002024-01-16T00:43:26.761ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012024-01-16T00:43:26.761Z+ for ZPOOL in "${ZPOOLS[@]}"
4022024-01-16T00:43:26.761Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032024-01-16T00:43:26.761ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042024-01-16T00:43:26.761Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052024-01-16T00:43:26.761Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062024-01-16T00:43:26.761Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072024-01-16T00:43:26.761Z0+0 records in
4082024-01-16T00:43:26.761Z0+0 records out
4092024-01-16T00:43:26.761Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102024-01-16T00:43:26.761Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112024-01-16T00:43:26.761Z+ set +x
4122024-01-16T00:43:26.761ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132024-01-16T00:43:26.761Z++ zpool list -o name
4142024-01-16T00:43:26.761Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152024-01-16T00:43:26.761Z+ [[ -z '' ]]
4162024-01-16T00:43:26.761Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172024-01-16T00:43:26.786Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182024-01-16T00:43:26.786Z+ set +x
4192024-01-16T00:43:26.810ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202024-01-16T00:43:26.810Z+ for ZPOOL in "${ZPOOLS[@]}"
4212024-01-16T00:43:26.810Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222024-01-16T00:43:26.810ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232024-01-16T00:43:26.811Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242024-01-16T00:43:26.811Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252024-01-16T00:43:26.811Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262024-01-16T00:43:26.811Z0+0 records in
4272024-01-16T00:43:26.811Z0+0 records out
4282024-01-16T00:43:26.811Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4292024-01-16T00:43:26.811Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302024-01-16T00:43:26.811Z+ set +x
4312024-01-16T00:43:26.811ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322024-01-16T00:43:26.811Z++ zpool list -o name
4332024-01-16T00:43:26.811Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342024-01-16T00:43:26.811Z+ [[ -z '' ]]
4352024-01-16T00:43:26.811Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362024-01-16T00:43:26.835Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372024-01-16T00:43:26.835Z+ set +x
4382024-01-16T00:43:26.835ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392024-01-16T00:43:26.835Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402024-01-16T00:43:26.835Z+ readarray -t ZPOOLS
4412024-01-16T00:43:26.835Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422024-01-16T00:43:26.860Z++ sed 's/[ ",]//g'
4432024-01-16T00:43:26.860Z+ for ZPOOL in "${ZPOOLS[@]}"
4442024-01-16T00:43:26.860Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452024-01-16T00:43:26.861ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462024-01-16T00:43:26.861Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472024-01-16T00:43:26.861Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482024-01-16T00:43:26.861Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492024-01-16T00:43:26.862Z0+0 records in
4502024-01-16T00:43:26.862Z0+0 records out
4512024-01-16T00:43:26.862Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4522024-01-16T00:43:26.862Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532024-01-16T00:43:26.862Z+ set +x
4542024-01-16T00:43:26.862ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552024-01-16T00:43:26.862Z++ zpool list -o name
4562024-01-16T00:43:26.862Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572024-01-16T00:43:26.862Z+ [[ -z '' ]]
4582024-01-16T00:43:26.862Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592024-01-16T00:43:26.892Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602024-01-16T00:43:26.892Z+ set +x
4612024-01-16T00:43:26.917ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622024-01-16T00:43:26.917Z+ for ZPOOL in "${ZPOOLS[@]}"
4632024-01-16T00:43:26.917Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642024-01-16T00:43:26.917ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652024-01-16T00:43:26.917Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662024-01-16T00:43:26.917Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672024-01-16T00:43:26.917Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682024-01-16T00:43:26.917Z0+0 records in
4692024-01-16T00:43:26.917Z0+0 records out
4702024-01-16T00:43:26.917Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4712024-01-16T00:43:26.917Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722024-01-16T00:43:26.917Z+ set +x
4732024-01-16T00:43:26.917ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742024-01-16T00:43:26.917Z++ zpool list -o name
4752024-01-16T00:43:26.917Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762024-01-16T00:43:26.917Z+ [[ -z '' ]]
4772024-01-16T00:43:26.917Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782024-01-16T00:43:26.954Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792024-01-16T00:43:26.954Z+ set +x
4802024-01-16T00:43:26.954ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812024-01-16T00:43:26.954Z+ [[ zone == \z\o\n\e ]]
4822024-01-16T00:43:26.954Z+ ensure_simulated_links igb0
4832024-01-16T00:43:26.954Z+ local PHYSICAL_LINK=igb0
4842024-01-16T00:43:26.954Z+ INDICES=("0" "1")
4852024-01-16T00:43:26.954Z+ for I in "${INDICES[@]}"
4862024-01-16T00:43:26.954Z++ get_simnet_name_if_exists net0
4872024-01-16T00:43:26.954Z++ dladm show-simnet -p -o LINK net0
4882024-01-16T00:43:26.954Z++ echo ''
4892024-01-16T00:43:26.954Z+ [[ -z '' ]]
4902024-01-16T00:43:26.954Z+ dladm create-simnet -t net0
4912024-01-16T00:43:26.980Z+ dladm create-simnet -t sc0_0
4922024-01-16T00:43:26.980Z+ dladm modify-simnet -t -p net0 sc0_0
4932024-01-16T00:43:26.980Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942024-01-16T00:43:26.980Z+ success 'Simnet net0/sc0_0 exists'
4952024-01-16T00:43:26.980Z+ set +x
4962024-01-16T00:43:26.980ZSimnet net0/sc0_0 exists
4972024-01-16T00:43:26.980Z+ for I in "${INDICES[@]}"
4982024-01-16T00:43:26.981Z++ get_simnet_name_if_exists net1
4992024-01-16T00:43:26.981Z++ dladm show-simnet -p -o LINK net1
5002024-01-16T00:43:26.981Z++ echo ''
5012024-01-16T00:43:26.981Z+ [[ -z '' ]]
5022024-01-16T00:43:26.981Z+ dladm create-simnet -t net1
5032024-01-16T00:43:26.981Z+ dladm create-simnet -t sc1_0
5042024-01-16T00:43:26.981Z+ dladm modify-simnet -t -p net1 sc1_0
5052024-01-16T00:43:26.981Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062024-01-16T00:43:27.005Z+ success 'Simnet net1/sc1_0 exists'
5072024-01-16T00:43:27.005Z+ set +x
5082024-01-16T00:43:27.005ZSimnet net1/sc1_0 exists
5092024-01-16T00:43:27.005Z++ get_vnic_name_if_exists sc0_1
5102024-01-16T00:43:27.005Z++ dladm show-vnic -p -o LINK sc0_1
5112024-01-16T00:43:27.005Z++ echo ''
5122024-01-16T00:43:27.005Z+ [[ -z '' ]]
5132024-01-16T00:43:27.005Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142024-01-16T00:43:27.005Z+ [[ -v PROMISC_FILT_OFF ]]
5152024-01-16T00:43:27.005Z+ success 'Vnic sc0_1 exists'
5162024-01-16T00:43:27.005Z+ set +x
5172024-01-16T00:43:27.006ZVnic sc0_1 exists
5182024-01-16T00:43:27.006Z+ warn_if_no_proxy_arp
5192024-01-16T00:43:27.006Z+ [[ -v PXA_START ]]
5202024-01-16T00:43:27.006Z+ [[ -v PXA_END ]]
5212024-01-16T00:43:27.006Z+ ensure_softnpu_zone
5222024-01-16T00:43:27.006Z+ zoneadm list
5232024-01-16T00:43:27.006Z+ grep -q sidecar_softnpu
5242024-01-16T00:43:27.006Z+ [[ -f out/npuzone/npuzone ]]
5252024-01-16T00:43:27.006Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit e3ea4b495ba0a71801ded0776ae4bbd31df57e26 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5262024-01-16T00:43:27.030Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5272024-01-16T00:43:31.612Zdownloading softnpu ...done
5282024-01-16T00:43:31.638Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/libsidecar_lite.so.sha256.txt
5292024-01-16T00:43:37.842Zdownloading asic_program.so ...done
5302024-01-16T00:43:37.866Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/scadm.sha256.txt
5312024-01-16T00:43:43.663Zdownloading scadm ...done
5322024-01-16T00:43:43.687Zcreating port tfportrear0_0 for sc0_0
5332024-01-16T00:43:43.687Zcreating port tfportqsfp0_0 for sc0_1
5342024-01-16T00:43:43.711Zsoftnpu zone setup
5352024-01-16T00:43:43.711Zconfigure zone
5362024-01-16T00:43:43.711Zinstall zone
5372024-01-16T00:43:44.771Zboot zone
5382024-01-16T00:43:47.874Z[sidecar_softnpu] svcs milestone/network
5392024-01-16T00:43:47.901ZSTATE STIME FMRI
5402024-01-16T00:43:47.901Zonline 0:43:45 svc:/milestone/network:default
5412024-01-16T00:43:47.901Z[sidecar_softnpu] /softnpu/npu start
5422024-01-16T00:43:47.926Z
5432024-01-16T00:43:47.950Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442024-01-16T00:43:47.950Z+ GATEWAY_IP=10.151.1.100
5452024-01-16T00:43:47.950ZUsing 10.151.1.100 as gateway ip
5462024-01-16T00:43:47.950Z+ echo 'Using 10.151.1.100 as gateway ip'
5472024-01-16T00:43:47.950Z+ [[ ! -v GATEWAY_MAC ]]
5482024-01-16T00:43:47.950Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492024-01-16T00:43:47.950Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502024-01-16T00:43:47.950Z+++ echo 18:c0:4d:0d:9f:b2
5512024-01-16T00:43:47.950Z+ COUNT=1
5522024-01-16T00:43:47.950Z+ [[ 1 -ne 1 ]]
5532024-01-16T00:43:47.950Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542024-01-16T00:43:47.952ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552024-01-16T00:43:47.952Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562024-01-16T00:43:47.952Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572024-01-16T00:43:47.976Z+ PXA_MAC=a8:e1:de:01:70:1d
5582024-01-16T00:43:47.976Z+ [[ -v PXA_START ]]
5592024-01-16T00:43:47.976Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602024-01-16T00:43:47.977Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612024-01-16T00:43:48.001Z+ z_scadm dump-state
5622024-01-16T00:43:48.001Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632024-01-16T00:43:48.058Zlocal v6:
5642024-01-16T00:43:48.058Zlocal v4:
5652024-01-16T00:43:48.058Zrouter v6:
5662024-01-16T00:43:48.058Zrouter v4_idx:
5672024-01-16T00:43:48.058Zrouter v4_routes:
5682024-01-16T00:43:48.058Zresolver v4:
5692024-01-16T00:43:48.058Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702024-01-16T00:43:48.058Zresolver v6:
5712024-01-16T00:43:48.058Znat v4:
5722024-01-16T00:43:48.058Znat_v6:
5732024-01-16T00:43:48.058Zport_mac:
5742024-01-16T00:43:48.058Zproxy arp:
5752024-01-16T00:43:48.058Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762024-01-16T00:43:48.059Z+ success 'softnpu zone exists'
5772024-01-16T00:43:48.059Z+ set +x
5782024-01-16T00:43:48.059Zsoftnpu zone exists
5792024-01-16T00:43:48.083Z
5802024-01-16T00:43:48.083Zreal 21.724480055
5812024-01-16T00:43:48.084Zuser 0.703941971
5822024-01-16T00:43:48.085Zsys 1.595245805
5832024-01-16T00:43:48.085Ztrap 0.000079082
5842024-01-16T00:43:48.085Ztflt 0.000222008
5852024-01-16T00:43:48.085Zdflt 0.001208710
5862024-01-16T00:43:48.085Zkflt 0.000000000
5872024-01-16T00:43:48.085Zlock 16:42.540988577
5882024-01-16T00:43:48.085Zslp 1:04.949409027
5892024-01-16T00:43:48.085Zlat 0.055608524
5902024-01-16T00:43:48.085Zstop 0.360992143
5912024-01-16T00:43:48.085Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922024-01-16T00:43:48.085Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932024-01-16T00:43:48.085Z+ SILO_NAME=recovery
5942024-01-16T00:43:48.085Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952024-01-16T00:43:48.085Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962024-01-16T00:43:48.085Z+ sed -i~ '
5972024-01-16T00:43:48.085Z /^external_dns_ips/c\
5982024-01-16T00:43:48.085Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992024-01-16T00:43:48.085Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002024-01-16T00:43:48.085Z /^first/c\
6012024-01-16T00:43:48.085Zfirst = "10.151.1.164"
6022024-01-16T00:43:48.085Z /^last/c\
6032024-01-16T00:43:48.085Zlast = "10.151.1.173"
6042024-01-16T00:43:48.086Z }
6052024-01-16T00:43:48.086Z /^\[rack_network_config/,/^$/ {
6062024-01-16T00:43:48.086Z /^infra_ip_first/c\
6072024-01-16T00:43:48.086Zinfra_ip_first = "10.151.1.174"
6082024-01-16T00:43:48.086Z /^infra_ip_last/c\
6092024-01-16T00:43:48.086Zinfra_ip_last = "10.151.1.174"
6102024-01-16T00:43:48.088Z }
6112024-01-16T00:43:48.088Z /^\[\[rack_network_config.ports/,/^$/ {
6122024-01-16T00:43:48.088Z /^routes/c\
6132024-01-16T00:43:48.088Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6142024-01-16T00:43:48.089Z /^addresses/c\
6152024-01-16T00:43:48.089Zaddresses = \["10.151.1.174/32"\]
6162024-01-16T00:43:48.089Z }
6172024-01-16T00:43:48.089Z' pkg/config-rss.toml
6182024-01-16T00:43:48.089Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192024-01-16T00:43:48.089Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202024-01-16T00:43:48.089Z+++ pkg/config-rss.toml Tue Jan 16 00:43:47 2024
6212024-01-16T00:43:48.089Z@@ -41,7 +41,7 @@
6222024-01-16T00:43:48.089Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232024-01-16T00:43:48.089Z # must be contained in one of the "internal services" IP Pool ranges listed
6242024-01-16T00:43:48.089Z # below.
6252024-01-16T00:43:48.089Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262024-01-16T00:43:48.089Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6272024-01-16T00:43:48.089Z
6282024-01-16T00:43:48.089Z # Initial TLS certificates for the external API
6292024-01-16T00:43:48.089Z #
6302024-01-16T00:43:48.089Z@@ -77,8 +77,8 @@
6312024-01-16T00:43:48.089Z #
6322024-01-16T00:43:48.089Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332024-01-16T00:43:48.089Z [[internal_services_ip_pool_ranges]]
6342024-01-16T00:43:48.089Z-first = "192.168.1.20"
6352024-01-16T00:43:48.089Z-last = "192.168.1.29"
6362024-01-16T00:43:48.089Z+first = "10.151.1.164"
6372024-01-16T00:43:48.090Z+last = "10.151.1.173"
6382024-01-16T00:43:48.090Z
6392024-01-16T00:43:48.090Z # TODO - this configuration is subject to change going forward. Ultimately these
6402024-01-16T00:43:48.090Z # parameters should be provided to the control plane via wicket, but we need to
6412024-01-16T00:43:48.090Z@@ -92,8 +92,8 @@
6422024-01-16T00:43:48.090Z # A range of IP addresses used by Boundary Services on the external network. In
6432024-01-16T00:43:48.090Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442024-01-16T00:43:48.090Z # With softnpu, only one address is used.
6452024-01-16T00:43:48.090Z-infra_ip_first = "192.168.1.30"
6462024-01-16T00:43:48.090Z-infra_ip_last = "192.168.1.30"
6472024-01-16T00:43:48.090Z+infra_ip_first = "10.151.1.174"
6482024-01-16T00:43:48.090Z+infra_ip_last = "10.151.1.174"
6492024-01-16T00:43:48.090Z
6502024-01-16T00:43:48.090Z # Configurations for BGP routers to run on the scrimlets.
6512024-01-16T00:43:48.090Z bgp = []
6522024-01-16T00:43:48.090Z@@ -101,9 +101,9 @@
6532024-01-16T00:43:48.090Z # You can configure multiple uplinks by repeating the following stanza
6542024-01-16T00:43:48.090Z [[rack_network_config.ports]]
6552024-01-16T00:43:48.090Z # Routes associated with this port.
6562024-01-16T00:43:48.090Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572024-01-16T00:43:48.090Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6582024-01-16T00:43:48.091Z # Addresses associated with this port.
6592024-01-16T00:43:48.091Z-addresses = ["192.168.1.30/32"]
6602024-01-16T00:43:48.091Z+addresses = ["10.151.1.174/32"]
6612024-01-16T00:43:48.091Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622024-01-16T00:43:48.091Z port = "qsfp0"
6632024-01-16T00:43:48.091Z # The speed of this port.
6642024-01-16T00:43:48.091Z+ true
6652024-01-16T00:43:48.091Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662024-01-16T00:43:48.091Za pkg/config-rss.toml 7K
6672024-01-16T00:43:48.091Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682024-01-16T00:43:48.091Z+ TLS_NAME=recovery.sys.oxide.test
6692024-01-16T00:43:48.091Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702024-01-16T00:43:49.730Z......+..............+...+.+..............+...+..........+...+..............+.+..+....+...+..+.+.....+....+.....+......+...+.......+.........+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.............+......+.+...+..+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+...+.........+.....+......+.............+......+...+.....+............+....+..+............+.......+.......................+...+............................+.........+..................+.....+....+...........+.............+.........+......+..+............+.........................+...+........+......+....+......+.....+.........+....+.....................+..............+.............+..+.+...........+...+.+...........+......+...+....+............+...........+..........+..+.......+...+......+.....+..........+......+...+......+.....+......+.+........+.........+...+...........................+...+...................+.........+..+....+...+..+...+.......+......+........+...............+...............+....+....................+..........+...+...+.....+.........+............+....+..+....+...+............+.....+......+...+......+......+...............+.......+...+..+......+....+...+..............+...+.+.........+...............+..+....+.....+......+.+..+.......+...+...........+.........+...+....+...+...+........+................+........+.............+.....+.......+..+....+...............+..+..........+.........+.....+..........+........+.+..+....+......+..+................+..+.........+.+..+..........+.....+...+...+.........+.......+...+...+.........+..................+..+......+.......+..+.............+..+.+.....+.............+..+....+........+......+......+.........+.+.....+....+..+...+......+.+..................+.....+.........+...+....+...........+.......+.....+......+.........+.+......+...............+.....+...+.......+...+...+........+..........+.........+......+.........+...........+........................+.......+...+...........+...............+......+.+............+...........+.+...........+....+........+.......+.....+.........+.+...+..+.+..+...+...+......+......+.......+......+..............+......+.+.......................+.+.....+..........+............+..+...+......................+...+...+.................+...+.+...+.................+................+..+.......+.....+.........+............+.........+....+..+.......+..+....+..............+.+.....+.......+..+.+......+..................+..+......+....+.........+.....+.......+..+...............+..........+......+..+.......+..+.+...+......+......+..............+..........+......+...+...........+....+...........+....+.........+............+...+...+...............+.....+...............+.............+.....+......+..........+...+.....+...+............................+.....+.........+..........+..+...+..........+......+.....+......+...+.+...........+...+......+.......+.....+...+....+..+................+...+..+...+.+........+...+..........+...+............+..+.......+.....+...+................+.....+......+...+.+.....+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712024-01-16T00:43:50.575Z......+.....+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+..+.+..+...+.+.....+.+........+............+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*............+...+....+.......................+....+...+...+.....+...+.......+........+......+...+.+.........+.....+.........................+..+.+..............+....+.................+...+......+...+..................+.......+...+........+...............+....+..+...................+.....+.+..............................+.....+.+...+..+....+........+......+..........+.....+......+..........+........+.+..+...+.+........+...+...+.+......+........+..........+.........+..+.........+....+.................+....+..+....+........+.+......+..............+.+...+......+............+.....+...+...................+..+.+........+......+....+..+.........+....+.....+.........+............+.+..+.+............+..+.+...........+.+..+...+...+.+...+..+.........+.......+..+......................+...+......+..+.........+.........+.+...........+....+...+..+.+.......................+....+..+.+.....+....+..........................+...+....+......+................................+..........+........+..........+......+.....+..................+...+..........+..+......+......+.........+.+........+...+..........+........+......+......+...........................+.......+......+.................+.+..+....+...+...+.....+..........+............+..+...+................+.....+.+........+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722024-01-16T00:43:50.610Z-----
6732024-01-16T00:43:50.610Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742024-01-16T00:43:50.610Za pkg/initial-tls-cert.pem 2K
6752024-01-16T00:43:50.610Za pkg/initial-tls-key.pem 4K
6762024-01-16T00:43:50.611Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772024-01-16T00:43:50.611Z+ rmdir pkg
6782024-01-16T00:43:50.611Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792024-01-16T00:43:50.611Z+ pfexec mkdir -p /usr/oxide
6802024-01-16T00:43:50.611Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6812024-01-16T00:43:51.312Z+ pfexec chmod +x /usr/oxide/oxide
6822024-01-16T00:43:51.336Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6832024-01-16T00:44:09.617Z+ OMICRON_NO_UNINSTALL=1
6842024-01-16T00:44:09.617Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6852024-01-16T00:44:09.642ZJan 16 00:44:09.135 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6862024-01-16T00:44:09.642ZJan 16 00:44:09.137 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6872024-01-16T00:44:09.642ZJan 16 00:44:09.137 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6882024-01-16T00:44:09.642ZJan 16 00:44:09.137 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6892024-01-16T00:44:09.642ZJan 16 00:44:09.137 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6902024-01-16T00:44:09.642ZJan 16 00:44:09.137 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6912024-01-16T00:44:09.642ZJan 16 00:44:09.137 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6922024-01-16T00:44:09.642ZJan 16 00:44:09.137 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6932024-01-16T00:44:09.642ZJan 16 00:44:09.137 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6942024-01-16T00:44:09.642ZJan 16 00:44:09.137 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6952024-01-16T00:44:09.642ZJan 16 00:44:09.137 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6962024-01-16T00:44:09.642ZJan 16 00:44:09.137 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6972024-01-16T00:44:09.643ZJan 16 00:44:09.137 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6982024-01-16T00:44:09.643ZJan 16 00:44:09.137 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6992024-01-16T00:44:09.643ZJan 16 00:44:09.138 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7002024-01-16T00:44:09.643ZJan 16 00:44:09.138 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7012024-01-16T00:44:09.643ZJan 16 00:44:09.138 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7022024-01-16T00:44:09.880ZJan 16 00:44:09.381 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7032024-01-16T00:44:09.918ZJan 16 00:44:09.419 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7042024-01-16T00:44:09.982ZJan 16 00:44:09.483 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7052024-01-16T00:44:10.110Z
7062024-01-16T00:44:10.136Zreal 0.488057041
7072024-01-16T00:44:10.136Zuser 0.101005656
7082024-01-16T00:44:10.136Zsys 1.952953662
7092024-01-16T00:44:10.136Ztrap 0.000015879
7102024-01-16T00:44:10.136Ztflt 0.000045740
7112024-01-16T00:44:10.136Zdflt 0.000577053
7122024-01-16T00:44:10.136Zkflt 0.000000000
7132024-01-16T00:44:10.136Zlock 43.708896267
7142024-01-16T00:44:10.136Zslp 0.685741763
7152024-01-16T00:44:10.137Zlat 0.003210355
7162024-01-16T00:44:10.137Zstop 0.209133468
7172024-01-16T00:44:10.137Z+ retry=0
7182024-01-16T00:44:10.137Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7192024-01-16T00:44:10.137Z+ [[ 0 -gt 30 ]]
7202024-01-16T00:44:10.137Z+ sleep 1
7212024-01-16T00:44:11.126Z+ retry=1
7222024-01-16T00:44:11.126Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232024-01-16T00:44:11.158Z+ [[ 1 -gt 30 ]]
7242024-01-16T00:44:11.158Z+ sleep 1
7252024-01-16T00:44:12.141Z+ retry=2
7262024-01-16T00:44:12.141Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272024-01-16T00:44:12.166Z+ [[ 2 -gt 30 ]]
7282024-01-16T00:44:12.166Z+ sleep 1
7292024-01-16T00:44:13.159Z+ retry=3
7302024-01-16T00:44:13.159Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312024-01-16T00:44:13.187Z+ [[ 3 -gt 30 ]]
7322024-01-16T00:44:13.187Z+ sleep 1
7332024-01-16T00:44:14.169Z+ retry=4
7342024-01-16T00:44:14.169Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352024-01-16T00:44:39.354Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7362024-01-16T00:44:39.382Zlocal v6:
7372024-01-16T00:44:39.382Zfe80::f47d:e4ff:fe63:391e
7382024-01-16T00:44:39.382Zfd00:99::1
7392024-01-16T00:44:39.382Zlocal v4:
7402024-01-16T00:44:39.382Z10.151.1.174
7412024-01-16T00:44:39.382Zrouter v6:
7422024-01-16T00:44:39.382Zfd00:1122:3344:2::/64 -> fe80::5c74:83ff:fe86:c364 (1)
7432024-01-16T00:44:39.382Zfd00:1122:3344:3::/64 -> fe80::5c74:83ff:fe86:c364 (1)
7442024-01-16T00:44:39.382Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::5c74:83ff:fe86:c364 (1)
7452024-01-16T00:44:39.382Zfd00:1122:3344:101::/64 -> fe80::5c74:83ff:fe86:c364 (1)
7462024-01-16T00:44:39.382Zfd00:1122:3344:1::/64 -> fe80::5c74:83ff:fe86:c364 (1)
7472024-01-16T00:44:39.383Zrouter v4_idx:
7482024-01-16T00:44:39.383Z0.0.0.0/0 -> 0
7492024-01-16T00:44:39.383Zrouter v4_routes:
7502024-01-16T00:44:39.383Z0 -> 10.151.1.100 (2)
7512024-01-16T00:44:39.409Zresolver v4:
7522024-01-16T00:44:39.409Z10.151.1.174 -> 7a:2b:1b:65:76:d0
7532024-01-16T00:44:39.409Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7542024-01-16T00:44:39.409Zresolver v6:
7552024-01-16T00:44:39.409Zfe80::f47d:e4ff:fe63:391e -> f6:7d:e4:63:39:1e
7562024-01-16T00:44:39.409Zfe80::5c74:83ff:fe86:c364 -> 5e:74:83:86:c3:64
7572024-01-16T00:44:39.409Znat v4:
7582024-01-16T00:44:39.410Znat_v6:
7592024-01-16T00:44:39.410Zport_mac:
7602024-01-16T00:44:39.410Z1: a8:40:25:51:e9:ec
7612024-01-16T00:44:39.410Z1000: a8:40:25:51:e9:eb
7622024-01-16T00:44:39.410Zproxy arp:
7632024-01-16T00:44:39.410Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7642024-01-16T00:44:39.410Z+ retry=0
7652024-01-16T00:44:39.410Z+++ zoneadm list -n
7662024-01-16T00:44:39.410Z+++ grep oxz_ntp
7672024-01-16T00:44:39.410Z++ pfexec svcs -z -Hostate oxide/ntp
7682024-01-16T00:44:39.410Zsvcs: zone '-Hostate': entity not found
7692024-01-16T00:44:39.410Z++ true
7702024-01-16T00:44:39.410Z+ [[ '' != online ]]
7712024-01-16T00:44:39.410Z+ [[ 0 -gt 60 ]]
7722024-01-16T00:44:39.410Z+ sleep 1
7732024-01-16T00:44:40.411Z+ retry=1
7742024-01-16T00:44:40.443Z+++ zoneadm list -n
7752024-01-16T00:44:40.443Z+++ grep oxz_ntp
7762024-01-16T00:44:40.443Z++ pfexec svcs -z -Hostate oxide/ntp
7772024-01-16T00:44:40.443Zsvcs: zone '-Hostate': entity not found
7782024-01-16T00:44:40.444Z++ true
7792024-01-16T00:44:40.444Z+ [[ '' != online ]]
7802024-01-16T00:44:40.444Z+ [[ 1 -gt 60 ]]
7812024-01-16T00:44:40.444Z+ sleep 1
7822024-01-16T00:44:41.429Z+ retry=2
7832024-01-16T00:44:41.454Z+++ zoneadm list -n
7842024-01-16T00:44:41.454Z+++ grep oxz_ntp
7852024-01-16T00:44:41.454Z++ pfexec svcs -z oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be -Hostate oxide/ntp
7862024-01-16T00:44:41.488Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7872024-01-16T00:44:41.513Z++ true
7882024-01-16T00:44:41.513Z+ [[ '' != online ]]
7892024-01-16T00:44:41.513Z+ [[ 2 -gt 60 ]]
7902024-01-16T00:44:41.513Z+ sleep 1
7912024-01-16T00:44:42.493Z+ retry=3
7922024-01-16T00:44:42.517Z+++ zoneadm list -n
7932024-01-16T00:44:42.517Z+++ grep oxz_ntp
7942024-01-16T00:44:42.518Z++ pfexec svcs -z oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be -Hostate oxide/ntp
7952024-01-16T00:44:42.543Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7962024-01-16T00:44:42.543Z++ true
7972024-01-16T00:44:42.543Z+ [[ '' != online ]]
7982024-01-16T00:44:42.543Z+ [[ 3 -gt 60 ]]
7992024-01-16T00:44:42.543Z+ sleep 1
8002024-01-16T00:44:43.523Z+ retry=4
8012024-01-16T00:44:43.549Z+++ zoneadm list -n
8022024-01-16T00:44:43.549Z+++ grep oxz_ntp
8032024-01-16T00:44:43.550Z++ pfexec svcs -z oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be -Hostate oxide/ntp
8042024-01-16T00:44:43.578Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8052024-01-16T00:44:43.578Z++ true
8062024-01-16T00:44:43.578Z+ [[ '' != online ]]
8072024-01-16T00:44:43.578Z+ [[ 4 -gt 60 ]]
8082024-01-16T00:44:43.578Z+ sleep 1
8092024-01-16T00:44:44.561Z+ retry=5
8102024-01-16T00:44:44.592Z+++ zoneadm list -n
8112024-01-16T00:44:44.592Z+++ grep oxz_ntp
8122024-01-16T00:44:44.592Z++ pfexec svcs -z oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be -Hostate oxide/ntp
8132024-01-16T00:44:44.592Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8142024-01-16T00:44:44.593Z++ true
8152024-01-16T00:44:44.593Z+ [[ '' != online ]]
8162024-01-16T00:44:44.593Z+ [[ 5 -gt 60 ]]
8172024-01-16T00:44:44.593Z+ sleep 1
8182024-01-16T00:44:45.585Z+ retry=6
8192024-01-16T00:44:45.612Z+++ zoneadm list -n
8202024-01-16T00:44:45.612Z+++ grep oxz_ntp
8212024-01-16T00:44:45.612Z++ pfexec svcs -z oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be -Hostate oxide/ntp
8222024-01-16T00:44:45.613Z+ [[ disabled != online ]]
8232024-01-16T00:44:45.613Z+ [[ 6 -gt 60 ]]
8242024-01-16T00:44:45.613Z+ sleep 1
8252024-01-16T00:44:46.609Z+ retry=7
8262024-01-16T00:44:46.634Z+++ zoneadm list -n
8272024-01-16T00:44:46.634Z+++ grep oxz_ntp
8282024-01-16T00:44:46.634Z++ pfexec svcs -z oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be -Hostate oxide/ntp
8292024-01-16T00:44:46.634Z+ [[ disabled != online ]]
8302024-01-16T00:44:46.634Z+ [[ 7 -gt 60 ]]
8312024-01-16T00:44:46.634Z+ sleep 1
8322024-01-16T00:44:47.631Z+ retry=8
8332024-01-16T00:44:47.656Z+++ zoneadm list -n
8342024-01-16T00:44:47.656Z+++ grep oxz_ntp
8352024-01-16T00:44:47.656Z++ pfexec svcs -z oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be -Hostate oxide/ntp
8362024-01-16T00:44:47.656Z+ [[ disabled != online ]]
8372024-01-16T00:44:47.656Z+ [[ 8 -gt 60 ]]
8382024-01-16T00:44:47.656Z+ sleep 1
8392024-01-16T00:44:48.652Z+ retry=9
8402024-01-16T00:44:48.677Z+++ zoneadm list -n
8412024-01-16T00:44:48.678Z+++ grep oxz_ntp
8422024-01-16T00:44:48.678Z++ pfexec svcs -z oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be -Hostate oxide/ntp
8432024-01-16T00:44:48.678Z+ [[ disabled != online ]]
8442024-01-16T00:44:48.678Z+ [[ 9 -gt 60 ]]
8452024-01-16T00:44:48.678Z+ sleep 1
8462024-01-16T00:44:49.674Z+ retry=10
8472024-01-16T00:44:49.704Z+++ zoneadm list -n
8482024-01-16T00:44:49.704Z+++ grep oxz_ntp
8492024-01-16T00:44:49.704Z++ pfexec svcs -z oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be -Hostate oxide/ntp
8502024-01-16T00:44:49.705Z+ [[ disabled != online ]]
8512024-01-16T00:44:49.705Z+ [[ 10 -gt 60 ]]
8522024-01-16T00:44:49.705Z+ sleep 1
8532024-01-16T00:44:52.052Z+ retry=11
8542024-01-16T00:44:52.083Z+++ zoneadm list -n
8552024-01-16T00:44:52.083Z+++ grep oxz_ntp
8562024-01-16T00:44:52.083Z++ pfexec svcs -z oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be -Hostate oxide/ntp
8572024-01-16T00:44:52.083Z+ [[ disabled != online ]]
8582024-01-16T00:44:52.083Z+ [[ 11 -gt 60 ]]
8592024-01-16T00:44:52.083Z+ sleep 1
8602024-01-16T00:44:52.185Z+ retry=12
8612024-01-16T00:44:52.185Z+++ zoneadm list -n
8622024-01-16T00:44:52.185Z+++ grep oxz_ntp
8632024-01-16T00:44:52.185Z++ pfexec svcs -z oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be -Hostate oxide/ntp
8642024-01-16T00:44:52.185Z+ [[ disabled != online ]]
8652024-01-16T00:44:52.185Z+ [[ 12 -gt 60 ]]
8662024-01-16T00:44:52.185Z+ sleep 1
8672024-01-16T00:44:52.742Z+ retry=13
8682024-01-16T00:44:52.777Z+++ zoneadm list -n
8692024-01-16T00:44:52.777Z+++ grep oxz_ntp
8702024-01-16T00:44:52.778Z++ pfexec svcs -z oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be -Hostate oxide/ntp
8712024-01-16T00:44:52.778Z+ [[ disabled != online ]]
8722024-01-16T00:44:52.778Z+ [[ 13 -gt 60 ]]
8732024-01-16T00:44:52.778Z+ sleep 1
8742024-01-16T00:44:53.765Z+ retry=14
8752024-01-16T00:44:53.857Z+++ zoneadm list -n
8762024-01-16T00:44:53.857Z+++ grep oxz_ntp
8772024-01-16T00:44:53.857Z++ pfexec svcs -z oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be -Hostate oxide/ntp
8782024-01-16T00:44:53.857Z+ [[ disabled != online ]]
8792024-01-16T00:44:53.858Z+ [[ 14 -gt 60 ]]
8802024-01-16T00:44:53.858Z+ sleep 1
 
8812024-01-16T00:44:54.564Zbuskin console login: Jan 16 00:44:53 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8822024-01-16T00:44:54.788Z+ retry=15
8832024-01-16T00:44:54.852Z+++ zoneadm list -n
8842024-01-16T00:44:54.852Z+++ grep oxz_ntp
8852024-01-16T00:44:54.852Z++ pfexec svcs -z oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be -Hostate oxide/ntp
8862024-01-16T00:44:54.852Z+ [[ online != online ]]
8872024-01-16T00:44:54.852Z+ echo 'Waited for chrony: 15s'
8882024-01-16T00:44:54.853ZWaited for chrony: 15s
8892024-01-16T00:44:54.853Z+ export RUST_BACKTRACE=1
8902024-01-16T00:44:54.853Z+ RUST_BACKTRACE=1
8912024-01-16T00:44:54.853Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8922024-01-16T00:44:54.853Z++ ./tests/bootstrap
8932024-01-16T00:44:54.853ZUsing external DNS server at 10.151.1.164:53
8942024-01-16T00:44:54.853Z2024-01-16 00:44:54.347647057 UTC: attempting to log into API
8952024-01-16T00:45:13.898Z2024-01-16 00:45:09.769669958 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8962024-01-16T00:45:13.898Z2024-01-16 00:45:10.770503031 UTC: attempting to log into API
8972024-01-16T00:45:25.925Z2024-01-16 00:45:25.803412322 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8982024-01-16T00:45:26.920Z2024-01-16 00:45:26.805326955 UTC: attempting to log into API
8992024-01-16T00:45:41.953Z2024-01-16 00:45:41.838260496 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9002024-01-16T00:45:42.958Z2024-01-16 00:45:42.839266440 UTC: attempting to log into API
9012024-01-16T00:45:57.986Z2024-01-16 00:45:57.871479745 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9022024-01-16T00:45:58.988Z2024-01-16 00:45:58.873174906 UTC: attempting to log into API
9032024-01-16T00:46:14.026Z2024-01-16 00:46:13.909177971 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9042024-01-16T00:46:15.045Z2024-01-16 00:46:14.912973335 UTC: attempting to log into API
9052024-01-16T00:46:30.055Z2024-01-16 00:46:29.985403700 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9062024-01-16T00:46:31.056Z2024-01-16 00:46:30.989209745 UTC: attempting to log into API
9072024-01-16T00:46:46.089Z2024-01-16 00:46:46.061694688 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9082024-01-16T00:46:47.089Z2024-01-16 00:46:47.065359627 UTC: attempting to log into API
9092024-01-16T00:47:02.122Z2024-01-16 00:47:02.108102892 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9102024-01-16T00:47:03.123Z2024-01-16 00:47:03.109224447 UTC: attempting to log into API
9112024-01-16T00:47:18.155Z2024-01-16 00:47:18.141387731 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9122024-01-16T00:47:19.160Z2024-01-16 00:47:19.142429215 UTC: attempting to log into API
9132024-01-16T00:47:34.514Z2024-01-16 00:47:34.468517959 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9142024-01-16T00:47:35.482Z2024-01-16 00:47:35.470237149 UTC: attempting to log into API
9152024-01-16T00:47:50.527Z2024-01-16 00:47:50.515008515 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9162024-01-16T00:47:51.531Z2024-01-16 00:47:51.516639765 UTC: attempting to log into API
9172024-01-16T00:48:06.569Z2024-01-16 00:48:06.557805956 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9182024-01-16T00:48:06.881ZJan 16 00:48:06 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9192024-01-16T00:48:07.572Z2024-01-16 00:48:07.558960041 UTC: attempting to log into API
 
9202024-01-16T00:48:12.908ZJan 16 00:48:12 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9212024-01-16T00:48:20.070ZJan 16 00:48:19 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9222024-01-16T00:48:22.602Z2024-01-16 00:48:22.591533305 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9232024-01-16T00:48:23.625Z2024-01-16 00:48:23.592596780 UTC: attempting to log into API
 
9242024-01-16T00:48:25.819ZJan 16 00:48:25 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9252024-01-16T00:48:33.927ZJan 16 00:48:33 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9262024-01-16T00:48:38.635Z2024-01-16 00:48:38.624597548 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9272024-01-16T00:48:39.639Z2024-01-16 00:48:39.625723053 UTC: attempting to log into API
9282024-01-16T00:48:54.669Z2024-01-16 00:48:54.657854553 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9292024-01-16T00:48:55.670Z2024-01-16 00:48:55.658879237 UTC: attempting to log into API
9302024-01-16T00:49:01.463Z2024-01-16 00:49:01.453027948 UTC: login succeeded
9312024-01-16T00:49:01.496Zwaiting for nexus to come up...
9322024-01-16T00:49:01.569Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9332024-01-16T00:49:01.751Zsetting up quotas...
9342024-01-16T00:49:01.792Zensuring datasets are ready...
9352024-01-16T00:49:04.364Zlet's roll.
9362024-01-16T00:49:04.396Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
9372024-01-16T00:49:04.397ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
9382024-01-16T00:49:04.397ZOXIDE_TOKEN=oxide-token-7d9aa85ea2b820e72ddb9bba31b528656fbe0c0b'
9392024-01-16T00:49:04.397Z++ OXIDE_HOST=https://recovery.sys.oxide.test
9402024-01-16T00:49:04.397Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
9412024-01-16T00:49:04.397Z++ OXIDE_TOKEN=oxide-token-7d9aa85ea2b820e72ddb9bba31b528656fbe0c0b
9422024-01-16T00:49:04.397Z+ export OXIDE_HOST OXIDE_TOKEN
9432024-01-16T00:49:04.397Z+ retry=0
9442024-01-16T00:49:04.398Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9452024-01-16T00:49:04.466Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
9462024-01-16T00:49:04.745Z{"status":"ok"}success
9472024-01-16T00:49:04.746ZProject {
9482024-01-16T00:49:04.746Z description: "some images",
9492024-01-16T00:49:04.746Z id: b72b10a3-7c9e-4356-9b4f-4e6ab142d4a5,
9502024-01-16T00:49:04.746Z name: Name(
9512024-01-16T00:49:04.746Z "images",
9522024-01-16T00:49:04.747Z ),
9532024-01-16T00:49:04.747Z time_created: 2024-01-16T00:49:04.522773Z,
9542024-01-16T00:49:04.747Z time_modified: 2024-01-16T00:49:04.522773Z,
9552024-01-16T00:49:04.747Z}
9562024-01-16T00:49:04.783Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
9572024-01-16T00:54:29.858Zdone!
9582024-01-16T00:54:29.960Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
9592024-01-16T00:54:30.089Zsuccess
9602024-01-16T00:54:30.089ZImage {
9612024-01-16T00:54:30.090Z block_size: ByteCount(
9622024-01-16T00:54:30.090Z 512,
9632024-01-16T00:54:30.090Z ),
9642024-01-16T00:54:30.117Z description: "debian 11 original base image",
9652024-01-16T00:54:30.117Z digest: None,
9662024-01-16T00:54:30.117Z id: 0413c25b-e509-4e23-a888-45e53d5f29a6,
9672024-01-16T00:54:30.118Z name: Name(
9682024-01-16T00:54:30.118Z "debian11",
9692024-01-16T00:54:30.118Z ),
9702024-01-16T00:54:30.118Z os: "debian",
9712024-01-16T00:54:30.118Z project_id: None,
9722024-01-16T00:54:30.118Z size: ByteCount(
9732024-01-16T00:54:30.119Z 2147483648,
9742024-01-16T00:54:30.120Z ),
9752024-01-16T00:54:30.120Z time_created: 2024-01-16T00:54:29.819321Z,
9762024-01-16T00:54:30.120Z time_modified: 2024-01-16T00:54:30.057249Z,
9772024-01-16T00:54:30.120Z url: None,
9782024-01-16T00:54:30.120Z version: "11",
9792024-01-16T00:54:30.120Z}
9802024-01-16T00:54:30.120Z+ rm ./tests/bootstrap
9812024-01-16T00:54:30.146Z+ for test_bin in tests/*
9822024-01-16T00:54:30.146Z+ ./tests/bootstrap-29cdd1ea464df3b1
9832024-01-16T00:54:30.171Z
9842024-01-16T00:54:30.171Zrunning 0 tests
9852024-01-16T00:54:30.172Z
9862024-01-16T00:54:30.172Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9872024-01-16T00:54:30.172Z
9882024-01-16T00:54:30.172Z+ for test_bin in tests/*
9892024-01-16T00:54:30.172Z+ ./tests/end_to_end_tests-bae04a45dbbda4ab
9902024-01-16T00:54:30.261Z
9912024-01-16T00:54:30.262Zrunning 2 tests
9922024-01-16T00:54:30.497Ztest no_spoof::no_spoof_here ... ok
9932024-01-16T00:55:03.760Ztest instance_launch::instance_launch ... ok
9942024-01-16T00:55:03.760Z
9952024-01-16T00:55:03.760Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 33.50s
9962024-01-16T00:55:03.760Z
9972024-01-16T00:55:03.785Z+ _exit_trap
9982024-01-16T00:55:03.785Z+ local status=0
9992024-01-16T00:55:03.785Z+ [[ 0 -eq 0 ]]
10002024-01-16T00:55:03.785Z+ exit 0
10012024-01-16T00:55:03.787Zprocess exited: duration 705570 ms, exit code 0
 
10022024-01-16T00:55:03.882Zfound 66 output files
10032024-01-16T00:55:03.935Zuploading: /var/svc/log/oxide-sled-agent:default.log (1858831 bytes)
10042024-01-16T00:55:07.046Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10052024-01-16T00:55:07.046Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_83a9976a-0bc9-4f66-a825-dbdbe89ba294/root/var/svc/log/oxide-cockroachdb:default.log (4406 bytes)
10062024-01-16T00:55:08.149Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_83a9976a-0bc9-4f66-a825-dbdbe89ba294/root/var/svc/log/oxide-cockroachdb:default.log
10072024-01-16T00:55:08.149Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_83a9976a-0bc9-4f66-a825-dbdbe89ba294/root/var/svc/log/oxide-zone-network-setup:default.log (1711 bytes)
10082024-01-16T00:55:09.247Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_83a9976a-0bc9-4f66-a825-dbdbe89ba294/root/var/svc/log/oxide-zone-network-setup:default.log
10092024-01-16T00:55:09.247Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1978b41-6627-4ea5-8306-d187357405ba/root/var/svc/log/oxide-crucible-agent:default.log (33296 bytes)
10102024-01-16T00:55:10.821Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1978b41-6627-4ea5-8306-d187357405ba/root/var/svc/log/oxide-crucible-agent:default.log
10112024-01-16T00:55:10.821Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1978b41-6627-4ea5-8306-d187357405ba/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10122024-01-16T00:55:11.928Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1978b41-6627-4ea5-8306-d187357405ba/root/var/svc/log/oxide-crucible-downstairs:default.log
10132024-01-16T00:55:11.928Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1978b41-6627-4ea5-8306-d187357405ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-80fa23be-3f58-474a-9aa2-39f123a27443.log (5621 bytes)
10142024-01-16T00:55:13.032Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1978b41-6627-4ea5-8306-d187357405ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-80fa23be-3f58-474a-9aa2-39f123a27443.log
10152024-01-16T00:55:13.032Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1978b41-6627-4ea5-8306-d187357405ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-b46b5b5c-cab4-4b7e-b69c-af6c1d6074b4.log (3050 bytes)
10162024-01-16T00:55:14.128Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1978b41-6627-4ea5-8306-d187357405ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-b46b5b5c-cab4-4b7e-b69c-af6c1d6074b4.log
10172024-01-16T00:55:14.129Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1978b41-6627-4ea5-8306-d187357405ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-e6c57224-24fd-4166-b4f5-b512afb221d7.log (2917 bytes)
10182024-01-16T00:55:15.222Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1978b41-6627-4ea5-8306-d187357405ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-e6c57224-24fd-4166-b4f5-b512afb221d7.log
10192024-01-16T00:55:15.222Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1978b41-6627-4ea5-8306-d187357405ba/root/var/svc/log/oxide-crucible-downstairs:snapshot-80fa23be-3f58-474a-9aa2-39f123a27443-25678b68-406e-4fe0-842e-cd8ae4bb49f5.log (5134 bytes)
10202024-01-16T00:55:16.318Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1978b41-6627-4ea5-8306-d187357405ba/root/var/svc/log/oxide-crucible-downstairs:snapshot-80fa23be-3f58-474a-9aa2-39f123a27443-25678b68-406e-4fe0-842e-cd8ae4bb49f5.log
10212024-01-16T00:55:16.319Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ec64e9d0-0a28-424c-8a11-866f99eec730/root/var/svc/log/oxide-crucible-agent:default.log (31150 bytes)
10222024-01-16T00:55:17.521Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ec64e9d0-0a28-424c-8a11-866f99eec730/root/var/svc/log/oxide-crucible-agent:default.log
10232024-01-16T00:55:17.521Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ec64e9d0-0a28-424c-8a11-866f99eec730/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10242024-01-16T00:55:18.654Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ec64e9d0-0a28-424c-8a11-866f99eec730/root/var/svc/log/oxide-crucible-downstairs:default.log
10252024-01-16T00:55:18.654Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ec64e9d0-0a28-424c-8a11-866f99eec730/root/var/svc/log/oxide-crucible-downstairs:downstairs-291d6d65-dd89-4823-8704-68021a70c574.log (3051 bytes)
10262024-01-16T00:55:19.762Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ec64e9d0-0a28-424c-8a11-866f99eec730/root/var/svc/log/oxide-crucible-downstairs:downstairs-291d6d65-dd89-4823-8704-68021a70c574.log
10272024-01-16T00:55:19.763Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ec64e9d0-0a28-424c-8a11-866f99eec730/root/var/svc/log/oxide-crucible-downstairs:downstairs-5782779e-8925-425d-88b0-41222753f71e.log (2920 bytes)
10282024-01-16T00:55:20.859Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ec64e9d0-0a28-424c-8a11-866f99eec730/root/var/svc/log/oxide-crucible-downstairs:downstairs-5782779e-8925-425d-88b0-41222753f71e.log
10292024-01-16T00:55:20.859Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ec64e9d0-0a28-424c-8a11-866f99eec730/root/var/svc/log/oxide-crucible-downstairs:downstairs-ca79c50a-b7bb-4982-ab50-21cc08f93d5f.log (5494 bytes)
10302024-01-16T00:55:21.952Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ec64e9d0-0a28-424c-8a11-866f99eec730/root/var/svc/log/oxide-crucible-downstairs:downstairs-ca79c50a-b7bb-4982-ab50-21cc08f93d5f.log
10312024-01-16T00:55:21.952Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_3c9ed70f-907d-4f61-b957-7afdb6873441/root/var/svc/log/oxide-crucible-pantry:default.log (1214607 bytes)
10322024-01-16T00:55:23.272Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_3c9ed70f-907d-4f61-b957-7afdb6873441/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 1214607 -> 1230459
10332024-01-16T00:55:24.348Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_3c9ed70f-907d-4f61-b957-7afdb6873441/root/var/svc/log/oxide-crucible-pantry:default.log
10342024-01-16T00:55:24.348Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e635cc13-2039-45bc-96b2-a3fcf1db7e23/root/var/svc/log/oxide-external_dns:default.log (26938 bytes)
10352024-01-16T00:55:25.515Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e635cc13-2039-45bc-96b2-a3fcf1db7e23/root/var/svc/log/oxide-external_dns:default.log
10362024-01-16T00:55:25.515Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_b8f9fe50-b389-4c61-9900-4eef4fa405e2/root/var/svc/log/oxide-oximeter:default.log (300173 bytes)
10372024-01-16T00:55:25.880Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_b8f9fe50-b389-4c61-9900-4eef4fa405e2/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 300173 -> 316408
10382024-01-16T00:55:27.000Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_b8f9fe50-b389-4c61-9900-4eef4fa405e2/root/var/svc/log/oxide-oximeter:default.log
10392024-01-16T00:55:27.000Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_18494817-844f-46d4-a75a-38cd11e8dc58/root/var/svc/log/oxide-cockroachdb:default.log (4379 bytes)
10402024-01-16T00:55:28.092Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_18494817-844f-46d4-a75a-38cd11e8dc58/root/var/svc/log/oxide-cockroachdb:default.log
10412024-01-16T00:55:28.093Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_18494817-844f-46d4-a75a-38cd11e8dc58/root/var/svc/log/oxide-zone-network-setup:default.log (1711 bytes)
10422024-01-16T00:55:29.200Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_18494817-844f-46d4-a75a-38cd11e8dc58/root/var/svc/log/oxide-zone-network-setup:default.log
10432024-01-16T00:55:29.225Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c0b921bb-7074-49fb-a551-8dfe4bd25c4c/root/var/svc/log/oxide-crucible-agent:default.log (14119 bytes)
10442024-01-16T00:55:30.323Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c0b921bb-7074-49fb-a551-8dfe4bd25c4c/root/var/svc/log/oxide-crucible-agent:default.log
10452024-01-16T00:55:30.323Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c0b921bb-7074-49fb-a551-8dfe4bd25c4c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10462024-01-16T00:55:31.429Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c0b921bb-7074-49fb-a551-8dfe4bd25c4c/root/var/svc/log/oxide-crucible-downstairs:default.log
10472024-01-16T00:55:31.429Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c0b921bb-7074-49fb-a551-8dfe4bd25c4c/root/var/svc/log/oxide-crucible-downstairs:downstairs-7f3dbfcd-d3b3-4db5-a71c-34a2c55ee947.log (5497 bytes)
10482024-01-16T00:55:32.613Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c0b921bb-7074-49fb-a551-8dfe4bd25c4c/root/var/svc/log/oxide-crucible-downstairs:downstairs-7f3dbfcd-d3b3-4db5-a71c-34a2c55ee947.log
10492024-01-16T00:55:32.649Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b6f2a824-73f9-4547-bcb9-20938808ca1c/root/var/svc/log/oxide-internal_dns:default.log (103143 bytes)
10502024-01-16T00:55:33.866Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b6f2a824-73f9-4547-bcb9-20938808ca1c/root/var/svc/log/oxide-internal_dns:default.log
10512024-01-16T00:55:33.867Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b6f2a824-73f9-4547-bcb9-20938808ca1c/root/var/svc/log/oxide-internal_dns:default.log.0 (10783 bytes)
10522024-01-16T00:55:34.982Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b6f2a824-73f9-4547-bcb9-20938808ca1c/root/var/svc/log/oxide-internal_dns:default.log.0
10532024-01-16T00:55:34.982Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9ede93a9-aa24-45ce-8f71-babae7e20be2/root/var/svc/log/oxide-nexus:default.log (762704 bytes)
10542024-01-16T00:55:35.848Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9ede93a9-aa24-45ce-8f71-babae7e20be2/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 762704 -> 833435
10552024-01-16T00:55:36.918Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9ede93a9-aa24-45ce-8f71-babae7e20be2/root/var/svc/log/oxide-nexus:default.log
10562024-01-16T00:55:36.918Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_975c69d0-41d8-49de-b78b-1e0f7725e413/root/var/svc/log/oxide-cockroachdb:default.log (4408 bytes)
10572024-01-16T00:55:38.025Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_975c69d0-41d8-49de-b78b-1e0f7725e413/root/var/svc/log/oxide-cockroachdb:default.log
10582024-01-16T00:55:38.025Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_975c69d0-41d8-49de-b78b-1e0f7725e413/root/var/svc/log/oxide-zone-network-setup:default.log (1711 bytes)
10592024-01-16T00:55:39.131Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_975c69d0-41d8-49de-b78b-1e0f7725e413/root/var/svc/log/oxide-zone-network-setup:default.log
10602024-01-16T00:55:39.131Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_2d15a09c-9ffe-4b36-a412-cbd26e71644e/root/var/svc/log/oxide-internal_dns:default.log (103148 bytes)
10612024-01-16T00:55:39.332Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_2d15a09c-9ffe-4b36-a412-cbd26e71644e/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 103148 -> 106904
10622024-01-16T00:55:40.403Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_2d15a09c-9ffe-4b36-a412-cbd26e71644e/root/var/svc/log/oxide-internal_dns:default.log
10632024-01-16T00:55:40.403Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_2d15a09c-9ffe-4b36-a412-cbd26e71644e/root/var/svc/log/oxide-internal_dns:default.log.0 (6725 bytes)
10642024-01-16T00:55:41.530Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_2d15a09c-9ffe-4b36-a412-cbd26e71644e/root/var/svc/log/oxide-internal_dns:default.log.0
10652024-01-16T00:55:41.530Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0fcaa4ec-3e03-4d11-bb54-aeee89f7cf8a/root/var/svc/log/oxide-crucible-agent:default.log (17553 bytes)
10662024-01-16T00:55:42.681Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0fcaa4ec-3e03-4d11-bb54-aeee89f7cf8a/root/var/svc/log/oxide-crucible-agent:default.log
10672024-01-16T00:55:42.682Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0fcaa4ec-3e03-4d11-bb54-aeee89f7cf8a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10682024-01-16T00:55:43.794Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0fcaa4ec-3e03-4d11-bb54-aeee89f7cf8a/root/var/svc/log/oxide-crucible-downstairs:default.log
10692024-01-16T00:55:43.795Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0fcaa4ec-3e03-4d11-bb54-aeee89f7cf8a/root/var/svc/log/oxide-crucible-downstairs:downstairs-13feffe5-6057-4f0f-8718-aef1ef8b5781.log (5623 bytes)
10702024-01-16T00:55:44.898Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0fcaa4ec-3e03-4d11-bb54-aeee89f7cf8a/root/var/svc/log/oxide-crucible-downstairs:downstairs-13feffe5-6057-4f0f-8718-aef1ef8b5781.log
10712024-01-16T00:55:44.898Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0fcaa4ec-3e03-4d11-bb54-aeee89f7cf8a/root/var/svc/log/oxide-crucible-downstairs:snapshot-13feffe5-6057-4f0f-8718-aef1ef8b5781-25678b68-406e-4fe0-842e-cd8ae4bb49f5.log (5140 bytes)
10722024-01-16T00:55:46.431Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_0fcaa4ec-3e03-4d11-bb54-aeee89f7cf8a/root/var/svc/log/oxide-crucible-downstairs:snapshot-13feffe5-6057-4f0f-8718-aef1ef8b5781-25678b68-406e-4fe0-842e-cd8ae4bb49f5.log
10732024-01-16T00:55:46.431Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8e5d6b7e-babe-4610-914e-3f156d5454eb/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10742024-01-16T00:55:47.551Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8e5d6b7e-babe-4610-914e-3f156d5454eb/root/var/svc/log/oxide-crucible-agent:default.log
10752024-01-16T00:55:47.551Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8e5d6b7e-babe-4610-914e-3f156d5454eb/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10762024-01-16T00:55:48.645Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_8e5d6b7e-babe-4610-914e-3f156d5454eb/root/var/svc/log/oxide-crucible-downstairs:default.log
10772024-01-16T00:55:48.645Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ae78c113-cdc9-4c21-a8a2-6f240fb98ed8/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10782024-01-16T00:55:49.751Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ae78c113-cdc9-4c21-a8a2-6f240fb98ed8/root/var/svc/log/oxide-crucible-agent:default.log
10792024-01-16T00:55:49.832Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ae78c113-cdc9-4c21-a8a2-6f240fb98ed8/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10802024-01-16T00:55:50.909Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_ae78c113-cdc9-4c21-a8a2-6f240fb98ed8/root/var/svc/log/oxide-crucible-downstairs:default.log
10812024-01-16T00:55:50.909Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_5c573ee7-fade-4d44-b19e-72662f045f1b/root/var/svc/log/oxide-nexus:default.log (36141251 bytes)
10822024-01-16T00:56:04.907Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_5c573ee7-fade-4d44-b19e-72662f045f1b/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 36141251 -> 36221860
10832024-01-16T00:56:06.396Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_5c573ee7-fade-4d44-b19e-72662f045f1b/root/var/svc/log/oxide-nexus:default.log
10842024-01-16T00:56:06.396Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_c01ea5eb-1445-47b9-b7e8-94691b3e377e/root/var/svc/log/oxide-external_dns:default.log (26945 bytes)
10852024-01-16T00:56:06.598Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_c01ea5eb-1445-47b9-b7e8-94691b3e377e/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 26945 -> 30723
10862024-01-16T00:56:07.669Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_c01ea5eb-1445-47b9-b7e8-94691b3e377e/root/var/svc/log/oxide-external_dns:default.log
10872024-01-16T00:56:07.670Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_400883f2-e500-4edd-bd82-3acb6e0075dd/root/var/svc/log/oxide-clickhouse:default.log (1123554 bytes)
10882024-01-16T00:56:09.134Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_400883f2-e500-4edd-bd82-3acb6e0075dd/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 1123554 -> 1318208
10892024-01-16T00:56:10.208Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_400883f2-e500-4edd-bd82-3acb6e0075dd/root/var/svc/log/oxide-clickhouse:default.log
10902024-01-16T00:56:10.208Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_400883f2-e500-4edd-bd82-3acb6e0075dd/root/var/svc/log/oxide-zone-network-setup:default.log (1705 bytes)
10912024-01-16T00:56:11.323Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_400883f2-e500-4edd-bd82-3acb6e0075dd/root/var/svc/log/oxide-zone-network-setup:default.log
10922024-01-16T00:56:11.323Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f3410a0c-763c-4a67-a54c-83aae1faf248/root/var/svc/log/oxide-cockroachdb:default.log (4408 bytes)
10932024-01-16T00:56:12.436Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f3410a0c-763c-4a67-a54c-83aae1faf248/root/var/svc/log/oxide-cockroachdb:default.log
10942024-01-16T00:56:12.436Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f3410a0c-763c-4a67-a54c-83aae1faf248/root/var/svc/log/oxide-zone-network-setup:default.log (1711 bytes)
10952024-01-16T00:56:13.554Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f3410a0c-763c-4a67-a54c-83aae1faf248/root/var/svc/log/oxide-zone-network-setup:default.log
10962024-01-16T00:56:13.554Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a669059-ccd2-418d-885f-719ccda2bdfd/root/var/svc/log/oxide-crucible-agent:default.log (27805 bytes)
10972024-01-16T00:56:14.700Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a669059-ccd2-418d-885f-719ccda2bdfd/root/var/svc/log/oxide-crucible-agent:default.log
10982024-01-16T00:56:14.700Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a669059-ccd2-418d-885f-719ccda2bdfd/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10992024-01-16T00:56:15.804Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a669059-ccd2-418d-885f-719ccda2bdfd/root/var/svc/log/oxide-crucible-downstairs:default.log
11002024-01-16T00:56:15.805Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a669059-ccd2-418d-885f-719ccda2bdfd/root/var/svc/log/oxide-crucible-downstairs:downstairs-7e3c7b86-c527-42fe-a41f-e3ed5d53967a.log (5495 bytes)
11012024-01-16T00:56:16.935Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a669059-ccd2-418d-885f-719ccda2bdfd/root/var/svc/log/oxide-crucible-downstairs:downstairs-7e3c7b86-c527-42fe-a41f-e3ed5d53967a.log
11022024-01-16T00:56:16.936Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a669059-ccd2-418d-885f-719ccda2bdfd/root/var/svc/log/oxide-crucible-downstairs:downstairs-ebe6f22f-d8bb-4e05-ad1f-de1b99354679.log (5620 bytes)
11032024-01-16T00:56:18.047Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a669059-ccd2-418d-885f-719ccda2bdfd/root/var/svc/log/oxide-crucible-downstairs:downstairs-ebe6f22f-d8bb-4e05-ad1f-de1b99354679.log
11042024-01-16T00:56:18.047Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a669059-ccd2-418d-885f-719ccda2bdfd/root/var/svc/log/oxide-crucible-downstairs:snapshot-ebe6f22f-d8bb-4e05-ad1f-de1b99354679-25678b68-406e-4fe0-842e-cd8ae4bb49f5.log (5138 bytes)
11052024-01-16T00:56:19.152Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a669059-ccd2-418d-885f-719ccda2bdfd/root/var/svc/log/oxide-crucible-downstairs:snapshot-ebe6f22f-d8bb-4e05-ad1f-de1b99354679-25678b68-406e-4fe0-842e-cd8ae4bb49f5.log
11062024-01-16T00:56:19.152Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2166fc1-2bc3-4ced-9e14-e17205cecee5/root/var/svc/log/oxide-crucible-agent:default.log (19612 bytes)
11072024-01-16T00:56:20.309Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2166fc1-2bc3-4ced-9e14-e17205cecee5/root/var/svc/log/oxide-crucible-agent:default.log
11082024-01-16T00:56:20.309Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2166fc1-2bc3-4ced-9e14-e17205cecee5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11092024-01-16T00:56:21.412Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2166fc1-2bc3-4ced-9e14-e17205cecee5/root/var/svc/log/oxide-crucible-downstairs:default.log
11102024-01-16T00:56:21.412Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2166fc1-2bc3-4ced-9e14-e17205cecee5/root/var/svc/log/oxide-crucible-downstairs:downstairs-1f4ebd4b-2e44-4699-9f13-914bae4ac6bf.log (3050 bytes)
11112024-01-16T00:56:22.523Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2166fc1-2bc3-4ced-9e14-e17205cecee5/root/var/svc/log/oxide-crucible-downstairs:downstairs-1f4ebd4b-2e44-4699-9f13-914bae4ac6bf.log
11122024-01-16T00:56:22.523Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2166fc1-2bc3-4ced-9e14-e17205cecee5/root/var/svc/log/oxide-crucible-downstairs:downstairs-9cd58f87-83b1-4ac9-8118-579c67a7e1bc.log (2921 bytes)
11132024-01-16T00:56:24.054Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e2166fc1-2bc3-4ced-9e14-e17205cecee5/root/var/svc/log/oxide-crucible-downstairs:downstairs-9cd58f87-83b1-4ac9-8118-579c67a7e1bc.log
11142024-01-16T00:56:24.087Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b3e7495e-387f-4bd7-949d-98445455098f/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
11152024-01-16T00:56:25.174Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b3e7495e-387f-4bd7-949d-98445455098f/root/var/svc/log/oxide-crucible-pantry:default.log
11162024-01-16T00:56:25.174Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_e180d29c-6433-4f4d-b172-108b0f35b39d/root/var/svc/log/oxide-nexus:default.log (890264 bytes)
11172024-01-16T00:56:26.380Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_e180d29c-6433-4f4d-b172-108b0f35b39d/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 890264 -> 969097
11182024-01-16T00:56:27.451Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_e180d29c-6433-4f4d-b172-108b0f35b39d/root/var/svc/log/oxide-nexus:default.log
11192024-01-16T00:56:27.452Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_d851de7d-1444-498c-87b7-c3b9e3656dd7/root/var/svc/log/oxide-cockroachdb:default.log (4406 bytes)
11202024-01-16T00:56:28.548Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_d851de7d-1444-498c-87b7-c3b9e3656dd7/root/var/svc/log/oxide-cockroachdb:default.log
11212024-01-16T00:56:28.548Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_d851de7d-1444-498c-87b7-c3b9e3656dd7/root/var/svc/log/oxide-zone-network-setup:default.log (1711 bytes)
11222024-01-16T00:56:29.645Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_d851de7d-1444-498c-87b7-c3b9e3656dd7/root/var/svc/log/oxide-zone-network-setup:default.log
11232024-01-16T00:56:29.670Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_51050378-db09-427d-941c-0c46f2654777/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
11242024-01-16T00:56:30.755Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_51050378-db09-427d-941c-0c46f2654777/root/var/svc/log/oxide-crucible-agent:default.log
11252024-01-16T00:56:30.755Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_51050378-db09-427d-941c-0c46f2654777/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11262024-01-16T00:56:31.930Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_51050378-db09-427d-941c-0c46f2654777/root/var/svc/log/oxide-crucible-downstairs:default.log
11272024-01-16T00:56:31.931Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_e7057193-95da-46f5-a30b-93bb296e1ccb/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
11282024-01-16T00:56:33.034Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_e7057193-95da-46f5-a30b-93bb296e1ccb/root/var/svc/log/oxide-crucible-pantry:default.log
11292024-01-16T00:56:33.034Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_e18a3f71-0073-4a2f-8010-6a2dfbd28c35/root/var/svc/log/oxide-internal_dns:default.log (103159 bytes)
11302024-01-16T00:56:33.225Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_e18a3f71-0073-4a2f-8010-6a2dfbd28c35/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 103159 -> 106914
11312024-01-16T00:56:34.326Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_e18a3f71-0073-4a2f-8010-6a2dfbd28c35/root/var/svc/log/oxide-internal_dns:default.log
11322024-01-16T00:56:34.326Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_e18a3f71-0073-4a2f-8010-6a2dfbd28c35/root/var/svc/log/oxide-internal_dns:default.log.0 (14844 bytes)
11332024-01-16T00:56:35.447Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_e18a3f71-0073-4a2f-8010-6a2dfbd28c35/root/var/svc/log/oxide-internal_dns:default.log.0
11342024-01-16T00:56:35.448Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be/root/var/svc/log/oxide-ntp:default.log (335 bytes)
11352024-01-16T00:56:36.543Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be/root/var/svc/log/oxide-ntp:default.log
11362024-01-16T00:56:36.543Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be/root/var/svc/log/oxide-ntp:default.log.0 (847 bytes)
11372024-01-16T00:56:37.644Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be/root/var/svc/log/oxide-ntp:default.log.0
11382024-01-16T00:56:37.645Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be/root/var/log/chrony/measurements.log (18906 bytes)
11392024-01-16T00:56:37.762Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be/root/var/log/chrony/measurements.log" changed size mid upload: 18906 -> 21372
11402024-01-16T00:56:38.844Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be/root/var/log/chrony/measurements.log
11412024-01-16T00:56:38.844Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be/root/var/log/chrony/statistics.log (12285 bytes)
11422024-01-16T00:56:38.921Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be/root/var/log/chrony/statistics.log" changed size mid upload: 12285 -> 14274
11432024-01-16T00:56:39.993Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be/root/var/log/chrony/statistics.log
11442024-01-16T00:56:39.994Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be/root/var/log/chrony/tracking.log (1986 bytes)
11452024-01-16T00:56:40.044Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be/root/var/log/chrony/tracking.log" changed size mid upload: 1986 -> 2250
11462024-01-16T00:56:41.112Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e51bd172-e0f5-436e-9ddf-aefb2b31e4be/root/var/log/chrony/tracking.log