01HGE8W1AZ3G96E51P143MQTSR: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HGE8XM3EGAG67YX36AEMT63Z

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-29T20:17:30.998Zjob dependencies complete; ready to run (waiting for 48 m 0 s)
22023-11-29T20:17:31.045Zjob assigned to worker 01HGEAW29M9EPB3YZD7C0AD0G1 (queued for 0 s)
32023-11-29T20:17:31.052Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-11-29T20:17:31.103Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-11-29T20:17:31.142Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-11-29T20:17:31.179Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-29T20:17:31.213Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-11-29T20:17:31.255ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-29T20:17:31.288Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-29T20:17:31.329ZOxide Helios Version helios-2.0.22318 64-bit
112023-11-29T20:17:31.375ZConfiguring devices.
122023-11-29T20:17:31.420Z#################################################
132023-11-29T20:17:31.455Z#### ############# ##
142023-11-29T20:17:31.510Z### ### ############ ##
152023-11-29T20:17:31.554Z## ### # ## ### ## Oxide Computer Company ##
162023-11-29T20:17:31.587Z## ## ## ### # ### ##
172023-11-29T20:17:31.622Z## # ### #### #### This Station Under ##
182023-11-29T20:17:31.677Z### ### #### # ### Computer Control ##
192023-11-29T20:17:31.710Z#### #### ### ## ##
202023-11-29T20:17:31.754Z##################################################
212023-11-29T20:17:31.788ZHostname: buskin
222023-11-29T20:17:31.823ZLoading smf(7) serice descriptions:
232023-11-29T20:17:33.102Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-8d69322849e0fa66.gz
242023-11-29T20:17:35.087Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-8d69322849e0fa66.gz
252023-11-29T20:17:35.088Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-29T20:17:53.011Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-29T20:17:53.011Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7d7c590276acd289.gz
282023-11-29T20:18:11.412Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7d7c590276acd289.gz
292023-11-29T20:18:11.412Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-29T20:18:12.336Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-29T20:18:12.336Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-29T20:18:15.643Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-29T20:18:15.674Zdownloading input: /input/package/work/version.txt
342023-11-29T20:18:15.674Zdownloaded input: /input/package/work/version.txt
352023-11-29T20:18:15.674Zdownloading input: /input/package/work/package.tar.gz
362023-11-29T20:18:31.285Zdownloaded input: /input/package/work/package.tar.gz
372023-11-29T20:18:31.285Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-29T20:18:53.377Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-29T20:18:53.378Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-29T20:19:00.694Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-29T20:19:00.724Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-29T20:19:42.803Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-29T20:19:42.803Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-29T20:20:26.121Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-29T20:20:26.121Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-29T20:20:53.806Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-29T20:20:53.806Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-29T20:20:58.046Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-29T20:20:58.069Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-29T20:21:07.412Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-29T20:21:07.412Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-29T20:21:13.769Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-29T20:21:13.769Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-29T20:21:19.129Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-29T20:21:19.130Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-29T20:21:19.168Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-29T20:21:19.194Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-29T20:21:25.074Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-29T20:21:25.074Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-29T20:21:28.941Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-29T20:21:28.941Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-29T20:21:49.951Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-29T20:21:49.951Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-29T20:22:05.332Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-29T20:22:05.332Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-29T20:22:05.380Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-29T20:22:05.405Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-29T20:22:12.210Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-29T20:22:12.210Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-29T20:22:30.869Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-29T20:22:30.870Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-29T20:23:13.327Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-29T20:23:13.358Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-29T20:23:42.784Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-29T20:23:42.810Zstarting task 0: "setup"
762023-11-29T20:23:42.811Z++ uname -s
772023-11-29T20:23:42.811Z+ kern=SunOS
782023-11-29T20:23:42.811Z+ case "$kern" in
792023-11-29T20:23:42.811Z+ groupadd -g 12345 build
802023-11-29T20:23:42.811Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-29T20:23:44.803Z+ zfs create -o mountpoint=/work rpool/work
822023-11-29T20:23:44.828Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-29T20:23:44.828Z+ home_fs=zfs
842023-11-29T20:23:44.828Z+ [[ zfs == autofs ]]
852023-11-29T20:23:44.828Z+ mkdir -p /home/build
862023-11-29T20:23:44.828Z+ chown build:build /home/build /work
872023-11-29T20:23:46.827Z+ chmod 0700 /home/build /work
882023-11-29T20:23:46.851Zprocess exited: duration 4045 ms, exit code 0
 
892023-11-29T20:23:46.900Zstarting task 1: "authentication"
902023-11-29T20:23:46.944Zprocess exited: duration 44 ms, exit code 0
 
912023-11-29T20:23:46.991Zstarting task 2: "build"
922023-11-29T20:23:47.015Z+ trap _exit_trap EXIT
932023-11-29T20:23:47.015Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-29T20:23:47.194Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-29T20:23:47.194Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-29T20:23:47.194Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-29T20:23:47.194Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-29T20:23:47.194Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-29T20:23:47.194Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-29T20:23:47.194Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-29T20:23:47.194Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-29T20:23:47.218Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-29T20:23:47.218Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-29T20:23:47.218Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-29T20:23:47.218Z+ [[ -d /opt/oxide ]]
1062023-11-29T20:23:47.218Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-29T20:23:47.218Z+ pfexec mkdir /opt/oxide
1082023-11-29T20:23:47.218Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-29T20:23:47.218Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-29T20:23:47.218Z+ cd /opt/oxide-underneath
1112023-11-29T20:23:47.219Z+ pfexec tar ceEp@/f - .
1122023-11-29T20:23:47.219Z+ cd /opt/oxide
1132023-11-29T20:23:47.219Z+ pfexec tar xveEp@/f -
1142023-11-29T20:23:47.317Zx ., 0 bytes, 0 tape blocks
1152023-11-29T20:23:47.317Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-29T20:23:47.317Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-29T20:23:47.317Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-29T20:23:47.341Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-29T20:23:47.341Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-29T20:23:47.365Z+ pfexec mkdir /opt/oxide/work
1212023-11-29T20:23:47.365Z+ pfexec chown build:build /opt/oxide/work
1222023-11-29T20:23:47.365Z+ cd /opt/oxide/work
1232023-11-29T20:23:47.365Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-29T20:23:48.911ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-29T20:23:48.911Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-11-29T20:23:48.911Zx out/omicron-sled-agent.tar, 112128512 bytes, 219001 tape blocks
1272023-11-29T20:23:48.911Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-29T20:23:48.911Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-11-29T20:23:48.911Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-11-29T20:23:48.911Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-11-29T20:23:48.911Zx target/release/omicron-package, 42568624 bytes, 83142 tape blocks
1322023-11-29T20:23:48.911Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-11-29T20:23:48.911Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-29T20:23:48.911Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-29T20:23:48.911Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-29T20:23:48.911Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-29T20:23:48.911Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-29T20:23:48.911Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-29T20:23:48.948Z
1402023-11-29T20:23:48.949Zreal 1.547799559
1412023-11-29T20:23:48.949Zuser 1.393963723
1422023-11-29T20:23:48.949Zsys 0.272794502
1432023-11-29T20:23:48.949Ztrap 0.000059650
1442023-11-29T20:23:48.949Ztflt 0.000000000
1452023-11-29T20:23:48.949Zdflt 0.000000000
1462023-11-29T20:23:48.949Zkflt 0.000000000
1472023-11-29T20:23:48.949Zlock 0.000000000
1482023-11-29T20:23:48.949Zslp 1.392797084
1492023-11-29T20:23:48.949Zlat 0.033711161
1502023-11-29T20:23:48.949Zstop 0.000130503
1512023-11-29T20:23:48.949Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-29T20:23:49.967Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-29T20:23:49.991Z+ mkdir tests
1542023-11-29T20:23:49.991Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-29T20:23:49.991Z+ ptime -m gunzip
1562023-11-29T20:23:49.991Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8d69322849e0fa66
1572023-11-29T20:23:50.128Z
1582023-11-29T20:23:50.152Zreal 0.138289878
1592023-11-29T20:23:50.152Zuser 0.126370080
1602023-11-29T20:23:50.152Zsys 0.011688051
1612023-11-29T20:23:50.152Ztrap 0.000008039
1622023-11-29T20:23:50.152Ztflt 0.000000000
1632023-11-29T20:23:50.152Zdflt 0.000000000
1642023-11-29T20:23:50.152Zkflt 0.000000000
1652023-11-29T20:23:50.152Zlock 0.000000000
1662023-11-29T20:23:50.152Zslp 0.000023230
1672023-11-29T20:23:50.152Zlat 0.000015820
1682023-11-29T20:23:50.152Zstop 0.000123552
1692023-11-29T20:23:50.152Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8d69322849e0fa66
1702023-11-29T20:23:50.152Z+ chmod a+x tests/bootstrap-8d69322849e0fa66
1712023-11-29T20:23:50.152Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-29T20:23:50.152Z+ ptime -m gunzip
1732023-11-29T20:23:50.152Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-29T20:23:52.312Z
1752023-11-29T20:23:52.335Zreal 2.171370107
1762023-11-29T20:23:52.336Zuser 2.041965716
1772023-11-29T20:23:52.336Zsys 0.129002962
1782023-11-29T20:23:52.336Ztrap 0.000169863
1792023-11-29T20:23:52.336Ztflt 0.000000000
1802023-11-29T20:23:52.336Zdflt 0.000000000
1812023-11-29T20:23:52.336Zkflt 0.000000000
1822023-11-29T20:23:52.336Zlock 0.000000000
1832023-11-29T20:23:52.336Zslp 0.000018860
1842023-11-29T20:23:52.336Zlat 0.000030701
1852023-11-29T20:23:52.336Zstop 0.000128093
1862023-11-29T20:23:52.336Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-29T20:23:52.336Z+ chmod a+x tests/bootstrap
1882023-11-29T20:23:52.336Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-29T20:23:52.336Z+ ptime -m gunzip
1902023-11-29T20:23:52.336Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7d7c590276acd289
1912023-11-29T20:23:54.565Z
1922023-11-29T20:23:54.635Zreal 2.240484099
1932023-11-29T20:23:54.635Zuser 2.107584130
1942023-11-29T20:23:54.635Zsys 0.132520671
1952023-11-29T20:23:54.635Ztrap 0.000158293
1962023-11-29T20:23:54.635Ztflt 0.000000000
1972023-11-29T20:23:54.635Zdflt 0.000000000
1982023-11-29T20:23:54.635Zkflt 0.000000000
1992023-11-29T20:23:54.635Zlock 0.000000000
2002023-11-29T20:23:54.635Zslp 0.000023140
2012023-11-29T20:23:54.635Zlat 0.000022630
2022023-11-29T20:23:54.635Zstop 0.000122423
2032023-11-29T20:23:54.635Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7d7c590276acd289
2042023-11-29T20:23:54.635Z+ chmod a+x tests/end_to_end_tests-7d7c590276acd289
2052023-11-29T20:23:54.635Z+ bmat address ls
2062023-11-29T20:23:54.635ZNAME CIDR FIRST LAST COUNT
2072023-11-29T20:23:54.636Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-11-29T20:23:54.636Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-29T20:23:54.660Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-11-29T20:23:54.660Z+ EXTRA_IP_START=10.151.1.164
2112023-11-29T20:23:54.660Z+ EXTRA_IP_END=10.151.1.227
2122023-11-29T20:23:54.660Z+ EXTRA_IP_COUNT=64
2132023-11-29T20:23:54.660Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-29T20:23:54.660Z+ EXTRA_IP_BASE=10.151.1
2152023-11-29T20:23:54.660Z+ EXTRA_IP_FOCTET=164
2162023-11-29T20:23:54.661Z+ EXTRA_IP_LOCTET=227
2172023-11-29T20:23:54.661Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-11-29T20:23:54.661Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-11-29T20:23:54.661Z+ DNS_IP1=10.151.1.164
2202023-11-29T20:23:54.661Z+ DNS_IP2=10.151.1.165
2212023-11-29T20:23:54.661Z+ UPLINK_IP=10.151.1.174
2222023-11-29T20:23:54.661Z+ IPPOOL_START=10.151.1.175
2232023-11-29T20:23:54.661Z+ IPPOOL_END=10.151.1.227
2242023-11-29T20:23:54.661Z++ ipadm show-addr -po type,addr
2252023-11-29T20:23:54.661Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-29T20:23:54.661Z+ GATEWAY_IP=10.151.1.100
2272023-11-29T20:23:54.661Z+ [[ -n 10.151.1.100 ]]
2282023-11-29T20:23:54.661Z+ ping -s 10.151.1.100 56 1
2292023-11-29T20:23:54.661ZPING 10.151.1.100: 56 data bytes
2302023-11-29T20:23:54.661Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
2312023-11-29T20:23:54.661Z
2322023-11-29T20:23:54.661Z----10.151.1.100 PING Statistics----
2332023-11-29T20:23:54.661Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-29T20:23:54.661Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-11-29T20:23:54.661Z++ arp -an
2362023-11-29T20:23:54.661Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-11-29T20:23:54.661Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-11-29T20:23:54.661Z+ routeadm -e ipv4-forwarding -u
2392023-11-29T20:23:54.729Z+ PXA_START=10.151.1.164
2402023-11-29T20:23:54.729Z+ PXA_END=10.151.1.227
2412023-11-29T20:23:54.764Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-29T20:23:54.764Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-29T20:23:54.925Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-29T20:23:54.927Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-29T20:23:54.951Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-29T20:23:54.951Z++ cd ./tools
2472023-11-29T20:23:54.951Z++ pwd
2482023-11-29T20:23:54.951Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-29T20:23:54.951Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-29T20:23:54.951Z+ SOFTNPU_MODE=zone
2512023-11-29T20:23:54.951Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-29T20:23:54.951Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-29T20:23:54.951Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-29T20:23:54.951Z++ dladm show-phys -p -o LINK
2552023-11-29T20:23:54.951Z++ head -1
2562023-11-29T20:23:54.951Z+ PHYSICAL_LINK=igb0
2572023-11-29T20:23:54.951Z+ echo 'Using igb0 as physical link'
2582023-11-29T20:23:54.951ZUsing igb0 as physical link
2592023-11-29T20:23:54.951Z+ ensure_run_as_root
2602023-11-29T20:23:54.951Z++ id -u
2612023-11-29T20:23:54.951Z+ [[ 0 -ne 0 ]]
2622023-11-29T20:23:54.951Z+ ensure_zpools
2632023-11-29T20:23:54.951Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-29T20:23:54.951Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-11-29T20:23:54.951Z+ readarray -t ZPOOLS
2662023-11-29T20:23:54.951Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-11-29T20:23:54.976Z++ sed 's/[ ",]//g'
2682023-11-29T20:23:54.976Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-11-29T20:23:54.976Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-11-29T20:23:54.976ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-11-29T20:23:54.976Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-11-29T20:23:54.977Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-11-29T20:23:54.977Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-11-29T20:23:55.002Z0+0 records in
2752023-11-29T20:23:55.002Z0+0 records out
2762023-11-29T20:23:55.002Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2772023-11-29T20:23:55.002Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-11-29T20:23:55.002Z+ set +x
2792023-11-29T20:23:55.002ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-11-29T20:23:55.002Z++ zpool list -o name
2812023-11-29T20:23:55.002Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-11-29T20:23:55.002Z+ [[ -z '' ]]
2832023-11-29T20:23:55.002Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-11-29T20:23:55.031Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-11-29T20:23:55.117Z+ set +x
2862023-11-29T20:23:55.117ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-11-29T20:23:55.117Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-11-29T20:23:55.117Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-11-29T20:23:55.117ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-11-29T20:23:55.117Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-11-29T20:23:55.117Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-11-29T20:23:55.117Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-11-29T20:23:55.119Z0+0 records in
2942023-11-29T20:23:55.119Z0+0 records out
2952023-11-29T20:23:55.120Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-11-29T20:23:55.120Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-11-29T20:23:55.120Z+ set +x
2982023-11-29T20:23:55.120ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-11-29T20:23:55.120Z++ zpool list -o name
3002023-11-29T20:23:55.120Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-11-29T20:23:55.120Z+ [[ -z '' ]]
3022023-11-29T20:23:55.120Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-11-29T20:23:55.147Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-11-29T20:23:55.147Z+ set +x
3052023-11-29T20:23:55.147ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-11-29T20:23:55.147Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-11-29T20:23:55.147Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-11-29T20:23:55.147ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3092023-11-29T20:23:55.147Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-11-29T20:23:55.147Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-11-29T20:23:55.147Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-11-29T20:23:55.147Z0+0 records in
3132023-11-29T20:23:55.147Z0+0 records out
3142023-11-29T20:23:55.147Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3152023-11-29T20:23:55.147Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-11-29T20:23:55.147Z+ set +x
3172023-11-29T20:23:55.147ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-11-29T20:23:55.148Z++ zpool list -o name
3192023-11-29T20:23:55.148Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-11-29T20:23:55.148Z+ [[ -z '' ]]
3212023-11-29T20:23:55.148Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-11-29T20:23:55.148Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-11-29T20:23:55.148Z+ set +x
3242023-11-29T20:23:55.148ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-11-29T20:23:55.148Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-11-29T20:23:55.148Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-11-29T20:23:55.148ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-11-29T20:23:55.148Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-11-29T20:23:55.148Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-11-29T20:23:55.150Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-11-29T20:23:55.150Z0+0 records in
3322023-11-29T20:23:55.150Z0+0 records out
3332023-11-29T20:23:55.150Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-11-29T20:23:55.150Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-11-29T20:23:55.150Z+ set +x
3362023-11-29T20:23:55.150ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-11-29T20:23:55.150Z++ zpool list -o name
3382023-11-29T20:23:55.150Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-11-29T20:23:55.150Z+ [[ -z '' ]]
3402023-11-29T20:23:55.150Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-11-29T20:23:55.173Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-11-29T20:23:55.173Z+ set +x
3432023-11-29T20:23:55.174ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-11-29T20:23:55.174Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-11-29T20:23:55.174Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-11-29T20:23:55.174ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-11-29T20:23:55.174Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-11-29T20:23:55.174Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-11-29T20:23:55.174Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-11-29T20:23:55.174Z0+0 records in
3512023-11-29T20:23:55.174Z0+0 records out
3522023-11-29T20:23:55.174Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3532023-11-29T20:23:55.174Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-11-29T20:23:55.174Z+ set +x
3552023-11-29T20:23:55.174ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-11-29T20:23:55.174Z++ zpool list -o name
3572023-11-29T20:23:55.174Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-11-29T20:23:55.174Z+ [[ -z '' ]]
3592023-11-29T20:23:55.174Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-11-29T20:23:55.200Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-11-29T20:23:55.200Z+ set +x
3622023-11-29T20:23:55.224ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-11-29T20:23:55.224Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-11-29T20:23:55.224Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-11-29T20:23:55.224ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-11-29T20:23:55.224Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-11-29T20:23:55.224Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-11-29T20:23:55.224Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-11-29T20:23:55.224Z0+0 records in
3702023-11-29T20:23:55.224Z0+0 records out
3712023-11-29T20:23:55.224Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3722023-11-29T20:23:55.224Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-11-29T20:23:55.224Z+ set +x
3742023-11-29T20:23:55.224ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-11-29T20:23:55.224Z++ zpool list -o name
3762023-11-29T20:23:55.224Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-11-29T20:23:55.224Z+ [[ -z '' ]]
3782023-11-29T20:23:55.224Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-11-29T20:23:55.314Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-11-29T20:23:55.315Z+ set +x
3812023-11-29T20:23:55.315ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-11-29T20:23:55.315Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-11-29T20:23:55.315Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-11-29T20:23:55.315ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-11-29T20:23:55.315Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-11-29T20:23:55.315Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-11-29T20:23:55.315Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-11-29T20:23:55.315Z0+0 records in
3892023-11-29T20:23:55.315Z0+0 records out
3902023-11-29T20:23:55.315Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-11-29T20:23:55.315Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-11-29T20:23:55.315Z+ set +x
3932023-11-29T20:23:55.315ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-11-29T20:23:55.315Z++ zpool list -o name
3952023-11-29T20:23:55.339Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-11-29T20:23:55.339Z+ [[ -z '' ]]
3972023-11-29T20:23:55.339Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-11-29T20:23:55.340Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-11-29T20:23:55.340Z+ set +x
4002023-11-29T20:23:55.340ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-11-29T20:23:55.340Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-29T20:23:55.340Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-11-29T20:23:55.340ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-11-29T20:23:55.340Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-11-29T20:23:55.340Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-11-29T20:23:55.340Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-11-29T20:23:55.340Z0+0 records in
4082023-11-29T20:23:55.340Z0+0 records out
4092023-11-29T20:23:55.340Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-11-29T20:23:55.340Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-11-29T20:23:55.340Z+ set +x
4122023-11-29T20:23:55.340ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-11-29T20:23:55.340Z++ zpool list -o name
4142023-11-29T20:23:55.340Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-11-29T20:23:55.340Z+ [[ -z '' ]]
4162023-11-29T20:23:55.340Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-11-29T20:23:55.340Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-11-29T20:23:55.340Z+ set +x
4192023-11-29T20:23:55.340ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-11-29T20:23:55.340Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-11-29T20:23:55.340Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-11-29T20:23:55.340ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-11-29T20:23:55.341Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-11-29T20:23:55.341Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-11-29T20:23:55.341Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-11-29T20:23:55.341Z0+0 records in
4272023-11-29T20:23:55.341Z0+0 records out
4282023-11-29T20:23:55.341Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4292023-11-29T20:23:55.341Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-11-29T20:23:55.341Z+ set +x
4312023-11-29T20:23:55.341ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-11-29T20:23:55.365Z++ zpool list -o name
4332023-11-29T20:23:55.365Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-11-29T20:23:55.365Z+ [[ -z '' ]]
4352023-11-29T20:23:55.365Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-11-29T20:23:55.442Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-11-29T20:23:55.442Z+ set +x
4382023-11-29T20:23:55.442ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-11-29T20:23:55.442Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-11-29T20:23:55.442Z+ readarray -t ZPOOLS
4412023-11-29T20:23:55.442Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-11-29T20:23:55.442Z++ sed 's/[ ",]//g'
4432023-11-29T20:23:55.442Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-11-29T20:23:55.442Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-11-29T20:23:55.443ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-11-29T20:23:55.443Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-11-29T20:23:55.443Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-11-29T20:23:55.443Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-11-29T20:23:55.443Z0+0 records in
4502023-11-29T20:23:55.443Z0+0 records out
4512023-11-29T20:23:55.443Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4522023-11-29T20:23:55.443Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-11-29T20:23:55.443Z+ set +x
4542023-11-29T20:23:55.443ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-11-29T20:23:55.443Z++ zpool list -o name
4562023-11-29T20:23:55.467Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-11-29T20:23:55.467Z+ [[ -z '' ]]
4582023-11-29T20:23:55.467Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-11-29T20:23:55.467Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-11-29T20:23:55.467Z+ set +x
4612023-11-29T20:23:55.467ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-11-29T20:23:55.467Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-11-29T20:23:55.467Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-11-29T20:23:55.467ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-11-29T20:23:55.467Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-11-29T20:23:55.467Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-11-29T20:23:55.468Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-11-29T20:23:55.468Z0+0 records in
4692023-11-29T20:23:55.468Z0+0 records out
4702023-11-29T20:23:55.468Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4712023-11-29T20:23:55.468Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-11-29T20:23:55.468Z+ set +x
4732023-11-29T20:23:55.468ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-11-29T20:23:55.468Z++ zpool list -o name
4752023-11-29T20:23:55.468Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-11-29T20:23:55.468Z+ [[ -z '' ]]
4772023-11-29T20:23:55.468Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-11-29T20:23:55.552Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-11-29T20:23:55.552Z+ set +x
4802023-11-29T20:23:55.552ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-11-29T20:23:55.552Z+ [[ zone == \z\o\n\e ]]
4822023-11-29T20:23:55.552Z+ ensure_simulated_links igb0
4832023-11-29T20:23:55.552Z+ local PHYSICAL_LINK=igb0
4842023-11-29T20:23:55.552Z+ INDICES=("0" "1")
4852023-11-29T20:23:55.552Z+ for I in "${INDICES[@]}"
4862023-11-29T20:23:55.552Z++ get_simnet_name_if_exists net0
4872023-11-29T20:23:55.552Z++ dladm show-simnet -p -o LINK net0
4882023-11-29T20:23:55.552Z++ echo ''
4892023-11-29T20:23:55.552Z+ [[ -z '' ]]
4902023-11-29T20:23:55.552Z+ dladm create-simnet -t net0
4912023-11-29T20:23:55.552Z+ dladm create-simnet -t sc0_0
4922023-11-29T20:23:55.552Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-11-29T20:23:55.552Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-11-29T20:23:55.635Z+ success 'Simnet net0/sc0_0 exists'
4952023-11-29T20:23:55.635Z+ set +x
4962023-11-29T20:23:55.635ZSimnet net0/sc0_0 exists
4972023-11-29T20:23:55.635Z+ for I in "${INDICES[@]}"
4982023-11-29T20:23:55.635Z++ get_simnet_name_if_exists net1
4992023-11-29T20:23:55.635Z++ dladm show-simnet -p -o LINK net1
5002023-11-29T20:23:55.635Z++ echo ''
5012023-11-29T20:23:55.635Z+ [[ -z '' ]]
5022023-11-29T20:23:55.635Z+ dladm create-simnet -t net1
5032023-11-29T20:23:55.635Z+ dladm create-simnet -t sc1_0
5042023-11-29T20:23:55.635Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-11-29T20:23:55.635Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-11-29T20:23:55.635Z+ success 'Simnet net1/sc1_0 exists'
5072023-11-29T20:23:55.635Z+ set +x
5082023-11-29T20:23:55.635ZSimnet net1/sc1_0 exists
5092023-11-29T20:23:55.635Z++ get_vnic_name_if_exists sc0_1
5102023-11-29T20:23:55.635Z++ dladm show-vnic -p -o LINK sc0_1
5112023-11-29T20:23:55.635Z++ echo ''
5122023-11-29T20:23:55.635Z+ [[ -z '' ]]
5132023-11-29T20:23:55.635Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-11-29T20:23:55.635Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-11-29T20:23:55.635Z+ success 'Vnic sc0_1 exists'
5162023-11-29T20:23:55.636Z+ set +x
5172023-11-29T20:23:55.637ZVnic sc0_1 exists
5182023-11-29T20:23:55.637Z+ warn_if_no_proxy_arp
5192023-11-29T20:23:55.637Z+ [[ -v PXA_START ]]
5202023-11-29T20:23:55.637Z+ [[ -v PXA_END ]]
5212023-11-29T20:23:55.637Z+ ensure_softnpu_zone
5222023-11-29T20:23:55.637Z+ zoneadm list
5232023-11-29T20:23:55.637Z+ grep -q sidecar_softnpu
5242023-11-29T20:23:55.637Z+ [[ -f out/npuzone/npuzone ]]
5252023-11-29T20:23:55.637Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5262023-11-29T20:23:55.637Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
5272023-11-29T20:24:01.859Zdownloading softnpu ...done
5282023-11-29T20:24:01.884Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5292023-11-29T20:24:08.216Zdownloading asic_program.so ...done
5302023-11-29T20:24:08.240Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-29T20:24:14.985Zdownloading scadm ...done
5322023-11-29T20:24:15.008Zcreating port tfportrear0_0 for sc0_0
5332023-11-29T20:24:15.008Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-29T20:24:15.032Zsoftnpu zone setup
5352023-11-29T20:24:15.032Zconfigure zone
5362023-11-29T20:24:15.056Zinstall zone
5372023-11-29T20:24:16.082Zboot zone
5382023-11-29T20:24:19.193Z[sidecar_softnpu] svcs milestone/network
5392023-11-29T20:24:19.220ZSTATE STIME FMRI
5402023-11-29T20:24:19.220Zonline 20:24:17 svc:/milestone/network:default
5412023-11-29T20:24:19.220Z[sidecar_softnpu] /softnpu/npu start
5422023-11-29T20:24:19.246Z
5432023-11-29T20:24:19.270Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-29T20:24:19.270Z+ GATEWAY_IP=10.151.1.100
5452023-11-29T20:24:19.270Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-11-29T20:24:19.270ZUsing 10.151.1.100 as gateway ip
5472023-11-29T20:24:19.270Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-29T20:24:19.270Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-11-29T20:24:19.270Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-29T20:24:19.270Z+++ echo 18:c0:4d:0d:9f:b2
5512023-11-29T20:24:19.270Z+ COUNT=1
5522023-11-29T20:24:19.270Z+ [[ 1 -ne 1 ]]
5532023-11-29T20:24:19.270Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-11-29T20:24:19.270ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-11-29T20:24:19.271Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-11-29T20:24:19.271Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-11-29T20:24:19.295Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-29T20:24:19.295Z+ [[ -v PXA_START ]]
5592023-11-29T20:24:19.295Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-11-29T20:24:19.295Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-11-29T20:24:19.318Z+ z_scadm dump-state
5622023-11-29T20:24:19.318Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-29T20:24:19.343Zlocal v6:
5642023-11-29T20:24:19.343Zlocal v4:
5652023-11-29T20:24:19.343Zrouter v6:
5662023-11-29T20:24:19.343Zrouter v4_idx:
5672023-11-29T20:24:19.343Zrouter v4_routes:
5682023-11-29T20:24:19.343Zresolver v4:
5692023-11-29T20:24:19.343Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-11-29T20:24:19.343Zresolver v6:
5712023-11-29T20:24:19.343Znat v4:
5722023-11-29T20:24:19.343Znat_v6:
5732023-11-29T20:24:19.343Zport_mac:
5742023-11-29T20:24:19.343Zproxy arp:
5752023-11-29T20:24:19.343Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-11-29T20:24:19.367Z+ success 'softnpu zone exists'
5772023-11-29T20:24:19.367Z+ set +x
5782023-11-29T20:24:19.367Zsoftnpu zone exists
5792023-11-29T20:24:19.367Z
5802023-11-29T20:24:19.367Zreal 24.483593947
5812023-11-29T20:24:19.367Zuser 0.694165379
5822023-11-29T20:24:19.367Zsys 1.616502087
5832023-11-29T20:24:19.367Ztrap 0.000075745
5842023-11-29T20:24:19.367Ztflt 0.000102607
5852023-11-29T20:24:19.367Zdflt 0.001108992
5862023-11-29T20:24:19.367Zkflt 0.000000000
5872023-11-29T20:24:19.367Zlock 18:56.218482337
5882023-11-29T20:24:19.367Zslp 1:13.247760496
5892023-11-29T20:24:19.367Zlat 0.078820330
5902023-11-29T20:24:19.367Zstop 0.342761962
5912023-11-29T20:24:19.367Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-29T20:24:19.391Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-29T20:24:19.391Z+ SILO_NAME=recovery
5942023-11-29T20:24:19.391Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-29T20:24:19.391Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-29T20:24:19.391Z+ sed -i~ '
5972023-11-29T20:24:19.391Z /^external_dns_ips/c\
5982023-11-29T20:24:19.391Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-11-29T20:24:19.391Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-29T20:24:19.391Z /^first/c\
6012023-11-29T20:24:19.391Zfirst = "10.151.1.164"
6022023-11-29T20:24:19.391Z /^last/c\
6032023-11-29T20:24:19.391Zlast = "10.151.1.173"
6042023-11-29T20:24:19.391Z }
6052023-11-29T20:24:19.391Z /^\[rack_network_config/,/^$/ {
6062023-11-29T20:24:19.391Z /^infra_ip_first/c\
6072023-11-29T20:24:19.391Zinfra_ip_first = "10.151.1.174"
6082023-11-29T20:24:19.391Z /^infra_ip_last/c\
6092023-11-29T20:24:19.392Zinfra_ip_last = "10.151.1.174"
6102023-11-29T20:24:19.392Z }
6112023-11-29T20:24:19.393Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-11-29T20:24:19.393Z /^routes/c\
6132023-11-29T20:24:19.393Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6142023-11-29T20:24:19.393Z /^addresses/c\
6152023-11-29T20:24:19.393Zaddresses = \["10.151.1.174/32"\]
6162023-11-29T20:24:19.393Z }
6172023-11-29T20:24:19.393Z' pkg/config-rss.toml
6182023-11-29T20:24:19.418Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-11-29T20:24:19.418Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-11-29T20:24:19.418Z+++ pkg/config-rss.toml Wed Nov 29 20:24:19 2023
6212023-11-29T20:24:19.418Z@@ -41,7 +41,7 @@
6222023-11-29T20:24:19.418Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-11-29T20:24:19.418Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-11-29T20:24:19.418Z # below.
6252023-11-29T20:24:19.418Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-11-29T20:24:19.418Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6272023-11-29T20:24:19.418Z
6282023-11-29T20:24:19.418Z # Initial TLS certificates for the external API
6292023-11-29T20:24:19.418Z #
6302023-11-29T20:24:19.418Z@@ -77,8 +77,8 @@
6312023-11-29T20:24:19.418Z #
6322023-11-29T20:24:19.418Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-11-29T20:24:19.418Z [[internal_services_ip_pool_ranges]]
6342023-11-29T20:24:19.418Z-first = "192.168.1.20"
6352023-11-29T20:24:19.418Z-last = "192.168.1.29"
6362023-11-29T20:24:19.418Z+first = "10.151.1.164"
6372023-11-29T20:24:19.418Z+last = "10.151.1.173"
6382023-11-29T20:24:19.418Z
6392023-11-29T20:24:19.418Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-11-29T20:24:19.418Z # parameters should be provided to the control plane via wicket, but we need to
6412023-11-29T20:24:19.418Z@@ -92,8 +92,8 @@
6422023-11-29T20:24:19.418Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-11-29T20:24:19.418Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-11-29T20:24:19.418Z # With softnpu, only one address is used.
6452023-11-29T20:24:19.418Z-infra_ip_first = "192.168.1.30"
6462023-11-29T20:24:19.418Z-infra_ip_last = "192.168.1.30"
6472023-11-29T20:24:19.418Z+infra_ip_first = "10.151.1.174"
6482023-11-29T20:24:19.418Z+infra_ip_last = "10.151.1.174"
6492023-11-29T20:24:19.418Z
6502023-11-29T20:24:19.418Z # Configurations for BGP routers to run on the scrimlets.
6512023-11-29T20:24:19.418Z bgp = []
6522023-11-29T20:24:19.418Z@@ -101,9 +101,9 @@
6532023-11-29T20:24:19.418Z # You can configure multiple uplinks by repeating the following stanza
6542023-11-29T20:24:19.418Z [[rack_network_config.ports]]
6552023-11-29T20:24:19.418Z # Routes associated with this port.
6562023-11-29T20:24:19.418Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-11-29T20:24:19.418Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6582023-11-29T20:24:19.418Z # Addresses associated with this port.
6592023-11-29T20:24:19.418Z-addresses = ["192.168.1.30/32"]
6602023-11-29T20:24:19.418Z+addresses = ["10.151.1.174/32"]
6612023-11-29T20:24:19.418Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-11-29T20:24:19.418Z port = "qsfp0"
6632023-11-29T20:24:19.418Z # The speed of this port.
6642023-11-29T20:24:19.418Z+ true
6652023-11-29T20:24:19.419Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-11-29T20:24:19.419Za pkg/config-rss.toml 7K
6672023-11-29T20:24:19.419Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-11-29T20:24:19.419Z+ TLS_NAME=recovery.sys.oxide.test
6692023-11-29T20:24:19.419Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-11-29T20:24:20.083Z..+.....+....+......+..............+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.+...............+..............+.......+...+..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+......+.........+....+.....+......+..................+.............+..+.......+.....+.+.........+..............+.........+...+....+.....................+..+...+.............+..+.......+...+..............+....+.....+.+.....+......+.......+..+.........+..........+..+...+.......+...+....................+.........+.......+.....+...+.....................+.+........+......+.......+............+...+.........+...+........+...+.........+.+.........+...+.........+........+...............+......+............+.......+..+....+..+.......+..+.............+.....+...+.......+..+.+..+.......+..............+...............+.......+.................+.+......+.....+...+.+.................+............+.+......+......+..+...+....+...+..........................+.........+.+.....................+........+..........+...+........+............+.........+.+..+...+.........+.....................+....+...+..+....+..+............+.......+......+..............+.+..+............+......+.........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-11-29T20:24:20.271Z..........+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+...........+....+...+...+.....+...+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+...................+..................+......+......+...+......+......+.....+.+....................+....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-11-29T20:24:20.295Z-----
6732023-11-29T20:24:20.295Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-11-29T20:24:20.319Za pkg/initial-tls-cert.pem 2K
6752023-11-29T20:24:20.319Za pkg/initial-tls-key.pem 4K
6762023-11-29T20:24:20.319Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-11-29T20:24:20.319Z+ rmdir pkg
6782023-11-29T20:24:20.319Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-11-29T20:24:20.319Z+ pfexec mkdir -p /usr/oxide
6802023-11-29T20:24:20.319Z+ pfexec rm -f /usr/oxide/tcpproxy
6812023-11-29T20:24:20.319Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6822023-11-29T20:24:20.602Z+ pfexec chmod +x /usr/oxide/tcpproxy
6832023-11-29T20:24:20.625Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6842023-11-29T20:24:20.625Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6852023-11-29T20:24:20.625Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6862023-11-29T20:24:20.725Z+ OMICRON_NO_UNINSTALL=1
6872023-11-29T20:24:20.725Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6882023-11-29T20:24:20.766ZNov 29 20:24:20.968 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6892023-11-29T20:24:20.766ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6902023-11-29T20:24:20.766ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6912023-11-29T20:24:20.766ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6922023-11-29T20:24:20.766ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6932023-11-29T20:24:20.766ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6942023-11-29T20:24:20.766ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6952023-11-29T20:24:20.766ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6962023-11-29T20:24:20.766ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6972023-11-29T20:24:20.766ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6982023-11-29T20:24:20.766ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6992023-11-29T20:24:20.766ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7002023-11-29T20:24:20.766ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7012023-11-29T20:24:20.766ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
7022023-11-29T20:24:20.767ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
7032023-11-29T20:24:20.767ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7042023-11-29T20:24:20.767ZNov 29 20:24:20.970 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7052023-11-29T20:24:20.938ZNov 29 20:24:21.168 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-11-29T20:24:20.972ZNov 29 20:24:21.202 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-29T20:24:21.030ZNov 29 20:24:21.259 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-29T20:24:21.157Z
7092023-11-29T20:24:21.180Zreal 0.427762370
7102023-11-29T20:24:21.180Zuser 0.098370027
7112023-11-29T20:24:21.180Zsys 1.725750694
7122023-11-29T20:24:21.180Ztrap 0.000007836
7132023-11-29T20:24:21.180Ztflt 0.000059598
7142023-11-29T20:24:21.180Zdflt 0.000372616
7152023-11-29T20:24:21.180Zkflt 0.000000000
7162023-11-29T20:24:21.181Zlock 38.318618361
7172023-11-29T20:24:21.181Zslp 0.634992751
7182023-11-29T20:24:21.181Zlat 0.002796243
7192023-11-29T20:24:21.181Zstop 0.198438523
7202023-11-29T20:24:21.181Z+ retry=0
7212023-11-29T20:24:21.181Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-29T20:24:21.181Z+ [[ 0 -gt 30 ]]
7232023-11-29T20:24:21.181Z+ sleep 1
7242023-11-29T20:24:22.172Z+ retry=1
7252023-11-29T20:24:22.172Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-29T20:24:22.195Z+ [[ 1 -gt 30 ]]
7272023-11-29T20:24:22.195Z+ sleep 1
7282023-11-29T20:24:23.185Z+ retry=2
7292023-11-29T20:24:23.185Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-29T20:24:23.208Z+ [[ 2 -gt 30 ]]
7312023-11-29T20:24:23.209Z+ sleep 1
7322023-11-29T20:24:24.198Z+ retry=3
7332023-11-29T20:24:24.198Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-11-29T20:24:24.222Z+ [[ 3 -gt 30 ]]
7352023-11-29T20:24:24.222Z+ sleep 1
7362023-11-29T20:24:25.213Z+ retry=4
7372023-11-29T20:24:25.213Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-11-29T20:24:44.975Z+ [[ 4 -gt 30 ]]
7392023-11-29T20:24:44.975Z+ sleep 1
7402023-11-29T20:24:45.979Z+ retry=5
7412023-11-29T20:24:45.979Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7422023-11-29T20:24:46.003Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7432023-11-29T20:24:46.026Zlocal v6:
7442023-11-29T20:24:46.026Zfd00:99::1
7452023-11-29T20:24:46.026Zfe80::e07b:d0ff:fef9:982
7462023-11-29T20:24:46.026Zlocal v4:
7472023-11-29T20:24:46.026Z10.151.1.174
7482023-11-29T20:24:46.027Zrouter v6:
7492023-11-29T20:24:46.027Zrouter v4_idx:
7502023-11-29T20:24:46.027Z0.0.0.0/0 -> 0
7512023-11-29T20:24:46.027Zrouter v4_routes:
7522023-11-29T20:24:46.027Z0 -> 10.151.1.100 (2)
7532023-11-29T20:24:46.027Zresolver v4:
7542023-11-29T20:24:46.027Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7552023-11-29T20:24:46.027Zresolver v6:
7562023-11-29T20:24:46.027Zfe80::e07b:d0ff:fef9:982 -> e2:7b:d0:f9:09:82
7572023-11-29T20:24:46.027Zfe80::787e:91ff:feb6:9303 -> 7a:7e:91:b6:93:03
7582023-11-29T20:24:46.027Znat v4:
7592023-11-29T20:24:46.027Znat_v6:
7602023-11-29T20:24:46.027Zport_mac:
7612023-11-29T20:24:46.027Z1000: a8:40:25:32:7d:ca
7622023-11-29T20:24:46.027Z1: a8:40:25:32:7d:cb
7632023-11-29T20:24:46.027Zproxy arp:
7642023-11-29T20:24:46.028Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7652023-11-29T20:24:46.028Z+ retry=0
7662023-11-29T20:24:46.028Z+++ zoneadm list -n
7672023-11-29T20:24:46.028Z+++ grep oxz_ntp
7682023-11-29T20:24:46.052Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-11-29T20:24:46.052Zsvcs: zone '-Hostate': entity not found
7702023-11-29T20:24:46.052Z++ true
7712023-11-29T20:24:46.052Z+ [[ '' != online ]]
7722023-11-29T20:24:46.052Z+ [[ 0 -gt 60 ]]
7732023-11-29T20:24:46.052Z+ sleep 1
7742023-11-29T20:24:47.040Z+ retry=1
7752023-11-29T20:24:47.064Z+++ zoneadm list -n
7762023-11-29T20:24:47.064Z+++ grep oxz_ntp
7772023-11-29T20:24:47.064Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-11-29T20:24:47.064Zsvcs: zone '-Hostate': entity not found
7792023-11-29T20:24:47.064Z++ true
7802023-11-29T20:24:47.064Z+ [[ '' != online ]]
7812023-11-29T20:24:47.064Z+ [[ 1 -gt 60 ]]
7822023-11-29T20:24:47.064Z+ sleep 1
7832023-11-29T20:24:48.061Z+ retry=2
7842023-11-29T20:24:48.085Z+++ zoneadm list -n
7852023-11-29T20:24:48.085Z+++ grep oxz_ntp
7862023-11-29T20:24:48.085Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-11-29T20:24:48.085Zsvcs: zone '-Hostate': entity not found
7882023-11-29T20:24:48.085Z++ true
7892023-11-29T20:24:48.085Z+ [[ '' != online ]]
7902023-11-29T20:24:48.085Z+ [[ 2 -gt 60 ]]
7912023-11-29T20:24:48.085Z+ sleep 1
7922023-11-29T20:24:49.084Z+ retry=3
7932023-11-29T20:24:49.107Z+++ zoneadm list -n
7942023-11-29T20:24:49.107Z+++ grep oxz_ntp
7952023-11-29T20:24:49.107Z++ pfexec svcs -z oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9 -Hostate oxide/ntp
7962023-11-29T20:24:49.131Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7972023-11-29T20:24:49.131Z++ true
7982023-11-29T20:24:49.131Z+ [[ '' != online ]]
7992023-11-29T20:24:49.131Z+ [[ 3 -gt 60 ]]
8002023-11-29T20:24:49.131Z+ sleep 1
8012023-11-29T20:24:50.116Z+ retry=4
8022023-11-29T20:24:50.139Z+++ zoneadm list -n
8032023-11-29T20:24:50.139Z+++ grep oxz_ntp
8042023-11-29T20:24:50.139Z++ pfexec svcs -z oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9 -Hostate oxide/ntp
8052023-11-29T20:24:50.163Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8062023-11-29T20:24:50.163Z++ true
8072023-11-29T20:24:50.163Z+ [[ '' != online ]]
8082023-11-29T20:24:50.163Z+ [[ 4 -gt 60 ]]
8092023-11-29T20:24:50.163Z+ sleep 1
8102023-11-29T20:24:51.146Z+ retry=5
8112023-11-29T20:24:51.170Z+++ zoneadm list -n
8122023-11-29T20:24:51.170Z+++ grep oxz_ntp
8132023-11-29T20:24:51.170Z++ pfexec svcs -z oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9 -Hostate oxide/ntp
8142023-11-29T20:24:51.194Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8152023-11-29T20:24:51.194Z++ true
8162023-11-29T20:24:51.194Z+ [[ '' != online ]]
8172023-11-29T20:24:51.194Z+ [[ 5 -gt 60 ]]
8182023-11-29T20:24:51.194Z+ sleep 1
8192023-11-29T20:24:52.176Z+ retry=6
8202023-11-29T20:24:52.207Z+++ zoneadm list -n
8212023-11-29T20:24:52.207Z+++ grep oxz_ntp
8222023-11-29T20:24:52.207Z++ pfexec svcs -z oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9 -Hostate oxide/ntp
8232023-11-29T20:24:52.207Z+ [[ disabled != online ]]
8242023-11-29T20:24:52.207Z+ [[ 6 -gt 60 ]]
8252023-11-29T20:24:52.207Z+ sleep 1
8262023-11-29T20:24:53.197Z+ retry=7
8272023-11-29T20:24:53.221Z+++ zoneadm list -n
8282023-11-29T20:24:53.221Z+++ grep oxz_ntp
8292023-11-29T20:24:53.221Z++ pfexec svcs -z oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9 -Hostate oxide/ntp
8302023-11-29T20:24:53.221Z+ [[ disabled != online ]]
8312023-11-29T20:24:53.221Z+ [[ 7 -gt 60 ]]
8322023-11-29T20:24:53.221Z+ sleep 1
8332023-11-29T20:24:54.220Z+ retry=8
8342023-11-29T20:24:54.243Z+++ zoneadm list -n
8352023-11-29T20:24:54.243Z+++ grep oxz_ntp
8362023-11-29T20:24:54.243Z++ pfexec svcs -z oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9 -Hostate oxide/ntp
8372023-11-29T20:24:54.243Z+ [[ disabled != online ]]
8382023-11-29T20:24:54.243Z+ [[ 8 -gt 60 ]]
8392023-11-29T20:24:54.243Z+ sleep 1
8402023-11-29T20:24:55.242Z+ retry=9
8412023-11-29T20:24:55.265Z+++ zoneadm list -n
8422023-11-29T20:24:55.265Z+++ grep oxz_ntp
8432023-11-29T20:24:55.265Z++ pfexec svcs -z oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9 -Hostate oxide/ntp
8442023-11-29T20:24:55.265Z+ [[ disabled != online ]]
8452023-11-29T20:24:55.265Z+ [[ 9 -gt 60 ]]
8462023-11-29T20:24:55.265Z+ sleep 1
8472023-11-29T20:24:56.263Z+ retry=10
8482023-11-29T20:24:56.287Z+++ zoneadm list -n
8492023-11-29T20:24:56.287Z+++ grep oxz_ntp
8502023-11-29T20:24:56.287Z++ pfexec svcs -z oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9 -Hostate oxide/ntp
8512023-11-29T20:24:56.287Z+ [[ disabled != online ]]
8522023-11-29T20:24:56.287Z+ [[ 10 -gt 60 ]]
8532023-11-29T20:24:56.287Z+ sleep 1
8542023-11-29T20:24:57.284Z+ retry=11
8552023-11-29T20:24:57.312Z+++ zoneadm list -n
8562023-11-29T20:24:57.312Z+++ grep oxz_ntp
8572023-11-29T20:24:57.312Z++ pfexec svcs -z oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9 -Hostate oxide/ntp
8582023-11-29T20:24:57.312Z+ [[ disabled != online ]]
8592023-11-29T20:24:57.312Z+ [[ 11 -gt 60 ]]
8602023-11-29T20:24:57.312Z+ sleep 1
8612023-11-29T20:24:58.305Z+ retry=12
8622023-11-29T20:24:58.328Z+++ zoneadm list -n
8632023-11-29T20:24:58.328Z+++ grep oxz_ntp
8642023-11-29T20:24:58.328Z++ pfexec svcs -z oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9 -Hostate oxide/ntp
8652023-11-29T20:24:58.329Z+ [[ disabled != online ]]
8662023-11-29T20:24:58.329Z+ [[ 12 -gt 60 ]]
8672023-11-29T20:24:58.329Z+ sleep 1
8682023-11-29T20:24:59.327Z+ retry=13
8692023-11-29T20:24:59.351Z+++ zoneadm list -n
8702023-11-29T20:24:59.351Z+++ grep oxz_ntp
8712023-11-29T20:24:59.351Z++ pfexec svcs -z oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9 -Hostate oxide/ntp
8722023-11-29T20:24:59.351Z+ [[ disabled != online ]]
8732023-11-29T20:24:59.351Z+ [[ 13 -gt 60 ]]
8742023-11-29T20:24:59.351Z+ sleep 1
8752023-11-29T20:25:00.347Z+ retry=14
8762023-11-29T20:25:00.371Z+++ zoneadm list -n
8772023-11-29T20:25:00.371Z+++ grep oxz_ntp
8782023-11-29T20:25:00.371Z++ pfexec svcs -z oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9 -Hostate oxide/ntp
8792023-11-29T20:25:00.371Z+ [[ disabled != online ]]
8802023-11-29T20:25:00.371Z+ [[ 14 -gt 60 ]]
8812023-11-29T20:25:00.371Z+ sleep 1
 
8822023-11-29T20:25:00.515Zbuskin console login: Nov 29 20:25:00 bukin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8832023-11-29T20:25:01.370Z+ retry=15
8842023-11-29T20:25:01.394Z+++ zoneadm list -n
8852023-11-29T20:25:01.394Z+++ grep oxz_ntp
8862023-11-29T20:25:01.395Z++ pfexec svcs -z oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9 -Hostate oxide/ntp
8872023-11-29T20:25:01.395Z+ [[ online != online ]]
8882023-11-29T20:25:01.395Z+ echo 'Waited for chrony: 15s'
8892023-11-29T20:25:01.395ZWaited for chrony: 15s
8902023-11-29T20:25:01.395Z+ export RUST_BACKTRACE=1
8912023-11-29T20:25:01.395Z+ RUST_BACKTRACE=1
8922023-11-29T20:25:01.395Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8932023-11-29T20:25:01.395Z+ ./tests/bootstrap
8942023-11-29T20:25:01.419ZUsing external DNS server at 10.151.1.164:53
8952023-11-29T20:25:01.419Z2023-11-29 20:25:01.648837563 UTC: attempting to log into API
8962023-11-29T20:25:16.456Z2023-11-29 20:25:16.532601765 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8972023-11-29T20:25:17.457Z2023-11-29 20:25:17.534295883 UTC: attempting to log into API
8982023-11-29T20:25:32.490Z2023-11-29 20:25:32.567462479 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8992023-11-29T20:25:33.491Z2023-11-29 20:25:33.568353027 UTC: attempting to log into API
9002023-11-29T20:25:48.524Z2023-11-29 20:25:48.601534404 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9012023-11-29T20:25:49.525Z2023-11-29 20:25:49.602460642 UTC: attempting to log into API
9022023-11-29T20:26:04.559Z2023-11-29 20:26:04.635241059 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9032023-11-29T20:26:05.559Z2023-11-29 20:26:05.635941302 UTC: attempting to log into API
9042023-11-29T20:26:20.591Z2023-11-29 20:26:20.644444527 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9052023-11-29T20:26:21.592Z2023-11-29 20:26:21.642653746 UTC: attempting to log into API
9062023-11-29T20:26:36.625Z2023-11-29 20:26:36.634847286 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9072023-11-29T20:26:37.626Z2023-11-29 20:26:37.634017449 UTC: attempting to log into API
9082023-11-29T20:26:52.660Z2023-11-29 20:26:52.649847434 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9092023-11-29T20:26:53.662Z2023-11-29 20:26:53.651458969 UTC: attempting to log into API
9102023-11-29T20:27:09.060Z2023-11-29 20:27:08.684645616 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9112023-11-29T20:27:09.696Z2023-11-29 20:27:09.686371505 UTC: attempting to log into API
9122023-11-29T20:27:24.729Z2023-11-29 20:27:24.718602126 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9132023-11-29T20:27:25.729Z2023-11-29 20:27:25.719497534 UTC: attempting to log into API
9142023-11-29T20:27:40.763Z2023-11-29 20:27:40.752502187 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9152023-11-29T20:27:41.763Z2023-11-29 20:27:41.752819940 UTC: attempting to log into API
9162023-11-29T20:27:56.795Z2023-11-29 20:27:56.785103065 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9172023-11-29T20:27:57.796Z2023-11-29 20:27:57.786254090 UTC: attempting to log into API
9182023-11-29T20:28:12.828Z2023-11-29 20:28:12.818342390 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9192023-11-29T20:28:13.830Z2023-11-29 20:28:13.819584836 UTC: attempting to log into API
9202023-11-29T20:28:28.877Z2023-11-29 20:28:28.861518665 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9212023-11-29T20:28:29.874Z2023-11-29 20:28:29.862389132 UTC: attempting to log into API
 
9222023-11-29T20:28:36.990ZNov 29 20:28:36 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9232023-11-29T20:28:43.601ZNov 29 20:28:42 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9242023-11-29T20:28:44.925Z2023-11-29 20:28:44.913590285 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9252023-11-29T20:28:45.940Z2023-11-29 20:28:45.915493898 UTC: attempting to log into API
 
9262023-11-29T20:28:50.188ZNov 29 20:28:49 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9272023-11-29T20:28:56.256ZNov 29 20:28:55 bukin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9282023-11-29T20:29:00.997Z2023-11-29 20:29:00.947865523 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9292023-11-29T20:29:01.816ZNov 29 20:29:01 bukin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9302023-11-29T20:29:01.963Z2023-11-29 20:29:01.948957076 UTC: attempting to log into API
9312023-11-29T20:29:12.782Z2023-11-29 20:29:12.752996380 UTC: login succeeded
9322023-11-29T20:29:12.811Zwaiting for nexus to come up...
9332023-11-29T20:29:12.866Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9342023-11-29T20:29:12.918Zensuring datasets are ready...
9352023-11-29T20:29:15.836Zlet's roll.
9362023-11-29T20:29:15.859Z+ rm ./tests/bootstrap
9372023-11-29T20:29:15.930Z+ for test_bin in tests/*
9382023-11-29T20:29:15.930Z+ ./tests/bootstrap-8d69322849e0fa66
9392023-11-29T20:29:15.954Z
9402023-11-29T20:29:15.954Zrunning 0 tests
9412023-11-29T20:29:15.954Z
9422023-11-29T20:29:15.954Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9432023-11-29T20:29:15.954Z
9442023-11-29T20:29:15.954Z+ for test_bin in tests/*
9452023-11-29T20:29:15.954Z+ ./tests/end_to_end_tests-7d7c590276acd289
9462023-11-29T20:29:15.977Z
9472023-11-29T20:29:15.977Zrunning 2 tests
9482023-11-29T20:29:16.169Ztest no_spoof::no_spoof_here ... ok
9492023-11-29T20:30:15.955Ztest instance_launch::instance_launch has been running for over 60 seconds
9502023-11-29T20:32:11.685Ztest instance_launch::instance_launch ... ok
9512023-11-29T20:32:11.685Z
9522023-11-29T20:32:11.685Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 175.73s
9532023-11-29T20:32:11.685Z
9542023-11-29T20:32:11.709Z+ _exit_trap
9552023-11-29T20:32:11.710Z+ local status=0
9562023-11-29T20:32:11.710Z+ [[ 0 -eq 0 ]]
9572023-11-29T20:32:11.710Z+ exit 0
9582023-11-29T20:32:11.710Zprocess exited: duration 504700 ms, exit code 0
 
9592023-11-29T20:32:11.811Zfound 90 output files
9602023-11-29T20:32:11.811Zuploading: /var/svc/log/oxide-sled-agent:default.log (744792 bytes)
9612023-11-29T20:32:12.089Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 744792 -> 745191
9622023-11-29T20:32:13.216Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9632023-11-29T20:32:13.216Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d439675e-0ddd-4713-a123-b78150e9f04e/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9642023-11-29T20:32:14.247Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d439675e-0ddd-4713-a123-b78150e9f04e/root/var/svc/log/oxide-crucible-agent:default.log
9652023-11-29T20:32:14.248Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d439675e-0ddd-4713-a123-b78150e9f04e/root/var/svc/log/oxide-crucible-agent:default.log.0 (14107 bytes)
9662023-11-29T20:32:15.365Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d439675e-0ddd-4713-a123-b78150e9f04e/root/var/svc/log/oxide-crucible-agent:default.log.0
9672023-11-29T20:32:15.388Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d439675e-0ddd-4713-a123-b78150e9f04e/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9682023-11-29T20:32:16.437Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d439675e-0ddd-4713-a123-b78150e9f04e/root/var/svc/log/oxide-crucible-downstairs:default.log
9692023-11-29T20:32:16.437Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d439675e-0ddd-4713-a123-b78150e9f04e/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9702023-11-29T20:32:17.554Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d439675e-0ddd-4713-a123-b78150e9f04e/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9712023-11-29T20:32:17.554Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d439675e-0ddd-4713-a123-b78150e9f04e/root/var/svc/log/oxide-crucible-downstairs:downstairs-bf25a7cb-8581-44e0-b6ba-36b06aa8f601.log (0 bytes)
9722023-11-29T20:32:18.626Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d439675e-0ddd-4713-a123-b78150e9f04e/root/var/svc/log/oxide-crucible-downstairs:downstairs-bf25a7cb-8581-44e0-b6ba-36b06aa8f601.log
9732023-11-29T20:32:18.626Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d439675e-0ddd-4713-a123-b78150e9f04e/root/var/svc/log/oxide-crucible-downstairs:downstairs-bf25a7cb-8581-44e0-b6ba-36b06aa8f601.log.0 (3051 bytes)
9742023-11-29T20:32:19.720Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d439675e-0ddd-4713-a123-b78150e9f04e/root/var/svc/log/oxide-crucible-downstairs:downstairs-bf25a7cb-8581-44e0-b6ba-36b06aa8f601.log.0
9752023-11-29T20:32:19.720Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e51e68b3-48d6-43b9-84c9-34c5c3f461f0/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9762023-11-29T20:32:20.792Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e51e68b3-48d6-43b9-84c9-34c5c3f461f0/root/var/svc/log/oxide-crucible-agent:default.log
9772023-11-29T20:32:20.817Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e51e68b3-48d6-43b9-84c9-34c5c3f461f0/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
9782023-11-29T20:32:22.479Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e51e68b3-48d6-43b9-84c9-34c5c3f461f0/root/var/svc/log/oxide-crucible-agent:default.log.0
9792023-11-29T20:32:22.480Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e51e68b3-48d6-43b9-84c9-34c5c3f461f0/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9802023-11-29T20:32:23.551Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e51e68b3-48d6-43b9-84c9-34c5c3f461f0/root/var/svc/log/oxide-crucible-downstairs:default.log
9812023-11-29T20:32:23.551Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e51e68b3-48d6-43b9-84c9-34c5c3f461f0/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9822023-11-29T20:32:24.646Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e51e68b3-48d6-43b9-84c9-34c5c3f461f0/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9832023-11-29T20:32:24.646Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_9fef3e37-39fe-4c64-b9f9-1449384896fe/root/var/svc/log/oxide-external_dns:default.log (11326 bytes)
9842023-11-29T20:32:25.823Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_9fef3e37-39fe-4c64-b9f9-1449384896fe/root/var/svc/log/oxide-external_dns:default.log
9852023-11-29T20:32:25.823Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_9fef3e37-39fe-4c64-b9f9-1449384896fe/root/var/svc/log/oxide-external_dns:default.log.0 (4283 bytes)
9862023-11-29T20:32:26.917Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_9fef3e37-39fe-4c64-b9f9-1449384896fe/root/var/svc/log/oxide-external_dns:default.log.0
9872023-11-29T20:32:26.943Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_763de280-084c-4c51-a117-e30cd5c44e55/root/var/svc/log/oxide-nexus:default.log (339118 bytes)
9882023-11-29T20:32:27.562Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_763de280-084c-4c51-a117-e30cd5c44e55/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 339118 -> 340334
9892023-11-29T20:32:28.632Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_763de280-084c-4c51-a117-e30cd5c44e55/root/var/svc/log/oxide-nexus:default.log
9902023-11-29T20:32:28.632Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_763de280-084c-4c51-a117-e30cd5c44e55/root/var/svc/log/oxide-nexus:default.log.0 (355756 bytes)
9912023-11-29T20:32:30.799Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_763de280-084c-4c51-a117-e30cd5c44e55/root/var/svc/log/oxide-nexus:default.log.0
9922023-11-29T20:32:30.799Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9da0cdd9-e775-4812-978a-d3d0994abdcd/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9932023-11-29T20:32:31.870Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9da0cdd9-e775-4812-978a-d3d0994abdcd/root/var/svc/log/oxide-cockroachdb:default.log
9942023-11-29T20:32:31.870Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9da0cdd9-e775-4812-978a-d3d0994abdcd/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
9952023-11-29T20:32:32.964Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9da0cdd9-e775-4812-978a-d3d0994abdcd/root/var/svc/log/oxide-cockroachdb:default.log.0
9962023-11-29T20:32:32.964Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_62dab281-3e87-49cc-8694-9eb0aa8d4f68/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
9972023-11-29T20:32:34.061Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_62dab281-3e87-49cc-8694-9eb0aa8d4f68/root/var/svc/log/oxide-crucible-pantry:default.log
9982023-11-29T20:32:34.061Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_62dab281-3e87-49cc-8694-9eb0aa8d4f68/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
9992023-11-29T20:32:35.165Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_62dab281-3e87-49cc-8694-9eb0aa8d4f68/root/var/svc/log/oxide-crucible-pantry:default.log.0
10002023-11-29T20:32:35.165Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_7d5eb78f-584a-4f88-9f3f-4e74f3d36c94/root/var/svc/log/oxide-internal_dns:default.log (11264 bytes)
10012023-11-29T20:32:36.284Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_7d5eb78f-584a-4f88-9f3f-4e74f3d36c94/root/var/svc/log/oxide-internal_dns:default.log
10022023-11-29T20:32:36.284Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_7d5eb78f-584a-4f88-9f3f-4e74f3d36c94/root/var/svc/log/oxide-internal_dns:default.log.0 (104942 bytes)
10032023-11-29T20:32:37.680Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_7d5eb78f-584a-4f88-9f3f-4e74f3d36c94/root/var/svc/log/oxide-internal_dns:default.log.0
10042023-11-29T20:32:37.704Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f626e9c2-a47b-4aec-be19-f8291db30246/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10052023-11-29T20:32:38.750Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f626e9c2-a47b-4aec-be19-f8291db30246/root/var/svc/log/oxide-cockroachdb:default.log
10062023-11-29T20:32:38.750Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f626e9c2-a47b-4aec-be19-f8291db30246/root/var/svc/log/oxide-cockroachdb:default.log.0 (5370 bytes)
10072023-11-29T20:32:39.842Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f626e9c2-a47b-4aec-be19-f8291db30246/root/var/svc/log/oxide-cockroachdb:default.log.0
10082023-11-29T20:32:39.843Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_67b14861-27dd-49a9-8bbc-20c2a1f99a0a/root/var/svc/log/oxide-crucible-agent:default.log (4323 bytes)
10092023-11-29T20:32:40.934Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_67b14861-27dd-49a9-8bbc-20c2a1f99a0a/root/var/svc/log/oxide-crucible-agent:default.log
10102023-11-29T20:32:40.935Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_67b14861-27dd-49a9-8bbc-20c2a1f99a0a/root/var/svc/log/oxide-crucible-agent:default.log.0 (9365 bytes)
10112023-11-29T20:32:42.050Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_67b14861-27dd-49a9-8bbc-20c2a1f99a0a/root/var/svc/log/oxide-crucible-agent:default.log.0
10122023-11-29T20:32:42.075Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_67b14861-27dd-49a9-8bbc-20c2a1f99a0a/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10132023-11-29T20:32:43.120Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_67b14861-27dd-49a9-8bbc-20c2a1f99a0a/root/var/svc/log/oxide-crucible-downstairs:default.log
10142023-11-29T20:32:43.120Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_67b14861-27dd-49a9-8bbc-20c2a1f99a0a/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10152023-11-29T20:32:44.214Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_67b14861-27dd-49a9-8bbc-20c2a1f99a0a/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10162023-11-29T20:32:44.214Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_67b14861-27dd-49a9-8bbc-20c2a1f99a0a/root/var/svc/log/oxide-crucible-downstairs:downstairs-24ba5c47-7c93-4aef-afda-c98ce46a9e57.log (1004 bytes)
10172023-11-29T20:32:45.308Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_67b14861-27dd-49a9-8bbc-20c2a1f99a0a/root/var/svc/log/oxide-crucible-downstairs:downstairs-24ba5c47-7c93-4aef-afda-c98ce46a9e57.log
10182023-11-29T20:32:45.309Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_67b14861-27dd-49a9-8bbc-20c2a1f99a0a/root/var/svc/log/oxide-crucible-downstairs:downstairs-24ba5c47-7c93-4aef-afda-c98ce46a9e57.log.0 (4525 bytes)
10192023-11-29T20:32:46.421Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_67b14861-27dd-49a9-8bbc-20c2a1f99a0a/root/var/svc/log/oxide-crucible-downstairs:downstairs-24ba5c47-7c93-4aef-afda-c98ce46a9e57.log.0
10202023-11-29T20:32:46.422Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-agent:default.log (4320 bytes)
10212023-11-29T20:32:47.514Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-agent:default.log
10222023-11-29T20:32:47.537Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-agent:default.log.0 (20507 bytes)
10232023-11-29T20:32:48.680Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-agent:default.log.0
10242023-11-29T20:32:48.705Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10252023-11-29T20:32:49.760Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-downstairs:default.log
10262023-11-29T20:32:49.760Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10272023-11-29T20:32:50.852Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10282023-11-29T20:32:50.852Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-downstairs:downstairs-7be35dfa-9c29-42f8-8c97-f088bd2d3201.log (1006 bytes)
10292023-11-29T20:32:51.944Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-downstairs:downstairs-7be35dfa-9c29-42f8-8c97-f088bd2d3201.log
10302023-11-29T20:32:51.944Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-downstairs:downstairs-7be35dfa-9c29-42f8-8c97-f088bd2d3201.log.0 (4531 bytes)
10312023-11-29T20:32:53.037Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-downstairs:downstairs-7be35dfa-9c29-42f8-8c97-f088bd2d3201.log.0
10322023-11-29T20:32:53.037Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-downstairs:downstairs-c9611241-4105-4538-a476-b264fa5b6c4c.log (0 bytes)
10332023-11-29T20:32:54.531Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-downstairs:downstairs-c9611241-4105-4538-a476-b264fa5b6c4c.log
10342023-11-29T20:32:54.531Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-downstairs:downstairs-c9611241-4105-4538-a476-b264fa5b6c4c.log.0 (3047 bytes)
10352023-11-29T20:32:55.623Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_bd233faf-5f94-4005-b731-2a5932771d2e/root/var/svc/log/oxide-crucible-downstairs:downstairs-c9611241-4105-4538-a476-b264fa5b6c4c.log.0
10362023-11-29T20:32:55.648Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_22dfb67d-fd16-45a6-8bab-05c878d3dd40/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10372023-11-29T20:32:56.693Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_22dfb67d-fd16-45a6-8bab-05c878d3dd40/root/var/svc/log/oxide-cockroachdb:default.log
10382023-11-29T20:32:56.693Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_22dfb67d-fd16-45a6-8bab-05c878d3dd40/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
10392023-11-29T20:32:57.817Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_22dfb67d-fd16-45a6-8bab-05c878d3dd40/root/var/svc/log/oxide-cockroachdb:default.log.0
10402023-11-29T20:32:57.817Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_2612f9ca-ff5d-4908-a32e-1daa76e9267b/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10412023-11-29T20:32:59.295Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_2612f9ca-ff5d-4908-a32e-1daa76e9267b/root/var/svc/log/oxide-cockroachdb:default.log
10422023-11-29T20:32:59.295Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_2612f9ca-ff5d-4908-a32e-1daa76e9267b/root/var/svc/log/oxide-cockroachdb:default.log.0 (5370 bytes)
10432023-11-29T20:33:00.389Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_2612f9ca-ff5d-4908-a32e-1daa76e9267b/root/var/svc/log/oxide-cockroachdb:default.log.0
10442023-11-29T20:33:00.389Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b0c1c157-3fc0-4aa2-948c-acf5162a654f/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10452023-11-29T20:33:01.459Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b0c1c157-3fc0-4aa2-948c-acf5162a654f/root/var/svc/log/oxide-crucible-agent:default.log
10462023-11-29T20:33:01.459Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b0c1c157-3fc0-4aa2-948c-acf5162a654f/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10472023-11-29T20:33:02.560Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b0c1c157-3fc0-4aa2-948c-acf5162a654f/root/var/svc/log/oxide-crucible-agent:default.log.0
10482023-11-29T20:33:02.560Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b0c1c157-3fc0-4aa2-948c-acf5162a654f/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10492023-11-29T20:33:03.634Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b0c1c157-3fc0-4aa2-948c-acf5162a654f/root/var/svc/log/oxide-crucible-downstairs:default.log
10502023-11-29T20:33:03.634Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b0c1c157-3fc0-4aa2-948c-acf5162a654f/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10512023-11-29T20:33:04.729Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b0c1c157-3fc0-4aa2-948c-acf5162a654f/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10522023-11-29T20:33:04.753Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_8f0119f9-25f1-44b3-b825-ae5bc7212bb6/root/var/svc/log/oxide-external_dns:default.log (11333 bytes)
10532023-11-29T20:33:04.797Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_8f0119f9-25f1-44b3-b825-ae5bc7212bb6/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 11333 -> 15110
10542023-11-29T20:33:05.866Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_8f0119f9-25f1-44b3-b825-ae5bc7212bb6/root/var/svc/log/oxide-external_dns:default.log
10552023-11-29T20:33:05.866Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_8f0119f9-25f1-44b3-b825-ae5bc7212bb6/root/var/svc/log/oxide-external_dns:default.log.0 (4281 bytes)
10562023-11-29T20:33:06.959Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_8f0119f9-25f1-44b3-b825-ae5bc7212bb6/root/var/svc/log/oxide-external_dns:default.log.0
10572023-11-29T20:33:06.983Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_ebc571a0-fbcd-42bf-b2d2-67a4a77711e7/root/var/svc/log/oxide-nexus:default.log (339120 bytes)
10582023-11-29T20:33:07.184Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_ebc571a0-fbcd-42bf-b2d2-67a4a77711e7/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 339120 -> 408046
10592023-11-29T20:33:08.254Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_ebc571a0-fbcd-42bf-b2d2-67a4a77711e7/root/var/svc/log/oxide-nexus:default.log
10602023-11-29T20:33:08.254Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_ebc571a0-fbcd-42bf-b2d2-67a4a77711e7/root/var/svc/log/oxide-nexus:default.log.0 (541164 bytes)
10612023-11-29T20:33:09.646Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_ebc571a0-fbcd-42bf-b2d2-67a4a77711e7/root/var/svc/log/oxide-nexus:default.log.0
10622023-11-29T20:33:09.646Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9fa0a121-bb07-4627-ac0e-4ac22848433a/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10632023-11-29T20:33:10.716Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9fa0a121-bb07-4627-ac0e-4ac22848433a/root/var/svc/log/oxide-crucible-agent:default.log
10642023-11-29T20:33:10.740Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9fa0a121-bb07-4627-ac0e-4ac22848433a/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10652023-11-29T20:33:11.809Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9fa0a121-bb07-4627-ac0e-4ac22848433a/root/var/svc/log/oxide-crucible-agent:default.log.0
10662023-11-29T20:33:11.834Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9fa0a121-bb07-4627-ac0e-4ac22848433a/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10672023-11-29T20:33:12.878Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9fa0a121-bb07-4627-ac0e-4ac22848433a/root/var/svc/log/oxide-crucible-downstairs:default.log
10682023-11-29T20:33:12.878Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9fa0a121-bb07-4627-ac0e-4ac22848433a/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10692023-11-29T20:33:13.972Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_9fa0a121-bb07-4627-ac0e-4ac22848433a/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10702023-11-29T20:33:13.972Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c7407b19-f62b-49bd-85e4-6b75a99bcac5/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10712023-11-29T20:33:15.047Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c7407b19-f62b-49bd-85e4-6b75a99bcac5/root/var/svc/log/oxide-crucible-agent:default.log
10722023-11-29T20:33:15.047Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c7407b19-f62b-49bd-85e4-6b75a99bcac5/root/var/svc/log/oxide-crucible-agent:default.log.0 (14579 bytes)
10732023-11-29T20:33:16.185Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c7407b19-f62b-49bd-85e4-6b75a99bcac5/root/var/svc/log/oxide-crucible-agent:default.log.0
10742023-11-29T20:33:16.185Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c7407b19-f62b-49bd-85e4-6b75a99bcac5/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10752023-11-29T20:33:17.259Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c7407b19-f62b-49bd-85e4-6b75a99bcac5/root/var/svc/log/oxide-crucible-downstairs:default.log
10762023-11-29T20:33:17.260Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c7407b19-f62b-49bd-85e4-6b75a99bcac5/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10772023-11-29T20:33:18.393Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c7407b19-f62b-49bd-85e4-6b75a99bcac5/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10782023-11-29T20:33:18.393Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c7407b19-f62b-49bd-85e4-6b75a99bcac5/root/var/svc/log/oxide-crucible-downstairs:downstairs-2143bf71-8a60-4b8b-a357-ffd1db2f8a94.log (0 bytes)
10792023-11-29T20:33:19.879Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c7407b19-f62b-49bd-85e4-6b75a99bcac5/root/var/svc/log/oxide-crucible-downstairs:downstairs-2143bf71-8a60-4b8b-a357-ffd1db2f8a94.log
10802023-11-29T20:33:19.879Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c7407b19-f62b-49bd-85e4-6b75a99bcac5/root/var/svc/log/oxide-crucible-downstairs:downstairs-2143bf71-8a60-4b8b-a357-ffd1db2f8a94.log.0 (3050 bytes)
10812023-11-29T20:33:20.975Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_c7407b19-f62b-49bd-85e4-6b75a99bcac5/root/var/svc/log/oxide-crucible-downstairs:downstairs-2143bf71-8a60-4b8b-a357-ffd1db2f8a94.log.0
10822023-11-29T20:33:20.975Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_63542db9-64dc-4b5c-8f5a-7ff376bcf363/root/var/svc/log/oxide-clickhouse:default.log (399222 bytes)
10832023-11-29T20:33:22.370Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_63542db9-64dc-4b5c-8f5a-7ff376bcf363/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 399222 -> 603007
10842023-11-29T20:33:23.441Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_63542db9-64dc-4b5c-8f5a-7ff376bcf363/root/var/svc/log/oxide-clickhouse:default.log
10852023-11-29T20:33:23.441Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_63542db9-64dc-4b5c-8f5a-7ff376bcf363/root/var/svc/log/oxide-clickhouse:default.log.0 (261214 bytes)
10862023-11-29T20:33:25.155Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_63542db9-64dc-4b5c-8f5a-7ff376bcf363/root/var/svc/log/oxide-clickhouse:default.log.0
10872023-11-29T20:33:25.179Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_91431279-d7f9-4d0c-9e37-54926296721e/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10882023-11-29T20:33:40.463Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_91431279-d7f9-4d0c-9e37-54926296721e/root/var/svc/log/oxide-crucible-pantry:default.log
10892023-11-29T20:33:40.786Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_91431279-d7f9-4d0c-9e37-54926296721e/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10902023-11-29T20:33:41.951Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_91431279-d7f9-4d0c-9e37-54926296721e/root/var/svc/log/oxide-crucible-pantry:default.log.0
10912023-11-29T20:33:41.982Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_dc8544e3-54c2-4c74-a6ae-0af4683bb08a/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10922023-11-29T20:33:43.045Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_dc8544e3-54c2-4c74-a6ae-0af4683bb08a/root/var/svc/log/oxide-crucible-pantry:default.log
10932023-11-29T20:33:43.045Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_dc8544e3-54c2-4c74-a6ae-0af4683bb08a/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10942023-11-29T20:33:44.167Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_dc8544e3-54c2-4c74-a6ae-0af4683bb08a/root/var/svc/log/oxide-crucible-pantry:default.log.0
10952023-11-29T20:33:44.167Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_c8723a5e-a944-41ad-91cd-baca14b4d426/root/var/svc/log/oxide-internal_dns:default.log (11268 bytes)
10962023-11-29T20:33:44.292Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_c8723a5e-a944-41ad-91cd-baca14b4d426/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 11268 -> 15025
10972023-11-29T20:33:45.371Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_c8723a5e-a944-41ad-91cd-baca14b4d426/root/var/svc/log/oxide-internal_dns:default.log
10982023-11-29T20:33:45.371Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_c8723a5e-a944-41ad-91cd-baca14b4d426/root/var/svc/log/oxide-internal_dns:default.log.0 (103412 bytes)
10992023-11-29T20:33:46.741Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_c8723a5e-a944-41ad-91cd-baca14b4d426/root/var/svc/log/oxide-internal_dns:default.log.0
11002023-11-29T20:33:46.769Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_72114e38-14e5-4752-be87-331063511174/root/var/svc/log/oxide-crucible-agent:default.log (4323 bytes)
11012023-11-29T20:33:47.862Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_72114e38-14e5-4752-be87-331063511174/root/var/svc/log/oxide-crucible-agent:default.log
11022023-11-29T20:33:47.862Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_72114e38-14e5-4752-be87-331063511174/root/var/svc/log/oxide-crucible-agent:default.log.0 (9793 bytes)
11032023-11-29T20:33:49.022Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_72114e38-14e5-4752-be87-331063511174/root/var/svc/log/oxide-crucible-agent:default.log.0
11042023-11-29T20:33:49.053Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_72114e38-14e5-4752-be87-331063511174/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11052023-11-29T20:33:50.101Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_72114e38-14e5-4752-be87-331063511174/root/var/svc/log/oxide-crucible-downstairs:default.log
11062023-11-29T20:33:50.127Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_72114e38-14e5-4752-be87-331063511174/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11072023-11-29T20:33:51.645Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_72114e38-14e5-4752-be87-331063511174/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11082023-11-29T20:33:51.645Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_72114e38-14e5-4752-be87-331063511174/root/var/svc/log/oxide-crucible-downstairs:downstairs-90448ebe-2436-4a3d-8694-48adbf123ab9.log (1005 bytes)
11092023-11-29T20:33:53.142Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_72114e38-14e5-4752-be87-331063511174/root/var/svc/log/oxide-crucible-downstairs:downstairs-90448ebe-2436-4a3d-8694-48adbf123ab9.log
11102023-11-29T20:33:53.169Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_72114e38-14e5-4752-be87-331063511174/root/var/svc/log/oxide-crucible-downstairs:downstairs-90448ebe-2436-4a3d-8694-48adbf123ab9.log.0 (4530 bytes)
11112023-11-29T20:33:54.249Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_72114e38-14e5-4752-be87-331063511174/root/var/svc/log/oxide-crucible-downstairs:downstairs-90448ebe-2436-4a3d-8694-48adbf123ab9.log.0
11122023-11-29T20:33:54.249Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b9c084c6-61ef-43de-b319-cc1da37d0d81/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11132023-11-29T20:33:55.321Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b9c084c6-61ef-43de-b319-cc1da37d0d81/root/var/svc/log/oxide-crucible-agent:default.log
11142023-11-29T20:33:55.321Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b9c084c6-61ef-43de-b319-cc1da37d0d81/root/var/svc/log/oxide-crucible-agent:default.log.0 (3434 bytes)
11152023-11-29T20:33:56.415Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b9c084c6-61ef-43de-b319-cc1da37d0d81/root/var/svc/log/oxide-crucible-agent:default.log.0
11162023-11-29T20:33:56.443Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b9c084c6-61ef-43de-b319-cc1da37d0d81/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11172023-11-29T20:33:57.495Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b9c084c6-61ef-43de-b319-cc1da37d0d81/root/var/svc/log/oxide-crucible-downstairs:default.log
11182023-11-29T20:33:57.495Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b9c084c6-61ef-43de-b319-cc1da37d0d81/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11192023-11-29T20:33:58.592Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b9c084c6-61ef-43de-b319-cc1da37d0d81/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11202023-11-29T20:33:58.592Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b6528faf-01ff-4e12-98e6-c7a2f0946a05/root/var/svc/log/oxide-internal_dns:default.log (11263 bytes)
11212023-11-29T20:33:58.660Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b6528faf-01ff-4e12-98e6-c7a2f0946a05/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 11263 -> 15020
11222023-11-29T20:33:59.733Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b6528faf-01ff-4e12-98e6-c7a2f0946a05/root/var/svc/log/oxide-internal_dns:default.log
11232023-11-29T20:33:59.740Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b6528faf-01ff-4e12-98e6-c7a2f0946a05/root/var/svc/log/oxide-internal_dns:default.log.0 (104633 bytes)
11242023-11-29T20:34:00.936Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b6528faf-01ff-4e12-98e6-c7a2f0946a05/root/var/svc/log/oxide-internal_dns:default.log.0
11252023-11-29T20:34:00.936Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_00a91b23-03f1-47d2-9c3e-800970be4182/root/var/svc/log/oxide-oximeter:default.log (113679 bytes)
11262023-11-29T20:34:01.148Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_00a91b23-03f1-47d2-9c3e-800970be4182/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 113679 -> 195535
11272023-11-29T20:34:02.229Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_00a91b23-03f1-47d2-9c3e-800970be4182/root/var/svc/log/oxide-oximeter:default.log
11282023-11-29T20:34:02.229Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_00a91b23-03f1-47d2-9c3e-800970be4182/root/var/svc/log/oxide-oximeter:default.log.0 (54744 bytes)
11292023-11-29T20:34:03.412Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_00a91b23-03f1-47d2-9c3e-800970be4182/root/var/svc/log/oxide-oximeter:default.log.0
11302023-11-29T20:34:03.442Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1549fd46-7fa7-499f-8144-8a4dc1a7365e/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11312023-11-29T20:34:04.483Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1549fd46-7fa7-499f-8144-8a4dc1a7365e/root/var/svc/log/oxide-cockroachdb:default.log
11322023-11-29T20:34:04.484Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1549fd46-7fa7-499f-8144-8a4dc1a7365e/root/var/svc/log/oxide-cockroachdb:default.log.0 (5343 bytes)
11332023-11-29T20:34:05.580Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1549fd46-7fa7-499f-8144-8a4dc1a7365e/root/var/svc/log/oxide-cockroachdb:default.log.0
11342023-11-29T20:34:05.580Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_178c1718-3251-46c6-af83-a62b7359374f/root/var/svc/log/oxide-nexus:default.log (1575085 bytes)
11352023-11-29T20:34:07.463Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_178c1718-3251-46c6-af83-a62b7359374f/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1575085 -> 1717727
11362023-11-29T20:34:08.543Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_178c1718-3251-46c6-af83-a62b7359374f/root/var/svc/log/oxide-nexus:default.log
11372023-11-29T20:34:08.543Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_178c1718-3251-46c6-af83-a62b7359374f/root/var/svc/log/oxide-nexus:default.log.0 (2690737 bytes)
11382023-11-29T20:34:12.483Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_178c1718-3251-46c6-af83-a62b7359374f/root/var/svc/log/oxide-nexus:default.log.0
11392023-11-29T20:34:12.483Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9/root/var/svc/log/oxide-ntp:default.log (70 bytes)
11402023-11-29T20:34:14.644Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9/root/var/svc/log/oxide-ntp:default.log
11412023-11-29T20:34:14.644Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9/root/var/svc/log/oxide-ntp:default.log.0 (1048 bytes)
11422023-11-29T20:34:15.744Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9/root/var/svc/log/oxide-ntp:default.log.0
11432023-11-29T20:34:15.744Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9/root/var/log/chrony/measurements.log (15070 bytes)
11442023-11-29T20:34:15.836Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9/root/var/log/chrony/measurements.log" changed size mid upload: 15070 -> 19180
11452023-11-29T20:34:16.906Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9/root/var/log/chrony/measurements.log
11462023-11-29T20:34:16.906Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9/root/var/log/chrony/statistics.log (6903 bytes)
11472023-11-29T20:34:16.975Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9/root/var/log/chrony/statistics.log" changed size mid upload: 6903 -> 9828
11482023-11-29T20:34:18.054Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9/root/var/log/chrony/statistics.log
11492023-11-29T20:34:18.054Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9/root/var/log/chrony/tracking.log (1721 bytes)
11502023-11-29T20:34:18.515Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9/root/var/log/chrony/tracking.log" changed size mid upload: 1721 -> 1985
11512023-11-29T20:34:19.596Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_94322fdc-314c-4c73-a729-f5fe4d49c3d9/root/var/log/chrony/tracking.log