01HGF6JF2FYH1HBYGTK8V0XNFR: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HGF6K8GSDN454TRAR3Q2MZSB

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-30T04:53:22.776Zjob dependencies complete; ready to run (waiting for 45 m 14 s)
22023-11-30T04:57:05.714Zjob assigned to worker 01HGF9695JBSTWTH2KP064EXVQ (queued for 3 m 42 s)
32023-11-30T04:57:05.775Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-11-30T04:57:05.828Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-11-30T04:57:05.871Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-11-30T04:57:05.908Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-30T04:57:05.949Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-11-30T04:57:05.985ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-30T04:57:06.028Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-30T04:57:06.067ZOxide Helios Version helios-2.0.22318 64-bit
112023-11-30T04:57:06.108ZConfiguring device.
122023-11-30T04:57:06.153Z#################################################
132023-11-30T04:57:06.193Z#### ############# ##
142023-11-30T04:57:06.247Z### ### ############ ##
152023-11-30T04:57:06.282Z## ### # ## ### ## Oxide Computer Company ##
162023-11-30T04:57:06.315Z## ## ## ### # ### ##
172023-11-30T04:57:06.349Z## # ### #### #### This Station Under ##
182023-11-30T04:57:06.382Z### ### #### # ### Computer Control ##
192023-11-30T04:57:06.416Z#### #### ### ## ##
202023-11-30T04:57:06.450Z##################################################
212023-11-30T04:57:06.483ZHostname: sock
222023-11-30T04:57:06.522ZLoading smf(7) serice descriptions:
232023-11-30T04:57:07.541Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-8d69322849e0fa66.gz
242023-11-30T04:57:10.311Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-8d69322849e0fa66.gz
252023-11-30T04:57:10.311Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-30T04:57:31.263Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-30T04:57:31.263Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7d7c590276acd289.gz
282023-11-30T04:57:53.687Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7d7c590276acd289.gz
292023-11-30T04:57:53.713Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-30T04:57:54.648Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-30T04:57:54.648Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-30T04:57:58.880Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-30T04:57:58.904Zdownloading input: /input/package/work/version.txt
342023-11-30T04:57:58.944Zdownloaded input: /input/package/work/version.txt
352023-11-30T04:57:58.944Zdownloading input: /input/package/work/package.tar.gz
362023-11-30T04:58:17.982Zdownloaded input: /input/package/work/package.tar.gz
372023-11-30T04:58:18.006Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-30T04:58:42.657Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-30T04:58:42.657Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-30T04:58:49.999Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-30T04:58:50.025Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-30T04:59:34.694Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-30T04:59:34.694Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-30T05:00:14.680Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-30T05:00:14.686Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-30T05:00:37.418Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-30T05:00:37.419Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-30T05:00:42.167Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-30T05:00:42.167Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-30T05:00:50.064Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-30T05:00:50.066Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-30T05:00:55.336Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-30T05:00:55.336Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-30T05:00:59.784Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-30T05:00:59.784Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-30T05:00:59.822Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-30T05:00:59.822Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-30T05:01:03.296Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-30T05:01:03.296Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-30T05:01:07.003Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-30T05:01:07.003Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-30T05:01:20.738Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-30T05:01:20.738Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-30T05:01:34.628Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-30T05:01:34.628Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-30T05:01:34.664Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-30T05:01:34.687Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-30T05:01:40.923Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-30T05:01:40.923Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-30T05:01:53.817Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-30T05:01:53.817Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-30T05:02:40.295Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-30T05:02:40.295Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-30T05:03:12.041Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-30T05:03:12.065Zstarting task 0: "setup"
762023-11-30T05:03:12.065Z++ uname -s
772023-11-30T05:03:12.065Z+ kern=SunOS
782023-11-30T05:03:12.066Z+ case "$kern" in
792023-11-30T05:03:12.066Z+ groupadd -g 12345 build
802023-11-30T05:03:12.066Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-30T05:03:14.059Z+ zfs create -o mountpoint=/work rpool/work
822023-11-30T05:03:14.094Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-30T05:03:14.118Z+ home_fs=zfs
842023-11-30T05:03:14.119Z+ [[ zfs == autofs ]]
852023-11-30T05:03:14.119Z+ mkdir -p /home/build
862023-11-30T05:03:14.119Z+ chown build:build /home/build /work
872023-11-30T05:03:16.103Z+ chmod 0700 /home/build /work
882023-11-30T05:03:16.127Zprocess exited: duration 4065 ms, exit code 0
 
892023-11-30T05:03:16.176Zstarting task 1: "authentication"
902023-11-30T05:03:16.219Zprocess exited: duration 43 ms, exit code 0
 
912023-11-30T05:03:16.266Zstarting task 2: "build"
922023-11-30T05:03:16.289Z+ trap _exit_trap EXIT
932023-11-30T05:03:16.289Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-30T05:03:16.479Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-30T05:03:16.479Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-30T05:03:16.479Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-30T05:03:16.479Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-30T05:03:16.479Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-30T05:03:16.479Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-30T05:03:16.479Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-30T05:03:16.479Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-30T05:03:16.479Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-30T05:03:16.480Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-30T05:03:16.507Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-30T05:03:16.507Z+ [[ -d /opt/oxide ]]
1062023-11-30T05:03:16.507Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-30T05:03:16.507Z+ pfexec mkdir /opt/oxide
1082023-11-30T05:03:16.507Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-30T05:03:16.507Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-30T05:03:16.507Z+ cd /opt/oxide-underneath
1112023-11-30T05:03:16.508Z+ pfexec tar ceEp@/f - .
1122023-11-30T05:03:16.508Z+ cd /opt/oxide
1132023-11-30T05:03:16.508Z+ pfexec tar xveEp@/f -
1142023-11-30T05:03:16.597Zx ., 0 bytes, 0 tape blocks
1152023-11-30T05:03:16.597Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-30T05:03:16.597Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-30T05:03:16.597Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-30T05:03:16.621Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-30T05:03:16.621Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-30T05:03:16.644Z+ pfexec mkdir /opt/oxide/work
1212023-11-30T05:03:16.644Z+ pfexec chown build:build /opt/oxide/work
1222023-11-30T05:03:16.644Z+ cd /opt/oxide/work
1232023-11-30T05:03:16.644Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-30T05:03:18.180ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-30T05:03:18.181Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-11-30T05:03:18.181Zx out/omicron-sled-agent.tar, 112139264 bytes, 219022 tape blocks
1272023-11-30T05:03:18.181Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-30T05:03:18.181Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-11-30T05:03:18.181Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-11-30T05:03:18.181Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-11-30T05:03:18.181Zx target/release/omicron-package, 42567616 bytes, 83140 tape blocks
1322023-11-30T05:03:18.181Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-11-30T05:03:18.181Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-30T05:03:18.181Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-30T05:03:18.181Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-30T05:03:18.181Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-30T05:03:18.181Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-30T05:03:18.181Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-30T05:03:18.205Z
1402023-11-30T05:03:18.205Zreal 1.546096986
1412023-11-30T05:03:18.205Zuser 1.390633774
1422023-11-30T05:03:18.205Zsys 0.274882358
1432023-11-30T05:03:18.205Ztrap 0.000042479
1442023-11-30T05:03:18.205Ztflt 0.000000000
1452023-11-30T05:03:18.205Zdflt 0.000000000
1462023-11-30T05:03:18.205Zkflt 0.000000000
1472023-11-30T05:03:18.205Zlock 0.000000000
1482023-11-30T05:03:18.205Zslp 1.390524579
1492023-11-30T05:03:18.205Zlat 0.033983576
1502023-11-30T05:03:18.205Zstop 0.000121421
1512023-11-30T05:03:18.205Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-30T05:03:19.259Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-30T05:03:19.287Z+ mkdir tests
1542023-11-30T05:03:19.287Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-30T05:03:19.287Z+ ptime -m gunzip
1562023-11-30T05:03:19.287Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8d69322849e0fa66
1572023-11-30T05:03:19.414Z
1582023-11-30T05:03:19.437Zreal 0.139934690
1592023-11-30T05:03:19.437Zuser 0.129065616
1602023-11-30T05:03:19.437Zsys 0.010609057
1612023-11-30T05:03:19.437Ztrap 0.000009229
1622023-11-30T05:03:19.437Ztflt 0.000000000
1632023-11-30T05:03:19.437Zdflt 0.000000000
1642023-11-30T05:03:19.437Zkflt 0.000000000
1652023-11-30T05:03:19.438Zlock 0.000000000
1662023-11-30T05:03:19.438Zslp 0.000019040
1672023-11-30T05:03:19.438Zlat 0.000022311
1682023-11-30T05:03:19.438Zstop 0.000124352
1692023-11-30T05:03:19.438Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8d69322849e0fa66
1702023-11-30T05:03:19.438Z+ chmod a+x tests/bootstrap-8d69322849e0fa66
1712023-11-30T05:03:19.438Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-30T05:03:19.438Z+ ptime -m gunzip
1732023-11-30T05:03:19.438Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-30T05:03:21.603Z
1752023-11-30T05:03:21.629Zreal 2.175700847
1762023-11-30T05:03:21.630Zuser 2.041438016
1772023-11-30T05:03:21.630Zsys 0.133865242
1782023-11-30T05:03:21.630Ztrap 0.000167494
1792023-11-30T05:03:21.630Ztflt 0.000000000
1802023-11-30T05:03:21.630Zdflt 0.000000000
1812023-11-30T05:03:21.630Zkflt 0.000000000
1822023-11-30T05:03:21.630Zlock 0.000000000
1832023-11-30T05:03:21.630Zslp 0.000023920
1842023-11-30T05:03:21.630Zlat 0.000022770
1852023-11-30T05:03:21.630Zstop 0.000123902
1862023-11-30T05:03:21.630Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-30T05:03:21.630Z+ chmod a+x tests/bootstrap
1882023-11-30T05:03:21.630Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-30T05:03:21.630Z+ ptime -m gunzip
1902023-11-30T05:03:21.630Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7d7c590276acd289
1912023-11-30T05:03:23.854Z
1922023-11-30T05:03:23.877Zreal 2.240804268
1932023-11-30T05:03:23.878Zuser 2.107995460
1942023-11-30T05:03:23.878Zsys 0.132427681
1952023-11-30T05:03:23.878Ztrap 0.000148882
1962023-11-30T05:03:23.878Ztflt 0.000000000
1972023-11-30T05:03:23.878Zdflt 0.000000000
1982023-11-30T05:03:23.878Zkflt 0.000000000
1992023-11-30T05:03:23.878Zlock 0.000000000
2002023-11-30T05:03:23.878Zslp 0.000021290
2012023-11-30T05:03:23.878Zlat 0.000019989
2022023-11-30T05:03:23.878Zstop 0.000131753
2032023-11-30T05:03:23.878Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7d7c590276acd289
2042023-11-30T05:03:23.878Z+ chmod a+x tests/end_to_end_tests-7d7c590276acd289
2052023-11-30T05:03:23.878Z+ bmat address ls
2062023-11-30T05:03:23.878ZNAME CIDR FIRST LAST COUNT
2072023-11-30T05:03:23.878Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-11-30T05:03:23.878Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-30T05:03:23.902Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-11-30T05:03:23.902Z+ EXTRA_IP_START=10.151.2.164
2112023-11-30T05:03:23.902Z+ EXTRA_IP_END=10.151.2.227
2122023-11-30T05:03:23.902Z+ EXTRA_IP_COUNT=64
2132023-11-30T05:03:23.902Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-30T05:03:23.902Z+ EXTRA_IP_BASE=10.151.2
2152023-11-30T05:03:23.902Z+ EXTRA_IP_FOCTET=164
2162023-11-30T05:03:23.902Z+ EXTRA_IP_LOCTET=227
2172023-11-30T05:03:23.902Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-11-30T05:03:23.902Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-11-30T05:03:23.902Z+ DNS_IP1=10.151.2.164
2202023-11-30T05:03:23.902Z+ DNS_IP2=10.151.2.165
2212023-11-30T05:03:23.902Z+ UPLINK_IP=10.151.2.174
2222023-11-30T05:03:23.902Z+ IPPOOL_START=10.151.2.175
2232023-11-30T05:03:23.903Z+ IPPOOL_END=10.151.2.227
2242023-11-30T05:03:23.903Z++ ipadm show-addr -po type,addr
2252023-11-30T05:03:23.903Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-30T05:03:23.903Z+ GATEWAY_IP=10.151.2.100
2272023-11-30T05:03:23.903Z+ [[ -n 10.151.2.100 ]]
2282023-11-30T05:03:23.903Z+ ping -s 10.151.2.100 56 1
2292023-11-30T05:03:23.903ZPING 10.151.2.100: 56 data bytes
2302023-11-30T05:03:23.903Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2312023-11-30T05:03:23.903Z
2322023-11-30T05:03:23.903Z----10.151.2.100 PING Statistics----
2332023-11-30T05:03:23.903Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-30T05:03:23.903Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-11-30T05:03:23.904Z++ arp -an
2362023-11-30T05:03:23.928Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-11-30T05:03:23.928Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-11-30T05:03:23.928Z+ routeadm -e ipv4-forwarding -u
2392023-11-30T05:03:24.045Z+ PXA_START=10.151.2.164
2402023-11-30T05:03:24.045Z+ PXA_END=10.151.2.227
2412023-11-30T05:03:24.070Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-30T05:03:24.070Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-30T05:03:24.151Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-30T05:03:24.151Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-30T05:03:24.175Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-30T05:03:24.175Z++ cd ./tools
2472023-11-30T05:03:24.175Z++ pwd
2482023-11-30T05:03:24.175Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-30T05:03:24.175Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-30T05:03:24.175Z+ SOFTNPU_MODE=zone
2512023-11-30T05:03:24.175Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-30T05:03:24.175Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-30T05:03:24.175Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-30T05:03:24.175Z++ dladm show-phys -p -o LINK
2552023-11-30T05:03:24.175Z++ head -1
2562023-11-30T05:03:24.234Z+ PHYSICAL_LINK=igb0
2572023-11-30T05:03:24.258Z+ echo 'Using igb0 as physical link'
2582023-11-30T05:03:24.258ZUsing igb0 as physical link
2592023-11-30T05:03:24.258Z+ ensure_run_as_root
2602023-11-30T05:03:24.258Z++ id -u
2612023-11-30T05:03:24.258Z+ [[ 0 -ne 0 ]]
2622023-11-30T05:03:24.258Z+ ensure_zpools
2632023-11-30T05:03:24.258Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-30T05:03:24.258Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-11-30T05:03:24.258Z+ readarray -t ZPOOLS
2662023-11-30T05:03:24.258Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-11-30T05:03:24.258Z++ sed 's/[ ",]//g'
2682023-11-30T05:03:24.258Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-11-30T05:03:24.258Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-11-30T05:03:24.258ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-11-30T05:03:24.258Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-11-30T05:03:24.258Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-11-30T05:03:24.258Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-11-30T05:03:24.282Z0+0 records in
2752023-11-30T05:03:24.282Z0+0 records out
2762023-11-30T05:03:24.282Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2772023-11-30T05:03:24.282Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-11-30T05:03:24.282Z+ set +x
2792023-11-30T05:03:24.282ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-11-30T05:03:24.282Z++ zpool list -o name
2812023-11-30T05:03:24.282Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-11-30T05:03:24.306Z+ [[ -z '' ]]
2832023-11-30T05:03:24.308Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-11-30T05:03:24.331Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-11-30T05:03:24.331Z+ set +x
2862023-11-30T05:03:24.331ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-11-30T05:03:24.331Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-11-30T05:03:24.331Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-11-30T05:03:24.331ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-11-30T05:03:24.331Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-11-30T05:03:24.331Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-11-30T05:03:24.331Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-11-30T05:03:24.331Z0+0 records in
2942023-11-30T05:03:24.331Z0+0 records out
2952023-11-30T05:03:24.331Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-11-30T05:03:24.331Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-11-30T05:03:24.331Z+ set +x
2982023-11-30T05:03:24.331ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-11-30T05:03:24.331Z++ zpool list -o name
3002023-11-30T05:03:24.355Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-11-30T05:03:24.355Z+ [[ -z '' ]]
3022023-11-30T05:03:24.355Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-11-30T05:03:24.379Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-11-30T05:03:24.379Z+ set +x
3052023-11-30T05:03:24.379ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-11-30T05:03:24.379ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3072023-11-30T05:03:24.379Z+ for ZPOOL in "${ZPOOLS[@]}"
3082023-11-30T05:03:24.380Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3092023-11-30T05:03:24.380Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-11-30T05:03:24.380Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-11-30T05:03:24.380Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-11-30T05:03:24.380Z0+0 records in
3132023-11-30T05:03:24.380Z0+0 records out
3142023-11-30T05:03:24.380Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3152023-11-30T05:03:24.380Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-11-30T05:03:24.380Z+ set +x
3172023-11-30T05:03:24.380ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-11-30T05:03:24.380Z++ zpool list -o name
3192023-11-30T05:03:24.380Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-11-30T05:03:24.404Z+ [[ -z '' ]]
3212023-11-30T05:03:24.404Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-11-30T05:03:24.427Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-11-30T05:03:24.427Z+ set +x
3242023-11-30T05:03:24.427ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-11-30T05:03:24.427Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-11-30T05:03:24.427Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-11-30T05:03:24.427ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-11-30T05:03:24.427Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-11-30T05:03:24.427Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-11-30T05:03:24.427Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-11-30T05:03:24.427Z0+0 records in
3322023-11-30T05:03:24.429Z0+0 records out
3332023-11-30T05:03:24.429Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-11-30T05:03:24.429Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-11-30T05:03:24.429Z+ set +x
3362023-11-30T05:03:24.429ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-11-30T05:03:24.429Z++ zpool list -o name
3382023-11-30T05:03:24.429Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-11-30T05:03:24.429Z+ [[ -z '' ]]
3402023-11-30T05:03:24.429Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-11-30T05:03:24.453Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-11-30T05:03:24.453Z+ set +x
3432023-11-30T05:03:24.453ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-11-30T05:03:24.453Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-11-30T05:03:24.453Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-11-30T05:03:24.453ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-11-30T05:03:24.453Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-11-30T05:03:24.453Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-11-30T05:03:24.453Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-11-30T05:03:24.453Z0+0 records in
3512023-11-30T05:03:24.453Z0+0 records out
3522023-11-30T05:03:24.453Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3532023-11-30T05:03:24.453Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-11-30T05:03:24.453Z+ set +x
3552023-11-30T05:03:24.453ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-11-30T05:03:24.477Z++ zpool list -o name
3572023-11-30T05:03:24.478Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-11-30T05:03:24.478Z+ [[ -z '' ]]
3592023-11-30T05:03:24.478Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-11-30T05:03:24.502Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-11-30T05:03:24.502Z+ set +x
3622023-11-30T05:03:24.502ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-11-30T05:03:24.502Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-11-30T05:03:24.502Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-11-30T05:03:24.502ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-11-30T05:03:24.502Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-11-30T05:03:24.502Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-11-30T05:03:24.503Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-11-30T05:03:24.503Z0+0 records in
3702023-11-30T05:03:24.503Z0+0 records out
3712023-11-30T05:03:24.503Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3722023-11-30T05:03:24.503Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-11-30T05:03:24.503Z+ set +x
3742023-11-30T05:03:24.503ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-11-30T05:03:24.503Z++ zpool list -o name
3762023-11-30T05:03:24.503Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-11-30T05:03:24.526Z+ [[ -z '' ]]
3782023-11-30T05:03:24.526Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-11-30T05:03:24.550Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-11-30T05:03:24.550Z+ set +x
3812023-11-30T05:03:24.550ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-11-30T05:03:24.551Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-11-30T05:03:24.551Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-11-30T05:03:24.551ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-11-30T05:03:24.551Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-11-30T05:03:24.551Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-11-30T05:03:24.551Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-11-30T05:03:24.551Z0+0 records in
3892023-11-30T05:03:24.551Z0+0 records out
3902023-11-30T05:03:24.551Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-11-30T05:03:24.551Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-11-30T05:03:24.551Z+ set +x
3932023-11-30T05:03:24.551ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-11-30T05:03:24.551Z++ zpool list -o name
3952023-11-30T05:03:24.551Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-11-30T05:03:24.551Z+ [[ -z '' ]]
3972023-11-30T05:03:24.551Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-11-30T05:03:24.579Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-11-30T05:03:24.579Z+ set +x
4002023-11-30T05:03:24.602ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-11-30T05:03:24.602Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-30T05:03:24.602ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4032023-11-30T05:03:24.602Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4042023-11-30T05:03:24.602Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-11-30T05:03:24.602Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-11-30T05:03:24.603Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-11-30T05:03:24.603Z0+0 records in
4082023-11-30T05:03:24.603Z0+0 records out
4092023-11-30T05:03:24.603Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-11-30T05:03:24.603Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-11-30T05:03:24.603Z+ set +x
4122023-11-30T05:03:24.603ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-11-30T05:03:24.603Z++ zpool list -o name
4142023-11-30T05:03:24.603Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-11-30T05:03:24.603Z+ [[ -z '' ]]
4162023-11-30T05:03:24.603Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-11-30T05:03:24.630Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-11-30T05:03:24.630Z+ set +x
4192023-11-30T05:03:24.630ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-11-30T05:03:24.630Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-11-30T05:03:24.630Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-11-30T05:03:24.630ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-11-30T05:03:24.630Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-11-30T05:03:24.630Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-11-30T05:03:24.630Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-11-30T05:03:24.630Z0+0 records in
4272023-11-30T05:03:24.630Z0+0 records out
4282023-11-30T05:03:24.630Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4292023-11-30T05:03:24.630Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-11-30T05:03:24.630Z+ set +x
4312023-11-30T05:03:24.630ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-11-30T05:03:24.657Z++ zpool list -o name
4332023-11-30T05:03:24.657Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-11-30T05:03:24.657Z+ [[ -z '' ]]
4352023-11-30T05:03:24.657Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-11-30T05:03:24.681Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-11-30T05:03:24.681Z+ set +x
4382023-11-30T05:03:24.681ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-11-30T05:03:24.681Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-11-30T05:03:24.681Z+ readarray -t ZPOOLS
4412023-11-30T05:03:24.681Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-11-30T05:03:24.681Z++ sed 's/[ ",]//g'
4432023-11-30T05:03:24.681Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-11-30T05:03:24.681Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-11-30T05:03:24.681ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-11-30T05:03:24.681Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-11-30T05:03:24.681Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-11-30T05:03:24.681Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-11-30T05:03:24.681Z0+0 records in
4502023-11-30T05:03:24.681Z0+0 records out
4512023-11-30T05:03:24.681Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4522023-11-30T05:03:24.681Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-11-30T05:03:24.681Z+ set +x
4542023-11-30T05:03:24.681ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-11-30T05:03:24.706Z++ zpool list -o name
4562023-11-30T05:03:24.706Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-11-30T05:03:24.709Z+ [[ -z '' ]]
4582023-11-30T05:03:24.709Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-11-30T05:03:24.733Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-11-30T05:03:24.733Z+ set +x
4612023-11-30T05:03:24.733ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-11-30T05:03:24.733Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-11-30T05:03:24.733Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-11-30T05:03:24.733ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-11-30T05:03:24.733Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-11-30T05:03:24.733Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-11-30T05:03:24.733Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-11-30T05:03:24.733Z0+0 records in
4692023-11-30T05:03:24.733Z0+0 records out
4702023-11-30T05:03:24.733Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4712023-11-30T05:03:24.733Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-11-30T05:03:24.733Z+ set +x
4732023-11-30T05:03:24.733ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-11-30T05:03:24.733Z++ zpool list -o name
4752023-11-30T05:03:24.733Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-11-30T05:03:24.757Z+ [[ -z '' ]]
4772023-11-30T05:03:24.757Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-11-30T05:03:24.780Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-11-30T05:03:24.781Z+ set +x
4802023-11-30T05:03:24.781ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-11-30T05:03:24.781Z+ [[ zone == \z\o\n\e ]]
4822023-11-30T05:03:24.781Z+ ensure_simulated_links igb0
4832023-11-30T05:03:24.781Z+ local PHYSICAL_LINK=igb0
4842023-11-30T05:03:24.781Z+ INDICES=("0" "1")
4852023-11-30T05:03:24.781Z+ for I in "${INDICES[@]}"
4862023-11-30T05:03:24.781Z++ get_simnet_name_if_exists net0
4872023-11-30T05:03:24.781Z++ dladm show-simnet -p -o LINK net0
4882023-11-30T05:03:24.781Z++ echo ''
4892023-11-30T05:03:24.781Z+ [[ -z '' ]]
4902023-11-30T05:03:24.781Z+ dladm create-simnet -t net0
4912023-11-30T05:03:24.781Z+ dladm create-simnet -t sc0_0
4922023-11-30T05:03:24.805Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-11-30T05:03:24.805Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-11-30T05:03:24.805Z+ success 'Simnet net0/sc0_0 exists'
4952023-11-30T05:03:24.805Z+ set +x
4962023-11-30T05:03:24.805ZSimnet net0/sc0_0 exists
4972023-11-30T05:03:24.805Z+ for I in "${INDICES[@]}"
4982023-11-30T05:03:24.805Z++ get_simnet_name_if_exists net1
4992023-11-30T05:03:24.805Z++ dladm show-simnet -p -o LINK net1
5002023-11-30T05:03:24.805Z++ echo ''
5012023-11-30T05:03:24.805Z+ [[ -z '' ]]
5022023-11-30T05:03:24.805Z+ dladm create-simnet -t net1
5032023-11-30T05:03:24.805Z+ dladm create-simnet -t sc1_0
5042023-11-30T05:03:24.805Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-11-30T05:03:24.829Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-11-30T05:03:24.829Z+ success 'Simnet net1/sc1_0 exists'
5072023-11-30T05:03:24.829Z+ set +x
5082023-11-30T05:03:24.829ZSimnet net1/sc1_0 exists
5092023-11-30T05:03:24.829Z++ get_vnic_name_if_exists sc0_1
5102023-11-30T05:03:24.829Z++ dladm show-vnic -p -o LINK sc0_1
5112023-11-30T05:03:24.829Z++ echo ''
5122023-11-30T05:03:24.829Z+ [[ -z '' ]]
5132023-11-30T05:03:24.829Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-11-30T05:03:24.829Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-11-30T05:03:24.829Z+ success 'Vnic sc0_1 exists'
5162023-11-30T05:03:24.829Z+ set +x
5172023-11-30T05:03:24.829ZVnic sc0_1 exists
5182023-11-30T05:03:24.829Z+ warn_if_no_proxy_arp
5192023-11-30T05:03:24.829Z+ [[ -v PXA_START ]]
5202023-11-30T05:03:24.829Z+ [[ -v PXA_END ]]
5212023-11-30T05:03:24.829Z+ ensure_softnpu_zone
5222023-11-30T05:03:24.829Z+ zoneadm list
5232023-11-30T05:03:24.829Z+ grep -q sidecar_softnpu
5242023-11-30T05:03:24.853Z+ [[ -f out/npuzone/npuzone ]]
5252023-11-30T05:03:24.853Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5262023-11-30T05:03:24.853Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
5272023-11-30T05:03:30.607Zdownloading softnpu ...done
5282023-11-30T05:03:30.631Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5292023-11-30T05:03:35.350Zdownloading asic_program.so ...done
5302023-11-30T05:03:35.375Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-30T05:03:43.259Zdownloading scadm ...done
5322023-11-30T05:03:43.282Zcreating port tfportrear0_0 for sc0_0
5332023-11-30T05:03:43.283Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-30T05:03:43.312Zsoftnpu zone setup
5352023-11-30T05:03:43.312Zconfigure zone
5362023-11-30T05:03:43.339Zinstall zone
5372023-11-30T05:03:44.335Zboot zone
5382023-11-30T05:03:47.444Z[sidecar_softnpu] svcs milestone/network
5392023-11-30T05:03:47.494ZSTATE STIME FMRI
5402023-11-30T05:03:47.494Zonline 5:03:45 svc:/milestone/network:default
5412023-11-30T05:03:47.494Z[sidecar_softnpu] /softnpu/npu start
5422023-11-30T05:03:47.518Z
5432023-11-30T05:03:47.518Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-30T05:03:47.518Z+ GATEWAY_IP=10.151.2.100
5452023-11-30T05:03:47.518Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-11-30T05:03:47.518ZUsing 10.151.2.100 as gateway ip
5472023-11-30T05:03:47.518Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-30T05:03:47.518Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-11-30T05:03:47.518Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-30T05:03:47.518Z+++ echo 18:c0:4d:0d:a0:2a
5512023-11-30T05:03:47.518Z+ COUNT=1
5522023-11-30T05:03:47.518Z+ [[ 1 -ne 1 ]]
5532023-11-30T05:03:47.518Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-11-30T05:03:47.518ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-11-30T05:03:47.518Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-11-30T05:03:47.518Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-11-30T05:03:47.542Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-30T05:03:47.542Z+ [[ -v PXA_START ]]
5592023-11-30T05:03:47.542Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-11-30T05:03:47.542Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-11-30T05:03:47.569Z+ z_scadm dump-state
5622023-11-30T05:03:47.569Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-30T05:03:47.595Zlocal v6:
5642023-11-30T05:03:47.595Zlocal v4:
5652023-11-30T05:03:47.595Zrouter v6:
5662023-11-30T05:03:47.595Zrouter v4_idx:
5672023-11-30T05:03:47.597Zrouter v4_routes:
5682023-11-30T05:03:47.598Zresolver v4:
5692023-11-30T05:03:47.598Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-11-30T05:03:47.598Zresolver v6:
5712023-11-30T05:03:47.598Znat v4:
5722023-11-30T05:03:47.598Znat_v6:
5732023-11-30T05:03:47.598Zport_mac:
5742023-11-30T05:03:47.622Zproxy arp:
5752023-11-30T05:03:47.622Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-11-30T05:03:47.622Z+ success 'softnpu zone exists'
5772023-11-30T05:03:47.622Z+ set +x
5782023-11-30T05:03:47.622Zsoftnpu zone exists
5792023-11-30T05:03:47.622Z
5802023-11-30T05:03:47.622Zreal 23.445390728
5812023-11-30T05:03:47.622Zuser 0.699459435
5822023-11-30T05:03:47.622Zsys 1.597886778
5832023-11-30T05:03:47.622Ztrap 0.000075129
5842023-11-30T05:03:47.622Ztflt 0.000277361
5852023-11-30T05:03:47.622Zdflt 0.000989353
5862023-11-30T05:03:47.622Zkflt 0.000000000
5872023-11-30T05:03:47.622Zlock 18:05.411696543
5882023-11-30T05:03:47.622Zslp 1:10.153570520
5892023-11-30T05:03:47.622Zlat 0.072135276
5902023-11-30T05:03:47.622Zstop 0.357618974
5912023-11-30T05:03:47.622Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-30T05:03:47.647Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-30T05:03:47.647Z+ SILO_NAME=recovery
5942023-11-30T05:03:47.647Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-30T05:03:47.647Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-30T05:03:47.647Z+ sed -i~ '
5972023-11-30T05:03:47.647Z /^external_dns_ips/c\
5982023-11-30T05:03:47.647Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-11-30T05:03:47.647Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-30T05:03:47.647Z /^first/c\
6012023-11-30T05:03:47.647Zfirst = "10.151.2.164"
6022023-11-30T05:03:47.647Z /^last/c\
6032023-11-30T05:03:47.647Zlast = "10.151.2.173"
6042023-11-30T05:03:47.647Z }
6052023-11-30T05:03:47.647Z /^\[rack_network_config/,/^$/ {
6062023-11-30T05:03:47.647Z /^infra_ip_first/c\
6072023-11-30T05:03:47.647Zinfra_ip_first = "10.151.2.174"
6082023-11-30T05:03:47.647Z /^infra_ip_last/c\
6092023-11-30T05:03:47.647Zinfra_ip_last = "10.151.2.174"
6102023-11-30T05:03:47.647Z }
6112023-11-30T05:03:47.647Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-11-30T05:03:47.647Z /^routes/c\
6132023-11-30T05:03:47.647Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6142023-11-30T05:03:47.647Z /^addresses/c\
6152023-11-30T05:03:47.647Zaddresses = \["10.151.2.174/32"\]
6162023-11-30T05:03:47.647Z }
6172023-11-30T05:03:47.647Z' pkg/config-rss.toml
6182023-11-30T05:03:47.647Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-11-30T05:03:47.647Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-11-30T05:03:47.647Z+++ pkg/config-rss.toml Thu Nov 30 05:03:47 2023
6212023-11-30T05:03:47.648Z@@ -41,7 +41,7 @@
6222023-11-30T05:03:47.648Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-11-30T05:03:47.648Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-11-30T05:03:47.648Z # below.
6252023-11-30T05:03:47.649Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-11-30T05:03:47.649Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6272023-11-30T05:03:47.649Z
6282023-11-30T05:03:47.649Z # Initial TLS certificates for the external API
6292023-11-30T05:03:47.649Z #
6302023-11-30T05:03:47.649Z@@ -77,8 +77,8 @@
6312023-11-30T05:03:47.649Z #
6322023-11-30T05:03:47.649Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-11-30T05:03:47.649Z [[internal_services_ip_pool_ranges]]
6342023-11-30T05:03:47.649Z-first = "192.168.1.20"
6352023-11-30T05:03:47.649Z-last = "192.168.1.29"
6362023-11-30T05:03:47.649Z+first = "10.151.2.164"
6372023-11-30T05:03:47.649Z+last = "10.151.2.173"
6382023-11-30T05:03:47.649Z
6392023-11-30T05:03:47.649Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-11-30T05:03:47.649Z # parameters should be provided to the control plane via wicket, but we need to
6412023-11-30T05:03:47.649Z@@ -92,8 +92,8 @@
6422023-11-30T05:03:47.649Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-11-30T05:03:47.649Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-11-30T05:03:47.649Z # With softnpu, only one address is used.
6452023-11-30T05:03:47.649Z-infra_ip_first = "192.168.1.30"
6462023-11-30T05:03:47.649Z-infra_ip_last = "192.168.1.30"
6472023-11-30T05:03:47.649Z+infra_ip_first = "10.151.2.174"
6482023-11-30T05:03:47.649Z+infra_ip_last = "10.151.2.174"
6492023-11-30T05:03:47.650Z
6502023-11-30T05:03:47.650Z # Configurations for BGP routers to run on the scrimlets.
6512023-11-30T05:03:47.650Z bgp = []
6522023-11-30T05:03:47.650Z@@ -101,9 +101,9 @@
6532023-11-30T05:03:47.651Z # You can configure multiple uplinks by repeating the following stanza
6542023-11-30T05:03:47.651Z [[rack_network_config.ports]]
6552023-11-30T05:03:47.651Z # Routes associated with this port.
6562023-11-30T05:03:47.651Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-11-30T05:03:47.651Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6582023-11-30T05:03:47.651Z # Addresses associated with this port.
6592023-11-30T05:03:47.651Z-addresses = ["192.168.1.30/32"]
6602023-11-30T05:03:47.651Z+addresses = ["10.151.2.174/32"]
6612023-11-30T05:03:47.651Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-11-30T05:03:47.651Z port = "qsfp0"
6632023-11-30T05:03:47.651Z # The speed of this port.
6642023-11-30T05:03:47.651Z+ true
6652023-11-30T05:03:47.651Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-11-30T05:03:47.675Za pkg/config-rss.toml 7K
6672023-11-30T05:03:47.675Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-11-30T05:03:47.675Z+ TLS_NAME=recovery.sys.oxide.test
6692023-11-30T05:03:47.675Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-11-30T05:03:48.346Z......+....+.....+...+.......+..+...+......+....+........+...+.+...+.....................+.....+.+.....+................+.....+.........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+....+......+...+.....+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+...+......+......+................+..+.+...........+...+.......+...+......+......+...+..+........................+...................+.................+...+.......+.....+...+.............+...+.....................+......+...............+..+......+.+...+......+...............+..+.......+...............+.....+..........+..+...+......+......+.+...+.....+.+...........+....+...........+............+...+.......+.....+..........+.....+...........................+.+........+...............+.+......+...+.....+................+...+........+...+.......+...............+.........+.....+...............+.+..+...+...+.+......+........+.......+..+...+..................+....+...+.....+....+.....+.+.................+......+......+......+....+..................+.................+....+..+.......+......+......+...+..+...+.............+........+.......+..............+......+..........+..............+...+.......+......+.........+......+..+.......+.........+......+...........+....+.....+...............+......+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-11-30T05:03:48.605Z.+....+.....+.......+...........+....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+..+.......+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+......+.............+............+...+..+.............+............+..+..........+...........+............+...+.+......+........+.......+...+.........+......+......+...+............+....................+....+..............+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-11-30T05:03:48.628Z-----
6732023-11-30T05:03:48.628Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-11-30T05:03:48.628Za pkg/initial-tls-cert.pem 2K
6752023-11-30T05:03:48.628Za pkg/initial-tls-key.pem 4K
6762023-11-30T05:03:48.628Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-11-30T05:03:48.656Z+ rmdir pkg
6782023-11-30T05:03:48.656Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-11-30T05:03:48.656Z+ pfexec mkdir -p /usr/oxide
6802023-11-30T05:03:48.656Z+ pfexec rm -f /usr/oxide/tcpproxy
6812023-11-30T05:03:48.656Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6822023-11-30T05:03:48.909Z+ pfexec chmod +x /usr/oxide/tcpproxy
6832023-11-30T05:03:48.933Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6842023-11-30T05:03:48.933Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6852023-11-30T05:03:48.934Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6862023-11-30T05:03:49.031Z+ OMICRON_NO_UNINSTALL=1
6872023-11-30T05:03:49.031Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6882023-11-30T05:03:49.055ZNov 30 05:03:48.864 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6892023-11-30T05:03:49.055ZNov 30 05:03:48.866 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6902023-11-30T05:03:49.055ZNov 30 05:03:48.867 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6912023-11-30T05:03:49.055ZNov 30 05:03:48.867 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6922023-11-30T05:03:49.055ZNov 30 05:03:48.867 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6932023-11-30T05:03:49.055ZNov 30 05:03:48.867 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6942023-11-30T05:03:49.055ZNov 30 05:03:48.867 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6952023-11-30T05:03:49.056ZNov 30 05:03:48.867 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6962023-11-30T05:03:49.056ZNov 30 05:03:48.867 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6972023-11-30T05:03:49.056ZNov 30 05:03:48.867 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6982023-11-30T05:03:49.056ZNov 30 05:03:48.867 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6992023-11-30T05:03:49.056ZNov 30 05:03:48.867 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7002023-11-30T05:03:49.056ZNov 30 05:03:48.867 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7012023-11-30T05:03:49.056ZNov 30 05:03:48.867 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7022023-11-30T05:03:49.056ZNov 30 05:03:48.867 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7032023-11-30T05:03:49.056ZNov 30 05:03:48.867 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7042023-11-30T05:03:49.057ZNov 30 05:03:48.867 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7052023-11-30T05:03:49.270ZNov 30 05:03:49.090 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-11-30T05:03:49.306ZNov 30 05:03:49.125 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-30T05:03:49.365ZNov 30 05:03:49.185 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-30T05:03:49.499Z
7092023-11-30T05:03:49.533Zreal 0.458454438
7102023-11-30T05:03:49.533Zuser 0.097863042
7112023-11-30T05:03:49.533Zsys 1.812675405
7122023-11-30T05:03:49.533Ztrap 0.000002489
7132023-11-30T05:03:49.533Ztflt 0.000021589
7142023-11-30T05:03:49.533Zdflt 0.000337754
7152023-11-30T05:03:49.533Zkflt 0.000000000
7162023-11-30T05:03:49.533Zlock 41.204901067
7172023-11-30T05:03:49.533Zslp 0.671362494
7182023-11-30T05:03:49.533Zlat 0.002958119
7192023-11-30T05:03:49.533Zstop 0.212524232
7202023-11-30T05:03:49.534Z+ retry=0
7212023-11-30T05:03:49.534Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-30T05:03:49.534Z+ [[ 0 -gt 30 ]]
7232023-11-30T05:03:49.534Z+ sleep 1
7242023-11-30T05:03:50.509Z+ retry=1
7252023-11-30T05:03:50.509Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-30T05:03:50.601Z+ [[ 1 -gt 30 ]]
7272023-11-30T05:03:50.601Z+ sleep 1
7282023-11-30T05:03:51.524Z+ retry=2
7292023-11-30T05:03:51.524Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-30T05:03:51.550Z+ [[ 2 -gt 30 ]]
7312023-11-30T05:03:51.550Z+ sleep 1
7322023-11-30T05:03:52.898Z+ retry=3
7332023-11-30T05:03:52.898Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-11-30T05:03:52.898Z+ [[ 3 -gt 30 ]]
7352023-11-30T05:03:52.898Z+ sleep 1
7362023-11-30T05:03:53.553Z+ retry=4
7372023-11-30T05:03:53.553Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-11-30T05:04:13.432Z+ [[ 4 -gt 30 ]]
7392023-11-30T05:04:13.432Z+ sleep 1
7402023-11-30T05:04:14.436Z+ retry=5
7412023-11-30T05:04:14.436Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7422023-11-30T05:04:14.460Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7432023-11-30T05:04:14.484Zlocal v6:
7442023-11-30T05:04:14.484Zfd00:99::1
7452023-11-30T05:04:14.485Zfe80::c8e7:a5ff:fef8:3bb7
7462023-11-30T05:04:14.485Zlocal v4:
7472023-11-30T05:04:14.485Z10.151.2.174
7482023-11-30T05:04:14.485Zrouter v6:
7492023-11-30T05:04:14.485Zrouter v4_idx:
7502023-11-30T05:04:14.485Z0.0.0.0/0 -> 0
7512023-11-30T05:04:14.485Zrouter v4_routes:
7522023-11-30T05:04:14.485Z0 -> 10.151.2.100 (2)
7532023-11-30T05:04:14.485Zresolver v4:
7542023-11-30T05:04:14.485Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7552023-11-30T05:04:14.485Z10.151.2.174 -> 7e:34:04:e8:d0:43
7562023-11-30T05:04:14.485Zresolver v6:
7572023-11-30T05:04:14.485Zfe80::c8e7:a5ff:fef8:3bb7 -> ca:e7:a5:f8:3b:b7
7582023-11-30T05:04:14.485Zfe80::58d6:e0ff:feb2:8c03 -> 5a:d6:e0:b2:8c:03
7592023-11-30T05:04:14.485Znat v4:
7602023-11-30T05:04:14.485Znat_v6:
7612023-11-30T05:04:14.485Zport_mac:
7622023-11-30T05:04:14.485Z1000: a8:40:25:74:8d:3f
7632023-11-30T05:04:14.485Z1: a8:40:25:74:8d:40
7642023-11-30T05:04:14.485Zproxy arp:
7652023-11-30T05:04:14.485Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7662023-11-30T05:04:14.485Z+ retry=0
7672023-11-30T05:04:14.485Z+++ zoneadm list -n
7682023-11-30T05:04:14.485Z+++ grep oxz_ntp
7692023-11-30T05:04:14.515Z++ pfexec svcs -z -Hostate oxide/ntp
7702023-11-30T05:04:14.515Zsvcs: zone '-Hostate': entity not found
7712023-11-30T05:04:14.515Z++ true
7722023-11-30T05:04:14.515Z+ [[ '' != online ]]
7732023-11-30T05:04:14.515Z+ [[ 0 -gt 60 ]]
7742023-11-30T05:04:14.515Z+ sleep 1
7752023-11-30T05:04:15.497Z+ retry=1
7762023-11-30T05:04:15.526Z+++ zoneadm list -n
7772023-11-30T05:04:15.526Z+++ grep oxz_ntp
7782023-11-30T05:04:15.526Z++ pfexec svcs -z -Hostate oxide/ntp
7792023-11-30T05:04:15.526Zsvcs: zone '-Hostate': entity not found
7802023-11-30T05:04:15.526Z++ true
7812023-11-30T05:04:15.526Z+ [[ '' != online ]]
7822023-11-30T05:04:15.526Z+ [[ 1 -gt 60 ]]
7832023-11-30T05:04:15.526Z+ sleep 1
7842023-11-30T05:04:16.517Z+ retry=2
7852023-11-30T05:04:16.540Z+++ zoneadm list -n
7862023-11-30T05:04:16.540Z+++ grep oxz_ntp
7872023-11-30T05:04:16.540Z++ pfexec svcs -z -Hostate oxide/ntp
7882023-11-30T05:04:16.540Zsvcs: zone '-Hostate': entity not found
7892023-11-30T05:04:16.540Z++ true
7902023-11-30T05:04:16.540Z+ [[ '' != online ]]
7912023-11-30T05:04:16.540Z+ [[ 2 -gt 60 ]]
7922023-11-30T05:04:16.540Z+ sleep 1
7932023-11-30T05:04:17.540Z+ retry=3
7942023-11-30T05:04:17.564Z+++ zoneadm list -n
7952023-11-30T05:04:17.564Z+++ grep oxz_ntp
7962023-11-30T05:04:17.564Z++ pfexec svcs -z -Hostate oxide/ntp
7972023-11-30T05:04:17.564Zsvcs: zone '-Hostate': entity not found
7982023-11-30T05:04:17.564Z++ true
7992023-11-30T05:04:17.564Z+ [[ '' != online ]]
8002023-11-30T05:04:17.564Z+ [[ 3 -gt 60 ]]
8012023-11-30T05:04:17.564Z+ sleep 1
8022023-11-30T05:04:18.561Z+ retry=4
8032023-11-30T05:04:18.585Z+++ zoneadm list -n
8042023-11-30T05:04:18.585Z+++ grep oxz_ntp
8052023-11-30T05:04:18.585Z++ pfexec svcs -z -Hostate oxide/ntp
8062023-11-30T05:04:18.585Zsvcs: zone '-Hostate': entity not found
8072023-11-30T05:04:18.585Z++ true
8082023-11-30T05:04:18.585Z+ [[ '' != online ]]
8092023-11-30T05:04:18.585Z+ [[ 4 -gt 60 ]]
8102023-11-30T05:04:18.585Z+ sleep 1
8112023-11-30T05:04:19.579Z+ retry=5
8122023-11-30T05:04:19.607Z+++ zoneadm list -n
8132023-11-30T05:04:19.607Z+++ grep oxz_ntp
8142023-11-30T05:04:19.607Z++ pfexec svcs -z -Hostate oxide/ntp
8152023-11-30T05:04:19.607Zsvcs: zone '-Hostate': entity not found
8162023-11-30T05:04:19.607Z++ true
8172023-11-30T05:04:19.607Z+ [[ '' != online ]]
8182023-11-30T05:04:19.607Z+ [[ 5 -gt 60 ]]
8192023-11-30T05:04:19.607Z+ sleep 1
8202023-11-30T05:04:20.600Z+ retry=6
8212023-11-30T05:04:20.624Z+++ zoneadm list -n
8222023-11-30T05:04:20.624Z+++ grep oxz_ntp
8232023-11-30T05:04:20.624Z++ pfexec svcs -z -Hostate oxide/ntp
8242023-11-30T05:04:20.624Zsvcs: zone '-Hostate': entity not found
8252023-11-30T05:04:20.624Z++ true
8262023-11-30T05:04:20.624Z+ [[ '' != online ]]
8272023-11-30T05:04:20.624Z+ [[ 6 -gt 60 ]]
8282023-11-30T05:04:20.624Z+ sleep 1
8292023-11-30T05:04:21.623Z+ retry=7
8302023-11-30T05:04:21.647Z+++ zoneadm list -n
8312023-11-30T05:04:21.647Z+++ grep oxz_ntp
8322023-11-30T05:04:21.647Z++ pfexec svcs -z -Hostate oxide/ntp
8332023-11-30T05:04:21.647Zsvcs: zone '-Hostate': entity not found
8342023-11-30T05:04:21.647Z++ true
8352023-11-30T05:04:21.647Z+ [[ '' != online ]]
8362023-11-30T05:04:21.647Z+ [[ 7 -gt 60 ]]
8372023-11-30T05:04:21.647Z+ sleep 1
8382023-11-30T05:04:22.645Z+ retry=8
8392023-11-30T05:04:22.669Z+++ zoneadm list -n
8402023-11-30T05:04:22.669Z+++ grep oxz_ntp
8412023-11-30T05:04:22.670Z++ pfexec svcs -z oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8 -Hostate oxide/ntp
8422023-11-30T05:04:22.696Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8432023-11-30T05:04:22.696Z++ true
8442023-11-30T05:04:22.696Z+ [[ '' != online ]]
8452023-11-30T05:04:22.696Z+ [[ 8 -gt 60 ]]
8462023-11-30T05:04:22.696Z+ sleep 1
8472023-11-30T05:04:23.675Z+ retry=9
8482023-11-30T05:04:23.699Z+++ zoneadm list -n
8492023-11-30T05:04:23.699Z+++ grep oxz_ntp
8502023-11-30T05:04:23.699Z++ pfexec svcs -z oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8 -Hostate oxide/ntp
8512023-11-30T05:04:23.722Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8522023-11-30T05:04:23.722Z++ true
8532023-11-30T05:04:23.722Z+ [[ '' != online ]]
8542023-11-30T05:04:23.722Z+ [[ 9 -gt 60 ]]
8552023-11-30T05:04:23.722Z+ sleep 1
8562023-11-30T05:04:24.708Z+ retry=10
8572023-11-30T05:04:24.735Z+++ zoneadm list -n
8582023-11-30T05:04:24.735Z+++ grep oxz_ntp
8592023-11-30T05:04:24.735Z++ pfexec svcs -z oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8 -Hostate oxide/ntp
8602023-11-30T05:04:24.735Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8612023-11-30T05:04:24.735Z++ true
8622023-11-30T05:04:24.761Z+ [[ '' != online ]]
8632023-11-30T05:04:24.762Z+ [[ 10 -gt 60 ]]
8642023-11-30T05:04:24.762Z+ sleep 1
8652023-11-30T05:04:25.736Z+ retry=11
8662023-11-30T05:04:25.760Z+++ zoneadm list -n
8672023-11-30T05:04:25.760Z+++ grep oxz_ntp
8682023-11-30T05:04:25.760Z++ pfexec svcs -z oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8 -Hostate oxide/ntp
8692023-11-30T05:04:25.760Z+ [[ disabled != online ]]
8702023-11-30T05:04:25.760Z+ [[ 11 -gt 60 ]]
8712023-11-30T05:04:25.760Z+ sleep 1
8722023-11-30T05:04:26.758Z+ retry=12
8732023-11-30T05:04:26.782Z+++ zoneadm list -n
8742023-11-30T05:04:26.782Z+++ grep oxz_ntp
8752023-11-30T05:04:26.782Z++ pfexec svcs -z oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8 -Hostate oxide/ntp
8762023-11-30T05:04:26.782Z+ [[ disabled != online ]]
8772023-11-30T05:04:26.782Z+ [[ 12 -gt 60 ]]
8782023-11-30T05:04:26.782Z+ sleep 1
8792023-11-30T05:04:27.781Z+ retry=13
8802023-11-30T05:04:27.805Z+++ zoneadm list -n
8812023-11-30T05:04:27.805Z+++ grep oxz_ntp
8822023-11-30T05:04:27.807Z++ pfexec svcs -z oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8 -Hostate oxide/ntp
8832023-11-30T05:04:27.807Z+ [[ disabled != online ]]
8842023-11-30T05:04:27.807Z+ [[ 13 -gt 60 ]]
8852023-11-30T05:04:27.807Z+ sleep 1
8862023-11-30T05:04:28.805Z+ retry=14
8872023-11-30T05:04:28.833Z+++ zoneadm list -n
8882023-11-30T05:04:28.833Z+++ grep oxz_ntp
8892023-11-30T05:04:28.833Z++ pfexec svcs -z oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8 -Hostate oxide/ntp
8902023-11-30T05:04:28.833Z+ [[ disabled != online ]]
8912023-11-30T05:04:28.833Z+ [[ 14 -gt 60 ]]
8922023-11-30T05:04:28.833Z+ sleep 1
8932023-11-30T05:04:29.829Z+ retry=15
8942023-11-30T05:04:29.857Z+++ zoneadm list -n
8952023-11-30T05:04:29.857Z+++ grep oxz_ntp
8962023-11-30T05:04:29.857Z++ pfexec svcs -z oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8 -Hostate oxide/ntp
8972023-11-30T05:04:29.857Z+ [[ disabled != online ]]
8982023-11-30T05:04:29.857Z+ [[ 15 -gt 60 ]]
8992023-11-30T05:04:29.857Z+ sleep 1
9002023-11-30T05:04:30.847Z+ retry=16
9012023-11-30T05:04:30.870Z+++ zoneadm list -n
9022023-11-30T05:04:30.870Z+++ grep oxz_ntp
9032023-11-30T05:04:30.870Z++ pfexec svcs -z oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8 -Hostate oxide/ntp
9042023-11-30T05:04:30.870Z+ [[ disabled != online ]]
9052023-11-30T05:04:30.870Z+ [[ 16 -gt 60 ]]
9062023-11-30T05:04:30.870Z+ sleep 1
9072023-11-30T05:04:31.871Z+ retry=17
9082023-11-30T05:04:31.895Z+++ zoneadm list -n
9092023-11-30T05:04:31.898Z+++ grep oxz_ntp
9102023-11-30T05:04:31.898Z++ pfexec svcs -z oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8 -Hostate oxide/ntp
9112023-11-30T05:04:31.898Z+ [[ disabled != online ]]
9122023-11-30T05:04:31.898Z+ [[ 17 -gt 60 ]]
9132023-11-30T05:04:31.898Z+ sleep 1
9142023-11-30T05:04:32.894Z+ retry=18
9152023-11-30T05:04:32.931Z+++ zoneadm list -n
9162023-11-30T05:04:32.931Z+++ grep oxz_ntp
9172023-11-30T05:04:32.931Z++ pfexec svcs -z oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8 -Hostate oxide/ntp
9182023-11-30T05:04:32.931Z+ [[ disabled != online ]]
9192023-11-30T05:04:32.931Z+ [[ 18 -gt 60 ]]
9202023-11-30T05:04:32.931Z+ sleep 1
 
9212023-11-30T05:04:33.748Zsock console login: Nov 30 05:04:33 sok ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9222023-11-30T05:04:34.105Z+ retry=19
9232023-11-30T05:04:34.139Z+++ zoneadm list -n
9242023-11-30T05:04:34.139Z+++ grep oxz_ntp
9252023-11-30T05:04:34.139Z++ pfexec svcs -z oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8 -Hostate oxide/ntp
9262023-11-30T05:04:34.139Z+ [[ disabled != online ]]
9272023-11-30T05:04:34.139Z+ [[ 19 -gt 60 ]]
9282023-11-30T05:04:34.139Z+ sleep 1
9292023-11-30T05:04:34.939Z+ retry=20
9302023-11-30T05:04:35.420Z+++ zoneadm list -n
9312023-11-30T05:04:35.420Z+++ grep oxz_ntp
9322023-11-30T05:04:35.420Z++ pfexec svcs -z oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8 -Hostate oxide/ntp
9332023-11-30T05:04:35.420Z+ [[ online != online ]]
9342023-11-30T05:04:35.420Z+ echo 'Waited for chrony: 20s'
9352023-11-30T05:04:35.420ZWaited for chrony: 20s
9362023-11-30T05:04:35.420Z+ export RUST_BACKTRACE=1
9372023-11-30T05:04:35.523Z+ RUST_BACKTRACE=1
9382023-11-30T05:04:35.523Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9392023-11-30T05:04:35.523Z+ ./tests/bootstrap
9402023-11-30T05:04:35.523ZUsing external DNS server at 10.151.2.164:53
9412023-11-30T05:04:35.523Z2023-11-30 05:04:34.807573263 UTC: attempting to log into API
9422023-11-30T05:05:03.424Z2023-11-30 05:04:49.581218555 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9432023-11-30T05:05:03.424Z2023-11-30 05:04:50.582303137 UTC: attempting to log into API
9442023-11-30T05:05:06.058Z2023-11-30 05:05:05.614517105 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9452023-11-30T05:05:07.058Z2023-11-30 05:05:06.616178530 UTC: attempting to log into API
9462023-11-30T05:05:22.093Z2023-11-30 05:05:22.060918500 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9472023-11-30T05:05:23.093Z2023-11-30 05:05:23.064739225 UTC: attempting to log into API
9482023-11-30T05:05:38.126Z2023-11-30 05:05:38.112878803 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9492023-11-30T05:05:39.128Z2023-11-30 05:05:39.114657840 UTC: attempting to log into API
 
9502023-11-30T05:05:46.683ZNov 30 05:05:46 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9512023-11-30T05:05:53.771ZNov 30 05:05:53 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9522023-11-30T05:05:54.161Z2023-11-30 05:05:54.148420639 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9532023-11-30T05:05:55.162Z2023-11-30 05:05:55.149491050 UTC: attempting to log into API
 
9542023-11-30T05:05:59.582ZNov 30 05:05:58 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
9552023-11-30T05:06:06.606ZNov 30 05:06:05 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9562023-11-30T05:06:10.195Z2023-11-30 05:06:10.182779147 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9572023-11-30T05:06:11.199Z2023-11-30 05:06:11.183806728 UTC: attempting to log into API
 
9582023-11-30T05:06:13.847ZNov 30 05:06:13 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9592023-11-30T05:06:26.229Z2023-11-30 05:06:26.217250845 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9602023-11-30T05:06:27.230Z2023-11-30 05:06:27.218086681 UTC: attempting to log into API
9612023-11-30T05:06:33.050Z2023-11-30 05:06:33.022383330 UTC: login succeeded
9622023-11-30T05:06:33.074Zwaiting for nexus to come up...
9632023-11-30T05:06:33.132Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9642023-11-30T05:06:33.173Zensuring datasets are ready...
9652023-11-30T05:06:35.907Zlet's roll.
9662023-11-30T05:06:35.936Z+ rm ./tests/bootstrap
9672023-11-30T05:06:35.972Z+ for test_bin in tests/*
9682023-11-30T05:06:35.972Z+ ./tests/bootstrap-8d69322849e0fa66
9692023-11-30T05:06:35.996Z
9702023-11-30T05:06:35.996Zrunning 0 tests
9712023-11-30T05:06:35.996Z
9722023-11-30T05:06:35.996Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9732023-11-30T05:06:35.996Z
9742023-11-30T05:06:35.996Z+ for test_bin in tests/*
9752023-11-30T05:06:35.996Z+ ./tests/end_to_end_tests-7d7c590276acd289
9762023-11-30T05:06:35.996Z
9772023-11-30T05:06:35.996Zrunning 2 tests
9782023-11-30T05:06:36.214Ztest no_spoof::no_spoof_here ... ok
9792023-11-30T05:07:35.996Ztest instance_launch::instance_launch has been running for over 60 seconds
9802023-11-30T05:09:32.945Ztest instance_launch::instance_launch ... ok
9812023-11-30T05:09:32.945Z
9822023-11-30T05:09:32.945Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 176.95s
9832023-11-30T05:09:32.945Z
9842023-11-30T05:09:32.969Z+ _exit_trap
9852023-11-30T05:09:32.969Z+ local status=0
9862023-11-30T05:09:32.969Z+ [[ 0 -eq 0 ]]
9872023-11-30T05:09:32.969Z+ exit 0
9882023-11-30T05:09:32.969Zprocess exited: duration 376687 ms, exit code 0
 
9892023-11-30T05:09:33.073Zfound 47 output files
9902023-11-30T05:09:33.073Zuploading: /var/svc/log/oxide-sled-agent:default.log (757141 bytes)
9912023-11-30T05:09:35.488Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9922023-11-30T05:09:35.488Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2378f003-eb07-4d50-a743-a6c6fbb54853/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
9932023-11-30T05:09:36.607Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2378f003-eb07-4d50-a743-a6c6fbb54853/root/var/svc/log/oxide-cockroachdb:default.log
9942023-11-30T05:09:36.608Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d58ded9e-4855-403c-aab1-b21f8a163eaf/root/var/svc/log/oxide-crucible-agent:default.log (24363 bytes)
9952023-11-30T05:09:37.787Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d58ded9e-4855-403c-aab1-b21f8a163eaf/root/var/svc/log/oxide-crucible-agent:default.log
9962023-11-30T05:09:37.787Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d58ded9e-4855-403c-aab1-b21f8a163eaf/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9972023-11-30T05:09:38.880Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d58ded9e-4855-403c-aab1-b21f8a163eaf/root/var/svc/log/oxide-crucible-downstairs:default.log
9982023-11-30T05:09:38.880Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d58ded9e-4855-403c-aab1-b21f8a163eaf/root/var/svc/log/oxide-crucible-downstairs:downstairs-2a7e325b-84f5-420e-b7f6-bb8da428040c.log (5536 bytes)
9992023-11-30T05:09:39.997Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d58ded9e-4855-403c-aab1-b21f8a163eaf/root/var/svc/log/oxide-crucible-downstairs:downstairs-2a7e325b-84f5-420e-b7f6-bb8da428040c.log
10002023-11-30T05:09:39.997Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d58ded9e-4855-403c-aab1-b21f8a163eaf/root/var/svc/log/oxide-crucible-downstairs:downstairs-ba1ea613-69fb-4e38-b0ad-10efe8450d1e.log (3049 bytes)
10012023-11-30T05:09:41.093Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d58ded9e-4855-403c-aab1-b21f8a163eaf/root/var/svc/log/oxide-crucible-downstairs:downstairs-ba1ea613-69fb-4e38-b0ad-10efe8450d1e.log
10022023-11-30T05:09:41.118Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_47b98128-dc9c-408e-a106-99fbb794492c/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10032023-11-30T05:09:42.186Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_47b98128-dc9c-408e-a106-99fbb794492c/root/var/svc/log/oxide-crucible-pantry:default.log
10042023-11-30T05:09:42.186Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_6677d13f-ea09-4183-b059-01d5bed3f937/root/var/svc/log/oxide-external_dns:default.log (15612 bytes)
10052023-11-30T05:09:43.322Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_6677d13f-ea09-4183-b059-01d5bed3f937/root/var/svc/log/oxide-external_dns:default.log
10062023-11-30T05:09:43.322Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_19f094b3-24e1-4305-b75b-41dbc104b5ed/root/var/svc/log/oxide-nexus:default.log (4667379 bytes)
10072023-11-30T05:09:46.322Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_19f094b3-24e1-4305-b75b-41dbc104b5ed/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4667379 -> 4671539
10082023-11-30T05:09:47.399Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_19f094b3-24e1-4305-b75b-41dbc104b5ed/root/var/svc/log/oxide-nexus:default.log
10092023-11-30T05:09:47.424Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_59d459fa-2980-4a31-853a-78639f130d49/root/var/svc/log/oxide-nexus:default.log (542867 bytes)
10102023-11-30T05:09:48.227Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_59d459fa-2980-4a31-853a-78639f130d49/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 542867 -> 547634
10112023-11-30T05:09:49.295Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_59d459fa-2980-4a31-853a-78639f130d49/root/var/svc/log/oxide-nexus:default.log
10122023-11-30T05:09:49.295Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_89dab195-3c28-4c1c-8043-7ddf2e3719d5/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10132023-11-30T05:09:50.399Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_89dab195-3c28-4c1c-8043-7ddf2e3719d5/root/var/svc/log/oxide-crucible-agent:default.log
10142023-11-30T05:09:50.399Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_89dab195-3c28-4c1c-8043-7ddf2e3719d5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10152023-11-30T05:09:51.490Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_89dab195-3c28-4c1c-8043-7ddf2e3719d5/root/var/svc/log/oxide-crucible-downstairs:default.log
10162023-11-30T05:09:51.491Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8d278c30-8aba-4af3-a762-23e0e8020e30/root/var/svc/log/oxide-crucible-agent:default.log (14578 bytes)
10172023-11-30T05:09:52.632Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8d278c30-8aba-4af3-a762-23e0e8020e30/root/var/svc/log/oxide-crucible-agent:default.log
10182023-11-30T05:09:52.632Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8d278c30-8aba-4af3-a762-23e0e8020e30/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10192023-11-30T05:09:53.730Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8d278c30-8aba-4af3-a762-23e0e8020e30/root/var/svc/log/oxide-crucible-downstairs:default.log
10202023-11-30T05:09:53.731Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8d278c30-8aba-4af3-a762-23e0e8020e30/root/var/svc/log/oxide-crucible-downstairs:downstairs-59e23f55-3ca9-4803-9ef0-552ef34f1d69.log (5533 bytes)
10212023-11-30T05:09:54.844Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8d278c30-8aba-4af3-a762-23e0e8020e30/root/var/svc/log/oxide-crucible-downstairs:downstairs-59e23f55-3ca9-4803-9ef0-552ef34f1d69.log
10222023-11-30T05:09:54.869Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_f10b9315-299e-4af8-82fb-de05bfb3013c/root/var/svc/log/oxide-external_dns:default.log (15608 bytes)
10232023-11-30T05:09:55.979Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_f10b9315-299e-4af8-82fb-de05bfb3013c/root/var/svc/log/oxide-external_dns:default.log
10242023-11-30T05:09:55.979Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_5f8042dd-9f3d-4174-aaf2-9731815af290/root/var/svc/log/oxide-internal_dns:default.log (103132 bytes)
10252023-11-30T05:09:57.329Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_5f8042dd-9f3d-4174-aaf2-9731815af290/root/var/svc/log/oxide-internal_dns:default.log
10262023-11-30T05:09:57.353Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c039a133-2c36-41e2-a03e-a4aa250c3668/root/var/svc/log/oxide-nexus:default.log (512180 bytes)
10272023-11-30T05:09:58.141Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c039a133-2c36-41e2-a03e-a4aa250c3668/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 512180 -> 517557
10282023-11-30T05:09:59.618Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c039a133-2c36-41e2-a03e-a4aa250c3668/root/var/svc/log/oxide-nexus:default.log
10292023-11-30T05:09:59.618Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_d3ed68f7-5725-40b4-a8fd-03303641dc90/root/var/svc/log/oxide-clickhouse:default.log (672928 bytes)
10302023-11-30T05:10:00.608Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_d3ed68f7-5725-40b4-a8fd-03303641dc90/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 672928 -> 743713
10312023-11-30T05:10:01.678Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_d3ed68f7-5725-40b4-a8fd-03303641dc90/root/var/svc/log/oxide-clickhouse:default.log
10322023-11-30T05:10:01.678Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_c288ef13-5ad0-48d5-ab70-28e5376cd5f8/root/var/svc/log/oxide-crucible-pantry:default.log (2337 bytes)
10332023-11-30T05:10:02.775Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_c288ef13-5ad0-48d5-ab70-28e5376cd5f8/root/var/svc/log/oxide-crucible-pantry:default.log
10342023-11-30T05:10:02.775Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_81a5afb6-128a-4d78-99fb-b8e63fb2d2e1/root/var/svc/log/oxide-oximeter:default.log (176029 bytes)
10352023-11-30T05:10:03.270Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_81a5afb6-128a-4d78-99fb-b8e63fb2d2e1/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 176029 -> 197446
10362023-11-30T05:10:04.340Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_81a5afb6-128a-4d78-99fb-b8e63fb2d2e1/root/var/svc/log/oxide-oximeter:default.log
10372023-11-30T05:10:04.340Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_329da024-0f70-4617-a7fd-657d5a0360fa/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10382023-11-30T05:10:05.434Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_329da024-0f70-4617-a7fd-657d5a0360fa/root/var/svc/log/oxide-cockroachdb:default.log
10392023-11-30T05:10:05.467Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_9a6c7477-3d47-4e27-98a4-0818ccb32c63/root/var/svc/log/oxide-internal_dns:default.log (106862 bytes)
10402023-11-30T05:10:06.816Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_9a6c7477-3d47-4e27-98a4-0818ccb32c63/root/var/svc/log/oxide-internal_dns:default.log
10412023-11-30T05:10:06.844Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_06164783-74d7-4c1a-a37c-d94e3d246f62/root/var/svc/log/oxide-cockroachdb:default.log (5341 bytes)
10422023-11-30T05:10:07.912Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_06164783-74d7-4c1a-a37c-d94e3d246f62/root/var/svc/log/oxide-cockroachdb:default.log
10432023-11-30T05:10:07.912Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_6bbfcf3e-0a47-4494-b7b2-5e5a55a2f027/root/var/svc/log/oxide-cockroachdb:default.log (5443 bytes)
10442023-11-30T05:10:09.006Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_6bbfcf3e-0a47-4494-b7b2-5e5a55a2f027/root/var/svc/log/oxide-cockroachdb:default.log
10452023-11-30T05:10:09.006Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_706a16d2-8117-4b2c-9f15-7241545fd7a9/root/var/svc/log/oxide-crucible-agent:default.log (14114 bytes)
10462023-11-30T05:10:10.142Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_706a16d2-8117-4b2c-9f15-7241545fd7a9/root/var/svc/log/oxide-crucible-agent:default.log
10472023-11-30T05:10:10.166Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_706a16d2-8117-4b2c-9f15-7241545fd7a9/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10482023-11-30T05:10:11.248Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_706a16d2-8117-4b2c-9f15-7241545fd7a9/root/var/svc/log/oxide-crucible-downstairs:default.log
10492023-11-30T05:10:11.248Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_706a16d2-8117-4b2c-9f15-7241545fd7a9/root/var/svc/log/oxide-crucible-downstairs:downstairs-90946a8d-b13d-4f2b-a9db-81428cf83ddf.log (3050 bytes)
10502023-11-30T05:10:12.751Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_706a16d2-8117-4b2c-9f15-7241545fd7a9/root/var/svc/log/oxide-crucible-downstairs:downstairs-90946a8d-b13d-4f2b-a9db-81428cf83ddf.log
10512023-11-30T05:10:12.751Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_d04a0034-3491-44ce-a90a-43f6b6188d04/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10522023-11-30T05:10:13.845Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_d04a0034-3491-44ce-a90a-43f6b6188d04/root/var/svc/log/oxide-crucible-pantry:default.log
10532023-11-30T05:10:13.873Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_5c79806c-e162-4221-b724-86eb46f77e34/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10542023-11-30T05:10:14.937Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_5c79806c-e162-4221-b724-86eb46f77e34/root/var/svc/log/oxide-cockroachdb:default.log
10552023-11-30T05:10:14.962Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7c620dab-ad98-4c48-af78-7625cd4d6322/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10562023-11-30T05:10:16.030Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7c620dab-ad98-4c48-af78-7625cd4d6322/root/var/svc/log/oxide-crucible-agent:default.log
10572023-11-30T05:10:16.054Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7c620dab-ad98-4c48-af78-7625cd4d6322/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10582023-11-30T05:10:17.122Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_7c620dab-ad98-4c48-af78-7625cd4d6322/root/var/svc/log/oxide-crucible-downstairs:default.log
10592023-11-30T05:10:17.122Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_aca4c8c4-8cd1-4af0-a67d-354a75119c3f/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10602023-11-30T05:10:18.215Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_aca4c8c4-8cd1-4af0-a67d-354a75119c3f/root/var/svc/log/oxide-crucible-agent:default.log
10612023-11-30T05:10:18.216Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_aca4c8c4-8cd1-4af0-a67d-354a75119c3f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10622023-11-30T05:10:19.309Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_aca4c8c4-8cd1-4af0-a67d-354a75119c3f/root/var/svc/log/oxide-crucible-downstairs:default.log
10632023-11-30T05:10:19.334Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8/root/var/svc/log/oxide-ntp:default.log (1244 bytes)
10642023-11-30T05:10:20.817Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8/root/var/svc/log/oxide-ntp:default.log
10652023-11-30T05:10:20.817Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ebb6e177-18c3-48cf-ab42-2fcfb3de4637/root/var/svc/log/oxide-crucible-agent:default.log (14119 bytes)
10662023-11-30T05:10:22.367Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ebb6e177-18c3-48cf-ab42-2fcfb3de4637/root/var/svc/log/oxide-crucible-agent:default.log
10672023-11-30T05:10:22.391Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ebb6e177-18c3-48cf-ab42-2fcfb3de4637/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10682023-11-30T05:10:23.460Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ebb6e177-18c3-48cf-ab42-2fcfb3de4637/root/var/svc/log/oxide-crucible-downstairs:default.log
10692023-11-30T05:10:23.460Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ebb6e177-18c3-48cf-ab42-2fcfb3de4637/root/var/svc/log/oxide-crucible-downstairs:downstairs-72667f94-4ec9-4760-8d86-1cd928a4b58e.log (3049 bytes)
10702023-11-30T05:10:24.555Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ebb6e177-18c3-48cf-ab42-2fcfb3de4637/root/var/svc/log/oxide-crucible-downstairs:downstairs-72667f94-4ec9-4760-8d86-1cd928a4b58e.log
10712023-11-30T05:10:24.555Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_e57f30ef-3b2b-41ca-b556-374db85e6a95/root/var/svc/log/oxide-internal_dns:default.log (100288 bytes)
10722023-11-30T05:10:24.866Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_e57f30ef-3b2b-41ca-b556-374db85e6a95/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 100288 -> 104043
10732023-11-30T05:10:25.937Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_e57f30ef-3b2b-41ca-b556-374db85e6a95/root/var/svc/log/oxide-internal_dns:default.log
10742023-11-30T05:10:25.962Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_36fa88ac-e37f-43ce-9f09-a2b3d9980d4d/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10752023-11-30T05:10:27.458Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_36fa88ac-e37f-43ce-9f09-a2b3d9980d4d/root/var/svc/log/oxide-crucible-agent:default.log
10762023-11-30T05:10:27.458Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_36fa88ac-e37f-43ce-9f09-a2b3d9980d4d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10772023-11-30T05:10:28.551Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_36fa88ac-e37f-43ce-9f09-a2b3d9980d4d/root/var/svc/log/oxide-crucible-downstairs:default.log
10782023-11-30T05:10:28.551Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5765b79d-36e9-4cb4-b065-c23ef20649ad/root/var/svc/log/oxide-crucible-agent:default.log (13686 bytes)
10792023-11-30T05:10:29.691Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5765b79d-36e9-4cb4-b065-c23ef20649ad/root/var/svc/log/oxide-crucible-agent:default.log
10802023-11-30T05:10:29.691Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5765b79d-36e9-4cb4-b065-c23ef20649ad/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10812023-11-30T05:10:31.258Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5765b79d-36e9-4cb4-b065-c23ef20649ad/root/var/svc/log/oxide-crucible-downstairs:default.log
10822023-11-30T05:10:31.282Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5765b79d-36e9-4cb4-b065-c23ef20649ad/root/var/svc/log/oxide-crucible-downstairs:downstairs-b7731fae-0fcc-42d2-8262-bd72a1a700f9.log (5535 bytes)
10832023-11-30T05:10:32.373Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5765b79d-36e9-4cb4-b065-c23ef20649ad/root/var/svc/log/oxide-crucible-downstairs:downstairs-b7731fae-0fcc-42d2-8262-bd72a1a700f9.log
10842023-11-30T05:10:32.397Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8/root/var/log/chrony/measurements.log (11645 bytes)
10852023-11-30T05:10:32.464Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8/root/var/log/chrony/measurements.log" changed size mid upload: 11645 -> 13015
10862023-11-30T05:10:33.951Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8/root/var/log/chrony/measurements.log
10872023-11-30T05:10:33.976Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8/root/var/log/chrony/statistics.log (7020 bytes)
10882023-11-30T05:10:33.976Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8/root/var/log/chrony/statistics.log" changed size mid upload: 7020 -> 8658
10892023-11-30T05:10:34.689Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8/root/var/log/chrony/statistics.log
10902023-11-30T05:10:34.689Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8/root/var/log/chrony/tracking.log (1723 bytes)
10912023-11-30T05:10:34.737Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8/root/var/log/chrony/tracking.log" changed size mid upload: 1723 -> 1855
10922023-11-30T05:10:35.811Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_5b835c0e-67d6-4f14-acfd-6060950001f8/root/var/log/chrony/tracking.log