01K3A4V8F0DEHSGCBRAABSPZWF: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-*.log*",
        "%/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
        "/tmp/kstat/*.kstat",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.37",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01K3A4W6Q07GN5MWR75RKMPH3P

Tags:

Output:

SEQ GLOBAL TIME DETAILS
12025-08-23T01:00:50.976Zjob dependencies complete; ready to run (waiting for 38 m 25 s)
22025-08-23T01:00:50.981Zjob assigned to worker 01K3A6GP2GPWN7MRP6ZW34NWYX [factory lab, buskin/16436] (queued for 0 s)
32025-08-23T01:00:51.231Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42025-08-23T01:00:51.277Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52025-08-23T01:00:51.313Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62025-08-23T01:00:51.346Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72025-08-23T01:00:51.380Zhttp://172.20.0.5:9969/postboot/buskin... ok
82025-08-23T01:00:51.436ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92025-08-23T01:00:51.469Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d5
102025-08-23T01:00:51.503ZOxide Helios Version helios-2.0.23457 64-bit
112025-08-23T01:00:51.537ZConfiguring devices.
122025-08-23T01:00:51.570Z##################################################
132025-08-23T01:00:51.633Z#### ############# ##
142025-08-23T01:00:51.966Z### ### ############ ##
152025-08-23T01:00:52.050Z## ### # ## ### ## Oxide Computer Company ##
162025-08-23T01:00:52.084Z## ## ## ### # ### ##
172025-08-23T01:00:52.117Z## # ### #### #### This Station Under ##
182025-08-23T01:00:52.150Z### ### #### # ### Computer Control ##
192025-08-23T01:00:52.184Z#### #### ### ## ##
202025-08-23T01:00:52.218Z##################################################
212025-08-23T01:00:52.251ZHostname: buskin
222025-08-23T01:00:52.285ZLoading smf(7) service descriptions:
232025-08-23T01:00:53.243Zdownloading input: /input/package/work/package.tar.gz
242025-08-23T01:07:17.084Zdownloaded input: /input/package/work/package.tar.gz
 
252025-08-23T01:07:17.084Zstarting task 0: "setup"
262025-08-23T01:07:17.109Z++ uname -s
272025-08-23T01:07:17.109Z+ kern=SunOS
282025-08-23T01:07:17.109Z+ build_user=build
292025-08-23T01:07:17.109Z+ build_uid=12345
302025-08-23T01:07:17.109Z+ work_dir=/work
312025-08-23T01:07:17.109Z+ input_dir=/input
322025-08-23T01:07:17.109Z+ [[ 0 == 12345 ]]
332025-08-23T01:07:17.109Z+ case "$kern" in
342025-08-23T01:07:17.109Z+ groupadd -g 12345 build
352025-08-23T01:07:17.109Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
362025-08-23T01:07:19.033Z+ zfs create -o mountpoint=/work rpool/work
372025-08-23T01:07:19.082Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
382025-08-23T01:07:19.106Z+ home_fs=zfs
392025-08-23T01:07:19.106Z+ [[ zfs == autofs ]]
402025-08-23T01:07:19.106Z+ mkdir -p /home/build
412025-08-23T01:07:19.106Z+ chown build:build /home/build /work
422025-08-23T01:07:21.089Z+ chmod 0700 /home/build /work
432025-08-23T01:07:21.114Zprocess exited: duration 4077 ms, exit code 0
 
442025-08-23T01:07:21.162Zstarting task 1: "authentication"
452025-08-23T01:07:21.207Zprocess exited: duration 44 ms, exit code 0
 
462025-08-23T01:07:21.256Zstarting task 2: "build"
472025-08-23T01:07:21.281Z+ trap _exit_trap EXIT
482025-08-23T01:07:21.281Z+ OPTE_COMMIT=
492025-08-23T01:07:21.281Z+ [[ x != \x ]]
502025-08-23T01:07:21.281Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
512025-08-23T01:07:21.375Zsecpolicy_fs_linkdir: 0x55 = 0x55
522025-08-23T01:07:21.375Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
532025-08-23T01:07:21.375Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
542025-08-23T01:07:21.375Zsecpolicy_fs_linkdir+3: 0 = 0xe5
552025-08-23T01:07:21.375Zsecpolicy_fs_linkdir+4: 0 = 0xb8
562025-08-23T01:07:21.375Zsecpolicy_fs_linkdir+5: 0 = 0x1
572025-08-23T01:07:21.375Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
582025-08-23T01:07:21.375Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
592025-08-23T01:07:21.375Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
602025-08-23T01:07:21.376Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
612025-08-23T01:07:21.376Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
622025-08-23T01:07:21.400Z+ [[ -d /opt/oxide ]]
632025-08-23T01:07:21.400Z+ pfexec mv /opt/oxide /opt/oxide-underneath
642025-08-23T01:07:21.400Z+ pfexec mkdir /opt/oxide
652025-08-23T01:07:21.400Z+ pfexec mount -F tmpfs -O swap /opt/oxide
662025-08-23T01:07:21.400Z+ [[ -d /opt/oxide-underneath ]]
672025-08-23T01:07:21.400Z+ cd /opt/oxide-underneath
682025-08-23T01:07:21.400Z+ pfexec tar ceEp@/f - .
692025-08-23T01:07:21.400Z+ cd /opt/oxide
702025-08-23T01:07:21.400Z+ pfexec tar xveEp@/f -
712025-08-23T01:07:21.473Zx ., 0 bytes, 0 tape blocks
722025-08-23T01:07:21.473Zx ./opte, 0 bytes, 0 tape blocks
732025-08-23T01:07:21.473Zx ./opte/bin, 0 bytes, 0 tape blocks
742025-08-23T01:07:21.473Zx ./opte/bin/opteadm, 20875880 bytes, 40774 tape blocks
752025-08-23T01:07:21.497Z+ pfexec rm -rf /opt/oxide-underneath
762025-08-23T01:07:21.497Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
772025-08-23T01:07:21.497Z+ pfexec mkdir /opt/oxide/work
782025-08-23T01:07:21.497Z+ pfexec chown build:build /opt/oxide/work
792025-08-23T01:07:21.497Z+ cd /opt/oxide/work
802025-08-23T01:07:21.497Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
812025-08-23T01:07:40.864ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
822025-08-23T01:07:40.865Zx .github/buildomat/ci-env.sh, 196 bytes, 1 tape blocks
832025-08-23T01:07:40.865Zx out/target/test, 107 bytes, 1 tape blocks
842025-08-23T01:07:40.865Zx out/npuzone/npuzone, 23699288 bytes, 46288 tape blocks
852025-08-23T01:07:40.865Zx package-manifest.toml, 34353 bytes, 68 tape blocks
862025-08-23T01:07:40.865Zx smf/sled-agent/non-gimlet/config.toml, 4942 bytes, 10 tape blocks
872025-08-23T01:07:40.865Zx target/release/omicron-package, 42308160 bytes, 82634 tape blocks
882025-08-23T01:07:40.865Zx target/release/xtask, 12078456 bytes, 23591 tape blocks
892025-08-23T01:07:40.865Zx target/debug/bootstrap, 251046192 bytes, 490325 tape blocks
902025-08-23T01:07:40.865Zx tests/bootstrap-057076fe054df040, 24201664 bytes, 47269 tape blocks
912025-08-23T01:07:40.865Zx tests/commtest-ed10da61875733fe, 26338936 bytes, 51444 tape blocks
922025-08-23T01:07:40.865Zx tests/dhcp_server-60f4352cb9479535, 24201672 bytes, 47269 tape blocks
932025-08-23T01:07:40.865Zx tests/end_to_end_tests-8572aa8b9b995115, 352765048 bytes, 688995 tape blocks
942025-08-23T01:07:40.865Zx out/clickhouse.tar.gz, 326351610 bytes, 637406 tape blocks
952025-08-23T01:07:40.865Zx out/clickhouse_keeper.tar.gz, 309636282 bytes, 604759 tape blocks
962025-08-23T01:07:40.865Zx out/clickhouse_server.tar.gz, 326350304 bytes, 637403 tape blocks
972025-08-23T01:07:40.865Zx out/cockroachdb.tar.gz, 166976529 bytes, 326127 tape blocks
982025-08-23T01:07:40.865Zx out/crucible-dtrace.tar, 101376 bytes, 198 tape blocks
992025-08-23T01:07:40.865Zx out/crucible-pantry-zone.tar.gz, 39281140 bytes, 76721 tape blocks
1002025-08-23T01:07:40.865Zx out/crucible-zone.tar.gz, 49319751 bytes, 96328 tape blocks
1012025-08-23T01:07:40.865Zx out/external-dns.tar.gz, 52603407 bytes, 102742 tape blocks
1022025-08-23T01:07:40.865Zx out/internal-dns.tar.gz, 52603007 bytes, 102741 tape blocks
1032025-08-23T01:07:40.865Zx out/mg-ddm-gz.tar, 93026304 bytes, 181692 tape blocks
1042025-08-23T01:07:40.865Zx out/nexus.tar.gz, 137428411 bytes, 268415 tape blocks
1052025-08-23T01:07:40.865Zx out/ntp.tar.gz, 34212010 bytes, 66821 tape blocks
1062025-08-23T01:07:40.865Zx out/omicron-gateway-softnpu.tar.gz, 32018012 bytes, 62536 tape blocks
1072025-08-23T01:07:40.865Zx out/omicron-sled-agent.tar, 146146816 bytes, 285443 tape blocks
1082025-08-23T01:07:40.865Zx out/overlay.tar.gz, 3007 bytes, 6 tape blocks
1092025-08-23T01:07:40.865Zx out/oximeter.tar.gz, 65311740 bytes, 127562 tape blocks
1102025-08-23T01:07:40.865Zx out/oxlog.tar, 4100096 bytes, 8008 tape blocks
1112025-08-23T01:07:40.865Zx out/probe.tar.gz, 3109568 bytes, 6074 tape blocks
1122025-08-23T01:07:40.865Zx out/propolis-server.tar.gz, 70231356 bytes, 137171 tape blocks
1132025-08-23T01:07:40.865Zx out/pumpkind-gz.tar, 27287040 bytes, 53295 tape blocks
1142025-08-23T01:07:40.865Zx out/switch-softnpu.tar.gz, 352066623 bytes, 687631 tape blocks
1152025-08-23T01:07:40.890Z
1162025-08-23T01:07:40.890Zreal 19.371260868
1172025-08-23T01:07:40.890Zuser 17.341909577
1182025-08-23T01:07:40.890Zsys 3.422829059
1192025-08-23T01:07:40.890Ztrap 0.000837350
1202025-08-23T01:07:40.890Ztflt 0.000000000
1212025-08-23T01:07:40.890Zdflt 0.000000000
1222025-08-23T01:07:40.890Zkflt 0.000000000
1232025-08-23T01:07:40.890Zlock 0.000000000
1242025-08-23T01:07:40.891Zslp 17.483465022
1252025-08-23T01:07:40.891Zlat 0.491290090
1262025-08-23T01:07:40.891Zstop 0.000125672
1272025-08-23T01:07:40.891Z+ source .github/buildomat/ci-env.sh
1282025-08-23T01:07:40.891Z++ export CARGO_TERM_COLOR=always
1292025-08-23T01:07:40.891Z++ CARGO_TERM_COLOR=always
1302025-08-23T01:07:40.891Z+ bmat address ls
1312025-08-23T01:07:40.891ZNAME CIDR FIRST LAST COUNT
1322025-08-23T01:07:40.891Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
1332025-08-23T01:07:40.891Z++ bmat address ls -f extra -Ho first,last,count
1342025-08-23T01:07:40.891Z+ set -- 10.151.1.164 10.151.1.227 64
1352025-08-23T01:07:40.891Z+ EXTRA_IP_START=10.151.1.164
1362025-08-23T01:07:40.891Z+ EXTRA_IP_END=10.151.1.227
1372025-08-23T01:07:40.891Z+ EXTRA_IP_COUNT=64
1382025-08-23T01:07:40.891Z+ (( EXTRA_IP_COUNT >= 32 ))
1392025-08-23T01:07:40.891Z+ EXTRA_IP_BASE=10.151.1
1402025-08-23T01:07:40.891Z+ EXTRA_IP_FOCTET=164
1412025-08-23T01:07:40.891Z+ EXTRA_IP_LOCTET=227
1422025-08-23T01:07:40.891Z+ SERVICE_IP_POOL_START=10.151.1.164
1432025-08-23T01:07:40.891Z+ SERVICE_IP_POOL_END=10.151.1.173
1442025-08-23T01:07:40.891Z+ DNS_IP1=10.151.1.164
1452025-08-23T01:07:40.891Z+ DNS_IP2=10.151.1.165
1462025-08-23T01:07:40.891Z+ UPLINK_IP=10.151.1.174
1472025-08-23T01:07:40.891Z+ IPPOOL_START=10.151.1.175
1482025-08-23T01:07:40.891Z+ IPPOOL_END=10.151.1.227
1492025-08-23T01:07:40.891Z++ ipadm show-addr -po type,addr
1502025-08-23T01:07:40.891Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
1512025-08-23T01:07:40.891Z+ GATEWAY_IP=10.151.1.100
1522025-08-23T01:07:40.891Z+ [[ -n 10.151.1.100 ]]
1532025-08-23T01:07:40.891Z+ ping -s 10.151.1.100 56 1
1542025-08-23T01:07:40.891ZPING 10.151.1.100: 56 data bytes
1552025-08-23T01:07:40.891Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
1562025-08-23T01:07:40.891Z
1572025-08-23T01:07:40.891Z----10.151.1.100 PING Statistics----
1582025-08-23T01:07:40.891Z1 packets transmitted, 1 packets received, 0% packet loss
1592025-08-23T01:07:40.891Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
1602025-08-23T01:07:40.891Z++ arp -an
1612025-08-23T01:07:40.891Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
1622025-08-23T01:07:40.916Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
1632025-08-23T01:07:40.916Z+ routeadm -e ipv4-forwarding -u
1642025-08-23T01:07:40.992Z+ PXA_START=10.151.1.164
1652025-08-23T01:07:40.992Z+ PXA_END=10.151.1.227
1662025-08-23T01:07:40.992Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
1672025-08-23T01:07:41.158Z+ ptime -m pfexec ./target/release/xtask virtual-hardware --vdev-dir /scratch create --gateway-ip 10.151.1.100 --gateway-mac 18:c0:4d:0d:9f:b2 --pxa-start 10.151.1.164 --pxa-end 10.151.1.227
1682025-08-23T01:07:41.183ZCouldn't find Cargo.toml, using /opt/oxide/work as workspace root
1692025-08-23T01:07:41.250Zcreating virtual hardware
1702025-08-23T01:07:41.275Zcreating /scratch/m2_0.vdev
1712025-08-23T01:07:41.275Zcreating /scratch/m2_1.vdev
1722025-08-23T01:07:41.275Zcreating /scratch/u2_0.vdev
1732025-08-23T01:07:41.275Zcreating /scratch/u2_1.vdev
1742025-08-23T01:07:41.275Zcreating /scratch/u2_2.vdev
1752025-08-23T01:07:41.275Zcreating /scratch/u2_3.vdev
1762025-08-23T01:07:41.275Zcreating /scratch/u2_4.vdev
1772025-08-23T01:07:41.275Zcreating /scratch/u2_5.vdev
1782025-08-23T01:07:41.275Zcreating /scratch/u2_6.vdev
1792025-08-23T01:07:41.275Zcreating /scratch/u2_7.vdev
1802025-08-23T01:07:41.275Zcreating /scratch/u2_8.vdev
1812025-08-23T01:07:41.275ZSimnet net0/sc0_0 exists
1822025-08-23T01:07:41.299ZSimnet net1/sc1_0 exists
1832025-08-23T01:07:41.299ZVnic sc0_1 exists
1842025-08-23T01:07:55.480ZUsing 10.151.1.100 as gateway ip
1852025-08-23T01:07:55.480Zusing 18:C0:4D:0D:9F:B2 as gateway mac
1862025-08-23T01:07:55.480Zconfiguring SoftNPU ARP entry
1872025-08-23T01:07:55.506Zconfiguring SoftNPU proxy ARP
1882025-08-23T01:07:55.557ZSoftNPU state:
1892025-08-23T01:07:55.557Z local v6:
1902025-08-23T01:07:55.557Z local v4:
1912025-08-23T01:07:55.557Z router v6:
1922025-08-23T01:07:55.557Z router v4_idx:
1932025-08-23T01:07:55.557Z router v4_routes:
1942025-08-23T01:07:55.557Z resolver v4:
1952025-08-23T01:07:55.557Z 10.151.1.100 -> 18:c0:4d:0d:9f:b2
1962025-08-23T01:07:55.557Z resolver v6:
1972025-08-23T01:07:55.557Z nat v4:
1982025-08-23T01:07:55.557Z nat_v6:
1992025-08-23T01:07:55.557Z port_mac:
2002025-08-23T01:07:55.557Z proxy arp:
2012025-08-23T01:07:55.557Z 10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
2022025-08-23T01:07:55.557Zcreated virtual hardware
2032025-08-23T01:07:55.582Z
2042025-08-23T01:07:55.582Zreal 14.395696109
2052025-08-23T01:07:55.582Zuser 0.637376543
2062025-08-23T01:07:55.582Zsys 3.086851422
2072025-08-23T01:07:55.582Ztrap 0.000055286
2082025-08-23T01:07:55.582Ztflt 0.000124144
2092025-08-23T01:07:55.582Zdflt 0.000700229
2102025-08-23T01:07:55.582Zkflt 0.000000000
2112025-08-23T01:07:55.582Zlock 11:10.136486378
2122025-08-23T01:07:55.582Zslp 43.496741889
2132025-08-23T01:07:55.582Zlat 0.107892838
2142025-08-23T01:07:55.582Zstop 0.393231042
2152025-08-23T01:07:55.582Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml pkg/config.toml
2162025-08-23T01:07:55.606Z+ sed -E -i~ 's/(m2|u2)(.*\.vdev)/\/scratch\/\1\2/g' pkg/config.toml
2172025-08-23T01:07:55.654Z+ diff -u pkg/config.toml~ pkg/config.toml
2182025-08-23T01:07:55.654Z--- pkg/config.toml~ Mon Jul 24 01:21:28 2006
2192025-08-23T01:07:55.654Z+++ pkg/config.toml Sat Aug 23 01:07:55 2025
2202025-08-23T01:07:55.654Z@@ -29,18 +29,18 @@
2212025-08-23T01:07:55.654Z # These paths have the prefix of either "u2" or "m2", followed by an underscore,
2222025-08-23T01:07:55.654Z # followed by a string that is embedded into their fake serial values.
2232025-08-23T01:07:55.654Z vdevs = [
2242025-08-23T01:07:55.654Z- "m2_0.vdev",
2252025-08-23T01:07:55.654Z- "m2_1.vdev",
2262025-08-23T01:07:55.654Z+ "/scratch/m2_0.vdev",
2272025-08-23T01:07:55.654Z+ "/scratch/m2_1.vdev",
2282025-08-23T01:07:55.655Z
2292025-08-23T01:07:55.655Z- "u2_0.vdev",
2302025-08-23T01:07:55.655Z- "u2_1.vdev",
2312025-08-23T01:07:55.655Z- "u2_2.vdev",
2322025-08-23T01:07:55.655Z- "u2_3.vdev",
2332025-08-23T01:07:55.655Z- "u2_4.vdev",
2342025-08-23T01:07:55.655Z- "u2_5.vdev",
2352025-08-23T01:07:55.655Z- "u2_6.vdev",
2362025-08-23T01:07:55.655Z- "u2_7.vdev",
2372025-08-23T01:07:55.655Z- "u2_8.vdev",
2382025-08-23T01:07:55.655Z+ "/scratch/u2_0.vdev",
2392025-08-23T01:07:55.655Z+ "/scratch/u2_1.vdev",
2402025-08-23T01:07:55.655Z+ "/scratch/u2_2.vdev",
2412025-08-23T01:07:55.655Z+ "/scratch/u2_3.vdev",
2422025-08-23T01:07:55.655Z+ "/scratch/u2_4.vdev",
2432025-08-23T01:07:55.655Z+ "/scratch/u2_5.vdev",
2442025-08-23T01:07:55.655Z+ "/scratch/u2_6.vdev",
2452025-08-23T01:07:55.655Z+ "/scratch/u2_7.vdev",
2462025-08-23T01:07:55.655Z+ "/scratch/u2_8.vdev",
2472025-08-23T01:07:55.655Z ]
2482025-08-23T01:07:55.655Z
2492025-08-23T01:07:55.655Z # The amount of memory held back for services which exist between zero and one
2502025-08-23T01:07:55.655Z+ true
2512025-08-23T01:07:55.655Z++ grep -c -E 'u2.*\.vdev' pkg/config.toml
2522025-08-23T01:07:55.655Z+ EXPECTED_ZPOOL_COUNT=9
2532025-08-23T01:07:55.655Z+ echo 'expected number of zpools is 9'
2542025-08-23T01:07:55.655Zexpected number of zpools is 9
2552025-08-23T01:07:55.655Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
2562025-08-23T01:07:55.655Z+ SILO_NAME=recovery
2572025-08-23T01:07:55.655Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
2582025-08-23T01:07:55.655Z+ EXTERNAL_DNS_DOMAIN=oxide.test
2592025-08-23T01:07:55.655Z+ sed -i~ '
2602025-08-23T01:07:55.655Z /^external_dns_ips/c\
2612025-08-23T01:07:55.655Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
2622025-08-23T01:07:55.655Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
2632025-08-23T01:07:55.655Z /^first/c\
2642025-08-23T01:07:55.655Zfirst = "10.151.1.164"
2652025-08-23T01:07:55.655Z /^last/c\
2662025-08-23T01:07:55.655Zlast = "10.151.1.173"
2672025-08-23T01:07:55.656Z }
2682025-08-23T01:07:55.656Z /^infra_ip_first/c\
2692025-08-23T01:07:55.656Zinfra_ip_first = "10.151.1.174"
2702025-08-23T01:07:55.656Z /^infra_ip_last/c\
2712025-08-23T01:07:55.656Zinfra_ip_last = "10.151.1.174"
2722025-08-23T01:07:55.656Z /^\[\[rack_network_config.ports/,/^$/ {
2732025-08-23T01:07:55.656Z /^routes/c\
2742025-08-23T01:07:55.656Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
2752025-08-23T01:07:55.656Z /^addresses/c\
2762025-08-23T01:07:55.656Zaddresses = \[{address = "10.151.1.174/24"} \]
2772025-08-23T01:07:55.656Z }
2782025-08-23T01:07:55.656Z' pkg/config-rss.toml
2792025-08-23T01:07:55.656Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
2802025-08-23T01:07:55.656Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
2812025-08-23T01:07:55.656Z+++ pkg/config-rss.toml Sat Aug 23 01:07:55 2025
2822025-08-23T01:07:55.656Z@@ -33,7 +33,7 @@
2832025-08-23T01:07:55.656Z # the DNS domain delegated to the rack by the customer. Each of these addresses
2842025-08-23T01:07:55.656Z # must be contained in one of the "internal services" IP Pool ranges listed
2852025-08-23T01:07:55.656Z # below.
2862025-08-23T01:07:55.656Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
2872025-08-23T01:07:55.656Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
2882025-08-23T01:07:55.656Z
2892025-08-23T01:07:55.656Z # Initial TLS certificates for the external API
2902025-08-23T01:07:55.656Z #
2912025-08-23T01:07:55.656Z@@ -69,8 +69,8 @@
2922025-08-23T01:07:55.656Z #
2932025-08-23T01:07:55.656Z # For more on this and what to put here, see docs/how-to-run.adoc.
2942025-08-23T01:07:55.656Z [[internal_services_ip_pool_ranges]]
2952025-08-23T01:07:55.656Z-first = "192.168.1.20"
2962025-08-23T01:07:55.656Z-last = "192.168.1.29"
2972025-08-23T01:07:55.656Z+first = "10.151.1.164"
2982025-08-23T01:07:55.656Z+last = "10.151.1.173"
2992025-08-23T01:07:55.656Z
3002025-08-23T01:07:55.656Z # TODO - this configuration is subject to change going forward. Ultimately these
3012025-08-23T01:07:55.656Z # parameters should be provided to the control plane via wicket, but we need to
3022025-08-23T01:07:55.656Z@@ -91,8 +91,8 @@
3032025-08-23T01:07:55.656Z # A range of IP addresses used by Boundary Services on the external network. In
3042025-08-23T01:07:55.656Z # a real system, these would be addresses of the uplink ports on the Sidecar.
3052025-08-23T01:07:55.656Z # With softnpu, only one address is used.
3062025-08-23T01:07:55.657Z-infra_ip_first = "192.168.1.30"
3072025-08-23T01:07:55.657Z-infra_ip_last = "192.168.1.30"
3082025-08-23T01:07:55.657Z+infra_ip_first = "10.151.1.174"
3092025-08-23T01:07:55.657Z+infra_ip_last = "10.151.1.174"
3102025-08-23T01:07:55.657Z
3112025-08-23T01:07:55.657Z # Configurations for BGP routers to run on the scrimlets.
3122025-08-23T01:07:55.657Z bgp = []
3132025-08-23T01:07:55.657Z@@ -100,9 +100,9 @@
3142025-08-23T01:07:55.657Z # You can configure multiple uplinks by repeating the following stanza
3152025-08-23T01:07:55.657Z [[rack_network_config.ports]]
3162025-08-23T01:07:55.657Z # Routes associated with this port.
3172025-08-23T01:07:55.682Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
3182025-08-23T01:07:55.682Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
3192025-08-23T01:07:55.682Z # Addresses associated with this port.
3202025-08-23T01:07:55.682Z-addresses = [{address = "192.168.1.30/24"}]
3212025-08-23T01:07:55.682Z+addresses = [{address = "10.151.1.174/24"} ]
3222025-08-23T01:07:55.682Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
3232025-08-23T01:07:55.682Z port = "qsfp0"
3242025-08-23T01:07:55.682Z # The speed of this port.
3252025-08-23T01:07:55.682Z+ true
3262025-08-23T01:07:55.682Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml pkg/config.toml
3272025-08-23T01:07:55.682Za pkg/config-rss.toml 8K
3282025-08-23T01:07:55.682Za pkg/config.toml 5K
3292025-08-23T01:07:55.682Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~ pkg/config.toml pkg/config.toml~
3302025-08-23T01:07:55.682Z+ TLS_NAME=recovery.sys.oxide.test
3312025-08-23T01:07:55.682Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
3322025-08-23T01:07:56.031Z.....+..+...+...+.+.....+.+........+....+............+...+..+...+...+...+....+.....+.+.....+......+...+.......+.....+.+.........+.....+...+...+....+...+...........+....+...+..+.+..............+..........+...+..+...+.........+....+......+......+...+..+...+....+........+.......+..+.........+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.....+....+......+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...........+..........+.........+..+.........+.............+..................+..+...+.........+...+.+.....+.+........................+.....+.......+..+..........+..+..........+.................................+.....+...+...+.+..................+..+.+..+......+.........+.......+...+...........+....+........+.........+.+.................+.+.....+............+...+.......+...+...........+..........+.....+......+.........+.+........+.........................+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
3332025-08-23T01:07:57.126Z..+.........+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.....+....+..+....+..............+.......+........+.+..+...+.+......+.........+.........+.....+....+........+....+......+.....+.......+.....+.......+...............+........+...+...+......+.+...+......+.....+.+..+.+..............+......+..........+...........+.+.....+.........+...+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+.....+...+....+...+..+....+.................+...+.+.........+....................+......+......+.........+.+............+......+......+...+...........+.+......+.....+....+.....+...+...............+...+...+............+.......+........+.......+......+...........+......+.......+...+...+.....+....+......+..............+......+................+.....+.+...+..+............+.+..+.........+....+..+.......+...........+.........+.............+.....+...+....+..+.+.........+.........+..............+............+....+.....+.+........+.......+......+...............+..+...+....+...+...........+.+......................................+.+.....+.+........+...+........................+....+..+...+......+.+........+....+...........+.........+..........+.....................+..+...+......+.+.........+......+.....+....+...+.....+...+...+....+......+.....+......+..................+.+.....+.......+..............+...............+.+............+............+.....+....+..+...................+....................................+..+.+..........................+........................+......+...+...+....+........+...+...............+.......+..+...............+..........+........+.......+...........+...+............+...............+..................+......+...+.......+.........+..+....+.....+.............+..+.+............+..+...................+..+.......+............+......+..+.......+......+........+.......+.....+....+.....+...............+.............+......+.....+.+........+.+.........+..+....+...............+..+.......+........+.........+......+...................+...+..+...+.....................+.+...+..+.........................+.....+......+.........................+............+.....+.......+...+...+..............+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
3342025-08-23T01:07:57.150Z-----
3352025-08-23T01:07:57.150Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
3362025-08-23T01:07:57.174Za pkg/initial-tls-cert.pem 2K
3372025-08-23T01:07:57.174Za pkg/initial-tls-key.pem 4K
3382025-08-23T01:07:57.174Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
3392025-08-23T01:07:57.174Z+ rmdir pkg
3402025-08-23T01:07:57.174Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
3412025-08-23T01:07:57.174Z+ pfexec mkdir -p /usr/oxide
3422025-08-23T01:07:57.175Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.1
3432025-08-23T01:07:57.750Z+ pfexec chmod +x /usr/oxide/oxide
3442025-08-23T01:07:57.775Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
3452025-08-23T01:08:16.104Z+ OMICRON_NO_UNINSTALL=1
3462025-08-23T01:08:16.104Z+ ptime -m pfexec ./target/release/omicron-package -t test install
3472025-08-23T01:08:16.129ZLogging to: /opt/oxide/work/out/LOG
3482025-08-23T01:08:16.858Z
3492025-08-23T01:08:16.858Zreal 0.747937530
3502025-08-23T01:08:16.858Zuser 0.161048366
3512025-08-23T01:08:16.882Zsys 4.395972194
3522025-08-23T01:08:16.882Ztrap 0.000008770
3532025-08-23T01:08:16.882Ztflt 0.000020709
3542025-08-23T01:08:16.882Zdflt 0.000365700
3552025-08-23T01:08:16.882Zkflt 0.000000000
3562025-08-23T01:08:16.882Zlock 1:07.252564214
3572025-08-23T01:08:16.882Zslp 0.926338444
3582025-08-23T01:08:16.882Zlat 0.037491278
3592025-08-23T01:08:16.882Zstop 0.053672544
3602025-08-23T01:08:16.882Z+ retry=0
3612025-08-23T01:08:16.882Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
3622025-08-23T01:08:16.882Z+ [[ 0 -gt 30 ]]
3632025-08-23T01:08:16.883Z+ sleep 1
3642025-08-23T01:08:17.872Z+ retry=1
3652025-08-23T01:08:17.872Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
3662025-08-23T01:08:17.897Z+ [[ 1 -gt 30 ]]
3672025-08-23T01:08:17.897Z+ sleep 1
3682025-08-23T01:08:18.887Z+ retry=2
3692025-08-23T01:08:18.887Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
3702025-08-23T01:08:18.911Z+ [[ 2 -gt 30 ]]
3712025-08-23T01:08:18.911Z+ sleep 1
3722025-08-23T01:08:19.901Z+ retry=3
3732025-08-23T01:08:19.902Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
3742025-08-23T01:08:39.077Z+ [[ 3 -gt 30 ]]
3752025-08-23T01:08:39.077Z+ sleep 1
3762025-08-23T01:08:40.080Z+ retry=4
3772025-08-23T01:08:40.080Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
3782025-08-23T01:08:40.104Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
3792025-08-23T01:08:40.129Zlocal v6:
3802025-08-23T01:08:40.129Zfda1:c8cc:2b8c:f87c::1
3812025-08-23T01:08:40.129Zfe80::4858:23ff:fe00:af0b
3822025-08-23T01:08:40.129Zlocal v4:
3832025-08-23T01:08:40.129Z10.151.1.174
3842025-08-23T01:08:40.129Zrouter v6:
3852025-08-23T01:08:40.129Zrouter v4_idx:
3862025-08-23T01:08:40.129Zrouter v4_routes:
3872025-08-23T01:08:40.129Zresolver v4:
3882025-08-23T01:08:40.129Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
3892025-08-23T01:08:40.129Zresolver v6:
3902025-08-23T01:08:40.129Zfe80::74f7:68ff:fef2:738d -> 76:f7:68:f2:73:8d
3912025-08-23T01:08:40.129Zfe80::4858:23ff:fe00:af0b -> 4a:58:23:00:af:0b
3922025-08-23T01:08:40.129Znat v4:
3932025-08-23T01:08:40.129Znat_v6:
3942025-08-23T01:08:40.129Zport_mac:
3952025-08-23T01:08:40.129Z1: a8:40:25:79:d6:fd
3962025-08-23T01:08:40.129Z2: a8:40:25:79:d7:1d
3972025-08-23T01:08:40.129Z1000: a8:40:25:79:d6:fc
3982025-08-23T01:08:40.129Zproxy arp:
3992025-08-23T01:08:40.129Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
4002025-08-23T01:08:40.129Z+ retry=0
4012025-08-23T01:08:40.129Z+++ zoneadm list -n
4022025-08-23T01:08:40.154Z+++ grep oxz_ntp
4032025-08-23T01:08:40.154Z++ pfexec svcs -z -Hostate oxide/ntp
4042025-08-23T01:08:40.154Zsvcs: zone '-Hostate': entity not found
4052025-08-23T01:08:40.154Z++ true
4062025-08-23T01:08:40.154Z+ [[ '' != online ]]
4072025-08-23T01:08:40.154Z+ [[ 0 -gt 60 ]]
4082025-08-23T01:08:40.154Z+ sleep 1
4092025-08-23T01:08:41.144Z+ retry=1
4102025-08-23T01:08:41.169Z+++ zoneadm list -n
4112025-08-23T01:08:41.169Z+++ grep oxz_ntp
4122025-08-23T01:08:41.169Z++ pfexec svcs -z -Hostate oxide/ntp
4132025-08-23T01:08:41.169Zsvcs: zone '-Hostate': entity not found
4142025-08-23T01:08:41.169Z++ true
4152025-08-23T01:08:41.169Z+ [[ '' != online ]]
4162025-08-23T01:08:41.169Z+ [[ 1 -gt 60 ]]
4172025-08-23T01:08:41.169Z+ sleep 1
4182025-08-23T01:08:42.168Z+ retry=2
4192025-08-23T01:08:42.192Z+++ zoneadm list -n
4202025-08-23T01:08:42.192Z+++ grep oxz_ntp
4212025-08-23T01:08:42.192Z++ pfexec svcs -z -Hostate oxide/ntp
4222025-08-23T01:08:42.192Zsvcs: zone '-Hostate': entity not found
4232025-08-23T01:08:42.192Z++ true
4242025-08-23T01:08:42.192Z+ [[ '' != online ]]
4252025-08-23T01:08:42.192Z+ [[ 2 -gt 60 ]]
4262025-08-23T01:08:42.192Z+ sleep 1
4272025-08-23T01:08:43.194Z+ retry=3
4282025-08-23T01:08:43.221Z+++ zoneadm list -n
4292025-08-23T01:08:43.221Z+++ grep oxz_ntp
4302025-08-23T01:08:43.221Z++ pfexec svcs -z oxz_ntp_eeead36c-c34c-4b1c-ab35-f4d96f953220 -Hostate oxide/ntp
4312025-08-23T01:08:43.245Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
4322025-08-23T01:08:43.245Z++ true
4332025-08-23T01:08:43.246Z+ [[ '' != online ]]
4342025-08-23T01:08:43.246Z+ [[ 3 -gt 60 ]]
4352025-08-23T01:08:43.246Z+ sleep 1
4362025-08-23T01:08:44.225Z+ retry=4
4372025-08-23T01:08:44.250Z+++ zoneadm list -n
4382025-08-23T01:08:44.250Z+++ grep oxz_ntp
4392025-08-23T01:08:44.250Z++ pfexec svcs -z oxz_ntp_eeead36c-c34c-4b1c-ab35-f4d96f953220 -Hostate oxide/ntp
4402025-08-23T01:08:44.274Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
4412025-08-23T01:08:44.274Z++ true
4422025-08-23T01:08:44.274Z+ [[ '' != online ]]
4432025-08-23T01:08:44.274Z+ [[ 4 -gt 60 ]]
4442025-08-23T01:08:44.274Z+ sleep 1
4452025-08-23T01:08:45.255Z+ retry=5
4462025-08-23T01:08:45.281Z+++ zoneadm list -n
4472025-08-23T01:08:45.281Z+++ grep oxz_ntp
4482025-08-23T01:08:45.281Z++ pfexec svcs -z oxz_ntp_eeead36c-c34c-4b1c-ab35-f4d96f953220 -Hostate oxide/ntp
4492025-08-23T01:08:45.305Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
4502025-08-23T01:08:45.305Z++ true
4512025-08-23T01:08:45.305Z+ [[ '' != online ]]
4522025-08-23T01:08:45.305Z+ [[ 5 -gt 60 ]]
4532025-08-23T01:08:45.305Z+ sleep 1
4542025-08-23T01:08:46.285Z+ retry=6
4552025-08-23T01:08:46.309Z+++ zoneadm list -n
4562025-08-23T01:08:46.309Z+++ grep oxz_ntp
4572025-08-23T01:08:46.309Z++ pfexec svcs -z oxz_ntp_eeead36c-c34c-4b1c-ab35-f4d96f953220 -Hostate oxide/ntp
4582025-08-23T01:08:46.309Z+ [[ offline != online ]]
4592025-08-23T01:08:46.309Z+ [[ 6 -gt 60 ]]
4602025-08-23T01:08:46.309Z+ sleep 1
4612025-08-23T01:08:47.308Z+ retry=7
4622025-08-23T01:08:47.333Z+++ zoneadm list -n
4632025-08-23T01:08:47.333Z+++ grep oxz_ntp
4642025-08-23T01:08:47.333Z++ pfexec svcs -z oxz_ntp_eeead36c-c34c-4b1c-ab35-f4d96f953220 -Hostate oxide/ntp
4652025-08-23T01:08:47.333Z+ [[ offline != online ]]
4662025-08-23T01:08:47.333Z+ [[ 7 -gt 60 ]]
4672025-08-23T01:08:47.333Z+ sleep 1
4682025-08-23T01:08:48.331Z+ retry=8
4692025-08-23T01:08:48.355Z+++ zoneadm list -n
4702025-08-23T01:08:48.356Z+++ grep oxz_ntp
4712025-08-23T01:08:48.356Z++ pfexec svcs -z oxz_ntp_eeead36c-c34c-4b1c-ab35-f4d96f953220 -Hostate oxide/ntp
4722025-08-23T01:08:48.356Z+ [[ offline != online ]]
4732025-08-23T01:08:48.356Z+ [[ 8 -gt 60 ]]
4742025-08-23T01:08:48.356Z+ sleep 1
4752025-08-23T01:08:49.352Z+ retry=9
4762025-08-23T01:08:49.376Z+++ zoneadm list -n
4772025-08-23T01:08:49.376Z+++ grep oxz_ntp
4782025-08-23T01:08:49.376Z++ pfexec svcs -z oxz_ntp_eeead36c-c34c-4b1c-ab35-f4d96f953220 -Hostate oxide/ntp
4792025-08-23T01:08:49.376Z+ [[ offline != online ]]
4802025-08-23T01:08:49.376Z+ [[ 9 -gt 60 ]]
4812025-08-23T01:08:49.376Z+ sleep 1
4822025-08-23T01:08:50.373Z+ retry=10
4832025-08-23T01:08:50.398Z+++ zoneadm list -n
4842025-08-23T01:08:50.398Z+++ grep oxz_ntp
4852025-08-23T01:08:50.398Z++ pfexec svcs -z oxz_ntp_eeead36c-c34c-4b1c-ab35-f4d96f953220 -Hostate oxide/ntp
4862025-08-23T01:08:50.398Z+ [[ offline != online ]]
4872025-08-23T01:08:50.398Z+ [[ 10 -gt 60 ]]
4882025-08-23T01:08:50.398Z+ sleep 1
4892025-08-23T01:08:51.395Z+ retry=11
4902025-08-23T01:08:51.419Z+++ zoneadm list -n
4912025-08-23T01:08:51.419Z+++ grep oxz_ntp
4922025-08-23T01:08:51.419Z++ pfexec svcs -z oxz_ntp_eeead36c-c34c-4b1c-ab35-f4d96f953220 -Hostate oxide/ntp
4932025-08-23T01:08:51.420Z+ [[ offline != online ]]
4942025-08-23T01:08:51.420Z+ [[ 11 -gt 60 ]]
4952025-08-23T01:08:51.420Z+ sleep 1
4962025-08-23T01:08:52.418Z+ retry=12
4972025-08-23T01:08:52.442Z+++ zoneadm list -n
4982025-08-23T01:08:52.442Z+++ grep oxz_ntp
4992025-08-23T01:08:52.442Z++ pfexec svcs -z oxz_ntp_eeead36c-c34c-4b1c-ab35-f4d96f953220 -Hostate oxide/ntp
5002025-08-23T01:08:52.442Z+ [[ offline != online ]]
5012025-08-23T01:08:52.442Z+ [[ 12 -gt 60 ]]
5022025-08-23T01:08:52.442Z+ sleep 1
 
5032025-08-23T01:08:53.316Zbuskin console login: Aug 23 01:08:52 buskin ip: ip: joining multicasts failed (4) on opte0 - will use link layer broadcasts for multicast
 
5042025-08-23T01:08:53.440Z+ retry=13
5052025-08-23T01:08:53.465Z+++ zoneadm list -n
5062025-08-23T01:08:53.465Z+++ grep oxz_ntp
5072025-08-23T01:08:53.465Z++ pfexec svcs -z oxz_ntp_eeead36c-c34c-4b1c-ab35-f4d96f953220 -Hostate oxide/ntp
5082025-08-23T01:08:53.465Z+ [[ online != online ]]
5092025-08-23T01:08:53.465Z+ echo 'Waited for chrony: 13s'
5102025-08-23T01:08:53.465Z+ retry=0
5112025-08-23T01:08:53.465ZWaited for chrony: 13s
5122025-08-23T01:08:53.465Z+ zoneadm list
5132025-08-23T01:08:53.465Z+ grep nexus
5142025-08-23T01:08:53.489Z+ [[ 0 -gt 300 ]]
5152025-08-23T01:08:53.489Z+ sleep 1
5162025-08-23T01:08:54.469Z+ retry=1
5172025-08-23T01:08:54.494Z+ zoneadm list
5182025-08-23T01:08:54.494Z+ grep nexus
5192025-08-23T01:08:54.494Z+ [[ 1 -gt 300 ]]
5202025-08-23T01:08:54.494Z+ sleep 1
5212025-08-23T01:08:55.480Z+ retry=2
5222025-08-23T01:08:55.504Z+ zoneadm list
5232025-08-23T01:08:55.505Z+ grep nexus
5242025-08-23T01:08:55.505Z+ [[ 2 -gt 300 ]]
5252025-08-23T01:08:55.505Z+ sleep 1
5262025-08-23T01:08:56.491Z+ retry=3
5272025-08-23T01:08:56.515Z+ zoneadm list
5282025-08-23T01:08:56.515Z+ grep nexus
5292025-08-23T01:08:56.515Z+ [[ 3 -gt 300 ]]
5302025-08-23T01:08:56.516Z+ sleep 1
5312025-08-23T01:08:57.502Z+ retry=4
5322025-08-23T01:08:57.526Z+ zoneadm list
5332025-08-23T01:08:57.526Z+ grep nexus
5342025-08-23T01:08:57.526Z+ [[ 4 -gt 300 ]]
5352025-08-23T01:08:57.526Z+ sleep 1
5362025-08-23T01:08:58.512Z+ retry=5
5372025-08-23T01:08:59.543Z+ zoneadm list
5382025-08-23T01:08:59.543Z+ grep nexus
5392025-08-23T01:08:59.543Z+ [[ 5 -gt 300 ]]
5402025-08-23T01:08:59.543Z+ sleep 1
5412025-08-23T01:08:59.570Z+ retry=6
5422025-08-23T01:08:59.570Z+ zoneadm list
5432025-08-23T01:08:59.570Z+ grep nexus
5442025-08-23T01:08:59.570Z+ [[ 6 -gt 300 ]]
5452025-08-23T01:08:59.570Z+ sleep 1
5462025-08-23T01:09:00.575Z+ retry=7
5472025-08-23T01:09:00.601Z+ zoneadm list
5482025-08-23T01:09:00.601Z+ grep nexus
5492025-08-23T01:09:00.601Z+ [[ 7 -gt 300 ]]
5502025-08-23T01:09:00.601Z+ sleep 1
5512025-08-23T01:09:01.618Z+ retry=8
5522025-08-23T01:09:01.645Z+ zoneadm list
5532025-08-23T01:09:01.645Z+ grep nexus
5542025-08-23T01:09:01.646Z+ [[ 8 -gt 300 ]]
5552025-08-23T01:09:01.646Z+ sleep 1
5562025-08-23T01:09:02.553Z+ retry=9
5572025-08-23T01:09:02.577Z+ zoneadm list
5582025-08-23T01:09:02.577Z+ grep nexus
5592025-08-23T01:09:02.577Z+ [[ 9 -gt 300 ]]
5602025-08-23T01:09:02.577Z+ sleep 1
5612025-08-23T01:09:03.563Z+ retry=10
5622025-08-23T01:09:03.588Z+ zoneadm list
5632025-08-23T01:09:03.588Z+ grep nexus
5642025-08-23T01:09:03.588Z+ [[ 10 -gt 300 ]]
5652025-08-23T01:09:03.588Z+ sleep 1
5662025-08-23T01:09:04.574Z+ retry=11
5672025-08-23T01:09:04.598Z+ zoneadm list
5682025-08-23T01:09:04.598Z+ grep nexus
5692025-08-23T01:09:04.598Z+ [[ 11 -gt 300 ]]
5702025-08-23T01:09:04.598Z+ sleep 1
5712025-08-23T01:09:05.584Z+ retry=12
5722025-08-23T01:09:05.608Z+ zoneadm list
5732025-08-23T01:09:05.608Z+ grep nexus
5742025-08-23T01:09:05.609Z+ [[ 12 -gt 300 ]]
5752025-08-23T01:09:05.609Z+ sleep 1
5762025-08-23T01:09:06.594Z+ retry=13
5772025-08-23T01:09:06.619Z+ zoneadm list
5782025-08-23T01:09:06.619Z+ grep nexus
5792025-08-23T01:09:06.619Z+ [[ 13 -gt 300 ]]
5802025-08-23T01:09:06.619Z+ sleep 1
5812025-08-23T01:09:07.609Z+ retry=14
5822025-08-23T01:09:07.634Z+ zoneadm list
5832025-08-23T01:09:07.634Z+ grep nexus
5842025-08-23T01:09:07.634Z+ [[ 14 -gt 300 ]]
5852025-08-23T01:09:07.634Z+ sleep 1
5862025-08-23T01:09:08.621Z+ retry=15
5872025-08-23T01:09:08.646Z+ zoneadm list
5882025-08-23T01:09:08.646Z+ grep nexus
5892025-08-23T01:09:08.646Z+ [[ 15 -gt 300 ]]
5902025-08-23T01:09:08.646Z+ sleep 1
5912025-08-23T01:09:09.634Z+ retry=16
5922025-08-23T01:09:09.678Z+ zoneadm list
5932025-08-23T01:09:09.678Z+ grep nexus
5942025-08-23T01:09:09.703Z+ [[ 16 -gt 300 ]]
5952025-08-23T01:09:09.703Z+ sleep 1
5962025-08-23T01:09:10.717Z+ retry=17
5972025-08-23T01:09:10.742Z+ zoneadm list
5982025-08-23T01:09:10.742Z+ grep nexus
5992025-08-23T01:09:10.742Z+ [[ 17 -gt 300 ]]
6002025-08-23T01:09:10.742Z+ sleep 1
6012025-08-23T01:09:11.744Z+ retry=18
6022025-08-23T01:09:11.768Z+ zoneadm list
6032025-08-23T01:09:11.768Z+ grep nexus
6042025-08-23T01:09:11.768Z+ [[ 18 -gt 300 ]]
6052025-08-23T01:09:11.768Z+ sleep 1
6062025-08-23T01:09:12.767Z+ retry=19
6072025-08-23T01:09:12.793Z+ zoneadm list
6082025-08-23T01:09:12.793Z+ grep nexus
6092025-08-23T01:09:12.793Z+ [[ 19 -gt 300 ]]
6102025-08-23T01:09:12.793Z+ sleep 1
6112025-08-23T01:09:13.779Z+ retry=20
6122025-08-23T01:09:13.803Z+ zoneadm list
6132025-08-23T01:09:13.803Z+ grep nexus
6142025-08-23T01:09:13.803Z+ [[ 20 -gt 300 ]]
6152025-08-23T01:09:13.803Z+ sleep 1
6162025-08-23T01:09:14.790Z+ retry=21
6172025-08-23T01:09:14.815Z+ zoneadm list
6182025-08-23T01:09:14.815Z+ grep nexus
6192025-08-23T01:09:14.815Z+ [[ 21 -gt 300 ]]
6202025-08-23T01:09:14.815Z+ sleep 1
6212025-08-23T01:09:15.811Z+ retry=22
6222025-08-23T01:09:15.835Z+ zoneadm list
6232025-08-23T01:09:15.835Z+ grep nexus
6242025-08-23T01:09:15.835Z+ [[ 22 -gt 300 ]]
6252025-08-23T01:09:15.835Z+ sleep 1
6262025-08-23T01:09:16.823Z+ retry=23
6272025-08-23T01:09:16.847Z+ zoneadm list
6282025-08-23T01:09:16.847Z+ grep nexus
6292025-08-23T01:09:16.847Z+ [[ 23 -gt 300 ]]
6302025-08-23T01:09:16.847Z+ sleep 1
6312025-08-23T01:09:17.836Z+ retry=24
6322025-08-23T01:09:17.860Z+ zoneadm list
6332025-08-23T01:09:17.860Z+ grep nexus
6342025-08-23T01:09:17.860Z+ [[ 24 -gt 300 ]]
6352025-08-23T01:09:17.860Z+ sleep 1
6362025-08-23T01:09:18.847Z+ retry=25
6372025-08-23T01:09:18.871Z+ zoneadm list
6382025-08-23T01:09:18.871Z+ grep nexus
6392025-08-23T01:09:18.872Z+ [[ 25 -gt 300 ]]
6402025-08-23T01:09:18.872Z+ sleep 1
6412025-08-23T01:09:19.860Z+ retry=26
6422025-08-23T01:09:19.884Z+ zoneadm list
6432025-08-23T01:09:19.884Z+ grep nexus
6442025-08-23T01:09:19.884Z+ [[ 26 -gt 300 ]]
6452025-08-23T01:09:19.884Z+ sleep 1
6462025-08-23T01:09:20.872Z+ retry=27
6472025-08-23T01:09:20.896Z+ zoneadm list
6482025-08-23T01:09:20.896Z+ grep nexus
6492025-08-23T01:09:20.896Z+ [[ 27 -gt 300 ]]
6502025-08-23T01:09:20.896Z+ sleep 1
6512025-08-23T01:09:21.881Z+ retry=28
6522025-08-23T01:09:21.905Z+ zoneadm list
6532025-08-23T01:09:21.905Z+ grep nexus
6542025-08-23T01:09:21.905Z+ [[ 28 -gt 300 ]]
6552025-08-23T01:09:21.905Z+ sleep 1
6562025-08-23T01:09:22.893Z+ retry=29
6572025-08-23T01:09:22.918Z+ zoneadm list
6582025-08-23T01:09:22.918Z+ grep nexus
6592025-08-23T01:09:22.918Z+ [[ 29 -gt 300 ]]
6602025-08-23T01:09:22.918Z+ sleep 1
6612025-08-23T01:09:23.906Z+ retry=30
6622025-08-23T01:09:23.931Z+ zoneadm list
6632025-08-23T01:09:23.931Z+ grep nexus
6642025-08-23T01:09:23.931Z+ [[ 30 -gt 300 ]]
6652025-08-23T01:09:23.931Z+ sleep 1
6662025-08-23T01:09:24.921Z+ retry=31
6672025-08-23T01:09:24.946Z+ zoneadm list
6682025-08-23T01:09:24.946Z+ grep nexus
6692025-08-23T01:09:24.946Z+ [[ 31 -gt 300 ]]
6702025-08-23T01:09:24.946Z+ sleep 1
6712025-08-23T01:09:25.936Z+ retry=32
6722025-08-23T01:09:25.960Z+ zoneadm list
6732025-08-23T01:09:25.960Z+ grep nexus
6742025-08-23T01:09:25.960Z+ [[ 32 -gt 300 ]]
6752025-08-23T01:09:25.960Z+ sleep 1
6762025-08-23T01:09:26.948Z+ retry=33
6772025-08-23T01:09:26.972Z+ zoneadm list
6782025-08-23T01:09:26.972Z+ grep nexus
6792025-08-23T01:09:26.972Z+ [[ 33 -gt 300 ]]
6802025-08-23T01:09:26.972Z+ sleep 1
6812025-08-23T01:09:27.963Z+ retry=34
 
6822025-08-23T01:09:29.154ZWARNING: panicked at lib/opte/src/ddi/mblk.rs:250:14:
6832025-08-23T01:09:29.198Zsomehow failed to get an mblk...
6842025-08-23T01:09:29.232Zpanic[cpu25]/thread=fffffe85e5eabc40: panicked at lib/opte/src/ddi/mblk.rs:250:14:
6852025-08-23T01:09:29.300Zsomehow failed to get an mblk...
6862025-08-23T01:09:29.446Zfffffe00c5684d70 xde:_RNvCsdLrOTpD4Wds_7___rustc17rust_begin_unwind+16d ()
6872025-08-23T01:09:29.805Zfffffe00c5684da0 xde:__rust_no_alloc_shim_is_unstable+33fef877 ()
6882025-08-23T01:09:29.840Zfffffe00c5684e00 xde:_ZN4core9panicking13assert_failed17h9869c6814042c667E+0 ()
6892025-08-23T01:09:29.874Zfffffe00c5684f10 xde:_ZN4opte6engine6packet8EmitSpec5apply17hf50034de8a8e2697E+e90 ()
6902025-08-23T01:09:29.907Zfffffe00c56853e0 xde:xde_mc_tx+393c ()
6912025-08-23T01:09:29.940Zfffffe00c5685410 mac:mac_ring_tx+50 ()
6922025-08-23T01:09:29.974Zfffffe00c5685460 mac:mac_provider_tx+6d ()
6932025-08-23T01:09:30.007Zfffffe00c5685520 mac:mac_tx+28b ()
6942025-08-23T01:09:30.041Zfffffe00c5685570 dld:str_mdata_fastpath_put+8e ()
6952025-08-23T01:09:30.074Zfffffe00c5685680 ip:ip_xmit+9ce ()
6962025-08-23T01:09:30.108Zfffffe00c56858d0 ip:ire_send_wire_v4+3c2 ()
6972025-08-23T01:09:30.163Zfffffe00c5685950 ip:conn_ip_output+18f ()
6982025-08-23T01:09:30.197Zfffffe00c5685a30 ip:udp_output_connected+131 ()
6992025-08-23T01:09:30.230Zfffffe00c5685ad0 ip:udp_send+1bf ()
7002025-08-23T01:09:30.263Zfffffe00c5685b60 sockfs:so_sendmsg+201 ()
7012025-08-23T01:09:30.297Zfffffe00c5685bc0 sockfs:socket_sendmsg+65 ()
7022025-08-23T01:09:30.331Zfffffe00c5685c60 sockfs:sendit+186 ()
7032025-08-23T01:09:30.364Zfffffe00c5685f00 sockfs:sendmsg+2c6 ()
7042025-08-23T01:09:30.398Zfffffe00c5685f10 unix:brand_sys_syscall+1fe ()
7052025-08-23T01:09:30.431Zskipping system dump - no dump device configured
7062025-08-23T01:09:30.464Zrebooting...
7072025-08-23T01:10:17.446Z^[[2J^[[1;1H^[[8;25;80t^[[2J^[[1;1H^[[1;1H^[[0;37;40m^[[2J^[[1;1H^[[25;
7082025-08-23T01:10:18.338Z8H98^[[1;1H^[[25;78H9D^[[1;1H^[[25;78H9A^[[1;1H^[[25;78H9C^[[1;1H^[[25;78
7092025-08-23T01:10:18.376Z9D^[[1;1H^[[25;78H9A^[[1;1H^[[25;78H9C^[[1;1H^[[25;78HB4^[[1;1H^[[25;78HB
7102025-08-23T01:10:20.958Z^[[1;1H^[[25;78HB4^[[1;1H^[[1;1H^[[1;1HPress <DEL> to enter setup, <F1
7112025-08-23T01:10:22.419Z> Display Boot Menu, <F12> Force Network Boot ^[[25;78H92^[[2;1H^[
7122025-08-23T01:10:22.466Z25;78HA0^[[2;1H^[[25;78HA0^[[2;1H^[[25;78H99^[[2;1H^[[25;78H92^[[2;1H^[[2
7132025-08-23T01:10:22.510Z;78H92^[[2;1H^[[25;78H72^[[2;1H^[[25;78HA0^[[2;1H^[[25;78HA2^[[2;1H^[[25;
7142025-08-23T01:10:22.544Z8HA2^[[2;1H^[[25;78HA2^[[2;1H^[[25;78HA2^[[2;1H^[[25;78HA2^[[2;1H^[[25;78
7152025-08-23T01:10:23.153ZA2^[[2;1H^[[25;78HA2^[[2;1H^[[25;78HA2^[[2;1H^[[25;78H92^[[2;1H^[[25;
7162025-08-23T01:10:23.189Z8HB2^[[2;1H^[[2J^[[1;1H^[[01;00H^[[0m^[[2;37;40m
7172025-08-23T01:10:23.257Zt GE v1.5.53 ^[[02;00HP
7182025-08-23T01:10:23.300ZE 2.1 Build 092 (WfM 2.0) ^[
7192025-08-23T01:10:23.334Z03;00HPress Ctrl+S to enter the Setup Menu. ^[
7202025-08-23T01:10:23.652Z04;00H ^[
7212025-08-23T01:10:23.693Z05;00H ^[
7222025-08-23T01:10:23.738Z06;00H ^[
7232025-08-23T01:10:23.783Z07;00H ^[
7242025-08-23T01:10:23.817Z08;00H ^[
7252025-08-23T01:10:23.851Z09;00H ^[
7262025-08-23T01:10:23.885Z10;00H ^[
7272025-08-23T01:10:23.918Z11;00H ^[
7282025-08-23T01:10:23.952Z12;00H ^[
7292025-08-23T01:10:23.986Z13;00H ^[
7302025-08-23T01:10:24.020Z14;00H ^[
7312025-08-23T01:10:24.064Z15;00H ^[
7322025-08-23T01:10:24.131Z16;00H ^[
7332025-08-23T01:10:24.175Z17;00H ^[
7342025-08-23T01:10:24.220Z18;00H ^[
7352025-08-23T01:10:24.254Z19;00H ^[
7362025-08-23T01:10:24.288Z20;00H ^[
7372025-08-23T01:10:24.330Z21;00H ^[
7382025-08-23T01:10:24.367Z22;00H ^[
7392025-08-23T01:10:24.399Z23;00H ^[
7402025-08-23T01:10:24.443Z24;00H ^[
7412025-08-23T01:10:24.478Z24;00H^[[03;39H^[[01;00HInitializing Intel(R) Boot Agent GE v1.5.53
7422025-08-23T01:10:24.512Z^[[03;00HPress Ctrl+S to enter
7432025-08-23T01:10:24.546Zthe Setup Menu.. ^[[03;39
7442025-08-23T01:10:25.736Z^[[01;00HInitializing Intel(R) Boot Agent GE v1.5.53
7452025-08-23T01:10:25.770Z^[[02;00HPXE 2.1 Build 092 (WfM 2.0)
7462025-08-23T01:10:25.804Z^[[03;00H
7472025-08-23T01:10:25.837Z^[
7482025-08-23T01:10:25.872Z04;00H ^[
7492025-08-23T01:10:25.905Z05;00HInitializing Intel(R) Boot Agent GE v1.5.53 ^[
7502025-08-23T01:10:26.232Z06;00HPXE 2.1 Build 092 (WfM 2.0) ^[
7512025-08-23T01:10:26.275Z07;00HPress Ctrl+S to enter the Setup Menu. ^[
7522025-08-23T01:10:26.309Z08;00H ^[
7532025-08-23T01:10:26.343Z09;00H ^[
7542025-08-23T01:10:26.377Z10;00H ^[
7552025-08-23T01:10:26.410Z11;00H ^[
7562025-08-23T01:10:26.444Z12;00H ^[
7572025-08-23T01:10:26.477Z13;00H ^[
7582025-08-23T01:10:26.966Z14;00H ^[
7592025-08-23T01:10:27.033Z15;00H ^[
7602025-08-23T01:10:27.110Z16;00H ^[
7612025-08-23T01:10:27.426Z17;00H ^[
7622025-08-23T01:10:27.458Z18;00H ^[
7632025-08-23T01:10:27.493Z19;00H ^[
7642025-08-23T01:10:27.527Z20;00H ^[
7652025-08-23T01:10:27.563Z21;00H ^[
7662025-08-23T01:10:27.600Z22;00H ^[
7672025-08-23T01:10:27.634Z23;00H ^[
7682025-08-23T01:10:27.667Z24;00H ^[
7692025-08-23T01:10:27.701Z24;00H^[[07;39H^[[07;00HPress Ctrl+S to enter the Setup Menu..
7702025-08-23T01:10:30.565Z^[[07;39H^[[2J^[[1;1H^[[8;25;80t^[
7712025-08-23T01:10:30.603Z2J^[[1;1H^[[1;1H^[[0;37;40mPress <DEL> to enter setup, <F10> Display
7722025-08-23T01:10:30.636ZBoot Menu, <F12> Force Network Boot B2^[
7732025-08-23T01:10:30.670Z2;1H^[[2;1H^[[2;1H
7742025-08-23T01:10:30.704Z^[[3;1H^[[3;1HBIOS Version: MZ32-AR0-
7752025-08-23T01:10:30.738Z0-M06 ^[[4;1H^[[4
7762025-08-23T01:10:31.066Z1HBIOS Date: 07/10/2021 ^[
7772025-08-23T01:10:31.119Z5;1H^[[5;1HProcessor Type: AMD EPYC 7413 24-Core Processor
7782025-08-23T01:10:31.164Z^[[6;1H^[[6;1HProcessor Speed: 2650 MHz
7792025-08-23T01:10:31.197Z^[[7;1H^[[7;1HTot
7802025-08-23T01:10:31.231Zl Memory: 131072MB (DDR4 3200) ^[
7812025-08-23T01:10:31.265Z8;1H^[[8;1HBMC Firmware Version: 12.53.07
7822025-08-23T01:10:31.310Z^[[9;1H^[[9;1HBMC IP : 172.20.4.57
7832025-08-23T01:10:31.354Z^[[10;1H
7842025-08-23T01:10:31.421Z^[
7852025-08-23T01:10:31.455Z11;1H^[[11;1H
7862025-08-23T01:10:31.500Z^[[12;1H^[[12;1HUSB Devices total: 0 KBDs
7872025-08-23T01:10:31.533Z0 MICE, 0 MASS, 3 HUBs ^[[13;1H^[[13
7882025-08-23T01:10:31.567Z1H ^[
7892025-08-23T01:10:31.601Z14;1H^[[14;1HDetected ATA/ATAPI Devices...
7902025-08-23T01:10:31.900Z^[[15;1H^[[15;1H
7912025-08-23T01:10:32.081Z^[[16;1H^[[16
7922025-08-23T01:10:32.149Z1HDetected NVME Devices... ^[
7932025-08-23T01:10:32.184Z17;1H^[[17;1HHP SSD EX950 1TB Size: 1024.2GB
7942025-08-23T01:10:32.217Z^[[18;1H^[[18;1HHP SSD EX950 1TB Size: 10
7952025-08-23T01:10:35.651Z4.2GB ^[[19;1H^[[19
7962025-08-23T01:10:36.092Z1HPreparing Bios Setup configuration.... ^[
7972025-08-23T01:10:36.173Z20;1H^[[20;1HImporting Bios settings from BMC.
7982025-08-23T01:10:36.261Z^[[21;1H^[[21;1H
7992025-08-23T01:10:38.873Z^[[2J^[[1;1H^[
8002025-08-23T01:10:56.374Z2J^[[1;1H^[[2J^[[1;1HiPXE initialising devices...ok
8012025-08-23T01:10:56.412Z^[[0m
8022025-08-23T01:10:56.446Z^[[1miPXE 1.21.1+ (g8bff5)^[[0m -- Open Source Network Boot Firmwar
8032025-08-23T01:10:56.480Z-- ^[[36mhttp://ipxe.org^[[0m
8042025-08-23T01:10:56.513ZFeatures: DNS HTTP iSCSI TFTP SRP AoE ELF MBOOT PXE bzImage Menu PXEXT
8052025-08-23T01:10:58.525Znet0: 18:c0:4d:0d:9f:b2 using i350 on 0000:c3:00.0 (Ethernet) [open]
8062025-08-23T01:10:58.559Z[Link:up, TX:0 TXE:1 RX:0 RXE:0]
8072025-08-23T01:10:58.592Z[TXE: 1 x "Network unreachable (http://ipxe.org/28086011)"]
8082025-08-23T01:11:01.504ZConfiguring (net0 18:c0:4d:0d:9f:b2)...... ok
8092025-08-23T01:11:01.548Znet0: 10.151.1.100/255.255.255.0 gw 10.151.1.1
8102025-08-23T01:11:01.582ZFilename: http://172.20.0.5:9969/ipxe/buskin
8112025-08-23T01:11:01.835Zhttp://172.20.0.5:9969/ipxe/buskin... ok
8122025-08-23T01:11:02.201Zbuskin : 557 bytes [script]
8132025-08-23T01:12:53.486Zworker failed without completing job
8142025-08-23T01:12:53.488Ztask 2 was incomplete, marked failed