01HDMDFD9BGS4X65N8Q44T8YZF: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HDMDG240PZE5YT8H3WBQH2GH

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-25T22:52:42.074Zjob dependencies complete; ready to run (waiting for 54 m 51 s)
22023-10-25T22:53:24.515Zjob assigned to worker 01HDMGMPVYRNMY241ABCVR60WF (queued for 42 s)
32023-10-25T22:53:24.538Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-25T22:53:24.582Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-10-25T22:53:24.617Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-10-25T22:53:24.650Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-25T22:53:24.803Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-10-25T22:53:24.885ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-25T22:53:24.918Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-25T22:53:24.952ZOxide Helios Version helios-2.0.22241 64-bit
112023-10-25T22:53:24.985ZConfiguring devices.
122023-10-25T22:53:25.019Z#################################################
132023-10-25T22:53:25.475Z#### ############# ##
142023-10-25T22:53:25.508Z### ### ############ ##
152023-10-25T22:53:25.541Z## ### # ## ### ## Oxide Computer Company ##
162023-10-25T22:53:25.575Z## ## ## ### # ### ##
172023-10-25T22:53:25.624Z## # ### #### #### This Station Under ##
182023-10-25T22:53:25.666Z### ### #### # ### Computer Control ##
192023-10-25T22:53:25.709Z#### #### ### ## ##
202023-10-25T22:53:25.776Z##################################################
212023-10-25T22:53:25.808ZHostname: buskin
222023-10-25T22:53:25.842ZLoading smf(7) serice descriptions:
232023-10-25T22:53:26.970Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-7505a19fa56a5ab1.gz
242023-10-25T22:53:28.396Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-7505a19fa56a5ab1.gz
252023-10-25T22:53:28.420Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-25T22:53:46.664Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-25T22:53:46.699Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a486f79ff30d68c1.gz
282023-10-25T22:54:39.336Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a486f79ff30d68c1.gz
292023-10-25T22:54:39.362Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-25T22:54:40.772Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-25T22:54:40.796Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-25T22:54:45.136Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-25T22:54:45.170Zdownloading input: /input/package/work/version.txt
342023-10-25T22:54:45.208Zdownloaded input: /input/package/work/version.txt
352023-10-25T22:54:45.241Zdownloading input: /input/package/work/package.tar.gz
362023-10-25T22:55:05.564Zdownloaded input: /input/package/work/package.tar.gz
372023-10-25T22:55:05.591Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-25T22:55:40.768Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-25T22:55:40.800Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-25T22:55:48.808Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-25T22:55:48.832Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-25T22:56:46.631Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-25T22:56:46.663Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-25T22:57:33.826Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-25T22:57:33.867Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-25T22:58:03.097Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-25T22:58:03.121Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-25T22:58:13.233Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-25T22:58:13.264Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-25T22:58:22.529Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-25T22:58:22.552Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-25T22:58:29.453Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-25T22:58:29.483Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-25T22:58:34.742Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-25T22:58:34.765Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-25T22:58:34.788Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-25T22:58:34.812Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-25T22:58:40.165Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-25T22:58:40.191Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-25T22:58:47.794Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-25T22:58:47.828Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-25T22:59:10.417Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-25T22:59:10.441Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-25T22:59:33.350Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-25T22:59:33.376Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-25T22:59:33.425Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-25T22:59:33.457Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-25T22:59:37.973Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-25T22:59:37.996Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-25T22:59:57.962Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-25T22:59:57.993Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-25T23:00:47.462Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-25T23:00:47.503Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-25T23:01:49.361Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-25T23:01:49.446Zstarting task 0: "setup"
762023-10-25T23:01:50.120Z++ uname -s
772023-10-25T23:01:50.152Z+ kern=SunOS
782023-10-25T23:01:50.202Z+ case "$kern" in
792023-10-25T23:01:50.244Z+ groupadd -g 12345 build
802023-10-25T23:01:50.300Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-25T23:01:51.503Z+ zfs create -o mountpoint=/work rpool/work
822023-10-25T23:01:51.540Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-25T23:01:51.573Z+ home_fs=zfs
842023-10-25T23:01:51.617Z+ [[ zfs == autofs ]]
852023-10-25T23:01:51.641Z+ mkdir -p /home/build
862023-10-25T23:01:51.666Z+ chown build:build /home/build /work
872023-10-25T23:01:53.526Z+ chmod 0700 /home/build /work
882023-10-25T23:01:53.559Zprocess exited: duration 4049 ms, exit code 0
 
892023-10-25T23:01:53.619Zstarting task 1: "authentication"
902023-10-25T23:01:53.690Zprocess exited: duration 46 ms, exit code 0
 
912023-10-25T23:01:53.737Zstarting task 2: "build"
922023-10-25T23:01:53.768Z+ trap _exit_trap EXIT
932023-10-25T23:01:53.795Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-25T23:01:53.974Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-25T23:01:53.998Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-25T23:01:54.022Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-25T23:01:54.045Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-25T23:01:54.069Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-25T23:01:54.099Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-25T23:01:54.126Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-25T23:01:54.151Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-25T23:01:54.186Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-25T23:01:54.210Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-25T23:01:54.238Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-25T23:01:54.261Z+ [[ -d /opt/oxide ]]
1062023-10-25T23:01:54.285Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-25T23:01:54.308Z+ pfexec mkdir /opt/oxide
1082023-10-25T23:01:54.340Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-25T23:01:54.367Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-25T23:01:54.395Z+ cd /opt/oxide-underneath
1112023-10-25T23:01:54.423Z+ pfexec tar ceEp@/f - .
1122023-10-25T23:01:54.456Z+ cd /opt/oxide
1132023-10-25T23:01:54.486Z+ pfexec tar xveEp@/f -
1142023-10-25T23:01:54.514Zx ., 0 bytes, 0 tape blocks
1152023-10-25T23:01:54.545Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-25T23:01:54.571Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-25T23:01:54.598Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-10-25T23:01:54.628Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-25T23:01:54.659Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-25T23:01:55.123Z+ pfexec mkdir /opt/oxide/work
1212023-10-25T23:01:55.147Z+ pfexec chown build:build /opt/oxide/work
1222023-10-25T23:01:55.171Z+ cd /opt/oxide/work
1232023-10-25T23:01:55.213Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-25T23:01:55.683ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-25T23:01:55.714Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1262023-10-25T23:01:55.751Zx out/omicron-sled-agent.tar, 115192320 bytes, 224985 tape blocks
1272023-10-25T23:01:55.778Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-25T23:01:55.808Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-10-25T23:01:55.837Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-10-25T23:01:55.863Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-25T23:01:55.889Zx target/release/omicron-package, 38200896 bytes, 74612 tape blocks
1322023-10-25T23:01:55.913Zx tools/create_virtual_hardware.sh, 2974 bytes, 6 tape blocks
1332023-10-25T23:01:55.946Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-25T23:01:55.969Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-25T23:01:55.993Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-25T23:01:56.017Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-25T23:01:56.041Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-25T23:01:56.065Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-25T23:01:56.089Z
1402023-10-25T23:01:56.113Zreal 1.547558989
1412023-10-25T23:01:56.138Zuser 1.393723003
1422023-10-25T23:01:56.162Zsys 0.271451206
1432023-10-25T23:01:56.186Ztrap 0.000061781
1442023-10-25T23:01:56.210Ztflt 0.000000000
1452023-10-25T23:01:56.269Zdflt 0.000000000
1462023-10-25T23:01:56.292Zkflt 0.000000000
1472023-10-25T23:01:56.316Zlock 0.000000000
1482023-10-25T23:01:56.342Zslp 1.392766219
1492023-10-25T23:01:56.372Zlat 0.035124226
1502023-10-25T23:01:56.403Zstop 0.000113653
1512023-10-25T23:01:56.432Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-25T23:01:56.710Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-25T23:01:56.738Z+ mkdir tests
1542023-10-25T23:01:56.761Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-25T23:01:56.790Z+ ptime -m gunzip
1562023-10-25T23:01:56.817Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-7505a19fa56a5ab1
1572023-10-25T23:01:56.870Z
1582023-10-25T23:01:56.893Zreal 0.140497878
1592023-10-25T23:01:56.966Zuser 0.129937429
1602023-10-25T23:01:56.994Zsys 0.010300879
1612023-10-25T23:01:57.017Ztrap 0.000013180
1622023-10-25T23:01:57.041Ztflt 0.000000000
1632023-10-25T23:01:57.072Zdflt 0.000000000
1642023-10-25T23:01:57.096Zkflt 0.000000000
1652023-10-25T23:01:57.130Zlock 0.000000000
1662023-10-25T23:01:57.159Zslp 0.000020060
1672023-10-25T23:01:57.186Zlat 0.000024250
1682023-10-25T23:01:57.216Zstop 0.000131993
1692023-10-25T23:01:57.239Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-7505a19fa56a5ab1
1702023-10-25T23:01:57.271Z+ chmod a+x tests/bootstrap-7505a19fa56a5ab1
1712023-10-25T23:01:57.295Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-25T23:01:57.321Z+ ptime -m gunzip
1732023-10-25T23:01:57.345Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-25T23:01:59.116Z
1752023-10-25T23:01:59.140Zreal 2.235814709
1762023-10-25T23:01:59.169Zuser 2.101261049
1772023-10-25T23:01:59.201Zsys 0.134137598
1782023-10-25T23:01:59.225Ztrap 0.000199254
1792023-10-25T23:01:59.248Ztflt 0.000000000
1802023-10-25T23:01:59.273Zdflt 0.000000000
1812023-10-25T23:01:59.297Zkflt 0.000000000
1822023-10-25T23:01:59.332Zlock 0.000000000
1832023-10-25T23:01:59.366Zslp 0.000023770
1842023-10-25T23:01:59.405Zlat 0.000016150
1852023-10-25T23:01:59.441Zstop 0.000113023
1862023-10-25T23:01:59.466Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-25T23:01:59.491Z+ chmod a+x tests/bootstrap
1882023-10-25T23:01:59.516Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-25T23:01:59.540Z+ ptime -m gunzip
1902023-10-25T23:01:59.564Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a486f79ff30d68c1
1912023-10-25T23:02:04.020Z
1922023-10-25T23:02:04.129Zreal 2.560433119
1932023-10-25T23:02:04.274Zuser 2.399094383
1942023-10-25T23:02:04.329Zsys 0.160924776
1952023-10-25T23:02:04.371Ztrap 0.000189364
1962023-10-25T23:02:04.432Ztflt 0.000000000
1972023-10-25T23:02:04.461Zdflt 0.000000000
1982023-10-25T23:02:04.498Zkflt 0.000000000
1992023-10-25T23:02:04.529Zlock 0.000000000
2002023-10-25T23:02:04.562Zslp 0.000020090
2012023-10-25T23:02:04.592Zlat 0.000026570
2022023-10-25T23:02:04.658Zstop 0.000116673
2032023-10-25T23:02:04.692Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a486f79ff30d68c1
2042023-10-25T23:02:04.734Z+ chmod a+x tests/end_to_end_tests-a486f79ff30d68c1
2052023-10-25T23:02:04.777Z+ bmat address ls
2062023-10-25T23:02:04.853ZNAME CIDR FIRST LAST COUNT
2072023-10-25T23:02:04.945Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-10-25T23:02:04.979Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-25T23:02:05.007Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-10-25T23:02:05.462Z+ EXTRA_IP_START=10.151.1.164
2112023-10-25T23:02:05.508Z+ EXTRA_IP_END=10.151.1.227
2122023-10-25T23:02:05.541Z+ EXTRA_IP_COUNT=64
2132023-10-25T23:02:05.567Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-10-25T23:02:05.595Z+ EXTRA_IP_BASE=10.151.1
2152023-10-25T23:02:05.626Z+ EXTRA_IP_FOCTET=164
2162023-10-25T23:02:05.652Z+ EXTRA_IP_LOCTET=227
2172023-10-25T23:02:05.678Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-10-25T23:02:05.706Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-10-25T23:02:05.738Z+ DNS_IP1=10.151.1.164
2202023-10-25T23:02:05.767Z+ DNS_IP2=10.151.1.165
2212023-10-25T23:02:05.792Z+ UPLINK_IP=10.151.1.174
2222023-10-25T23:02:05.820Z+ IPPOOL_START=10.151.1.175
2232023-10-25T23:02:05.863Z+ IPPOOL_END=10.151.1.227
2242023-10-25T23:02:05.927Z++ ipadm show-addr -po type,addr
2252023-10-25T23:02:05.952Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-10-25T23:02:05.980Z+ GATEWAY_IP=10.151.1.100
2272023-10-25T23:02:06.004Z+ [[ -n 10.151.1.100 ]]
2282023-10-25T23:02:06.029Z+ ping -s 10.151.1.100 56 1
2292023-10-25T23:02:06.065ZPING 10.151.1.100: 56 data bytes
2302023-10-25T23:02:06.090Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.012 ms
2312023-10-25T23:02:06.113Z
2322023-10-25T23:02:06.138Z----10.151.1.100 PING Statistics----
2332023-10-25T23:02:06.194Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-10-25T23:02:06.219Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2352023-10-25T23:02:06.242Z++ arp -an
2362023-10-25T23:02:06.267Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-10-25T23:02:06.302Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-10-25T23:02:06.330Z+ routeadm -e ipv4-forwarding -u
2392023-10-25T23:02:06.357Z+ PXA_START=10.151.1.164
2402023-10-25T23:02:06.385Z+ PXA_END=10.151.1.227
2412023-10-25T23:02:06.410Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-10-25T23:02:06.435Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-10-25T23:02:06.462Z+ ZPOOL_VDEV_DIR=/scratch
2442023-10-25T23:02:06.486Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-10-25T23:02:06.511Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-10-25T23:02:06.535Z++ cd ./tools
2472023-10-25T23:02:06.562Z++ pwd
2482023-10-25T23:02:06.591Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-10-25T23:02:06.620Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-10-25T23:02:06.653Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-10-25T23:02:06.678Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-10-25T23:02:07.120Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-10-25T23:02:07.196Z++ dladm show-phys -p -o LINK
2542023-10-25T23:02:07.631Z++ head -1
2552023-10-25T23:02:07.665Z+ PHYSICAL_LINK=igb0
2562023-10-25T23:02:08.127Z+ echo 'Using igb0 as physical link'
2572023-10-25T23:02:08.151ZUsing igb0 as physical link
2582023-10-25T23:02:08.177Z+ ensure_run_as_root
2592023-10-25T23:02:08.204Z++ id -u
2602023-10-25T23:02:08.231Z+ [[ 0 -ne 0 ]]
2612023-10-25T23:02:08.255Z+ ensure_zpools
2622023-10-25T23:02:08.279Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-10-25T23:02:08.391Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-10-25T23:02:08.425Z+ readarray -t ZPOOLS
2652023-10-25T23:02:08.457Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-10-25T23:02:08.483Z++ sed 's/[ ",]//g'
2672023-10-25T23:02:08.511Z+ for ZPOOL in "${ZPOOLS[@]}"
2682023-10-25T23:02:08.543Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2692023-10-25T23:02:08.569ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2702023-10-25T23:02:08.595Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2712023-10-25T23:02:08.630ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2722023-10-25T23:02:08.655Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-10-25T23:02:08.680ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2742023-10-25T23:02:08.704Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2752023-10-25T23:02:08.728ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2762023-10-25T23:02:08.753Z0+0 records in
2772023-10-25T23:02:08.777ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2782023-10-25T23:02:08.802Z0+0 records out
2792023-10-25T23:02:09.241ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2802023-10-25T23:02:09.280Z0 bytes (0 iB) transferred in 0.000007 secs (0 iB/sec)
2812023-10-25T23:02:09.311ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2822023-10-25T23:02:09.337Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2832023-10-25T23:02:09.446ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2842023-10-25T23:02:09.478Z+ set +x
2852023-10-25T23:02:09.503ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2862023-10-25T23:02:09.935Z++ zpool list -o name
2872023-10-25T23:02:09.964ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2882023-10-25T23:02:10.407Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2892023-10-25T23:02:10.507ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2902023-10-25T23:02:10.532Z+ [[ -z '' ]]
2912023-10-25T23:02:10.557ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2922023-10-25T23:02:10.582Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2932023-10-25T23:02:10.608ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2942023-10-25T23:02:10.634Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2952023-10-25T23:02:10.659ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2962023-10-25T23:02:10.684Z+ set +x
2972023-10-25T23:02:10.708ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2982023-10-25T23:02:10.735Z+ for ZPOOL in "${ZPOOLS[@]}"
2992023-10-25T23:02:10.760ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3002023-10-25T23:02:10.784Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3012023-10-25T23:02:10.808ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3022023-10-25T23:02:10.831Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-10-25T23:02:10.855ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3042023-10-25T23:02:10.886Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3052023-10-25T23:02:10.917ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3062023-10-25T23:02:10.950Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3072023-10-25T23:02:10.981ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3082023-10-25T23:02:11.011Z0+0 records in
3092023-10-25T23:02:11.045ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3102023-10-25T23:02:11.072Z0+0 records out
3112023-10-25T23:02:11.095ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3122023-10-25T23:02:11.122Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3132023-10-25T23:02:11.152ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3142023-10-25T23:02:11.177Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3152023-10-25T23:02:11.203ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3162023-10-25T23:02:11.233Z+ set +x
3172023-10-25T23:02:11.261ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3182023-10-25T23:02:11.290Z++ zpool list -o name
3192023-10-25T23:02:11.319ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3202023-10-25T23:02:11.359Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3212023-10-25T23:02:11.396ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3222023-10-25T23:02:11.426Z+ [[ -z '' ]]
3232023-10-25T23:02:11.480ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3242023-10-25T23:02:11.631Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3252023-10-25T23:02:11.660ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3262023-10-25T23:02:11.689Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3272023-10-25T23:02:11.715ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3282023-10-25T23:02:11.740Z+ set +x
3292023-10-25T23:02:11.767ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3302023-10-25T23:02:11.791Z+ for ZPOOL in "${ZPOOLS[@]}"
3312023-10-25T23:02:11.815ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3322023-10-25T23:02:11.839Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3332023-10-25T23:02:11.864ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3342023-10-25T23:02:11.889Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3352023-10-25T23:02:11.915ZSimnet net0/sc0_0 exists
3362023-10-25T23:02:11.993Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3372023-10-25T23:02:12.021ZSimnet net1/sc1_0 exists
3382023-10-25T23:02:12.048Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3392023-10-25T23:02:12.079ZVnic sc0_1 exists
3402023-10-25T23:02:12.102Z0+0 records in
3412023-10-25T23:02:12.128Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3422023-10-25T23:02:12.152Z0+0 records out
3432023-10-25T23:02:12.177Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3442023-10-25T23:02:12.201Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3452023-10-25T23:02:12.226Z+ set +x
3462023-10-25T23:02:12.250Z++ zpool list -o name
3472023-10-25T23:02:12.275Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3482023-10-25T23:02:12.302Z+ [[ -z '' ]]
3492023-10-25T23:02:12.328Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-10-25T23:02:12.358Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-10-25T23:02:12.384Z+ set +x
3522023-10-25T23:02:12.408Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-10-25T23:02:12.431Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-10-25T23:02:12.455Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-10-25T23:02:12.485Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-10-25T23:02:12.509Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-10-25T23:02:12.535Z0+0 records in
3582023-10-25T23:02:12.559Z0+0 records out
3592023-10-25T23:02:12.583Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3602023-10-25T23:02:12.611Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-10-25T23:02:12.680Z+ set +x
3622023-10-25T23:02:12.733Z++ zpool list -o name
3632023-10-25T23:02:12.783Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-10-25T23:02:12.841Z+ [[ -z '' ]]
3652023-10-25T23:02:12.871Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-10-25T23:02:12.922Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-10-25T23:02:12.952Z+ set +x
3682023-10-25T23:02:12.978Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-10-25T23:02:13.002Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-10-25T23:02:13.027Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-10-25T23:02:13.061Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-10-25T23:02:13.085Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-10-25T23:02:13.109Z0+0 records in
3742023-10-25T23:02:13.134Z0+0 records out
3752023-10-25T23:02:13.158Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3762023-10-25T23:02:13.182Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-10-25T23:02:13.206Z+ set +x
3782023-10-25T23:02:13.231Z++ zpool list -o name
3792023-10-25T23:02:13.254Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-10-25T23:02:13.279Z+ [[ -z '' ]]
3812023-10-25T23:02:13.314Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-10-25T23:02:13.338Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-10-25T23:02:13.363Z+ set +x
3842023-10-25T23:02:13.386Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-10-25T23:02:13.410Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862023-10-25T23:02:13.434Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872023-10-25T23:02:13.457Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882023-10-25T23:02:13.481Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892023-10-25T23:02:13.505Z0+0 records in
3902023-10-25T23:02:13.531Z0+0 records out
3912023-10-25T23:02:13.556Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-10-25T23:02:13.583Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932023-10-25T23:02:13.608Z+ set +x
3942023-10-25T23:02:13.632Z++ zpool list -o name
3952023-10-25T23:02:13.657Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3962023-10-25T23:02:13.744Z+ [[ -z '' ]]
3972023-10-25T23:02:14.184Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3982023-10-25T23:02:14.647Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3992023-10-25T23:02:14.674Z+ set +x
4002023-10-25T23:02:14.701Z+ for ZPOOL in "${ZPOOLS[@]}"
4012023-10-25T23:02:14.725Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4022023-10-25T23:02:14.802Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4032023-10-25T23:02:14.825Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4042023-10-25T23:02:14.850Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4052023-10-25T23:02:14.874Z0+0 records in
4062023-10-25T23:02:14.898Z0+0 records out
4072023-10-25T23:02:14.922Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4082023-10-25T23:02:14.946Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4092023-10-25T23:02:14.969Z+ set +x
4102023-10-25T23:02:14.993Z++ zpool list -o name
4112023-10-25T23:02:15.019Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4122023-10-25T23:02:15.043Z+ [[ -z '' ]]
4132023-10-25T23:02:15.068Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4142023-10-25T23:02:15.093Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4152023-10-25T23:02:15.550Z+ set +x
4162023-10-25T23:02:16.437Z+ for ZPOOL in "${ZPOOLS[@]}"
4172023-10-25T23:02:16.462Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4182023-10-25T23:02:16.486Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192023-10-25T23:02:16.510Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4202023-10-25T23:02:16.535Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4212023-10-25T23:02:16.973Z0+0 records in
4222023-10-25T23:02:17.001Z0+0 records out
4232023-10-25T23:02:17.033Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4242023-10-25T23:02:17.226Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4252023-10-25T23:02:17.252Z+ set +x
4262023-10-25T23:02:17.280Z++ zpool list -o name
4272023-10-25T23:02:17.304Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4282023-10-25T23:02:17.327Z+ [[ -z '' ]]
4292023-10-25T23:02:17.351Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4302023-10-25T23:02:17.375Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4312023-10-25T23:02:17.399Z+ set +x
4322023-10-25T23:02:17.431Z+ for ZPOOL in "${ZPOOLS[@]}"
4332023-10-25T23:02:17.465Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4342023-10-25T23:02:17.489Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4352023-10-25T23:02:17.515Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4362023-10-25T23:02:17.541Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4372023-10-25T23:02:17.567Z0+0 records in
4382023-10-25T23:02:17.592Z0+0 records out
4392023-10-25T23:02:17.617Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4402023-10-25T23:02:17.659Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4412023-10-25T23:02:17.683Z+ set +x
4422023-10-25T23:02:17.707Z++ zpool list -o name
4432023-10-25T23:02:18.073Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4442023-10-25T23:02:18.111Z+ [[ -z '' ]]
4452023-10-25T23:02:18.140Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4462023-10-25T23:02:18.173Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4472023-10-25T23:02:18.197Z+ set +x
4482023-10-25T23:02:18.224Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4492023-10-25T23:02:18.251Z+ readarray -t ZPOOLS
4502023-10-25T23:02:18.280Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4512023-10-25T23:02:18.304Z++ sed 's/[ ",]//g'
4522023-10-25T23:02:18.327Z+ for ZPOOL in "${ZPOOLS[@]}"
4532023-10-25T23:02:18.352Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4542023-10-25T23:02:18.377Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4552023-10-25T23:02:18.405Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4562023-10-25T23:02:18.438Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4572023-10-25T23:02:18.474Z0+0 records in
4582023-10-25T23:02:18.511Z0+0 records out
4592023-10-25T23:02:18.551Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4602023-10-25T23:02:19.010Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4612023-10-25T23:02:19.039Z+ set +x
4622023-10-25T23:02:19.075Z++ zpool list -o name
4632023-10-25T23:02:19.178Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4642023-10-25T23:02:19.202Z+ [[ -z '' ]]
4652023-10-25T23:02:19.230Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-10-25T23:02:19.260Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4672023-10-25T23:02:19.311Z+ set +x
4682023-10-25T23:02:19.335Z+ for ZPOOL in "${ZPOOLS[@]}"
4692023-10-25T23:02:19.373Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4702023-10-25T23:02:19.402Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4712023-10-25T23:02:19.426Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4722023-10-25T23:02:19.450Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4732023-10-25T23:02:19.482Z0+0 records in
4742023-10-25T23:02:19.509Z0+0 records out
4752023-10-25T23:02:19.533Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4762023-10-25T23:02:19.559Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4772023-10-25T23:02:19.583Z+ set +x
4782023-10-25T23:02:19.607Z++ zpool list -o name
4792023-10-25T23:02:19.632Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4802023-10-25T23:02:19.660Z+ [[ -z '' ]]
4812023-10-25T23:02:19.685Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4822023-10-25T23:02:19.712Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4832023-10-25T23:02:19.736Z+ set +x
4842023-10-25T23:02:19.761Z+ ensure_simulated_links igb0
4852023-10-25T23:02:19.791Z+ local PHYSICAL_LINK=igb0
4862023-10-25T23:02:19.817Z+ INDICES=("0" "1")
4872023-10-25T23:02:19.841Z+ for I in "${INDICES[@]}"
4882023-10-25T23:02:19.865Z++ get_simnet_name_if_exists net0
4892023-10-25T23:02:19.889Z++ dladm show-simnet -p -o LINK net0
4902023-10-25T23:02:20.335Z++ echo ''
4912023-10-25T23:02:20.369Z+ [[ -z '' ]]
4922023-10-25T23:02:20.397Z+ dladm create-simnet -t net0
4932023-10-25T23:02:20.424Z+ dladm create-simnet -t sc0_0
4942023-10-25T23:02:20.450Z+ dladm modify-simnet -t -p net0 sc0_0
4952023-10-25T23:02:20.478Z+ dladm set-linkprop -p mtu=9000 sc0_0
4962023-10-25T23:02:20.525Z+ success 'Simnet net0/sc0_0 exists'
4972023-10-25T23:02:20.552Z+ set +x
4982023-10-25T23:02:20.576Z+ for I in "${INDICES[@]}"
4992023-10-25T23:02:20.600Z++ get_simnet_name_if_exists net1
5002023-10-25T23:02:20.624Zdownloading softnpu ...done
5012023-10-25T23:02:20.648Z++ dladm show-simnet -p -o LINK net1
5022023-10-25T23:02:20.672Z++ echo ''
5032023-10-25T23:02:20.696Z+ [[ -z '' ]]
5042023-10-25T23:02:20.722Z+ dladm create-simnet -t net1
5052023-10-25T23:02:20.750Z+ dladm create-simnet -t sc1_0
5062023-10-25T23:02:20.787Z+ dladm modify-simnet -t -p net1 sc1_0
5072023-10-25T23:02:20.812Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5082023-10-25T23:02:20.836Z+ dladm set-linkprop -p mtu=9000 sc1_0
5092023-10-25T23:02:20.859Z+ success 'Simnet net1/sc1_0 exists'
5102023-10-25T23:02:20.883Z+ set +x
5112023-10-25T23:02:20.908Z++ get_vnic_name_if_exists sc0_1
5122023-10-25T23:02:20.932Z++ dladm show-vnic -p -o LINK sc0_1
5132023-10-25T23:02:20.957Z++ echo ''
5142023-10-25T23:02:21.393Z+ [[ -z '' ]]
5152023-10-25T23:02:21.427Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162023-10-25T23:02:21.528Z+ [[ -v PROMISC_FILT_OFF ]]
5172023-10-25T23:02:21.559Z+ success 'Vnic sc0_1 exists'
5182023-10-25T23:02:21.590Z+ set +x
5192023-10-25T23:02:21.615Z+ warn_if_no_proxy_arp
5202023-10-25T23:02:21.646Z+ [[ -v PXA_START ]]
5212023-10-25T23:02:22.519Z+ [[ -v PXA_END ]]
5222023-10-25T23:02:22.562Z+ ensure_softnpu_zone
5232023-10-25T23:02:22.598Z+ zoneadm list
5242023-10-25T23:02:22.625Z+ grep -q sidecar_softnpu
5252023-10-25T23:02:22.658Z+ [[ -f out/npuzone/npuzone ]]
5262023-10-25T23:02:22.798Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch main
5272023-10-25T23:02:24.204Zdownloading asic_program.so ...done
5282023-10-25T23:02:24.679Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5292023-10-25T23:02:32.606Zdownloading scadm ...done
5302023-10-25T23:02:32.661Zcreating port tfportrear0_0 for sc0_0
5312023-10-25T23:02:32.705Zcreating port tfportqsfp0_0 for sc0_1
5322023-10-25T23:02:32.735Zsoftnpu zone setup
5332023-10-25T23:02:32.761Zconfigure zone
5342023-10-25T23:02:32.788Zinstall zone
5352023-10-25T23:02:33.648Zboot zone
5362023-10-25T23:02:36.761Z[sidecar_softnpu] svcs milestone/network
5372023-10-25T23:02:36.788ZSTATE STIME FMRI
5382023-10-25T23:02:36.813Zonline 23:02:34 svc:/milestone/network:default
5392023-10-25T23:02:36.837Z[sidecar_softnpu] /softnpu/npu start
5402023-10-25T23:02:36.861Z
5412023-10-25T23:02:36.885Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5422023-10-25T23:02:36.936Z+ GATEWAY_IP=10.151.1.100
5432023-10-25T23:02:36.960Z+ echo 'Using 10.151.1.100 as gateway ip'
5442023-10-25T23:02:37.021ZUsing 10.151.1.100 as gateway ip
5452023-10-25T23:02:37.061Z+ [[ ! -v GATEWAY_MAC ]]
5462023-10-25T23:02:37.502Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5472023-10-25T23:02:37.530Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5482023-10-25T23:02:37.557Z+++ echo 18:c0:4d:0d:9f:b2
5492023-10-25T23:02:37.582Z+ COUNT=1
5502023-10-25T23:02:37.612Z+ [[ 1 -ne 1 ]]
5512023-10-25T23:02:37.639Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5522023-10-25T23:02:37.670ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5532023-10-25T23:02:37.701Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5542023-10-25T23:02:37.726Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5552023-10-25T23:02:37.751Z+ PXA_MAC=a8:e1:de:01:70:1d
5562023-10-25T23:02:37.776Z+ [[ -v PXA_START ]]
5572023-10-25T23:02:37.804Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5582023-10-25T23:02:37.829Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5592023-10-25T23:02:37.854Z+ z_scadm dump-state
5602023-10-25T23:02:37.886Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5612023-10-25T23:02:37.919Zlocal v6:
5622023-10-25T23:02:37.945Zlocal v4:
5632023-10-25T23:02:37.969Zrouter v6:
5642023-10-25T23:02:37.998Zrouter v4_idx:
5652023-10-25T23:02:38.024Zrouter v4_routes:
5662023-10-25T23:02:38.049Zresolver v4:
5672023-10-25T23:02:38.101Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5682023-10-25T23:02:38.148Zresolver v6:
5692023-10-25T23:02:38.173Znat v4:
5702023-10-25T23:02:38.198Znat_v6:
5712023-10-25T23:02:38.223Zport_mac:
5722023-10-25T23:02:38.248Zproxy arp:
5732023-10-25T23:02:38.273Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5742023-10-25T23:02:38.297Z+ success 'softnpu zone exists'
5752023-10-25T23:02:38.322Z+ set +x
5762023-10-25T23:02:38.346Zsoftnpu zone exists
5772023-10-25T23:02:38.377Z
5782023-10-25T23:02:38.434Zreal 32.244774483
5792023-10-25T23:02:38.463Zuser 0.712617253
5802023-10-25T23:02:38.495Zsys 1.622791907
5812023-10-25T23:02:38.529Ztrap 0.000087117
5822023-10-25T23:02:38.562Ztflt 0.000126994
5832023-10-25T23:02:38.592Zdflt 0.001181377
5842023-10-25T23:02:38.621Zkflt 0.000000000
5852023-10-25T23:02:38.652Zlock 25:17.571431455
5862023-10-25T23:02:38.677Zslp 1:35.850696571
5872023-10-25T23:02:38.702Zlat 0.069195505
5882023-10-25T23:02:38.726Zstop 0.349148089
5892023-10-25T23:02:38.750Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5902023-10-25T23:02:38.780Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5912023-10-25T23:02:39.248Z+ SILO_NAME=recovery
5922023-10-25T23:02:39.283Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-10-25T23:02:39.308Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5942023-10-25T23:02:39.333Z+ sed -i~ '
5952023-10-25T23:02:39.358Z /^external_dns_ips/c\
5962023-10-25T23:02:39.382Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5972023-10-25T23:02:39.811Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5982023-10-25T23:02:39.835Z /^first/c\
5992023-10-25T23:02:39.861Zfirst = "10.151.1.164"
6002023-10-25T23:02:39.899Z /^last/c\
6012023-10-25T23:02:39.931Zlast = "10.151.1.173"
6022023-10-25T23:02:39.973Z }
6032023-10-25T23:02:40.012Z /^\[rack_network_config/,/^$/ {
6042023-10-25T23:02:40.040Z /^infra_ip_first/c\
6052023-10-25T23:02:40.072Zinfra_ip_first = "10.151.1.174"
6062023-10-25T23:02:40.097Z /^infra_ip_last/c\
6072023-10-25T23:02:40.122Zinfra_ip_last = "10.151.1.174"
6082023-10-25T23:02:40.150Z }
6092023-10-25T23:02:40.183Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6102023-10-25T23:02:40.221Z /^\[\[rack_network_config.ports/,/^$/ {
6112023-10-25T23:02:40.373Z+++ pkg/config-rss.toml Wed Oct 25 23:02:36 2023
6122023-10-25T23:02:40.401Z /^routes/c\
6132023-10-25T23:02:40.431Z@@ -41,7 +41,7 @@
6142023-10-25T23:02:40.458Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6152023-10-25T23:02:40.483Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6162023-10-25T23:02:40.511Z /^addresses/c\
6172023-10-25T23:02:40.537Z # must be contained in one of the "internal services" IP Pool ranges listed
6182023-10-25T23:02:40.561Zaddresses = \["10.151.1.174/32"\]
6192023-10-25T23:02:40.586Z # below.
6202023-10-25T23:02:40.610Z }
6212023-10-25T23:02:40.636Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6222023-10-25T23:02:40.670Z' pkg/config-rss.toml
6232023-10-25T23:02:40.695Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6242023-10-25T23:02:40.723Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6252023-10-25T23:02:40.747Z
6262023-10-25T23:02:40.772Z+ true
6272023-10-25T23:02:40.796Z # Initial TLS certificates for the external API
6282023-10-25T23:02:40.820Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6292023-10-25T23:02:40.845Z #
6302023-10-25T23:02:40.872Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6312023-10-25T23:02:40.897Z@@ -77,8 +77,8 @@
6322023-10-25T23:02:40.950Z+ TLS_NAME=recovery.sys.oxide.test
6332023-10-25T23:02:40.975Z #
6342023-10-25T23:02:41.007Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6352023-10-25T23:02:41.033Z # For more on this and what to put here, see docs/how-to-run.adoc.
6362023-10-25T23:02:41.057Z.+.....+......+..................+.+.........+........+...+.+.....+.+.....+..........+..+...+.......+..+...+.+.....+.+...+...+..+............+....+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*................+...............+.....+...+...............+.+..+...+......+...+......+.......+...+..+.......+.....+...+.......+..+...+.......+...+.....+...............+.......+.....+.+..+.+.....+....+..+.........+.+...+..+.+...+.....+.......+.................+.............+............+..+.+..............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+.+...+......+.....+...+..........+......+.....+....+.....+.......+...........+...+.+......+........+..........+...........+......+.......+..+.+..................+...........+.+..+...+............................+............+..+......+.........+......+....+...+........+.+...+.................+......+............+..........+.....+................+.....+.......+..+.+......+...........+...+....+......+..+..........+.........+........+................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6372023-10-25T23:02:41.082Z [[internal_services_ip_pool_ranges]]
6382023-10-25T23:02:41.122Z..+..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+....+.....+.+...+.........+...+.........+.....+....+......+..................+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+...+....+...............+...+..+......+.+......+..+............+.+........+.+.........+...........+.+.........+...+.....+.+..............+.+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6392023-10-25T23:02:41.146Z-first = "192.168.1.20"
6402023-10-25T23:02:41.171Z-----
6412023-10-25T23:02:41.198Z-last = "192.168.1.29"
6422023-10-25T23:02:41.669Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6432023-10-25T23:02:41.702Z+first = "10.151.1.164"
6442023-10-25T23:02:41.730Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6452023-10-25T23:02:41.765Z+last = "10.151.1.173"
6462023-10-25T23:02:41.795Z+ rmdir pkg
6472023-10-25T23:02:41.824Z
6482023-10-25T23:02:41.851Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6492023-10-25T23:02:41.877Z # TODO - this configuration is subject to change going forward. Ultimately these
6502023-10-25T23:02:41.905Z+ pfexec mkdir -p /usr/oxide
6512023-10-25T23:02:41.943Z # parameters should be provided to the control plane via wicket, but we need to
6522023-10-25T23:02:41.969Z+ pfexec rm -f /usr/oxide/tcpproxy
6532023-10-25T23:02:42.084Z@@ -92,8 +92,8 @@
6542023-10-25T23:02:42.109Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6552023-10-25T23:02:42.137Z # A range of IP addresses used by Boundary Services on the external network. In
6562023-10-25T23:02:42.163Z+ pfexec chmod +x /usr/oxide/tcpproxy
6572023-10-25T23:02:42.188Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6582023-10-25T23:02:42.215Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6592023-10-25T23:02:42.240Z # With softnpu, only one address is used.
6602023-10-25T23:02:42.264Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6612023-10-25T23:02:42.288Z-infra_ip_first = "192.168.1.30"
6622023-10-25T23:02:42.312Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6632023-10-25T23:02:42.336Z-infra_ip_last = "192.168.1.30"
6642023-10-25T23:02:42.366Z+ OMICRON_NO_UNINSTALL=1
6652023-10-25T23:02:42.394Z+infra_ip_first = "10.151.1.174"
6662023-10-25T23:02:42.438Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6672023-10-25T23:02:42.509Z+infra_ip_last = "10.151.1.174"
6682023-10-25T23:02:42.534ZOct 25 23:02:37.421 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6692023-10-25T23:02:42.559Z
6702023-10-25T23:02:42.583ZOct 25 23:02:37.423 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6712023-10-25T23:02:42.612Z # Configurations for BGP routers to run on the scrimlets.
6722023-10-25T23:02:42.656ZOct 25 23:02:37.423 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6732023-10-25T23:02:42.683Z bgp = []
6742023-10-25T23:02:42.712ZOct 25 23:02:37.423 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6752023-10-25T23:02:42.745Z@@ -101,9 +101,9 @@
6762023-10-25T23:02:42.774ZOct 25 23:02:37.423 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6772023-10-25T23:02:42.799Z # You can configure multiple uplinks by repeating the following stanza
6782023-10-25T23:02:42.824ZOct 25 23:02:37.423 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6792023-10-25T23:02:42.848Z [[rack_network_config.ports]]
6802023-10-25T23:02:42.876ZOct 25 23:02:37.423 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6812023-10-25T23:02:42.900Z # Routes associated with this port.
6822023-10-25T23:02:42.925ZOct 25 23:02:37.423 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6832023-10-25T23:02:42.950Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6842023-10-25T23:02:42.976ZOct 25 23:02:37.423 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6852023-10-25T23:02:43.003Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6862023-10-25T23:02:43.028ZOct 25 23:02:37.423 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6872023-10-25T23:02:43.052Z # Addresses associated with this port.
6882023-10-25T23:02:43.076ZOct 25 23:02:37.423 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6892023-10-25T23:02:43.100Z-addresses = ["192.168.1.30/32"]
6902023-10-25T23:02:43.124ZOct 25 23:02:37.423 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6912023-10-25T23:02:43.149Z+addresses = ["10.151.1.174/32"]
6922023-10-25T23:02:43.183ZOct 25 23:02:37.423 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6932023-10-25T23:02:43.213Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6942023-10-25T23:02:43.237ZOct 25 23:02:37.423 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6952023-10-25T23:02:43.261Z port = "qsfp0"
6962023-10-25T23:02:43.286ZOct 25 23:02:37.423 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6972023-10-25T23:02:43.310Z # The speed of this port.
6982023-10-25T23:02:43.334ZOct 25 23:02:37.424 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6992023-10-25T23:02:43.359Za pkg/config-rss.toml 7K
7002023-10-25T23:02:43.384ZOct 25 23:02:37.424 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7012023-10-25T23:02:43.408Za pkg/initial-tls-cert.pem 2K
7022023-10-25T23:02:43.435ZOct 25 23:02:37.653 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7032023-10-25T23:02:43.471Za pkg/initial-tls-key.pem 4K
7042023-10-25T23:02:43.624ZOct 25 23:02:37.688 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7052023-10-25T23:02:43.649ZOct 25 23:02:37.750 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7062023-10-25T23:02:43.684Z
7072023-10-25T23:02:43.722Zreal 0.469635902
7082023-10-25T23:02:43.760Zuser 0.095787963
7092023-10-25T23:02:43.784Zsys 1.863837821
7102023-10-25T23:02:43.808Ztrap 0.000011298
7112023-10-25T23:02:43.833Ztflt 0.000011609
7122023-10-25T23:02:43.857Zdflt 0.000704993
7132023-10-25T23:02:43.885Zkflt 0.000005789
7142023-10-25T23:02:43.909Zlock 42.114704436
7152023-10-25T23:02:43.934Zslp 0.671871759
7162023-10-25T23:02:44.401Zlat 0.002789772
7172023-10-25T23:02:44.426Zstop 0.220282327
7182023-10-25T23:02:44.452Z+ retry=0
7192023-10-25T23:02:44.477Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7202023-10-25T23:02:44.503Z+ [[ 0 -gt 30 ]]
7212023-10-25T23:02:44.529Z+ sleep 1
7222023-10-25T23:02:44.554Z+ retry=1
7232023-10-25T23:02:44.578Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7242023-10-25T23:02:44.603Z+ [[ 1 -gt 30 ]]
7252023-10-25T23:02:44.660Z+ sleep 1
7262023-10-25T23:02:44.695Z+ retry=2
7272023-10-25T23:02:44.725Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7282023-10-25T23:03:05.740Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7292023-10-25T23:03:05.771Zlocal v6:
7302023-10-25T23:03:05.796Zfe80::280c:19ff:fe9b:4475
7312023-10-25T23:03:05.820Zfd00:99::1
7322023-10-25T23:03:05.844Zlocal v4:
7332023-10-25T23:03:05.879Z10.151.1.174
7342023-10-25T23:03:05.904Zrouter v6:
7352023-10-25T23:03:05.928Zfd00:1122:3344:101::/64 -> fe80::6cff:bdff:fec0:aa9d (1)
7362023-10-25T23:03:05.966Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::6cff:bdff:fec0:aa9d (1)
7372023-10-25T23:03:05.991Zrouter v4_idx:
7382023-10-25T23:03:06.015Z0.0.0.0/0 -> 0
7392023-10-25T23:03:06.041Zrouter v4_routes:
7402023-10-25T23:03:06.066Z0 -> 10.151.1.100 (2)
7412023-10-25T23:03:06.091Zresolver v4:
7422023-10-25T23:03:06.116Z10.151.1.174 -> f2:74:a1:15:e4:47
7432023-10-25T23:03:06.139Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7442023-10-25T23:03:06.165Zresolver v6:
7452023-10-25T23:03:06.193Zfe80::6cff:bdff:fec0:aa9d -> 6e:ff:bd:c0:aa:9d
7462023-10-25T23:03:06.218Zfe80::280c:19ff:fe9b:4475 -> 2a:0c:19:9b:44:75
7472023-10-25T23:03:06.242Znat v4:
7482023-10-25T23:03:06.266Znat_v6:
7492023-10-25T23:03:06.289Zport_mac:
7502023-10-25T23:03:06.319Z1: a8:40:25:23:0c:95
7512023-10-25T23:03:06.349Z1000: a8:40:25:23:0c:94
7522023-10-25T23:03:06.378Zproxy arp:
7532023-10-25T23:03:06.406Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7542023-10-25T23:03:06.430Z+ retry=0
7552023-10-25T23:03:06.456Z+++ zoneadm list -n
7562023-10-25T23:03:06.486Z+++ grep oxz_ntp
7572023-10-25T23:03:06.512Z++ pfexec svcs -z -Hostate oxide/ntp
7582023-10-25T23:03:06.536Zsvcs: zone '-Hostate': entity not found
7592023-10-25T23:03:06.562Z++ true
7602023-10-25T23:03:06.586Z+ [[ '' != online ]]
7612023-10-25T23:03:06.611Z+ [[ 0 -gt 60 ]]
7622023-10-25T23:03:06.637Z+ sleep 1
7632023-10-25T23:03:06.794Z+ retry=1
7642023-10-25T23:03:06.822Z+++ zoneadm list -n
7652023-10-25T23:03:06.846Z+++ grep oxz_ntp
7662023-10-25T23:03:06.870Z++ pfexec svcs -z -Hostate oxide/ntp
7672023-10-25T23:03:06.898Zsvcs: zone '-Hostate': entity not found
7682023-10-25T23:03:07.331Z++ true
7692023-10-25T23:03:07.405Z+ [[ '' != online ]]
7702023-10-25T23:03:07.433Z+ [[ 1 -gt 60 ]]
7712023-10-25T23:03:07.868Z+ sleep 1
7722023-10-25T23:03:07.901Z+ retry=2
7732023-10-25T23:03:07.926Z+++ zoneadm list -n
7742023-10-25T23:03:07.953Z+++ grep oxz_ntp
7752023-10-25T23:03:08.389Z++ pfexec svcs -z -Hostate oxide/ntp
7762023-10-25T23:03:08.415Zsvcs: zone '-Hostate': entity not found
7772023-10-25T23:03:08.440Z++ true
7782023-10-25T23:03:08.464Z+ [[ '' != online ]]
7792023-10-25T23:03:08.488Z+ [[ 2 -gt 60 ]]
7802023-10-25T23:03:08.521Z+ sleep 1
7812023-10-25T23:03:08.838Z+ retry=3
7822023-10-25T23:03:08.866Z+++ zoneadm list -n
7832023-10-25T23:03:08.893Z+++ grep oxz_ntp
7842023-10-25T23:03:08.919Z++ pfexec svcs -z -Hostate oxide/ntp
7852023-10-25T23:03:08.944Zsvcs: zone '-Hostate': entity not found
7862023-10-25T23:03:08.967Z++ true
7872023-10-25T23:03:09.009Z+ [[ '' != online ]]
7882023-10-25T23:03:09.050Z+ [[ 3 -gt 60 ]]
7892023-10-25T23:03:09.074Z+ sleep 1
7902023-10-25T23:03:09.856Z+ retry=4
7912023-10-25T23:03:09.880Z+++ zoneadm list -n
7922023-10-25T23:03:09.904Z+++ grep oxz_ntp
7932023-10-25T23:03:09.928Z++ pfexec svcs -z -Hostate oxide/ntp
7942023-10-25T23:03:09.952Zsvcs: zone '-Hostate': entity not found
7952023-10-25T23:03:09.975Z++ true
7962023-10-25T23:03:10.000Z+ [[ '' != online ]]
7972023-10-25T23:03:10.025Z+ [[ 4 -gt 60 ]]
7982023-10-25T23:03:10.076Z+ sleep 1
7992023-10-25T23:03:10.878Z+ retry=5
8002023-10-25T23:03:10.902Z+++ zoneadm list -n
8012023-10-25T23:03:10.926Z+++ grep oxz_ntp
8022023-10-25T23:03:10.950Z++ pfexec svcs -z -Hostate oxide/ntp
8032023-10-25T23:03:10.977Zsvcs: zone '-Hostate': entity not found
8042023-10-25T23:03:11.005Z++ true
8052023-10-25T23:03:11.031Z+ [[ '' != online ]]
8062023-10-25T23:03:11.055Z+ [[ 5 -gt 60 ]]
8072023-10-25T23:03:11.101Z+ sleep 1
8082023-10-25T23:03:11.932Z+ retry=6
8092023-10-25T23:03:11.956Z+++ zoneadm list -n
8102023-10-25T23:03:11.984Z+++ grep oxz_ntp
8112023-10-25T23:03:12.427Z++ pfexec svcs -z -Hostate oxide/ntp
8122023-10-25T23:03:12.857Zsvcs: zone '-Hostate': entity not found
8132023-10-25T23:03:12.884Z++ true
8142023-10-25T23:03:12.910Z+ [[ '' != online ]]
8152023-10-25T23:03:12.937Z+ [[ 6 -gt 60 ]]
8162023-10-25T23:03:12.968Z+ sleep 1
8172023-10-25T23:03:12.999Z+ retry=7
8182023-10-25T23:03:13.028Z+++ zoneadm list -n
8192023-10-25T23:03:13.052Z+++ grep oxz_ntp
8202023-10-25T23:03:13.078Z++ pfexec svcs -z -Hostate oxide/ntp
8212023-10-25T23:03:13.103Zsvcs: zone '-Hostate': entity not found
8222023-10-25T23:03:13.157Z++ true
8232023-10-25T23:03:13.182Z+ [[ '' != online ]]
8242023-10-25T23:03:13.209Z+ [[ 7 -gt 60 ]]
8252023-10-25T23:03:13.652Z+ sleep 1
8262023-10-25T23:03:13.943Z+ retry=8
8272023-10-25T23:03:13.968Z+++ zoneadm list -n
8282023-10-25T23:03:13.992Z+++ grep oxz_ntp
8292023-10-25T23:03:14.015Z++ pfexec svcs -z -Hostate oxide/ntp
8302023-10-25T23:03:14.049Zsvcs: zone '-Hostate': entity not found
8312023-10-25T23:03:14.078Z++ true
8322023-10-25T23:03:14.102Z+ [[ '' != online ]]
8332023-10-25T23:03:14.127Z+ [[ 8 -gt 60 ]]
8342023-10-25T23:03:14.151Z+ sleep 1
8352023-10-25T23:03:14.975Z+ retry=9
8362023-10-25T23:03:15.000Z+++ zoneadm list -n
8372023-10-25T23:03:15.024Z+++ grep oxz_ntp
8382023-10-25T23:03:15.047Z++ pfexec svcs -z oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e -Hostate oxide/ntp
8392023-10-25T23:03:15.071Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8402023-10-25T23:03:15.095Z++ true
8412023-10-25T23:03:15.126Z+ [[ '' != online ]]
8422023-10-25T23:03:15.154Z+ [[ 9 -gt 60 ]]
8432023-10-25T23:03:15.177Z+ sleep 1
8442023-10-25T23:03:17.830Z+ retry=10
8452023-10-25T23:03:18.012Z+++ zoneadm list -n
8462023-10-25T23:03:18.054Z+++ grep oxz_ntp
8472023-10-25T23:03:18.183Z++ pfexec svcs -z oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e -Hostate oxide/ntp
8482023-10-25T23:03:18.219Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8492023-10-25T23:03:18.261Z++ true
8502023-10-25T23:03:18.296Z+ [[ '' != online ]]
8512023-10-25T23:03:18.328Z+ [[ 10 -gt 60 ]]
8522023-10-25T23:03:18.355Z+ sleep 1
8532023-10-25T23:03:18.384Z+ retry=11
8542023-10-25T23:03:18.429Z+++ zoneadm list -n
8552023-10-25T23:03:18.504Z+++ grep oxz_ntp
8562023-10-25T23:03:18.537Z++ pfexec svcs -z oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e -Hostate oxide/ntp
8572023-10-25T23:03:18.571Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8582023-10-25T23:03:19.029Z++ true
8592023-10-25T23:03:19.057Z+ [[ '' != online ]]
8602023-10-25T23:03:19.094Z+ [[ 11 -gt 60 ]]
8612023-10-25T23:03:19.124Z+ sleep 1
8622023-10-25T23:03:19.161Z+ retry=12
8632023-10-25T23:03:19.194Z+++ zoneadm list -n
8642023-10-25T23:03:19.227Z+++ grep oxz_ntp
8652023-10-25T23:03:19.286Z++ pfexec svcs -z oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e -Hostate oxide/ntp
8662023-10-25T23:03:19.329Z+ [[ disabled != online ]]
8672023-10-25T23:03:19.361Z+ [[ 12 -gt 60 ]]
8682023-10-25T23:03:19.409Z+ sleep 1
8692023-10-25T23:03:19.463Z+ retry=13
8702023-10-25T23:03:19.515Z+++ zoneadm list -n
8712023-10-25T23:03:19.545Z+++ grep oxz_ntp
8722023-10-25T23:03:19.571Z++ pfexec svcs -z oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e -Hostate oxide/ntp
8732023-10-25T23:03:19.596Z+ [[ disabled != online ]]
8742023-10-25T23:03:19.623Z+ [[ 13 -gt 60 ]]
8752023-10-25T23:03:19.672Z+ sleep 1
8762023-10-25T23:03:20.100Z+ retry=14
8772023-10-25T23:03:20.142Z+++ zoneadm list -n
8782023-10-25T23:03:20.167Z+++ grep oxz_ntp
8792023-10-25T23:03:20.190Z++ pfexec svcs -z oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e -Hostate oxide/ntp
8802023-10-25T23:03:20.219Z+ [[ disabled != online ]]
8812023-10-25T23:03:20.256Z+ [[ 14 -gt 60 ]]
8822023-10-25T23:03:20.287Z+ sleep 1
8832023-10-25T23:03:21.125Z+ retry=15
8842023-10-25T23:03:21.151Z+++ zoneadm list -n
8852023-10-25T23:03:21.179Z+++ grep oxz_ntp
8862023-10-25T23:03:21.208Z++ pfexec svcs -z oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e -Hostate oxide/ntp
8872023-10-25T23:03:21.234Z+ [[ disabled != online ]]
8882023-10-25T23:03:21.259Z+ [[ 15 -gt 60 ]]
8892023-10-25T23:03:21.291Z+ sleep 1
8902023-10-25T23:03:22.145Z+ retry=16
8912023-10-25T23:03:22.170Z+++ zoneadm list -n
8922023-10-25T23:03:22.195Z+++ grep oxz_ntp
8932023-10-25T23:03:22.229Z++ pfexec svcs -z oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e -Hostate oxide/ntp
8942023-10-25T23:03:22.260Z+ [[ disabled != online ]]
8952023-10-25T23:03:22.728Z+ [[ 16 -gt 60 ]]
8962023-10-25T23:03:23.621Z+ sleep 1
8972023-10-25T23:03:23.649Z+ retry=17
8982023-10-25T23:03:23.675Z+++ zoneadm list -n
8992023-10-25T23:03:23.703Z+++ grep oxz_ntp
9002023-10-25T23:03:23.729Z++ pfexec svcs -z oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e -Hostate oxide/ntp
9012023-10-25T23:03:24.167Z+ [[ disabled != online ]]
9022023-10-25T23:03:24.197Z+ [[ 17 -gt 60 ]]
9032023-10-25T23:03:24.230Z+ sleep 1
9042023-10-25T23:03:24.255Z+ retry=18
9052023-10-25T23:03:24.279Z+++ zoneadm list -n
9062023-10-25T23:03:24.303Z+++ grep oxz_ntp
9072023-10-25T23:03:24.327Z++ pfexec svcs -z oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e -Hostate oxide/ntp
9082023-10-25T23:03:24.350Z+ [[ disabled != online ]]
9092023-10-25T23:03:24.375Z+ [[ 18 -gt 60 ]]
9102023-10-25T23:03:24.399Z+ sleep 1
9112023-10-25T23:03:25.210Z+ retry=19
9122023-10-25T23:03:25.245Z+++ zoneadm list -n
9132023-10-25T23:03:25.697Z+++ grep oxz_ntp
9142023-10-25T23:03:25.731Z++ pfexec svcs -z oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e -Hostate oxide/ntp
9152023-10-25T23:03:25.767Z+ [[ disabled != online ]]
9162023-10-25T23:03:25.802Z+ [[ 19 -gt 60 ]]
 
9172023-10-25T23:03:25.829Zbuskin console login: Oct 25 23:03:24 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9182023-10-25T23:03:25.841Z+ sleep 1
9192023-10-25T23:03:26.233Z+ retry=20
9202023-10-25T23:03:26.261Z+++ zoneadm list -n
9212023-10-25T23:03:26.299Z+++ grep oxz_ntp
9222023-10-25T23:03:26.752Z++ pfexec svcs -z oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e -Hostate oxide/ntp
9232023-10-25T23:03:27.219Z+ [[ online != online ]]
9242023-10-25T23:03:27.245Z+ echo 'Waited for chrony: 20s'
9252023-10-25T23:03:27.269Z+ export RUST_BACKTRACE=1
9262023-10-25T23:03:27.294Z+ RUST_BACKTRACE=1
9272023-10-25T23:03:27.318ZWaited for chrony: 20s
9282023-10-25T23:03:27.405Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9292023-10-25T23:03:27.430Z+ ./tests/bootstrap
9302023-10-25T23:03:27.454ZUsing external DNS server at 10.151.1.164:53
9312023-10-25T23:03:27.478Z2023-10-25 23:03:25.628552902 UTC: attempting to log into API
9322023-10-25T23:03:41.726Z2023-10-25 23:03:40.838422177 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9332023-10-25T23:03:42.320Z2023-10-25 23:03:41.839604633 UTC: attempting to log into API
9342023-10-25T23:04:14.701Z2023-10-25 23:03:56.873143901 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9352023-10-25T23:04:14.805Z2023-10-25 23:03:57.875061130 UTC: attempting to log into API
9362023-10-25T23:04:14.950Z2023-10-25 23:04:12.907625379 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9372023-10-25T23:04:15.432Z2023-10-25 23:04:13.908772574 UTC: attempting to log into API
9382023-10-25T23:04:29.435Z2023-10-25 23:04:28.955952924 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9392023-10-25T23:04:30.438Z2023-10-25 23:04:29.957945905 UTC: attempting to log into API
 
9402023-10-25T23:04:41.718ZOct 25 23:04:41 bukin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9412023-10-25T23:04:45.501Z2023-10-25 23:04:45.344989272 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9422023-10-25T23:04:50.706ZOct 25 23:04:47 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9432023-10-25T23:04:50.714Z2023-10-25 23:04:46.348677194 UTC: attempting to log into API
 
9442023-10-25T23:04:55.414ZOct 25 23:04:54 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9452023-10-25T23:05:01.572Z2023-10-25 23:05:01.422225491 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9462023-10-25T23:05:10.443Z2023-10-25 23:05:02.425822596 UTC: attempting to log into API
 
9472023-10-25T23:05:10.451ZOct 25 23:05:02 bukin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
9482023-10-25T23:05:10.640ZOct 25 23:05:08 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9492023-10-25T23:05:17.566Z2023-10-25 23:05:17.499371831 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9502023-10-25T23:05:18.566Z2023-10-25 23:05:18.503151029 UTC: attempting to log into API
9512023-10-25T23:05:33.596Z2023-10-25 23:05:33.576749778 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9522023-10-25T23:05:34.910Z2023-10-25 23:05:34.580511766 UTC: attempting to log into API
9532023-10-25T23:05:40.391Z2023-10-25 23:05:40.387215107 UTC: login succeeded
9542023-10-25T23:05:40.423Zwaiting for nexus to come up...
9552023-10-25T23:05:40.484Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9562023-10-25T23:05:40.529Zensuring datasets are ready...
9572023-10-25T23:05:43.131Zlet's roll.
9582023-10-25T23:05:43.169Z+ rm ./tests/bootstrap
9592023-10-25T23:05:43.221Z+ for test_bin in tests/*
9602023-10-25T23:05:43.249Z+ ./tests/bootstrap-7505a19fa56a5ab1
9612023-10-25T23:05:43.281Z
9622023-10-25T23:05:43.331Zrunning 0 tests
9632023-10-25T23:05:43.419Z
9642023-10-25T23:05:43.452Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9652023-10-25T23:05:43.500Z
9662023-10-25T23:05:43.536Z+ for test_bin in tests/*
9672023-10-25T23:05:43.567Z+ ./tests/end_to_end_tests-a486f79ff30d68c1
9682023-10-25T23:05:43.597Z
9692023-10-25T23:05:43.628Zrunning 2 tests
9702023-10-25T23:05:43.717Ztest no_spoof::no_spoof_here ... ok
9712023-10-25T23:06:43.255Ztest instance_launch::instance_launch has been running for over 60 seconds
9722023-10-25T23:08:40.949Ztest instance_launch::instance_launch ... ok
9732023-10-25T23:08:40.988Z
9742023-10-25T23:08:41.015Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 177.70s
9752023-10-25T23:08:41.040Z
9762023-10-25T23:08:41.065Z+ _exit_trap
9772023-10-25T23:08:41.090Z+ local status=0
9782023-10-25T23:08:41.115Z+ [[ 0 -eq 0 ]]
9792023-10-25T23:08:41.140Z+ exit 0
9802023-10-25T23:08:41.165Zprocess exited: duration 407187 ms, exit code 0
 
9812023-10-25T23:08:41.244Zfound 47 output files
9822023-10-25T23:08:41.305Zuploading: /var/svc/log/oxide-sled-agent:default.log (436630 bytes)
9832023-10-25T23:08:42.024Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 436630 -> 437030
9842023-10-25T23:08:43.096Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9852023-10-25T23:08:43.124Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9f16cf09-3a23-48f2-ac02-10cb784336c4/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
9862023-10-25T23:08:44.606Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9f16cf09-3a23-48f2-ac02-10cb784336c4/root/var/svc/log/oxide-cockroachdb:default.log
9872023-10-25T23:08:44.647Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a7862d60-f735-4f2c-b1f9-55e04f663d6b/root/var/svc/log/oxide-crucible-agent:default.log (14114 bytes)
9882023-10-25T23:08:45.760Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a7862d60-f735-4f2c-b1f9-55e04f663d6b/root/var/svc/log/oxide-crucible-agent:default.log
9892023-10-25T23:08:45.785Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a7862d60-f735-4f2c-b1f9-55e04f663d6b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9902023-10-25T23:08:46.876Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a7862d60-f735-4f2c-b1f9-55e04f663d6b/root/var/svc/log/oxide-crucible-downstairs:default.log
9912023-10-25T23:08:46.907Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a7862d60-f735-4f2c-b1f9-55e04f663d6b/root/var/svc/log/oxide-crucible-downstairs:downstairs-4b2bc609-54d2-4900-98fb-cd384a7bd51f.log (3049 bytes)
9922023-10-25T23:08:48.003Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a7862d60-f735-4f2c-b1f9-55e04f663d6b/root/var/svc/log/oxide-crucible-downstairs:downstairs-4b2bc609-54d2-4900-98fb-cd384a7bd51f.log
9932023-10-25T23:08:48.038Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_92ac9c10-1360-4511-9b25-e3ddd943ee6f/root/var/svc/log/oxide-internal_dns:default.log (102328 bytes)
9942023-10-25T23:08:50.492Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_92ac9c10-1360-4511-9b25-e3ddd943ee6f/root/var/svc/log/oxide-internal_dns:default.log
9952023-10-25T23:08:50.525Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_2fe7fb16-b039-4e91-8e16-5fc5b4303851/root/var/svc/log/oxide-oximeter:default.log (172336 bytes)
9962023-10-25T23:08:51.154Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_2fe7fb16-b039-4e91-8e16-5fc5b4303851/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 172336 -> 178585
9972023-10-25T23:08:52.251Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_2fe7fb16-b039-4e91-8e16-5fc5b4303851/root/var/svc/log/oxide-oximeter:default.log
9982023-10-25T23:08:52.288Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a86b3e6a-112f-481e-8688-0bafcbf8bddf/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
9992023-10-25T23:08:53.791Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a86b3e6a-112f-481e-8688-0bafcbf8bddf/root/var/svc/log/oxide-cockroachdb:default.log
10002023-10-25T23:08:53.824Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6ef5f5ca-31d8-4b1a-86de-0089c22b6602/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10012023-10-25T23:08:55.327Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6ef5f5ca-31d8-4b1a-86de-0089c22b6602/root/var/svc/log/oxide-crucible-agent:default.log
10022023-10-25T23:08:55.367Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6ef5f5ca-31d8-4b1a-86de-0089c22b6602/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10032023-10-25T23:08:56.855Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6ef5f5ca-31d8-4b1a-86de-0089c22b6602/root/var/svc/log/oxide-crucible-downstairs:default.log
10042023-10-25T23:08:56.880Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_701d8f1c-1379-4d34-8b03-fa897243a50a/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10052023-10-25T23:08:57.956Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_701d8f1c-1379-4d34-8b03-fa897243a50a/root/var/svc/log/oxide-crucible-agent:default.log
10062023-10-25T23:08:57.982Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_701d8f1c-1379-4d34-8b03-fa897243a50a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10072023-10-25T23:08:59.057Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_701d8f1c-1379-4d34-8b03-fa897243a50a/root/var/svc/log/oxide-crucible-downstairs:default.log
10082023-10-25T23:08:59.092Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e/root/var/svc/log/oxide-ntp:default.log (1464 bytes)
10092023-10-25T23:09:00.178Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e/root/var/svc/log/oxide-ntp:default.log
10102023-10-25T23:09:00.217Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_45727201-b9fe-457e-9772-8e02011fdc79/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10112023-10-25T23:09:01.379Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_45727201-b9fe-457e-9772-8e02011fdc79/root/var/svc/log/oxide-cockroachdb:default.log
10122023-10-25T23:09:01.410Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_64730a42-6396-44ba-8b35-cf170d850813/root/var/svc/log/oxide-crucible-agent:default.log (14117 bytes)
10132023-10-25T23:09:02.536Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_64730a42-6396-44ba-8b35-cf170d850813/root/var/svc/log/oxide-crucible-agent:default.log
10142023-10-25T23:09:02.563Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_64730a42-6396-44ba-8b35-cf170d850813/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10152023-10-25T23:09:03.641Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_64730a42-6396-44ba-8b35-cf170d850813/root/var/svc/log/oxide-crucible-downstairs:default.log
10162023-10-25T23:09:03.666Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_64730a42-6396-44ba-8b35-cf170d850813/root/var/svc/log/oxide-crucible-downstairs:downstairs-fc4d996f-5ad0-40cd-97bf-805a92301206.log (3047 bytes)
10172023-10-25T23:09:04.738Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_64730a42-6396-44ba-8b35-cf170d850813/root/var/svc/log/oxide-crucible-downstairs:downstairs-fc4d996f-5ad0-40cd-97bf-805a92301206.log
10182023-10-25T23:09:04.765Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_c043a9ea-86c3-43ab-b037-36053a4624da/root/var/svc/log/oxide-clickhouse:default.log (606758 bytes)
10192023-10-25T23:09:05.782Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_c043a9ea-86c3-43ab-b037-36053a4624da/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 606758 -> 667398
10202023-10-25T23:09:06.864Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_c043a9ea-86c3-43ab-b037-36053a4624da/root/var/svc/log/oxide-clickhouse:default.log
10212023-10-25T23:09:06.892Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_179382bb-fe45-4bdc-9ae8-248a5887e1a3/root/var/svc/log/oxide-crucible-agent:default.log (14579 bytes)
10222023-10-25T23:09:08.665Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_179382bb-fe45-4bdc-9ae8-248a5887e1a3/root/var/svc/log/oxide-crucible-agent:default.log
10232023-10-25T23:09:08.702Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_179382bb-fe45-4bdc-9ae8-248a5887e1a3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10242023-10-25T23:09:09.790Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_179382bb-fe45-4bdc-9ae8-248a5887e1a3/root/var/svc/log/oxide-crucible-downstairs:default.log
10252023-10-25T23:09:09.819Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_179382bb-fe45-4bdc-9ae8-248a5887e1a3/root/var/svc/log/oxide-crucible-downstairs:downstairs-41a6714e-4629-4219-bfef-5ec88c746976.log (3050 bytes)
10262023-10-25T23:09:10.921Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_179382bb-fe45-4bdc-9ae8-248a5887e1a3/root/var/svc/log/oxide-crucible-downstairs:downstairs-41a6714e-4629-4219-bfef-5ec88c746976.log
10272023-10-25T23:09:10.955Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_9e7eb16b-0f1e-4d84-93f0-11d501ca6d70/root/var/svc/log/oxide-crucible-agent:default.log (14113 bytes)
10282023-10-25T23:09:12.359Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_9e7eb16b-0f1e-4d84-93f0-11d501ca6d70/root/var/svc/log/oxide-crucible-agent:default.log
10292023-10-25T23:09:12.390Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_9e7eb16b-0f1e-4d84-93f0-11d501ca6d70/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10302023-10-25T23:09:13.232Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_9e7eb16b-0f1e-4d84-93f0-11d501ca6d70/root/var/svc/log/oxide-crucible-downstairs:default.log
10312023-10-25T23:09:13.262Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_9e7eb16b-0f1e-4d84-93f0-11d501ca6d70/root/var/svc/log/oxide-crucible-downstairs:downstairs-9505a40c-ef66-444e-bba0-97791b926dcf.log (5533 bytes)
10322023-10-25T23:09:14.376Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_9e7eb16b-0f1e-4d84-93f0-11d501ca6d70/root/var/svc/log/oxide-crucible-downstairs:downstairs-9505a40c-ef66-444e-bba0-97791b926dcf.log
10332023-10-25T23:09:14.405Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b7ad9162-9e9e-467b-a692-1bdc39969684/root/var/svc/log/oxide-crucible-agent:default.log (14117 bytes)
10342023-10-25T23:09:15.939Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b7ad9162-9e9e-467b-a692-1bdc39969684/root/var/svc/log/oxide-crucible-agent:default.log
10352023-10-25T23:09:15.969Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b7ad9162-9e9e-467b-a692-1bdc39969684/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10362023-10-25T23:09:17.500Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b7ad9162-9e9e-467b-a692-1bdc39969684/root/var/svc/log/oxide-crucible-downstairs:default.log
10372023-10-25T23:09:17.537Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b7ad9162-9e9e-467b-a692-1bdc39969684/root/var/svc/log/oxide-crucible-downstairs:downstairs-7d8d57d9-492c-4b6d-ab08-6670e03a8b9a.log (5535 bytes)
10382023-10-25T23:09:19.152Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b7ad9162-9e9e-467b-a692-1bdc39969684/root/var/svc/log/oxide-crucible-downstairs:downstairs-7d8d57d9-492c-4b6d-ab08-6670e03a8b9a.log
10392023-10-25T23:09:19.204Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_58b4da41-de6f-458b-93d0-596819f581dc/root/var/svc/log/oxide-nexus:default.log (4594366 bytes)
10402023-10-25T23:09:23.479Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_58b4da41-de6f-458b-93d0-596819f581dc/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4594366 -> 4654984
10412023-10-25T23:09:25.097Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_58b4da41-de6f-458b-93d0-596819f581dc/root/var/svc/log/oxide-nexus:default.log
10422023-10-25T23:09:25.140Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_566f66f3-dfae-4825-af44-c214dc3c5a8b/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
10432023-10-25T23:09:26.247Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_566f66f3-dfae-4825-af44-c214dc3c5a8b/root/var/svc/log/oxide-cockroachdb:default.log
10442023-10-25T23:09:26.271Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_bbe938ff-3363-4c68-80b1-38277773e91a/root/var/svc/log/oxide-external_dns:default.log (15609 bytes)
10452023-10-25T23:09:26.364Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_bbe938ff-3363-4c68-80b1-38277773e91a/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15609 -> 19386
10462023-10-25T23:09:27.446Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_bbe938ff-3363-4c68-80b1-38277773e91a/root/var/svc/log/oxide-external_dns:default.log
10472023-10-25T23:09:27.475Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_ebcefb04-45b1-4cca-8107-79b30e6e152b/root/var/svc/log/oxide-external_dns:default.log (15613 bytes)
10482023-10-25T23:09:28.060Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_ebcefb04-45b1-4cca-8107-79b30e6e152b/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15613 -> 19389
10492023-10-25T23:09:29.149Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_ebcefb04-45b1-4cca-8107-79b30e6e152b/root/var/svc/log/oxide-external_dns:default.log
10502023-10-25T23:09:29.177Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_c3dd58c8-0b67-4ed7-874c-22a65ac30a10/root/var/svc/log/oxide-internal_dns:default.log (99565 bytes)
10512023-10-25T23:09:29.908Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_c3dd58c8-0b67-4ed7-874c-22a65ac30a10/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99565 -> 103318
10522023-10-25T23:09:30.981Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_c3dd58c8-0b67-4ed7-874c-22a65ac30a10/root/var/svc/log/oxide-internal_dns:default.log
10532023-10-25T23:09:31.012Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9601f99f-4c60-40bf-8886-d07a92365fc7/root/var/svc/log/oxide-crucible-agent:default.log (14112 bytes)
10542023-10-25T23:09:32.130Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9601f99f-4c60-40bf-8886-d07a92365fc7/root/var/svc/log/oxide-crucible-agent:default.log
10552023-10-25T23:09:32.155Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9601f99f-4c60-40bf-8886-d07a92365fc7/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10562023-10-25T23:09:33.229Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9601f99f-4c60-40bf-8886-d07a92365fc7/root/var/svc/log/oxide-crucible-downstairs:default.log
10572023-10-25T23:09:33.254Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9601f99f-4c60-40bf-8886-d07a92365fc7/root/var/svc/log/oxide-crucible-downstairs:downstairs-71b2a10e-f14e-4226-a0e4-d8d4a74440cc.log (5534 bytes)
10582023-10-25T23:09:34.359Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9601f99f-4c60-40bf-8886-d07a92365fc7/root/var/svc/log/oxide-crucible-downstairs:downstairs-71b2a10e-f14e-4226-a0e4-d8d4a74440cc.log
10592023-10-25T23:09:34.436Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_df84ea73-3b90-4609-95aa-256501715480/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10602023-10-25T23:09:35.922Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_df84ea73-3b90-4609-95aa-256501715480/root/var/svc/log/oxide-crucible-pantry:default.log
10612023-10-25T23:09:35.952Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_fd66048d-61bf-4b44-a36f-a5cdd3acdbb3/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10622023-10-25T23:09:37.030Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_fd66048d-61bf-4b44-a36f-a5cdd3acdbb3/root/var/svc/log/oxide-crucible-pantry:default.log
10632023-10-25T23:09:37.055Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_4af024ea-0d43-4f6d-b576-b5e683f1cb1c/root/var/svc/log/oxide-internal_dns:default.log (99891 bytes)
10642023-10-25T23:09:37.346Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_4af024ea-0d43-4f6d-b576-b5e683f1cb1c/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99891 -> 103646
10652023-10-25T23:09:38.421Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_4af024ea-0d43-4f6d-b576-b5e683f1cb1c/root/var/svc/log/oxide-internal_dns:default.log
10662023-10-25T23:09:38.856Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_b863e85b-2975-4287-b3d0-7e3868599b59/root/var/svc/log/oxide-nexus:default.log (498740 bytes)
10672023-10-25T23:09:39.861Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_b863e85b-2975-4287-b3d0-7e3868599b59/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 498740 -> 559960
10682023-10-25T23:09:40.934Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_b863e85b-2975-4287-b3d0-7e3868599b59/root/var/svc/log/oxide-nexus:default.log
10692023-10-25T23:09:40.970Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_e758eb1f-72c2-4704-b3e6-98c95d915c68/root/var/svc/log/oxide-nexus:default.log (503114 bytes)
10702023-10-25T23:09:41.926Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_e758eb1f-72c2-4704-b3e6-98c95d915c68/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 503114 -> 564939
10712023-10-25T23:09:42.969Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_e758eb1f-72c2-4704-b3e6-98c95d915c68/root/var/svc/log/oxide-nexus:default.log
10722023-10-25T23:09:42.994Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f174fdfd-88d2-460f-8882-284e3b697970/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10732023-10-25T23:09:44.075Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f174fdfd-88d2-460f-8882-284e3b697970/root/var/svc/log/oxide-crucible-agent:default.log
10742023-10-25T23:09:44.110Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f174fdfd-88d2-460f-8882-284e3b697970/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10752023-10-25T23:09:45.179Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f174fdfd-88d2-460f-8882-284e3b697970/root/var/svc/log/oxide-crucible-downstairs:default.log
10762023-10-25T23:09:45.206Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_d89eb5cb-9319-4e50-8bfe-8ce5ae177aab/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10772023-10-25T23:09:46.287Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_d89eb5cb-9319-4e50-8bfe-8ce5ae177aab/root/var/svc/log/oxide-crucible-pantry:default.log
10782023-10-25T23:09:46.312Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_241fab19-0c3c-417e-950f-d8724ac06cf3/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
10792023-10-25T23:09:47.385Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_241fab19-0c3c-417e-950f-d8724ac06cf3/root/var/svc/log/oxide-cockroachdb:default.log
10802023-10-25T23:09:47.417Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e/root/var/log/chrony/measurements.log (11645 bytes)
10812023-10-25T23:09:47.463Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e/root/var/log/chrony/measurements.log" changed size mid upload: 11645 -> 13289
10822023-10-25T23:09:48.539Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e/root/var/log/chrony/measurements.log
10832023-10-25T23:09:48.568Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e/root/var/log/chrony/statistics.log (6786 bytes)
10842023-10-25T23:09:48.614Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e/root/var/log/chrony/statistics.log" changed size mid upload: 6786 -> 8190
10852023-10-25T23:09:49.702Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e/root/var/log/chrony/statistics.log
10862023-10-25T23:09:49.733Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e/root/var/log/chrony/tracking.log (1988 bytes)
10872023-10-25T23:09:49.762Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e/root/var/log/chrony/tracking.log" changed size mid upload: 1988 -> 2120
10882023-10-25T23:09:50.833Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_8c867ea2-3821-48e2-b2dd-a22d849df50e/root/var/log/chrony/tracking.log