01HER4RYPK2SZ75HRQY0BPJ1W2: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HER4TBDH99XVA8V7Z8HNG0TR

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-08T19:53:23.754Zjob dependencies complete; ready to run (waiting for 54 m 33 s)
22023-11-08T20:11:18.944Zjob assigned to worker 01HER8XTG78MQYPTZB1DEFG445 (queued for 17 m 55 s)
32023-11-08T20:11:19.148Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-11-08T20:11:19.234Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-11-08T20:11:19.314Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-11-08T20:11:19.349Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-08T20:11:19.383Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-11-08T20:11:19.418ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-08T20:11:19.453Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-08T20:11:19.488ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-08T20:11:19.525ZConfiguring devices.
122023-11-08T20:11:19.565Z#################################################
132023-11-08T20:11:19.604Z#### ############# ##
142023-11-08T20:11:19.636Z### ### ############ ##
152023-11-08T20:11:19.669Z## ### # ## ### ## Oxide Computer Company ##
162023-11-08T20:11:19.737Z## ## ## ### # ### ##
172023-11-08T20:11:19.769Z## # ### #### #### This Station Under ##
182023-11-08T20:11:19.803Z### ### #### # ### Computer Control ##
192023-11-08T20:11:19.837Z#### #### ### ## ##
202023-11-08T20:11:19.870Z##################################################
212023-11-08T20:11:19.903ZHostname: sock
222023-11-08T20:11:19.937ZLoading smf(7) serice descriptions:
232023-11-08T20:11:20.891Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-95f6f69deb9d73aa.gz
242023-11-08T20:11:22.269Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-95f6f69deb9d73aa.gz
252023-11-08T20:11:22.302Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-08T20:11:48.111Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-08T20:11:48.137Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cc026c1de0ba49ab.gz
282023-11-08T20:12:20.293Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cc026c1de0ba49ab.gz
292023-11-08T20:12:20.322Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-08T20:12:21.627Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-08T20:12:21.661Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-08T20:12:25.820Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-08T20:12:25.846Zdownloading input: /input/package/work/version.txt
342023-11-08T20:12:25.908Zdownloaded input: /input/package/work/version.txt
352023-11-08T20:12:25.935Zdownloading input: /input/package/work/package.tar.gz
362023-11-08T20:13:05.104Zdownloaded input: /input/package/work/package.tar.gz
372023-11-08T20:13:05.131Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-08T20:13:30.176Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-08T20:13:30.207Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-08T20:13:37.827Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-08T20:13:37.851Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-08T20:14:14.150Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-08T20:14:14.178Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-08T20:14:52.552Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-08T20:14:52.577Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-08T20:15:15.748Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-08T20:15:15.799Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-08T20:15:21.741Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-08T20:15:21.767Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-08T20:15:30.291Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-08T20:15:30.317Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-08T20:15:35.443Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-08T20:15:35.935Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-08T20:16:09.197Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-08T20:16:09.251Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-08T20:16:09.495Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-08T20:16:09.537Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-08T20:16:13.193Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-08T20:16:13.218Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-08T20:16:16.840Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-08T20:16:16.865Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-08T20:16:29.980Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-08T20:16:30.008Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-08T20:16:42.979Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-08T20:16:43.004Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-08T20:16:43.071Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-08T20:16:43.095Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-08T20:16:46.996Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-08T20:16:47.021Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-08T20:16:59.819Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-08T20:16:59.844Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-08T20:17:45.604Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-08T20:17:45.629Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-08T20:18:19.582Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-08T20:18:19.609Zstarting task 0: "setup"
762023-11-08T20:18:19.643Z++ uname -s
772023-11-08T20:18:19.668Z+ kern=SunOS
782023-11-08T20:18:19.694Z+ case "$kern" in
792023-11-08T20:18:19.719Z+ groupadd -g 12345 build
802023-11-08T20:18:19.750Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-08T20:18:21.652Z+ zfs create -o mountpoint=/work rpool/work
822023-11-08T20:18:21.681Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-08T20:18:21.706Z+ home_fs=zfs
842023-11-08T20:18:21.733Z+ [[ zfs == autofs ]]
852023-11-08T20:18:21.757Z+ mkdir -p /home/build
862023-11-08T20:18:21.783Z+ chown build:build /home/build /work
872023-11-08T20:18:23.690Z+ chmod 0700 /home/build /work
882023-11-08T20:18:23.714Zprocess exited: duration 4060 ms, exit code 0
 
892023-11-08T20:18:23.766Zstarting task 1: "authentication"
902023-11-08T20:18:23.836Zprocess exited: duration 44 ms, exit code 0
 
912023-11-08T20:18:23.889Zstarting task 2: "build"
922023-11-08T20:18:23.921Z+ trap _exit_trap EXIT
932023-11-08T20:18:23.956Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-08T20:18:24.121Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-08T20:18:24.146Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-08T20:18:24.170Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-08T20:18:24.195Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-08T20:18:24.222Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-08T20:18:24.247Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-08T20:18:24.271Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-08T20:18:24.297Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-08T20:18:24.322Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-08T20:18:24.353Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-08T20:18:24.384Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-08T20:18:24.849Z+ [[ -d /opt/oxide ]]
1062023-11-08T20:18:24.878Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-08T20:18:24.906Z+ pfexec mkdir /opt/oxide
1082023-11-08T20:18:24.931Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-08T20:18:25.364Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-08T20:18:25.390Z+ cd /opt/oxide-underneath
1112023-11-08T20:18:25.415Z+ pfexec tar ceEp@/f - .
1122023-11-08T20:18:25.440Z+ cd /opt/oxide
1132023-11-08T20:18:25.464Z+ pfexec tar xveEp@/f -
1142023-11-08T20:18:25.489Zx ., 0 bytes, 0 tape blocks
1152023-11-08T20:18:25.515Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-08T20:18:25.539Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-08T20:18:25.576Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-08T20:18:25.600Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-08T20:18:25.625Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-08T20:18:25.651Z+ pfexec mkdir /opt/oxide/work
1212023-11-08T20:18:25.675Z+ pfexec chown build:build /opt/oxide/work
1222023-11-08T20:18:25.699Z+ cd /opt/oxide/work
1232023-11-08T20:18:25.723Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-08T20:18:26.376ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-08T20:18:26.838Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1262023-11-08T20:18:26.863Zx out/omicron-sled-agent.tar, 122315776 bytes, 238898 tape blocks
1272023-11-08T20:18:26.887Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-08T20:18:26.911Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-11-08T20:18:26.936Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-11-08T20:18:26.960Zx smf/sled-agent/non-gimlet/config.toml, 3102 bytes, 7 tape blocks
1312023-11-08T20:18:26.985Zx target/release/omicron-package, 42420592 bytes, 82853 tape blocks
1322023-11-08T20:18:27.010Zx tools/create_virtual_hardware.sh, 3067 bytes, 6 tape blocks
1332023-11-08T20:18:27.035Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-08T20:18:27.062Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-08T20:18:27.087Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-08T20:18:27.112Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-08T20:18:27.136Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-08T20:18:27.160Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-08T20:18:27.189Z
1402023-11-08T20:18:27.213Zreal 1.612887308
1412023-11-08T20:18:27.240Zuser 1.448817579
1422023-11-08T20:18:27.684Zsys 0.286451036
1432023-11-08T20:18:27.709Ztrap 0.000048741
1442023-11-08T20:18:27.733Ztflt 0.000000000
1452023-11-08T20:18:27.767Zdflt 0.000000000
1462023-11-08T20:18:27.797Zkflt 0.000000000
1472023-11-08T20:18:27.823Zlock 0.000000000
1482023-11-08T20:18:27.849Zslp 1.452693554
1492023-11-08T20:18:27.873Zlat 0.035508307
1502023-11-08T20:18:27.898Zstop 0.000126012
1512023-11-08T20:18:27.927Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-08T20:18:27.954Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-08T20:18:27.980Z+ mkdir tests
1542023-11-08T20:18:28.005Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-08T20:18:28.033Z+ ptime -m gunzip
1562023-11-08T20:18:28.057Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-95f6f69deb9d73aa
1572023-11-08T20:18:28.082Z
1582023-11-08T20:18:28.107Zreal 0.137531390
1592023-11-08T20:18:28.133Zuser 0.127132249
1602023-11-08T20:18:28.157Zsys 0.010156604
1612023-11-08T20:18:28.584Ztrap 0.000013110
1622023-11-08T20:18:28.611Ztflt 0.000000000
1632023-11-08T20:18:28.654Zdflt 0.000000000
1642023-11-08T20:18:28.679Zkflt 0.000000000
1652023-11-08T20:18:28.705Zlock 0.000000000
1662023-11-08T20:18:28.731Zslp 0.000016940
1672023-11-08T20:18:28.755Zlat 0.000020760
1682023-11-08T20:18:28.780Zstop 0.000123872
1692023-11-08T20:18:28.814Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-95f6f69deb9d73aa
1702023-11-08T20:18:28.841Z+ chmod a+x tests/bootstrap-95f6f69deb9d73aa
1712023-11-08T20:18:28.865Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-08T20:18:28.900Z+ ptime -m gunzip
1732023-11-08T20:18:28.930Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-08T20:18:29.471Z
1752023-11-08T20:18:29.496Zreal 2.325905651
1762023-11-08T20:18:29.520Zuser 2.182391564
1772023-11-08T20:18:29.546Zsys 0.143084107
1782023-11-08T20:18:29.570Ztrap 0.000193925
1792023-11-08T20:18:29.594Ztflt 0.000000000
1802023-11-08T20:18:29.619Zdflt 0.000000000
1812023-11-08T20:18:29.646Zkflt 0.000000000
1822023-11-08T20:18:29.694Zlock 0.000000000
1832023-11-08T20:18:29.719Zslp 0.000025000
1842023-11-08T20:18:29.743Zlat 0.000031830
1852023-11-08T20:18:29.768Zstop 0.000121392
1862023-11-08T20:18:29.792Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-08T20:18:29.816Z+ chmod a+x tests/bootstrap
1882023-11-08T20:18:29.841Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-08T20:18:29.865Z+ ptime -m gunzip
1902023-11-08T20:18:29.889Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cc026c1de0ba49ab
1912023-11-08T20:18:41.318Z
1922023-11-08T20:18:41.597Zreal 2.669887024
1932023-11-08T20:18:42.196Zuser 2.511440633
1942023-11-08T20:18:42.365Zsys 0.157990343
1952023-11-08T20:18:42.581Ztrap 0.000214902
1962023-11-08T20:18:42.719Ztflt 0.000000000
1972023-11-08T20:18:42.795Zdflt 0.000000000
1982023-11-08T20:18:42.852Zkflt 0.000000000
1992023-11-08T20:18:42.909Zlock 0.000000000
2002023-11-08T20:18:43.031Zslp 0.000025180
2012023-11-08T20:18:43.176Zlat 0.000024120
2022023-11-08T20:18:43.240Zstop 0.000126882
2032023-11-08T20:18:43.371Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cc026c1de0ba49ab
2042023-11-08T20:18:43.434Z+ chmod a+x tests/end_to_end_tests-cc026c1de0ba49ab
2052023-11-08T20:18:43.476Z+ bmat address ls
2062023-11-08T20:18:43.528ZNAME CIDR FIRST LAST COUNT
2072023-11-08T20:18:43.560Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-11-08T20:18:43.596Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-08T20:18:43.629Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-11-08T20:18:43.673Z+ EXTRA_IP_START=10.151.2.164
2112023-11-08T20:18:43.708Z+ EXTRA_IP_END=10.151.2.227
2122023-11-08T20:18:43.742Z+ EXTRA_IP_COUNT=64
2132023-11-08T20:18:43.796Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-08T20:18:43.860Z+ EXTRA_IP_BASE=10.151.2
2152023-11-08T20:18:43.893Z+ EXTRA_IP_FOCTET=164
2162023-11-08T20:18:43.944Z+ EXTRA_IP_LOCTET=227
2172023-11-08T20:18:44.008Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-11-08T20:18:44.047Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-11-08T20:18:44.110Z+ DNS_IP1=10.151.2.164
2202023-11-08T20:18:44.182Z+ DNS_IP2=10.151.2.165
2212023-11-08T20:18:44.216Z+ UPLINK_IP=10.151.2.174
2222023-11-08T20:18:44.284Z+ IPPOOL_START=10.151.2.175
2232023-11-08T20:18:44.584Z+ IPPOOL_END=10.151.2.227
2242023-11-08T20:18:44.633Z++ ipadm show-addr -po type,addr
2252023-11-08T20:18:44.664Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-08T20:18:44.700Z+ GATEWAY_IP=10.151.2.100
2272023-11-08T20:18:44.732Z+ [[ -n 10.151.2.100 ]]
2282023-11-08T20:18:44.776Z+ ping -s 10.151.2.100 56 1
2292023-11-08T20:18:44.804ZPING 10.151.2.100: 56 data bytes
2302023-11-08T20:18:44.835Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.014 ms
2312023-11-08T20:18:44.870Z
2322023-11-08T20:18:44.903Z----10.151.2.100 PING Statistics----
2332023-11-08T20:18:44.935Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-08T20:18:44.968Zround-trip (ms) min/avg/max/stddev = 0.014/0.014/0.014/-nan
2352023-11-08T20:18:44.997Z++ arp -an
2362023-11-08T20:18:45.030Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-11-08T20:18:45.057Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-11-08T20:18:45.094Z+ routeadm -e ipv4-forwarding -u
2392023-11-08T20:18:45.124Z+ PXA_START=10.151.2.164
2402023-11-08T20:18:45.152Z+ PXA_END=10.151.2.227
2412023-11-08T20:18:45.183Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-08T20:18:45.622Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-08T20:18:45.646Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-08T20:18:45.671Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-08T20:18:45.696Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-08T20:18:45.722Z++ cd ./tools
2472023-11-08T20:18:45.747Z++ pwd
2482023-11-08T20:18:45.773Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-08T20:18:45.797Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-08T20:18:45.848Z+ SOFTNPU_MODE=zone
2512023-11-08T20:18:45.873Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-08T20:18:45.900Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-08T20:18:45.934Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-08T20:18:45.975Z++ dladm show-phys -p -o LINK
2552023-11-08T20:18:46.000Z++ head -1
2562023-11-08T20:18:46.024Z+ PHYSICAL_LINK=igb0
2572023-11-08T20:18:46.049Z+ echo 'Using igb0 as physical link'
2582023-11-08T20:18:46.074ZUsing igb0 as physical link
2592023-11-08T20:18:46.098Z+ ensure_run_as_root
2602023-11-08T20:18:46.122Z++ id -u
2612023-11-08T20:18:46.147Z+ [[ 0 -ne 0 ]]
2622023-11-08T20:18:46.171Z+ ensure_zpools
2632023-11-08T20:18:46.195Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-08T20:18:46.220ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2652023-11-08T20:18:46.244Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2662023-11-08T20:18:46.269ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2672023-11-08T20:18:46.294Z+ readarray -t ZPOOLS
2682023-11-08T20:18:46.319ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2692023-11-08T20:18:46.343Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2702023-11-08T20:18:46.369ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2712023-11-08T20:18:46.402Z++ sed 's/[ ",]//g'
2722023-11-08T20:18:46.439ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2732023-11-08T20:18:46.464Z+ for ZPOOL in "${ZPOOLS[@]}"
2742023-11-08T20:18:46.515ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2752023-11-08T20:18:46.542Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2762023-11-08T20:18:46.569ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2772023-11-08T20:18:46.594Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2782023-11-08T20:18:46.621ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2792023-11-08T20:18:46.645Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2802023-11-08T20:18:46.670ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2812023-11-08T20:18:46.694Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2822023-11-08T20:18:46.721ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2832023-11-08T20:18:46.749Z0+0 records in
2842023-11-08T20:18:46.773ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2852023-11-08T20:18:46.801Z0+0 records out
2862023-11-08T20:18:46.828ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2872023-11-08T20:18:46.858Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2882023-11-08T20:18:46.884ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2892023-11-08T20:18:46.913Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2902023-11-08T20:18:46.940ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2912023-11-08T20:18:46.968Z+ set +x
2922023-11-08T20:18:47.004ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2932023-11-08T20:18:47.035Z++ zpool list -o name
2942023-11-08T20:18:47.063ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
2952023-11-08T20:18:47.090Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2962023-11-08T20:18:47.116ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
2972023-11-08T20:18:47.141Z+ [[ -z '' ]]
2982023-11-08T20:18:47.165ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
2992023-11-08T20:18:47.190Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
3002023-11-08T20:18:47.215ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3012023-11-08T20:18:47.239Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
3022023-11-08T20:18:47.263ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3032023-11-08T20:18:47.288Z+ set +x
3042023-11-08T20:18:47.312ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3052023-11-08T20:18:47.336Z+ for ZPOOL in "${ZPOOLS[@]}"
3062023-11-08T20:18:47.361ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3072023-11-08T20:18:47.385Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-11-08T20:18:47.409ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3092023-11-08T20:18:47.434Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-11-08T20:18:47.898ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3112023-11-08T20:18:47.923Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3122023-11-08T20:18:47.947ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3132023-11-08T20:18:47.971Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3142023-11-08T20:18:47.995ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3152023-11-08T20:18:48.020Z0+0 records in
3162023-11-08T20:18:48.047ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3172023-11-08T20:18:48.072Z0+0 records out
3182023-11-08T20:18:48.098ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3192023-11-08T20:18:48.125Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3202023-11-08T20:18:48.156ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3212023-11-08T20:18:48.181Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3222023-11-08T20:18:48.205ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3232023-11-08T20:18:48.230Z+ set +x
3242023-11-08T20:18:48.254ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3252023-11-08T20:18:48.278Z++ zpool list -o name
3262023-11-08T20:18:48.302ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3272023-11-08T20:18:48.327Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3282023-11-08T20:18:48.352ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3292023-11-08T20:18:48.377Z+ [[ -z '' ]]
3302023-11-08T20:18:48.401ZSimnet net0/sc0_0 exists
3312023-11-08T20:18:48.426Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3322023-11-08T20:18:48.450ZSimnet net1/sc1_0 exists
3332023-11-08T20:18:48.476Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3342023-11-08T20:18:48.501ZVnic sc0_1 exists
3352023-11-08T20:18:48.525Z+ set +x
3362023-11-08T20:18:48.550Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3372023-11-08T20:18:48.589Z+ for ZPOOL in "${ZPOOLS[@]}"
3382023-11-08T20:18:48.615Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3392023-11-08T20:18:48.641Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3402023-11-08T20:18:48.666Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3412023-11-08T20:18:48.696Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3422023-11-08T20:18:48.721Z0+0 records in
3432023-11-08T20:18:48.747Z0+0 records out
3442023-11-08T20:18:48.773Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3452023-11-08T20:18:48.797Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3462023-11-08T20:18:48.822Z+ set +x
3472023-11-08T20:18:48.847Z++ zpool list -o name
3482023-11-08T20:18:48.871Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3492023-11-08T20:18:48.897Z+ [[ -z '' ]]
3502023-11-08T20:18:48.921Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3512023-11-08T20:18:48.945Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3522023-11-08T20:18:48.970Z+ set +x
3532023-11-08T20:18:48.994Z+ for ZPOOL in "${ZPOOLS[@]}"
3542023-11-08T20:18:49.019Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552023-11-08T20:18:49.043Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562023-11-08T20:18:49.068Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572023-11-08T20:18:49.092Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582023-11-08T20:18:49.116Z0+0 records in
3592023-11-08T20:18:49.140Z0+0 records out
3602023-11-08T20:18:49.165Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3612023-11-08T20:18:49.190Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622023-11-08T20:18:49.215Z+ set +x
3632023-11-08T20:18:49.239Z++ zpool list -o name
3642023-11-08T20:18:49.263Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3652023-11-08T20:18:49.287Z+ [[ -z '' ]]
3662023-11-08T20:18:49.722Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3672023-11-08T20:18:49.753Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3682023-11-08T20:18:49.777Z+ set +x
3692023-11-08T20:18:49.801Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-11-08T20:18:49.826Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3712023-11-08T20:18:49.850Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3722023-11-08T20:18:49.878Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3732023-11-08T20:18:49.903Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3742023-11-08T20:18:49.928Z0+0 records in
3752023-11-08T20:18:49.952Z0+0 records out
3762023-11-08T20:18:49.977Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3772023-11-08T20:18:50.002Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3782023-11-08T20:18:50.026Z+ set +x
3792023-11-08T20:18:50.050Z++ zpool list -o name
3802023-11-08T20:18:50.074Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3812023-11-08T20:18:50.098Z+ [[ -z '' ]]
3822023-11-08T20:18:50.123Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3832023-11-08T20:18:50.147Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3842023-11-08T20:18:50.172Z+ set +x
3852023-11-08T20:18:50.196Z+ for ZPOOL in "${ZPOOLS[@]}"
3862023-11-08T20:18:50.220Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-11-08T20:18:50.244Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-11-08T20:18:50.268Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-11-08T20:18:50.293Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-11-08T20:18:50.317Z0+0 records in
3912023-11-08T20:18:50.341Z0+0 records out
3922023-11-08T20:18:50.365Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3932023-11-08T20:18:50.389Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942023-11-08T20:18:50.413Z+ set +x
3952023-11-08T20:18:50.437Z++ zpool list -o name
3962023-11-08T20:18:50.461Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972023-11-08T20:18:50.485Z+ [[ -z '' ]]
3982023-11-08T20:18:50.510Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992023-11-08T20:18:50.534Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002023-11-08T20:18:50.558Z+ set +x
4012023-11-08T20:18:50.587Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-08T20:18:50.645Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4032023-11-08T20:18:50.672Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-11-08T20:18:50.696Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4052023-11-08T20:18:50.721Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4062023-11-08T20:18:50.747Z0+0 records in
4072023-11-08T20:18:50.774Z0+0 records out
4082023-11-08T20:18:50.798Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4092023-11-08T20:18:50.824Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4102023-11-08T20:18:50.848Z+ set +x
4112023-11-08T20:18:50.900Z++ zpool list -o name
4122023-11-08T20:18:50.927Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4132023-11-08T20:18:50.959Z+ [[ -z '' ]]
4142023-11-08T20:18:50.984Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4152023-11-08T20:18:51.009Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4162023-11-08T20:18:51.033Z+ set +x
4172023-11-08T20:18:51.058Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-11-08T20:18:51.084Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4192023-11-08T20:18:51.109Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4202023-11-08T20:18:51.134Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4212023-11-08T20:18:51.159Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4222023-11-08T20:18:51.183Z0+0 records in
4232023-11-08T20:18:51.207Z0+0 records out
4242023-11-08T20:18:51.232Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4252023-11-08T20:18:51.257Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4262023-11-08T20:18:51.281Z+ set +x
4272023-11-08T20:18:51.305Z++ zpool list -o name
4282023-11-08T20:18:51.329Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4292023-11-08T20:18:51.354Z+ [[ -z '' ]]
4302023-11-08T20:18:51.378Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4312023-11-08T20:18:51.402Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4322023-11-08T20:18:51.427Z+ set +x
4332023-11-08T20:18:51.452Z+ for ZPOOL in "${ZPOOLS[@]}"
4342023-11-08T20:18:51.476Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4352023-11-08T20:18:51.506Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-11-08T20:18:51.533Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4372023-11-08T20:18:51.557Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4382023-11-08T20:18:51.582Z0+0 records in
4392023-11-08T20:18:51.608Z0+0 records out
4402023-11-08T20:18:51.632Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4412023-11-08T20:18:51.683Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4422023-11-08T20:18:51.712Z+ set +x
4432023-11-08T20:18:51.744Z++ zpool list -o name
4442023-11-08T20:18:51.772Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4452023-11-08T20:18:51.799Z+ [[ -z '' ]]
4462023-11-08T20:18:51.828Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4472023-11-08T20:18:51.855Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4482023-11-08T20:18:51.882Z+ set +x
4492023-11-08T20:18:51.909Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4502023-11-08T20:18:51.941Z+ readarray -t ZPOOLS
4512023-11-08T20:18:51.984Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4522023-11-08T20:18:52.013Z++ sed 's/[ ",]//g'
4532023-11-08T20:18:52.042Z+ for ZPOOL in "${ZPOOLS[@]}"
4542023-11-08T20:18:52.071Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4552023-11-08T20:18:52.098Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4562023-11-08T20:18:52.126Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4572023-11-08T20:18:52.156Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4582023-11-08T20:18:52.187Z0+0 records in
4592023-11-08T20:18:52.221Z0+0 records out
4602023-11-08T20:18:52.250Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4612023-11-08T20:18:52.274Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4622023-11-08T20:18:52.299Z+ set +x
4632023-11-08T20:18:52.324Z++ zpool list -o name
4642023-11-08T20:18:52.349Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4652023-11-08T20:18:52.374Z+ [[ -z '' ]]
4662023-11-08T20:18:52.402Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4672023-11-08T20:18:52.429Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4682023-11-08T20:18:52.454Z+ set +x
4692023-11-08T20:18:52.483Z+ for ZPOOL in "${ZPOOLS[@]}"
4702023-11-08T20:18:52.507Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4712023-11-08T20:18:52.531Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4722023-11-08T20:18:52.559Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4732023-11-08T20:18:52.586Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4742023-11-08T20:18:52.612Z0+0 records in
4752023-11-08T20:18:52.642Z0+0 records out
4762023-11-08T20:18:52.723Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4772023-11-08T20:18:52.747Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4782023-11-08T20:18:52.772Z+ set +x
4792023-11-08T20:18:52.797Z++ zpool list -o name
4802023-11-08T20:18:52.822Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4812023-11-08T20:18:52.847Z+ [[ -z '' ]]
4822023-11-08T20:18:52.872Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4832023-11-08T20:18:52.900Zdownloading softnpu ...done
4842023-11-08T20:18:52.927Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4852023-11-08T20:18:52.952Z+ set +x
4862023-11-08T20:18:52.977Z+ [[ zone == \z\o\n\e ]]
4872023-11-08T20:18:53.003Z+ ensure_simulated_links igb0
4882023-11-08T20:18:53.027Z+ local PHYSICAL_LINK=igb0
4892023-11-08T20:18:53.052Z+ INDICES=("0" "1")
4902023-11-08T20:18:53.078Z+ for I in "${INDICES[@]}"
4912023-11-08T20:18:53.104Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
4922023-11-08T20:18:53.142Z++ get_simnet_name_if_exists net0
4932023-11-08T20:18:53.172Z++ dladm show-simnet -p -o LINK net0
4942023-11-08T20:18:53.197Z++ echo ''
4952023-11-08T20:18:53.221Z+ [[ -z '' ]]
4962023-11-08T20:18:53.245Z+ dladm create-simnet -t net0
4972023-11-08T20:18:53.269Z+ dladm create-simnet -t sc0_0
4982023-11-08T20:18:53.294Z+ dladm modify-simnet -t -p net0 sc0_0
4992023-11-08T20:18:53.319Z+ dladm set-linkprop -p mtu=9000 sc0_0
5002023-11-08T20:18:53.343Z+ success 'Simnet net0/sc0_0 exists'
5012023-11-08T20:18:53.368Z+ set +x
5022023-11-08T20:18:53.392Z+ for I in "${INDICES[@]}"
5032023-11-08T20:18:53.418Z++ get_simnet_name_if_exists net1
5042023-11-08T20:18:53.442Z++ dladm show-simnet -p -o LINK net1
5052023-11-08T20:18:53.468Z++ echo ''
5062023-11-08T20:18:53.494Z+ [[ -z '' ]]
5072023-11-08T20:18:53.518Z+ dladm create-simnet -t net1
5082023-11-08T20:18:53.543Z+ dladm create-simnet -t sc1_0
5092023-11-08T20:18:53.567Z+ dladm modify-simnet -t -p net1 sc1_0
5102023-11-08T20:18:53.591Z+ dladm set-linkprop -p mtu=9000 sc1_0
5112023-11-08T20:18:53.617Z+ success 'Simnet net1/sc1_0 exists'
5122023-11-08T20:18:53.642Z+ set +x
5132023-11-08T20:18:53.666Z++ get_vnic_name_if_exists sc0_1
5142023-11-08T20:18:53.693Z++ dladm show-vnic -p -o LINK sc0_1
5152023-11-08T20:18:53.754Z++ echo ''
5162023-11-08T20:18:53.779Z+ [[ -z '' ]]
5172023-11-08T20:18:53.803Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5182023-11-08T20:18:53.829Z+ [[ -v PROMISC_FILT_OFF ]]
5192023-11-08T20:18:53.853Z+ success 'Vnic sc0_1 exists'
5202023-11-08T20:18:53.878Z+ set +x
5212023-11-08T20:18:53.902Z+ warn_if_no_proxy_arp
5222023-11-08T20:18:53.929Z+ [[ -v PXA_START ]]
5232023-11-08T20:18:54.366Z+ [[ -v PXA_END ]]
5242023-11-08T20:18:54.392Z+ ensure_softnpu_zone
5252023-11-08T20:18:54.416Z+ zoneadm list
5262023-11-08T20:18:54.440Z+ grep -q sidecar_softnpu
5272023-11-08T20:18:54.464Z+ [[ -f out/npuzone/npuzone ]]
5282023-11-08T20:18:54.488Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch main
5292023-11-08T20:18:57.442Zdownloading asic_program.so ...done
5302023-11-08T20:18:57.492Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-08T20:19:06.853Zdownloading scadm ...done
5322023-11-08T20:19:06.878Zcreating port tfportrear0_0 for sc0_0
5332023-11-08T20:19:06.902Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-08T20:19:06.927Zsoftnpu zone setup
5352023-11-08T20:19:06.952Zconfigure zone
5362023-11-08T20:19:06.979Zinstall zone
5372023-11-08T20:19:07.946Zboot zone
5382023-11-08T20:19:11.050Z[sidecar_softnpu] svcs milestone/network
5392023-11-08T20:19:11.085ZSTATE STIME FMRI
5402023-11-08T20:19:11.112Zonline 20:19:09 svc:/milestone/network:default
5412023-11-08T20:19:11.136Z[sidecar_softnpu] /softnpu/npu start
5422023-11-08T20:19:11.160Z
5432023-11-08T20:19:11.187Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-08T20:19:11.212Z+ GATEWAY_IP=10.151.2.100
5452023-11-08T20:19:11.237Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-11-08T20:19:11.261ZUsing 10.151.2.100 as gateway ip
5472023-11-08T20:19:11.296Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-08T20:19:11.320Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-11-08T20:19:11.345Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-08T20:19:11.369Z+++ echo 18:c0:4d:0d:a0:2a
5512023-11-08T20:19:11.394Z+ COUNT=1
5522023-11-08T20:19:11.418Z+ [[ 1 -ne 1 ]]
5532023-11-08T20:19:11.443Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-11-08T20:19:11.469ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-11-08T20:19:11.496Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-11-08T20:19:11.522Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-11-08T20:19:11.547Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-08T20:19:11.571Z+ [[ -v PXA_START ]]
5592023-11-08T20:19:11.596Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-11-08T20:19:11.621Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-11-08T20:19:11.645Z+ z_scadm dump-state
5622023-11-08T20:19:11.671Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-08T20:19:11.697Zlocal v6:
5642023-11-08T20:19:11.722Zlocal v4:
5652023-11-08T20:19:11.746Zrouter v6:
5662023-11-08T20:19:11.770Zrouter v4_idx:
5672023-11-08T20:19:11.795Zrouter v4_routes:
5682023-11-08T20:19:11.820Zresolver v4:
5692023-11-08T20:19:11.844Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-11-08T20:19:11.868Zresolver v6:
5712023-11-08T20:19:11.893Znat v4:
5722023-11-08T20:19:11.917Znat_v6:
5732023-11-08T20:19:11.945Zport_mac:
5742023-11-08T20:19:11.992Zproxy arp:
5752023-11-08T20:19:12.018Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-11-08T20:19:12.045Z+ success 'softnpu zone exists'
5772023-11-08T20:19:12.070Z+ set +x
5782023-11-08T20:19:12.095Zsoftnpu zone exists
5792023-11-08T20:19:12.119Z
5802023-11-08T20:19:12.143Zreal 28.458625771
5812023-11-08T20:19:12.168Zuser 0.696498095
5822023-11-08T20:19:12.192Zsys 1.586738620
5832023-11-08T20:19:12.217Ztrap 0.000080258
5842023-11-08T20:19:12.242Ztflt 0.000137297
5852023-11-08T20:19:12.266Zdflt 0.001094464
5862023-11-08T20:19:12.290Zkflt 0.000000000
5872023-11-08T20:19:12.321Zlock 22:14.256916955
5882023-11-08T20:19:12.346Zslp 1:24.706466206
5892023-11-08T20:19:12.370Zlat 0.075368152
5902023-11-08T20:19:12.396Zstop 0.367918770
5912023-11-08T20:19:12.422Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-08T20:19:12.450Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-08T20:19:12.475Z+ SILO_NAME=recovery
5942023-11-08T20:19:12.500Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-08T20:19:12.524Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-08T20:19:12.548Z+ sed -i~ '
5972023-11-08T20:19:12.572Z /^external_dns_ips/c\
5982023-11-08T20:19:12.597Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-11-08T20:19:12.622Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-08T20:19:12.646Z /^first/c\
6012023-11-08T20:19:12.670Zfirst = "10.151.2.164"
6022023-11-08T20:19:12.695Z /^last/c\
6032023-11-08T20:19:12.719Zlast = "10.151.2.173"
6042023-11-08T20:19:12.743Z }
6052023-11-08T20:19:12.767Z /^\[rack_network_config/,/^$/ {
6062023-11-08T20:19:12.791Z /^infra_ip_first/c\
6072023-11-08T20:19:12.815Zinfra_ip_first = "10.151.2.174"
6082023-11-08T20:19:12.839Z /^infra_ip_last/c\
6092023-11-08T20:19:12.863Zinfra_ip_last = "10.151.2.174"
6102023-11-08T20:19:12.888Z }
6112023-11-08T20:19:12.913Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6122023-11-08T20:19:12.936Z /^\[\[rack_network_config.ports/,/^$/ {
6132023-11-08T20:19:12.962Z+++ pkg/config-rss.toml Wed Nov 8 20:19:11 2023
6142023-11-08T20:19:12.987Z /^routes/c\
6152023-11-08T20:19:13.010Z@@ -41,7 +41,7 @@
6162023-11-08T20:19:13.035Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6172023-11-08T20:19:13.060Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6182023-11-08T20:19:13.085Z /^addresses/c\
6192023-11-08T20:19:13.110Z # must be contained in one of the "internal services" IP Pool ranges listed
6202023-11-08T20:19:13.134Zaddresses = \["10.151.2.174/32"\]
6212023-11-08T20:19:13.159Z # below.
6222023-11-08T20:19:13.183Z }
6232023-11-08T20:19:13.208Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6242023-11-08T20:19:13.232Z' pkg/config-rss.toml
6252023-11-08T20:19:13.263Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6262023-11-08T20:19:13.291Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6272023-11-08T20:19:13.315Z
6282023-11-08T20:19:13.346Z+ true
6292023-11-08T20:19:13.870Z # Initial TLS certificates for the external API
6302023-11-08T20:19:13.907Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6312023-11-08T20:19:13.934Z #
6322023-11-08T20:19:13.961Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6332023-11-08T20:19:14.389Z@@ -77,8 +77,8 @@
6342023-11-08T20:19:14.414Z+ TLS_NAME=recovery.sys.oxide.test
6352023-11-08T20:19:14.438Z #
6362023-11-08T20:19:14.463Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6372023-11-08T20:19:14.487Z # For more on this and what to put here, see docs/how-to-run.adoc.
6382023-11-08T20:19:14.512Z....+....+.....+.+..+......+.+.....+...+.+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.+.........+...+........+...+....+......+.....+.+.....+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+...+......+...+......+.+.........+..+............+...+...+.+.........+..............+..........+......+.....+.......+..+.......+.....+...+...............+...+....+..............+............+............+...+....+.........+.........+...........+.+.....+....+...............+............+..+.......+.....+.....................+....+......................................+.......+......+........+...............+....+...+.....+.......+...+.....+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6392023-11-08T20:19:14.536Z [[internal_services_ip_pool_ranges]]
6402023-11-08T20:19:14.560Z-first = "192.168.1.20"
6412023-11-08T20:19:14.584Z-last = "192.168.1.29"
6422023-11-08T20:19:14.608Z+first = "10.151.2.164"
6432023-11-08T20:19:14.632Z.....+....+...+..+....+............+..+.+.....+.........+.......+.........+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+.........+...+........+...............+.............+.....+.......+..+...+..........+.........+...+...+...+........................+.....+...+....+...............+......+..+......+......+...+..................+....+......+.....+...+.......+...+..+.......+.....+...+...+........................+.+.....+......+...+...+...................+...+..+...............+.+..+.+.....................+......+...+..+....+.....+................+........+......+.+..+......+..........+.........+.....................+...............+...............+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6442023-11-08T20:19:14.657Z+last = "10.151.2.173"
6452023-11-08T20:19:14.681Z-----
6462023-11-08T20:19:14.706Z
6472023-11-08T20:19:14.730Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6482023-11-08T20:19:14.754Z # TODO - this configuration is subject to change going forward. Ultimately these
6492023-11-08T20:19:14.778Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6502023-11-08T20:19:14.802Z # parameters should be provided to the control plane via wicket, but we need to
6512023-11-08T20:19:14.831Z+ rmdir pkg
6522023-11-08T20:19:14.859Z@@ -92,8 +92,8 @@
6532023-11-08T20:19:14.884Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6542023-11-08T20:19:14.908Z # A range of IP addresses used by Boundary Services on the external network. In
6552023-11-08T20:19:14.932Z+ pfexec mkdir -p /usr/oxide
6562023-11-08T20:19:14.957Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6572023-11-08T20:19:14.981Z+ pfexec rm -f /usr/oxide/tcpproxy
6582023-11-08T20:19:15.006Z # With softnpu, only one address is used.
6592023-11-08T20:19:15.032Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6602023-11-08T20:19:15.059Z-infra_ip_first = "192.168.1.30"
6612023-11-08T20:19:15.092Z+ pfexec chmod +x /usr/oxide/tcpproxy
6622023-11-08T20:19:15.116Z-infra_ip_last = "192.168.1.30"
6632023-11-08T20:19:15.141Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6642023-11-08T20:19:15.166Z+infra_ip_first = "10.151.2.174"
6652023-11-08T20:19:15.191Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6662023-11-08T20:19:15.216Z+infra_ip_last = "10.151.2.174"
6672023-11-08T20:19:15.241Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6682023-11-08T20:19:15.266Z
6692023-11-08T20:19:15.290Z+ OMICRON_NO_UNINSTALL=1
6702023-11-08T20:19:15.314Z # Configurations for BGP routers to run on the scrimlets.
6712023-11-08T20:19:15.338Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6722023-11-08T20:19:15.364Z bgp = []
6732023-11-08T20:19:15.401ZNov 08 20:19:12.929 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6742023-11-08T20:19:15.426Z@@ -101,9 +101,9 @@
6752023-11-08T20:19:15.451ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6762023-11-08T20:19:15.476Z # You can configure multiple uplinks by repeating the following stanza
6772023-11-08T20:19:15.501ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6782023-11-08T20:19:15.525Z [[rack_network_config.ports]]
6792023-11-08T20:19:15.549ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6802023-11-08T20:19:15.574Z # Routes associated with this port.
6812023-11-08T20:19:15.598ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6822023-11-08T20:19:15.623Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6832023-11-08T20:19:15.648ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6842023-11-08T20:19:15.672Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6852023-11-08T20:19:15.698ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6862023-11-08T20:19:15.725Z # Addresses associated with this port.
6872023-11-08T20:19:15.749ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6882023-11-08T20:19:15.773Z-addresses = ["192.168.1.30/32"]
6892023-11-08T20:19:15.798ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6902023-11-08T20:19:15.823Z+addresses = ["10.151.2.174/32"]
6912023-11-08T20:19:15.855ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6922023-11-08T20:19:15.930Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6932023-11-08T20:19:15.955ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6942023-11-08T20:19:15.979Z port = "qsfp0"
6952023-11-08T20:19:16.003ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6962023-11-08T20:19:16.028Z # The speed of this port.
6972023-11-08T20:19:16.052ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6982023-11-08T20:19:16.077Za pkg/config-rss.toml 7K
6992023-11-08T20:19:16.103ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7002023-11-08T20:19:16.127Za pkg/initial-tls-cert.pem 2K
7012023-11-08T20:19:16.153ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7022023-11-08T20:19:16.179Za pkg/initial-tls-key.pem 4K
7032023-11-08T20:19:16.203ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7042023-11-08T20:19:16.228ZNov 08 20:19:12.931 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7052023-11-08T20:19:16.252ZNov 08 20:19:13.162 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-11-08T20:19:16.279ZNov 08 20:19:13.198 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-08T20:19:16.316ZNov 08 20:19:13.265 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-08T20:19:16.345Z
7092023-11-08T20:19:16.388Zreal 0.475170584
7102023-11-08T20:19:16.440Zuser 0.098895227
7112023-11-08T20:19:16.466Zsys 1.837669106
7122023-11-08T20:19:16.490Ztrap 0.000000400
7132023-11-08T20:19:16.515Ztflt 0.000037560
7142023-11-08T20:19:16.539Zdflt 0.000342778
7152023-11-08T20:19:16.564Zkflt 0.000000000
7162023-11-08T20:19:16.588Zlock 42.706284461
7172023-11-08T20:19:16.613Zslp 0.688188846
7182023-11-08T20:19:16.638Zlat 0.002860302
7192023-11-08T20:19:16.662Zstop 0.220111582
7202023-11-08T20:19:16.686Z+ retry=0
7212023-11-08T20:19:16.710Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-08T20:19:16.734Z+ [[ 0 -gt 30 ]]
7232023-11-08T20:19:16.758Z+ sleep 1
7242023-11-08T20:19:16.782Z+ retry=1
7252023-11-08T20:19:16.807Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-08T20:19:16.832Z+ [[ 1 -gt 30 ]]
7272023-11-08T20:19:16.863Z+ sleep 1
7282023-11-08T20:19:16.891Z+ retry=2
7292023-11-08T20:19:16.915Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-08T20:19:40.151Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7312023-11-08T20:19:40.180Zlocal v6:
7322023-11-08T20:19:40.209Zfd00:99::1
7332023-11-08T20:19:40.235Zfe80::e80d:d0ff:fe56:386b
7342023-11-08T20:19:40.259Zlocal v4:
7352023-11-08T20:19:40.284Z10.151.2.174
7362023-11-08T20:19:40.308Zrouter v6:
7372023-11-08T20:19:40.333Zfd00:1122:3344:101::/64 -> fe80::947b:62ff:fe2c:9fae (1)
7382023-11-08T20:19:40.357Zfdb0:18c0:4d0d:a02a::/64 -> fe80::947b:62ff:fe2c:9fae (1)
7392023-11-08T20:19:40.381Zrouter v4_idx:
7402023-11-08T20:19:40.406Z0.0.0.0/0 -> 0
7412023-11-08T20:19:40.430Zrouter v4_routes:
7422023-11-08T20:19:40.455Z0 -> 10.151.2.100 (2)
7432023-11-08T20:19:40.479Zresolver v4:
7442023-11-08T20:19:40.504Z10.151.2.174 -> 7e:7b:71:45:89:43
7452023-11-08T20:19:40.533Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7462023-11-08T20:19:40.558Zresolver v6:
7472023-11-08T20:19:40.583Zfe80::e80d:d0ff:fe56:386b -> ea:0d:d0:56:38:6b
7482023-11-08T20:19:40.607Zfe80::947b:62ff:fe2c:9fae -> 96:7b:62:2c:9f:ae
7492023-11-08T20:19:40.632Znat v4:
7502023-11-08T20:19:40.660Znat_v6:
7512023-11-08T20:19:40.686Zport_mac:
7522023-11-08T20:19:40.710Z1000: a8:40:25:48:75:db
7532023-11-08T20:19:40.737Z1: a8:40:25:48:75:dc
7542023-11-08T20:19:40.763Zproxy arp:
7552023-11-08T20:19:40.787Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7562023-11-08T20:19:40.811Z+ retry=0
7572023-11-08T20:19:40.836Z+++ zoneadm list -n
7582023-11-08T20:19:40.862Z+++ grep oxz_ntp
7592023-11-08T20:19:40.886Z++ pfexec svcs -z -Hostate oxide/ntp
7602023-11-08T20:19:40.955Zsvcs: zone '-Hostate': entity not found
7612023-11-08T20:19:40.981Z++ true
7622023-11-08T20:19:41.005Z+ [[ '' != online ]]
7632023-11-08T20:19:41.030Z+ [[ 0 -gt 60 ]]
7642023-11-08T20:19:41.057Z+ sleep 1
7652023-11-08T20:19:41.206Z+ retry=1
7662023-11-08T20:19:41.231Z+++ zoneadm list -n
7672023-11-08T20:19:41.255Z+++ grep oxz_ntp
7682023-11-08T20:19:41.279Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-11-08T20:19:41.303Zsvcs: zone '-Hostate': entity not found
7702023-11-08T20:19:41.329Z++ true
7712023-11-08T20:19:41.353Z+ [[ '' != online ]]
7722023-11-08T20:19:41.377Z+ [[ 1 -gt 60 ]]
7732023-11-08T20:19:41.402Z+ sleep 1
7742023-11-08T20:19:42.228Z+ retry=2
7752023-11-08T20:19:42.253Z+++ zoneadm list -n
7762023-11-08T20:19:42.277Z+++ grep oxz_ntp
7772023-11-08T20:19:42.302Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-11-08T20:19:42.327Zsvcs: zone '-Hostate': entity not found
7792023-11-08T20:19:42.351Z++ true
7802023-11-08T20:19:42.375Z+ [[ '' != online ]]
7812023-11-08T20:19:42.400Z+ [[ 2 -gt 60 ]]
7822023-11-08T20:19:42.430Z+ sleep 1
7832023-11-08T20:19:43.250Z+ retry=3
7842023-11-08T20:19:43.274Z+++ zoneadm list -n
7852023-11-08T20:19:43.298Z+++ grep oxz_ntp
7862023-11-08T20:19:43.323Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-11-08T20:19:43.348Zsvcs: zone '-Hostate': entity not found
7882023-11-08T20:19:43.375Z++ true
7892023-11-08T20:19:43.400Z+ [[ '' != online ]]
7902023-11-08T20:19:43.425Z+ [[ 3 -gt 60 ]]
7912023-11-08T20:19:43.449Z+ sleep 1
7922023-11-08T20:19:44.286Z+ retry=4
7932023-11-08T20:19:44.318Z+++ zoneadm list -n
7942023-11-08T20:19:44.343Z+++ grep oxz_ntp
7952023-11-08T20:19:44.371Z++ pfexec svcs -z -Hostate oxide/ntp
7962023-11-08T20:19:44.395Zsvcs: zone '-Hostate': entity not found
7972023-11-08T20:19:44.421Z++ true
7982023-11-08T20:19:44.447Z+ [[ '' != online ]]
7992023-11-08T20:19:44.473Z+ [[ 4 -gt 60 ]]
8002023-11-08T20:19:44.499Z+ sleep 1
8012023-11-08T20:19:45.296Z+ retry=5
8022023-11-08T20:19:45.320Z+++ zoneadm list -n
8032023-11-08T20:19:45.345Z+++ grep oxz_ntp
8042023-11-08T20:19:45.369Z++ pfexec svcs -z -Hostate oxide/ntp
8052023-11-08T20:19:45.393Zsvcs: zone '-Hostate': entity not found
8062023-11-08T20:19:45.418Z++ true
8072023-11-08T20:19:45.442Z+ [[ '' != online ]]
8082023-11-08T20:19:45.466Z+ [[ 5 -gt 60 ]]
8092023-11-08T20:19:45.491Z+ sleep 1
8102023-11-08T20:19:46.317Z+ retry=6
8112023-11-08T20:19:46.341Z+++ zoneadm list -n
8122023-11-08T20:19:46.365Z+++ grep oxz_ntp
8132023-11-08T20:19:46.389Z++ pfexec svcs -z -Hostate oxide/ntp
8142023-11-08T20:19:46.414Zsvcs: zone '-Hostate': entity not found
8152023-11-08T20:19:46.438Z++ true
8162023-11-08T20:19:46.462Z+ [[ '' != online ]]
8172023-11-08T20:19:46.487Z+ [[ 6 -gt 60 ]]
8182023-11-08T20:19:46.512Z+ sleep 1
8192023-11-08T20:19:47.338Z+ retry=7
8202023-11-08T20:19:47.363Z+++ zoneadm list -n
8212023-11-08T20:19:47.388Z+++ grep oxz_ntp
8222023-11-08T20:19:47.412Z++ pfexec svcs -z -Hostate oxide/ntp
8232023-11-08T20:19:47.438Zsvcs: zone '-Hostate': entity not found
8242023-11-08T20:19:47.462Z++ true
8252023-11-08T20:19:47.486Z+ [[ '' != online ]]
8262023-11-08T20:19:47.519Z+ [[ 7 -gt 60 ]]
8272023-11-08T20:19:47.545Z+ sleep 1
8282023-11-08T20:19:48.359Z+ retry=8
8292023-11-08T20:19:48.384Z+++ zoneadm list -n
8302023-11-08T20:19:48.408Z+++ grep oxz_ntp
8312023-11-08T20:19:48.432Z++ pfexec svcs -z -Hostate oxide/ntp
8322023-11-08T20:19:48.456Zsvcs: zone '-Hostate': entity not found
8332023-11-08T20:19:48.481Z++ true
8342023-11-08T20:19:48.507Z+ [[ '' != online ]]
8352023-11-08T20:19:48.531Z+ [[ 8 -gt 60 ]]
8362023-11-08T20:19:48.556Z+ sleep 1
8372023-11-08T20:19:49.397Z+ retry=9
8382023-11-08T20:19:49.424Z+++ zoneadm list -n
8392023-11-08T20:19:49.451Z+++ grep oxz_ntp
8402023-11-08T20:19:49.476Z++ pfexec svcs -z -Hostate oxide/ntp
8412023-11-08T20:19:49.500Zsvcs: zone '-Hostate': entity not found
8422023-11-08T20:19:49.525Z++ true
8432023-11-08T20:19:49.550Z+ [[ '' != online ]]
8442023-11-08T20:19:49.574Z+ [[ 9 -gt 60 ]]
8452023-11-08T20:19:49.599Z+ sleep 1
8462023-11-08T20:19:50.815Z+ retry=10
8472023-11-08T20:19:50.844Z+++ zoneadm list -n
8482023-11-08T20:19:50.883Z+++ grep oxz_ntp
8492023-11-08T20:19:50.909Z++ pfexec svcs -z oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b -Hostate oxide/ntp
8502023-11-08T20:19:50.935Zsvcs: Could not bind to repository server for zone oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b: repository server unavailable
8512023-11-08T20:19:50.964Z++ true
8522023-11-08T20:19:50.990Z+ [[ '' != online ]]
8532023-11-08T20:19:51.015Z+ [[ 10 -gt 60 ]]
8542023-11-08T20:19:51.040Z+ sleep 1
8552023-11-08T20:19:51.425Z+ retry=11
8562023-11-08T20:19:51.487Z+++ zoneadm list -n
8572023-11-08T20:19:51.512Z+++ grep oxz_ntp
8582023-11-08T20:19:51.537Z++ pfexec svcs -z oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b -Hostate oxide/ntp
8592023-11-08T20:19:51.562Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8602023-11-08T20:19:51.587Z++ true
8612023-11-08T20:19:51.612Z+ [[ '' != online ]]
8622023-11-08T20:19:51.638Z+ [[ 11 -gt 60 ]]
8632023-11-08T20:19:51.664Z+ sleep 1
8642023-11-08T20:19:52.452Z+ retry=12
8652023-11-08T20:19:52.477Z+++ zoneadm list -n
8662023-11-08T20:19:52.529Z+++ grep oxz_ntp
8672023-11-08T20:19:52.553Z++ pfexec svcs -z oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b -Hostate oxide/ntp
8682023-11-08T20:19:52.578Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8692023-11-08T20:19:52.606Z++ true
8702023-11-08T20:19:52.632Z+ [[ '' != online ]]
8712023-11-08T20:19:52.657Z+ [[ 12 -gt 60 ]]
8722023-11-08T20:19:52.682Z+ sleep 1
8732023-11-08T20:19:53.480Z+ retry=13
8742023-11-08T20:19:53.504Z+++ zoneadm list -n
8752023-11-08T20:19:53.529Z+++ grep oxz_ntp
8762023-11-08T20:19:53.567Z++ pfexec svcs -z oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b -Hostate oxide/ntp
8772023-11-08T20:19:53.592Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8782023-11-08T20:19:53.616Z++ true
8792023-11-08T20:19:53.646Z+ [[ '' != online ]]
8802023-11-08T20:19:53.675Z+ [[ 13 -gt 60 ]]
8812023-11-08T20:19:53.707Z+ sleep 1
8822023-11-08T20:19:54.509Z+ retry=14
8832023-11-08T20:19:54.536Z+++ zoneadm list -n
8842023-11-08T20:19:54.562Z+++ grep oxz_ntp
8852023-11-08T20:19:54.617Z++ pfexec svcs -z oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b -Hostate oxide/ntp
8862023-11-08T20:19:54.648Z+ [[ disabled != online ]]
8872023-11-08T20:19:54.673Z+ [[ 14 -gt 60 ]]
8882023-11-08T20:19:54.697Z+ sleep 1
8892023-11-08T20:19:55.530Z+ retry=15
8902023-11-08T20:19:55.555Z+++ zoneadm list -n
8912023-11-08T20:19:55.579Z+++ grep oxz_ntp
8922023-11-08T20:19:55.605Z++ pfexec svcs -z oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b -Hostate oxide/ntp
8932023-11-08T20:19:55.658Z+ [[ disabled != online ]]
8942023-11-08T20:19:55.684Z+ [[ 15 -gt 60 ]]
8952023-11-08T20:19:55.713Z+ sleep 1
8962023-11-08T20:19:56.552Z+ retry=16
8972023-11-08T20:19:56.577Z+++ zoneadm list -n
8982023-11-08T20:19:56.601Z+++ grep oxz_ntp
8992023-11-08T20:19:56.625Z++ pfexec svcs -z oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b -Hostate oxide/ntp
9002023-11-08T20:19:56.649Z+ [[ disabled != online ]]
9012023-11-08T20:19:56.709Z+ [[ 16 -gt 60 ]]
9022023-11-08T20:19:56.733Z+ sleep 1
9032023-11-08T20:19:57.575Z+ retry=17
9042023-11-08T20:19:57.604Z+++ zoneadm list -n
9052023-11-08T20:19:57.631Z+++ grep oxz_ntp
9062023-11-08T20:19:57.668Z++ pfexec svcs -z oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b -Hostate oxide/ntp
9072023-11-08T20:19:57.697Z+ [[ disabled != online ]]
9082023-11-08T20:19:57.742Z+ [[ 17 -gt 60 ]]
9092023-11-08T20:19:57.796Z+ sleep 1
9102023-11-08T20:19:58.598Z+ retry=18
9112023-11-08T20:19:58.623Z+++ zoneadm list -n
9122023-11-08T20:19:58.651Z+++ grep oxz_ntp
9132023-11-08T20:19:58.676Z++ pfexec svcs -z oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b -Hostate oxide/ntp
9142023-11-08T20:19:58.702Z+ [[ disabled != online ]]
9152023-11-08T20:19:58.729Z+ [[ 18 -gt 60 ]]
9162023-11-08T20:19:58.770Z+ sleep 1
9172023-11-08T20:19:59.621Z+ retry=19
9182023-11-08T20:19:59.646Z+++ zoneadm list -n
9192023-11-08T20:19:59.670Z+++ grep oxz_ntp
9202023-11-08T20:19:59.695Z++ pfexec svcs -z oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b -Hostate oxide/ntp
9212023-11-08T20:19:59.731Z+ [[ disabled != online ]]
9222023-11-08T20:20:00.198Z+ [[ 19 -gt 60 ]]
9232023-11-08T20:20:00.224Z+ sleep 1
9242023-11-08T20:20:00.643Z+ retry=20
9252023-11-08T20:20:00.669Z+++ zoneadm list -n
9262023-11-08T20:20:00.704Z+++ grep oxz_ntp
9272023-11-08T20:20:00.734Z++ pfexec svcs -z oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b -Hostate oxide/ntp
9282023-11-08T20:20:00.758Z+ [[ disabled != online ]]
9292023-11-08T20:20:00.784Z+ [[ 20 -gt 60 ]]
9302023-11-08T20:20:00.812Z+ sleep 1
9312023-11-08T20:20:01.665Z+ retry=21
9322023-11-08T20:20:01.689Z+++ zoneadm list -n
9332023-11-08T20:20:01.714Z+++ grep oxz_ntp
9342023-11-08T20:20:01.739Z++ pfexec svcs -z oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b -Hostate oxide/ntp
9352023-11-08T20:20:01.764Z+ [[ disabled != online ]]
9362023-11-08T20:20:01.789Z+ [[ 21 -gt 60 ]]
9372023-11-08T20:20:01.814Z+ sleep 1
 
9382023-11-08T20:20:02.030Zsock console login: Nov 8 20:20:01 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9392023-11-08T20:20:02.687Z+ retry=22
9402023-11-08T20:20:02.712Z+++ zoneadm list -n
9412023-11-08T20:20:02.744Z+++ grep oxz_ntp
9422023-11-08T20:20:02.772Z++ pfexec svcs -z oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b -Hostate oxide/ntp
9432023-11-08T20:20:02.801Z+ [[ online != online ]]
9442023-11-08T20:20:02.832Z+ echo 'Waited for chrony: 22s'
9452023-11-08T20:20:02.859ZWaited for chrony: 22s
9462023-11-08T20:20:02.884Z+ export RUST_BACKTRACE=1
9472023-11-08T20:20:02.909Z+ RUST_BACKTRACE=1
9482023-11-08T20:20:02.934Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9492023-11-08T20:20:02.960Z+ ./tests/bootstrap
9502023-11-08T20:20:02.985ZUsing external DNS server at 10.151.2.164:53
9512023-11-08T20:20:03.013Z2023-11-08 20:20:03.186597651 UTC: attempting to log into API
9522023-11-08T20:20:17.774Z2023-11-08 20:20:17.785887063 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9532023-11-08T20:20:18.775Z2023-11-08 20:20:18.784650876 UTC: attempting to log into API
9542023-11-08T20:20:33.813Z2023-11-08 20:20:33.801211765 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9552023-11-08T20:20:34.812Z2023-11-08 20:20:34.802383142 UTC: attempting to log into API
9562023-11-08T20:20:49.852Z2023-11-08 20:20:49.843828144 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9572023-11-08T20:20:50.852Z2023-11-08 20:20:50.844979360 UTC: attempting to log into API
9582023-11-08T20:21:05.886Z2023-11-08 20:21:05.878682254 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9592023-11-08T20:21:06.889Z2023-11-08 20:21:06.879402051 UTC: attempting to log into API
 
9602023-11-08T20:21:17.593ZNov 8 20:21:16 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9612023-11-08T20:21:21.943Z2023-11-08 20:21:21.930216571 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9622023-11-08T20:21:22.976Z2023-11-08 20:21:22.931345387 UTC: attempting to log into API
 
9632023-11-08T20:21:25.437ZNov 8 20:21:24 sok ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9642023-11-08T20:21:33.363ZNov 8 20:21:32 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9652023-11-08T20:21:37.972Z2023-11-08 20:21:37.964898811 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9662023-11-08T20:21:38.910ZNov 8 20:21:38 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9672023-11-08T20:21:38.973Z2023-11-08 20:21:38.965976226 UTC: attempting to log into API
 
9682023-11-08T20:21:44.991ZNov 8 20:21:44 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9692023-11-08T20:21:54.007Z2023-11-08 20:21:53.999467757 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9702023-11-08T20:21:55.008Z2023-11-08 20:21:55.000571913 UTC: attempting to log into API
9712023-11-08T20:22:10.041Z2023-11-08 20:22:10.033732926 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9722023-11-08T20:22:11.042Z2023-11-08 20:22:11.034849372 UTC: attempting to log into API
9732023-11-08T20:22:21.083Z2023-11-08 20:22:21.075428050 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: tcp connect error: Connection refused (os error 146)
9742023-11-08T20:22:22.084Z2023-11-08 20:22:22.075823401 UTC: attempting to log into API
9752023-11-08T20:22:22.871Z2023-11-08 20:22:22.858904621 UTC: login succeeded
9762023-11-08T20:22:22.901Zwaiting for nexus to come up...
9772023-11-08T20:22:22.965Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9782023-11-08T20:22:23.020Zensuring datasets are ready...
9792023-11-08T20:22:25.440Zlet's roll.
9802023-11-08T20:22:25.467Z+ rm ./tests/bootstrap
9812023-11-08T20:22:25.948Z+ for test_bin in tests/*
9822023-11-08T20:22:25.972Z+ ./tests/bootstrap-95f6f69deb9d73aa
9832023-11-08T20:22:26.003Z
9842023-11-08T20:22:26.029Zrunning 0 tests
9852023-11-08T20:22:26.079Z
9862023-11-08T20:22:26.103Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9872023-11-08T20:22:26.127Z
9882023-11-08T20:22:26.151Z+ for test_bin in tests/*
9892023-11-08T20:22:26.174Z+ ./tests/end_to_end_tests-cc026c1de0ba49ab
9902023-11-08T20:22:26.198Z
9912023-11-08T20:22:26.221Zrunning 2 tests
9922023-11-08T20:22:26.245Ztest no_spoof::no_spoof_here ... ok
9932023-11-08T20:23:25.563Ztest instance_launch::instance_launch has been running for over 60 seconds
9942023-11-08T20:25:17.147Ztest instance_launch::instance_launch ... ok
9952023-11-08T20:25:17.173Z
9962023-11-08T20:25:17.199Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 171.57s
9972023-11-08T20:25:17.226Z
9982023-11-08T20:25:17.252Z+ _exit_trap
9992023-11-08T20:25:17.281Z+ local status=0
10002023-11-08T20:25:17.312Z+ [[ 0 -eq 0 ]]
10012023-11-08T20:25:17.345Z+ exit 0
10022023-11-08T20:25:17.372Zprocess exited: duration 413220 ms, exit code 0
 
10032023-11-08T20:25:17.459Zfound 47 output files
10042023-11-08T20:25:17.510Zuploading: /var/svc/log/oxide-sled-agent:default.log (577981 bytes)
10052023-11-08T20:25:22.343Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 577981 -> 579232
10062023-11-08T20:25:23.416Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10072023-11-08T20:25:23.524Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_357c1017-e59c-44d0-8e0e-9bcca83f6576/root/var/svc/log/oxide-crucible-agent:default.log (14148 bytes)
10082023-11-08T20:25:25.190Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_357c1017-e59c-44d0-8e0e-9bcca83f6576/root/var/svc/log/oxide-crucible-agent:default.log
10092023-11-08T20:25:26.361Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_357c1017-e59c-44d0-8e0e-9bcca83f6576/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10102023-11-08T20:25:31.636Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_357c1017-e59c-44d0-8e0e-9bcca83f6576/root/var/svc/log/oxide-crucible-downstairs:default.log
10112023-11-08T20:25:55.120Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_357c1017-e59c-44d0-8e0e-9bcca83f6576/root/var/svc/log/oxide-crucible-downstairs:downstairs-55beece4-03e9-422d-892c-b224ddfae43d.log (5537 bytes)
10122023-11-08T20:25:56.908Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_357c1017-e59c-44d0-8e0e-9bcca83f6576/root/var/svc/log/oxide-crucible-downstairs:downstairs-55beece4-03e9-422d-892c-b224ddfae43d.log
10132023-11-08T20:25:58.237Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_908ffef8-216f-42b4-95b0-7c918bd67a1a/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
10142023-11-08T20:25:58.305Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_908ffef8-216f-42b4-95b0-7c918bd67a1a/root/var/svc/log/oxide-cockroachdb:default.log
10152023-11-08T20:25:58.386Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a75df259-93eb-4d42-b0af-6d2cd65ffb66/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
10162023-11-08T20:25:59.259Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a75df259-93eb-4d42-b0af-6d2cd65ffb66/root/var/svc/log/oxide-crucible-agent:default.log
10172023-11-08T20:25:59.304Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a75df259-93eb-4d42-b0af-6d2cd65ffb66/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10182023-11-08T20:26:00.461Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a75df259-93eb-4d42-b0af-6d2cd65ffb66/root/var/svc/log/oxide-crucible-downstairs:default.log
10192023-11-08T20:26:00.508Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a75df259-93eb-4d42-b0af-6d2cd65ffb66/root/var/svc/log/oxide-crucible-downstairs:downstairs-43db95a9-4658-4544-8d6c-1b34105a5bc3.log (3051 bytes)
10202023-11-08T20:26:01.617Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a75df259-93eb-4d42-b0af-6d2cd65ffb66/root/var/svc/log/oxide-crucible-downstairs:downstairs-43db95a9-4658-4544-8d6c-1b34105a5bc3.log
10212023-11-08T20:26:01.648Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_99938ce4-affc-481a-be09-c9048c593388/root/var/svc/log/oxide-internal_dns:default.log (99997 bytes)
10222023-11-08T20:26:02.092Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_99938ce4-affc-481a-be09-c9048c593388/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99997 -> 103751
10232023-11-08T20:26:03.210Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_99938ce4-affc-481a-be09-c9048c593388/root/var/svc/log/oxide-internal_dns:default.log
10242023-11-08T20:26:03.247Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f912c231-90c2-4c28-89a8-a2dc9760e1b7/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10252023-11-08T20:26:04.379Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f912c231-90c2-4c28-89a8-a2dc9760e1b7/root/var/svc/log/oxide-cockroachdb:default.log
10262023-11-08T20:26:04.412Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_38e91c99-a817-44a1-9d3f-a32b34620728/root/var/svc/log/oxide-external_dns:default.log (15610 bytes)
10272023-11-08T20:26:04.509Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_38e91c99-a817-44a1-9d3f-a32b34620728/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15610 -> 19386
10282023-11-08T20:26:05.620Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_38e91c99-a817-44a1-9d3f-a32b34620728/root/var/svc/log/oxide-external_dns:default.log
10292023-11-08T20:26:05.656Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_897d469d-e137-43a8-badb-6b99ab037748/root/var/svc/log/oxide-clickhouse:default.log (766559 bytes)
10302023-11-08T20:26:06.804Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_897d469d-e137-43a8-badb-6b99ab037748/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 766559 -> 927365
10312023-11-08T20:26:07.899Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_897d469d-e137-43a8-badb-6b99ab037748/root/var/svc/log/oxide-clickhouse:default.log
10322023-11-08T20:26:08.432Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_550b704d-6d4c-4093-8297-20488163906a/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10332023-11-08T20:26:09.443Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_550b704d-6d4c-4093-8297-20488163906a/root/var/svc/log/oxide-crucible-agent:default.log
10342023-11-08T20:26:09.489Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_550b704d-6d4c-4093-8297-20488163906a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10352023-11-08T20:26:11.056Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_550b704d-6d4c-4093-8297-20488163906a/root/var/svc/log/oxide-crucible-downstairs:default.log
10362023-11-08T20:26:11.094Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_63b242a9-cc32-4fd1-a7a8-e793e60e3266/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10372023-11-08T20:26:12.170Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_63b242a9-cc32-4fd1-a7a8-e793e60e3266/root/var/svc/log/oxide-cockroachdb:default.log
10382023-11-08T20:26:12.209Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_28e41597-d1ca-41d0-9c52-3d5d2e1163da/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10392023-11-08T20:26:13.765Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_28e41597-d1ca-41d0-9c52-3d5d2e1163da/root/var/svc/log/oxide-crucible-pantry:default.log
10402023-11-08T20:26:13.802Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_decbed82-0fd2-4928-a26f-32ba013ec72a/root/var/svc/log/oxide-external_dns:default.log (15614 bytes)
10412023-11-08T20:26:13.895Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_decbed82-0fd2-4928-a26f-32ba013ec72a/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15614 -> 19388
10422023-11-08T20:26:14.968Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_decbed82-0fd2-4928-a26f-32ba013ec72a/root/var/svc/log/oxide-external_dns:default.log
10432023-11-08T20:26:14.993Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_d8b38e89-83b1-4e80-bb97-395629635517/root/var/svc/log/oxide-nexus:default.log (514396 bytes)
10442023-11-08T20:26:16.534Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_d8b38e89-83b1-4e80-bb97-395629635517/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 514396 -> 576223
10452023-11-08T20:26:17.620Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_d8b38e89-83b1-4e80-bb97-395629635517/root/var/svc/log/oxide-nexus:default.log
10462023-11-08T20:26:17.655Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b142a81-e9ec-4aed-a201-8276cf4736f3/root/var/svc/log/oxide-crucible-agent:default.log (14580 bytes)
10472023-11-08T20:26:18.786Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b142a81-e9ec-4aed-a201-8276cf4736f3/root/var/svc/log/oxide-crucible-agent:default.log
10482023-11-08T20:26:18.812Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b142a81-e9ec-4aed-a201-8276cf4736f3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10492023-11-08T20:26:19.890Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b142a81-e9ec-4aed-a201-8276cf4736f3/root/var/svc/log/oxide-crucible-downstairs:default.log
10502023-11-08T20:26:19.915Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b142a81-e9ec-4aed-a201-8276cf4736f3/root/var/svc/log/oxide-crucible-downstairs:downstairs-59487200-45e6-48fc-9d37-d685661de6aa.log (5532 bytes)
10512023-11-08T20:26:21.410Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b142a81-e9ec-4aed-a201-8276cf4736f3/root/var/svc/log/oxide-crucible-downstairs:downstairs-59487200-45e6-48fc-9d37-d685661de6aa.log
10522023-11-08T20:26:21.438Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_9146fedd-a20c-495d-8b1a-9dc53b1a0c34/root/var/svc/log/oxide-internal_dns:default.log (107152 bytes)
10532023-11-08T20:26:21.739Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_9146fedd-a20c-495d-8b1a-9dc53b1a0c34/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 107152 -> 110906
10542023-11-08T20:26:23.223Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_9146fedd-a20c-495d-8b1a-9dc53b1a0c34/root/var/svc/log/oxide-internal_dns:default.log
10552023-11-08T20:26:23.262Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_407a6d9f-b239-4103-95d8-024c63bb53cd/root/var/svc/log/oxide-nexus:default.log (4673838 bytes)
10562023-11-08T20:26:28.058Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_407a6d9f-b239-4103-95d8-024c63bb53cd/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4673838 -> 4736280
10572023-11-08T20:26:29.157Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_407a6d9f-b239-4103-95d8-024c63bb53cd/root/var/svc/log/oxide-nexus:default.log
10582023-11-08T20:26:29.183Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_182c3019-8408-4cd8-8735-5df29c24ca36/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
10592023-11-08T20:26:30.398Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_182c3019-8408-4cd8-8735-5df29c24ca36/root/var/svc/log/oxide-cockroachdb:default.log
10602023-11-08T20:26:30.424Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8b2c2178-c397-43b2-87ec-0a0911fe8af8/root/var/svc/log/oxide-crucible-agent:default.log (13684 bytes)
10612023-11-08T20:26:32.087Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8b2c2178-c397-43b2-87ec-0a0911fe8af8/root/var/svc/log/oxide-crucible-agent:default.log
10622023-11-08T20:26:32.111Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8b2c2178-c397-43b2-87ec-0a0911fe8af8/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10632023-11-08T20:26:33.167Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8b2c2178-c397-43b2-87ec-0a0911fe8af8/root/var/svc/log/oxide-crucible-downstairs:default.log
10642023-11-08T20:26:33.665Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8b2c2178-c397-43b2-87ec-0a0911fe8af8/root/var/svc/log/oxide-crucible-downstairs:downstairs-377e6d52-9f89-4e21-897d-8b6dfc6ba060.log (3051 bytes)
10652023-11-08T20:26:34.750Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_8b2c2178-c397-43b2-87ec-0a0911fe8af8/root/var/svc/log/oxide-crucible-downstairs:downstairs-377e6d52-9f89-4e21-897d-8b6dfc6ba060.log
10662023-11-08T20:26:34.782Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_c7c4a254-0e85-4b65-a95a-da6d5fefb4be/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10672023-11-08T20:26:36.252Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_c7c4a254-0e85-4b65-a95a-da6d5fefb4be/root/var/svc/log/oxide-crucible-pantry:default.log
10682023-11-08T20:26:36.276Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_e7c862ed-b330-40df-a2a9-8f75364c1361/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10692023-11-08T20:26:37.359Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_e7c862ed-b330-40df-a2a9-8f75364c1361/root/var/svc/log/oxide-crucible-pantry:default.log
10702023-11-08T20:26:37.384Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_eb94b5de-f204-4216-8cee-fcc10f32a203/root/var/svc/log/oxide-nexus:default.log (518767 bytes)
10712023-11-08T20:26:38.242Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_eb94b5de-f204-4216-8cee-fcc10f32a203/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 518767 -> 581816
10722023-11-08T20:26:39.309Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_eb94b5de-f204-4216-8cee-fcc10f32a203/root/var/svc/log/oxide-nexus:default.log
10732023-11-08T20:26:39.333Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_845cbd30-2e2f-4760-8d40-b655b9957cc3/root/var/svc/log/oxide-crucible-agent:default.log (3433 bytes)
10742023-11-08T20:26:40.406Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_845cbd30-2e2f-4760-8d40-b655b9957cc3/root/var/svc/log/oxide-crucible-agent:default.log
10752023-11-08T20:26:40.430Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_845cbd30-2e2f-4760-8d40-b655b9957cc3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10762023-11-08T20:26:41.507Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_845cbd30-2e2f-4760-8d40-b655b9957cc3/root/var/svc/log/oxide-crucible-downstairs:default.log
10772023-11-08T20:26:41.531Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f0eef30a-8cd2-443c-80f2-66a5c384f838/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10782023-11-08T20:26:42.603Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f0eef30a-8cd2-443c-80f2-66a5c384f838/root/var/svc/log/oxide-crucible-agent:default.log
10792023-11-08T20:26:42.627Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f0eef30a-8cd2-443c-80f2-66a5c384f838/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10802023-11-08T20:26:43.734Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f0eef30a-8cd2-443c-80f2-66a5c384f838/root/var/svc/log/oxide-crucible-downstairs:default.log
10812023-11-08T20:26:43.782Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b/root/var/svc/log/oxide-ntp:default.log (1116 bytes)
10822023-11-08T20:26:44.868Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b/root/var/svc/log/oxide-ntp:default.log
10832023-11-08T20:26:44.892Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_a7922c2d-c0dc-4d07-8814-e3074299ed03/root/var/svc/log/oxide-oximeter:default.log (195461 bytes)
10842023-11-08T20:26:45.420Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_a7922c2d-c0dc-4d07-8814-e3074299ed03/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 195461 -> 270702
10852023-11-08T20:26:46.494Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_a7922c2d-c0dc-4d07-8814-e3074299ed03/root/var/svc/log/oxide-oximeter:default.log
10862023-11-08T20:26:46.518Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c0016b5c-b3b2-4d57-be5a-b8e2a1d7f9da/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10872023-11-08T20:26:47.592Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c0016b5c-b3b2-4d57-be5a-b8e2a1d7f9da/root/var/svc/log/oxide-cockroachdb:default.log
10882023-11-08T20:26:47.621Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_29bcba02-eb43-4e84-a05d-b507426bcbab/root/var/svc/log/oxide-crucible-agent:default.log (23931 bytes)
10892023-11-08T20:26:48.762Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_29bcba02-eb43-4e84-a05d-b507426bcbab/root/var/svc/log/oxide-crucible-agent:default.log
10902023-11-08T20:26:48.786Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_29bcba02-eb43-4e84-a05d-b507426bcbab/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10912023-11-08T20:26:49.860Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_29bcba02-eb43-4e84-a05d-b507426bcbab/root/var/svc/log/oxide-crucible-downstairs:default.log
10922023-11-08T20:26:49.891Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_29bcba02-eb43-4e84-a05d-b507426bcbab/root/var/svc/log/oxide-crucible-downstairs:downstairs-0dc82a22-d012-41ad-a8fd-b240dc3db0bd.log (3049 bytes)
10932023-11-08T20:26:50.974Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_29bcba02-eb43-4e84-a05d-b507426bcbab/root/var/svc/log/oxide-crucible-downstairs:downstairs-0dc82a22-d012-41ad-a8fd-b240dc3db0bd.log
10942023-11-08T20:26:50.998Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_29bcba02-eb43-4e84-a05d-b507426bcbab/root/var/svc/log/oxide-crucible-downstairs:downstairs-c03899c7-186d-4237-9181-32398aa228a5.log (5534 bytes)
10952023-11-08T20:26:52.129Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_29bcba02-eb43-4e84-a05d-b507426bcbab/root/var/svc/log/oxide-crucible-downstairs:downstairs-c03899c7-186d-4237-9181-32398aa228a5.log
10962023-11-08T20:26:52.169Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_32340b6b-85fb-44c2-b2f1-e5e37354436d/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10972023-11-08T20:27:07.364Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_32340b6b-85fb-44c2-b2f1-e5e37354436d/root/var/svc/log/oxide-crucible-agent:default.log
10982023-11-08T20:27:07.774Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_32340b6b-85fb-44c2-b2f1-e5e37354436d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10992023-11-08T20:27:11.751Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_32340b6b-85fb-44c2-b2f1-e5e37354436d/root/var/svc/log/oxide-crucible-downstairs:default.log
11002023-11-08T20:27:11.847Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_20655c0b-9730-4c9b-8e49-d750b256560d/root/var/svc/log/oxide-internal_dns:default.log (105156 bytes)
11012023-11-08T20:27:11.918Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_20655c0b-9730-4c9b-8e49-d750b256560d/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 105156 -> 112666
11022023-11-08T20:27:13.018Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_20655c0b-9730-4c9b-8e49-d750b256560d/root/var/svc/log/oxide-internal_dns:default.log
11032023-11-08T20:27:13.106Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b/root/var/log/chrony/measurements.log (10960 bytes)
11042023-11-08T20:27:13.179Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b/root/var/log/chrony/measurements.log" changed size mid upload: 10960 -> 13563
11052023-11-08T20:27:14.351Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b/root/var/log/chrony/measurements.log
11062023-11-08T20:27:14.388Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b/root/var/log/chrony/statistics.log (6435 bytes)
11072023-11-08T20:27:14.465Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b/root/var/log/chrony/statistics.log" changed size mid upload: 6435 -> 8892
11082023-11-08T20:27:15.605Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b/root/var/log/chrony/statistics.log
11092023-11-08T20:27:15.708Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b/root/var/log/chrony/tracking.log (1589 bytes)
11102023-11-08T20:27:15.773Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b/root/var/log/chrony/tracking.log" changed size mid upload: 1589 -> 1721
11112023-11-08T20:27:16.877Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_c4e54596-4365-49ab-8312-8e96676c7e0b/root/var/log/chrony/tracking.log