01HDH8T5NRKWVZH2RC38C2ANAR: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HDH8TVCRH6V71ZA1HRX5ZBTQ

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-24T17:29:50.976Zjob dependencies complete; ready to run (waiting for 51 m 13 s)
22023-10-24T17:30:35.488Zjob assigned to worker 01HDHBRTFMTEP5KKGNX36CWCYW (queued for 44 s)
32023-10-24T17:30:35.546Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-24T17:30:35.599Zxcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd
52023-10-24T17:30:35.638Zhttp://172.20.0.5:9969/ipxe/buskin... ok
62023-10-24T17:30:35.673Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
72023-10-24T17:30:35.707Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
82023-10-24T17:30:35.745Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
92023-10-24T17:30:35.784Zhttp://172.20.0.5:9969/postboot/buskin... ok
102023-10-24T17:30:35.818ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
112023-10-24T17:30:35.857Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
122023-10-24T17:30:35.892ZOxide Helios Version helios-2.0.22241 64-bit
132023-10-24T17:30:35.932ZConfiguring devices.
142023-10-24T17:30:35.969Z##################################################
152023-10-24T17:30:36.008Z#### ############# ##
162023-10-24T17:30:36.048Z### ### ############ ##
172023-10-24T17:30:36.086Z## ### # ## ### ## Oxide Computer Company ##
182023-10-24T17:30:36.124Z## ## ## ### # ### ##
192023-10-24T17:30:36.166Z## # ### #### #### This Station Under ##
202023-10-24T17:30:36.203Z### ### #### # ### Computer Control ##
212023-10-24T17:30:36.244Z#### #### ### ## ##
222023-10-24T17:30:36.286Z##################################################
232023-10-24T17:30:36.324ZHostname: buskin
242023-10-24T17:30:36.359ZLoading smf(7) serice descriptions:
252023-10-24T17:30:37.383Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622.gz
262023-10-24T17:30:39.752Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622.gz
272023-10-24T17:30:39.781Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
282023-10-24T17:31:03.235Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
292023-10-24T17:31:03.261Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280.gz
302023-10-24T17:31:36.970Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280.gz
312023-10-24T17:31:36.998Zdownloading input: /input/ci-tools/work/caboose-util.gz
322023-10-24T17:31:38.154Zdownloaded input: /input/ci-tools/work/caboose-util.gz
332023-10-24T17:31:38.182Zdownloading input: /input/ci-tools/work/tufaceous.gz
342023-10-24T17:31:40.977Zdownloaded input: /input/ci-tools/work/tufaceous.gz
352023-10-24T17:31:41.037Zdownloading input: /input/package/work/version.txt
362023-10-24T17:31:41.094Zdownloaded input: /input/package/work/version.txt
372023-10-24T17:31:41.203Zdownloading input: /input/package/work/package.tar.gz
382023-10-24T17:32:01.729Zdownloaded input: /input/package/work/package.tar.gz
392023-10-24T17:32:01.755Zdownloading input: /input/package/work/global-zone-packages.tar.gz
402023-10-24T17:32:39.817Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
412023-10-24T17:32:39.849Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
422023-10-24T17:32:48.349Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
432023-10-24T17:32:48.374Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
442023-10-24T17:33:23.945Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
452023-10-24T17:33:23.987Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
462023-10-24T17:34:06.558Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
472023-10-24T17:34:06.586Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
482023-10-24T17:34:32.070Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
492023-10-24T17:34:32.098Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
502023-10-24T17:34:37.796Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
512023-10-24T17:34:37.823Zdownloading input: /input/package/work/zones/crucible.tar.gz
522023-10-24T17:34:47.827Zdownloaded input: /input/package/work/zones/crucible.tar.gz
532023-10-24T17:34:47.852Zdownloading input: /input/package/work/zones/external-dns.tar.gz
542023-10-24T17:34:53.281Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
552023-10-24T17:34:53.305Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
562023-10-24T17:34:59.071Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
572023-10-24T17:34:59.097Zdownloading input: /input/package/work/zones/ntp.tar.gz
582023-10-24T17:34:59.122Zdownloaded input: /input/package/work/zones/ntp.tar.gz
592023-10-24T17:34:59.147Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
602023-10-24T17:35:04.548Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
612023-10-24T17:35:04.574Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
622023-10-24T17:35:08.610Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
632023-10-24T17:35:08.638Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
642023-10-24T17:35:23.258Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
652023-10-24T17:35:23.283Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
662023-10-24T17:35:38.733Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
672023-10-24T17:35:38.810Zdownloading input: /input/package/work/zones/overlay.tar.gz
682023-10-24T17:35:38.933Zdownloaded input: /input/package/work/zones/overlay.tar.gz
692023-10-24T17:35:38.964Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
702023-10-24T17:35:43.212Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
712023-10-24T17:35:43.241Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
722023-10-24T17:35:57.131Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
732023-10-24T17:35:57.154Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
742023-10-24T17:36:49.764Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
752023-10-24T17:36:49.791Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
762023-10-24T17:37:33.734Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
772023-10-24T17:37:34.361Zstarting task 0: "setup"
782023-10-24T17:37:34.613Z++ uname -s
792023-10-24T17:37:34.716Z+ kern=SunOS
802023-10-24T17:37:34.876Z+ case "$kern" in
812023-10-24T17:37:35.348Z+ groupadd -g 12345 build
822023-10-24T17:37:35.398Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
832023-10-24T17:37:36.424Z+ zfs create -o mountpoint=/work rpool/work
842023-10-24T17:37:36.466Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
852023-10-24T17:37:37.034Z+ home_fs=zfs
862023-10-24T17:37:37.086Z+ [[ zfs == autofs ]]
872023-10-24T17:37:37.140Z+ mkdir -p /home/build
882023-10-24T17:37:37.196Z+ chown build:build /home/build /work
892023-10-24T17:37:38.988Z+ chmod 0700 /home/build /work
902023-10-24T17:37:39.104Zprocess exited: duration 4051 ms, exit code 0
 
912023-10-24T17:37:40.383Zstarting task 1: "authentication"
922023-10-24T17:37:40.491Zprocess exited: duration 67 ms, exit code 0
 
932023-10-24T17:37:40.571Zstarting task 2: "build"
942023-10-24T17:37:40.613Z+ trap _exit_trap EXIT
952023-10-24T17:37:40.650Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
962023-10-24T17:37:40.812Zsecpolicy_fs_linkdir: 0x55 = 0x55
972023-10-24T17:37:42.090Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
982023-10-24T17:37:42.122Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
992023-10-24T17:37:42.158Zsecpolicy_fs_linkdir+3: 0 = 0xe5
1002023-10-24T17:37:42.196Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1012023-10-24T17:37:42.226Zsecpolicy_fs_linkdir+5: 0 = 0x1
1022023-10-24T17:37:42.702Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1032023-10-24T17:37:43.503Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1042023-10-24T17:37:43.630Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1052023-10-24T17:37:43.806Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1062023-10-24T17:37:43.972Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1072023-10-24T17:37:44.264Z+ [[ -d /opt/oxide ]]
1082023-10-24T17:37:44.754Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1092023-10-24T17:37:44.793Z+ pfexec mkdir /opt/oxide
1102023-10-24T17:37:44.839Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1112023-10-24T17:37:44.884Z+ [[ -d /opt/oxide-underneath ]]
1122023-10-24T17:37:44.921Z+ cd /opt/oxide-underneath
1132023-10-24T17:37:44.953Z+ pfexec tar ceEp@/f - .
1142023-10-24T17:37:44.982Z+ cd /opt/oxide
1152023-10-24T17:37:45.011Z+ pfexec tar xveEp@/f -
1162023-10-24T17:37:45.056Zx ., 0 bytes, 0 tape blocks
1172023-10-24T17:37:45.089Zx ./opte, 0 bytes, 0 tape blocks
1182023-10-24T17:37:45.125Zx ./opte/bin, 0 bytes, 0 tape blocks
1192023-10-24T17:37:45.162Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1202023-10-24T17:37:45.199Z+ pfexec rm -rf /opt/oxide-underneath
1212023-10-24T17:37:45.230Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1222023-10-24T17:37:45.261Z+ pfexec mkdir /opt/oxide/work
1232023-10-24T17:37:45.298Z+ pfexec chown build:build /opt/oxide/work
1242023-10-24T17:37:45.344Z+ cd /opt/oxide/work
1252023-10-24T17:37:45.386Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1262023-10-24T17:37:45.432ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1272023-10-24T17:37:45.892Zx out/mg-ddm-gz.tar, 63638528 bytes, 124294 tape blocks
1282023-10-24T17:37:46.337Zx out/omicron-sled-agent.tar, 115460608 bytes, 225509 tape blocks
1292023-10-24T17:37:46.372Zx out/target/test, 75 bytes, 1 tape blocks
1302023-10-24T17:37:46.409Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1312023-10-24T17:37:46.451Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1322023-10-24T17:37:46.986Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1332023-10-24T17:37:47.017Zx target/release/omicron-package, 38196032 bytes, 74602 tape blocks
1342023-10-24T17:37:47.047Zx tools/create_virtual_hardware.sh, 2986 bytes, 6 tape blocks
1352023-10-24T17:37:47.118Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1362023-10-24T17:37:47.145Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1372023-10-24T17:37:47.173Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1382023-10-24T17:37:47.204Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1392023-10-24T17:37:47.657Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1402023-10-24T17:37:47.700Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1412023-10-24T17:37:47.732Z
1422023-10-24T17:37:47.836Zreal 1.552332739
1432023-10-24T17:37:47.862Zuser 1.397078546
1442023-10-24T17:37:47.893Zsys 0.274340889
1452023-10-24T17:37:48.169Ztrap 0.000067360
1462023-10-24T17:37:48.219Ztflt 0.000000000
1472023-10-24T17:37:48.259Zdflt 0.000000000
1482023-10-24T17:37:48.296Zkflt 0.000000000
1492023-10-24T17:37:48.334Zlock 0.000000000
1502023-10-24T17:37:48.383Zslp 1.396031917
1512023-10-24T17:37:48.417Zlat 0.035265939
1522023-10-24T17:37:48.452Zstop 0.000120831
1532023-10-24T17:37:48.489Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1542023-10-24T17:37:48.519Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1552023-10-24T17:37:48.544Z+ mkdir tests
1562023-10-24T17:37:48.574Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1572023-10-24T17:37:48.604Z+ ptime -m gunzip
1582023-10-24T17:37:48.642Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622
1592023-10-24T17:37:48.684Z
1602023-10-24T17:37:48.712Zreal 0.140552167
1612023-10-24T17:37:48.738Zuser 0.129437719
1622023-10-24T17:37:48.763Zsys 0.010859062
1632023-10-24T17:37:48.793Ztrap 0.000013220
1642023-10-24T17:37:48.821Ztflt 0.000000000
1652023-10-24T17:37:48.851Zdflt 0.000000000
1662023-10-24T17:37:48.881Zkflt 0.000000000
1672023-10-24T17:37:48.916Zlock 0.000000000
1682023-10-24T17:37:48.950Zslp 0.000019850
1692023-10-24T17:37:48.988Zlat 0.000022810
1702023-10-24T17:37:49.023Zstop 0.000135212
1712023-10-24T17:37:49.052Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-49e31e75b930b622
1722023-10-24T17:37:49.085Z+ chmod a+x tests/bootstrap-49e31e75b930b622
1732023-10-24T17:37:49.137Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1742023-10-24T17:37:49.168Z+ ptime -m gunzip
1752023-10-24T17:37:49.203Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1762023-10-24T17:37:49.231Z
1772023-10-24T17:37:49.255Zreal 2.237617594
1782023-10-24T17:37:49.283Zuser 2.102605342
1792023-10-24T17:37:49.317Zsys 0.134616162
1802023-10-24T17:37:49.347Ztrap 0.000169844
1812023-10-24T17:37:49.373Ztflt 0.000000000
1822023-10-24T17:37:49.816Zdflt 0.000000000
1832023-10-24T17:37:49.848Zkflt 0.000000000
1842023-10-24T17:37:49.881Zlock 0.000000000
1852023-10-24T17:37:49.915Zslp 0.000011400
1862023-10-24T17:37:49.942Zlat 0.000022780
1872023-10-24T17:37:49.967Zstop 0.000128122
1882023-10-24T17:37:49.998Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1892023-10-24T17:37:50.039Z+ chmod a+x tests/bootstrap
1902023-10-24T17:37:50.077Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1912023-10-24T17:37:50.539Z+ ptime -m gunzip
1922023-10-24T17:37:50.576Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280
1932023-10-24T17:37:50.607Z
1942023-10-24T17:37:50.644Zreal 2.539869167
1952023-10-24T17:37:50.678Zuser 2.396790175
1962023-10-24T17:37:50.715Zsys 0.142633164
1972023-10-24T17:37:50.753Ztrap 0.000216293
1982023-10-24T17:37:51.226Ztflt 0.000000000
1992023-10-24T17:37:51.258Zdflt 0.000000000
2002023-10-24T17:37:51.285Zkflt 0.000000000
2012023-10-24T17:37:51.319Zlock 0.000000000
2022023-10-24T17:37:51.352Zslp 0.000023520
2032023-10-24T17:37:51.377Zlat 0.000023290
2042023-10-24T17:37:51.407Zstop 0.000122292
2052023-10-24T17:37:51.444Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a91cf9af3457d280
2062023-10-24T17:37:51.483Z+ chmod a+x tests/end_to_end_tests-a91cf9af3457d280
2072023-10-24T17:37:51.949Z+ bmat address ls
2082023-10-24T17:37:51.975ZNAME CIDR FIRST LAST COUNT
2092023-10-24T17:37:52.416Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2102023-10-24T17:37:52.452Z++ bmat address ls -f extra -Ho first,last,count
2112023-10-24T17:37:52.479Z+ set -- 10.151.1.164 10.151.1.227 64
2122023-10-24T17:37:52.510Z+ EXTRA_IP_START=10.151.1.164
2132023-10-24T17:37:52.535Z+ EXTRA_IP_END=10.151.1.227
2142023-10-24T17:37:52.613Z+ EXTRA_IP_COUNT=64
2152023-10-24T17:37:52.638Z+ (( EXTRA_IP_COUNT >= 32 ))
2162023-10-24T17:37:52.669Z+ EXTRA_IP_BASE=10.151.1
2172023-10-24T17:37:52.701Z+ EXTRA_IP_FOCTET=164
2182023-10-24T17:37:52.733Z+ EXTRA_IP_LOCTET=227
2192023-10-24T17:37:52.767Z+ SERVICE_IP_POOL_START=10.151.1.164
2202023-10-24T17:37:52.817Z+ SERVICE_IP_POOL_END=10.151.1.173
2212023-10-24T17:37:52.846Z+ DNS_IP1=10.151.1.164
2222023-10-24T17:37:52.877Z+ DNS_IP2=10.151.1.165
2232023-10-24T17:37:52.921ZPING 10.151.1.100: 56 data bytes
2242023-10-24T17:37:52.954Z+ UPLINK_IP=10.151.1.174
2252023-10-24T17:37:52.987Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
2262023-10-24T17:37:53.026Z+ IPPOOL_START=10.151.1.175
2272023-10-24T17:37:53.066Z
2282023-10-24T17:37:53.112Z+ IPPOOL_END=10.151.1.227
2292023-10-24T17:37:53.146Z----10.151.1.100 PING Statistics----
2302023-10-24T17:37:53.586Z++ ipadm show-addr -po type,addr
2312023-10-24T17:37:53.621Z1 packets transmitted, 1 packets received, 0% packet loss
2322023-10-24T17:37:53.651Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2332023-10-24T17:37:53.680Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2342023-10-24T17:37:53.710Z+ GATEWAY_IP=10.151.1.100
2352023-10-24T17:37:53.740ZUsing igb0 as physical link
2362023-10-24T17:37:53.770Z+ [[ -n 10.151.1.100 ]]
2372023-10-24T17:37:53.797ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2382023-10-24T17:37:53.824Z+ ping -s 10.151.1.100 56 1
2392023-10-24T17:37:53.859ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2402023-10-24T17:37:53.892Z++ arp -an
2412023-10-24T17:37:54.392ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2422023-10-24T17:37:54.428Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2432023-10-24T17:37:54.880ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2442023-10-24T17:37:54.918Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2452023-10-24T17:37:54.951ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2462023-10-24T17:37:54.979Z+ routeadm -e ipv4-forwarding -u
2472023-10-24T17:37:55.006ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2482023-10-24T17:37:55.036Z+ PXA_START=10.151.1.164
2492023-10-24T17:37:55.074ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2502023-10-24T17:37:55.103Z+ PXA_END=10.151.1.227
2512023-10-24T17:37:55.128ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2522023-10-24T17:37:55.152Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2532023-10-24T17:37:55.181ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2542023-10-24T17:37:55.212Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2552023-10-24T17:37:55.247ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2562023-10-24T17:37:55.284Z+ ZPOOL_VDEV_DIR=/scratch
2572023-10-24T17:37:55.335ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2582023-10-24T17:37:55.424Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2592023-10-24T17:37:55.461ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2602023-10-24T17:37:55.938Z+++ dirname ./tools/create_virtual_hardware.sh
2612023-10-24T17:37:55.977ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2622023-10-24T17:37:56.017Z++ cd ./tools
2632023-10-24T17:37:56.045ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2642023-10-24T17:37:56.076Z++ pwd
2652023-10-24T17:37:56.111ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2662023-10-24T17:37:56.154Z+ SOURCE_DIR=/opt/oxide/work/tools
2672023-10-24T17:37:56.187ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
2682023-10-24T17:37:56.222Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2692023-10-24T17:37:56.254ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
2702023-10-24T17:37:56.287Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2712023-10-24T17:37:56.725ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
2722023-10-24T17:37:56.757Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2732023-10-24T17:37:56.788ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
2742023-10-24T17:37:56.820Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2752023-10-24T17:37:56.855ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
2762023-10-24T17:37:56.887Z++ dladm show-phys -p -o LINK
2772023-10-24T17:37:56.927ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
2782023-10-24T17:37:56.961Z++ head -1
2792023-10-24T17:37:56.988ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
2802023-10-24T17:37:57.017Z+ PHYSICAL_LINK=igb0
2812023-10-24T17:37:57.047ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
2822023-10-24T17:37:57.083Z+ echo 'Using igb0 as physical link'
2832023-10-24T17:37:57.109ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
2842023-10-24T17:37:57.134Z+ ensure_run_as_root
2852023-10-24T17:37:57.163ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
2862023-10-24T17:37:57.629Z++ id -u
2872023-10-24T17:37:57.783ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
2882023-10-24T17:37:57.809Z+ [[ 0 -ne 0 ]]
2892023-10-24T17:37:57.833ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
2902023-10-24T17:37:57.858Z+ ensure_zpools
2912023-10-24T17:37:57.883ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
2922023-10-24T17:37:57.909Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2932023-10-24T17:37:57.938ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2942023-10-24T17:37:57.969Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2952023-10-24T17:37:58.007ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2962023-10-24T17:37:58.059Z+ readarray -t ZPOOLS
2972023-10-24T17:37:58.085ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
2982023-10-24T17:37:58.123Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2992023-10-24T17:37:58.150ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3002023-10-24T17:37:58.174Z++ sed 's/[ ",]//g'
3012023-10-24T17:37:58.200ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3022023-10-24T17:37:58.230Z+ for ZPOOL in "${ZPOOLS[@]}"
3032023-10-24T17:37:58.256ZSimnet net0/sc0_0 exists
3042023-10-24T17:37:58.280Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
3052023-10-24T17:37:58.305ZSimnet net1/sc1_0 exists
3062023-10-24T17:37:58.329Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
3072023-10-24T17:37:58.357ZVnic sc0_1 exists
3082023-10-24T17:37:58.381Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
3092023-10-24T17:37:58.406Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3102023-10-24T17:37:58.431Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
3112023-10-24T17:37:58.465Z0+0 records in
3122023-10-24T17:37:58.913Z0+0 records out
3132023-10-24T17:37:58.944Z0 bytes (0 iB) transferred in 0.000008 secs (0 iB/sec)
3142023-10-24T17:37:58.986Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
3152023-10-24T17:37:59.021Z+ set +x
3162023-10-24T17:37:59.047Z++ zpool list -o name
3172023-10-24T17:37:59.077Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
3182023-10-24T17:37:59.107Z+ [[ -z '' ]]
3192023-10-24T17:37:59.147Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
3202023-10-24T17:37:59.584Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
3212023-10-24T17:37:59.611Z+ set +x
3222023-10-24T17:37:59.655Z+ for ZPOOL in "${ZPOOLS[@]}"
3232023-10-24T17:37:59.682Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3242023-10-24T17:37:59.711Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3252023-10-24T17:37:59.744Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3262023-10-24T17:37:59.774Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3272023-10-24T17:37:59.799Z0+0 records in
3282023-10-24T17:37:59.823Z0+0 records out
3292023-10-24T17:37:59.850Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3302023-10-24T17:37:59.880Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3312023-10-24T17:37:59.918Z+ set +x
3322023-10-24T17:37:59.949Z++ zpool list -o name
3332023-10-24T17:37:59.975Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3342023-10-24T17:38:00.000Z+ [[ -z '' ]]
3352023-10-24T17:38:00.025Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3362023-10-24T17:38:00.050Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3372023-10-24T17:38:00.074Z+ set +x
3382023-10-24T17:38:00.099Z+ for ZPOOL in "${ZPOOLS[@]}"
3392023-10-24T17:38:00.143Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3402023-10-24T17:38:00.179Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-10-24T17:38:00.218Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3422023-10-24T17:38:00.261Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3432023-10-24T17:38:00.305Z0+0 records in
3442023-10-24T17:38:00.340Z0+0 records out
3452023-10-24T17:38:00.376Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3462023-10-24T17:38:00.402Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3472023-10-24T17:38:00.431Z+ set +x
3482023-10-24T17:38:00.460Z++ zpool list -o name
3492023-10-24T17:38:00.487Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3502023-10-24T17:38:00.512Z+ [[ -z '' ]]
3512023-10-24T17:38:00.536Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3522023-10-24T17:38:00.567Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3532023-10-24T17:38:00.634Z+ set +x
3542023-10-24T17:38:00.665Z+ for ZPOOL in "${ZPOOLS[@]}"
3552023-10-24T17:38:00.690Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3562023-10-24T17:38:00.722Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3572023-10-24T17:38:00.752Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3582023-10-24T17:38:00.801Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3592023-10-24T17:38:00.832Z0+0 records in
3602023-10-24T17:38:00.871Z0+0 records out
3612023-10-24T17:38:00.938Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3622023-10-24T17:38:04.622Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3632023-10-24T17:38:04.875Z+ set +x
3642023-10-24T17:38:04.962Z++ zpool list -o name
3652023-10-24T17:38:05.032Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3662023-10-24T17:38:05.480Z+ [[ -z '' ]]
3672023-10-24T17:38:05.510Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3682023-10-24T17:38:05.540Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3692023-10-24T17:38:05.580Z+ set +x
3702023-10-24T17:38:05.623Z+ for ZPOOL in "${ZPOOLS[@]}"
3712023-10-24T17:38:05.681Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3722023-10-24T17:38:05.744Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3732023-10-24T17:38:05.792Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3742023-10-24T17:38:05.850Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3752023-10-24T17:38:06.300Z0+0 records in
3762023-10-24T17:38:06.348Z0+0 records out
3772023-10-24T17:38:06.380Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3782023-10-24T17:38:06.427Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3792023-10-24T17:38:06.459Z+ set +x
3802023-10-24T17:38:06.493Z++ zpool list -o name
3812023-10-24T17:38:06.518Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3822023-10-24T17:38:06.544Z+ [[ -z '' ]]
3832023-10-24T17:38:06.578Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3842023-10-24T17:38:06.616Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3852023-10-24T17:38:06.651Z+ set +x
3862023-10-24T17:38:06.686Z+ for ZPOOL in "${ZPOOLS[@]}"
3872023-10-24T17:38:06.712Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3882023-10-24T17:38:06.745Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3892023-10-24T17:38:06.777Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3902023-10-24T17:38:06.811Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3912023-10-24T17:38:06.850Z0+0 records in
3922023-10-24T17:38:06.874Z0+0 records out
3932023-10-24T17:38:06.899Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3942023-10-24T17:38:06.923Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3952023-10-24T17:38:06.951Z+ set +x
3962023-10-24T17:38:06.977Z++ zpool list -o name
3972023-10-24T17:38:07.009Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3982023-10-24T17:38:07.037Z+ [[ -z '' ]]
3992023-10-24T17:38:07.062Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
4002023-10-24T17:38:07.096Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4012023-10-24T17:38:07.129Z+ set +x
4022023-10-24T17:38:07.156Z+ for ZPOOL in "${ZPOOLS[@]}"
4032023-10-24T17:38:07.186Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4042023-10-24T17:38:07.214Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4052023-10-24T17:38:07.241Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4062023-10-24T17:38:07.268Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4072023-10-24T17:38:07.304Z0+0 records in
4082023-10-24T17:38:07.330Z0+0 records out
4092023-10-24T17:38:07.355Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4102023-10-24T17:38:07.379Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4112023-10-24T17:38:07.438Z+ set +x
4122023-10-24T17:38:07.468Z++ zpool list -o name
4132023-10-24T17:38:07.492Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4142023-10-24T17:38:07.523Z+ [[ -z '' ]]
4152023-10-24T17:38:07.556Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4162023-10-24T17:38:07.586Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4172023-10-24T17:38:07.622Z+ set +x
4182023-10-24T17:38:07.651Z+ for ZPOOL in "${ZPOOLS[@]}"
4192023-10-24T17:38:07.688Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4202023-10-24T17:38:07.724Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4212023-10-24T17:38:07.757Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4222023-10-24T17:38:07.795Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4232023-10-24T17:38:07.826Z0+0 records in
4242023-10-24T17:38:07.859Z0+0 records out
4252023-10-24T17:38:07.891Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4262023-10-24T17:38:07.918Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4272023-10-24T17:38:07.950Z+ set +x
4282023-10-24T17:38:08.030Zdownloading softnpu ...done
4292023-10-24T17:38:08.063Z++ zpool list -o name
4302023-10-24T17:38:08.100Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/libsidecar_lite.so.sha256.txt
4312023-10-24T17:38:08.125Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4322023-10-24T17:38:08.151Z+ [[ -z '' ]]
4332023-10-24T17:38:08.176Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4342023-10-24T17:38:08.202Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4352023-10-24T17:38:08.228Z+ set +x
4362023-10-24T17:38:08.253Z+ for ZPOOL in "${ZPOOLS[@]}"
4372023-10-24T17:38:08.279Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4382023-10-24T17:38:08.314Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4392023-10-24T17:38:09.664Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4402023-10-24T17:38:09.694Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4412023-10-24T17:38:09.722Z0+0 records in
4422023-10-24T17:38:09.747Z0+0 records out
4432023-10-24T17:38:09.772Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4442023-10-24T17:38:09.797Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4452023-10-24T17:38:09.821Z+ set +x
4462023-10-24T17:38:09.847Z++ zpool list -o name
4472023-10-24T17:38:09.875Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4482023-10-24T17:38:09.907Z+ [[ -z '' ]]
4492023-10-24T17:38:09.944Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4502023-10-24T17:38:09.972Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4512023-10-24T17:38:09.999Z+ set +x
4522023-10-24T17:38:10.031Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4532023-10-24T17:38:10.113Z+ readarray -t ZPOOLS
4542023-10-24T17:38:10.147Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4552023-10-24T17:38:10.173Z++ sed 's/[ ",]//g'
4562023-10-24T17:38:10.198Z+ for ZPOOL in "${ZPOOLS[@]}"
4572023-10-24T17:38:10.226Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4582023-10-24T17:38:10.253Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-10-24T17:38:10.281Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4602023-10-24T17:38:10.313Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4612023-10-24T17:38:10.349Z0+0 records in
4622023-10-24T17:38:10.379Z0+0 records out
4632023-10-24T17:38:10.403Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4642023-10-24T17:38:10.430Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4652023-10-24T17:38:10.463Z+ set +x
4662023-10-24T17:38:10.500Z++ zpool list -o name
4672023-10-24T17:38:10.531Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4682023-10-24T17:38:10.578Z+ [[ -z '' ]]
4692023-10-24T17:38:10.606Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4702023-10-24T17:38:10.638Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4712023-10-24T17:38:11.085Z+ set +x
4722023-10-24T17:38:11.137Z+ for ZPOOL in "${ZPOOLS[@]}"
4732023-10-24T17:38:11.164Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4742023-10-24T17:38:11.191Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4752023-10-24T17:38:11.221Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4762023-10-24T17:38:11.245Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4772023-10-24T17:38:11.270Z0+0 records in
4782023-10-24T17:38:11.296Z0+0 records out
4792023-10-24T17:38:11.334Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4802023-10-24T17:38:11.372Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4812023-10-24T17:38:11.412Z+ set +x
4822023-10-24T17:38:11.441Z++ zpool list -o name
4832023-10-24T17:38:11.466Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4842023-10-24T17:38:11.491Z+ [[ -z '' ]]
4852023-10-24T17:38:11.520Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4862023-10-24T17:38:11.551Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4872023-10-24T17:38:11.579Z+ set +x
4882023-10-24T17:38:11.603Z+ ensure_simulated_links igb0
4892023-10-24T17:38:11.628Z+ local PHYSICAL_LINK=igb0
4902023-10-24T17:38:11.652Z+ INDICES=("0" "1")
4912023-10-24T17:38:11.676Z+ for I in "${INDICES[@]}"
4922023-10-24T17:38:11.701Z++ get_simnet_name_if_exists net0
4932023-10-24T17:38:11.726Z++ dladm show-simnet -p -o LINK net0
4942023-10-24T17:38:11.754Z++ echo ''
4952023-10-24T17:38:11.783Z+ [[ -z '' ]]
4962023-10-24T17:38:11.818Z+ dladm create-simnet -t net0
4972023-10-24T17:38:11.855Z+ dladm create-simnet -t sc0_0
4982023-10-24T17:38:11.889Z+ dladm modify-simnet -t -p net0 sc0_0
4992023-10-24T17:38:11.914Z+ dladm set-linkprop -p mtu=9000 sc0_0
5002023-10-24T17:38:11.940Z+ success 'Simnet net0/sc0_0 exists'
5012023-10-24T17:38:11.969Z+ set +x
5022023-10-24T17:38:12.005Z+ for I in "${INDICES[@]}"
5032023-10-24T17:38:12.031Z++ get_simnet_name_if_exists net1
5042023-10-24T17:38:12.057Z++ dladm show-simnet -p -o LINK net1
5052023-10-24T17:38:12.081Z++ echo ''
5062023-10-24T17:38:12.107Z+ [[ -z '' ]]
5072023-10-24T17:38:12.168Z+ dladm create-simnet -t net1
5082023-10-24T17:38:12.197Z+ dladm create-simnet -t sc1_0
5092023-10-24T17:38:12.226Z+ dladm modify-simnet -t -p net1 sc1_0
5102023-10-24T17:38:12.260Z+ dladm set-linkprop -p mtu=9000 sc1_0
5112023-10-24T17:38:12.295Z+ success 'Simnet net1/sc1_0 exists'
5122023-10-24T17:38:12.327Z+ set +x
5132023-10-24T17:38:12.363Z++ get_vnic_name_if_exists sc0_1
5142023-10-24T17:38:12.428Z++ dladm show-vnic -p -o LINK sc0_1
5152023-10-24T17:38:12.459Z++ echo ''
5162023-10-24T17:38:12.488Z+ [[ -z '' ]]
5172023-10-24T17:38:12.515Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5182023-10-24T17:38:12.541Z+ [[ -v PROMISC_FILT_OFF ]]
5192023-10-24T17:38:12.571Z+ success 'Vnic sc0_1 exists'
5202023-10-24T17:38:12.607Z+ set +x
5212023-10-24T17:38:12.637Z+ warn_if_no_proxy_arp
5222023-10-24T17:38:12.663Z+ [[ -v PXA_START ]]
5232023-10-24T17:38:12.693Z+ [[ -v PXA_END ]]
5242023-10-24T17:38:12.724Z+ ensure_softnpu_zone
5252023-10-24T17:38:12.751Z+ zoneadm list
5262023-10-24T17:38:12.779Z+ grep -q sidecar_softnpu
5272023-10-24T17:38:13.229Z+ [[ -f out/npuzone/npuzone ]]
5282023-10-24T17:38:13.263Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch omicron-tracking
5292023-10-24T17:38:13.828Zdownloading asic_program.so ...done
5302023-10-24T17:38:13.871Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/scadm.sha256.txt
5312023-10-24T17:38:21.938Zdownloading scadm ...done
5322023-10-24T17:38:21.962Zcreating port tfportrear0_0 for sc0_0
5332023-10-24T17:38:21.986Zcreating port tfportqsfp0_0 for sc0_1
5342023-10-24T17:38:22.015Zsoftnpu zone setup
5352023-10-24T17:38:22.043Zconfigure zone
5362023-10-24T17:38:22.068Zinstall zone
5372023-10-24T17:38:23.023Zboot zone
5382023-10-24T17:38:26.135Z[sidecar_softnpu] svcs milestone/network
5392023-10-24T17:38:26.569ZSTATE STIME FMRI
5402023-10-24T17:38:26.600Zonline 17:38:24 svc:/milestone/network:default
5412023-10-24T17:38:26.626Z[sidecar_softnpu] /softnpu/npu start
5422023-10-24T17:38:26.650Z
5432023-10-24T17:38:26.674Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-10-24T17:38:26.701Z+ GATEWAY_IP=10.151.1.100
5452023-10-24T17:38:26.727Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-10-24T17:38:26.754ZUsing 10.151.1.100 as gateway ip
5472023-10-24T17:38:26.783Z+ [[ ! -v GATEWAY_MAC ]]
5482023-10-24T17:38:26.809Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-10-24T17:38:26.870Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-10-24T17:38:26.898Z+++ echo 18:c0:4d:0d:9f:b2
5512023-10-24T17:38:26.924Z+ COUNT=1
5522023-10-24T17:38:26.952Z+ [[ 1 -ne 1 ]]
5532023-10-24T17:38:26.977Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-10-24T17:38:27.010ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-10-24T17:38:27.039Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-10-24T17:38:27.069Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-10-24T17:38:27.093Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-10-24T17:38:27.119Z+ [[ -v PXA_START ]]
5592023-10-24T17:38:27.146Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-10-24T17:38:27.178Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-10-24T17:38:27.218Z+ z_scadm dump-state
5622023-10-24T17:38:27.244Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-10-24T17:38:27.288Zlocal v6:
5642023-10-24T17:38:27.326Zlocal v4:
5652023-10-24T17:38:27.354Zrouter v6:
5662023-10-24T17:38:27.380Zrouter v4:
5672023-10-24T17:38:27.408Zresolver v4:
5682023-10-24T17:38:27.436Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5692023-10-24T17:38:27.465Zresolver v6:
5702023-10-24T17:38:27.530Znat v4:
5712023-10-24T17:38:27.556Znat_v6:
5722023-10-24T17:38:27.582Zport_mac:
5732023-10-24T17:38:27.606Zproxy arp:
5742023-10-24T17:38:27.638Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5752023-10-24T17:38:27.665Z+ success 'softnpu zone exists'
5762023-10-24T17:38:27.690Z+ set +x
5772023-10-24T17:38:27.717Zsoftnpu zone exists
5782023-10-24T17:38:27.745Z
5792023-10-24T17:38:27.769Zreal 37.409676257
5802023-10-24T17:38:27.794Zuser 0.722062118
5812023-10-24T17:38:27.825Zsys 1.616759878
5822023-10-24T17:38:27.852Ztrap 0.000094214
5832023-10-24T17:38:27.890Ztflt 0.000162676
5842023-10-24T17:38:27.915Zdflt 0.000618659
5852023-10-24T17:38:27.940Zkflt 0.000000000
5862023-10-24T17:38:27.965Zlock 29:26.992319561
5872023-10-24T17:38:27.990Zslp 1:51.445357859
5882023-10-24T17:38:28.016Zlat 0.081563563
5892023-10-24T17:38:28.041Zstop 0.365005456
5902023-10-24T17:38:28.070Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5912023-10-24T17:38:28.105Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5922023-10-24T17:38:28.150Z+ SILO_NAME=recovery
5932023-10-24T17:38:28.175Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5942023-10-24T17:38:28.207Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5952023-10-24T17:38:28.240Z+ sed -i~ '
5962023-10-24T17:38:28.269Z /^external_dns_ips/c\
5972023-10-24T17:38:28.298Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5982023-10-24T17:38:28.323Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5992023-10-24T17:38:28.761Z /^first/c\
6002023-10-24T17:38:28.788Zfirst = "10.151.1.164"
6012023-10-24T17:38:28.813Z /^last/c\
6022023-10-24T17:38:28.837Zlast = "10.151.1.173"
6032023-10-24T17:38:28.862Z }
6042023-10-24T17:38:28.886Z /^\[rack_network_config/,/^$/ {
6052023-10-24T17:38:28.915Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6062023-10-24T17:38:28.943Z /^infra_ip_first/c\
6072023-10-24T17:38:28.972Z+++ pkg/config-rss.toml Tue Oct 24 17:38:26 2023
6082023-10-24T17:38:29.000Zinfra_ip_first = "10.151.1.174"
6092023-10-24T17:38:29.031Z@@ -41,7 +41,7 @@
6102023-10-24T17:38:29.056Z /^infra_ip_last/c\
6112023-10-24T17:38:29.081Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6122023-10-24T17:38:29.106Zinfra_ip_last = "10.151.1.174"
6132023-10-24T17:38:29.136Z # must be contained in one of the "internal services" IP Pool ranges listed
6142023-10-24T17:38:29.162Z }
6152023-10-24T17:38:29.187Z # below.
6162023-10-24T17:38:29.211Z /^\[\[rack_network_config.ports/,/^$/ {
6172023-10-24T17:38:29.238Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6182023-10-24T17:38:29.264Z /^routes/c\
6192023-10-24T17:38:29.292Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6202023-10-24T17:38:29.318Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6212023-10-24T17:38:29.342Z
6222023-10-24T17:38:29.367Z /^addresses/c\
6232023-10-24T17:38:29.396Z # Initial TLS certificates for the external API
6242023-10-24T17:38:29.430Zaddresses = \["10.151.1.174/32"\]
6252023-10-24T17:38:29.473Z #
6262023-10-24T17:38:29.513Z }
6272023-10-24T17:38:29.541Z@@ -77,8 +77,8 @@
6282023-10-24T17:38:29.570Z' pkg/config-rss.toml
6292023-10-24T17:38:29.603Z #
6302023-10-24T17:38:29.630Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6312023-10-24T17:38:29.654Z # For more on this and what to put here, see docs/how-to-run.adoc.
6322023-10-24T17:38:29.679Z+ true
6332023-10-24T17:38:29.724Z [[internal_services_ip_pool_ranges]]
6342023-10-24T17:38:29.774Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6352023-10-24T17:38:29.802Z-first = "192.168.1.20"
6362023-10-24T17:38:29.826Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6372023-10-24T17:38:29.855Z-last = "192.168.1.29"
6382023-10-24T17:38:30.180Z+ TLS_NAME=recovery.sys.oxide.test
6392023-10-24T17:38:32.769Z+first = "10.151.1.164"
6402023-10-24T17:38:33.070Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6412023-10-24T17:38:33.617Z+last = "10.151.1.173"
6422023-10-24T17:38:33.945Z......+......+.+..+...+.........+......+...+....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.............+.+...........+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+..+....+...+.....+............+......+...................+......+.....+..........+..+..................+.............+.....+.+......+........+......+.+..+..........+........+......................+............+...+.....+...+..........+.....+..........+.........+........+.......+...+...+.........+...+..+.................................+.......+......+.....+...+.+.....+...............+......+.........+.............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6432023-10-24T17:38:34.055Z
6442023-10-24T17:38:34.104Z....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+...+.+..+....+.....+.............+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+....+......+....................+...+......+.............+......+..+...+................+.....+...+.......+......+..+.+......+.........+......+.........+..+............+...+...+......................+.....+...............+..........+............+.....................+.....+.......+..+.......+...........+...................+.....+....+.....+.......+...+..+...+....+..+.+........+.........+...+..................+....+...+...............+...+....................+.+...+............+.....+...+.+......+........+.......+...........+...........................................+.................+.......+..+....+.....+.+......+.....+...+.......+........+............+.........+...............+...+.......+...........+..................+......+...+...+...+.+........+.......+.....+.+............+........+....+...+...........+......................+..................+.........+..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6452023-10-24T17:38:34.146Z # TODO - this configuration is subject to change going forward. Ultimately these
6462023-10-24T17:38:34.197Z-----
6472023-10-24T17:38:34.245Z # parameters should be provided to the control plane via wicket, but we need to
6482023-10-24T17:38:34.285Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6492023-10-24T17:38:34.329Z@@ -92,8 +92,8 @@
6502023-10-24T17:38:34.360Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6512023-10-24T17:38:34.393Z # A range of IP addresses used by Boundary Services on the external network. In
6522023-10-24T17:38:34.419Z+ rmdir pkg
6532023-10-24T17:38:34.445Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6542023-10-24T17:38:34.473Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6552023-10-24T17:38:34.506Z # With softnpu, only one address is used.
6562023-10-24T17:38:34.534Z+ pfexec mkdir -p /usr/oxide
6572023-10-24T17:38:34.559Z-infra_ip_first = "192.168.1.30"
6582023-10-24T17:38:34.590Z+ pfexec rm -f /usr/oxide/tcpproxy
6592023-10-24T17:38:34.615Z-infra_ip_last = "192.168.1.30"
6602023-10-24T17:38:35.080Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6612023-10-24T17:38:35.118Z+infra_ip_first = "10.151.1.174"
6622023-10-24T17:38:35.146Z+ pfexec chmod +x /usr/oxide/tcpproxy
6632023-10-24T17:38:35.175Z+infra_ip_last = "10.151.1.174"
6642023-10-24T17:38:35.202Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6652023-10-24T17:38:35.228Z
6662023-10-24T17:38:35.255Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6672023-10-24T17:38:35.295Z # Configurations for BGP routers to run on the scrimlets.
6682023-10-24T17:38:35.330Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6692023-10-24T17:38:35.359Z bgp = []
6702023-10-24T17:38:35.409Z+ OMICRON_NO_UNINSTALL=1
6712023-10-24T17:38:35.438Z@@ -101,9 +101,9 @@
6722023-10-24T17:38:35.465Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6732023-10-24T17:38:35.493Z # You can configure multiple uplinks by repeating the following stanza
6742023-10-24T17:38:35.541ZOct 24 17:38:27.584 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6752023-10-24T17:38:35.571Z [[rack_network_config.ports]]
6762023-10-24T17:38:35.599ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6772023-10-24T17:38:35.627Z # Routes associated with this port.
6782023-10-24T17:38:35.662ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6792023-10-24T17:38:35.690Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6802023-10-24T17:38:35.719ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6812023-10-24T17:38:35.750Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6822023-10-24T17:38:35.789ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6832023-10-24T17:38:35.819Z # Addresses associated with this port.
6842023-10-24T17:38:35.847ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6852023-10-24T17:38:35.919Z-addresses = ["192.168.1.30/32"]
6862023-10-24T17:38:35.949ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6872023-10-24T17:38:35.987Z+addresses = ["10.151.1.174/32"]
6882023-10-24T17:38:36.023ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6892023-10-24T17:38:36.053Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6902023-10-24T17:38:36.080ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6912023-10-24T17:38:36.111Z port = "qsfp0"
6922023-10-24T17:38:36.140ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6932023-10-24T17:38:36.165Z # The speed of this port.
6942023-10-24T17:38:36.189ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6952023-10-24T17:38:36.218Za pkg/config-rss.toml 7K
6962023-10-24T17:38:36.259ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6972023-10-24T17:38:36.293Za pkg/initial-tls-cert.pem 2K
6982023-10-24T17:38:36.319ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6992023-10-24T17:38:36.357Za pkg/initial-tls-key.pem 4K
7002023-10-24T17:38:36.387ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7012023-10-24T17:38:36.415ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7022023-10-24T17:38:36.861ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7032023-10-24T17:38:36.890ZOct 24 17:38:27.586 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7042023-10-24T17:38:36.918ZOct 24 17:38:27.846 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7052023-10-24T17:38:36.946ZOct 24 17:38:27.880 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7062023-10-24T17:38:37.094ZOct 24 17:38:27.940 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7072023-10-24T17:38:37.127Z
7082023-10-24T17:38:37.155Zreal 0.492982440
7092023-10-24T17:38:37.187Zuser 0.096436974
7102023-10-24T17:38:37.242Zsys 2.044505418
7112023-10-24T17:38:37.283Ztrap 0.000000430
7122023-10-24T17:38:37.901Ztflt 0.000714781
7132023-10-24T17:38:37.936Zdflt 0.000448440
7142023-10-24T17:38:37.967Zkflt 0.000000000
7152023-10-24T17:38:37.999Zlock 44.354155613
7162023-10-24T17:38:38.024Zslp 0.707786647
7172023-10-24T17:38:38.052Zlat 0.002701267
7182023-10-24T17:38:38.147Zstop 0.200477917
7192023-10-24T17:38:38.176Z+ retry=0
7202023-10-24T17:38:38.206Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7212023-10-24T17:38:38.242Z+ [[ 0 -gt 30 ]]
7222023-10-24T17:38:38.273Z+ sleep 1
7232023-10-24T17:38:38.299Z+ retry=1
7242023-10-24T17:38:38.325Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7252023-10-24T17:38:38.352Z+ [[ 1 -gt 30 ]]
7262023-10-24T17:38:38.385Z+ sleep 1
7272023-10-24T17:38:38.412Z+ retry=2
7282023-10-24T17:38:38.438Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7292023-10-24T17:38:55.429Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7302023-10-24T17:38:55.949Zlocal v6:
7312023-10-24T17:38:55.975Zfd00:99::1
7322023-10-24T17:38:56.000Zfe80::34a5:f8ff:fe33:d2e1
7332023-10-24T17:38:56.026Zlocal v4:
7342023-10-24T17:38:56.057Z10.151.1.174
7352023-10-24T17:38:56.085Zrouter v6:
7362023-10-24T17:38:56.121Zfd00:1122:3344:101::/64 -> fe80::1402:3dff:fe0e:2fb (1) vid=0
7372023-10-24T17:38:56.149Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::1402:3dff:fe0e:2fb (1) vid=0
7382023-10-24T17:38:56.190Zrouter v4:
7392023-10-24T17:38:56.222Z0.0.0.0/0 -> 10.151.1.100 (2) vid=0
7402023-10-24T17:38:56.249Zresolver v4:
7412023-10-24T17:38:56.289Z10.151.1.174 -> 12:67:d7:c8:67:a7
7422023-10-24T17:38:56.322Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7432023-10-24T17:38:56.804Zresolver v6:
7442023-10-24T17:38:56.840Zfe80::34a5:f8ff:fe33:d2e1 -> 36:a5:f8:33:d2:e1
7452023-10-24T17:38:57.319Zfe80::1402:3dff:fe0e:2fb -> 16:02:3d:0e:02:fb
7462023-10-24T17:38:57.348Znat v4:
7472023-10-24T17:38:57.377Znat_v6:
7482023-10-24T17:38:57.402Zport_mac:
7492023-10-24T17:38:57.546Z1000: a8:40:25:06:95:4e
7502023-10-24T17:38:57.573Z1: a8:40:25:06:95:4f
7512023-10-24T17:38:57.602Zproxy arp:
7522023-10-24T17:38:57.628Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7532023-10-24T17:38:57.653Z+ retry=0
7542023-10-24T17:38:57.685Z+++ zoneadm list -n
7552023-10-24T17:38:57.719Z+++ grep oxz_ntp
7562023-10-24T17:38:57.744Z++ pfexec svcs -z -Hostate oxide/ntp
7572023-10-24T17:38:57.769Zsvcs: zone '-Hostate': entity not found
7582023-10-24T17:38:57.793Z++ true
7592023-10-24T17:38:57.818Z+ [[ '' != online ]]
7602023-10-24T17:38:57.842Z+ [[ 0 -gt 60 ]]
7612023-10-24T17:38:57.866Z+ sleep 1
7622023-10-24T17:38:57.897Z+ retry=1
7632023-10-24T17:38:58.374Z+++ zoneadm list -n
7642023-10-24T17:38:58.403Z+++ grep oxz_ntp
7652023-10-24T17:38:58.428Z++ pfexec svcs -z -Hostate oxide/ntp
7662023-10-24T17:38:58.453Zsvcs: zone '-Hostate': entity not found
7672023-10-24T17:38:58.481Z++ true
7682023-10-24T17:38:58.516Z+ [[ '' != online ]]
7692023-10-24T17:38:58.546Z+ [[ 1 -gt 60 ]]
7702023-10-24T17:38:58.572Z+ sleep 1
7712023-10-24T17:38:58.597Z+ retry=2
7722023-10-24T17:38:58.621Z+++ zoneadm list -n
7732023-10-24T17:38:58.646Z+++ grep oxz_ntp
7742023-10-24T17:38:58.670Z++ pfexec svcs -z -Hostate oxide/ntp
7752023-10-24T17:38:58.697Zsvcs: zone '-Hostate': entity not found
7762023-10-24T17:38:58.726Z++ true
7772023-10-24T17:38:58.752Z+ [[ '' != online ]]
7782023-10-24T17:38:58.781Z+ [[ 2 -gt 60 ]]
7792023-10-24T17:38:59.215Z+ sleep 1
7802023-10-24T17:38:59.242Z+ retry=3
7812023-10-24T17:38:59.275Z+++ zoneadm list -n
7822023-10-24T17:38:59.302Z+++ grep oxz_ntp
7832023-10-24T17:38:59.338Z++ pfexec svcs -z -Hostate oxide/ntp
7842023-10-24T17:38:59.365Zsvcs: zone '-Hostate': entity not found
7852023-10-24T17:38:59.390Z++ true
7862023-10-24T17:38:59.445Z+ [[ '' != online ]]
7872023-10-24T17:38:59.474Z+ [[ 3 -gt 60 ]]
7882023-10-24T17:38:59.509Z+ sleep 1
7892023-10-24T17:38:59.540Z+ retry=4
7902023-10-24T17:38:59.577Z+++ zoneadm list -n
7912023-10-24T17:38:59.609Z+++ grep oxz_ntp
7922023-10-24T17:38:59.636Z++ pfexec svcs -z -Hostate oxide/ntp
7932023-10-24T17:38:59.664Zsvcs: zone '-Hostate': entity not found
7942023-10-24T17:38:59.690Z++ true
7952023-10-24T17:38:59.714Z+ [[ '' != online ]]
7962023-10-24T17:38:59.741Z+ [[ 4 -gt 60 ]]
7972023-10-24T17:38:59.766Z+ sleep 1
7982023-10-24T17:39:00.535Z+ retry=5
7992023-10-24T17:39:00.561Z+++ zoneadm list -n
8002023-10-24T17:39:00.587Z+++ grep oxz_ntp
8012023-10-24T17:39:00.613Z++ pfexec svcs -z -Hostate oxide/ntp
8022023-10-24T17:39:00.642Zsvcs: zone '-Hostate': entity not found
8032023-10-24T17:39:00.669Z++ true
8042023-10-24T17:39:00.695Z+ [[ '' != online ]]
8052023-10-24T17:39:00.720Z+ [[ 5 -gt 60 ]]
8062023-10-24T17:39:00.745Z+ sleep 1
8072023-10-24T17:39:01.570Z+ retry=6
8082023-10-24T17:39:01.607Z+++ zoneadm list -n
8092023-10-24T17:39:01.642Z+++ grep oxz_ntp
8102023-10-24T17:39:01.712Z++ pfexec svcs -z -Hostate oxide/ntp
8112023-10-24T17:39:01.757Zsvcs: zone '-Hostate': entity not found
8122023-10-24T17:39:01.795Z++ true
8132023-10-24T17:39:01.820Z+ [[ '' != online ]]
8142023-10-24T17:39:01.845Z+ [[ 6 -gt 60 ]]
8152023-10-24T17:39:01.875Z+ sleep 1
8162023-10-24T17:39:02.580Z+ retry=7
8172023-10-24T17:39:02.606Z+++ zoneadm list -n
8182023-10-24T17:39:02.647Z+++ grep oxz_ntp
8192023-10-24T17:39:02.675Z++ pfexec svcs -z -Hostate oxide/ntp
8202023-10-24T17:39:02.701Zsvcs: zone '-Hostate': entity not found
8212023-10-24T17:39:02.726Z++ true
8222023-10-24T17:39:02.750Z+ [[ '' != online ]]
8232023-10-24T17:39:02.775Z+ [[ 7 -gt 60 ]]
8242023-10-24T17:39:02.801Z+ sleep 1
8252023-10-24T17:39:03.603Z+ retry=8
8262023-10-24T17:39:03.631Z+++ zoneadm list -n
8272023-10-24T17:39:03.661Z+++ grep oxz_ntp
8282023-10-24T17:39:03.688Z++ pfexec svcs -z -Hostate oxide/ntp
8292023-10-24T17:39:03.713Zsvcs: zone '-Hostate': entity not found
8302023-10-24T17:39:03.739Z++ true
8312023-10-24T17:39:03.768Z+ [[ '' != online ]]
8322023-10-24T17:39:03.795Z+ [[ 8 -gt 60 ]]
8332023-10-24T17:39:03.820Z+ sleep 1
8342023-10-24T17:39:04.654Z+ retry=9
8352023-10-24T17:39:04.680Z+++ zoneadm list -n
8362023-10-24T17:39:04.713Z+++ grep oxz_ntp
8372023-10-24T17:39:04.748Z++ pfexec svcs -z oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0 -Hostate oxide/ntp
8382023-10-24T17:39:04.780Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8392023-10-24T17:39:04.810Z++ true
8402023-10-24T17:39:04.844Z+ [[ '' != online ]]
8412023-10-24T17:39:04.873Z+ [[ 9 -gt 60 ]]
8422023-10-24T17:39:04.933Z+ sleep 1
8432023-10-24T17:39:05.656Z+ retry=10
8442023-10-24T17:39:05.743Z+++ zoneadm list -n
8452023-10-24T17:39:05.769Z+++ grep oxz_ntp
8462023-10-24T17:39:05.798Z++ pfexec svcs -z oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0 -Hostate oxide/ntp
8472023-10-24T17:39:05.896Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8482023-10-24T17:39:05.973Z++ true
8492023-10-24T17:39:05.999Z+ [[ '' != online ]]
8502023-10-24T17:39:06.024Z+ [[ 10 -gt 60 ]]
8512023-10-24T17:39:06.048Z+ sleep 1
8522023-10-24T17:39:06.726Z+ retry=11
8532023-10-24T17:39:06.831Z+++ zoneadm list -n
8542023-10-24T17:39:06.858Z+++ grep oxz_ntp
8552023-10-24T17:39:06.883Z++ pfexec svcs -z oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0 -Hostate oxide/ntp
8562023-10-24T17:39:06.908Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8572023-10-24T17:39:06.933Z++ true
8582023-10-24T17:39:06.959Z+ [[ '' != online ]]
8592023-10-24T17:39:06.984Z+ [[ 11 -gt 60 ]]
8602023-10-24T17:39:07.415Z+ sleep 1
8612023-10-24T17:39:07.717Z+ retry=12
8622023-10-24T17:39:07.742Z+++ zoneadm list -n
8632023-10-24T17:39:07.766Z+++ grep oxz_ntp
8642023-10-24T17:39:07.791Z++ pfexec svcs -z oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0 -Hostate oxide/ntp
8652023-10-24T17:39:07.815Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8662023-10-24T17:39:07.840Z++ true
8672023-10-24T17:39:07.909Z+ [[ '' != online ]]
8682023-10-24T17:39:07.935Z+ [[ 12 -gt 60 ]]
8692023-10-24T17:39:07.960Z+ sleep 1
8702023-10-24T17:39:08.740Z+ retry=13
8712023-10-24T17:39:08.765Z+++ zoneadm list -n
8722023-10-24T17:39:08.790Z+++ grep oxz_ntp
8732023-10-24T17:39:08.814Z++ pfexec svcs -z oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0 -Hostate oxide/ntp
8742023-10-24T17:39:08.838Z+ [[ disabled != online ]]
8752023-10-24T17:39:08.863Z+ [[ 13 -gt 60 ]]
8762023-10-24T17:39:08.888Z+ sleep 1
8772023-10-24T17:39:09.762Z+ retry=14
8782023-10-24T17:39:09.786Z+++ zoneadm list -n
8792023-10-24T17:39:09.811Z+++ grep oxz_ntp
8802023-10-24T17:39:09.835Z++ pfexec svcs -z oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0 -Hostate oxide/ntp
8812023-10-24T17:39:09.859Z+ [[ disabled != online ]]
8822023-10-24T17:39:09.883Z+ [[ 14 -gt 60 ]]
8832023-10-24T17:39:09.908Z+ sleep 1
8842023-10-24T17:39:10.784Z+ retry=15
8852023-10-24T17:39:10.810Z+++ zoneadm list -n
8862023-10-24T17:39:10.835Z+++ grep oxz_ntp
8872023-10-24T17:39:10.860Z++ pfexec svcs -z oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0 -Hostate oxide/ntp
8882023-10-24T17:39:10.886Z+ [[ disabled != online ]]
8892023-10-24T17:39:10.912Z+ [[ 15 -gt 60 ]]
8902023-10-24T17:39:10.939Z+ sleep 1
8912023-10-24T17:39:11.805Z+ retry=16
8922023-10-24T17:39:11.829Z+++ zoneadm list -n
8932023-10-24T17:39:11.854Z+++ grep oxz_ntp
8942023-10-24T17:39:11.878Z++ pfexec svcs -z oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0 -Hostate oxide/ntp
8952023-10-24T17:39:11.903Z+ [[ disabled != online ]]
8962023-10-24T17:39:11.927Z+ [[ 16 -gt 60 ]]
8972023-10-24T17:39:11.953Z+ sleep 1
8982023-10-24T17:39:12.827Z+ retry=17
8992023-10-24T17:39:12.852Z+++ zoneadm list -n
9002023-10-24T17:39:12.877Z+++ grep oxz_ntp
9012023-10-24T17:39:12.903Z++ pfexec svcs -z oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0 -Hostate oxide/ntp
9022023-10-24T17:39:12.928Z+ [[ disabled != online ]]
9032023-10-24T17:39:12.953Z+ [[ 17 -gt 60 ]]
9042023-10-24T17:39:12.978Z+ sleep 1
9052023-10-24T17:39:13.848Z+ retry=18
9062023-10-24T17:39:13.872Z+++ zoneadm list -n
9072023-10-24T17:39:13.897Z+++ grep oxz_ntp
9082023-10-24T17:39:13.924Z++ pfexec svcs -z oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0 -Hostate oxide/ntp
9092023-10-24T17:39:13.952Z+ [[ disabled != online ]]
9102023-10-24T17:39:13.979Z+ [[ 18 -gt 60 ]]
9112023-10-24T17:39:14.023Z+ sleep 1
9122023-10-24T17:39:14.870Z+ retry=19
9132023-10-24T17:39:14.894Z+++ zoneadm list -n
9142023-10-24T17:39:14.928Z+++ grep oxz_ntp
9152023-10-24T17:39:14.970Z++ pfexec svcs -z oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0 -Hostate oxide/ntp
9162023-10-24T17:39:15.021Z+ [[ disabled != online ]]
9172023-10-24T17:39:15.058Z+ [[ 19 -gt 60 ]]
9182023-10-24T17:39:15.103Z+ sleep 1
 
9192023-10-24T17:39:15.343Zbuskin console login: Oct 24 17:39:14 bukin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9202023-10-24T17:39:15.892Z+ retry=20
9212023-10-24T17:39:15.917Z+++ zoneadm list -n
9222023-10-24T17:39:15.942Z+++ grep oxz_ntp
9232023-10-24T17:39:15.966Z++ pfexec svcs -z oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0 -Hostate oxide/ntp
9242023-10-24T17:39:15.993Z+ [[ online != online ]]
9252023-10-24T17:39:16.019Z+ echo 'Waited for chrony: 20s'
9262023-10-24T17:39:16.044Z+ export RUST_BACKTRACE=1
9272023-10-24T17:39:16.073ZWaited for chrony: 20s
9282023-10-24T17:39:16.101Z+ RUST_BACKTRACE=1
9292023-10-24T17:39:16.130Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9302023-10-24T17:39:16.155Z+ ./tests/bootstrap
9312023-10-24T17:39:16.180ZUsing external DNS server at 10.151.1.164:53
9322023-10-24T17:39:16.205Z2023-10-24 17:39:15.812414240 UTC: attempting to log into API
9332023-10-24T17:39:30.982Z2023-10-24 17:39:30.979039217 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9342023-10-24T17:39:32.009Z2023-10-24 17:39:31.977302041 UTC: attempting to log into API
9352023-10-24T17:39:47.013Z2023-10-24 17:39:47.001417464 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9362023-10-24T17:39:48.014Z2023-10-24 17:39:48.002343208 UTC: attempting to log into API
9372023-10-24T17:40:03.047Z2023-10-24 17:40:03.041208741 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9382023-10-24T17:40:04.049Z2023-10-24 17:40:04.042182464 UTC: attempting to log into API
9392023-10-24T17:40:19.087Z2023-10-24 17:40:19.075520304 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9402023-10-24T17:40:20.084Z2023-10-24 17:40:20.076769564 UTC: attempting to log into API
9412023-10-24T17:40:35.120Z2023-10-24 17:40:35.107548142 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9422023-10-24T17:40:36.119Z2023-10-24 17:40:36.108850160 UTC: attempting to log into API
9432023-10-24T17:40:51.175Z2023-10-24 17:40:51.164316380 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9442023-10-24T17:40:52.176Z2023-10-24 17:40:52.165467137 UTC: attempting to log into API
 
9452023-10-24T17:40:53.266ZOct 24 17:40:52 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9462023-10-24T17:40:59.999ZOct 24 17:40:59 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9472023-10-24T17:41:05.710ZOct 24 17:41:05 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9482023-10-24T17:41:07.213Z2023-10-24 17:41:07.197970143 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9492023-10-24T17:41:08.209Z2023-10-24 17:41:08.198764163 UTC: attempting to log into API
 
9502023-10-24T17:41:11.165ZOct 24 17:41:10 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9512023-10-24T17:41:18.401ZOct 24 17:41:17 bukin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9522023-10-24T17:41:23.244Z2023-10-24 17:41:23.230744790 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9532023-10-24T17:41:24.593Z2023-10-24 17:41:24.231834406 UTC: attempting to log into API
9542023-10-24T17:41:39.277Z2023-10-24 17:41:39.267335329 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9552023-10-24T17:41:40.278Z2023-10-24 17:41:40.268112370 UTC: attempting to log into API
9562023-10-24T17:41:41.056Z2023-10-24 17:41:41.046741713 UTC: login succeeded
9572023-10-24T17:41:41.089Zwaiting for nexus to come up...
9582023-10-24T17:41:41.151Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9592023-10-24T17:41:41.204Zensuring datasets are ready...
9602023-10-24T17:41:43.866Zlet's roll.
9612023-10-24T17:41:43.895Z+ rm ./tests/bootstrap
9622023-10-24T17:41:43.949Z+ for test_bin in tests/*
9632023-10-24T17:41:43.982Z+ ./tests/bootstrap-49e31e75b930b622
9642023-10-24T17:41:44.009Z
9652023-10-24T17:41:44.034Zrunning 0 tests
9662023-10-24T17:41:44.063Z
9672023-10-24T17:41:44.090Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9682023-10-24T17:41:44.118Z
9692023-10-24T17:41:44.148Z+ for test_bin in tests/*
9702023-10-24T17:41:44.178Z+ ./tests/end_to_end_tests-a91cf9af3457d280
9712023-10-24T17:41:44.209Z
9722023-10-24T17:41:44.239Zrunning 2 tests
9732023-10-24T17:41:44.269Ztest no_spoof::no_spoof_here ... ok
9742023-10-24T17:42:43.973Ztest instance_launch::instance_launch has been running for over 60 seconds
9752023-10-24T17:44:41.749Ztest instance_launch::instance_launch ... ok
9762023-10-24T17:44:41.779Z
9772023-10-24T17:44:41.803Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 177.78s
9782023-10-24T17:44:41.828Z
9792023-10-24T17:44:41.852Z+ _exit_trap
9802023-10-24T17:44:41.877Z+ local status=0
9812023-10-24T17:44:41.901Z+ [[ 0 -eq 0 ]]
9822023-10-24T17:44:41.925Z+ exit 0
9832023-10-24T17:44:41.950Zprocess exited: duration 421155 ms, exit code 0
 
9842023-10-24T17:44:42.094Zfound 47 output files
9852023-10-24T17:44:42.270Zuploading: /var/svc/log/oxide-sled-agent:default.log (446570 bytes)
9862023-10-24T17:44:42.360Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 446570 -> 446970
9872023-10-24T17:44:43.393Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9882023-10-24T17:44:43.425Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_3d93b0d9-d890-451f-93f7-c623977b0248/root/var/svc/log/oxide-nexus:default.log (485329 bytes)
9892023-10-24T17:44:49.146Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_3d93b0d9-d890-451f-93f7-c623977b0248/root/var/svc/log/oxide-nexus:default.log
9902023-10-24T17:44:49.241Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_b980e819-fa3f-4f12-91c1-123a687d466f/root/var/svc/log/oxide-clickhouse:default.log (676240 bytes)
9912023-10-24T17:44:49.454Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_b980e819-fa3f-4f12-91c1-123a687d466f/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 676240 -> 702187
9922023-10-24T17:44:50.527Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_b980e819-fa3f-4f12-91c1-123a687d466f/root/var/svc/log/oxide-clickhouse:default.log
9932023-10-24T17:44:50.552Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2180ccaf-8975-4480-b9ce-caddfd91a175/root/var/svc/log/oxide-crucible-agent:default.log (14579 bytes)
9942023-10-24T17:44:51.641Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2180ccaf-8975-4480-b9ce-caddfd91a175/root/var/svc/log/oxide-crucible-agent:default.log
9952023-10-24T17:44:51.666Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2180ccaf-8975-4480-b9ce-caddfd91a175/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9962023-10-24T17:44:52.733Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2180ccaf-8975-4480-b9ce-caddfd91a175/root/var/svc/log/oxide-crucible-downstairs:default.log
9972023-10-24T17:44:52.757Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2180ccaf-8975-4480-b9ce-caddfd91a175/root/var/svc/log/oxide-crucible-downstairs:downstairs-c80fa30f-4f6a-447f-baf8-846fa2e3f8f5.log (3049 bytes)
9982023-10-24T17:44:53.835Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2180ccaf-8975-4480-b9ce-caddfd91a175/root/var/svc/log/oxide-crucible-downstairs:downstairs-c80fa30f-4f6a-447f-baf8-846fa2e3f8f5.log
9992023-10-24T17:44:53.859Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_e8a82cce-83b4-4a43-8099-379e5d5622f0/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10002023-10-24T17:44:54.933Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_e8a82cce-83b4-4a43-8099-379e5d5622f0/root/var/svc/log/oxide-crucible-pantry:default.log
10012023-10-24T17:44:54.958Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_5c8d0ec7-8e5f-4e1c-abb6-c99a5ed866ae/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10022023-10-24T17:44:56.026Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_5c8d0ec7-8e5f-4e1c-abb6-c99a5ed866ae/root/var/svc/log/oxide-cockroachdb:default.log
10032023-10-24T17:44:56.050Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_0b1052f8-1c6e-4e76-b7a8-38db75c986e0/root/var/svc/log/oxide-crucible-agent:default.log (13686 bytes)
10042023-10-24T17:44:57.165Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_0b1052f8-1c6e-4e76-b7a8-38db75c986e0/root/var/svc/log/oxide-crucible-agent:default.log
10052023-10-24T17:44:57.198Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_0b1052f8-1c6e-4e76-b7a8-38db75c986e0/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10062023-10-24T17:44:58.272Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_0b1052f8-1c6e-4e76-b7a8-38db75c986e0/root/var/svc/log/oxide-crucible-downstairs:default.log
10072023-10-24T17:44:58.297Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_0b1052f8-1c6e-4e76-b7a8-38db75c986e0/root/var/svc/log/oxide-crucible-downstairs:downstairs-b7c754ce-83c5-4649-a32f-adf652d53c35.log (5535 bytes)
10082023-10-24T17:44:59.374Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_0b1052f8-1c6e-4e76-b7a8-38db75c986e0/root/var/svc/log/oxide-crucible-downstairs:downstairs-b7c754ce-83c5-4649-a32f-adf652d53c35.log
10092023-10-24T17:44:59.400Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_929b4b09-3e9d-4741-b3ba-4d8749102e59/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10102023-10-24T17:45:00.467Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_929b4b09-3e9d-4741-b3ba-4d8749102e59/root/var/svc/log/oxide-crucible-agent:default.log
10112023-10-24T17:45:00.492Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_929b4b09-3e9d-4741-b3ba-4d8749102e59/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10122023-10-24T17:45:01.559Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_929b4b09-3e9d-4741-b3ba-4d8749102e59/root/var/svc/log/oxide-crucible-downstairs:default.log
10132023-10-24T17:45:01.585Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_bc9518ea-845a-4ac1-9ccc-e8652deaed36/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10142023-10-24T17:45:02.654Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_bc9518ea-845a-4ac1-9ccc-e8652deaed36/root/var/svc/log/oxide-crucible-agent:default.log
10152023-10-24T17:45:02.680Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_bc9518ea-845a-4ac1-9ccc-e8652deaed36/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10162023-10-24T17:45:02.704Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_bc9518ea-845a-4ac1-9ccc-e8652deaed36/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10172023-10-24T17:45:03.749Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_bc9518ea-845a-4ac1-9ccc-e8652deaed36/root/var/svc/log/oxide-crucible-downstairs:default.log
10182023-10-24T17:45:03.773Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_eed571dd-8d5a-421a-b82d-bcf8b82b7f25/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10192023-10-24T17:45:03.797Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_eed571dd-8d5a-421a-b82d-bcf8b82b7f25/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 3436 -> 0
10202023-10-24T17:45:04.847Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_eed571dd-8d5a-421a-b82d-bcf8b82b7f25/root/var/svc/log/oxide-crucible-agent:default.log
10212023-10-24T17:45:04.871Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_eed571dd-8d5a-421a-b82d-bcf8b82b7f25/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10222023-10-24T17:45:04.900Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_eed571dd-8d5a-421a-b82d-bcf8b82b7f25/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10232023-10-24T17:45:05.959Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_eed571dd-8d5a-421a-b82d-bcf8b82b7f25/root/var/svc/log/oxide-crucible-downstairs:default.log
10242023-10-24T17:45:05.984Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_5eaccb70-19d0-410f-a47e-915b0451771b/root/var/svc/log/oxide-external_dns:default.log (15614 bytes)
10252023-10-24T17:45:06.008Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_5eaccb70-19d0-410f-a47e-915b0451771b/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15614 -> 0
10262023-10-24T17:45:07.055Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_5eaccb70-19d0-410f-a47e-915b0451771b/root/var/svc/log/oxide-external_dns:default.log
10272023-10-24T17:45:07.080Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_807f2e33-7c55-4b46-afdd-cdcbb5ca677d/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10282023-10-24T17:45:07.104Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_807f2e33-7c55-4b46-afdd-cdcbb5ca677d/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 2338 -> 0
10292023-10-24T17:45:08.156Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_807f2e33-7c55-4b46-afdd-cdcbb5ca677d/root/var/svc/log/oxide-crucible-pantry:default.log
10302023-10-24T17:45:08.181Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_0e2c63ce-6484-4683-a6dd-68bd73b99491/root/var/svc/log/oxide-internal_dns:default.log (102320 bytes)
10312023-10-24T17:45:08.205Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_0e2c63ce-6484-4683-a6dd-68bd73b99491/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 102320 -> 0
10322023-10-24T17:45:09.254Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_0e2c63ce-6484-4683-a6dd-68bd73b99491/root/var/svc/log/oxide-internal_dns:default.log
10332023-10-24T17:45:09.278Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_7e6331e9-3a11-4a45-a00e-a23aae0dcab6/root/var/svc/log/oxide-nexus:default.log (495698 bytes)
10342023-10-24T17:45:09.302Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_7e6331e9-3a11-4a45-a00e-a23aae0dcab6/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 495698 -> 608
10352023-10-24T17:45:10.378Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_7e6331e9-3a11-4a45-a00e-a23aae0dcab6/root/var/svc/log/oxide-nexus:default.log
10362023-10-24T17:45:10.404Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_4acb5951-5375-4921-be03-c372181b23f4/root/var/svc/log/oxide-oximeter:default.log (187366 bytes)
10372023-10-24T17:45:10.458Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_4acb5951-5375-4921-be03-c372181b23f4/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 187366 -> 7831
10382023-10-24T17:45:11.559Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_4acb5951-5375-4921-be03-c372181b23f4/root/var/svc/log/oxide-oximeter:default.log
10392023-10-24T17:45:11.584Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_958012bd-15b0-4950-9e01-64ccecc743c9/root/var/svc/log/oxide-internal_dns:default.log (103546 bytes)
10402023-10-24T17:45:11.608Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_958012bd-15b0-4950-9e01-64ccecc743c9/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 103546 -> 0
10412023-10-24T17:45:12.657Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_958012bd-15b0-4950-9e01-64ccecc743c9/root/var/svc/log/oxide-internal_dns:default.log
10422023-10-24T17:45:12.680Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_0b053114-f326-4ae3-9a5e-ba32ed5645b8/root/var/svc/log/oxide-nexus:default.log (4716052 bytes)
10432023-10-24T17:45:12.705Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_0b053114-f326-4ae3-9a5e-ba32ed5645b8/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4716052 -> 609
10442023-10-24T17:45:13.777Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_0b053114-f326-4ae3-9a5e-ba32ed5645b8/root/var/svc/log/oxide-nexus:default.log
10452023-10-24T17:45:13.801Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3f96ce4a-8d96-4788-bc26-3e25c9285afe/root/var/svc/log/oxide-crucible-agent:default.log (14114 bytes)
10462023-10-24T17:45:13.825Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3f96ce4a-8d96-4788-bc26-3e25c9285afe/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 14114 -> 0
10472023-10-24T17:45:14.879Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3f96ce4a-8d96-4788-bc26-3e25c9285afe/root/var/svc/log/oxide-crucible-agent:default.log
10482023-10-24T17:45:14.904Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3f96ce4a-8d96-4788-bc26-3e25c9285afe/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10492023-10-24T17:45:15.316Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3f96ce4a-8d96-4788-bc26-3e25c9285afe/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10502023-10-24T17:45:16.389Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3f96ce4a-8d96-4788-bc26-3e25c9285afe/root/var/svc/log/oxide-crucible-downstairs:default.log
10512023-10-24T17:45:16.413Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3f96ce4a-8d96-4788-bc26-3e25c9285afe/root/var/svc/log/oxide-crucible-downstairs:downstairs-2c37b4a5-87c2-4a3e-bee3-47847840fc4f.log (3050 bytes)
10522023-10-24T17:45:16.437Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3f96ce4a-8d96-4788-bc26-3e25c9285afe/root/var/svc/log/oxide-crucible-downstairs:downstairs-2c37b4a5-87c2-4a3e-bee3-47847840fc4f.log" changed size mid upload: 3050 -> 0
10532023-10-24T17:45:17.487Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3f96ce4a-8d96-4788-bc26-3e25c9285afe/root/var/svc/log/oxide-crucible-downstairs:downstairs-2c37b4a5-87c2-4a3e-bee3-47847840fc4f.log
10542023-10-24T17:45:17.511Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b2c27b66-f9f1-4a58-8f48-02ff61f9f586/root/var/svc/log/oxide-crucible-agent:default.log (24829 bytes)
10552023-10-24T17:45:17.534Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b2c27b66-f9f1-4a58-8f48-02ff61f9f586/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 24829 -> 0
10562023-10-24T17:45:18.583Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b2c27b66-f9f1-4a58-8f48-02ff61f9f586/root/var/svc/log/oxide-crucible-agent:default.log
10572023-10-24T17:45:18.607Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b2c27b66-f9f1-4a58-8f48-02ff61f9f586/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10582023-10-24T17:45:18.631Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b2c27b66-f9f1-4a58-8f48-02ff61f9f586/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10592023-10-24T17:45:19.680Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b2c27b66-f9f1-4a58-8f48-02ff61f9f586/root/var/svc/log/oxide-crucible-downstairs:default.log
10602023-10-24T17:45:19.703Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b2c27b66-f9f1-4a58-8f48-02ff61f9f586/root/var/svc/log/oxide-crucible-downstairs:downstairs-4d4e7740-392b-4865-ae4f-5da1e93601e6.log (3049 bytes)
10612023-10-24T17:45:19.727Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b2c27b66-f9f1-4a58-8f48-02ff61f9f586/root/var/svc/log/oxide-crucible-downstairs:downstairs-4d4e7740-392b-4865-ae4f-5da1e93601e6.log" changed size mid upload: 3049 -> 0
10622023-10-24T17:45:20.800Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b2c27b66-f9f1-4a58-8f48-02ff61f9f586/root/var/svc/log/oxide-crucible-downstairs:downstairs-4d4e7740-392b-4865-ae4f-5da1e93601e6.log
10632023-10-24T17:45:20.824Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b2c27b66-f9f1-4a58-8f48-02ff61f9f586/root/var/svc/log/oxide-crucible-downstairs:downstairs-a1c185a4-2018-44f0-bbd8-cf25c6953784.log (5533 bytes)
10642023-10-24T17:45:20.854Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b2c27b66-f9f1-4a58-8f48-02ff61f9f586/root/var/svc/log/oxide-crucible-downstairs:downstairs-a1c185a4-2018-44f0-bbd8-cf25c6953784.log" changed size mid upload: 5533 -> 0
10652023-10-24T17:45:21.910Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_b2c27b66-f9f1-4a58-8f48-02ff61f9f586/root/var/svc/log/oxide-crucible-downstairs:downstairs-a1c185a4-2018-44f0-bbd8-cf25c6953784.log
10662023-10-24T17:45:21.934Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0ed05f52-6719-4227-a1e7-36a8bc8311db/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
10672023-10-24T17:45:21.961Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0ed05f52-6719-4227-a1e7-36a8bc8311db/root/var/svc/log/oxide-cockroachdb:default.log" changed size mid upload: 5343 -> 0
10682023-10-24T17:45:23.005Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0ed05f52-6719-4227-a1e7-36a8bc8311db/root/var/svc/log/oxide-cockroachdb:default.log
10692023-10-24T17:45:23.029Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e7e6e37c-d5b5-4662-a447-9659b73f5fba/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10702023-10-24T17:45:23.054Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e7e6e37c-d5b5-4662-a447-9659b73f5fba/root/var/svc/log/oxide-cockroachdb:default.log" changed size mid upload: 5369 -> 0
10712023-10-24T17:45:24.105Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e7e6e37c-d5b5-4662-a447-9659b73f5fba/root/var/svc/log/oxide-cockroachdb:default.log
10722023-10-24T17:45:24.130Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f9d86c20-436b-4f32-a81d-d1f3d4bc1a8b/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10732023-10-24T17:45:24.154Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f9d86c20-436b-4f32-a81d-d1f3d4bc1a8b/root/var/svc/log/oxide-cockroachdb:default.log" changed size mid upload: 5369 -> 0
10742023-10-24T17:45:25.211Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f9d86c20-436b-4f32-a81d-d1f3d4bc1a8b/root/var/svc/log/oxide-cockroachdb:default.log
10752023-10-24T17:45:25.235Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_abfa60b4-5f72-49eb-897b-6390cc58c84c/root/var/svc/log/oxide-crucible-agent:default.log (14119 bytes)
10762023-10-24T17:45:25.259Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_abfa60b4-5f72-49eb-897b-6390cc58c84c/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 14119 -> 0
10772023-10-24T17:45:26.422Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_abfa60b4-5f72-49eb-897b-6390cc58c84c/root/var/svc/log/oxide-crucible-agent:default.log
10782023-10-24T17:45:26.482Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_abfa60b4-5f72-49eb-897b-6390cc58c84c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10792023-10-24T17:45:26.505Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_abfa60b4-5f72-49eb-897b-6390cc58c84c/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10802023-10-24T17:45:27.556Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_abfa60b4-5f72-49eb-897b-6390cc58c84c/root/var/svc/log/oxide-crucible-downstairs:default.log
10812023-10-24T17:45:27.580Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_abfa60b4-5f72-49eb-897b-6390cc58c84c/root/var/svc/log/oxide-crucible-downstairs:downstairs-c2df7ee7-1a07-43e6-817b-98fbfa30fae6.log (5535 bytes)
10822023-10-24T17:45:27.603Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_abfa60b4-5f72-49eb-897b-6390cc58c84c/root/var/svc/log/oxide-crucible-downstairs:downstairs-c2df7ee7-1a07-43e6-817b-98fbfa30fae6.log" changed size mid upload: 5535 -> 0
10832023-10-24T17:45:28.651Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_abfa60b4-5f72-49eb-897b-6390cc58c84c/root/var/svc/log/oxide-crucible-downstairs:downstairs-c2df7ee7-1a07-43e6-817b-98fbfa30fae6.log
10842023-10-24T17:45:28.675Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ae834b01-961b-4e4b-89cb-fa20da0dfdde/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10852023-10-24T17:45:28.703Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ae834b01-961b-4e4b-89cb-fa20da0dfdde/root/var/svc/log/oxide-crucible-agent:default.log" changed size mid upload: 3436 -> 0
10862023-10-24T17:45:29.753Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ae834b01-961b-4e4b-89cb-fa20da0dfdde/root/var/svc/log/oxide-crucible-agent:default.log
10872023-10-24T17:45:29.779Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ae834b01-961b-4e4b-89cb-fa20da0dfdde/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10882023-10-24T17:45:29.805Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ae834b01-961b-4e4b-89cb-fa20da0dfdde/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
10892023-10-24T17:45:30.855Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ae834b01-961b-4e4b-89cb-fa20da0dfdde/root/var/svc/log/oxide-crucible-downstairs:default.log
10902023-10-24T17:45:30.879Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_2303dccc-c179-432f-92f5-afa8b85664e4/root/var/svc/log/oxide-external_dns:default.log (15615 bytes)
10912023-10-24T17:45:30.903Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_2303dccc-c179-432f-92f5-afa8b85664e4/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15615 -> 3777
10922023-10-24T17:45:31.976Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_2303dccc-c179-432f-92f5-afa8b85664e4/root/var/svc/log/oxide-external_dns:default.log
10932023-10-24T17:45:31.999Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_61886626-1d97-40db-b676-685b5e907008/root/var/svc/log/oxide-internal_dns:default.log (99571 bytes)
10942023-10-24T17:45:32.046Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_61886626-1d97-40db-b676-685b5e907008/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99571 -> 8796
10952023-10-24T17:45:33.135Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_61886626-1d97-40db-b676-685b5e907008/root/var/svc/log/oxide-internal_dns:default.log
10962023-10-24T17:45:33.159Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0/root/var/svc/log/oxide-ntp:default.log (1244 bytes)
10972023-10-24T17:45:33.182Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0/root/var/svc/log/oxide-ntp:default.log" changed size mid upload: 1244 -> 0
10982023-10-24T17:45:34.222Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0/root/var/svc/log/oxide-ntp:default.log
10992023-10-24T17:45:34.246Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_dc67e7e8-315e-4133-b7e6-8861d06f810f/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
11002023-10-24T17:45:34.271Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_dc67e7e8-315e-4133-b7e6-8861d06f810f/root/var/svc/log/oxide-cockroachdb:default.log" changed size mid upload: 5369 -> 0
11012023-10-24T17:45:35.314Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_dc67e7e8-315e-4133-b7e6-8861d06f810f/root/var/svc/log/oxide-cockroachdb:default.log
11022023-10-24T17:45:35.338Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_8300213d-f6d4-4e4b-a1ae-37602d1d922d/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
11032023-10-24T17:45:35.361Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_8300213d-f6d4-4e4b-a1ae-37602d1d922d/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 2338 -> 0
11042023-10-24T17:45:36.409Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_8300213d-f6d4-4e4b-a1ae-37602d1d922d/root/var/svc/log/oxide-crucible-pantry:default.log
11052023-10-24T17:45:36.440Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0/root/var/log/chrony/measurements.log (17536 bytes)
11062023-10-24T17:45:36.504Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0/root/var/log/chrony/measurements.log" changed size mid upload: 17536 -> 20139
11072023-10-24T17:45:37.576Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0/root/var/log/chrony/measurements.log
11082023-10-24T17:45:37.601Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0/root/var/log/chrony/statistics.log (9594 bytes)
11092023-10-24T17:45:37.648Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0/root/var/log/chrony/statistics.log" changed size mid upload: 9594 -> 11349
11102023-10-24T17:45:38.721Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0/root/var/log/chrony/statistics.log
11112023-10-24T17:45:38.745Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0/root/var/log/chrony/tracking.log (1855 bytes)
11122023-10-24T17:45:38.769Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0/root/var/log/chrony/tracking.log" changed size mid upload: 1855 -> 2119
11132023-10-24T17:45:39.846Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_a531f9ed-474c-42f7-88f5-27c7dfb62df0/root/var/log/chrony/tracking.log