01HDNM3G1CYS3DBSXF8ZDJFXQY: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HDNM84HDS691XCYH6880SQ9N

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-26T10:22:12.393Zjob dependencies complete; ready to run (waiting for 1 h 7 m 7 s)
22023-10-26T11:33:28.681Zjob assigned to worker 01HDNW48SATKG939YKERA7J37B (queued for 1 h 11 m 16 s)
32023-10-26T11:33:28.747Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-26T11:33:28.792Zxcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd
52023-10-26T11:33:28.825Zhttp://172.20.0.5:9969/ipxe/buskin... ok
62023-10-26T11:33:29.263Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
72023-10-26T11:33:29.308Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
82023-10-26T11:33:29.349Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
92023-10-26T11:33:29.383Zhttp://172.20.0.5:9969/postboot/buskin... ok
102023-10-26T11:33:29.428ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
112023-10-26T11:33:29.484Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
122023-10-26T11:33:29.540ZOxide Helios Version helios-2.0.22241 64-bit
132023-10-26T11:33:29.573ZConfiguring device.
142023-10-26T11:33:29.713Z##################################################
152023-10-26T11:33:29.904Z#### ############# ##
162023-10-26T11:33:29.974Z### ### ############ ##
172023-10-26T11:33:30.017Z## ### # ## ### ## Oxide Computer Company ##
182023-10-26T11:33:30.072Z## ## ## ### # ### ##
192023-10-26T11:33:30.108Z## # ### #### #### This Station Under ##
202023-10-26T11:33:30.144Z### ### #### # ### Computer Control ##
212023-10-26T11:33:30.179Z#### #### ### ## ##
222023-10-26T11:33:30.218Z##################################################
232023-10-26T11:33:30.253ZHostname: buskin
242023-10-26T11:33:30.286ZLoading smf(7) serice descriptions:
252023-10-26T11:33:31.223Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-ce25c26538ce669c.gz
262023-10-26T11:33:33.353Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-ce25c26538ce669c.gz
272023-10-26T11:33:33.378Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
282023-10-26T11:33:56.941Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
292023-10-26T11:33:56.974Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-bcb3df1a66c8d052.gz
302023-10-26T11:34:22.493Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-bcb3df1a66c8d052.gz
312023-10-26T11:34:22.520Zdownloading input: /input/ci-tools/work/caboose-util.gz
322023-10-26T11:34:23.539Zdownloaded input: /input/ci-tools/work/caboose-util.gz
332023-10-26T11:34:23.563Zdownloading input: /input/ci-tools/work/tufaceous.gz
342023-10-26T11:34:27.699Zdownloaded input: /input/ci-tools/work/tufaceous.gz
352023-10-26T11:34:27.742Zdownloading input: /input/package/work/version.txt
362023-10-26T11:34:27.771Zdownloaded input: /input/package/work/version.txt
372023-10-26T11:34:27.799Zdownloading input: /input/package/work/package.tar.gz
382023-10-26T11:34:46.167Zdownloaded input: /input/package/work/package.tar.gz
392023-10-26T11:34:46.453Zdownloading input: /input/package/work/global-zone-packages.tar.gz
402023-10-26T11:35:45.125Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
412023-10-26T11:35:45.879Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
422023-10-26T11:35:53.552Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
432023-10-26T11:35:53.576Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
442023-10-26T11:36:48.527Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
452023-10-26T11:36:48.563Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
462023-10-26T11:37:41.695Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
472023-10-26T11:37:41.729Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
482023-10-26T11:38:16.789Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
492023-10-26T11:38:16.815Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
502023-10-26T11:38:31.062Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
512023-10-26T11:38:31.111Zdownloading input: /input/package/work/zones/crucible.tar.gz
522023-10-26T11:38:40.537Zdownloaded input: /input/package/work/zones/crucible.tar.gz
532023-10-26T11:38:49.093Zdownloading input: /input/package/work/zones/external-dns.tar.gz
542023-10-26T11:38:54.778Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
552023-10-26T11:38:54.804Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
562023-10-26T11:39:00.547Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
572023-10-26T11:39:00.572Zdownloading input: /input/package/work/zones/ntp.tar.gz
582023-10-26T11:39:00.602Zdownloaded input: /input/package/work/zones/ntp.tar.gz
592023-10-26T11:39:00.626Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
602023-10-26T11:39:05.235Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
612023-10-26T11:39:05.265Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
622023-10-26T11:39:10.444Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
632023-10-26T11:39:10.469Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
642023-10-26T11:39:24.671Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
652023-10-26T11:39:24.726Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
662023-10-26T11:39:38.215Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
672023-10-26T11:39:38.267Zdownloading input: /input/package/work/zones/overlay.tar.gz
682023-10-26T11:39:38.312Zdownloaded input: /input/package/work/zones/overlay.tar.gz
692023-10-26T11:39:38.345Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
702023-10-26T11:39:54.616Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
712023-10-26T11:39:54.693Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
722023-10-26T11:40:09.912Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
732023-10-26T11:40:09.939Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
742023-10-26T11:40:57.098Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
752023-10-26T11:40:57.123Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
762023-10-26T11:41:26.381Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
772023-10-26T11:41:26.409Zstarting task 0: "setup"
782023-10-26T11:41:26.447Z++ uname -s
792023-10-26T11:41:26.471Z+ kern=SunOS
802023-10-26T11:41:26.496Z+ case "$kern" in
812023-10-26T11:41:26.520Z+ groupadd -g 12345 build
822023-10-26T11:41:26.549Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
832023-10-26T11:41:28.454Z+ zfs create -o mountpoint=/work rpool/work
842023-10-26T11:41:28.916Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
852023-10-26T11:41:28.946Z+ home_fs=zfs
862023-10-26T11:41:28.972Z+ [[ zfs == autofs ]]
872023-10-26T11:41:28.999Z+ mkdir -p /home/build
882023-10-26T11:41:29.024Z+ chown build:build /home/build /work
892023-10-26T11:41:30.476Z+ chmod 0700 /home/build /work
902023-10-26T11:41:30.502Zprocess exited: duration 4045 ms, exit code 0
 
912023-10-26T11:41:30.551Zstarting task 1: "authentication"
922023-10-26T11:41:30.620Zprocess exited: duration 45 ms, exit code 0
 
932023-10-26T11:41:30.671Zstarting task 2: "build"
942023-10-26T11:41:30.703Z+ trap _exit_trap EXIT
952023-10-26T11:41:30.727Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
962023-10-26T11:41:30.899Zsecpolicy_fs_linkdir: 0x55 = 0x55
972023-10-26T11:41:30.924Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
982023-10-26T11:41:30.949Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
992023-10-26T11:41:30.977Zsecpolicy_fs_linkdir+3: 0 = 0xe5
1002023-10-26T11:41:31.002Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1012023-10-26T11:41:31.027Zsecpolicy_fs_linkdir+5: 0 = 0x1
1022023-10-26T11:41:31.053Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1032023-10-26T11:41:31.077Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1042023-10-26T11:41:31.106Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1052023-10-26T11:41:31.131Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1062023-10-26T11:41:31.155Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1072023-10-26T11:41:31.180Z+ [[ -d /opt/oxide ]]
1082023-10-26T11:41:31.204Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1092023-10-26T11:41:31.232Z+ pfexec mkdir /opt/oxide
1102023-10-26T11:41:31.257Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1112023-10-26T11:41:31.358Z+ [[ -d /opt/oxide-underneath ]]
1122023-10-26T11:41:31.383Z+ cd /opt/oxide-underneath
1132023-10-26T11:41:31.407Z+ pfexec tar ceEp@/f - .
1142023-10-26T11:41:31.433Z+ cd /opt/oxide
1152023-10-26T11:41:31.457Z+ pfexec tar xveEp@/f -
1162023-10-26T11:41:31.482Zx ., 0 bytes, 0 tape blocks
1172023-10-26T11:41:31.507Zx ./opte, 0 bytes, 0 tape blocks
1182023-10-26T11:41:31.531Zx ./opte/bin, 0 bytes, 0 tape blocks
1192023-10-26T11:41:31.558Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1202023-10-26T11:41:31.586Z+ pfexec rm -rf /opt/oxide-underneath
1212023-10-26T11:41:31.614Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1222023-10-26T11:41:31.640Z+ pfexec mkdir /opt/oxide/work
1232023-10-26T11:41:31.666Z+ pfexec chown build:build /opt/oxide/work
1242023-10-26T11:41:31.696Z+ cd /opt/oxide/work
1252023-10-26T11:41:31.724Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1262023-10-26T11:41:32.604ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1272023-10-26T11:41:32.634Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1282023-10-26T11:41:32.659Zx out/omicron-sled-agent.tar, 115091968 bytes, 224789 tape blocks
1292023-10-26T11:41:32.683Zx out/target/test, 75 bytes, 1 tape blocks
1302023-10-26T11:41:32.708Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1312023-10-26T11:41:32.732Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1322023-10-26T11:41:32.757Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1332023-10-26T11:41:32.782Zx target/release/omicron-package, 38187008 bytes, 74584 tape blocks
1342023-10-26T11:41:32.807Zx tools/create_virtual_hardware.sh, 2974 bytes, 6 tape blocks
1352023-10-26T11:41:32.831Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1362023-10-26T11:41:32.855Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1372023-10-26T11:41:32.882Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1382023-10-26T11:41:32.907Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1392023-10-26T11:41:32.932Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1402023-10-26T11:41:32.959Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1412023-10-26T11:41:32.984Z
1422023-10-26T11:41:33.010Zreal 1.545341343
1432023-10-26T11:41:33.035Zuser 1.394304866
1442023-10-26T11:41:33.060Zsys 0.267889502
1452023-10-26T11:41:33.085Ztrap 0.000045930
1462023-10-26T11:41:33.109Ztflt 0.000000000
1472023-10-26T11:41:33.133Zdflt 0.000000000
1482023-10-26T11:41:33.158Zkflt 0.000000000
1492023-10-26T11:41:33.183Zlock 0.000000000
1502023-10-26T11:41:33.207Zslp 1.393628138
1512023-10-26T11:41:33.234Zlat 0.032541185
1522023-10-26T11:41:33.261Zstop 0.000121861
1532023-10-26T11:41:33.287Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1542023-10-26T11:41:33.631Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1552023-10-26T11:41:33.655Z+ mkdir tests
1562023-10-26T11:41:33.680Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1572023-10-26T11:41:33.704Z+ ptime -m gunzip
1582023-10-26T11:41:33.728Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-ce25c26538ce669c
1592023-10-26T11:41:33.791Z
1602023-10-26T11:41:33.816Zreal 0.142071394
1612023-10-26T11:41:33.840Zuser 0.129662282
1622023-10-26T11:41:33.866Zsys 0.012170178
1632023-10-26T11:41:33.895Ztrap 0.000013310
1642023-10-26T11:41:34.008Ztflt 0.000000000
1652023-10-26T11:41:34.034Zdflt 0.000000000
1662023-10-26T11:41:34.058Zkflt 0.000000000
1672023-10-26T11:41:34.083Zlock 0.000000000
1682023-10-26T11:41:34.109Zslp 0.000022980
1692023-10-26T11:41:34.134Zlat 0.000018750
1702023-10-26T11:41:34.159Zstop 0.000116842
1712023-10-26T11:41:34.183Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-ce25c26538ce669c
1722023-10-26T11:41:34.208Z+ chmod a+x tests/bootstrap-ce25c26538ce669c
1732023-10-26T11:41:34.242Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1742023-10-26T11:41:34.267Z+ ptime -m gunzip
1752023-10-26T11:41:34.291Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1762023-10-26T11:41:36.100Z
1772023-10-26T11:41:36.124Zreal 2.295645745
1782023-10-26T11:41:36.183Zuser 2.151990160
1792023-10-26T11:41:36.207Zsys 0.143244531
1802023-10-26T11:41:36.232Ztrap 0.000175419
1812023-10-26T11:41:36.256Ztflt 0.000000000
1822023-10-26T11:41:36.280Zdflt 0.000000000
1832023-10-26T11:41:36.305Zkflt 0.000000000
1842023-10-26T11:41:36.329Zlock 0.000000000
1852023-10-26T11:41:36.353Zslp 0.000025340
1862023-10-26T11:41:36.377Zlat 0.000023590
1872023-10-26T11:41:36.407Zstop 0.000125032
1882023-10-26T11:41:36.435Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1892023-10-26T11:41:36.460Z+ chmod a+x tests/bootstrap
1902023-10-26T11:41:36.486Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1912023-10-26T11:41:36.511Z+ ptime -m gunzip
1922023-10-26T11:41:36.550Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-bcb3df1a66c8d052
1932023-10-26T11:41:38.653Z
1942023-10-26T11:41:38.678Zreal 2.540836810
1952023-10-26T11:41:38.703Zuser 2.399637579
1962023-10-26T11:41:38.728Zsys 0.140764215
1972023-10-26T11:41:38.756Ztrap 0.000197693
1982023-10-26T11:41:38.787Ztflt 0.000000000
1992023-10-26T11:41:39.222Zdflt 0.000000000
2002023-10-26T11:41:39.266Zkflt 0.000000000
2012023-10-26T11:41:39.294Zlock 0.000000000
2022023-10-26T11:41:39.319Zslp 0.000021000
2032023-10-26T11:41:39.344Zlat 0.000022020
2042023-10-26T11:41:39.369Zstop 0.000119291
2052023-10-26T11:41:39.408Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-bcb3df1a66c8d052
2062023-10-26T11:41:39.433Z+ chmod a+x tests/end_to_end_tests-bcb3df1a66c8d052
2072023-10-26T11:41:39.457Z+ bmat address ls
2082023-10-26T11:41:39.482ZNAME CIDR FIRST LAST COUNT
2092023-10-26T11:41:39.506Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2102023-10-26T11:41:39.530Z++ bmat address ls -f extra -Ho first,last,count
2112023-10-26T11:41:39.554Z+ set -- 10.151.1.164 10.151.1.227 64
2122023-10-26T11:41:39.578Z+ EXTRA_IP_START=10.151.1.164
2132023-10-26T11:41:39.602Z+ EXTRA_IP_END=10.151.1.227
2142023-10-26T11:41:39.626Z+ EXTRA_IP_COUNT=64
2152023-10-26T11:41:39.677Z+ (( EXTRA_IP_COUNT >= 32 ))
2162023-10-26T11:41:39.702Z+ EXTRA_IP_BASE=10.151.1
2172023-10-26T11:41:39.726Z+ EXTRA_IP_FOCTET=164
2182023-10-26T11:41:39.750Z+ EXTRA_IP_LOCTET=227
2192023-10-26T11:41:39.775Z+ SERVICE_IP_POOL_START=10.151.1.164
2202023-10-26T11:41:39.799Z+ SERVICE_IP_POOL_END=10.151.1.173
2212023-10-26T11:41:39.866Z+ DNS_IP1=10.151.1.164
2222023-10-26T11:41:39.896Z+ DNS_IP2=10.151.1.165
2232023-10-26T11:41:39.921Z+ UPLINK_IP=10.151.1.174
2242023-10-26T11:41:39.946Z+ IPPOOL_START=10.151.1.175
2252023-10-26T11:41:39.970Z+ IPPOOL_END=10.151.1.227
2262023-10-26T11:41:39.995Z++ ipadm show-addr -po type,addr
2272023-10-26T11:41:40.019Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2282023-10-26T11:41:40.044Z+ GATEWAY_IP=10.151.1.100
2292023-10-26T11:41:40.075Z+ [[ -n 10.151.1.100 ]]
2302023-10-26T11:41:40.102Z+ ping -s 10.151.1.100 56 1
2312023-10-26T11:41:40.128ZPING 10.151.1.100: 56 data bytes
2322023-10-26T11:41:40.154Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.013 ms
2332023-10-26T11:41:40.217Z
2342023-10-26T11:41:40.243Z----10.151.1.100 PING Statistics----
2352023-10-26T11:41:40.267Z1 packets transmitted, 1 packets received, 0% packet loss
2362023-10-26T11:41:40.292Zround-trip (ms) min/avg/max/stddev = 0.013/0.013/0.013/-nan
2372023-10-26T11:41:40.316Z++ arp -an
2382023-10-26T11:41:40.340Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2392023-10-26T11:41:40.364Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2402023-10-26T11:41:40.388Z+ routeadm -e ipv4-forwarding -u
2412023-10-26T11:41:40.413Z+ PXA_START=10.151.1.164
2422023-10-26T11:41:40.438Z+ PXA_END=10.151.1.227
2432023-10-26T11:41:40.462Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2442023-10-26T11:41:40.487Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2452023-10-26T11:41:40.511Z+ ZPOOL_VDEV_DIR=/scratch
2462023-10-26T11:41:40.546Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2472023-10-26T11:41:40.576Z+++ dirname ./tools/create_virtual_hardware.sh
2482023-10-26T11:41:41.429Z++ cd ./tools
2492023-10-26T11:41:41.859Z++ pwd
2502023-10-26T11:41:42.028Z+ SOURCE_DIR=/opt/oxide/work/tools
2512023-10-26T11:41:42.061Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2522023-10-26T11:41:42.088Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2532023-10-26T11:41:42.114Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2542023-10-26T11:41:42.138Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2552023-10-26T11:41:42.163Z++ dladm show-phys -p -o LINK
2562023-10-26T11:41:42.188Z++ head -1
2572023-10-26T11:41:42.212Z+ PHYSICAL_LINK=igb0
2582023-10-26T11:41:42.236Z+ echo 'Using igb0 as physical link'
2592023-10-26T11:41:42.260ZUsing igb0 as physical link
2602023-10-26T11:41:42.284Z+ ensure_run_as_root
2612023-10-26T11:41:42.311Z++ id -u
2622023-10-26T11:41:42.339Z+ [[ 0 -ne 0 ]]
2632023-10-26T11:41:42.363Z+ ensure_zpools
2642023-10-26T11:41:42.388Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2652023-10-26T11:41:42.412ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2662023-10-26T11:41:42.436Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2672023-10-26T11:41:42.460ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2682023-10-26T11:41:42.487Z+ readarray -t ZPOOLS
2692023-10-26T11:41:42.514ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2702023-10-26T11:41:42.540Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2712023-10-26T11:41:42.570ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2722023-10-26T11:41:42.598Z++ sed 's/[ ",]//g'
2732023-10-26T11:41:42.623ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2742023-10-26T11:41:42.649Z+ for ZPOOL in "${ZPOOLS[@]}"
2752023-10-26T11:41:42.674ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2762023-10-26T11:41:42.699Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2772023-10-26T11:41:42.725ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2782023-10-26T11:41:42.752Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2792023-10-26T11:41:42.801ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2802023-10-26T11:41:42.829Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2812023-10-26T11:41:42.858ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2822023-10-26T11:41:42.888Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2832023-10-26T11:41:42.915ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2842023-10-26T11:41:42.939Z0+0 records in
2852023-10-26T11:41:42.963ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2862023-10-26T11:41:43.012Z0+0 records out
2872023-10-26T11:41:43.042ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2882023-10-26T11:41:43.068Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2892023-10-26T11:41:43.091ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-10-26T11:41:43.115Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2912023-10-26T11:41:43.143ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2922023-10-26T11:41:43.172Z+ set +x
2932023-10-26T11:41:43.200ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2942023-10-26T11:41:43.231Z++ zpool list -o name
2952023-10-26T11:41:43.260ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
2962023-10-26T11:41:43.696Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2972023-10-26T11:41:43.728ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
2982023-10-26T11:41:43.757Z+ [[ -z '' ]]
2992023-10-26T11:41:43.783ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3002023-10-26T11:41:43.809Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
3012023-10-26T11:41:43.877ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3022023-10-26T11:41:43.902Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
3032023-10-26T11:41:43.927ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3042023-10-26T11:41:43.955Z+ set +x
3052023-10-26T11:41:44.390ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3062023-10-26T11:41:44.428Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-10-26T11:41:44.465ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3082023-10-26T11:41:44.489Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3092023-10-26T11:41:44.513ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3102023-10-26T11:41:44.538Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3112023-10-26T11:41:44.562ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3122023-10-26T11:41:44.593Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3132023-10-26T11:41:44.630ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3142023-10-26T11:41:44.656Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3152023-10-26T11:41:44.681ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3162023-10-26T11:41:44.705Z0+0 records in
3172023-10-26T11:41:44.729ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3182023-10-26T11:41:44.753Z0+0 records out
3192023-10-26T11:41:44.777ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3202023-10-26T11:41:45.211Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3212023-10-26T11:41:45.237ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3222023-10-26T11:41:45.262Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3232023-10-26T11:41:45.287ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3242023-10-26T11:41:45.318Z+ set +x
3252023-10-26T11:41:45.342ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3262023-10-26T11:41:45.370Z++ zpool list -o name
3272023-10-26T11:41:45.397ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3282023-10-26T11:41:45.421Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3292023-10-26T11:41:45.446ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3302023-10-26T11:41:45.470Z+ [[ -z '' ]]
3312023-10-26T11:41:45.500ZSimnet net0/sc0_0 exists
3322023-10-26T11:41:45.533Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3332023-10-26T11:41:45.559ZSimnet net1/sc1_0 exists
3342023-10-26T11:41:45.587Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3352023-10-26T11:41:45.612ZVnic sc0_1 exists
3362023-10-26T11:41:45.640Z+ set +x
3372023-10-26T11:41:45.668Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3382023-10-26T11:41:45.695Z+ for ZPOOL in "${ZPOOLS[@]}"
3392023-10-26T11:41:45.730Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3402023-10-26T11:41:45.764Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-10-26T11:41:45.800Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3422023-10-26T11:41:45.824Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3432023-10-26T11:41:45.849Z0+0 records in
3442023-10-26T11:41:45.873Z0+0 records out
3452023-10-26T11:41:45.898Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3462023-10-26T11:41:45.979Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3472023-10-26T11:41:46.004Z+ set +x
3482023-10-26T11:41:46.028Z++ zpool list -o name
3492023-10-26T11:41:46.052Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3502023-10-26T11:41:46.078Z+ [[ -z '' ]]
3512023-10-26T11:41:46.102Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3522023-10-26T11:41:46.127Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3532023-10-26T11:41:46.151Z+ set +x
3542023-10-26T11:41:46.181Z+ for ZPOOL in "${ZPOOLS[@]}"
3552023-10-26T11:41:46.208Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3562023-10-26T11:41:46.689Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3572023-10-26T11:41:46.723Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3582023-10-26T11:41:46.752Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3592023-10-26T11:41:46.783Z0+0 records in
3602023-10-26T11:41:46.808Z0+0 records out
3612023-10-26T11:41:46.832Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3622023-10-26T11:41:46.857Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3632023-10-26T11:41:46.881Z+ set +x
3642023-10-26T11:41:46.906Z++ zpool list -o name
3652023-10-26T11:41:46.932Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3662023-10-26T11:41:46.957Z+ [[ -z '' ]]
3672023-10-26T11:41:47.012Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3682023-10-26T11:41:47.037Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3692023-10-26T11:41:47.065Z+ set +x
3702023-10-26T11:41:47.089Z+ for ZPOOL in "${ZPOOLS[@]}"
3712023-10-26T11:41:47.113Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3722023-10-26T11:41:47.149Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3732023-10-26T11:41:47.173Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3742023-10-26T11:41:47.196Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3752023-10-26T11:41:47.222Z0+0 records in
3762023-10-26T11:41:47.245Z0+0 records out
3772023-10-26T11:41:47.269Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3782023-10-26T11:41:47.293Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3792023-10-26T11:41:47.316Z+ set +x
3802023-10-26T11:41:47.340Z++ zpool list -o name
3812023-10-26T11:41:47.364Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3822023-10-26T11:41:47.387Z+ [[ -z '' ]]
3832023-10-26T11:41:47.412Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3842023-10-26T11:41:47.436Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3852023-10-26T11:41:47.459Z+ set +x
3862023-10-26T11:41:47.483Z+ for ZPOOL in "${ZPOOLS[@]}"
3872023-10-26T11:41:47.507Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3882023-10-26T11:41:47.531Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3892023-10-26T11:41:47.555Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3902023-10-26T11:41:47.992Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3912023-10-26T11:41:48.090Z0+0 records in
3922023-10-26T11:41:48.114Z0+0 records out
3932023-10-26T11:41:48.138Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3942023-10-26T11:41:48.162Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3952023-10-26T11:41:48.187Z+ set +x
3962023-10-26T11:41:48.211Z++ zpool list -o name
3972023-10-26T11:41:48.240Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3982023-10-26T11:41:48.274Z+ [[ -z '' ]]
3992023-10-26T11:41:48.301Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
4002023-10-26T11:41:48.327Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4012023-10-26T11:41:48.351Z+ set +x
4022023-10-26T11:41:48.375Z+ for ZPOOL in "${ZPOOLS[@]}"
4032023-10-26T11:41:48.406Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4042023-10-26T11:41:48.429Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4052023-10-26T11:41:48.453Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4062023-10-26T11:41:48.477Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4072023-10-26T11:41:48.500Z0+0 records in
4082023-10-26T11:41:48.524Z0+0 records out
4092023-10-26T11:41:48.550Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-10-26T11:41:48.574Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4112023-10-26T11:41:48.598Z+ set +x
4122023-10-26T11:41:48.621Z++ zpool list -o name
4132023-10-26T11:41:48.649Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4142023-10-26T11:41:49.088Z+ [[ -z '' ]]
4152023-10-26T11:41:49.121Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4162023-10-26T11:41:49.148Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4172023-10-26T11:41:49.173Z+ set +x
4182023-10-26T11:41:49.202Z+ for ZPOOL in "${ZPOOLS[@]}"
4192023-10-26T11:41:49.226Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4202023-10-26T11:41:49.251Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4212023-10-26T11:41:49.277Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4222023-10-26T11:41:49.303Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4232023-10-26T11:41:49.326Z0+0 records in
4242023-10-26T11:41:49.350Z0+0 records out
4252023-10-26T11:41:49.374Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4262023-10-26T11:41:49.398Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4272023-10-26T11:41:49.427Z+ set +x
4282023-10-26T11:41:49.455Z++ zpool list -o name
4292023-10-26T11:41:49.479Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4302023-10-26T11:41:49.502Z+ [[ -z '' ]]
4312023-10-26T11:41:49.527Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4322023-10-26T11:41:49.563Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4332023-10-26T11:41:49.591Z+ set +x
4342023-10-26T11:41:49.615Z+ for ZPOOL in "${ZPOOLS[@]}"
4352023-10-26T11:41:49.639Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4362023-10-26T11:41:49.663Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4372023-10-26T11:41:49.686Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4382023-10-26T11:41:49.711Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4392023-10-26T11:41:49.738Z0+0 records in
4402023-10-26T11:41:49.764Z0+0 records out
4412023-10-26T11:41:49.788Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4422023-10-26T11:41:49.812Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4432023-10-26T11:41:49.836Z+ set +x
4442023-10-26T11:41:49.865Z++ zpool list -o name
4452023-10-26T11:41:49.892Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4462023-10-26T11:41:49.916Z+ [[ -z '' ]]
4472023-10-26T11:41:49.942Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4482023-10-26T11:41:49.966Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4492023-10-26T11:41:49.995Z+ set +x
4502023-10-26T11:41:50.031Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4512023-10-26T11:41:50.058Z+ readarray -t ZPOOLS
4522023-10-26T11:41:50.082Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4532023-10-26T11:41:50.109Z++ sed 's/[ ",]//g'
4542023-10-26T11:41:50.158Zdownloading softnpu ...done
4552023-10-26T11:41:50.182Z+ for ZPOOL in "${ZPOOLS[@]}"
4562023-10-26T11:41:50.206Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
4572023-10-26T11:41:50.230Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4582023-10-26T11:41:50.254Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-10-26T11:41:50.278Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4602023-10-26T11:41:50.301Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4612023-10-26T11:41:50.325Z0+0 records in
4622023-10-26T11:41:50.349Z0+0 records out
4632023-10-26T11:41:50.373Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4642023-10-26T11:41:50.397Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4652023-10-26T11:41:50.424Z+ set +x
4662023-10-26T11:41:50.453Z++ zpool list -o name
4672023-10-26T11:41:50.483Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4682023-10-26T11:41:50.515Z+ [[ -z '' ]]
4692023-10-26T11:41:50.545Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4702023-10-26T11:41:50.569Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4712023-10-26T11:41:50.595Z+ set +x
4722023-10-26T11:41:50.620Z+ for ZPOOL in "${ZPOOLS[@]}"
4732023-10-26T11:41:50.644Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4742023-10-26T11:41:50.669Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4752023-10-26T11:41:50.694Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4762023-10-26T11:41:50.721Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4772023-10-26T11:41:50.747Z0+0 records in
4782023-10-26T11:41:50.772Z0+0 records out
4792023-10-26T11:41:50.796Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4802023-10-26T11:41:50.819Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4812023-10-26T11:41:50.843Z+ set +x
4822023-10-26T11:41:50.872Z++ zpool list -o name
4832023-10-26T11:41:52.120Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4842023-10-26T11:41:52.149Z+ [[ -z '' ]]
4852023-10-26T11:41:52.182Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4862023-10-26T11:41:52.222Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4872023-10-26T11:41:52.248Z+ set +x
4882023-10-26T11:41:52.273Z+ ensure_simulated_links igb0
4892023-10-26T11:41:52.366Z+ local PHYSICAL_LINK=igb0
4902023-10-26T11:41:52.390Z+ INDICES=("0" "1")
4912023-10-26T11:41:52.420Z+ for I in "${INDICES[@]}"
4922023-10-26T11:41:52.445Z++ get_simnet_name_if_exists net0
4932023-10-26T11:41:52.470Z++ dladm show-simnet -p -o LINK net0
4942023-10-26T11:41:52.494Z++ echo ''
4952023-10-26T11:41:52.518Z+ [[ -z '' ]]
4962023-10-26T11:41:52.543Z+ dladm create-simnet -t net0
4972023-10-26T11:41:52.567Z+ dladm create-simnet -t sc0_0
4982023-10-26T11:41:52.597Z+ dladm modify-simnet -t -p net0 sc0_0
4992023-10-26T11:41:52.643Z+ dladm set-linkprop -p mtu=9000 sc0_0
5002023-10-26T11:41:52.669Z+ success 'Simnet net0/sc0_0 exists'
5012023-10-26T11:41:52.693Z+ set +x
5022023-10-26T11:41:52.717Z+ for I in "${INDICES[@]}"
5032023-10-26T11:41:52.741Z++ get_simnet_name_if_exists net1
5042023-10-26T11:41:52.769Z++ dladm show-simnet -p -o LINK net1
5052023-10-26T11:41:52.795Z++ echo ''
5062023-10-26T11:41:52.819Z+ [[ -z '' ]]
5072023-10-26T11:41:52.842Z+ dladm create-simnet -t net1
5082023-10-26T11:41:52.866Z+ dladm create-simnet -t sc1_0
5092023-10-26T11:41:52.890Z+ dladm modify-simnet -t -p net1 sc1_0
5102023-10-26T11:41:52.913Z+ dladm set-linkprop -p mtu=9000 sc1_0
5112023-10-26T11:41:52.937Z+ success 'Simnet net1/sc1_0 exists'
5122023-10-26T11:41:52.965Z+ set +x
5132023-10-26T11:41:52.989Z++ get_vnic_name_if_exists sc0_1
5142023-10-26T11:41:53.013Z++ dladm show-vnic -p -o LINK sc0_1
5152023-10-26T11:41:53.448Z++ echo ''
5162023-10-26T11:41:53.473Z+ [[ -z '' ]]
5172023-10-26T11:41:53.498Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5182023-10-26T11:41:53.521Z+ [[ -v PROMISC_FILT_OFF ]]
5192023-10-26T11:41:53.545Z+ success 'Vnic sc0_1 exists'
5202023-10-26T11:41:53.568Z+ set +x
5212023-10-26T11:41:53.622Z+ warn_if_no_proxy_arp
5222023-10-26T11:41:53.647Z+ [[ -v PXA_START ]]
5232023-10-26T11:41:53.672Z+ [[ -v PXA_END ]]
5242023-10-26T11:41:53.697Z+ ensure_softnpu_zone
5252023-10-26T11:41:53.734Z+ zoneadm list
5262023-10-26T11:41:53.759Z+ grep -q sidecar_softnpu
5272023-10-26T11:41:53.783Z+ [[ -f out/npuzone/npuzone ]]
5282023-10-26T11:41:53.807Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch main
5292023-10-26T11:41:53.958Zdownloading asic_program.so ...done
5302023-10-26T11:41:54.009Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-10-26T11:42:00.914Zdownloading scadm ...done
5322023-10-26T11:42:00.940Zcreating port tfportrear0_0 for sc0_0
5332023-10-26T11:42:00.966Zcreating port tfportqsfp0_0 for sc0_1
5342023-10-26T11:42:00.994Zsoftnpu zone setup
5352023-10-26T11:42:01.023Zconfigure zone
5362023-10-26T11:42:01.048Zinstall zone
5372023-10-26T11:42:01.980Zboot zone
5382023-10-26T11:42:05.077Z[sidecar_softnpu] svcs milestone/network
5392023-10-26T11:42:05.106ZSTATE STIME FMRI
5402023-10-26T11:42:05.133Zonline 11:42:03 svc:/milestone/network:default
5412023-10-26T11:42:05.158Z[sidecar_softnpu] /softnpu/npu start
5422023-10-26T11:42:05.183Z
5432023-10-26T11:42:05.209Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-10-26T11:42:05.233Z+ GATEWAY_IP=10.151.1.100
5452023-10-26T11:42:05.258Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-10-26T11:42:05.283ZUsing 10.151.1.100 as gateway ip
5472023-10-26T11:42:05.310Z+ [[ ! -v GATEWAY_MAC ]]
5482023-10-26T11:42:05.334Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-10-26T11:42:05.362Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-10-26T11:42:05.388Z+++ echo 18:c0:4d:0d:9f:b2
5512023-10-26T11:42:05.413Z+ COUNT=1
5522023-10-26T11:42:05.437Z+ [[ 1 -ne 1 ]]
5532023-10-26T11:42:05.472Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-10-26T11:42:05.502ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-10-26T11:42:05.530Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-10-26T11:42:05.561Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-10-26T11:42:05.590Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-10-26T11:42:05.615Z+ [[ -v PXA_START ]]
5592023-10-26T11:42:05.639Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-10-26T11:42:05.665Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-10-26T11:42:05.689Z+ z_scadm dump-state
5622023-10-26T11:42:05.718Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-10-26T11:42:05.746Zlocal v6:
5642023-10-26T11:42:05.771Zlocal v4:
5652023-10-26T11:42:05.823Zrouter v6:
5662023-10-26T11:42:05.853Zrouter v4_idx:
5672023-10-26T11:42:05.877Zrouter v4_routes:
5682023-10-26T11:42:05.904Zresolver v4:
5692023-10-26T11:42:05.929Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-10-26T11:42:05.954Zresolver v6:
5712023-10-26T11:42:05.980Znat v4:
5722023-10-26T11:42:06.004Znat_v6:
5732023-10-26T11:42:06.031Zport_mac:
5742023-10-26T11:42:06.056Zproxy arp:
5752023-10-26T11:42:06.080Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-10-26T11:42:06.105Z+ success 'softnpu zone exists'
5772023-10-26T11:42:06.148Z+ set +x
5782023-10-26T11:42:06.174Zsoftnpu zone exists
5792023-10-26T11:42:06.199Z
5802023-10-26T11:42:06.225Zreal 26.245141502
5812023-10-26T11:42:06.253Zuser 0.708083642
5822023-10-26T11:42:06.279Zsys 1.520438942
5832023-10-26T11:42:06.304Ztrap 0.000107995
5842023-10-26T11:42:06.328Ztflt 0.000124935
5852023-10-26T11:42:06.352Zdflt 0.000936362
5862023-10-26T11:42:06.377Zkflt 0.000000000
5872023-10-26T11:42:06.403Zlock 20:32.773974113
5882023-10-26T11:42:06.459Zslp 1:17.978403132
5892023-10-26T11:42:06.489Zlat 0.054396958
5902023-10-26T11:42:06.513Zstop 0.366165213
5912023-10-26T11:42:06.538Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-10-26T11:42:06.567Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-10-26T11:42:06.591Z+ SILO_NAME=recovery
5942023-10-26T11:42:06.621Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-10-26T11:42:06.653Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-10-26T11:42:06.683Z+ sed -i~ '
5972023-10-26T11:42:06.707Z /^external_dns_ips/c\
5982023-10-26T11:42:06.733Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-10-26T11:42:06.758Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-10-26T11:42:06.784Z /^first/c\
6012023-10-26T11:42:06.810Zfirst = "10.151.1.164"
6022023-10-26T11:42:06.861Z /^last/c\
6032023-10-26T11:42:06.887Zlast = "10.151.1.173"
6042023-10-26T11:42:06.913Z }
6052023-10-26T11:42:06.938Z /^\[rack_network_config/,/^$/ {
6062023-10-26T11:42:06.968Z /^infra_ip_first/c\
6072023-10-26T11:42:06.993Zinfra_ip_first = "10.151.1.174"
6082023-10-26T11:42:07.017Z /^infra_ip_last/c\
6092023-10-26T11:42:07.043Zinfra_ip_last = "10.151.1.174"
6102023-10-26T11:42:07.068Z }
6112023-10-26T11:42:07.097Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6122023-10-26T11:42:07.125Z /^\[\[rack_network_config.ports/,/^$/ {
6132023-10-26T11:42:07.150Z+++ pkg/config-rss.toml Thu Oct 26 11:42:05 2023
6142023-10-26T11:42:07.175Z /^routes/c\
6152023-10-26T11:42:07.200Z@@ -41,7 +41,7 @@
6162023-10-26T11:42:07.225Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6172023-10-26T11:42:07.250Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6182023-10-26T11:42:07.284Z /^addresses/c\
6192023-10-26T11:42:07.310Z # must be contained in one of the "internal services" IP Pool ranges listed
6202023-10-26T11:42:07.335Zaddresses = \["10.151.1.174/32"\]
6212023-10-26T11:42:07.361Z # below.
6222023-10-26T11:42:07.385Z }
6232023-10-26T11:42:07.413Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6242023-10-26T11:42:07.437Z' pkg/config-rss.toml
6252023-10-26T11:42:07.461Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6262023-10-26T11:42:07.485Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6272023-10-26T11:42:07.559Z
6282023-10-26T11:42:07.585Z+ true
6292023-10-26T11:42:07.621Z # Initial TLS certificates for the external API
6302023-10-26T11:42:07.646Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6312023-10-26T11:42:07.670Z #
6322023-10-26T11:42:07.694Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6332023-10-26T11:42:07.719Z@@ -77,8 +77,8 @@
6342023-10-26T11:42:07.743Z+ TLS_NAME=recovery.sys.oxide.test
6352023-10-26T11:42:07.768Z #
6362023-10-26T11:42:07.792Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6372023-10-26T11:42:07.817Z # For more on this and what to put here, see docs/how-to-run.adoc.
6382023-10-26T11:42:07.841Z.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+....+.....+.+........+...+....+......+..+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+..+.............+...+..................+......+...........+....+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6392023-10-26T11:42:07.866Z [[internal_services_ip_pool_ranges]]
6402023-10-26T11:42:07.910Z-first = "192.168.1.20"
6412023-10-26T11:42:07.942Z-last = "192.168.1.29"
6422023-10-26T11:42:08.115Z+first = "10.151.1.164"
6432023-10-26T11:42:08.169Z+last = "10.151.1.173"
6442023-10-26T11:42:08.197Z
6452023-10-26T11:42:08.251Z # TODO - this configuration is subject to change going forward. Ultimately these
6462023-10-26T11:42:08.283Z # parameters should be provided to the control plane via wicket, but we need to
6472023-10-26T11:42:08.316Z@@ -92,8 +92,8 @@
6482023-10-26T11:42:08.364Z # A range of IP addresses used by Boundary Services on the external network. In
6492023-10-26T11:42:08.388Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6502023-10-26T11:42:08.430Z # With softnpu, only one address is used.
6512023-10-26T11:42:08.455Z-infra_ip_first = "192.168.1.30"
6522023-10-26T11:42:08.479Z-infra_ip_last = "192.168.1.30"
6532023-10-26T11:42:08.504Z..........+......+...+...+.........+....+...........+......+.........+.............+.....+......+.+...+...+........+.+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+....................+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+...............+.+......+.........+.....+......+.............+..+.............+..+............+...+...+.+........+...............+............+.+.....+...............+...+.........+..........+......+......+...+..................+...+......+.........+.....+....+...........+.......+...+..+...+......+.............+...........+....+...+..+..........+...+..............+.+..+.+..+...+....+...........+.+...........+.........+....+..+....+...+........+....+........+...+...................+...........+............+...+...+.+.........+.....+......+......+.......+...........+....+...+..............+..................+.............+.....+...+....+..................+.....+....+...+.....+.......+......+.........+...+............+......+...+...........+...................+..........................................+..+...+..................+.+.........+............+.....+..................+.........+.+...........+.........+...................+...+...+.....+.......+.....+............+..........+..+.........+......+......+............+.......+...+...........+..........+......+.....+.+..+.+..+.........................+........+.........................+..+..........+...............+..+......+......+....+..+.........+.+........+......+.........+.......+..+.+.....................+..............+....+..................+..+.......+......+.....+...+.+.....+....+.........+...............+..+..........+........+......+.......+..+.......+........+...........................+....+...............+...........+...+.+..............+............+.+..+.+...........+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6542023-10-26T11:42:08.570Z+infra_ip_first = "10.151.1.174"
6552023-10-26T11:42:08.603Z-----
6562023-10-26T11:42:08.637Z+infra_ip_last = "10.151.1.174"
6572023-10-26T11:42:08.662Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6582023-10-26T11:42:08.694Z
6592023-10-26T11:42:08.737Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6602023-10-26T11:42:08.771Z # Configurations for BGP routers to run on the scrimlets.
6612023-10-26T11:42:08.797Z+ rmdir pkg
6622023-10-26T11:42:08.825Z bgp = []
6632023-10-26T11:42:08.849Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6642023-10-26T11:42:08.875Z@@ -101,9 +101,9 @@
6652023-10-26T11:42:08.908Z+ pfexec mkdir -p /usr/oxide
6662023-10-26T11:42:08.975Z # You can configure multiple uplinks by repeating the following stanza
6672023-10-26T11:42:09.007Z+ pfexec rm -f /usr/oxide/tcpproxy
6682023-10-26T11:42:09.062Z [[rack_network_config.ports]]
6692023-10-26T11:42:09.110Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6702023-10-26T11:42:09.162Z # Routes associated with this port.
6712023-10-26T11:42:09.190Z+ pfexec chmod +x /usr/oxide/tcpproxy
6722023-10-26T11:42:09.266Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6732023-10-26T11:42:09.313Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6742023-10-26T11:42:09.345Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6752023-10-26T11:42:09.371Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6762023-10-26T11:42:09.413Z # Addresses associated with this port.
6772023-10-26T11:42:09.438Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6782023-10-26T11:42:09.464Z-addresses = ["192.168.1.30/32"]
6792023-10-26T11:42:09.503Z+ OMICRON_NO_UNINSTALL=1
6802023-10-26T11:42:09.588Z+addresses = ["10.151.1.174/32"]
6812023-10-26T11:42:17.891Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6822023-10-26T11:42:17.917Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6832023-10-26T11:42:17.943ZOct 26 11:42:07.163 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6842023-10-26T11:42:17.985Z port = "qsfp0"
6852023-10-26T11:42:18.011ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6862023-10-26T11:42:18.046Z # The speed of this port.
6872023-10-26T11:42:18.082ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6882023-10-26T11:42:18.115Za pkg/config-rss.toml 7K
6892023-10-26T11:42:18.140ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6902023-10-26T11:42:18.173Za pkg/initial-tls-cert.pem 2K
6912023-10-26T11:42:18.203ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6922023-10-26T11:42:18.234Za pkg/initial-tls-key.pem 4K
6932023-10-26T11:42:18.262ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6942023-10-26T11:42:18.291ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6952023-10-26T11:42:18.321ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6962023-10-26T11:42:18.346ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6972023-10-26T11:42:18.377ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6982023-10-26T11:42:18.415ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6992023-10-26T11:42:18.450ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7002023-10-26T11:42:18.478ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7012023-10-26T11:42:18.509ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7022023-10-26T11:42:18.537ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7032023-10-26T11:42:18.567ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7042023-10-26T11:42:18.595ZOct 26 11:42:07.165 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7052023-10-26T11:42:18.624ZOct 26 11:42:07.403 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-10-26T11:42:18.652ZOct 26 11:42:07.447 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-10-26T11:42:18.678ZOct 26 11:42:07.510 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-10-26T11:42:18.702Z
7092023-10-26T11:42:18.731Zreal 0.484480403
7102023-10-26T11:42:18.756Zuser 0.097340677
7112023-10-26T11:42:18.782Zsys 1.979563748
7122023-10-26T11:42:18.811Ztrap 0.000000200
7132023-10-26T11:42:18.885Ztflt 0.000604417
7142023-10-26T11:42:18.913Zdflt 0.000454761
7152023-10-26T11:42:18.945Zkflt 0.000000000
7162023-10-26T11:42:18.972Zlock 43.611730475
7172023-10-26T11:42:19.000Zslp 0.693054454
7182023-10-26T11:42:19.041Zlat 0.003521448
7192023-10-26T11:42:19.082Zstop 0.214885144
7202023-10-26T11:42:19.115Z+ retry=0
7212023-10-26T11:42:19.139Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-10-26T11:42:19.169Z+ [[ 0 -gt 30 ]]
7232023-10-26T11:42:19.212Z+ sleep 1
7242023-10-26T11:42:19.253Z+ retry=1
7252023-10-26T11:42:19.285Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-10-26T11:42:19.318Z+ [[ 1 -gt 30 ]]
7272023-10-26T11:42:19.346Z+ sleep 1
7282023-10-26T11:42:19.386Z+ retry=2
7292023-10-26T11:42:19.412Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-10-26T11:42:34.637Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7312023-10-26T11:42:34.663Zlocal v6:
7322023-10-26T11:42:34.687Zfe80::1c8e:56ff:fef8:79f9
7332023-10-26T11:42:34.711Zfd00:99::1
7342023-10-26T11:42:34.760Zlocal v4:
7352023-10-26T11:42:34.785Z10.151.1.174
7362023-10-26T11:42:34.809Zrouter v6:
7372023-10-26T11:42:34.834Zfd00:1122:3344:101::/64 -> fe80::142e:2fff:fe65:54a8 (1)
7382023-10-26T11:42:34.861Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::142e:2fff:fe65:54a8 (1)
7392023-10-26T11:42:34.885Zrouter v4_idx:
7402023-10-26T11:42:34.909Z0.0.0.0/0 -> 0
7412023-10-26T11:42:34.934Zrouter v4_routes:
7422023-10-26T11:42:34.960Z0 -> 10.151.1.100 (2)
7432023-10-26T11:42:34.984Zresolver v4:
7442023-10-26T11:42:35.009Z10.151.1.174 -> 3e:a2:d5:69:61:bf
7452023-10-26T11:42:35.035Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7462023-10-26T11:42:35.109Zresolver v6:
7472023-10-26T11:42:35.143Zfe80::142e:2fff:fe65:54a8 -> 16:2e:2f:65:54:a8
7482023-10-26T11:42:35.167Zfe80::1c8e:56ff:fef8:79f9 -> 1e:8e:56:f8:79:f9
7492023-10-26T11:42:35.192Znat v4:
7502023-10-26T11:42:35.216Znat_v6:
7512023-10-26T11:42:35.241Zport_mac:
7522023-10-26T11:42:35.265Z1000: a8:40:25:58:3a:98
7532023-10-26T11:42:35.290Z1: a8:40:25:58:3a:99
7542023-10-26T11:42:35.316Zproxy arp:
7552023-10-26T11:42:35.341Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7562023-10-26T11:42:35.365Z+ retry=0
7572023-10-26T11:42:35.391Z+++ zoneadm list -n
7582023-10-26T11:42:35.416Z+++ grep oxz_ntp
7592023-10-26T11:42:35.445Z++ pfexec svcs -z -Hostate oxide/ntp
7602023-10-26T11:42:35.470Zsvcs: zone '-Hostate': entity not found
7612023-10-26T11:42:35.495Z++ true
7622023-10-26T11:42:35.520Z+ [[ '' != online ]]
7632023-10-26T11:42:35.545Z+ [[ 0 -gt 60 ]]
7642023-10-26T11:42:35.569Z+ sleep 1
7652023-10-26T11:42:35.691Z+ retry=1
7662023-10-26T11:42:35.715Z+++ zoneadm list -n
7672023-10-26T11:42:35.739Z+++ grep oxz_ntp
7682023-10-26T11:42:35.764Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-10-26T11:42:35.803Zsvcs: zone '-Hostate': entity not found
7702023-10-26T11:42:35.830Z++ true
7712023-10-26T11:42:35.854Z+ [[ '' != online ]]
7722023-10-26T11:42:35.879Z+ [[ 1 -gt 60 ]]
7732023-10-26T11:42:35.903Z+ sleep 1
7742023-10-26T11:42:36.709Z+ retry=2
7752023-10-26T11:42:36.734Z+++ zoneadm list -n
7762023-10-26T11:42:36.758Z+++ grep oxz_ntp
7772023-10-26T11:42:36.783Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-10-26T11:42:36.808Zsvcs: zone '-Hostate': entity not found
7792023-10-26T11:42:36.851Z++ true
7802023-10-26T11:42:36.877Z+ [[ '' != online ]]
7812023-10-26T11:42:36.902Z+ [[ 2 -gt 60 ]]
7822023-10-26T11:42:36.926Z+ sleep 1
7832023-10-26T11:42:37.731Z+ retry=3
7842023-10-26T11:42:37.761Z+++ zoneadm list -n
7852023-10-26T11:42:37.785Z+++ grep oxz_ntp
7862023-10-26T11:42:37.811Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-10-26T11:42:37.836Zsvcs: zone '-Hostate': entity not found
7882023-10-26T11:42:37.893Z++ true
7892023-10-26T11:42:37.917Z+ [[ '' != online ]]
7902023-10-26T11:42:37.942Z+ [[ 3 -gt 60 ]]
7912023-10-26T11:42:37.966Z+ sleep 1
7922023-10-26T11:42:38.754Z+ retry=4
7932023-10-26T11:42:38.778Z+++ zoneadm list -n
7942023-10-26T11:42:38.803Z+++ grep oxz_ntp
7952023-10-26T11:42:38.828Z++ pfexec svcs -z -Hostate oxide/ntp
7962023-10-26T11:42:38.852Zsvcs: zone '-Hostate': entity not found
7972023-10-26T11:42:38.876Z++ true
7982023-10-26T11:42:38.931Z+ [[ '' != online ]]
7992023-10-26T11:42:38.957Z+ [[ 4 -gt 60 ]]
8002023-10-26T11:42:38.982Z+ sleep 1
8012023-10-26T11:42:39.775Z+ retry=5
8022023-10-26T11:42:39.800Z+++ zoneadm list -n
8032023-10-26T11:42:39.829Z+++ grep oxz_ntp
8042023-10-26T11:42:39.856Z++ pfexec svcs -z -Hostate oxide/ntp
8052023-10-26T11:42:39.880Zsvcs: zone '-Hostate': entity not found
8062023-10-26T11:42:39.905Z++ true
8072023-10-26T11:42:39.932Z+ [[ '' != online ]]
8082023-10-26T11:42:39.976Z+ [[ 5 -gt 60 ]]
8092023-10-26T11:42:40.003Z+ sleep 1
8102023-10-26T11:42:40.795Z+ retry=6
8112023-10-26T11:42:40.820Z+++ zoneadm list -n
8122023-10-26T11:42:40.845Z+++ grep oxz_ntp
8132023-10-26T11:42:40.870Z++ pfexec svcs -z -Hostate oxide/ntp
8142023-10-26T11:42:40.895Zsvcs: zone '-Hostate': entity not found
8152023-10-26T11:42:40.920Z++ true
8162023-10-26T11:42:40.948Z+ [[ '' != online ]]
8172023-10-26T11:42:40.972Z+ [[ 6 -gt 60 ]]
8182023-10-26T11:42:41.019Z+ sleep 1
8192023-10-26T11:42:41.816Z+ retry=7
8202023-10-26T11:42:41.840Z+++ zoneadm list -n
8212023-10-26T11:42:41.865Z+++ grep oxz_ntp
8222023-10-26T11:42:41.889Z++ pfexec svcs -z -Hostate oxide/ntp
8232023-10-26T11:42:41.914Zsvcs: zone '-Hostate': entity not found
8242023-10-26T11:42:41.940Z++ true
8252023-10-26T11:42:41.965Z+ [[ '' != online ]]
8262023-10-26T11:42:41.990Z+ [[ 7 -gt 60 ]]
8272023-10-26T11:42:42.014Z+ sleep 1
8282023-10-26T11:42:42.837Z+ retry=8
8292023-10-26T11:42:42.863Z+++ zoneadm list -n
8302023-10-26T11:42:42.888Z+++ grep oxz_ntp
8312023-10-26T11:42:42.913Z++ pfexec svcs -z -Hostate oxide/ntp
8322023-10-26T11:42:42.938Zsvcs: zone '-Hostate': entity not found
8332023-10-26T11:42:42.963Z++ true
8342023-10-26T11:42:42.988Z+ [[ '' != online ]]
8352023-10-26T11:42:43.013Z+ [[ 8 -gt 60 ]]
8362023-10-26T11:42:43.038Z+ sleep 1
8372023-10-26T11:42:44.429Z+ retry=9
8382023-10-26T11:42:44.455Z+++ zoneadm list -n
8392023-10-26T11:42:44.480Z+++ grep oxz_ntp
8402023-10-26T11:42:44.505Z++ pfexec svcs -z -Hostate oxide/ntp
8412023-10-26T11:42:44.530Zsvcs: zone '-Hostate': entity not found
8422023-10-26T11:42:44.555Z++ true
8432023-10-26T11:42:44.580Z+ [[ '' != online ]]
8442023-10-26T11:42:44.605Z+ [[ 9 -gt 60 ]]
8452023-10-26T11:42:44.630Z+ sleep 1
8462023-10-26T11:42:44.881Z+ retry=10
8472023-10-26T11:42:44.908Z+++ zoneadm list -n
8482023-10-26T11:42:44.934Z+++ grep oxz_ntp
8492023-10-26T11:42:44.959Z++ pfexec svcs -z oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793 -Hostate oxide/ntp
8502023-10-26T11:42:44.984Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8512023-10-26T11:42:45.008Z++ true
8522023-10-26T11:42:45.033Z+ [[ '' != online ]]
8532023-10-26T11:42:45.058Z+ [[ 10 -gt 60 ]]
8542023-10-26T11:42:45.082Z+ sleep 1
8552023-10-26T11:42:45.913Z+ retry=11
8562023-10-26T11:42:45.939Z+++ zoneadm list -n
8572023-10-26T11:42:45.964Z+++ grep oxz_ntp
8582023-10-26T11:42:45.988Z++ pfexec svcs -z oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793 -Hostate oxide/ntp
8592023-10-26T11:42:46.016Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8602023-10-26T11:42:46.041Z++ true
8612023-10-26T11:42:46.067Z+ [[ '' != online ]]
8622023-10-26T11:42:46.093Z+ [[ 11 -gt 60 ]]
8632023-10-26T11:42:46.117Z+ sleep 1
8642023-10-26T11:42:46.945Z+ retry=12
8652023-10-26T11:42:46.969Z+++ zoneadm list -n
8662023-10-26T11:42:46.994Z+++ grep oxz_ntp
8672023-10-26T11:42:47.019Z++ pfexec svcs -z oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793 -Hostate oxide/ntp
8682023-10-26T11:42:47.043Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8692023-10-26T11:42:47.068Z++ true
8702023-10-26T11:42:47.092Z+ [[ '' != online ]]
8712023-10-26T11:42:47.116Z+ [[ 12 -gt 60 ]]
8722023-10-26T11:42:47.141Z+ sleep 1
8732023-10-26T11:42:47.974Z+ retry=13
8742023-10-26T11:42:47.998Z+++ zoneadm list -n
8752023-10-26T11:42:48.024Z+++ grep oxz_ntp
8762023-10-26T11:42:48.049Z++ pfexec svcs -z oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793 -Hostate oxide/ntp
8772023-10-26T11:42:48.076Z+ [[ uninitialized != online ]]
8782023-10-26T11:42:48.100Z+ [[ 13 -gt 60 ]]
8792023-10-26T11:42:48.126Z+ sleep 1
8802023-10-26T11:42:48.996Z+ retry=14
8812023-10-26T11:42:49.023Z+++ zoneadm list -n
8822023-10-26T11:42:49.048Z+++ grep oxz_ntp
8832023-10-26T11:42:49.073Z++ pfexec svcs -z oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793 -Hostate oxide/ntp
8842023-10-26T11:42:49.099Z+ [[ disabled != online ]]
8852023-10-26T11:42:49.124Z+ [[ 14 -gt 60 ]]
8862023-10-26T11:42:49.148Z+ sleep 1
8872023-10-26T11:42:50.020Z+ retry=15
8882023-10-26T11:42:50.045Z+++ zoneadm list -n
8892023-10-26T11:42:50.069Z+++ grep oxz_ntp
8902023-10-26T11:42:50.094Z++ pfexec svcs -z oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793 -Hostate oxide/ntp
8912023-10-26T11:42:50.118Z+ [[ disabled != online ]]
8922023-10-26T11:42:50.143Z+ [[ 15 -gt 60 ]]
8932023-10-26T11:42:50.167Z+ sleep 1
8942023-10-26T11:42:51.041Z+ retry=16
8952023-10-26T11:42:51.066Z+++ zoneadm list -n
8962023-10-26T11:42:51.091Z+++ grep oxz_ntp
8972023-10-26T11:42:51.120Z++ pfexec svcs -z oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793 -Hostate oxide/ntp
8982023-10-26T11:42:51.145Z+ [[ disabled != online ]]
8992023-10-26T11:42:51.169Z+ [[ 16 -gt 60 ]]
9002023-10-26T11:42:51.194Z+ sleep 1
9012023-10-26T11:42:52.063Z+ retry=17
9022023-10-26T11:42:52.088Z+++ zoneadm list -n
9032023-10-26T11:42:52.113Z+++ grep oxz_ntp
9042023-10-26T11:42:52.137Z++ pfexec svcs -z oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793 -Hostate oxide/ntp
9052023-10-26T11:42:52.162Z+ [[ disabled != online ]]
9062023-10-26T11:42:52.186Z+ [[ 17 -gt 60 ]]
9072023-10-26T11:42:52.210Z+ sleep 1
9082023-10-26T11:42:53.085Z+ retry=18
9092023-10-26T11:42:53.113Z+++ zoneadm list -n
9102023-10-26T11:42:53.137Z+++ grep oxz_ntp
9112023-10-26T11:42:53.162Z++ pfexec svcs -z oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793 -Hostate oxide/ntp
9122023-10-26T11:42:53.186Z+ [[ disabled != online ]]
9132023-10-26T11:42:53.211Z+ [[ 18 -gt 60 ]]
9142023-10-26T11:42:53.236Z+ sleep 1
9152023-10-26T11:42:54.106Z+ retry=19
9162023-10-26T11:42:54.130Z+++ zoneadm list -n
9172023-10-26T11:42:54.153Z+++ grep oxz_ntp
9182023-10-26T11:42:54.177Z++ pfexec svcs -z oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793 -Hostate oxide/ntp
9192023-10-26T11:42:54.201Z+ [[ disabled != online ]]
9202023-10-26T11:42:54.225Z+ [[ 19 -gt 60 ]]
9212023-10-26T11:42:54.249Z+ sleep 1
9222023-10-26T11:42:55.128Z+ retry=20
9232023-10-26T11:42:55.152Z+++ zoneadm list -n
9242023-10-26T11:42:55.177Z+++ grep oxz_ntp
9252023-10-26T11:42:55.201Z++ pfexec svcs -z oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793 -Hostate oxide/ntp
9262023-10-26T11:42:55.254Z+ [[ disabled != online ]]
9272023-10-26T11:42:55.279Z+ [[ 20 -gt 60 ]]
9282023-10-26T11:42:55.304Z+ sleep 1
9292023-10-26T11:42:56.149Z+ retry=21
9302023-10-26T11:42:56.174Z+++ zoneadm list -n
9312023-10-26T11:42:56.199Z+++ grep oxz_ntp
9322023-10-26T11:42:56.224Z++ pfexec svcs -z oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793 -Hostate oxide/ntp
9332023-10-26T11:42:56.251Z+ [[ disabled != online ]]
9342023-10-26T11:42:56.304Z+ [[ 21 -gt 60 ]]
9352023-10-26T11:42:56.331Z+ sleep 1
 
9362023-10-26T11:42:56.742Zbuskin console login: Oct 26 11:42:56 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9372023-10-26T11:42:57.171Z+ retry=22
9382023-10-26T11:42:57.195Z+++ zoneadm list -n
9392023-10-26T11:42:57.219Z+++ grep oxz_ntp
9402023-10-26T11:42:57.243Z++ pfexec svcs -z oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793 -Hostate oxide/ntp
9412023-10-26T11:42:57.268Z+ [[ online != online ]]
9422023-10-26T11:42:57.292Z+ echo 'Waited for chrony: 22s'
9432023-10-26T11:42:57.317Z+ export RUST_BACKTRACE=1
9442023-10-26T11:42:57.341ZWaited for chrony: 22s
9452023-10-26T11:42:57.366Z+ RUST_BACKTRACE=1
9462023-10-26T11:42:57.392Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9472023-10-26T11:42:57.416Z+ ./tests/bootstrap
9482023-10-26T11:42:57.440ZUsing external DNS server at 10.151.1.164:53
9492023-10-26T11:42:57.464Z2023-10-26 11:42:57.428374519 UTC: attempting to log into API
9502023-10-26T11:43:12.255Z2023-10-26 11:43:12.341710291 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9512023-10-26T11:43:13.257Z2023-10-26 11:43:13.342553847 UTC: attempting to log into API
9522023-10-26T11:43:28.289Z2023-10-26 11:43:28.375048631 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9532023-10-26T11:43:29.290Z2023-10-26 11:43:29.375932777 UTC: attempting to log into API
9542023-10-26T11:43:44.323Z2023-10-26 11:43:44.408854337 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9552023-10-26T11:43:45.324Z2023-10-26 11:43:45.409622972 UTC: attempting to log into API
9562023-10-26T11:44:00.357Z2023-10-26 11:44:00.442830456 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9572023-10-26T11:44:01.359Z2023-10-26 11:44:01.443857695 UTC: attempting to log into API
9582023-10-26T11:44:16.392Z2023-10-26 11:44:16.454526225 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9592023-10-26T11:44:17.605Z2023-10-26 11:44:17.453682466 UTC: attempting to log into API
9602023-10-26T11:44:32.435Z2023-10-26 11:44:32.446992062 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9612023-10-26T11:44:33.430Z2023-10-26 11:44:33.446214304 UTC: attempting to log into API
9622023-10-26T11:44:48.464Z2023-10-26 11:44:48.450181870 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9632023-10-26T11:44:49.464Z2023-10-26 11:44:49.450988835 UTC: attempting to log into API
9642023-10-26T11:45:04.498Z2023-10-26 11:45:04.484149349 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9652023-10-26T11:45:05.499Z2023-10-26 11:45:05.484819922 UTC: attempting to log into API
9662023-10-26T11:45:20.532Z2023-10-26 11:45:20.511177010 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9672023-10-26T11:45:21.533Z2023-10-26 11:45:21.511968955 UTC: attempting to log into API
9682023-10-26T11:45:36.566Z2023-10-26 11:45:36.544705102 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9692023-10-26T11:45:37.567Z2023-10-26 11:45:37.545593408 UTC: attempting to log into API
9702023-10-26T11:45:52.601Z2023-10-26 11:45:52.579079527 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9712023-10-26T11:45:53.602Z2023-10-26 11:45:53.579905932 UTC: attempting to log into API
9722023-10-26T11:46:08.634Z2023-10-26 11:46:08.612225584 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9732023-10-26T11:46:09.634Z2023-10-26 11:46:09.613049599 UTC: attempting to log into API
9742023-10-26T11:46:24.668Z2023-10-26 11:46:24.646087647 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9752023-10-26T11:46:25.669Z2023-10-26 11:46:25.647911236 UTC: attempting to log into API
9762023-10-26T11:46:40.702Z2023-10-26 11:46:40.680844240 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9772023-10-26T11:46:41.703Z2023-10-26 11:46:41.681929789 UTC: attempting to log into API
9782023-10-26T11:46:56.736Z2023-10-26 11:46:56.715534502 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9792023-10-26T11:46:57.737Z2023-10-26 11:46:57.716264686 UTC: attempting to log into API
9802023-10-26T11:47:12.770Z2023-10-26 11:47:12.749389308 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9812023-10-26T11:47:13.772Z2023-10-26 11:47:13.750524828 UTC: attempting to log into API
9822023-10-26T11:47:28.803Z2023-10-26 11:47:28.782739492 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9832023-10-26T11:47:29.805Z2023-10-26 11:47:29.784155356 UTC: attempting to log into API
9842023-10-26T11:47:44.864Z2023-10-26 11:47:44.843322974 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9852023-10-26T11:47:45.891Z2023-10-26 11:47:45.844540955 UTC: attempting to log into API
 
9862023-10-26T11:47:55.613ZOct 26 11:47:55 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9872023-10-26T11:48:00.922Z2023-10-26 11:48:00.901162106 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9882023-10-26T11:48:01.848ZOct 26 11:48:01 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9892023-10-26T11:48:01.923Z2023-10-26 11:48:01.902117262 UTC: attempting to log into API
 
9902023-10-26T11:48:08.859ZOct 26 11:48:08 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9912023-10-26T11:48:15.592ZOct 26 11:48:15 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9922023-10-26T11:48:16.990Z2023-10-26 11:48:16.934583237 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9932023-10-26T11:48:17.955Z2023-10-26 11:48:17.935756297 UTC: attempting to log into API
 
9942023-10-26T11:48:21.299ZOct 26 11:48:20 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9952023-10-26T11:48:32.988Z2023-10-26 11:48:32.969548934 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9962023-10-26T11:48:33.990Z2023-10-26 11:48:33.971542804 UTC: attempting to log into API
9972023-10-26T11:48:39.771Z2023-10-26 11:48:39.753858534 UTC: login succeeded
9982023-10-26T11:48:39.804Zwaiting for nexus to come up...
9992023-10-26T11:48:39.869Zcreating IP pool... 10.151.1.175 - 10.151.1.227
10002023-10-26T11:48:39.921Zensuring datasets are ready...
10012023-10-26T11:48:42.277Zlet's roll.
10022023-10-26T11:48:42.301Z+ rm ./tests/bootstrap
10032023-10-26T11:48:42.350Z+ for test_bin in tests/*
10042023-10-26T11:48:42.374Z+ ./tests/bootstrap-ce25c26538ce669c
10052023-10-26T11:48:42.398Z
10062023-10-26T11:48:42.444Zrunning 0 tests
10072023-10-26T11:48:42.469Z
10082023-10-26T11:48:42.493Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10092023-10-26T11:48:42.517Z
10102023-10-26T11:48:42.542Z+ for test_bin in tests/*
10112023-10-26T11:48:42.568Z+ ./tests/end_to_end_tests-bcb3df1a66c8d052
10122023-10-26T11:48:42.593Z
10132023-10-26T11:48:42.619Zrunning 2 tests
10142023-10-26T11:48:42.644Ztest no_spoof::no_spoof_here ... ok
10152023-10-26T11:49:42.374Ztest instance_launch::instance_launch has been running for over 60 seconds
10162023-10-26T11:51:39.861Ztest instance_launch::instance_launch ... ok
10172023-10-26T11:51:39.886Z
10182023-10-26T11:51:39.910Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 177.49s
10192023-10-26T11:51:39.934Z
10202023-10-26T11:51:39.958Z+ _exit_trap
10212023-10-26T11:51:39.982Z+ local status=0
10222023-10-26T11:51:40.008Z+ [[ 0 -eq 0 ]]
10232023-10-26T11:51:40.032Z+ exit 0
10242023-10-26T11:51:40.057Zprocess exited: duration 609166 ms, exit code 0
 
10252023-10-26T11:51:40.165Zfound 51 output files
10262023-10-26T11:51:40.353Zuploading: /var/svc/log/oxide-sled-agent:default.log (498540 bytes)
10272023-10-26T11:51:41.425Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10282023-10-26T11:51:41.453Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_14cdd25a-63aa-4d82-838b-df41a9a3db1a/root/var/svc/log/oxide-crucible-agent:default.log (14576 bytes)
10292023-10-26T11:51:42.548Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_14cdd25a-63aa-4d82-838b-df41a9a3db1a/root/var/svc/log/oxide-crucible-agent:default.log
10302023-10-26T11:51:42.573Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_14cdd25a-63aa-4d82-838b-df41a9a3db1a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10312023-10-26T11:51:43.646Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_14cdd25a-63aa-4d82-838b-df41a9a3db1a/root/var/svc/log/oxide-crucible-downstairs:default.log
10322023-10-26T11:51:43.670Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_14cdd25a-63aa-4d82-838b-df41a9a3db1a/root/var/svc/log/oxide-crucible-downstairs:downstairs-e12ac101-a4ae-48de-b53d-169913e55ff5.log (3051 bytes)
10332023-10-26T11:51:44.743Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_14cdd25a-63aa-4d82-838b-df41a9a3db1a/root/var/svc/log/oxide-crucible-downstairs:downstairs-e12ac101-a4ae-48de-b53d-169913e55ff5.log
10342023-10-26T11:51:44.772Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_27d2ed01-98de-4a6a-ad99-67c6b3e6a103/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
10352023-10-26T11:51:45.845Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_27d2ed01-98de-4a6a-ad99-67c6b3e6a103/root/var/svc/log/oxide-cockroachdb:default.log
10362023-10-26T11:51:45.870Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_be13eb45-3243-4393-ae20-cda53a52aa99/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10372023-10-26T11:51:46.943Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_be13eb45-3243-4393-ae20-cda53a52aa99/root/var/svc/log/oxide-cockroachdb:default.log
10382023-10-26T11:51:46.967Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2da5a661-0ceb-4a45-a078-4ad45c578309/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
10392023-10-26T11:51:48.047Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2da5a661-0ceb-4a45-a078-4ad45c578309/root/var/svc/log/oxide-crucible-agent:default.log
10402023-10-26T11:51:48.072Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2da5a661-0ceb-4a45-a078-4ad45c578309/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10412023-10-26T11:51:49.144Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2da5a661-0ceb-4a45-a078-4ad45c578309/root/var/svc/log/oxide-crucible-downstairs:default.log
10422023-10-26T11:51:49.169Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_aabdc218-1b5e-4f9e-b23c-688222c35718/root/var/svc/log/oxide-crucible-agent:default.log (14112 bytes)
10432023-10-26T11:51:50.263Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_aabdc218-1b5e-4f9e-b23c-688222c35718/root/var/svc/log/oxide-crucible-agent:default.log
10442023-10-26T11:51:50.288Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_aabdc218-1b5e-4f9e-b23c-688222c35718/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10452023-10-26T11:51:51.360Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_aabdc218-1b5e-4f9e-b23c-688222c35718/root/var/svc/log/oxide-crucible-downstairs:default.log
10462023-10-26T11:51:51.384Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_aabdc218-1b5e-4f9e-b23c-688222c35718/root/var/svc/log/oxide-crucible-downstairs:downstairs-76106ecd-a7f9-4f1a-99a2-9beeaadb998c.log (5537 bytes)
10472023-10-26T11:51:52.456Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_aabdc218-1b5e-4f9e-b23c-688222c35718/root/var/svc/log/oxide-crucible-downstairs:downstairs-76106ecd-a7f9-4f1a-99a2-9beeaadb998c.log
10482023-10-26T11:51:52.480Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_97fe5252-e491-45c4-bdda-853c9391f082/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10492023-10-26T11:51:53.551Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_97fe5252-e491-45c4-bdda-853c9391f082/root/var/svc/log/oxide-crucible-pantry:default.log
10502023-10-26T11:51:53.576Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_4def7104-23a8-4143-869a-4734f2f1ae89/root/var/svc/log/oxide-oximeter:default.log (195844 bytes)
10512023-10-26T11:51:53.781Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_4def7104-23a8-4143-869a-4734f2f1ae89/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 195844 -> 209403
10522023-10-26T11:51:54.854Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_4def7104-23a8-4143-869a-4734f2f1ae89/root/var/svc/log/oxide-oximeter:default.log
10532023-10-26T11:51:54.878Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_40adb6b3-3d0e-4f89-9736-aa2697b6576d/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10542023-10-26T11:51:55.949Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_40adb6b3-3d0e-4f89-9736-aa2697b6576d/root/var/svc/log/oxide-cockroachdb:default.log
10552023-10-26T11:51:55.973Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4b414aec-40c9-434a-855b-5488f8542034/root/var/svc/log/oxide-crucible-agent:default.log (14112 bytes)
10562023-10-26T11:51:57.068Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4b414aec-40c9-434a-855b-5488f8542034/root/var/svc/log/oxide-crucible-agent:default.log
10572023-10-26T11:51:57.093Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4b414aec-40c9-434a-855b-5488f8542034/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10582023-10-26T11:51:58.166Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4b414aec-40c9-434a-855b-5488f8542034/root/var/svc/log/oxide-crucible-downstairs:default.log
10592023-10-26T11:51:58.191Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4b414aec-40c9-434a-855b-5488f8542034/root/var/svc/log/oxide-crucible-downstairs:downstairs-b85c25fa-74a4-4816-b3aa-f52e0d7a0ac8.log (3051 bytes)
10602023-10-26T11:51:59.268Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4b414aec-40c9-434a-855b-5488f8542034/root/var/svc/log/oxide-crucible-downstairs:downstairs-b85c25fa-74a4-4816-b3aa-f52e0d7a0ac8.log
10612023-10-26T11:51:59.306Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_c820b6b3-4c47-4932-b665-65eb644a3df4/root/var/svc/log/oxide-crucible-pantry:default.log (2337 bytes)
10622023-10-26T11:52:00.377Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_c820b6b3-4c47-4932-b665-65eb644a3df4/root/var/svc/log/oxide-crucible-pantry:default.log
10632023-10-26T11:52:00.401Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_82763286-4ce1-4182-8669-b25ca4f8d384/root/var/svc/log/oxide-external_dns:default.log (15613 bytes)
10642023-10-26T11:52:01.495Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_82763286-4ce1-4182-8669-b25ca4f8d384/root/var/svc/log/oxide-external_dns:default.log
10652023-10-26T11:52:01.521Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_ce123a22-9a68-4043-a4cc-35f28c49e838/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10662023-10-26T11:52:02.595Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_ce123a22-9a68-4043-a4cc-35f28c49e838/root/var/svc/log/oxide-cockroachdb:default.log
10672023-10-26T11:52:02.620Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_3cc3cd81-72cd-4c25-870c-8030ab7d0a30/root/var/svc/log/oxide-crucible-agent:default.log (14154 bytes)
10682023-10-26T11:52:03.714Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_3cc3cd81-72cd-4c25-870c-8030ab7d0a30/root/var/svc/log/oxide-crucible-agent:default.log
10692023-10-26T11:52:03.739Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_3cc3cd81-72cd-4c25-870c-8030ab7d0a30/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10702023-10-26T11:52:04.813Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_3cc3cd81-72cd-4c25-870c-8030ab7d0a30/root/var/svc/log/oxide-crucible-downstairs:default.log
10712023-10-26T11:52:04.839Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_3cc3cd81-72cd-4c25-870c-8030ab7d0a30/root/var/svc/log/oxide-crucible-downstairs:downstairs-63c02db5-57fd-4d40-ac99-d7f167718fe8.log (5535 bytes)
10722023-10-26T11:52:05.912Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_3cc3cd81-72cd-4c25-870c-8030ab7d0a30/root/var/svc/log/oxide-crucible-downstairs:downstairs-63c02db5-57fd-4d40-ac99-d7f167718fe8.log
10732023-10-26T11:52:05.942Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_56f80cd9-06cb-47f0-a119-f922fdfe62a0/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10742023-10-26T11:52:07.014Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_56f80cd9-06cb-47f0-a119-f922fdfe62a0/root/var/svc/log/oxide-crucible-agent:default.log
10752023-10-26T11:52:07.039Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_56f80cd9-06cb-47f0-a119-f922fdfe62a0/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10762023-10-26T11:52:08.112Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_56f80cd9-06cb-47f0-a119-f922fdfe62a0/root/var/svc/log/oxide-crucible-downstairs:default.log
10772023-10-26T11:52:08.136Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_09daa43f-f6b3-4f94-8260-58c2121f091c/root/var/svc/log/oxide-internal_dns:default.log (93568 bytes)
10782023-10-26T11:52:09.319Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_09daa43f-f6b3-4f94-8260-58c2121f091c/root/var/svc/log/oxide-internal_dns:default.log
10792023-10-26T11:52:09.344Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_09daa43f-f6b3-4f94-8260-58c2121f091c/root/var/svc/log/oxide-internal_dns:default.log.0 (6318 bytes)
10802023-10-26T11:52:10.417Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_09daa43f-f6b3-4f94-8260-58c2121f091c/root/var/svc/log/oxide-internal_dns:default.log.0
10812023-10-26T11:52:10.452Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_6a221681-1cac-435c-bfde-75faf502736c/root/var/svc/log/oxide-nexus:default.log (4485784 bytes)
10822023-10-26T11:52:12.077Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_6a221681-1cac-435c-bfde-75faf502736c/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4485784 -> 4487611
10832023-10-26T11:52:13.153Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_6a221681-1cac-435c-bfde-75faf502736c/root/var/svc/log/oxide-nexus:default.log
10842023-10-26T11:52:13.178Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_2a5436d5-4819-4b2e-a040-1020b6f8d925/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10852023-10-26T11:52:14.251Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_2a5436d5-4819-4b2e-a040-1020b6f8d925/root/var/svc/log/oxide-cockroachdb:default.log
10862023-10-26T11:52:14.276Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_aabc9001-447b-4ce1-a1d8-00c964b431df/root/var/svc/log/oxide-nexus:default.log (578782 bytes)
10872023-10-26T11:52:14.506Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_aabc9001-447b-4ce1-a1d8-00c964b431df/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 578782 -> 581214
10882023-10-26T11:52:15.579Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_aabc9001-447b-4ce1-a1d8-00c964b431df/root/var/svc/log/oxide-nexus:default.log
10892023-10-26T11:52:15.604Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793/root/var/svc/log/oxide-ntp:default.log (137 bytes)
10902023-10-26T11:52:16.676Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793/root/var/svc/log/oxide-ntp:default.log
10912023-10-26T11:52:16.700Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793/root/var/svc/log/oxide-ntp:default.log.0 (984 bytes)
10922023-10-26T11:52:18.804Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793/root/var/svc/log/oxide-ntp:default.log.0
10932023-10-26T11:52:18.829Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_26b963d5-3fe3-4e03-a0b3-71128808ed0a/root/var/svc/log/oxide-clickhouse:default.log (726664 bytes)
10942023-10-26T11:52:19.085Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_26b963d5-3fe3-4e03-a0b3-71128808ed0a/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 726664 -> 848003
10952023-10-26T11:52:20.157Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_26b963d5-3fe3-4e03-a0b3-71128808ed0a/root/var/svc/log/oxide-clickhouse:default.log
10962023-10-26T11:52:20.182Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1970534a-8b1b-40c1-b0da-c5b3330cb16e/root/var/svc/log/oxide-crucible-agent:default.log (13687 bytes)
10972023-10-26T11:52:21.285Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1970534a-8b1b-40c1-b0da-c5b3330cb16e/root/var/svc/log/oxide-crucible-agent:default.log
10982023-10-26T11:52:21.309Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1970534a-8b1b-40c1-b0da-c5b3330cb16e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10992023-10-26T11:52:22.370Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1970534a-8b1b-40c1-b0da-c5b3330cb16e/root/var/svc/log/oxide-crucible-downstairs:default.log
11002023-10-26T11:52:22.394Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1970534a-8b1b-40c1-b0da-c5b3330cb16e/root/var/svc/log/oxide-crucible-downstairs:downstairs-0e9251a9-4f80-4ece-a6b2-165964ad2c11.log (3050 bytes)
11012023-10-26T11:52:23.465Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1970534a-8b1b-40c1-b0da-c5b3330cb16e/root/var/svc/log/oxide-crucible-downstairs:downstairs-0e9251a9-4f80-4ece-a6b2-165964ad2c11.log
11022023-10-26T11:52:23.489Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1cc2d477-9147-4688-8723-6e51d3406ac8/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
11032023-10-26T11:52:24.559Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1cc2d477-9147-4688-8723-6e51d3406ac8/root/var/svc/log/oxide-crucible-agent:default.log
11042023-10-26T11:52:24.583Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1cc2d477-9147-4688-8723-6e51d3406ac8/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11052023-10-26T11:52:25.653Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1cc2d477-9147-4688-8723-6e51d3406ac8/root/var/svc/log/oxide-crucible-downstairs:default.log
11062023-10-26T11:52:25.677Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_47d6b7d2-c305-4d94-b7a2-9f038a234333/root/var/svc/log/oxide-crucible-agent:default.log (13683 bytes)
11072023-10-26T11:52:26.769Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_47d6b7d2-c305-4d94-b7a2-9f038a234333/root/var/svc/log/oxide-crucible-agent:default.log
11082023-10-26T11:52:26.793Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_47d6b7d2-c305-4d94-b7a2-9f038a234333/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11092023-10-26T11:52:27.862Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_47d6b7d2-c305-4d94-b7a2-9f038a234333/root/var/svc/log/oxide-crucible-downstairs:default.log
11102023-10-26T11:52:27.886Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_47d6b7d2-c305-4d94-b7a2-9f038a234333/root/var/svc/log/oxide-crucible-downstairs:downstairs-63a26c0c-32ca-40c2-af79-9cdae78bd796.log (5534 bytes)
11112023-10-26T11:52:28.956Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_47d6b7d2-c305-4d94-b7a2-9f038a234333/root/var/svc/log/oxide-crucible-downstairs:downstairs-63a26c0c-32ca-40c2-af79-9cdae78bd796.log
11122023-10-26T11:52:28.980Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_0ceb53f9-7fdf-4a48-a239-6bbbc0f2e49b/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
11132023-10-26T11:52:30.055Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_0ceb53f9-7fdf-4a48-a239-6bbbc0f2e49b/root/var/svc/log/oxide-crucible-pantry:default.log
11142023-10-26T11:52:30.081Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ab581266-2f11-462c-8f26-280c4872fc0e/root/var/svc/log/oxide-internal_dns:default.log (93568 bytes)
11152023-10-26T11:52:30.213Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ab581266-2f11-462c-8f26-280c4872fc0e/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93568 -> 97324
11162023-10-26T11:52:31.284Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ab581266-2f11-462c-8f26-280c4872fc0e/root/var/svc/log/oxide-internal_dns:default.log
11172023-10-26T11:52:31.309Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ab581266-2f11-462c-8f26-280c4872fc0e/root/var/svc/log/oxide-internal_dns:default.log.0 (8150 bytes)
11182023-10-26T11:52:32.403Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ab581266-2f11-462c-8f26-280c4872fc0e/root/var/svc/log/oxide-internal_dns:default.log.0
11192023-10-26T11:52:32.427Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_c36a9c2b-7a5c-42e1-9b85-b32f3fff6061/root/var/svc/log/oxide-external_dns:default.log (15612 bytes)
11202023-10-26T11:52:32.495Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_c36a9c2b-7a5c-42e1-9b85-b32f3fff6061/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15612 -> 19389
11212023-10-26T11:52:33.568Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_c36a9c2b-7a5c-42e1-9b85-b32f3fff6061/root/var/svc/log/oxide-external_dns:default.log
11222023-10-26T11:52:33.593Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f0090265-a298-4251-be6b-8656b97a0c05/root/var/svc/log/oxide-internal_dns:default.log (93572 bytes)
11232023-10-26T11:52:33.729Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f0090265-a298-4251-be6b-8656b97a0c05/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93572 -> 97327
11242023-10-26T11:52:34.801Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f0090265-a298-4251-be6b-8656b97a0c05/root/var/svc/log/oxide-internal_dns:default.log
11252023-10-26T11:52:34.825Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f0090265-a298-4251-be6b-8656b97a0c05/root/var/svc/log/oxide-internal_dns:default.log.0 (8750 bytes)
11262023-10-26T11:52:35.920Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f0090265-a298-4251-be6b-8656b97a0c05/root/var/svc/log/oxide-internal_dns:default.log.0
11272023-10-26T11:52:35.948Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8b75a744-8007-4d3f-97ed-1be76e80cd0a/root/var/svc/log/oxide-nexus:default.log (610885 bytes)
11282023-10-26T11:52:36.690Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8b75a744-8007-4d3f-97ed-1be76e80cd0a/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 610885 -> 672705
11292023-10-26T11:52:37.763Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8b75a744-8007-4d3f-97ed-1be76e80cd0a/root/var/svc/log/oxide-nexus:default.log
11302023-10-26T11:52:37.788Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793/root/var/log/chrony/measurements.log (19865 bytes)
11312023-10-26T11:52:37.857Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793/root/var/log/chrony/measurements.log" changed size mid upload: 19865 -> 21372
11322023-10-26T11:52:38.929Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793/root/var/log/chrony/measurements.log
11332023-10-26T11:52:38.953Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793/root/var/log/chrony/statistics.log (11466 bytes)
11342023-10-26T11:52:39.000Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793/root/var/log/chrony/statistics.log" changed size mid upload: 11466 -> 12870
11352023-10-26T11:52:40.073Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793/root/var/log/chrony/statistics.log
11362023-10-26T11:52:40.097Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793/root/var/log/chrony/tracking.log (1986 bytes)
11372023-10-26T11:52:40.122Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793/root/var/log/chrony/tracking.log" changed size mid upload: 1986 -> 2118
11382023-10-26T11:52:41.191Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_fd652d1e-9846-4054-947d-6756748ef793/root/var/log/chrony/tracking.log