01HE8NJDAR5QABSS3D5H57XJBD: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HE8NMJDG4K0Q86A46AZF1KSE

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-02T19:44:41.488Zjob dependencies complete; ready to run (waiting for 59 m 46 s)
22023-11-02T20:09:30.447Zjob assigned to worker 01HE8TDWXDGW9HDP4EFQH03T8K (queued for 24 m 48 s)
32023-11-02T20:09:30.529Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-11-02T20:09:30.577Zxcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd
52023-11-02T20:09:30.610Zhttp://172.20.0.5:9969/ipxe/buskin... ok
62023-11-02T20:09:30.651Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
72023-11-02T20:09:30.688Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
82023-11-02T20:09:30.729Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
92023-11-02T20:09:30.778Zhttp://172.20.0.5:9969/postboot/buskin... ok
102023-11-02T20:09:30.826ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
112023-11-02T20:09:30.871Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
122023-11-02T20:09:30.916ZOxide Helios Version helios-2.0.22241 64-bit
132023-11-02T20:09:30.969ZConfiguring devices.
142023-11-02T20:09:31.026Z##################################################
152023-11-02T20:09:31.063Z#### ############# ##
162023-11-02T20:09:31.100Z### ### ############ ##
172023-11-02T20:09:31.140Z## ### # ## ### ## Oxide Computer Company ##
182023-11-02T20:09:31.180Z## ## ## ### # ### ##
192023-11-02T20:09:31.842Z## # ### #### #### This Station Under ##
202023-11-02T20:09:31.906Z### ### #### # ### Computer Control ##
212023-11-02T20:09:31.950Z#### #### ### ## ##
222023-11-02T20:09:32.009Z##################################################
232023-11-02T20:09:32.053ZHostname: buskin
242023-11-02T20:09:32.090ZLoading smf(7) serice descriptions:
252023-11-02T20:09:32.422Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-13d7796e9c457a2a.gz
262023-11-02T20:09:36.310Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-13d7796e9c457a2a.gz
272023-11-02T20:09:36.338Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
282023-11-02T20:10:09.273Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
292023-11-02T20:10:09.303Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cc8f9cfe063d8adb.gz
302023-11-02T20:11:03.846Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cc8f9cfe063d8adb.gz
312023-11-02T20:11:03.871Zdownloading input: /input/ci-tools/work/caboose-util.gz
322023-11-02T20:11:05.258Zdownloaded input: /input/ci-tools/work/caboose-util.gz
332023-11-02T20:11:05.290Zdownloading input: /input/ci-tools/work/tufaceous.gz
342023-11-02T20:11:09.543Zdownloaded input: /input/ci-tools/work/tufaceous.gz
352023-11-02T20:11:09.632Zdownloading input: /input/package/work/version.txt
362023-11-02T20:11:09.668Zdownloaded input: /input/package/work/version.txt
372023-11-02T20:11:09.729Zdownloading input: /input/package/work/package.tar.gz
382023-11-02T20:11:54.689Zdownloaded input: /input/package/work/package.tar.gz
392023-11-02T20:11:54.713Zdownloading input: /input/package/work/global-zone-packages.tar.gz
402023-11-02T20:12:41.980Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
412023-11-02T20:12:42.008Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
422023-11-02T20:12:58.897Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
432023-11-02T20:12:58.969Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
442023-11-02T20:13:52.259Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
452023-11-02T20:13:52.328Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
462023-11-02T20:15:45.378Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
472023-11-02T20:15:45.406Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
482023-11-02T20:16:30.026Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
492023-11-02T20:16:30.053Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
502023-11-02T20:16:38.894Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
512023-11-02T20:16:38.917Zdownloading input: /input/package/work/zones/crucible.tar.gz
522023-11-02T20:16:57.111Zdownloaded input: /input/package/work/zones/crucible.tar.gz
532023-11-02T20:16:57.137Zdownloading input: /input/package/work/zones/external-dns.tar.gz
542023-11-02T20:17:10.893Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
552023-11-02T20:17:10.916Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
562023-11-02T20:17:22.674Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
572023-11-02T20:17:22.698Zdownloading input: /input/package/work/zones/ntp.tar.gz
582023-11-02T20:17:22.737Zdownloaded input: /input/package/work/zones/ntp.tar.gz
592023-11-02T20:17:22.761Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
602023-11-02T20:17:27.057Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
612023-11-02T20:17:35.542Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
622023-11-02T20:17:42.752Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
632023-11-02T20:17:42.788Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
642023-11-02T20:18:23.075Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
652023-11-02T20:18:23.955Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
662023-11-02T20:18:58.512Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
672023-11-02T20:18:58.543Zdownloading input: /input/package/work/zones/overlay.tar.gz
682023-11-02T20:18:58.656Zdownloaded input: /input/package/work/zones/overlay.tar.gz
692023-11-02T20:18:58.688Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
702023-11-02T20:19:05.530Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
712023-11-02T20:19:05.567Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
722023-11-02T20:19:37.335Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
732023-11-02T20:19:37.397Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
742023-11-02T20:21:14.101Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
752023-11-02T20:21:14.125Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
762023-11-02T20:22:14.445Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
772023-11-02T20:22:14.920Zstarting task 0: "setup"
782023-11-02T20:22:14.965Z++ uname -s
792023-11-02T20:22:14.999Z+ kern=SunOS
802023-11-02T20:22:15.041Z+ case "$kern" in
812023-11-02T20:22:15.081Z+ groupadd -g 12345 build
822023-11-02T20:22:15.118Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
832023-11-02T20:22:16.995Z+ zfs create -o mountpoint=/work rpool/work
842023-11-02T20:22:17.047Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
852023-11-02T20:22:17.094Z+ home_fs=zfs
862023-11-02T20:22:17.143Z+ [[ zfs == autofs ]]
872023-11-02T20:22:17.302Z+ mkdir -p /home/build
882023-11-02T20:22:17.889Z+ chown build:build /home/build /work
892023-11-02T20:22:19.031Z+ chmod 0700 /home/build /work
902023-11-02T20:22:19.055Zprocess exited: duration 4049 ms, exit code 0
 
912023-11-02T20:22:19.102Zstarting task 1: "authentication"
922023-11-02T20:22:19.180Zprocess exited: duration 47 ms, exit code 0
 
932023-11-02T20:22:19.232Zstarting task 2: "build"
942023-11-02T20:22:19.265Z+ trap _exit_trap EXIT
952023-11-02T20:22:19.293Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
962023-11-02T20:22:19.460Zsecpolicy_fs_linkdir: 0x55 = 0x55
972023-11-02T20:22:19.487Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
982023-11-02T20:22:19.513Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
992023-11-02T20:22:19.540Zsecpolicy_fs_linkdir+3: 0 = 0xe5
1002023-11-02T20:22:19.568Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1012023-11-02T20:22:19.595Zsecpolicy_fs_linkdir+5: 0 = 0x1
1022023-11-02T20:22:19.622Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1032023-11-02T20:22:19.649Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1042023-11-02T20:22:19.677Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1052023-11-02T20:22:19.706Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1062023-11-02T20:22:19.734Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1072023-11-02T20:22:19.763Z+ [[ -d /opt/oxide ]]
1082023-11-02T20:22:19.795Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1092023-11-02T20:22:19.832Z+ pfexec mkdir /opt/oxide
1102023-11-02T20:22:20.272Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1112023-11-02T20:22:20.301Z+ [[ -d /opt/oxide-underneath ]]
1122023-11-02T20:22:20.336Z+ cd /opt/oxide-underneath
1132023-11-02T20:22:20.376Z+ pfexec tar ceEp@/f - .
1142023-11-02T20:22:20.404Z+ cd /opt/oxide
1152023-11-02T20:22:20.430Z+ pfexec tar xveEp@/f -
1162023-11-02T20:22:20.454Zx ., 0 bytes, 0 tape blocks
1172023-11-02T20:22:20.499Zx ./opte, 0 bytes, 0 tape blocks
1182023-11-02T20:22:20.523Zx ./opte/bin, 0 bytes, 0 tape blocks
1192023-11-02T20:22:20.547Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1202023-11-02T20:22:20.572Z+ pfexec rm -rf /opt/oxide-underneath
1212023-11-02T20:22:20.596Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1222023-11-02T20:22:20.622Z+ pfexec mkdir /opt/oxide/work
1232023-11-02T20:22:20.651Z+ pfexec chown build:build /opt/oxide/work
1242023-11-02T20:22:20.678Z+ cd /opt/oxide/work
1252023-11-02T20:22:20.709Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1262023-11-02T20:22:21.234ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1272023-11-02T20:22:21.273Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1282023-11-02T20:22:21.309Zx out/omicron-sled-agent.tar, 122405376 bytes, 239073 tape blocks
1292023-11-02T20:22:21.338Zx out/target/test, 75 bytes, 1 tape blocks
1302023-11-02T20:22:21.363Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1312023-11-02T20:22:21.392Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1322023-11-02T20:22:21.421Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1332023-11-02T20:22:21.454Zx target/release/omicron-package, 41798416 bytes, 81638 tape blocks
1342023-11-02T20:22:21.485Zx tools/create_virtual_hardware.sh, 2974 bytes, 6 tape blocks
1352023-11-02T20:22:21.515Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1362023-11-02T20:22:21.548Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1372023-11-02T20:22:21.576Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1382023-11-02T20:22:21.607Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1392023-11-02T20:22:21.632Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1402023-11-02T20:22:21.656Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1412023-11-02T20:22:21.686Z
1422023-11-02T20:22:21.718Zreal 1.604190710
1432023-11-02T20:22:21.760Zuser 1.444331949
1442023-11-02T20:22:22.202Zsys 0.284643489
1452023-11-02T20:22:22.231Ztrap 0.000073360
1462023-11-02T20:22:22.256Ztflt 0.000000000
1472023-11-02T20:22:22.282Zdflt 0.000000000
1482023-11-02T20:22:22.306Zkflt 0.000000000
1492023-11-02T20:22:22.364Zlock 0.000000000
1502023-11-02T20:22:22.440Zslp 1.442817395
1512023-11-02T20:22:22.478Zlat 0.034579925
1522023-11-02T20:22:22.507Zstop 0.000121001
1532023-11-02T20:22:22.533Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1542023-11-02T20:22:22.562Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1552023-11-02T20:22:22.590Z+ mkdir tests
1562023-11-02T20:22:22.623Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1572023-11-02T20:22:23.063Z+ ptime -m gunzip
1582023-11-02T20:22:23.098Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-13d7796e9c457a2a
1592023-11-02T20:22:23.123Z
1602023-11-02T20:22:23.146Zreal 0.142410424
1612023-11-02T20:22:23.182Zuser 0.129975132
1622023-11-02T20:22:23.207Zsys 0.012185488
1632023-11-02T20:22:23.237Ztrap 0.000011419
1642023-11-02T20:22:23.268Ztflt 0.000000000
1652023-11-02T20:22:23.300Zdflt 0.000000000
1662023-11-02T20:22:23.331Zkflt 0.000000000
1672023-11-02T20:22:23.363Zlock 0.000000000
1682023-11-02T20:22:23.401Zslp 0.000023900
1692023-11-02T20:22:23.434Zlat 0.000022420
1702023-11-02T20:22:23.461Zstop 0.000127881
1712023-11-02T20:22:23.489Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-13d7796e9c457a2a
1722023-11-02T20:22:23.519Z+ chmod a+x tests/bootstrap-13d7796e9c457a2a
1732023-11-02T20:22:23.957Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1742023-11-02T20:22:23.988Z+ ptime -m gunzip
1752023-11-02T20:22:24.016Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1762023-11-02T20:22:24.686Z
1772023-11-02T20:22:24.712Zreal 2.249367274
1782023-11-02T20:22:24.739Zuser 2.113932579
1792023-11-02T20:22:24.768Zsys 0.135006768
1802023-11-02T20:22:24.794Ztrap 0.000192984
1812023-11-02T20:22:24.818Ztflt 0.000000000
1822023-11-02T20:22:24.842Zdflt 0.000000000
1832023-11-02T20:22:24.869Zkflt 0.000000000
1842023-11-02T20:22:24.896Zlock 0.000000000
1852023-11-02T20:22:24.924Zslp 0.000022170
1862023-11-02T20:22:24.959Zlat 0.000040780
1872023-11-02T20:22:24.991Zstop 0.000117242
1882023-11-02T20:22:25.021Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1892023-11-02T20:22:25.047Z+ chmod a+x tests/bootstrap
1902023-11-02T20:22:25.076Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1912023-11-02T20:22:25.518Z+ ptime -m gunzip
1922023-11-02T20:22:25.557Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cc8f9cfe063d8adb
1932023-11-02T20:22:27.255Z
1942023-11-02T20:22:27.279Zreal 2.557812194
1952023-11-02T20:22:27.303Zuser 2.412726505
1962023-11-02T20:22:27.334Zsys 0.144644741
1972023-11-02T20:22:27.499Ztrap 0.000197773
1982023-11-02T20:22:27.641Ztflt 0.000000000
1992023-11-02T20:22:27.675Zdflt 0.000000000
2002023-11-02T20:22:27.705Zkflt 0.000000000
2012023-11-02T20:22:27.739Zlock 0.000000000
2022023-11-02T20:22:27.768Zslp 0.000025940
2032023-11-02T20:22:27.807Zlat 0.000023600
2042023-11-02T20:22:27.842Zstop 0.000127331
2052023-11-02T20:22:27.886Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cc8f9cfe063d8adb
2062023-11-02T20:22:27.943Z+ chmod a+x tests/end_to_end_tests-cc8f9cfe063d8adb
2072023-11-02T20:22:27.994Z+ bmat address ls
2082023-11-02T20:22:28.029ZNAME CIDR FIRST LAST COUNT
2092023-11-02T20:22:28.058Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2102023-11-02T20:22:28.083Z++ bmat address ls -f extra -Ho first,last,count
2112023-11-02T20:22:28.107Z+ set -- 10.151.1.164 10.151.1.227 64
2122023-11-02T20:22:28.132Z+ EXTRA_IP_START=10.151.1.164
2132023-11-02T20:22:28.156Z+ EXTRA_IP_END=10.151.1.227
2142023-11-02T20:22:28.180Z+ EXTRA_IP_COUNT=64
2152023-11-02T20:22:28.206Z+ (( EXTRA_IP_COUNT >= 32 ))
2162023-11-02T20:22:28.231Z+ EXTRA_IP_BASE=10.151.1
2172023-11-02T20:22:28.263Z+ EXTRA_IP_FOCTET=164
2182023-11-02T20:22:28.699Z+ EXTRA_IP_LOCTET=227
2192023-11-02T20:22:28.726Z+ SERVICE_IP_POOL_START=10.151.1.164
2202023-11-02T20:22:28.760Z+ SERVICE_IP_POOL_END=10.151.1.173
2212023-11-02T20:22:28.799Z+ DNS_IP1=10.151.1.164
2222023-11-02T20:22:28.831Z+ DNS_IP2=10.151.1.165
2232023-11-02T20:22:28.863Z+ UPLINK_IP=10.151.1.174
2242023-11-02T20:22:28.890Z+ IPPOOL_START=10.151.1.175
2252023-11-02T20:22:28.921Z+ IPPOOL_END=10.151.1.227
2262023-11-02T20:22:28.952Z++ ipadm show-addr -po type,addr
2272023-11-02T20:22:28.979Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2282023-11-02T20:22:29.018Z+ GATEWAY_IP=10.151.1.100
2292023-11-02T20:22:29.055Z+ [[ -n 10.151.1.100 ]]
2302023-11-02T20:22:29.088Z+ ping -s 10.151.1.100 56 1
2312023-11-02T20:22:29.113ZPING 10.151.1.100: 56 data bytes
2322023-11-02T20:22:29.137Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.012 ms
2332023-11-02T20:22:29.175Z
2342023-11-02T20:22:29.205Z----10.151.1.100 PING Statistics----
2352023-11-02T20:22:29.241Z1 packets transmitted, 1 packets received, 0% packet loss
2362023-11-02T20:22:29.277Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2372023-11-02T20:22:29.306Z++ arp -an
2382023-11-02T20:22:29.334Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2392023-11-02T20:22:29.362Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2402023-11-02T20:22:29.393Z+ routeadm -e ipv4-forwarding -u
2412023-11-02T20:22:29.439Z+ PXA_START=10.151.1.164
2422023-11-02T20:22:29.471Z+ PXA_END=10.151.1.227
2432023-11-02T20:22:29.904Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2442023-11-02T20:22:29.944Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2452023-11-02T20:22:30.800Z+ ZPOOL_VDEV_DIR=/scratch
2462023-11-02T20:22:31.000Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2472023-11-02T20:22:31.039Z+++ dirname ./tools/create_virtual_hardware.sh
2482023-11-02T20:22:31.068Z++ cd ./tools
2492023-11-02T20:22:31.097Z++ pwd
2502023-11-02T20:22:31.126Z+ SOURCE_DIR=/opt/oxide/work/tools
2512023-11-02T20:22:31.156Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2522023-11-02T20:22:31.183Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2532023-11-02T20:22:31.617Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2542023-11-02T20:22:31.650Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2552023-11-02T20:22:31.676Z++ dladm show-phys -p -o LINK
2562023-11-02T20:22:31.703Z++ head -1
2572023-11-02T20:22:31.727Z+ PHYSICAL_LINK=igb0
2582023-11-02T20:22:31.750Z+ echo 'Using igb0 as physical link'
2592023-11-02T20:22:31.774ZUsing igb0 as physical link
2602023-11-02T20:22:31.802Z+ ensure_run_as_root
2612023-11-02T20:22:31.830Z++ id -u
2622023-11-02T20:22:31.857Z+ [[ 0 -ne 0 ]]
2632023-11-02T20:22:32.290Z+ ensure_zpools
2642023-11-02T20:22:32.325ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2652023-11-02T20:22:32.452Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2662023-11-02T20:22:33.096ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2672023-11-02T20:22:33.143Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2682023-11-02T20:22:33.181ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2692023-11-02T20:22:33.211Z+ readarray -t ZPOOLS
2702023-11-02T20:22:33.238ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2712023-11-02T20:22:33.265Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2722023-11-02T20:22:33.293ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2732023-11-02T20:22:33.321Z++ sed 's/[ ",]//g'
2742023-11-02T20:22:33.774ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2752023-11-02T20:22:33.802Z+ for ZPOOL in "${ZPOOLS[@]}"
2762023-11-02T20:22:33.830ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2772023-11-02T20:22:33.877Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2782023-11-02T20:22:33.902ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2792023-11-02T20:22:33.926Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2802023-11-02T20:22:33.962ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2812023-11-02T20:22:33.996Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2822023-11-02T20:22:34.028ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2832023-11-02T20:22:34.057Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2842023-11-02T20:22:34.085ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2852023-11-02T20:22:34.115Z0+0 records in
2862023-11-02T20:22:34.143ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2872023-11-02T20:22:34.168Z0+0 records out
2882023-11-02T20:22:34.194ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2892023-11-02T20:22:34.256Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2902023-11-02T20:22:34.296ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2912023-11-02T20:22:34.730Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2922023-11-02T20:22:34.758ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2932023-11-02T20:22:34.833Z+ set +x
2942023-11-02T20:22:34.861ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
2952023-11-02T20:22:34.888Z++ zpool list -o name
2962023-11-02T20:22:34.919ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
2972023-11-02T20:22:34.944Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2982023-11-02T20:22:34.974ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
2992023-11-02T20:22:35.004Z+ [[ -z '' ]]
3002023-11-02T20:22:35.032ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3012023-11-02T20:22:35.468Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
3022023-11-02T20:22:35.547ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3032023-11-02T20:22:35.634Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
3042023-11-02T20:22:36.125ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3052023-11-02T20:22:36.233Z+ set +x
3062023-11-02T20:22:36.290ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3072023-11-02T20:22:36.341Z+ for ZPOOL in "${ZPOOLS[@]}"
3082023-11-02T20:22:36.421ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3092023-11-02T20:22:36.481Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3102023-11-02T20:22:36.515ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3112023-11-02T20:22:36.571Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3122023-11-02T20:22:36.623ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3132023-11-02T20:22:36.711Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3142023-11-02T20:22:36.750ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3152023-11-02T20:22:36.782Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3162023-11-02T20:22:36.825ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3172023-11-02T20:22:36.899Z0+0 records in
3182023-11-02T20:22:36.953ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3192023-11-02T20:22:45.204Z0+0 records out
3202023-11-02T20:22:45.428ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3212023-11-02T20:22:45.873Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3222023-11-02T20:22:45.931ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3232023-11-02T20:22:46.369Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3242023-11-02T20:22:46.811ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3252023-11-02T20:22:46.850Z+ set +x
3262023-11-02T20:22:46.888ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3272023-11-02T20:22:46.922Z++ zpool list -o name
3282023-11-02T20:22:46.960ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3292023-11-02T20:22:47.020Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3302023-11-02T20:22:47.552ZSimnet net0/sc0_0 exists
3312023-11-02T20:22:47.996Z+ [[ -z '' ]]
3322023-11-02T20:22:48.029ZSimnet net1/sc1_0 exists
3332023-11-02T20:22:48.105Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3342023-11-02T20:22:48.153ZVnic sc0_1 exists
3352023-11-02T20:22:48.241Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3362023-11-02T20:22:48.317Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3372023-11-02T20:22:48.364Z+ set +x
3382023-11-02T20:22:48.409Z+ for ZPOOL in "${ZPOOLS[@]}"
3392023-11-02T20:22:48.459Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3402023-11-02T20:22:48.501Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-11-02T20:22:48.547Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3422023-11-02T20:22:48.601Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3432023-11-02T20:22:48.646Z0+0 records in
3442023-11-02T20:22:49.090Z0+0 records out
3452023-11-02T20:22:49.129Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3462023-11-02T20:22:50.511Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3472023-11-02T20:22:50.556Z+ set +x
3482023-11-02T20:22:50.639Z++ zpool list -o name
3492023-11-02T20:22:50.680Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3502023-11-02T20:22:50.706Z+ [[ -z '' ]]
3512023-11-02T20:22:50.730Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3522023-11-02T20:22:50.757Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3532023-11-02T20:22:50.796Z+ set +x
3542023-11-02T20:22:50.829Z+ for ZPOOL in "${ZPOOLS[@]}"
3552023-11-02T20:22:50.869Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3562023-11-02T20:22:50.895Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3572023-11-02T20:22:50.922Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3582023-11-02T20:22:50.946Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3592023-11-02T20:22:50.979Z0+0 records in
3602023-11-02T20:22:51.017Z0+0 records out
3612023-11-02T20:22:51.049Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3622023-11-02T20:22:51.076Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3632023-11-02T20:22:51.110Z+ set +x
3642023-11-02T20:22:51.147Z++ zpool list -o name
3652023-11-02T20:22:51.173Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3662023-11-02T20:22:51.200Z+ [[ -z '' ]]
3672023-11-02T20:22:51.246Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3682023-11-02T20:22:51.279Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3692023-11-02T20:22:51.314Z+ set +x
3702023-11-02T20:22:51.758Z+ for ZPOOL in "${ZPOOLS[@]}"
3712023-11-02T20:22:51.787Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3722023-11-02T20:22:51.816Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3732023-11-02T20:22:51.844Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3742023-11-02T20:22:52.275Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3752023-11-02T20:22:52.299Z0+0 records in
3762023-11-02T20:22:52.324Z0+0 records out
3772023-11-02T20:22:52.348Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3782023-11-02T20:22:52.374Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3792023-11-02T20:22:52.402Z+ set +x
3802023-11-02T20:22:52.427Z++ zpool list -o name
3812023-11-02T20:22:52.456Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3822023-11-02T20:22:52.536Z+ [[ -z '' ]]
3832023-11-02T20:22:52.585Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3842023-11-02T20:22:52.608Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3852023-11-02T20:22:52.633Zdownloading softnpu ...done
3862023-11-02T20:22:53.098Z+ set +x
3872023-11-02T20:22:53.124Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
3882023-11-02T20:22:53.154Z+ for ZPOOL in "${ZPOOLS[@]}"
3892023-11-02T20:22:53.184Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3902023-11-02T20:22:53.213Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3912023-11-02T20:22:53.655Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3922023-11-02T20:22:53.684Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3932023-11-02T20:22:53.718Z0+0 records in
3942023-11-02T20:22:53.742Z0+0 records out
3952023-11-02T20:22:53.766Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3962023-11-02T20:22:53.789Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3972023-11-02T20:22:54.226Z+ set +x
3982023-11-02T20:22:54.252Z++ zpool list -o name
3992023-11-02T20:22:54.278Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
4002023-11-02T20:22:54.303Z+ [[ -z '' ]]
4012023-11-02T20:22:54.327Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
4022023-11-02T20:22:54.352Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4032023-11-02T20:22:54.376Z+ set +x
4042023-11-02T20:22:54.809Z+ for ZPOOL in "${ZPOOLS[@]}"
4052023-11-02T20:22:54.837Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4062023-11-02T20:22:54.864Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4072023-11-02T20:22:54.894Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4082023-11-02T20:22:54.922Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4092023-11-02T20:22:54.956Z0+0 records in
4102023-11-02T20:22:54.987Z0+0 records out
4112023-11-02T20:22:55.014Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4122023-11-02T20:22:55.039Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4132023-11-02T20:22:55.074Z+ set +x
4142023-11-02T20:22:55.109Z++ zpool list -o name
4152023-11-02T20:22:55.136Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4162023-11-02T20:22:55.161Z+ [[ -z '' ]]
4172023-11-02T20:22:55.185Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4182023-11-02T20:22:55.211Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4192023-11-02T20:22:55.236Z+ set +x
4202023-11-02T20:22:55.277Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-11-02T20:22:55.302Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4222023-11-02T20:22:55.331Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4232023-11-02T20:22:55.358Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4242023-11-02T20:22:55.382Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4252023-11-02T20:22:55.420Z0+0 records in
4262023-11-02T20:22:55.454Z0+0 records out
4272023-11-02T20:22:55.911Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4282023-11-02T20:22:55.935Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4292023-11-02T20:22:55.970Z+ set +x
4302023-11-02T20:22:56.003Z++ zpool list -o name
4312023-11-02T20:22:56.029Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4322023-11-02T20:22:56.054Z+ [[ -z '' ]]
4332023-11-02T20:22:56.080Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4342023-11-02T20:22:56.103Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4352023-11-02T20:22:56.126Z+ set +x
4362023-11-02T20:22:56.153Z+ for ZPOOL in "${ZPOOLS[@]}"
4372023-11-02T20:22:56.177Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4382023-11-02T20:22:56.201Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4392023-11-02T20:22:56.225Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4402023-11-02T20:22:56.252Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4412023-11-02T20:22:56.279Z0+0 records in
4422023-11-02T20:22:56.323Z0+0 records out
4432023-11-02T20:22:56.368Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4442023-11-02T20:22:56.404Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4452023-11-02T20:22:56.444Z+ set +x
4462023-11-02T20:22:56.477Z++ zpool list -o name
4472023-11-02T20:22:56.510Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4482023-11-02T20:22:56.545Z+ [[ -z '' ]]
4492023-11-02T20:22:56.586Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4502023-11-02T20:22:56.616Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4512023-11-02T20:22:56.644Z+ set +x
4522023-11-02T20:22:56.672Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4532023-11-02T20:22:56.700Z+ readarray -t ZPOOLS
4542023-11-02T20:22:56.729Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4552023-11-02T20:22:56.756Z++ sed 's/[ ",]//g'
4562023-11-02T20:22:56.849Z+ for ZPOOL in "${ZPOOLS[@]}"
4572023-11-02T20:22:56.882Zdownloading asic_program.so ...done
4582023-11-02T20:22:56.913Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4592023-11-02T20:22:56.942Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
4602023-11-02T20:22:56.970Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4612023-11-02T20:22:57.002Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4622023-11-02T20:22:57.050Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4632023-11-02T20:22:57.092Z0+0 records in
4642023-11-02T20:22:57.123Z0+0 records out
4652023-11-02T20:22:57.160Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4662023-11-02T20:22:57.188Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4672023-11-02T20:22:57.216Z+ set +x
4682023-11-02T20:22:57.249Z++ zpool list -o name
4692023-11-02T20:22:57.272Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4702023-11-02T20:22:57.297Z+ [[ -z '' ]]
4712023-11-02T20:22:57.323Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4722023-11-02T20:22:57.352Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4732023-11-02T20:22:57.379Z+ set +x
4742023-11-02T20:22:57.407Z+ for ZPOOL in "${ZPOOLS[@]}"
4752023-11-02T20:22:57.437Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4762023-11-02T20:22:57.540Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4772023-11-02T20:22:57.596Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4782023-11-02T20:22:57.646Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4792023-11-02T20:22:57.696Z0+0 records in
4802023-11-02T20:22:58.168Z0+0 records out
4812023-11-02T20:22:58.198Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4822023-11-02T20:22:58.227Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4832023-11-02T20:22:58.261Z+ set +x
4842023-11-02T20:22:58.296Z++ zpool list -o name
4852023-11-02T20:22:58.328Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4862023-11-02T20:22:58.377Z+ [[ -z '' ]]
4872023-11-02T20:22:58.407Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4882023-11-02T20:22:58.439Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4892023-11-02T20:22:58.469Z+ set +x
4902023-11-02T20:22:58.498Z+ ensure_simulated_links igb0
4912023-11-02T20:22:58.524Z+ local PHYSICAL_LINK=igb0
4922023-11-02T20:22:58.552Z+ INDICES=("0" "1")
4932023-11-02T20:22:58.581Z+ for I in "${INDICES[@]}"
4942023-11-02T20:22:59.015Z++ get_simnet_name_if_exists net0
4952023-11-02T20:22:59.048Z++ dladm show-simnet -p -o LINK net0
4962023-11-02T20:22:59.086Z++ echo ''
4972023-11-02T20:22:59.125Z+ [[ -z '' ]]
4982023-11-02T20:22:59.157Z+ dladm create-simnet -t net0
4992023-11-02T20:22:59.189Z+ dladm create-simnet -t sc0_0
5002023-11-02T20:22:59.216Z+ dladm modify-simnet -t -p net0 sc0_0
5012023-11-02T20:22:59.660Z+ dladm set-linkprop -p mtu=9000 sc0_0
5022023-11-02T20:22:59.692Z+ success 'Simnet net0/sc0_0 exists'
5032023-11-02T20:22:59.722Z+ set +x
5042023-11-02T20:22:59.751Z+ for I in "${INDICES[@]}"
5052023-11-02T20:22:59.782Z++ get_simnet_name_if_exists net1
5062023-11-02T20:22:59.817Z++ dladm show-simnet -p -o LINK net1
5072023-11-02T20:22:59.844Z++ echo ''
5082023-11-02T20:22:59.869Z+ [[ -z '' ]]
5092023-11-02T20:22:59.893Z+ dladm create-simnet -t net1
5102023-11-02T20:22:59.917Z+ dladm create-simnet -t sc1_0
5112023-11-02T20:22:59.962Z+ dladm modify-simnet -t -p net1 sc1_0
5122023-11-02T20:22:59.988Z+ dladm set-linkprop -p mtu=9000 sc1_0
5132023-11-02T20:23:00.012Z+ success 'Simnet net1/sc1_0 exists'
5142023-11-02T20:23:00.036Z+ set +x
5152023-11-02T20:23:00.084Z++ get_vnic_name_if_exists sc0_1
5162023-11-02T20:23:00.116Z++ dladm show-vnic -p -o LINK sc0_1
5172023-11-02T20:23:00.558Z++ echo ''
5182023-11-02T20:23:00.593Z+ [[ -z '' ]]
5192023-11-02T20:23:00.624Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5202023-11-02T20:23:00.652Z+ [[ -v PROMISC_FILT_OFF ]]
5212023-11-02T20:23:00.703Z+ success 'Vnic sc0_1 exists'
5222023-11-02T20:23:00.827Z+ set +x
5232023-11-02T20:23:00.933Z+ warn_if_no_proxy_arp
5242023-11-02T20:23:01.377Z+ [[ -v PXA_START ]]
5252023-11-02T20:23:01.415Z+ [[ -v PXA_END ]]
5262023-11-02T20:23:01.453Z+ ensure_softnpu_zone
5272023-11-02T20:23:01.484Z+ zoneadm list
5282023-11-02T20:23:01.518Z+ grep -q sidecar_softnpu
5292023-11-02T20:23:01.553Z+ [[ -f out/npuzone/npuzone ]]
5302023-11-02T20:23:01.582Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch main
5312023-11-02T20:23:07.660Zdownloading scadm ...done
5322023-11-02T20:23:07.685Zcreating port tfportrear0_0 for sc0_0
5332023-11-02T20:23:07.709Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-02T20:23:07.738Zsoftnpu zone setup
5352023-11-02T20:23:07.767Zconfigure zone
5362023-11-02T20:23:07.801Zinstall zone
5372023-11-02T20:23:08.748Zboot zone
5382023-11-02T20:23:11.862Z[sidecar_softnpu] svcs milestone/network
5392023-11-02T20:23:11.892ZSTATE STIME FMRI
5402023-11-02T20:23:11.925Zonline 20:23:10 svc:/milestone/network:default
5412023-11-02T20:23:12.374Z[sidecar_softnpu] /softnpu/npu start
5422023-11-02T20:23:12.409Z
5432023-11-02T20:23:12.438Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-02T20:23:12.463Z+ GATEWAY_IP=10.151.1.100
5452023-11-02T20:23:12.489Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-11-02T20:23:12.514ZUsing 10.151.1.100 as gateway ip
5472023-11-02T20:23:12.544Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-02T20:23:12.583Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-11-02T20:23:12.654Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-02T20:23:12.686Z+++ echo 18:c0:4d:0d:9f:b2
5512023-11-02T20:23:12.710Z+ COUNT=1
5522023-11-02T20:23:12.734Z+ [[ 1 -ne 1 ]]
5532023-11-02T20:23:12.768Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-11-02T20:23:12.803ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-11-02T20:23:12.847Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-11-02T20:23:12.881Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-11-02T20:23:13.325Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-02T20:23:13.375Z+ [[ -v PXA_START ]]
5592023-11-02T20:23:13.402Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-11-02T20:23:13.427Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-11-02T20:23:13.453Z+ z_scadm dump-state
5622023-11-02T20:23:13.478Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-02T20:23:13.502Zlocal v6:
5642023-11-02T20:23:13.527Zlocal v4:
5652023-11-02T20:23:13.555Zrouter v6:
5662023-11-02T20:23:13.580Zrouter v4_idx:
5672023-11-02T20:23:14.013Zrouter v4_routes:
5682023-11-02T20:23:14.043Zresolver v4:
5692023-11-02T20:23:14.076Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-11-02T20:23:14.107Zresolver v6:
5712023-11-02T20:23:14.134Znat v4:
5722023-11-02T20:23:14.165Znat_v6:
5732023-11-02T20:23:14.195Zport_mac:
5742023-11-02T20:23:14.632Zproxy arp:
5752023-11-02T20:23:14.689Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-11-02T20:23:14.718Z+ success 'softnpu zone exists'
5772023-11-02T20:23:14.747Z+ set +x
5782023-11-02T20:23:14.770Zsoftnpu zone exists
5792023-11-02T20:23:14.804Z
5802023-11-02T20:23:14.831Zreal 44.430511218
5812023-11-02T20:23:14.871Zuser 0.705334829
5822023-11-02T20:23:14.895Zsys 1.609266868
5832023-11-02T20:23:14.919Ztrap 0.000084038
5842023-11-02T20:23:14.944Ztflt 0.000181125
5852023-11-02T20:23:14.968Zdflt 0.000718829
5862023-11-02T20:23:15.004Zkflt 0.000000000
5872023-11-02T20:23:15.032Zlock 35:03.509765910
5882023-11-02T20:23:15.056Zslp 2:12.671942342
5892023-11-02T20:23:15.089Zlat 0.069062193
5902023-11-02T20:23:15.115Zstop 0.364041679
5912023-11-02T20:23:15.139Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-02T20:23:15.189Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-02T20:23:15.228Z+ SILO_NAME=recovery
5942023-11-02T20:23:15.253Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-02T20:23:15.279Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-02T20:23:15.303Z+ sed -i~ '
5972023-11-02T20:23:15.329Z /^external_dns_ips/c\
5982023-11-02T20:23:15.353Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-11-02T20:23:15.378Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-02T20:23:15.406Z /^first/c\
6012023-11-02T20:23:15.434Zfirst = "10.151.1.164"
6022023-11-02T20:23:15.468Z /^last/c\
6032023-11-02T20:23:15.495Zlast = "10.151.1.173"
6042023-11-02T20:23:15.523Z }
6052023-11-02T20:23:15.548Z /^\[rack_network_config/,/^$/ {
6062023-11-02T20:23:15.582Z /^infra_ip_first/c\
6072023-11-02T20:23:15.614Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6082023-11-02T20:23:15.664Zinfra_ip_first = "10.151.1.174"
6092023-11-02T20:23:15.771Z+++ pkg/config-rss.toml Thu Nov 2 20:23:12 2023
6102023-11-02T20:23:15.803Z /^infra_ip_last/c\
6112023-11-02T20:23:15.838Z@@ -41,7 +41,7 @@
6122023-11-02T20:23:16.283Zinfra_ip_last = "10.151.1.174"
6132023-11-02T20:23:16.308Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6142023-11-02T20:23:16.333Z }
6152023-11-02T20:23:16.357Z # must be contained in one of the "internal services" IP Pool ranges listed
6162023-11-02T20:23:16.382Z /^\[\[rack_network_config.ports/,/^$/ {
6172023-11-02T20:23:16.412Z # below.
6182023-11-02T20:23:16.441Z /^routes/c\
6192023-11-02T20:23:16.471Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6202023-11-02T20:23:16.905Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6212023-11-02T20:23:16.933Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6222023-11-02T20:23:16.959Z /^addresses/c\
6232023-11-02T20:23:16.993Z
6242023-11-02T20:23:17.018Zaddresses = \["10.151.1.174/32"\]
6252023-11-02T20:23:17.046Z # Initial TLS certificates for the external API
6262023-11-02T20:23:17.090Z }
6272023-11-02T20:23:17.526Z #
6282023-11-02T20:23:17.560Z' pkg/config-rss.toml
6292023-11-02T20:23:17.618Z@@ -77,8 +77,8 @@
6302023-11-02T20:23:17.648Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6312023-11-02T20:23:17.679Z #
6322023-11-02T20:23:17.717Z+ true
6332023-11-02T20:23:17.749Z # For more on this and what to put here, see docs/how-to-run.adoc.
6342023-11-02T20:23:17.776Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6352023-11-02T20:23:17.802Z [[internal_services_ip_pool_ranges]]
6362023-11-02T20:23:17.858Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6372023-11-02T20:23:17.885Z-first = "192.168.1.20"
6382023-11-02T20:23:17.909Z+ TLS_NAME=recovery.sys.oxide.test
6392023-11-02T20:23:17.933Z-last = "192.168.1.29"
6402023-11-02T20:23:17.957Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6412023-11-02T20:23:17.983Z+first = "10.151.1.164"
6422023-11-02T20:23:18.038Z.+.....+....+...+..+................+......+.....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+....+......+.........+..+...+............+....+.............................+.+...........+...+.......+.....+..........+..+.+......+...+..+...................+......+.....+.+...+.....+......+.+...+......+.......................................+.........+........+....+......+..............+.+........+.+........+.+..+...+....+.....+............+......+...+......+...+............+...+.+.........+......+.....+.......+...+......+.........+.........+..+..........+...+...+............+..+......+............+.............+.....+.......+...........+.......+.....+...+.......+..+.......+......+..............+.+.....+.......+.....+...+..................+..........+......+...........+..........+.....+...+....+......+.........+..+.+.........+..................+.....+.+.....+.+......+...+.....+.........+.+..+...............+.......+..+....+.....+......+...............+...................+..+...+..........+....................+......+.+..............+.+..............+...+.......+.............................+.+.....+....+.....+.+......+.....+....+...+.....+......+.+....................+......+....+.........+.....+..........+....................+.......+...............+....................+.............+..+....+...+.....+...+......................+.....+................+.........+...+.........+...+......+.....+......+...+.......+........+.............+..............+.+..................+.....+.+..............+.+...+...........+..........+...........+....+..............................+.....+.......+...............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6432023-11-02T20:23:18.472Z+last = "10.151.1.173"
6442023-11-02T20:23:18.502Z..+..+.........+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+....+.....+...+.......+..............+.........+.......+.........+..+...+.+..............+....+.....+.............+...+..+....+.....+......+......................+..........................+....+...........+....+...........+.+......+...+.........+........+....+...+..+....+.........+.....+............+....+......+......+...+..+..........+..+.......+...........+.......+..........................+....+.....+...+..................+...................+......+.....................+...+.....+...+....+.................................+..+....+......+......+..+.+..+.......+...+........+.+...........+....+...........+...+.......+.................+.+..+.........+.........+.+........+.......+.....+.+.........+......+...+..+...............+...+............+.......+..+..........+...+...........+.+......+.....+...+....+............+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6452023-11-02T20:23:18.527Z
6462023-11-02T20:23:18.551Z-----
6472023-11-02T20:23:18.575Z # TODO - this configuration is subject to change going forward. Ultimately these
6482023-11-02T20:23:18.601Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6492023-11-02T20:23:18.628Z # parameters should be provided to the control plane via wicket, but we need to
6502023-11-02T20:23:18.652Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6512023-11-02T20:23:18.681Z@@ -92,8 +92,8 @@
6522023-11-02T20:23:18.716Z+ rmdir pkg
6532023-11-02T20:23:18.747Z # A range of IP addresses used by Boundary Services on the external network. In
6542023-11-02T20:23:18.776Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6552023-11-02T20:23:18.807Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6562023-11-02T20:23:18.832Z+ pfexec mkdir -p /usr/oxide
6572023-11-02T20:23:18.907Z # With softnpu, only one address is used.
6582023-11-02T20:23:19.344Z+ pfexec rm -f /usr/oxide/tcpproxy
6592023-11-02T20:23:19.772Z-infra_ip_first = "192.168.1.30"
6602023-11-02T20:23:19.796Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6612023-11-02T20:23:19.819Z-infra_ip_last = "192.168.1.30"
6622023-11-02T20:23:19.843Z+ pfexec chmod +x /usr/oxide/tcpproxy
6632023-11-02T20:23:19.866Z+infra_ip_first = "10.151.1.174"
6642023-11-02T20:23:19.890Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6652023-11-02T20:23:19.946Z+infra_ip_last = "10.151.1.174"
6662023-11-02T20:23:19.977Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6672023-11-02T20:23:20.429Z
6682023-11-02T20:23:20.465Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6692023-11-02T20:23:20.490Z # Configurations for BGP routers to run on the scrimlets.
6702023-11-02T20:23:20.514Z+ OMICRON_NO_UNINSTALL=1
6712023-11-02T20:23:20.537Z bgp = []
6722023-11-02T20:23:20.561Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6732023-11-02T20:23:20.585Z@@ -101,9 +101,9 @@
6742023-11-02T20:23:20.610ZNov 02 20:23:14.755 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6752023-11-02T20:23:20.636Z # You can configure multiple uplinks by repeating the following stanza
6762023-11-02T20:23:20.681ZNov 02 20:23:14.757 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6772023-11-02T20:23:21.148Z [[rack_network_config.ports]]
6782023-11-02T20:23:21.181ZNov 02 20:23:14.757 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6792023-11-02T20:23:21.205Z # Routes associated with this port.
6802023-11-02T20:23:21.229ZNov 02 20:23:14.758 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6812023-11-02T20:23:21.253Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6822023-11-02T20:23:21.278ZNov 02 20:23:14.758 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6832023-11-02T20:23:21.302Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6842023-11-02T20:23:21.336ZNov 02 20:23:14.758 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6852023-11-02T20:23:22.176Z # Addresses associated with this port.
6862023-11-02T20:23:22.204ZNov 02 20:23:14.758 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6872023-11-02T20:23:22.231Z-addresses = ["192.168.1.30/32"]
6882023-11-02T20:23:22.255ZNov 02 20:23:14.758 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6892023-11-02T20:23:22.281Z+addresses = ["10.151.1.174/32"]
6902023-11-02T20:23:22.314ZNov 02 20:23:14.758 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6912023-11-02T20:23:22.351Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6922023-11-02T20:23:22.490ZNov 02 20:23:14.758 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6932023-11-02T20:23:22.521Z port = "qsfp0"
6942023-11-02T20:23:22.561ZNov 02 20:23:14.758 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6952023-11-02T20:23:22.641Z # The speed of this port.
6962023-11-02T20:23:22.665ZNov 02 20:23:14.758 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6972023-11-02T20:23:22.732Za pkg/config-rss.toml 7K
6982023-11-02T20:23:22.756ZNov 02 20:23:14.758 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6992023-11-02T20:23:22.779Za pkg/initial-tls-cert.pem 2K
7002023-11-02T20:23:22.803ZNov 02 20:23:14.758 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7012023-11-02T20:23:22.833Za pkg/initial-tls-key.pem 4K
7022023-11-02T20:23:22.859ZNov 02 20:23:14.758 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7032023-11-02T20:23:22.890ZNov 02 20:23:14.758 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7042023-11-02T20:23:22.914ZNov 02 20:23:14.758 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7052023-11-02T20:23:22.989ZNov 02 20:23:14.977 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-11-02T20:23:23.035ZNov 02 20:23:15.013 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-02T20:23:23.125ZNov 02 20:23:15.076 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-02T20:23:23.564Z
7092023-11-02T20:23:23.588Zreal 0.459905795
7102023-11-02T20:23:23.612Zuser 0.098394038
7112023-11-02T20:23:23.636Zsys 1.745830688
7122023-11-02T20:23:23.665Ztrap 0.000000260
7132023-11-02T20:23:23.694Ztflt 0.000022040
7142023-11-02T20:23:24.163Zdflt 0.000906048
7152023-11-02T20:23:24.187Zkflt 0.000000000
7162023-11-02T20:23:24.210Zlock 41.352120759
7172023-11-02T20:23:24.239Zslp 0.667499466
7182023-11-02T20:23:24.264Zlat 0.002947864
7192023-11-02T20:23:24.288Zstop 0.222469674
7202023-11-02T20:23:24.311Z+ retry=0
7212023-11-02T20:23:24.340Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-02T20:23:24.372Z+ [[ 0 -gt 30 ]]
7232023-11-02T20:23:24.403Z+ sleep 1
7242023-11-02T20:23:24.432Z+ retry=1
7252023-11-02T20:23:24.458Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-02T20:23:24.482Z+ [[ 1 -gt 30 ]]
7272023-11-02T20:23:24.912Z+ sleep 1
7282023-11-02T20:23:24.940Z+ retry=2
7292023-11-02T20:23:24.973Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-02T20:23:56.773Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7312023-11-02T20:23:57.132Zlocal v6:
7322023-11-02T20:23:57.226Zfe80::8cb3:86ff:fe1a:d8cf
7332023-11-02T20:23:57.296Zfd00:99::1
7342023-11-02T20:23:57.337Zlocal v4:
7352023-11-02T20:23:57.365Z10.151.1.174
7362023-11-02T20:23:57.404Zrouter v6:
7372023-11-02T20:23:58.168Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::a87a:cff:fe1b:9ea7 (1)
7382023-11-02T20:23:58.213Zfd00:1122:3344:101::/64 -> fe80::a87a:cff:fe1b:9ea7 (1)
7392023-11-02T20:23:58.257Zrouter v4_idx:
7402023-11-02T20:23:58.293Z0.0.0.0/0 -> 0
7412023-11-02T20:23:58.497Zrouter v4_routes:
7422023-11-02T20:23:58.550Z0 -> 10.151.1.100 (2)
7432023-11-02T20:23:58.577Zresolver v4:
7442023-11-02T20:23:58.610Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7452023-11-02T20:23:58.644Z10.151.1.174 -> ce:a0:d8:b2:bd:e4
7462023-11-02T20:23:58.679Zresolver v6:
7472023-11-02T20:23:58.736Zfe80::a87a:cff:fe1b:9ea7 -> aa:7a:0c:1b:9e:a7
7482023-11-02T20:23:58.776Zfe80::8cb3:86ff:fe1a:d8cf -> 8e:b3:86:1a:d8:cf
7492023-11-02T20:23:58.878Znat v4:
7502023-11-02T20:23:59.318Znat_v6:
7512023-11-02T20:23:59.363Zport_mac:
7522023-11-02T20:23:59.401Z1: a8:40:25:48:94:7f
7532023-11-02T20:23:59.431Z1000: a8:40:25:48:94:7e
7542023-11-02T20:23:59.455Zproxy arp:
7552023-11-02T20:23:59.479Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7562023-11-02T20:23:59.504Z+ retry=0
7572023-11-02T20:23:59.532Z+++ zoneadm list -n
7582023-11-02T20:23:59.556Z+++ grep oxz_ntp
7592023-11-02T20:23:59.581Z++ pfexec svcs -z -Hostate oxide/ntp
7602023-11-02T20:23:59.607Zsvcs: zone '-Hostate': entity not found
7612023-11-02T20:23:59.639Z++ true
7622023-11-02T20:24:00.090Z+ [[ '' != online ]]
7632023-11-02T20:24:00.114Z+ [[ 0 -gt 60 ]]
7642023-11-02T20:24:00.138Z+ sleep 1
7652023-11-02T20:24:00.161Z+ retry=1
7662023-11-02T20:24:00.192Z+++ zoneadm list -n
7672023-11-02T20:24:00.217Z+++ grep oxz_ntp
7682023-11-02T20:24:00.240Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-11-02T20:24:00.267Zsvcs: zone '-Hostate': entity not found
7702023-11-02T20:24:00.291Z++ true
7712023-11-02T20:24:00.315Z+ [[ '' != online ]]
7722023-11-02T20:24:00.341Z+ [[ 1 -gt 60 ]]
7732023-11-02T20:24:00.367Z+ sleep 1
7742023-11-02T20:24:00.391Z+ retry=2
7752023-11-02T20:24:00.441Z+++ zoneadm list -n
7762023-11-02T20:24:00.467Z+++ grep oxz_ntp
7772023-11-02T20:24:00.500Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-11-02T20:24:00.536Zsvcs: zone '-Hostate': entity not found
7792023-11-02T20:24:00.578Z++ true
7802023-11-02T20:24:00.618Z+ [[ '' != online ]]
7812023-11-02T20:24:00.655Z+ [[ 2 -gt 60 ]]
7822023-11-02T20:24:00.700Z+ sleep 1
7832023-11-02T20:24:00.732Z+ retry=3
7842023-11-02T20:24:00.763Z+++ zoneadm list -n
7852023-11-02T20:24:00.791Z+++ grep oxz_ntp
7862023-11-02T20:24:00.819Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-11-02T20:24:00.856Zsvcs: zone '-Hostate': entity not found
7882023-11-02T20:24:00.884Z++ true
7892023-11-02T20:24:00.908Z+ [[ '' != online ]]
7902023-11-02T20:24:00.954Z+ [[ 3 -gt 60 ]]
7912023-11-02T20:24:00.980Z+ sleep 1
7922023-11-02T20:24:01.005Z+ retry=4
7932023-11-02T20:24:01.031Z+++ zoneadm list -n
7942023-11-02T20:24:01.058Z+++ grep oxz_ntp
7952023-11-02T20:24:01.094Z++ pfexec svcs -z -Hostate oxide/ntp
7962023-11-02T20:24:01.119Zsvcs: zone '-Hostate': entity not found
7972023-11-02T20:24:01.150Z++ true
7982023-11-02T20:24:01.187Z+ [[ '' != online ]]
7992023-11-02T20:24:01.217Z+ [[ 4 -gt 60 ]]
8002023-11-02T20:24:01.692Z+ sleep 1
8012023-11-02T20:24:01.725Z+ retry=5
8022023-11-02T20:24:01.754Z+++ zoneadm list -n
8032023-11-02T20:24:01.782Z+++ grep oxz_ntp
8042023-11-02T20:24:01.809Z++ pfexec svcs -z -Hostate oxide/ntp
8052023-11-02T20:24:01.835Zsvcs: zone '-Hostate': entity not found
8062023-11-02T20:24:01.862Z++ true
8072023-11-02T20:24:01.887Z+ [[ '' != online ]]
8082023-11-02T20:24:01.915Z+ [[ 5 -gt 60 ]]
8092023-11-02T20:24:01.943Z+ sleep 1
8102023-11-02T20:24:02.000Z+ retry=6
8112023-11-02T20:24:02.038Z+++ zoneadm list -n
8122023-11-02T20:24:02.071Z+++ grep oxz_ntp
8132023-11-02T20:24:02.105Z++ pfexec svcs -z -Hostate oxide/ntp
8142023-11-02T20:24:02.715Zsvcs: zone '-Hostate': entity not found
8152023-11-02T20:24:02.741Z++ true
8162023-11-02T20:24:02.765Z+ [[ '' != online ]]
8172023-11-02T20:24:02.790Z+ [[ 6 -gt 60 ]]
8182023-11-02T20:24:02.815Z+ sleep 1
8192023-11-02T20:24:02.846Z+ retry=7
8202023-11-02T20:24:02.875Z+++ zoneadm list -n
8212023-11-02T20:24:02.899Z+++ grep oxz_ntp
8222023-11-02T20:24:02.934Z++ pfexec svcs -z -Hostate oxide/ntp
8232023-11-02T20:24:02.969Zsvcs: zone '-Hostate': entity not found
8242023-11-02T20:24:02.996Z++ true
8252023-11-02T20:24:03.042Z+ [[ '' != online ]]
8262023-11-02T20:24:03.081Z+ [[ 7 -gt 60 ]]
8272023-11-02T20:24:03.107Z+ sleep 1
8282023-11-02T20:24:03.133Z+ retry=8
8292023-11-02T20:24:03.161Z+++ zoneadm list -n
8302023-11-02T20:24:03.185Z+++ grep oxz_ntp
8312023-11-02T20:24:03.210Z++ pfexec svcs -z -Hostate oxide/ntp
8322023-11-02T20:24:03.237Zsvcs: zone '-Hostate': entity not found
8332023-11-02T20:24:03.261Z++ true
8342023-11-02T20:24:03.295Z+ [[ '' != online ]]
8352023-11-02T20:24:03.322Z+ [[ 8 -gt 60 ]]
8362023-11-02T20:24:03.346Z+ sleep 1
8372023-11-02T20:24:03.370Z+ retry=9
8382023-11-02T20:24:03.394Z+++ zoneadm list -n
8392023-11-02T20:24:03.418Z+++ grep oxz_ntp
8402023-11-02T20:24:03.443Z++ pfexec svcs -z -Hostate oxide/ntp
8412023-11-02T20:24:03.474Zsvcs: zone '-Hostate': entity not found
8422023-11-02T20:24:03.501Z++ true
8432023-11-02T20:24:03.528Z+ [[ '' != online ]]
8442023-11-02T20:24:03.552Z+ [[ 9 -gt 60 ]]
8452023-11-02T20:24:03.578Z+ sleep 1
8462023-11-02T20:24:03.605Z+ retry=10
8472023-11-02T20:24:03.633Z+++ zoneadm list -n
8482023-11-02T20:24:03.664Z+++ grep oxz_ntp
8492023-11-02T20:24:03.689Z++ pfexec svcs -z -Hostate oxide/ntp
8502023-11-02T20:24:03.713Zsvcs: zone '-Hostate': entity not found
8512023-11-02T20:24:03.744Z++ true
8522023-11-02T20:24:04.594Z+ [[ '' != online ]]
8532023-11-02T20:24:04.622Z+ [[ 10 -gt 60 ]]
8542023-11-02T20:24:05.065Z+ sleep 1
8552023-11-02T20:24:05.530Z+ retry=11
8562023-11-02T20:24:05.560Z+++ zoneadm list -n
8572023-11-02T20:24:05.584Z+++ grep oxz_ntp
8582023-11-02T20:24:05.612Z++ pfexec svcs -z -Hostate oxide/ntp
8592023-11-02T20:24:05.639Zsvcs: zone '-Hostate': entity not found
8602023-11-02T20:24:05.664Z++ true
8612023-11-02T20:24:05.689Z+ [[ '' != online ]]
8622023-11-02T20:24:05.713Z+ [[ 11 -gt 60 ]]
8632023-11-02T20:24:05.741Z+ sleep 1
8642023-11-02T20:24:05.768Z+ retry=12
8652023-11-02T20:24:05.802Z+++ zoneadm list -n
8662023-11-02T20:24:05.828Z+++ grep oxz_ntp
8672023-11-02T20:24:05.861Z++ pfexec svcs -z -Hostate oxide/ntp
8682023-11-02T20:24:05.887Zsvcs: zone '-Hostate': entity not found
8692023-11-02T20:24:05.917Z++ true
8702023-11-02T20:24:05.953Z+ [[ '' != online ]]
8712023-11-02T20:24:05.992Z+ [[ 12 -gt 60 ]]
8722023-11-02T20:24:06.022Z+ sleep 1
8732023-11-02T20:24:06.052Z+ retry=13
8742023-11-02T20:24:06.084Z+++ zoneadm list -n
8752023-11-02T20:24:06.112Z+++ grep oxz_ntp
8762023-11-02T20:24:06.141Z++ pfexec svcs -z oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b -Hostate oxide/ntp
8772023-11-02T20:24:06.207Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8782023-11-02T20:24:06.235Z++ true
8792023-11-02T20:24:06.667Z+ [[ '' != online ]]
8802023-11-02T20:24:06.693Z+ [[ 13 -gt 60 ]]
8812023-11-02T20:24:06.718Z+ sleep 1
8822023-11-02T20:24:06.742Z+ retry=14
8832023-11-02T20:24:06.766Z+++ zoneadm list -n
8842023-11-02T20:24:06.790Z+++ grep oxz_ntp
8852023-11-02T20:24:06.817Z++ pfexec svcs -z oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b -Hostate oxide/ntp
8862023-11-02T20:24:06.841Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
 
8872023-11-02T20:24:06.857Zbuskin console login: Nov 2 20:24:06 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8882023-11-02T20:24:06.865Z++ true
8892023-11-02T20:24:06.888Z+ [[ '' != online ]]
8902023-11-02T20:24:06.914Z+ [[ 14 -gt 60 ]]
8912023-11-02T20:24:06.938Z+ sleep 1
8922023-11-02T20:24:06.988Z+ retry=15
8932023-11-02T20:24:07.023Z+++ zoneadm list -n
8942023-11-02T20:24:07.056Z+++ grep oxz_ntp
8952023-11-02T20:24:07.082Z++ pfexec svcs -z oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b -Hostate oxide/ntp
8962023-11-02T20:24:07.109Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8972023-11-02T20:24:07.137Z++ true
8982023-11-02T20:24:07.162Z+ [[ '' != online ]]
8992023-11-02T20:24:07.186Z+ [[ 15 -gt 60 ]]
9002023-11-02T20:24:07.237Z+ sleep 1
9012023-11-02T20:24:07.261Z+ retry=16
9022023-11-02T20:24:07.284Z+++ zoneadm list -n
9032023-11-02T20:24:07.316Z+++ grep oxz_ntp
9042023-11-02T20:24:07.340Z++ pfexec svcs -z oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b -Hostate oxide/ntp
9052023-11-02T20:24:07.367Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9062023-11-02T20:24:07.393Z++ true
9072023-11-02T20:24:07.421Z+ [[ '' != online ]]
9082023-11-02T20:24:07.460Z+ [[ 16 -gt 60 ]]
9092023-11-02T20:24:08.017Z+ sleep 1
9102023-11-02T20:24:08.050Z+ retry=17
9112023-11-02T20:24:08.083Z+++ zoneadm list -n
9122023-11-02T20:24:08.928Z+++ grep oxz_ntp
9132023-11-02T20:24:08.958Z++ pfexec svcs -z oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b -Hostate oxide/ntp
9142023-11-02T20:24:08.983Z+ [[ disabled != online ]]
9152023-11-02T20:24:09.427Z+ [[ 17 -gt 60 ]]
9162023-11-02T20:24:09.461Z+ sleep 1
9172023-11-02T20:24:09.907Z+ retry=18
9182023-11-02T20:24:09.932Z+++ zoneadm list -n
9192023-11-02T20:24:09.956Z+++ grep oxz_ntp
9202023-11-02T20:24:09.985Z++ pfexec svcs -z oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b -Hostate oxide/ntp
9212023-11-02T20:24:10.018Z+ [[ disabled != online ]]
9222023-11-02T20:24:10.052Z+ [[ 18 -gt 60 ]]
9232023-11-02T20:24:10.082Z+ sleep 1
9242023-11-02T20:24:10.107Z+ retry=19
9252023-11-02T20:24:10.131Z+++ zoneadm list -n
9262023-11-02T20:24:10.155Z+++ grep oxz_ntp
9272023-11-02T20:24:10.180Z++ pfexec svcs -z oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b -Hostate oxide/ntp
9282023-11-02T20:24:10.204Z+ [[ disabled != online ]]
9292023-11-02T20:24:10.228Z+ [[ 19 -gt 60 ]]
9302023-11-02T20:24:10.263Z+ sleep 1
9312023-11-02T20:24:10.295Z+ retry=20
9322023-11-02T20:24:10.757Z+++ zoneadm list -n
9332023-11-02T20:24:10.783Z+++ grep oxz_ntp
9342023-11-02T20:24:10.808Z++ pfexec svcs -z oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b -Hostate oxide/ntp
9352023-11-02T20:24:10.837Z+ [[ disabled != online ]]
9362023-11-02T20:24:10.865Z+ [[ 20 -gt 60 ]]
9372023-11-02T20:24:10.895Z+ sleep 1
9382023-11-02T20:24:10.925Z+ retry=21
9392023-11-02T20:24:10.955Z+++ zoneadm list -n
9402023-11-02T20:24:10.984Z+++ grep oxz_ntp
9412023-11-02T20:24:11.014Z++ pfexec svcs -z oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b -Hostate oxide/ntp
9422023-11-02T20:24:11.045Z+ [[ disabled != online ]]
9432023-11-02T20:24:11.072Z+ [[ 21 -gt 60 ]]
9442023-11-02T20:24:11.104Z+ sleep 1
9452023-11-02T20:24:11.133Z+ retry=22
9462023-11-02T20:24:11.163Z+++ zoneadm list -n
9472023-11-02T20:24:11.188Z+++ grep oxz_ntp
9482023-11-02T20:24:11.213Z++ pfexec svcs -z oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b -Hostate oxide/ntp
9492023-11-02T20:24:11.243Z+ [[ disabled != online ]]
9502023-11-02T20:24:11.272Z+ [[ 22 -gt 60 ]]
9512023-11-02T20:24:11.302Z+ sleep 1
9522023-11-02T20:24:11.347Z+ retry=23
9532023-11-02T20:24:11.509Z+++ zoneadm list -n
9542023-11-02T20:24:11.586Z+++ grep oxz_ntp
9552023-11-02T20:24:11.617Z++ pfexec svcs -z oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b -Hostate oxide/ntp
9562023-11-02T20:24:11.648Z+ [[ disabled != online ]]
9572023-11-02T20:24:11.677Z+ [[ 23 -gt 60 ]]
9582023-11-02T20:24:11.710Z+ sleep 1
9592023-11-02T20:24:11.737Z+ retry=24
9602023-11-02T20:24:11.762Z+++ zoneadm list -n
9612023-11-02T20:24:11.789Z+++ grep oxz_ntp
9622023-11-02T20:24:12.227Z++ pfexec svcs -z oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b -Hostate oxide/ntp
9632023-11-02T20:24:12.254Z+ [[ disabled != online ]]
9642023-11-02T20:24:12.300Z+ [[ 24 -gt 60 ]]
9652023-11-02T20:24:12.324Z+ sleep 1
9662023-11-02T20:24:12.348Z+ retry=25
9672023-11-02T20:24:12.386Z+++ zoneadm list -n
9682023-11-02T20:24:12.523Z+++ grep oxz_ntp
9692023-11-02T20:24:12.548Z++ pfexec svcs -z oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b -Hostate oxide/ntp
9702023-11-02T20:24:12.575Z+ [[ online != online ]]
9712023-11-02T20:24:12.603Z+ echo 'Waited for chrony: 25s'
9722023-11-02T20:24:12.628Z+ export RUST_BACKTRACE=1
9732023-11-02T20:24:12.826ZWaited for chrony: 25s
9742023-11-02T20:24:13.279Z+ RUST_BACKTRACE=1
9752023-11-02T20:24:13.310Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9762023-11-02T20:24:13.335Z+ ./tests/bootstrap
9772023-11-02T20:24:13.371ZUsing external DNS server at 10.151.1.164:53
9782023-11-02T20:24:13.397Z2023-11-02 20:24:08.071491661 UTC: attempting to log into API
9792023-11-02T20:24:22.590Z2023-11-02 20:24:22.547195481 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9802023-11-02T20:24:23.592Z2023-11-02 20:24:23.550977978 UTC: attempting to log into API
9812023-11-02T20:24:38.621Z2023-11-02 20:24:38.607116601 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9822023-11-02T20:24:39.627Z2023-11-02 20:24:39.608973570 UTC: attempting to log into API
9832023-11-02T20:24:54.659Z2023-11-02 20:24:54.646154392 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9842023-11-02T20:24:55.661Z2023-11-02 20:24:55.647898640 UTC: attempting to log into API
 
9852023-11-02T20:25:00.622ZNov 2 20:24:59 bukin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
9862023-11-02T20:25:19.935ZNov 2 20:25:06 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9872023-11-02T20:25:19.998Z2023-11-02 20:25:10.702961367 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9882023-11-02T20:25:20.048ZNov 2 20:25:13 bukin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9892023-11-02T20:25:20.091Z2023-11-02 20:25:11.705206453 UTC: attempting to log into API
 
9902023-11-02T20:25:20.631ZNov 2 20:25:20 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9912023-11-02T20:25:26.753Z2023-11-02 20:25:26.739121795 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9922023-11-02T20:25:27.757Z2023-11-02 20:25:27.740932004 UTC: attempting to log into API
 
9932023-11-02T20:25:28.452ZNov 2 20:25:27 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9942023-11-02T20:25:42.788Z2023-11-02 20:25:42.774042943 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9952023-11-02T20:25:43.791Z2023-11-02 20:25:43.775154072 UTC: attempting to log into API
9962023-11-02T20:25:44.688Z2023-11-02 20:25:44.672557260 UTC: login succeeded
9972023-11-02T20:25:44.719Zwaiting for nexus to come up...
9982023-11-02T20:25:44.788Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9992023-11-02T20:25:44.885Zensuring datasets are ready...
10002023-11-02T20:25:49.934Zlet's roll.
10012023-11-02T20:25:49.963Z+ rm ./tests/bootstrap
10022023-11-02T20:25:50.002Z+ for test_bin in tests/*
10032023-11-02T20:25:50.039Z+ ./tests/bootstrap-13d7796e9c457a2a
10042023-11-02T20:25:50.066Z
10052023-11-02T20:25:50.092Zrunning 0 tests
10062023-11-02T20:25:50.119Z
10072023-11-02T20:25:50.147Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10082023-11-02T20:25:50.175Z
10092023-11-02T20:25:50.201Z+ for test_bin in tests/*
10102023-11-02T20:25:50.226Z+ ./tests/end_to_end_tests-cc8f9cfe063d8adb
10112023-11-02T20:25:50.254Z
10122023-11-02T20:25:50.280Zrunning 2 tests
10132023-11-02T20:25:50.313Ztest no_spoof::no_spoof_here ... ok
10142023-11-02T20:26:50.027Ztest instance_launch::instance_launch has been running for over 60 seconds
10152023-11-02T20:29:06.009Ztest instance_launch::instance_launch ... ok
10162023-11-02T20:29:06.189Z
10172023-11-02T20:29:06.248Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 180.94s
10182023-11-02T20:29:07.023Z
10192023-11-02T20:29:07.056Z+ _exit_trap
10202023-11-02T20:29:07.112Z+ local status=0
10212023-11-02T20:29:07.154Z+ [[ 0 -eq 0 ]]
10222023-11-02T20:29:07.198Z+ exit 0
10232023-11-02T20:29:08.552Zprocess exited: duration 391705 ms, exit code 0
 
10242023-11-02T20:29:09.016Zfound 47 output files
10252023-11-02T20:29:09.047Zuploading: /var/svc/log/oxide-sled-agent:default.log (413262 bytes)
10262023-11-02T20:29:10.306Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10272023-11-02T20:29:10.334Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_24dd337d-3eaa-4a92-84d9-fd52ec451d3c/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
10282023-11-02T20:29:11.481Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_24dd337d-3eaa-4a92-84d9-fd52ec451d3c/root/var/svc/log/oxide-crucible-agent:default.log
10292023-11-02T20:29:11.519Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_24dd337d-3eaa-4a92-84d9-fd52ec451d3c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10302023-11-02T20:29:12.611Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_24dd337d-3eaa-4a92-84d9-fd52ec451d3c/root/var/svc/log/oxide-crucible-downstairs:default.log
10312023-11-02T20:29:12.649Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_24dd337d-3eaa-4a92-84d9-fd52ec451d3c/root/var/svc/log/oxide-crucible-downstairs:downstairs-cc3688b3-d187-47bc-9a26-951b1b41393c.log (5536 bytes)
10322023-11-02T20:29:13.742Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_24dd337d-3eaa-4a92-84d9-fd52ec451d3c/root/var/svc/log/oxide-crucible-downstairs:downstairs-cc3688b3-d187-47bc-9a26-951b1b41393c.log
10332023-11-02T20:29:13.769Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_96846d16-ba49-4bcf-a534-0aae27d07dd3/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10342023-11-02T20:29:14.866Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_96846d16-ba49-4bcf-a534-0aae27d07dd3/root/var/svc/log/oxide-crucible-agent:default.log
10352023-11-02T20:29:14.899Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_96846d16-ba49-4bcf-a534-0aae27d07dd3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10362023-11-02T20:29:16.395Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_96846d16-ba49-4bcf-a534-0aae27d07dd3/root/var/svc/log/oxide-crucible-downstairs:default.log
10372023-11-02T20:29:16.426Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d01f530d-5cee-48ec-adad-2b3785c905fa/root/var/svc/log/oxide-crucible-agent:default.log (15007 bytes)
10382023-11-02T20:29:17.233Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d01f530d-5cee-48ec-adad-2b3785c905fa/root/var/svc/log/oxide-crucible-agent:default.log
10392023-11-02T20:29:17.257Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d01f530d-5cee-48ec-adad-2b3785c905fa/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10402023-11-02T20:29:18.337Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d01f530d-5cee-48ec-adad-2b3785c905fa/root/var/svc/log/oxide-crucible-downstairs:default.log
10412023-11-02T20:29:18.369Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d01f530d-5cee-48ec-adad-2b3785c905fa/root/var/svc/log/oxide-crucible-downstairs:downstairs-b1e0a253-00eb-40f8-9640-9baa309ea459.log (3050 bytes)
10422023-11-02T20:29:19.452Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d01f530d-5cee-48ec-adad-2b3785c905fa/root/var/svc/log/oxide-crucible-downstairs:downstairs-b1e0a253-00eb-40f8-9640-9baa309ea459.log
10432023-11-02T20:29:19.479Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2744983b-8c1d-4710-af3f-416cd4f62eba/root/var/svc/log/oxide-cockroachdb:default.log (5342 bytes)
10442023-11-02T20:29:21.024Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2744983b-8c1d-4710-af3f-416cd4f62eba/root/var/svc/log/oxide-cockroachdb:default.log
10452023-11-02T20:29:21.470Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c7115179-2729-4180-8b37-6d2189ab4b9c/root/var/svc/log/oxide-internal_dns:default.log (99483 bytes)
10462023-11-02T20:29:22.504Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c7115179-2729-4180-8b37-6d2189ab4b9c/root/var/svc/log/oxide-internal_dns:default.log
10472023-11-02T20:29:22.544Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_93210010-858f-4649-b0e0-2d1080c83690/root/var/svc/log/oxide-nexus:default.log (4568348 bytes)
10482023-11-02T20:29:24.187Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_93210010-858f-4649-b0e0-2d1080c83690/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4568348 -> 4569566
10492023-11-02T20:29:25.256Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_93210010-858f-4649-b0e0-2d1080c83690/root/var/svc/log/oxide-nexus:default.log
10502023-11-02T20:29:25.282Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_32bf42b4-f447-4517-b253-dcc649fd0468/root/var/svc/log/oxide-crucible-agent:default.log (14118 bytes)
10512023-11-02T20:29:26.384Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_32bf42b4-f447-4517-b253-dcc649fd0468/root/var/svc/log/oxide-crucible-agent:default.log
10522023-11-02T20:29:26.420Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_32bf42b4-f447-4517-b253-dcc649fd0468/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10532023-11-02T20:29:27.545Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_32bf42b4-f447-4517-b253-dcc649fd0468/root/var/svc/log/oxide-crucible-downstairs:default.log
10542023-11-02T20:29:27.579Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_32bf42b4-f447-4517-b253-dcc649fd0468/root/var/svc/log/oxide-crucible-downstairs:downstairs-0d126b93-a214-45d1-832f-447c0c922c7a.log (5537 bytes)
10552023-11-02T20:29:28.654Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_32bf42b4-f447-4517-b253-dcc649fd0468/root/var/svc/log/oxide-crucible-downstairs:downstairs-0d126b93-a214-45d1-832f-447c0c922c7a.log
10562023-11-02T20:29:28.681Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7ccbc7d6-8126-4821-bbbc-25f4b2b865b7/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10572023-11-02T20:29:29.786Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7ccbc7d6-8126-4821-bbbc-25f4b2b865b7/root/var/svc/log/oxide-crucible-agent:default.log
10582023-11-02T20:29:29.832Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7ccbc7d6-8126-4821-bbbc-25f4b2b865b7/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10592023-11-02T20:29:30.907Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7ccbc7d6-8126-4821-bbbc-25f4b2b865b7/root/var/svc/log/oxide-crucible-downstairs:default.log
10602023-11-02T20:29:30.935Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_2a8e6483-1fa2-40b6-98ba-3cf65ebf25aa/root/var/svc/log/oxide-external_dns:default.log (15611 bytes)
10612023-11-02T20:29:31.004Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_2a8e6483-1fa2-40b6-98ba-3cf65ebf25aa/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15611 -> 19386
10622023-11-02T20:29:32.074Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_2a8e6483-1fa2-40b6-98ba-3cf65ebf25aa/root/var/svc/log/oxide-external_dns:default.log
10632023-11-02T20:29:32.099Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_58d0b491-e509-4496-89b0-48e20b837292/root/var/svc/log/oxide-clickhouse:default.log (799104 bytes)
10642023-11-02T20:29:32.583Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_58d0b491-e509-4496-89b0-48e20b837292/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 799104 -> 854043
10652023-11-02T20:29:33.653Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_58d0b491-e509-4496-89b0-48e20b837292/root/var/svc/log/oxide-clickhouse:default.log
10662023-11-02T20:29:33.677Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_95b4c51d-7f94-4c1b-a43d-5e62c5eae847/root/var/svc/log/oxide-nexus:default.log (481307 bytes)
10672023-11-02T20:29:34.542Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_95b4c51d-7f94-4c1b-a43d-5e62c5eae847/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 481307 -> 541307
10682023-11-02T20:29:35.627Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_95b4c51d-7f94-4c1b-a43d-5e62c5eae847/root/var/svc/log/oxide-nexus:default.log
10692023-11-02T20:29:35.665Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_66d326a7-062f-404e-9623-0ee379694f33/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10702023-11-02T20:29:36.763Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_66d326a7-062f-404e-9623-0ee379694f33/root/var/svc/log/oxide-cockroachdb:default.log
10712023-11-02T20:29:36.789Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_2b78c87a-9a4f-4135-a614-2a00229c0a85/root/var/svc/log/oxide-external_dns:default.log (15614 bytes)
10722023-11-02T20:29:36.867Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_2b78c87a-9a4f-4135-a614-2a00229c0a85/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15614 -> 19389
10732023-11-02T20:29:37.939Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_2b78c87a-9a4f-4135-a614-2a00229c0a85/root/var/svc/log/oxide-external_dns:default.log
10742023-11-02T20:29:37.980Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_96120ff3-07ae-43a4-8f50-a1d214d4994f/root/var/svc/log/oxide-nexus:default.log (506441 bytes)
10752023-11-02T20:29:38.421Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_96120ff3-07ae-43a4-8f50-a1d214d4994f/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 506441 -> 566438
10762023-11-02T20:29:39.531Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_96120ff3-07ae-43a4-8f50-a1d214d4994f/root/var/svc/log/oxide-nexus:default.log
10772023-11-02T20:29:39.560Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b/root/var/svc/log/oxide-ntp:default.log (984 bytes)
10782023-11-02T20:29:40.712Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b/root/var/svc/log/oxide-ntp:default.log
10792023-11-02T20:29:41.150Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_85ad233a-4063-4e49-b457-b49d190db927/root/var/svc/log/oxide-cockroachdb:default.log (5366 bytes)
10802023-11-02T20:29:41.833Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_85ad233a-4063-4e49-b457-b49d190db927/root/var/svc/log/oxide-cockroachdb:default.log
10812023-11-02T20:29:41.859Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_1797e739-1b33-484d-9376-57acf3b336b2/root/var/svc/log/oxide-internal_dns:default.log (98256 bytes)
10822023-11-02T20:29:42.219Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_1797e739-1b33-484d-9376-57acf3b336b2/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 98256 -> 102007
10832023-11-02T20:29:43.319Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_1797e739-1b33-484d-9376-57acf3b336b2/root/var/svc/log/oxide-internal_dns:default.log
10842023-11-02T20:29:43.824Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_a849527a-de17-4b55-a21a-15c09c3cb8b1/root/var/svc/log/oxide-internal_dns:default.log (97851 bytes)
10852023-11-02T20:29:43.848Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_a849527a-de17-4b55-a21a-15c09c3cb8b1/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 97851 -> 101607
10862023-11-02T20:29:44.626Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_a849527a-de17-4b55-a21a-15c09c3cb8b1/root/var/svc/log/oxide-internal_dns:default.log
10872023-11-02T20:29:44.675Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_4e7c19e8-2e3a-4404-add5-61f8d74d4662/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10882023-11-02T20:29:45.754Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_4e7c19e8-2e3a-4404-add5-61f8d74d4662/root/var/svc/log/oxide-cockroachdb:default.log
10892023-11-02T20:29:45.783Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_60092536-7fc3-4814-b026-f4a58a8adfa9/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10902023-11-02T20:29:48.452Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_60092536-7fc3-4814-b026-f4a58a8adfa9/root/var/svc/log/oxide-cockroachdb:default.log
10912023-11-02T20:29:48.485Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df418c00-a7f7-47ef-8eab-b687d21768d6/root/var/svc/log/oxide-crucible-agent:default.log (14576 bytes)
10922023-11-02T20:29:48.962Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df418c00-a7f7-47ef-8eab-b687d21768d6/root/var/svc/log/oxide-crucible-agent:default.log
10932023-11-02T20:29:49.449Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df418c00-a7f7-47ef-8eab-b687d21768d6/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10942023-11-02T20:29:50.550Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df418c00-a7f7-47ef-8eab-b687d21768d6/root/var/svc/log/oxide-crucible-downstairs:default.log
10952023-11-02T20:29:50.600Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df418c00-a7f7-47ef-8eab-b687d21768d6/root/var/svc/log/oxide-crucible-downstairs:downstairs-789a5a01-c732-492a-8563-647003ea96d0.log (3048 bytes)
10962023-11-02T20:29:52.559Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_df418c00-a7f7-47ef-8eab-b687d21768d6/root/var/svc/log/oxide-crucible-downstairs:downstairs-789a5a01-c732-492a-8563-647003ea96d0.log
10972023-11-02T20:29:53.094Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_d49eecc4-8bc0-4d0f-80cd-72e1b0adff9c/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10982023-11-02T20:29:53.667Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_d49eecc4-8bc0-4d0f-80cd-72e1b0adff9c/root/var/svc/log/oxide-crucible-pantry:default.log
10992023-11-02T20:29:53.702Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_678ca45f-a7e7-44ff-82ee-f9f04320b863/root/var/svc/log/oxide-oximeter:default.log (209132 bytes)
11002023-11-02T20:29:54.274Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_678ca45f-a7e7-44ff-82ee-f9f04320b863/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 209132 -> 247374
11012023-11-02T20:29:55.360Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_678ca45f-a7e7-44ff-82ee-f9f04320b863/root/var/svc/log/oxide-oximeter:default.log
11022023-11-02T20:29:55.387Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_4ecca63c-3b66-43bf-9668-235ab1f4fc70/root/var/svc/log/oxide-crucible-agent:default.log (14578 bytes)
11032023-11-02T20:29:56.519Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_4ecca63c-3b66-43bf-9668-235ab1f4fc70/root/var/svc/log/oxide-crucible-agent:default.log
11042023-11-02T20:29:56.545Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_4ecca63c-3b66-43bf-9668-235ab1f4fc70/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11052023-11-02T20:29:57.629Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_4ecca63c-3b66-43bf-9668-235ab1f4fc70/root/var/svc/log/oxide-crucible-downstairs:default.log
11062023-11-02T20:29:58.062Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_4ecca63c-3b66-43bf-9668-235ab1f4fc70/root/var/svc/log/oxide-crucible-downstairs:downstairs-3e1ae206-1004-4fe4-b313-5554acad2c34.log (3049 bytes)
11072023-11-02T20:29:58.741Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_4ecca63c-3b66-43bf-9668-235ab1f4fc70/root/var/svc/log/oxide-crucible-downstairs:downstairs-3e1ae206-1004-4fe4-b313-5554acad2c34.log
11082023-11-02T20:29:58.775Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9c9622e0-aba2-444e-8a85-19aeb5895302/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
11092023-11-02T20:30:00.274Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9c9622e0-aba2-444e-8a85-19aeb5895302/root/var/svc/log/oxide-crucible-agent:default.log
11102023-11-02T20:30:00.300Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9c9622e0-aba2-444e-8a85-19aeb5895302/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11112023-11-02T20:30:01.386Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9c9622e0-aba2-444e-8a85-19aeb5895302/root/var/svc/log/oxide-crucible-downstairs:default.log
11122023-11-02T20:30:01.411Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cd068562-09fb-4c8b-9e1b-d951d2068256/root/var/svc/log/oxide-crucible-agent:default.log (13686 bytes)
11132023-11-02T20:30:02.549Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cd068562-09fb-4c8b-9e1b-d951d2068256/root/var/svc/log/oxide-crucible-agent:default.log
11142023-11-02T20:30:02.583Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cd068562-09fb-4c8b-9e1b-d951d2068256/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11152023-11-02T20:30:02.985Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cd068562-09fb-4c8b-9e1b-d951d2068256/root/var/svc/log/oxide-crucible-downstairs:default.log" changed size mid upload: 75 -> 0
11162023-11-02T20:30:04.088Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cd068562-09fb-4c8b-9e1b-d951d2068256/root/var/svc/log/oxide-crucible-downstairs:default.log
11172023-11-02T20:30:04.121Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cd068562-09fb-4c8b-9e1b-d951d2068256/root/var/svc/log/oxide-crucible-downstairs:downstairs-6701c733-30d3-4c1c-bbdf-9a0540f43e0e.log (5536 bytes)
11182023-11-02T20:30:04.152Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cd068562-09fb-4c8b-9e1b-d951d2068256/root/var/svc/log/oxide-crucible-downstairs:downstairs-6701c733-30d3-4c1c-bbdf-9a0540f43e0e.log" changed size mid upload: 5536 -> 0
11192023-11-02T20:30:05.212Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cd068562-09fb-4c8b-9e1b-d951d2068256/root/var/svc/log/oxide-crucible-downstairs:downstairs-6701c733-30d3-4c1c-bbdf-9a0540f43e0e.log
11202023-11-02T20:30:05.249Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_23aa5fd7-d772-455d-b803-1e0e9f484007/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
11212023-11-02T20:30:05.280Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_23aa5fd7-d772-455d-b803-1e0e9f484007/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 2338 -> 0
11222023-11-02T20:30:06.334Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_23aa5fd7-d772-455d-b803-1e0e9f484007/root/var/svc/log/oxide-crucible-pantry:default.log
11232023-11-02T20:30:06.361Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_e3bdc2c5-5d1e-4d7c-bf7c-303a3bff61f5/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
11242023-11-02T20:30:06.390Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_e3bdc2c5-5d1e-4d7c-bf7c-303a3bff61f5/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 2333 -> 0
11252023-11-02T20:30:07.850Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_e3bdc2c5-5d1e-4d7c-bf7c-303a3bff61f5/root/var/svc/log/oxide-crucible-pantry:default.log
11262023-11-02T20:30:07.882Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b/root/var/log/chrony/measurements.log (8494 bytes)
11272023-11-02T20:30:08.775Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b/root/var/log/chrony/measurements.log" changed size mid upload: 8494 -> 9590
11282023-11-02T20:30:09.445Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b/root/var/log/chrony/measurements.log
11292023-11-02T20:30:09.471Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b/root/var/log/chrony/statistics.log (3861 bytes)
11302023-11-02T20:30:09.512Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b/root/var/log/chrony/statistics.log" changed size mid upload: 3861 -> 5031
11312023-11-02T20:30:11.006Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b/root/var/log/chrony/statistics.log
11322023-11-02T20:30:11.041Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b/root/var/log/chrony/tracking.log (1456 bytes)
11332023-11-02T20:30:11.066Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b/root/var/log/chrony/tracking.log" changed size mid upload: 1456 -> 1588
11342023-11-02T20:30:12.545Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_1da4786b-e9b7-49f5-afe8-4c9e95c0156b/root/var/log/chrony/tracking.log