01HF7TKMJJ8FRJ96WH13A51H14: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HF7TMA5W81ZTCAM88FNH2JTN

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-14T22:02:02.994Zjob dependencies complete; ready to run (waiting for 53 m 25 s)
22023-11-14T22:02:50.076Zjob assigned to worker 01HF7XPB441NJYKJEHX7SJ56A6 (queued for 47 s)
32023-11-14T22:02:50.325Zxcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd
42023-11-14T22:02:50.495Zhttp://172.20.0.5:9969/ipxe/sock... ok
52023-11-14T22:02:50.644Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
62023-11-14T22:02:50.688Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
72023-11-14T22:02:50.772Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
82023-11-14T22:02:50.817Zhttp://172.20.0.5:9969/postboot/sock... ok
92023-11-14T22:02:50.862ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
102023-11-14T22:02:50.907Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
112023-11-14T22:02:50.951ZOxide Helios Version helios-2.0.22241 64-bit
122023-11-14T22:02:50.996ZConfiguring devices.
132023-11-14T22:02:51.041Z#################################################
142023-11-14T22:02:51.086Z#### ############# ##
152023-11-14T22:02:51.139Z### ### ############ ##
162023-11-14T22:02:51.188Z## ### # ## ### ## Oxide Computer Company ##
172023-11-14T22:02:51.232Z## ## ## ### # ### ##
182023-11-14T22:02:51.275Z## # ### #### #### This Station Under ##
192023-11-14T22:02:51.331Z### ### #### # ### Computer Control ##
202023-11-14T22:02:51.375Z#### #### ### ## ##
212023-11-14T22:02:51.420Z##################################################
222023-11-14T22:02:51.870ZHostname: sock
232023-11-14T22:02:51.929ZLoading smf(7) service descriptions:
242023-11-14T22:02:52.408Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-54b36ccac73b662d.gz
252023-11-14T22:02:54.704Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-54b36ccac73b662d.gz
262023-11-14T22:02:54.738Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-14T22:03:24.431Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
282023-11-14T22:03:25.471Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-183be15994d1779e.gz
292023-11-14T22:04:20.658Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-183be15994d1779e.gz
302023-11-14T22:04:20.690Zdownloading input: /input/ci-tools/work/caboose-util.gz
312023-11-14T22:04:24.549Zdownloaded input: /input/ci-tools/work/caboose-util.gz
322023-11-14T22:04:24.582Zdownloading input: /input/ci-tools/work/tufaceous.gz
332023-11-14T22:04:33.196Zdownloaded input: /input/ci-tools/work/tufaceous.gz
342023-11-14T22:04:33.238Zdownloading input: /input/package/work/version.txt
352023-11-14T22:04:33.280Zdownloaded input: /input/package/work/version.txt
362023-11-14T22:04:33.316Zdownloading input: /input/package/work/package.tar.gz
372023-11-14T22:05:24.070Zdownloaded input: /input/package/work/package.tar.gz
382023-11-14T22:05:26.659Zdownloading input: /input/package/work/global-zone-packages.tar.gz
392023-11-14T22:06:13.370Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
402023-11-14T22:06:13.401Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-14T22:06:23.400Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
422023-11-14T22:06:23.434Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
432023-11-14T22:07:16.093Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
442023-11-14T22:07:16.125Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-14T22:07:56.656Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
462023-11-14T22:07:56.691Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-14T22:08:28.228Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
482023-11-14T22:08:28.254Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-14T22:08:35.069Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
502023-11-14T22:08:35.097Zdownloading input: /input/package/work/zones/crucible.tar.gz
512023-11-14T22:08:49.632Zdownloaded input: /input/package/work/zones/crucible.tar.gz
522023-11-14T22:08:49.690Zdownloading input: /input/package/work/zones/external-dns.tar.gz
532023-11-14T22:08:57.284Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
542023-11-14T22:08:57.309Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
552023-11-14T22:09:03.640Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
562023-11-14T22:09:04.080Zdownloading input: /input/package/work/zones/ntp.tar.gz
572023-11-14T22:09:04.112Zdownloaded input: /input/package/work/zones/ntp.tar.gz
582023-11-14T22:09:04.145Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-14T22:09:12.428Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
602023-11-14T22:09:12.453Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-14T22:09:16.973Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
622023-11-14T22:09:17.005Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-14T22:09:33.321Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
642023-11-14T22:09:33.348Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-14T22:09:55.519Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
662023-11-14T22:09:55.571Zdownloading input: /input/package/work/zones/overlay.tar.gz
672023-11-14T22:09:55.615Zdownloaded input: /input/package/work/zones/overlay.tar.gz
682023-11-14T22:09:55.652Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-14T22:10:04.737Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
702023-11-14T22:10:05.713Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
712023-11-14T22:10:45.109Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
722023-11-14T22:10:45.141Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
732023-11-14T22:11:35.267Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
742023-11-14T22:11:35.292Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
752023-11-14T22:12:25.808Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
762023-11-14T22:12:25.846Zstarting task 0: "setup"
772023-11-14T22:12:25.888Z++ uname -s
782023-11-14T22:12:25.920Z+ kern=SunOS
792023-11-14T22:12:25.951Z+ case "$kern" in
802023-11-14T22:12:25.983Z+ groupadd -g 12345 build
812023-11-14T22:12:26.015Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
822023-11-14T22:12:27.897Z+ zfs create -o mountpoint=/work rpool/work
832023-11-14T22:12:27.929Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
842023-11-14T22:12:27.960Z+ home_fs=zfs
852023-11-14T22:12:27.992Z+ [[ zfs == autofs ]]
862023-11-14T22:12:28.030Z+ mkdir -p /home/build
872023-11-14T22:12:28.062Z+ chown build:build /home/build /work
882023-11-14T22:12:29.927Z+ chmod 0700 /home/build /work
892023-11-14T22:12:29.959Zprocess exited: duration 4052 ms, exit code 0
 
902023-11-14T22:12:30.022Zstarting task 1: "authentication"
912023-11-14T22:12:30.106Zprocess exited: duration 52 ms, exit code 0
 
922023-11-14T22:12:30.172Zstarting task 2: "build"
932023-11-14T22:12:30.211Z+ trap _exit_trap EXIT
942023-11-14T22:12:30.243Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
952023-11-14T22:12:30.412Zsecpolicy_fs_linkdir: 0x55 = 0x55
962023-11-14T22:12:30.460Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
972023-11-14T22:12:30.493Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
982023-11-14T22:12:30.529Zsecpolicy_fs_linkdir+3: 0 = 0xe5
992023-11-14T22:12:30.560Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1002023-11-14T22:12:30.588Zsecpolicy_fs_linkdir+5: 0 = 0x1
1012023-11-14T22:12:30.612Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1022023-11-14T22:12:30.637Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1032023-11-14T22:12:30.662Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1042023-11-14T22:12:30.686Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1052023-11-14T22:12:30.710Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1062023-11-14T22:12:30.734Z+ [[ -d /opt/oxide ]]
1072023-11-14T22:12:30.759Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1082023-11-14T22:12:30.783Z+ pfexec mkdir /opt/oxide
1092023-11-14T22:12:30.807Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1102023-11-14T22:12:30.834Z+ [[ -d /opt/oxide-underneath ]]
1112023-11-14T22:12:30.862Z+ cd /opt/oxide-underneath
1122023-11-14T22:12:30.886Z+ pfexec tar ceEp@/f - .
1132023-11-14T22:12:30.911Z+ cd /opt/oxide
1142023-11-14T22:12:30.940Z+ pfexec tar xveEp@/f -
1152023-11-14T22:12:31.404Zx ., 0 bytes, 0 tape blocks
1162023-11-14T22:12:31.434Zx ./opte, 0 bytes, 0 tape blocks
1172023-11-14T22:12:31.463Zx ./opte/bin, 0 bytes, 0 tape blocks
1182023-11-14T22:12:31.490Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1192023-11-14T22:12:31.518Z+ pfexec rm -rf /opt/oxide-underneath
1202023-11-14T22:12:31.550Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1212023-11-14T22:12:31.575Z+ pfexec mkdir /opt/oxide/work
1222023-11-14T22:12:31.599Z+ pfexec chown build:build /opt/oxide/work
1232023-11-14T22:12:31.623Z+ cd /opt/oxide/work
1242023-11-14T22:12:31.648Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1252023-11-14T22:12:31.673Ztar: blocksize = 10
1262023-11-14T22:12:32.607ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1272023-11-14T22:12:32.733Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1282023-11-14T22:12:32.758Zx out/omicron-sled-agent.tar, 123741696 bytes, 241683 tape blocks
1292023-11-14T22:12:32.782Zx out/target/test, 75 bytes, 1 tape blocks
1302023-11-14T22:12:32.806Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1312023-11-14T22:12:32.831Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1322023-11-14T22:12:32.855Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1332023-11-14T22:12:32.879Zx target/release/omicron-package, 42425440 bytes, 82863 tape blocks
1342023-11-14T22:12:32.903Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1352023-11-14T22:12:32.928Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1362023-11-14T22:12:32.955Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1372023-11-14T22:12:32.982Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1382023-11-14T22:12:33.008Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1392023-11-14T22:12:33.034Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1402023-11-14T22:12:33.062Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1412023-11-14T22:12:33.086Z
1422023-11-14T22:12:33.110Zreal 1.623928096
1432023-11-14T22:12:33.134Zuser 1.459530632
1442023-11-14T22:12:33.159Zsys 0.305735486
1452023-11-14T22:12:33.183Ztrap 0.000060800
1462023-11-14T22:12:33.211Ztflt 0.000000000
1472023-11-14T22:12:33.237Zdflt 0.000000000
1482023-11-14T22:12:33.265Zkflt 0.000000000
1492023-11-14T22:12:33.291Zlock 0.000000000
1502023-11-14T22:12:33.327Zslp 1.438099037
1512023-11-14T22:12:33.354Zlat 0.042593800
1522023-11-14T22:12:33.378Zstop 0.000120562
1532023-11-14T22:12:33.403Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1542023-11-14T22:12:33.434Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1552023-11-14T22:12:33.462Z+ mkdir tests
1562023-11-14T22:12:33.492Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1572023-11-14T22:12:33.523Z+ ptime -m gunzip
1582023-11-14T22:12:33.549Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-54b36ccac73b662d
1592023-11-14T22:12:33.580Z
1602023-11-14T22:12:33.605Zreal 0.140378158
1612023-11-14T22:12:33.629Zuser 0.127710840
1622023-11-14T22:12:33.654Zsys 0.012438174
1632023-11-14T22:12:33.678Ztrap 0.000011919
1642023-11-14T22:12:33.746Ztflt 0.000000000
1652023-11-14T22:12:33.775Zdflt 0.000000000
1662023-11-14T22:12:33.801Zkflt 0.000000000
1672023-11-14T22:12:33.825Zlock 0.000000000
1682023-11-14T22:12:33.850Zslp 0.000023230
1692023-11-14T22:12:33.874Zlat 0.000015741
1702023-11-14T22:12:33.899Zstop 0.000115371
1712023-11-14T22:12:33.923Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-54b36ccac73b662d
1722023-11-14T22:12:33.947Z+ chmod a+x tests/bootstrap-54b36ccac73b662d
1732023-11-14T22:12:33.972Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1742023-11-14T22:12:34.007Z+ ptime -m gunzip
1752023-11-14T22:12:34.469Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1762023-11-14T22:12:35.834Z
1772023-11-14T22:12:35.862Zreal 2.349399927
1782023-11-14T22:12:35.891Zuser 2.208297163
1792023-11-14T22:12:35.915Zsys 0.140560206
1802023-11-14T22:12:35.942Ztrap 0.000243353
1812023-11-14T22:12:35.967Ztflt 0.000000000
1822023-11-14T22:12:35.995Zdflt 0.000000000
1832023-11-14T22:12:36.027Zkflt 0.000000000
1842023-11-14T22:12:36.053Zlock 0.000000000
1852023-11-14T22:12:36.488Zslp 0.000023970
1862023-11-14T22:12:36.530Zlat 0.000091121
1872023-11-14T22:12:36.556Zstop 0.000123612
1882023-11-14T22:12:36.582Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1892023-11-14T22:12:36.612Z+ chmod a+x tests/bootstrap
1902023-11-14T22:12:36.642Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1912023-11-14T22:12:36.667Z+ ptime -m gunzip
1922023-11-14T22:12:36.691Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-183be15994d1779e
1932023-11-14T22:12:38.482Z
1942023-11-14T22:12:38.520Zreal 2.639996097
1952023-11-14T22:12:38.546Zuser 2.489062083
1962023-11-14T22:12:38.603Zsys 0.150536219
1972023-11-14T22:12:38.628Ztrap 0.000171262
1982023-11-14T22:12:38.653Ztflt 0.000000000
1992023-11-14T22:12:38.678Zdflt 0.000000000
2002023-11-14T22:12:38.703Zkflt 0.000000000
2012023-11-14T22:12:38.749Zlock 0.000000000
2022023-11-14T22:12:38.775Zslp 0.000021920
2032023-11-14T22:12:38.807Zlat 0.000024270
2042023-11-14T22:12:38.846Zstop 0.000118451
2052023-11-14T22:12:38.891Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-183be15994d1779e
2062023-11-14T22:12:38.936Z+ chmod a+x tests/end_to_end_tests-183be15994d1779e
2072023-11-14T22:12:38.962Z+ bmat address ls
2082023-11-14T22:12:39.011ZNAME CIDR FIRST LAST COUNT
2092023-11-14T22:12:39.051Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2102023-11-14T22:12:39.095Z++ bmat address ls -f extra -Ho first,last,count
2112023-11-14T22:12:39.135Z+ set -- 10.151.2.164 10.151.2.227 64
2122023-11-14T22:12:39.171Z+ EXTRA_IP_START=10.151.2.164
2132023-11-14T22:12:39.235Z+ EXTRA_IP_END=10.151.2.227
2142023-11-14T22:12:39.292Z+ EXTRA_IP_COUNT=64
2152023-11-14T22:12:39.352Z+ (( EXTRA_IP_COUNT >= 32 ))
2162023-11-14T22:12:39.434Z+ EXTRA_IP_BASE=10.151.2
2172023-11-14T22:12:49.234Z+ EXTRA_IP_FOCTET=164
2182023-11-14T22:12:49.510Z+ EXTRA_IP_LOCTET=227
2192023-11-14T22:12:49.623Z+ SERVICE_IP_POOL_START=10.151.2.164
2202023-11-14T22:12:49.710Z+ SERVICE_IP_POOL_END=10.151.2.173
2212023-11-14T22:12:49.780Z+ DNS_IP1=10.151.2.164
2222023-11-14T22:12:49.936Z+ DNS_IP2=10.151.2.165
2232023-11-14T22:12:49.969Z+ UPLINK_IP=10.151.2.174
2242023-11-14T22:12:50.004Z+ IPPOOL_START=10.151.2.175
2252023-11-14T22:12:50.041Z+ IPPOOL_END=10.151.2.227
2262023-11-14T22:12:50.082Z++ ipadm show-addr -po type,addr
2272023-11-14T22:12:50.123Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2282023-11-14T22:12:50.148Z+ GATEWAY_IP=10.151.2.100
2292023-11-14T22:12:50.185Z+ [[ -n 10.151.2.100 ]]
2302023-11-14T22:12:50.212Z+ ping -s 10.151.2.100 56 1
2312023-11-14T22:12:50.250ZPING 10.151.2.100: 56 data bytes
2322023-11-14T22:12:50.325Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2332023-11-14T22:12:50.375Z
2342023-11-14T22:12:50.418Z----10.151.2.100 PING Statistics----
2352023-11-14T22:12:50.453Z1 packets transmitted, 1 packets received, 0% packet loss
2362023-11-14T22:12:50.478Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2372023-11-14T22:12:50.504Z++ arp -an
2382023-11-14T22:12:50.534Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2392023-11-14T22:12:50.576Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2402023-11-14T22:12:50.616Z+ routeadm -e ipv4-forwarding -u
2412023-11-14T22:12:50.649Z+ PXA_START=10.151.2.164
2422023-11-14T22:12:50.684Z+ PXA_END=10.151.2.227
2432023-11-14T22:12:50.723Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2442023-11-14T22:12:50.758Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2452023-11-14T22:12:51.217Z+ ZPOOL_VDEV_DIR=/scratch
2462023-11-14T22:12:51.271Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2472023-11-14T22:12:51.299Z+++ dirname ./tools/create_virtual_hardware.sh
2482023-11-14T22:12:51.371Z++ cd ./tools
2492023-11-14T22:12:51.417Z++ pwd
2502023-11-14T22:12:51.458Z+ SOURCE_DIR=/opt/oxide/work/tools
2512023-11-14T22:12:51.489Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2522023-11-14T22:12:51.524Z+ SOFTNPU_MODE=zone
2532023-11-14T22:12:51.549Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2542023-11-14T22:12:51.595Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2552023-11-14T22:12:51.643Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2562023-11-14T22:12:51.691Z++ dladm show-phys -p -o LINK
2572023-11-14T22:12:51.720Z++ head -1
2582023-11-14T22:12:51.747Z+ PHYSICAL_LINK=igb0
2592023-11-14T22:12:51.782Z+ echo 'Using igb0 as physical link'
2602023-11-14T22:12:51.813ZUsing igb0 as physical link
2612023-11-14T22:12:51.845Z+ ensure_run_as_root
2622023-11-14T22:12:51.885Z++ id -u
2632023-11-14T22:12:51.923Z+ [[ 0 -ne 0 ]]
2642023-11-14T22:12:51.953Z+ ensure_zpools
2652023-11-14T22:12:51.978Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2662023-11-14T22:12:52.006Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2672023-11-14T22:12:52.181Z+ readarray -t ZPOOLS
2682023-11-14T22:12:52.208Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2692023-11-14T22:12:52.259Z++ sed 's/[ ",]//g'
2702023-11-14T22:12:52.284Z+ for ZPOOL in "${ZPOOLS[@]}"
2712023-11-14T22:12:52.310ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2722023-11-14T22:12:52.349Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2732023-11-14T22:12:52.404ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2742023-11-14T22:12:52.484Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2752023-11-14T22:12:52.513ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2762023-11-14T22:12:52.538Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2772023-11-14T22:12:52.568ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2782023-11-14T22:12:52.597Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2792023-11-14T22:12:52.631ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2802023-11-14T22:12:52.670Z0+0 records in
2812023-11-14T22:12:52.707ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2822023-11-14T22:12:52.746Z0+0 records out
2832023-11-14T22:12:52.783ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2842023-11-14T22:12:52.813Z0 bytes (0 iB) transferred in 0.000008 secs (0 iB/sec)
2852023-11-14T22:12:52.843ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2862023-11-14T22:12:52.870Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2872023-11-14T22:12:52.930ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2882023-11-14T22:12:52.992Z+ set +x
2892023-11-14T22:12:53.017ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-11-14T22:12:53.048Z++ zpool list -o name
2912023-11-14T22:12:53.074ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2922023-11-14T22:12:53.098Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2932023-11-14T22:12:53.123ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2942023-11-14T22:12:53.148Z+ [[ -z '' ]]
2952023-11-14T22:12:53.172ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2962023-11-14T22:12:53.197Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2972023-11-14T22:12:53.221ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2982023-11-14T22:12:53.247Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2992023-11-14T22:12:54.495ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3002023-11-14T22:12:54.523Z+ set +x
3012023-11-14T22:12:54.554ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3022023-11-14T22:12:54.589Z+ for ZPOOL in "${ZPOOLS[@]}"
3032023-11-14T22:12:54.624ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3042023-11-14T22:12:54.649Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3052023-11-14T22:12:54.675ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3062023-11-14T22:12:54.699Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3072023-11-14T22:12:54.724ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3082023-11-14T22:12:54.748Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3092023-11-14T22:12:54.773ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3102023-11-14T22:12:54.798Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3112023-11-14T22:12:54.826ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3122023-11-14T22:12:54.854Z0+0 records in
3132023-11-14T22:12:54.882ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3142023-11-14T22:12:54.910Z0+0 records out
3152023-11-14T22:12:54.939ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3162023-11-14T22:12:54.967Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3172023-11-14T22:12:54.991ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3182023-11-14T22:12:55.018Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3192023-11-14T22:12:55.048ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3202023-11-14T22:12:55.076Z+ set +x
3212023-11-14T22:12:55.101ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3222023-11-14T22:12:55.126Z++ zpool list -o name
3232023-11-14T22:12:55.151ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3242023-11-14T22:12:55.177Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3252023-11-14T22:12:55.201ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3262023-11-14T22:12:55.229Z+ [[ -z '' ]]
3272023-11-14T22:12:55.254ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3282023-11-14T22:12:55.282Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-11-14T22:12:55.309ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3302023-11-14T22:12:55.337Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3312023-11-14T22:12:55.363ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3322023-11-14T22:12:55.390Z+ set +x
3332023-11-14T22:12:55.420ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3342023-11-14T22:12:55.447Z+ for ZPOOL in "${ZPOOLS[@]}"
3352023-11-14T22:12:55.520ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3362023-11-14T22:12:55.547Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3372023-11-14T22:12:55.572ZSimnet net0/sc0_0 exists
3382023-11-14T22:12:55.596Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3392023-11-14T22:12:55.620ZSimnet net1/sc1_0 exists
3402023-11-14T22:12:55.644Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3412023-11-14T22:12:55.670ZVnic sc0_1 exists
3422023-11-14T22:12:55.694Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3432023-11-14T22:12:55.718Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
3442023-11-14T22:12:55.743Z0+0 records in
3452023-11-14T22:12:55.767Z0+0 records out
3462023-11-14T22:12:55.791Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3472023-11-14T22:12:55.815Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3482023-11-14T22:12:55.839Z+ set +x
3492023-11-14T22:12:55.864Z++ zpool list -o name
3502023-11-14T22:12:55.890Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3512023-11-14T22:12:55.928Z+ [[ -z '' ]]
3522023-11-14T22:12:55.957Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3532023-11-14T22:12:55.982Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3542023-11-14T22:12:56.006Z+ set +x
3552023-11-14T22:12:56.047Z+ for ZPOOL in "${ZPOOLS[@]}"
3562023-11-14T22:12:56.072Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3572023-11-14T22:12:56.096Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3582023-11-14T22:12:56.121Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3592023-11-14T22:12:56.145Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3602023-11-14T22:12:56.170Z0+0 records in
3612023-11-14T22:12:56.195Z0+0 records out
3622023-11-14T22:12:56.219Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3632023-11-14T22:12:56.244Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3642023-11-14T22:12:56.268Z+ set +x
3652023-11-14T22:12:56.298Z++ zpool list -o name
3662023-11-14T22:12:56.327Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3672023-11-14T22:12:56.351Z+ [[ -z '' ]]
3682023-11-14T22:12:56.375Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3692023-11-14T22:12:56.399Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3702023-11-14T22:12:56.426Z+ set +x
3712023-11-14T22:12:56.451Z+ for ZPOOL in "${ZPOOLS[@]}"
3722023-11-14T22:12:56.481Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3732023-11-14T22:12:56.509Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3742023-11-14T22:12:56.555Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3752023-11-14T22:12:56.580Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3762023-11-14T22:12:56.604Z0+0 records in
3772023-11-14T22:12:56.628Z0+0 records out
3782023-11-14T22:12:56.653Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3792023-11-14T22:12:56.677Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3802023-11-14T22:12:56.702Z+ set +x
3812023-11-14T22:12:56.727Z++ zpool list -o name
3822023-11-14T22:12:56.752Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3832023-11-14T22:12:56.777Z+ [[ -z '' ]]
3842023-11-14T22:12:56.801Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3852023-11-14T22:12:56.826Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3862023-11-14T22:12:56.850Z+ set +x
3872023-11-14T22:12:56.874Z+ for ZPOOL in "${ZPOOLS[@]}"
3882023-11-14T22:12:56.901Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3892023-11-14T22:12:57.357Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3902023-11-14T22:12:57.381Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3912023-11-14T22:12:57.406Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3922023-11-14T22:12:57.980Z0+0 records in
3932023-11-14T22:12:58.004Z0+0 records out
3942023-11-14T22:12:58.029Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3952023-11-14T22:12:58.062Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3962023-11-14T22:12:58.089Z+ set +x
3972023-11-14T22:12:58.113Z++ zpool list -o name
3982023-11-14T22:12:58.137Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3992023-11-14T22:12:58.162Z+ [[ -z '' ]]
4002023-11-14T22:12:58.186Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
4012023-11-14T22:12:58.210Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4022023-11-14T22:12:58.235Z+ set +x
4032023-11-14T22:12:58.259Z+ for ZPOOL in "${ZPOOLS[@]}"
4042023-11-14T22:12:58.283Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4052023-11-14T22:12:58.310Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4062023-11-14T22:12:58.334Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4072023-11-14T22:12:58.358Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4082023-11-14T22:12:58.383Z0+0 records in
4092023-11-14T22:12:58.407Z0+0 records out
4102023-11-14T22:12:58.431Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4112023-11-14T22:12:58.455Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4122023-11-14T22:12:58.479Z+ set +x
4132023-11-14T22:12:58.509Z++ zpool list -o name
4142023-11-14T22:12:58.535Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4152023-11-14T22:12:58.559Z+ [[ -z '' ]]
4162023-11-14T22:12:58.994Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4172023-11-14T22:12:59.025Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4182023-11-14T22:12:59.050Z+ set +x
4192023-11-14T22:12:59.074Z+ for ZPOOL in "${ZPOOLS[@]}"
4202023-11-14T22:12:59.105Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4212023-11-14T22:12:59.136Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4222023-11-14T22:12:59.161Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4232023-11-14T22:12:59.187Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4242023-11-14T22:12:59.211Z0+0 records in
4252023-11-14T22:12:59.236Z0+0 records out
4262023-11-14T22:12:59.261Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4272023-11-14T22:12:59.285Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4282023-11-14T22:12:59.310Z+ set +x
4292023-11-14T22:12:59.744Z++ zpool list -o name
4302023-11-14T22:12:59.794Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4312023-11-14T22:12:59.819Z+ [[ -z '' ]]
4322023-11-14T22:12:59.844Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4332023-11-14T22:12:59.877Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4342023-11-14T22:12:59.904Z+ set +x
4352023-11-14T22:12:59.933Z+ for ZPOOL in "${ZPOOLS[@]}"
4362023-11-14T22:12:59.965Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4372023-11-14T22:13:00.000Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4382023-11-14T22:13:00.040Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4392023-11-14T22:13:00.065Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4402023-11-14T22:13:00.090Z0+0 records in
4412023-11-14T22:13:00.114Z0+0 records out
4422023-11-14T22:13:00.141Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4432023-11-14T22:13:00.167Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4442023-11-14T22:13:00.198Z+ set +x
4452023-11-14T22:13:00.227Z++ zpool list -o name
4462023-11-14T22:13:01.185Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4472023-11-14T22:13:01.220Z+ [[ -z '' ]]
4482023-11-14T22:13:01.248Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4492023-11-14T22:13:01.273Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4502023-11-14T22:13:01.297Z+ set +x
4512023-11-14T22:13:01.322Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4522023-11-14T22:13:01.347Z+ readarray -t ZPOOLS
4532023-11-14T22:13:01.371Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4542023-11-14T22:13:01.396Z++ sed 's/[ ",]//g'
4552023-11-14T22:13:01.421Z+ for ZPOOL in "${ZPOOLS[@]}"
4562023-11-14T22:13:01.445Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4572023-11-14T22:13:01.470Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4582023-11-14T22:13:01.495Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4592023-11-14T22:13:01.519Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4602023-11-14T22:13:01.543Z0+0 records in
4612023-11-14T22:13:01.568Z0+0 records out
4622023-11-14T22:13:01.594Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4632023-11-14T22:13:01.620Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4642023-11-14T22:13:01.653Z+ set +x
4652023-11-14T22:13:01.680Z++ zpool list -o name
4662023-11-14T22:13:01.707Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4672023-11-14T22:13:01.735Z+ [[ -z '' ]]
4682023-11-14T22:13:01.762Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4692023-11-14T22:13:01.787Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4702023-11-14T22:13:01.812Z+ set +x
4712023-11-14T22:13:01.837Z+ for ZPOOL in "${ZPOOLS[@]}"
4722023-11-14T22:13:01.864Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4732023-11-14T22:13:01.888Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4742023-11-14T22:13:01.913Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4752023-11-14T22:13:01.938Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4762023-11-14T22:13:01.963Z0+0 records in
4772023-11-14T22:13:01.989Z0+0 records out
4782023-11-14T22:13:02.014Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4792023-11-14T22:13:02.040Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4802023-11-14T22:13:02.066Z+ set +x
4812023-11-14T22:13:02.098Z++ zpool list -o name
4822023-11-14T22:13:02.125Zdownloading softnpu ...done
4832023-11-14T22:13:02.155Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4842023-11-14T22:13:02.182Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
4852023-11-14T22:13:02.216Z+ [[ -z '' ]]
4862023-11-14T22:13:02.242Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4872023-11-14T22:13:02.266Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4882023-11-14T22:13:02.291Z+ set +x
4892023-11-14T22:13:02.316Z+ [[ zone == \z\o\n\e ]]
4902023-11-14T22:13:02.341Z+ ensure_simulated_links igb0
4912023-11-14T22:13:02.366Z+ local PHYSICAL_LINK=igb0
4922023-11-14T22:13:02.391Z+ INDICES=("0" "1")
4932023-11-14T22:13:02.416Z+ for I in "${INDICES[@]}"
4942023-11-14T22:13:02.441Z++ get_simnet_name_if_exists net0
4952023-11-14T22:13:02.467Z++ dladm show-simnet -p -o LINK net0
4962023-11-14T22:13:02.496Z++ echo ''
4972023-11-14T22:13:02.527Z+ [[ -z '' ]]
4982023-11-14T22:13:02.612Z+ dladm create-simnet -t net0
4992023-11-14T22:13:02.642Z+ dladm create-simnet -t sc0_0
5002023-11-14T22:13:02.674Z+ dladm modify-simnet -t -p net0 sc0_0
5012023-11-14T22:13:02.706Z+ dladm set-linkprop -p mtu=9000 sc0_0
5022023-11-14T22:13:02.736Z+ success 'Simnet net0/sc0_0 exists'
5032023-11-14T22:13:02.768Z+ set +x
5042023-11-14T22:13:02.805Z+ for I in "${INDICES[@]}"
5052023-11-14T22:13:02.832Z++ get_simnet_name_if_exists net1
5062023-11-14T22:13:02.865Z++ dladm show-simnet -p -o LINK net1
5072023-11-14T22:13:02.893Z++ echo ''
5082023-11-14T22:13:02.925Z+ [[ -z '' ]]
5092023-11-14T22:13:02.952Z+ dladm create-simnet -t net1
5102023-11-14T22:13:02.977Z+ dladm create-simnet -t sc1_0
5112023-11-14T22:13:03.003Z+ dladm modify-simnet -t -p net1 sc1_0
5122023-11-14T22:13:03.028Z+ dladm set-linkprop -p mtu=9000 sc1_0
5132023-11-14T22:13:03.053Z+ success 'Simnet net1/sc1_0 exists'
5142023-11-14T22:13:03.081Z+ set +x
5152023-11-14T22:13:03.107Z++ get_vnic_name_if_exists sc0_1
5162023-11-14T22:13:03.136Z++ dladm show-vnic -p -o LINK sc0_1
5172023-11-14T22:13:03.166Z++ echo ''
5182023-11-14T22:13:03.604Z+ [[ -z '' ]]
5192023-11-14T22:13:03.630Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5202023-11-14T22:13:03.674Z+ [[ -v PROMISC_FILT_OFF ]]
5212023-11-14T22:13:03.700Z+ success 'Vnic sc0_1 exists'
5222023-11-14T22:13:03.725Z+ set +x
5232023-11-14T22:13:03.750Z+ warn_if_no_proxy_arp
5242023-11-14T22:13:03.795Z+ [[ -v PXA_START ]]
5252023-11-14T22:13:03.825Z+ [[ -v PXA_END ]]
5262023-11-14T22:13:03.850Z+ ensure_softnpu_zone
5272023-11-14T22:13:03.876Z+ zoneadm list
5282023-11-14T22:13:03.901Z+ grep -q sidecar_softnpu
5292023-11-14T22:13:03.927Z+ [[ -f out/npuzone/npuzone ]]
5302023-11-14T22:13:03.953Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5312023-11-14T22:13:05.689Zdownloading asic_program.so ...done
5322023-11-14T22:13:05.717Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5332023-11-14T22:13:15.672Zdownloading scadm ...done
5342023-11-14T22:13:15.699Zcreating port tfportrear0_0 for sc0_0
5352023-11-14T22:13:15.727Zcreating port tfportqsfp0_0 for sc0_1
5362023-11-14T22:13:15.760Zsoftnpu zone setup
5372023-11-14T22:13:15.793Zconfigure zone
5382023-11-14T22:13:15.844Zinstall zone
5392023-11-14T22:13:16.752Zboot zone
5402023-11-14T22:13:19.864Z[sidecar_softnpu] svcs milestone/network
5412023-11-14T22:13:19.895ZSTATE STIME FMRI
5422023-11-14T22:13:19.921Zonline 22:13:17 svc:/milestone/network:default
5432023-11-14T22:13:20.369Z[sidecar_softnpu] /softnpu/npu start
5442023-11-14T22:13:20.394Z
5452023-11-14T22:13:20.419Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5462023-11-14T22:13:20.444Z+ GATEWAY_IP=10.151.2.100
5472023-11-14T22:13:20.469Z+ echo 'Using 10.151.2.100 as gateway ip'
5482023-11-14T22:13:20.495ZUsing 10.151.2.100 as gateway ip
5492023-11-14T22:13:20.521Z+ [[ ! -v GATEWAY_MAC ]]
5502023-11-14T22:13:20.549Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5512023-11-14T22:13:20.584Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5522023-11-14T22:13:20.613Z+++ echo 18:c0:4d:0d:a0:2a
5532023-11-14T22:13:20.638Z+ COUNT=1
5542023-11-14T22:13:20.663Z+ [[ 1 -ne 1 ]]
5552023-11-14T22:13:20.688Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5562023-11-14T22:13:20.730ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5572023-11-14T22:13:20.757Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5582023-11-14T22:13:20.784Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5592023-11-14T22:13:21.222Z+ PXA_MAC=a8:e1:de:01:70:1d
5602023-11-14T22:13:21.279Z+ [[ -v PXA_START ]]
5612023-11-14T22:13:21.309Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5622023-11-14T22:13:21.335Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5632023-11-14T22:13:21.360Z+ z_scadm dump-state
5642023-11-14T22:13:21.385Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5652023-11-14T22:13:21.409Zlocal v6:
5662023-11-14T22:13:21.434Zlocal v4:
5672023-11-14T22:13:21.459Zrouter v6:
5682023-11-14T22:13:21.485Zrouter v4_idx:
5692023-11-14T22:13:21.510Zrouter v4_routes:
5702023-11-14T22:13:21.538Zresolver v4:
5712023-11-14T22:13:21.563Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5722023-11-14T22:13:21.587Zresolver v6:
5732023-11-14T22:13:21.614Znat v4:
5742023-11-14T22:13:21.639Znat_v6:
5752023-11-14T22:13:22.084Zport_mac:
5762023-11-14T22:13:22.110Zproxy arp:
5772023-11-14T22:13:22.134Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5782023-11-14T22:13:22.158Z+ success 'softnpu zone exists'
5792023-11-14T22:13:22.183Z+ set +x
5802023-11-14T22:13:22.208Zsoftnpu zone exists
5812023-11-14T22:13:22.232Z
5822023-11-14T22:13:22.257Zreal 41.192043069
5832023-11-14T22:13:22.282Zuser 0.720609950
5842023-11-14T22:13:22.329Zsys 1.585171461
5852023-11-14T22:13:22.382Ztrap 0.000079876
5862023-11-14T22:13:22.406Ztflt 0.000116537
5872023-11-14T22:13:22.431Zdflt 0.001275025
5882023-11-14T22:13:22.456Zkflt 0.000000000
5892023-11-14T22:13:22.491Zlock 32:07.297478803
5902023-11-14T22:13:22.582Zslp 2:03.503889177
5912023-11-14T22:13:22.621Zlat 0.059750179
5922023-11-14T22:13:22.685Zstop 0.356259309
5932023-11-14T22:13:22.712Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5942023-11-14T22:13:22.740Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-14T22:13:22.766Z+ SILO_NAME=recovery
5962023-11-14T22:13:22.806Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5972023-11-14T22:13:22.832Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5982023-11-14T22:13:22.859Z+ sed -i~ '
5992023-11-14T22:13:22.884Z /^external_dns_ips/c\
6002023-11-14T22:13:22.911Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6012023-11-14T22:13:22.937Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6022023-11-14T22:13:22.963Z /^first/c\
6032023-11-14T22:13:22.992Zfirst = "10.151.2.164"
6042023-11-14T22:13:23.018Z /^last/c\
6052023-11-14T22:13:23.045Zlast = "10.151.2.173"
6062023-11-14T22:13:23.069Z }
6072023-11-14T22:13:23.094Z /^\[rack_network_config/,/^$/ {
6082023-11-14T22:13:23.119Z /^infra_ip_first/c\
6092023-11-14T22:13:23.144Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6102023-11-14T22:13:23.169Zinfra_ip_first = "10.151.2.174"
6112023-11-14T22:13:23.194Z+++ pkg/config-rss.toml Tue Nov 14 22:13:19 2023
6122023-11-14T22:13:23.219Z /^infra_ip_last/c\
6132023-11-14T22:13:23.244Z@@ -41,7 +41,7 @@
6142023-11-14T22:13:23.268Zinfra_ip_last = "10.151.2.174"
6152023-11-14T22:13:23.293Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6162023-11-14T22:13:23.318Z }
6172023-11-14T22:13:23.364Z # must be contained in one of the "internal services" IP Pool ranges listed
6182023-11-14T22:13:23.391Z /^\[\[rack_network_config.ports/,/^$/ {
6192023-11-14T22:13:23.424Z # below.
6202023-11-14T22:13:23.449Z /^routes/c\
6212023-11-14T22:13:23.478Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6222023-11-14T22:13:23.503Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6232023-11-14T22:13:23.528Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6242023-11-14T22:13:23.554Z /^addresses/c\
6252023-11-14T22:13:23.579Z
6262023-11-14T22:13:23.605Zaddresses = \["10.151.2.174/32"\]
6272023-11-14T22:13:23.631Z # Initial TLS certificates for the external API
6282023-11-14T22:13:24.062Z }
6292023-11-14T22:13:24.087Z #
6302023-11-14T22:13:24.522Z' pkg/config-rss.toml
6312023-11-14T22:13:24.550Z@@ -77,8 +77,8 @@
6322023-11-14T22:13:24.575Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6332023-11-14T22:13:24.600Z #
6342023-11-14T22:13:24.625Z+ true
6352023-11-14T22:13:24.650Z # For more on this and what to put here, see docs/how-to-run.adoc.
6362023-11-14T22:13:24.682Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6372023-11-14T22:13:25.113Z [[internal_services_ip_pool_ranges]]
6382023-11-14T22:13:25.141Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6392023-11-14T22:13:25.167Z-first = "192.168.1.20"
6402023-11-14T22:13:25.195Z+ TLS_NAME=recovery.sys.oxide.test
6412023-11-14T22:13:25.220Z-last = "192.168.1.29"
6422023-11-14T22:13:25.245Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6432023-11-14T22:13:25.270Z+first = "10.151.2.164"
6442023-11-14T22:13:25.295Z......................+..........+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+..+......+......+.........+....+........+.......+.....+.......+......+...+..+.........+.+...+............+...+..+....+.....+...+.........................+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+.........+.......+........+...+....+.....+....+.........+..+......+....+...+..+......+....+..................+............+........+..........+...........+...............+............+.......+......+.................+...+.+.....+...............+.+............+...........+............+...+.......+.....+.............+..+...+....+.....+.....................+......+....+...........+.+.........+...+....................+...+.......+........+...+.+........+.+.....+.+.........+............+.....+....+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6452023-11-14T22:13:25.320Z+last = "10.151.2.173"
6462023-11-14T22:13:25.346Z
6472023-11-14T22:13:25.371Z # TODO - this configuration is subject to change going forward. Ultimately these
6482023-11-14T22:13:25.396Z # parameters should be provided to the control plane via wicket, but we need to
6492023-11-14T22:13:25.420Z@@ -92,8 +92,8 @@
6502023-11-14T22:13:25.517Z # A range of IP addresses used by Boundary Services on the external network. In
6512023-11-14T22:13:25.545Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6522023-11-14T22:13:25.571Z.+...+..+......+...+.+...+..+....+..+.......+...+............+............+......+........+.+...............+......+..+.+..+...+....+.....+......+.+.....+.........+.........+...+.......+..+......+.......+........+...+.+.....+.+...+.....+......+...+.......+...+.....+...+...+.+...+......+.....+...................+..+.........+.......+.........+......+...+.....+.+........+...+.........+...+......+.+.........+.........+..+....+..+....+.....+......+...+.+...+...+............+.....+.+.....+.+.....+.+.....+.......+..+...+......+....+......+.....+....+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+......+...+.........+.+......+...+...+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+.........+..+.......+...+..................+..............+.......+..................+..+......+.......+.................+..................+...+...........................+......+....+.....+.......+...+...+........+................+.....+....+.........+...+........+.......+........+................+........+.......+......+..............+.+..+...............+.+..............+.+............+........+................+...+...+.....+..................+......+.+..+...............+......+.+...+.....+...+...+..........+...............+.........+..+......+....+.....+.........+.......+..+.......+..................+............+.....+..........+......+.....+...+.........+..........+..+..................+.+...............+..+....+.......................+......+.+........+......+.+........+......+..........+.........+...+...+.........+.....+..........+.....+.............+...+...........+...+..........+............+.....+.........+....+...+...+.........+........+...+....+............+.....+.+........+.+......+.................+...+...+....+...........+.......+.....+....+........+.......+........+.+..+....+.....+.........+......+..........+......+..+......+.......+..................+..+.......+........+...+.+......+...+...............+....................+.......+..............+.........+....+.....+...+....+........+.+..+..........+........+....+.................+...+..................+....+...+..+...............+....+.........+..+......+...+.......+.................+.+.....+......+...+......+.........+.......+......+.....+............+.......+........+...+...+....+...............+......................................+....+.....+....+............+...+..+............+..................+....+........+.............+.....+...............+..........+........+.......+...+...........+................+...+......+......+..............+......+.......+.....+......+.......+........+....+.....................+.........+...............+...+..............+.........+....+...........+.+.....+.........+.........+.......+...+........+............+...+...........................+..........+...+........+...............+......+.+..............+...+......+....+...........+....+...+...+.....+..........+...........+.........+..........+..+...............+..................+.+...+...+.........+.........+.....+......+.............+...+..............................+..+.......+........+.......+...+...+..+.......+....................+.+...+.....+............+...........................................+..............+.+.....+....+..+...+.+.....+............+...............+....+...+..+....+.........+.....+...+....+.........+......+........+......+.+.........+......+........+...+...............+...+.+..................+..+..........+...+..+...................+............+....................+.........+.+........+.+..............+........................+...+....+......+.....+....+.....................+..+.+.....+....+..+...+.......+.....+....+.........+......+...........+.............+..................+..+.....................+.+...+..+......+..........+........+.......+...+..+............+.+..............+...+............................+..+...+....+.....................+..............+.......+........+................+..+...+..............................+......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6532023-11-14T22:13:25.601Z # With softnpu, only one address is used.
6542023-11-14T22:13:25.627Z-----
6552023-11-14T22:13:25.653Z-infra_ip_first = "192.168.1.30"
6562023-11-14T22:13:25.683Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6572023-11-14T22:13:25.715Z-infra_ip_last = "192.168.1.30"
6582023-11-14T22:13:25.742Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6592023-11-14T22:13:25.767Z+infra_ip_first = "10.151.2.174"
6602023-11-14T22:13:25.791Z+ rmdir pkg
6612023-11-14T22:13:25.817Z+infra_ip_last = "10.151.2.174"
6622023-11-14T22:13:25.842Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6632023-11-14T22:13:25.866Z
6642023-11-14T22:13:25.891Z+ pfexec mkdir -p /usr/oxide
6652023-11-14T22:13:25.916Z # Configurations for BGP routers to run on the scrimlets.
6662023-11-14T22:13:25.944Z+ pfexec rm -f /usr/oxide/tcpproxy
6672023-11-14T22:13:25.968Z bgp = []
6682023-11-14T22:13:25.996Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6692023-11-14T22:13:26.023Z@@ -101,9 +101,9 @@
6702023-11-14T22:13:26.053Z+ pfexec chmod +x /usr/oxide/tcpproxy
6712023-11-14T22:13:26.081Z # You can configure multiple uplinks by repeating the following stanza
6722023-11-14T22:13:26.105Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6732023-11-14T22:13:26.142Z [[rack_network_config.ports]]
6742023-11-14T22:13:26.173Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6752023-11-14T22:13:26.198Z # Routes associated with this port.
6762023-11-14T22:13:26.223Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6772023-11-14T22:13:26.247Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6782023-11-14T22:13:26.272Z+ OMICRON_NO_UNINSTALL=1
6792023-11-14T22:13:26.296Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6802023-11-14T22:13:26.321Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6812023-11-14T22:13:26.346Z # Addresses associated with this port.
6822023-11-14T22:13:26.371ZNov 14 22:13:22.583 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6832023-11-14T22:13:26.400Z-addresses = ["192.168.1.30/32"]
6842023-11-14T22:13:26.428ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6852023-11-14T22:13:26.453Z+addresses = ["10.151.2.174/32"]
6862023-11-14T22:13:26.891ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6872023-11-14T22:13:26.916Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6882023-11-14T22:13:26.941ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6892023-11-14T22:13:26.965Z port = "qsfp0"
6902023-11-14T22:13:26.990ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6912023-11-14T22:13:27.017Z # The speed of this port.
6922023-11-14T22:13:27.042ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6932023-11-14T22:13:27.067Za pkg/config-rss.toml 7K
6942023-11-14T22:13:27.092ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6952023-11-14T22:13:27.178Za pkg/initial-tls-cert.pem 2K
6962023-11-14T22:13:27.207ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6972023-11-14T22:13:27.232Za pkg/initial-tls-key.pem 4K
6982023-11-14T22:13:27.256ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6992023-11-14T22:13:27.298ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7002023-11-14T22:13:27.322ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7012023-11-14T22:13:27.346ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7022023-11-14T22:13:27.396ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7032023-11-14T22:13:27.420ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7042023-11-14T22:13:27.445ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7052023-11-14T22:13:27.469ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7062023-11-14T22:13:27.493ZNov 14 22:13:22.585 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7072023-11-14T22:13:27.517ZNov 14 22:13:22.810 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7082023-11-14T22:13:27.546ZNov 14 22:13:22.843 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7092023-11-14T22:13:27.607ZNov 14 22:13:22.907 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7102023-11-14T22:13:27.633Z
7112023-11-14T22:13:27.658Zreal 0.462491053
7122023-11-14T22:13:27.683Zuser 0.098653471
7132023-11-14T22:13:27.708Zsys 1.863000929
7142023-11-14T22:13:27.733Ztrap 0.000002199
7152023-11-14T22:13:27.758Ztflt 0.000051139
7162023-11-14T22:13:27.783Zdflt 0.000996239
7172023-11-14T22:13:27.808Zkflt 0.000000000
7182023-11-14T22:13:27.833Zlock 41.519496778
7192023-11-14T22:13:27.858Zslp 0.668403898
7202023-11-14T22:13:27.884Zlat 0.002910349
7212023-11-14T22:13:27.909Zstop 0.209012549
7222023-11-14T22:13:27.934Z+ retry=0
7232023-11-14T22:13:27.959Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7242023-11-14T22:13:27.987Z+ [[ 0 -gt 30 ]]
7252023-11-14T22:13:28.016Z+ sleep 1
7262023-11-14T22:13:28.043Z+ retry=1
7272023-11-14T22:13:28.069Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7282023-11-14T22:13:28.100Z+ [[ 1 -gt 30 ]]
7292023-11-14T22:13:28.141Z+ sleep 1
7302023-11-14T22:13:28.168Z+ retry=2
7312023-11-14T22:13:28.193Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7322023-11-14T22:14:03.447Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7332023-11-14T22:14:03.511Zlocal v6:
7342023-11-14T22:14:03.539Zfe80::9c91:afff:fe38:fe87
7352023-11-14T22:14:03.585Zfd00:99::1
7362023-11-14T22:14:03.640Zlocal v4:
7372023-11-14T22:14:03.672Z10.151.2.174
7382023-11-14T22:14:03.736Zrouter v6:
7392023-11-14T22:14:03.774Zfd00:1122:3344:101::/64 -> fe80::c0fa:4aff:fe95:7e39 (1)
7402023-11-14T22:14:03.805Zfdb0:18c0:4d0d:a02a::/64 -> fe80::c0fa:4aff:fe95:7e39 (1)
7412023-11-14T22:14:03.843Zrouter v4_idx:
7422023-11-14T22:14:03.874Z0.0.0.0/0 -> 0
7432023-11-14T22:14:03.901Zrouter v4_routes:
7442023-11-14T22:14:03.955Z0 -> 10.151.2.100 (2)
7452023-11-14T22:14:03.981Zresolver v4:
7462023-11-14T22:14:04.021Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7472023-11-14T22:14:04.088Z10.151.2.174 -> ca:d5:b0:20:a2:d1
7482023-11-14T22:14:04.127Zresolver v6:
7492023-11-14T22:14:04.170Zfe80::c0fa:4aff:fe95:7e39 -> c2:fa:4a:95:7e:39
7502023-11-14T22:14:04.208Zfe80::9c91:afff:fe38:fe87 -> 9e:91:af:38:fe:87
7512023-11-14T22:14:04.668Znat v4:
7522023-11-14T22:14:04.714Znat_v6:
7532023-11-14T22:14:04.750Zport_mac:
7542023-11-14T22:14:04.782Z1000: a8:40:25:52:05:10
7552023-11-14T22:14:04.814Z1: a8:40:25:52:05:11
7562023-11-14T22:14:04.841Zproxy arp:
7572023-11-14T22:14:04.868Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7582023-11-14T22:14:04.896Z+ retry=0
7592023-11-14T22:14:04.926Z+++ zoneadm list -n
7602023-11-14T22:14:04.986Z+++ grep oxz_ntp
7612023-11-14T22:14:05.194Z++ pfexec svcs -z -Hostate oxide/ntp
7622023-11-14T22:14:05.219Zsvcs: zone '-Hostate': entity not found
7632023-11-14T22:14:05.259Z++ true
7642023-11-14T22:14:05.290Z+ [[ '' != online ]]
7652023-11-14T22:14:05.319Z+ [[ 0 -gt 60 ]]
7662023-11-14T22:14:05.355Z+ sleep 1
7672023-11-14T22:14:05.962Z+ retry=1
7682023-11-14T22:14:06.033Z+++ zoneadm list -n
7692023-11-14T22:14:06.068Z+++ grep oxz_ntp
7702023-11-14T22:14:06.097Z++ pfexec svcs -z -Hostate oxide/ntp
7712023-11-14T22:14:06.123Zsvcs: zone '-Hostate': entity not found
7722023-11-14T22:14:06.150Z++ true
7732023-11-14T22:14:06.178Z+ [[ '' != online ]]
7742023-11-14T22:14:06.226Z+ [[ 1 -gt 60 ]]
7752023-11-14T22:14:06.253Z+ sleep 1
7762023-11-14T22:14:06.278Z+ retry=2
7772023-11-14T22:14:06.302Z+++ zoneadm list -n
7782023-11-14T22:14:06.327Z+++ grep oxz_ntp
7792023-11-14T22:14:06.351Z++ pfexec svcs -z -Hostate oxide/ntp
7802023-11-14T22:14:06.375Zsvcs: zone '-Hostate': entity not found
7812023-11-14T22:14:06.400Z++ true
7822023-11-14T22:14:06.433Z+ [[ '' != online ]]
7832023-11-14T22:14:06.463Z+ [[ 2 -gt 60 ]]
7842023-11-14T22:14:06.490Z+ sleep 1
7852023-11-14T22:14:06.515Z+ retry=3
7862023-11-14T22:14:06.540Z+++ zoneadm list -n
7872023-11-14T22:14:06.570Z+++ grep oxz_ntp
7882023-11-14T22:14:06.598Z++ pfexec svcs -z -Hostate oxide/ntp
7892023-11-14T22:14:06.623Zsvcs: zone '-Hostate': entity not found
7902023-11-14T22:14:06.648Z++ true
7912023-11-14T22:14:06.674Z+ [[ '' != online ]]
7922023-11-14T22:14:06.700Z+ [[ 3 -gt 60 ]]
7932023-11-14T22:14:06.729Z+ sleep 1
7942023-11-14T22:14:06.764Z+ retry=4
7952023-11-14T22:14:06.794Z+++ zoneadm list -n
7962023-11-14T22:14:06.826Z+++ grep oxz_ntp
7972023-11-14T22:14:06.864Z++ pfexec svcs -z -Hostate oxide/ntp
7982023-11-14T22:14:06.893Zsvcs: zone '-Hostate': entity not found
7992023-11-14T22:14:06.918Z++ true
8002023-11-14T22:14:06.943Z+ [[ '' != online ]]
8012023-11-14T22:14:06.967Z+ [[ 4 -gt 60 ]]
8022023-11-14T22:14:06.998Z+ sleep 1
8032023-11-14T22:14:07.029Z+ retry=5
8042023-11-14T22:14:07.059Z+++ zoneadm list -n
8052023-11-14T22:14:07.090Z+++ grep oxz_ntp
8062023-11-14T22:14:07.118Z++ pfexec svcs -z -Hostate oxide/ntp
8072023-11-14T22:14:07.149Zsvcs: zone '-Hostate': entity not found
8082023-11-14T22:14:07.175Z++ true
8092023-11-14T22:14:07.200Z+ [[ '' != online ]]
8102023-11-14T22:14:07.224Z+ [[ 5 -gt 60 ]]
8112023-11-14T22:14:07.259Z+ sleep 1
8122023-11-14T22:14:07.288Z+ retry=6
8132023-11-14T22:14:07.322Z+++ zoneadm list -n
8142023-11-14T22:14:07.361Z+++ grep oxz_ntp
8152023-11-14T22:14:07.435Z++ pfexec svcs -z -Hostate oxide/ntp
8162023-11-14T22:14:07.467Zsvcs: zone '-Hostate': entity not found
8172023-11-14T22:14:07.498Z++ true
8182023-11-14T22:14:07.524Z+ [[ '' != online ]]
8192023-11-14T22:14:07.549Z+ [[ 6 -gt 60 ]]
8202023-11-14T22:14:07.605Z+ sleep 1
8212023-11-14T22:14:07.641Z+ retry=7
8222023-11-14T22:14:07.668Z+++ zoneadm list -n
8232023-11-14T22:14:07.694Z+++ grep oxz_ntp
8242023-11-14T22:14:07.719Z++ pfexec svcs -z -Hostate oxide/ntp
8252023-11-14T22:14:07.752Zsvcs: zone '-Hostate': entity not found
8262023-11-14T22:14:07.779Z++ true
8272023-11-14T22:14:07.810Z+ [[ '' != online ]]
8282023-11-14T22:14:07.842Z+ [[ 7 -gt 60 ]]
8292023-11-14T22:14:07.875Z+ sleep 1
8302023-11-14T22:14:07.901Z+ retry=8
8312023-11-14T22:14:07.926Z+++ zoneadm list -n
8322023-11-14T22:14:07.951Z+++ grep oxz_ntp
8332023-11-14T22:14:07.977Z++ pfexec svcs -z -Hostate oxide/ntp
8342023-11-14T22:14:08.121Zsvcs: zone '-Hostate': entity not found
8352023-11-14T22:14:08.242Z++ true
8362023-11-14T22:14:08.366Z+ [[ '' != online ]]
8372023-11-14T22:14:08.478Z+ [[ 8 -gt 60 ]]
8382023-11-14T22:14:08.517Z+ sleep 1
8392023-11-14T22:14:08.571Z+ retry=9
8402023-11-14T22:14:08.618Z+++ zoneadm list -n
8412023-11-14T22:14:08.647Z+++ grep oxz_ntp
8422023-11-14T22:14:08.674Z++ pfexec svcs -z -Hostate oxide/ntp
8432023-11-14T22:14:08.701Zsvcs: zone '-Hostate': entity not found
8442023-11-14T22:14:08.729Z++ true
8452023-11-14T22:14:09.167Z+ [[ '' != online ]]
8462023-11-14T22:14:09.193Z+ [[ 9 -gt 60 ]]
8472023-11-14T22:14:09.217Z+ sleep 1
8482023-11-14T22:14:09.241Z+ retry=10
8492023-11-14T22:14:09.266Z+++ zoneadm list -n
8502023-11-14T22:14:09.291Z+++ grep oxz_ntp
8512023-11-14T22:14:09.320Z++ pfexec svcs -z oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe -Hostate oxide/ntp
8522023-11-14T22:14:09.760Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8532023-11-14T22:14:09.788Z++ true
8542023-11-14T22:14:09.813Z+ [[ '' != online ]]
8552023-11-14T22:14:09.838Z+ [[ 10 -gt 60 ]]
8562023-11-14T22:14:09.869Z+ sleep 1
8572023-11-14T22:14:09.896Z+ retry=11
8582023-11-14T22:14:09.932Z+++ zoneadm list -n
8592023-11-14T22:14:09.974Z+++ grep oxz_ntp
8602023-11-14T22:14:10.007Z++ pfexec svcs -z oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe -Hostate oxide/ntp
8612023-11-14T22:14:10.040Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8622023-11-14T22:14:10.068Z++ true
8632023-11-14T22:14:10.094Z+ [[ '' != online ]]
8642023-11-14T22:14:10.125Z+ [[ 11 -gt 60 ]]
8652023-11-14T22:14:10.154Z+ sleep 1
8662023-11-14T22:14:10.183Z+ retry=12
8672023-11-14T22:14:10.216Z+++ zoneadm list -n
8682023-11-14T22:14:10.244Z+++ grep oxz_ntp
8692023-11-14T22:14:10.288Z++ pfexec svcs -z oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe -Hostate oxide/ntp
8702023-11-14T22:14:10.316Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8712023-11-14T22:14:10.343Z++ true
 
8722023-11-14T22:14:10.686Zsock console login: Nov 14 22:14:09 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8732023-11-14T22:14:10.799Z+ [[ '' != online ]]
8742023-11-14T22:14:10.823Z+ [[ 12 -gt 60 ]]
8752023-11-14T22:14:10.848Z+ sleep 1
8762023-11-14T22:14:10.872Z+ retry=13
8772023-11-14T22:14:10.897Z+++ zoneadm list -n
8782023-11-14T22:14:10.921Z+++ grep oxz_ntp
8792023-11-14T22:14:10.945Z++ pfexec svcs -z oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe -Hostate oxide/ntp
8802023-11-14T22:14:10.970Z+ [[ disabled != online ]]
8812023-11-14T22:14:11.042Z+ [[ 13 -gt 60 ]]
8822023-11-14T22:14:11.071Z+ sleep 1
8832023-11-14T22:14:11.103Z+ retry=14
8842023-11-14T22:14:11.134Z+++ zoneadm list -n
8852023-11-14T22:14:11.167Z+++ grep oxz_ntp
8862023-11-14T22:14:11.197Z++ pfexec svcs -z oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe -Hostate oxide/ntp
8872023-11-14T22:14:11.228Z+ [[ disabled != online ]]
8882023-11-14T22:14:11.259Z+ [[ 14 -gt 60 ]]
8892023-11-14T22:14:11.287Z+ sleep 1
8902023-11-14T22:14:11.312Z+ retry=15
8912023-11-14T22:14:11.336Z+++ zoneadm list -n
8922023-11-14T22:14:11.361Z+++ grep oxz_ntp
8932023-11-14T22:14:11.387Z++ pfexec svcs -z oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe -Hostate oxide/ntp
8942023-11-14T22:14:11.411Z+ [[ disabled != online ]]
8952023-11-14T22:14:11.436Z+ [[ 15 -gt 60 ]]
8962023-11-14T22:14:11.461Z+ sleep 1
8972023-11-14T22:14:11.501Z+ retry=16
8982023-11-14T22:14:11.525Z+++ zoneadm list -n
8992023-11-14T22:14:11.551Z+++ grep oxz_ntp
9002023-11-14T22:14:11.577Z++ pfexec svcs -z oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe -Hostate oxide/ntp
9012023-11-14T22:14:11.603Z+ [[ disabled != online ]]
9022023-11-14T22:14:11.629Z+ [[ 16 -gt 60 ]]
9032023-11-14T22:14:11.654Z+ sleep 1
9042023-11-14T22:14:11.680Z+ retry=17
9052023-11-14T22:14:11.706Z+++ zoneadm list -n
9062023-11-14T22:14:11.732Z+++ grep oxz_ntp
9072023-11-14T22:14:11.757Z++ pfexec svcs -z oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe -Hostate oxide/ntp
9082023-11-14T22:14:11.782Z+ [[ disabled != online ]]
9092023-11-14T22:14:11.807Z+ [[ 17 -gt 60 ]]
9102023-11-14T22:14:11.832Z+ sleep 1
9112023-11-14T22:14:11.861Z+ retry=18
9122023-11-14T22:14:11.891Z+++ zoneadm list -n
9132023-11-14T22:14:11.940Z+++ grep oxz_ntp
9142023-11-14T22:14:11.982Z++ pfexec svcs -z oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe -Hostate oxide/ntp
9152023-11-14T22:14:12.019Z+ [[ disabled != online ]]
9162023-11-14T22:14:12.048Z+ [[ 18 -gt 60 ]]
9172023-11-14T22:14:12.077Z+ sleep 1
9182023-11-14T22:14:12.106Z+ retry=19
9192023-11-14T22:14:12.131Z+++ zoneadm list -n
9202023-11-14T22:14:12.155Z+++ grep oxz_ntp
9212023-11-14T22:14:12.189Z++ pfexec svcs -z oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe -Hostate oxide/ntp
9222023-11-14T22:14:12.216Z+ [[ disabled != online ]]
9232023-11-14T22:14:12.241Z+ [[ 19 -gt 60 ]]
9242023-11-14T22:14:12.266Z+ sleep 1
9252023-11-14T22:14:12.291Z+ retry=20
9262023-11-14T22:14:12.315Z+++ zoneadm list -n
9272023-11-14T22:14:12.389Z+++ grep oxz_ntp
9282023-11-14T22:14:12.421Z++ pfexec svcs -z oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe -Hostate oxide/ntp
9292023-11-14T22:14:12.851Z+ [[ online != online ]]
9302023-11-14T22:14:12.882Z+ echo 'Waited for chrony: 20s'
9312023-11-14T22:14:12.920Z+ export RUST_BACKTRACE=1
9322023-11-14T22:14:12.947ZWaited for chrony: 20s
9332023-11-14T22:14:12.971Z+ RUST_BACKTRACE=1
9342023-11-14T22:14:12.995Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9352023-11-14T22:14:13.020Z+ ./tests/bootstrap
9362023-11-14T22:14:13.048ZUsing external DNS server at 10.151.2.164:53
9372023-11-14T22:14:13.080Z2023-11-14 22:14:10.785635036 UTC: attempting to log into API
9382023-11-14T22:14:29.458Z2023-11-14 22:14:26.270477280 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9392023-11-14T22:14:29.487Z2023-11-14 22:14:27.268748262 UTC: attempting to log into API
9402023-11-14T22:14:42.250Z2023-11-14 22:14:42.262940438 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9412023-11-14T22:14:43.253Z2023-11-14 22:14:43.261690119 UTC: attempting to log into API
9422023-11-14T22:14:58.302Z2023-11-14 22:14:58.291218685 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9432023-11-14T22:14:59.309Z2023-11-14 22:14:59.293445761 UTC: attempting to log into API
9442023-11-14T22:15:14.351Z2023-11-14 22:15:14.340055444 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9452023-11-14T22:15:15.352Z2023-11-14 22:15:15.340949414 UTC: attempting to log into API
 
9462023-11-14T22:15:28.024ZNov 14 22:15:27 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9472023-11-14T22:15:30.415Z2023-11-14 22:15:30.398031946 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9482023-11-14T22:15:31.413Z2023-11-14 22:15:31.399043989 UTC: attempting to log into API
 
9492023-11-14T22:15:35.331ZNov 14 22:15:34 sok ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9502023-11-14T22:15:42.309ZNov 14 22:15:42 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9512023-11-14T22:15:46.441Z2023-11-14 22:15:46.431092700 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9522023-11-14T22:15:47.441Z2023-11-14 22:15:47.432094954 UTC: attempting to log into API
 
9532023-11-14T22:15:49.639ZNov 14 22:15:49 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9542023-11-14T22:15:56.825ZNov 14 22:15:56 sok ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9552023-11-14T22:16:02.478Z2023-11-14 22:16:02.466044027 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9562023-11-14T22:16:03.477Z2023-11-14 22:16:03.467750784 UTC: attempting to log into API
9572023-11-14T22:16:18.517Z2023-11-14 22:16:18.500242523 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9582023-11-14T22:16:20.097Z2023-11-14 22:16:19.501160374 UTC: attempting to log into API
9592023-11-14T22:16:34.543Z2023-11-14 22:16:34.533699015 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9602023-11-14T22:16:35.543Z2023-11-14 22:16:35.534702688 UTC: attempting to log into API
9612023-11-14T22:16:41.347Z2023-11-14 22:16:41.338108052 UTC: login succeeded
9622023-11-14T22:16:41.379Zwaiting for nexus to come up...
9632023-11-14T22:16:41.446Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9642023-11-14T22:16:41.497Zensuring datasets are ready...
9652023-11-14T22:16:44.069Zlet's roll.
9662023-11-14T22:16:44.094Z+ rm ./tests/bootstrap
9672023-11-14T22:16:44.137Z+ for test_bin in tests/*
9682023-11-14T22:16:44.162Z+ ./tests/bootstrap-54b36ccac73b662d
9692023-11-14T22:16:44.186Z
9702023-11-14T22:16:44.209Zrunning 0 tests
9712023-11-14T22:16:44.233Z
9722023-11-14T22:16:44.257Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9732023-11-14T22:16:44.299Z
9742023-11-14T22:16:44.346Z+ for test_bin in tests/*
9752023-11-14T22:16:44.370Z+ ./tests/end_to_end_tests-183be15994d1779e
9762023-11-14T22:16:44.400Z
9772023-11-14T22:16:44.437Zrunning 2 tests
9782023-11-14T22:16:44.468Ztest no_spoof::no_spoof_here ... ok
9792023-11-14T22:17:44.161Ztest instance_launch::instance_launch has been running for over 60 seconds
9802023-11-14T22:19:41.592Ztest instance_launch::instance_launch ... ok
9812023-11-14T22:19:41.616Z
9822023-11-14T22:19:41.639Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 177.43s
9832023-11-14T22:19:41.663Z
9842023-11-14T22:19:41.687Z+ _exit_trap
9852023-11-14T22:19:41.710Z+ local status=0
9862023-11-14T22:19:41.733Z+ [[ 0 -eq 0 ]]
9872023-11-14T22:19:41.764Z+ exit 0
9882023-11-14T22:19:41.789Zprocess exited: duration 431397 ms, exit code 0
 
9892023-11-14T22:19:41.872Zfound 56 output files
9902023-11-14T22:19:41.924Zuploading: /var/svc/log/oxide-sled-agent:default.log (674679 bytes)
9912023-11-14T22:19:43.395Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9922023-11-14T22:19:43.426Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9b8f576b-8022-47ee-be72-f88edb34fe1f/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9932023-11-14T22:19:44.478Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9b8f576b-8022-47ee-be72-f88edb34fe1f/root/var/svc/log/oxide-cockroachdb:default.log
9942023-11-14T22:19:44.528Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9b8f576b-8022-47ee-be72-f88edb34fe1f/root/var/svc/log/oxide-cockroachdb:default.log.0 (5366 bytes)
9952023-11-14T22:19:45.661Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9b8f576b-8022-47ee-be72-f88edb34fe1f/root/var/svc/log/oxide-cockroachdb:default.log.0
9962023-11-14T22:19:45.690Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_ce3d00f4-f5ef-4858-852f-6dc51e1d7203/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9972023-11-14T22:19:46.765Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_ce3d00f4-f5ef-4858-852f-6dc51e1d7203/root/var/svc/log/oxide-crucible-pantry:default.log
9982023-11-14T22:19:46.793Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f3c8a8d7-2bb9-4fd3-a0eb-bbd4dcfd63e6/root/var/svc/log/oxide-internal_dns:default.log (15030 bytes)
9992023-11-14T22:19:48.041Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f3c8a8d7-2bb9-4fd3-a0eb-bbd4dcfd63e6/root/var/svc/log/oxide-internal_dns:default.log
10002023-11-14T22:19:48.066Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f3c8a8d7-2bb9-4fd3-a0eb-bbd4dcfd63e6/root/var/svc/log/oxide-internal_dns:default.log.0 (100986 bytes)
10012023-11-14T22:19:50.079Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f3c8a8d7-2bb9-4fd3-a0eb-bbd4dcfd63e6/root/var/svc/log/oxide-internal_dns:default.log.0
10022023-11-14T22:19:50.105Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_18ff6f26-b88f-4994-8392-322a4a9036c1/root/var/svc/log/oxide-crucible-agent:default.log (14114 bytes)
10032023-11-14T22:19:51.203Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_18ff6f26-b88f-4994-8392-322a4a9036c1/root/var/svc/log/oxide-crucible-agent:default.log
10042023-11-14T22:19:51.228Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_18ff6f26-b88f-4994-8392-322a4a9036c1/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10052023-11-14T22:19:52.299Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_18ff6f26-b88f-4994-8392-322a4a9036c1/root/var/svc/log/oxide-crucible-downstairs:default.log
10062023-11-14T22:19:52.331Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_18ff6f26-b88f-4994-8392-322a4a9036c1/root/var/svc/log/oxide-crucible-downstairs:downstairs-2ecd4afc-a616-40c8-aab8-ceaeadbfb60c.log (3048 bytes)
10072023-11-14T22:19:53.406Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_18ff6f26-b88f-4994-8392-322a4a9036c1/root/var/svc/log/oxide-crucible-downstairs:downstairs-2ecd4afc-a616-40c8-aab8-ceaeadbfb60c.log
10082023-11-14T22:19:53.431Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c428618f-2cb3-46e0-939e-6576e50f1696/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10092023-11-14T22:19:54.501Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c428618f-2cb3-46e0-939e-6576e50f1696/root/var/svc/log/oxide-crucible-agent:default.log
10102023-11-14T22:19:54.525Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c428618f-2cb3-46e0-939e-6576e50f1696/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10112023-11-14T22:19:55.597Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c428618f-2cb3-46e0-939e-6576e50f1696/root/var/svc/log/oxide-crucible-downstairs:default.log
10122023-11-14T22:19:55.645Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_c2d6e029-93e4-41be-ab18-e2e480a54976/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10132023-11-14T22:19:56.708Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_c2d6e029-93e4-41be-ab18-e2e480a54976/root/var/svc/log/oxide-cockroachdb:default.log
10142023-11-14T22:19:56.734Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_c2d6e029-93e4-41be-ab18-e2e480a54976/root/var/svc/log/oxide-cockroachdb:default.log.0 (5443 bytes)
10152023-11-14T22:19:57.815Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_c2d6e029-93e4-41be-ab18-e2e480a54976/root/var/svc/log/oxide-cockroachdb:default.log.0
10162023-11-14T22:19:57.839Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7e769db4-5a0f-4ed6-9fa4-554610e19f27/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10172023-11-14T22:19:58.886Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7e769db4-5a0f-4ed6-9fa4-554610e19f27/root/var/svc/log/oxide-cockroachdb:default.log
10182023-11-14T22:19:58.910Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7e769db4-5a0f-4ed6-9fa4-554610e19f27/root/var/svc/log/oxide-cockroachdb:default.log.0 (5343 bytes)
10192023-11-14T22:19:59.982Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7e769db4-5a0f-4ed6-9fa4-554610e19f27/root/var/svc/log/oxide-cockroachdb:default.log.0
10202023-11-14T22:20:00.014Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4e989f58-0b35-4905-b57f-b15cb4af5091/root/var/svc/log/oxide-crucible-agent:default.log (13685 bytes)
10212023-11-14T22:20:01.118Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4e989f58-0b35-4905-b57f-b15cb4af5091/root/var/svc/log/oxide-crucible-agent:default.log
10222023-11-14T22:20:01.142Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4e989f58-0b35-4905-b57f-b15cb4af5091/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10232023-11-14T22:20:02.216Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4e989f58-0b35-4905-b57f-b15cb4af5091/root/var/svc/log/oxide-crucible-downstairs:default.log
10242023-11-14T22:20:02.242Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4e989f58-0b35-4905-b57f-b15cb4af5091/root/var/svc/log/oxide-crucible-downstairs:downstairs-29d63367-bee2-48ae-ab67-801152cca015.log (5534 bytes)
10252023-11-14T22:20:03.316Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4e989f58-0b35-4905-b57f-b15cb4af5091/root/var/svc/log/oxide-crucible-downstairs:downstairs-29d63367-bee2-48ae-ab67-801152cca015.log
10262023-11-14T22:20:03.759Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6f34dbd4-5819-4e25-b5af-63132ac727fc/root/var/svc/log/oxide-crucible-agent:default.log (13682 bytes)
10272023-11-14T22:20:04.449Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6f34dbd4-5819-4e25-b5af-63132ac727fc/root/var/svc/log/oxide-crucible-agent:default.log
10282023-11-14T22:20:04.476Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6f34dbd4-5819-4e25-b5af-63132ac727fc/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10292023-11-14T22:20:05.564Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6f34dbd4-5819-4e25-b5af-63132ac727fc/root/var/svc/log/oxide-crucible-downstairs:default.log
10302023-11-14T22:20:05.999Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6f34dbd4-5819-4e25-b5af-63132ac727fc/root/var/svc/log/oxide-crucible-downstairs:downstairs-4d95c6f1-ef6b-4b8e-8b78-e86a9f1e84af.log (3051 bytes)
10312023-11-14T22:20:07.080Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6f34dbd4-5819-4e25-b5af-63132ac727fc/root/var/svc/log/oxide-crucible-downstairs:downstairs-4d95c6f1-ef6b-4b8e-8b78-e86a9f1e84af.log
10322023-11-14T22:20:07.113Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fabbd57f-119c-4016-acbf-80de7b14c49d/root/var/svc/log/oxide-crucible-agent:default.log (13683 bytes)
10332023-11-14T22:20:08.227Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fabbd57f-119c-4016-acbf-80de7b14c49d/root/var/svc/log/oxide-crucible-agent:default.log
10342023-11-14T22:20:08.257Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fabbd57f-119c-4016-acbf-80de7b14c49d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10352023-11-14T22:20:11.717Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fabbd57f-119c-4016-acbf-80de7b14c49d/root/var/svc/log/oxide-crucible-downstairs:default.log
10362023-11-14T22:20:11.781Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fabbd57f-119c-4016-acbf-80de7b14c49d/root/var/svc/log/oxide-crucible-downstairs:downstairs-f92bc965-8900-43d1-aa3b-c04f7716094b.log (5536 bytes)
10372023-11-14T22:20:12.873Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fabbd57f-119c-4016-acbf-80de7b14c49d/root/var/svc/log/oxide-crucible-downstairs:downstairs-f92bc965-8900-43d1-aa3b-c04f7716094b.log
10382023-11-14T22:20:13.308Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe/root/var/svc/log/oxide-ntp:default.log (267 bytes)
10392023-11-14T22:20:14.378Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe/root/var/svc/log/oxide-ntp:default.log
10402023-11-14T22:20:14.403Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe/root/var/svc/log/oxide-ntp:default.log.0 (1046 bytes)
10412023-11-14T22:20:15.473Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe/root/var/svc/log/oxide-ntp:default.log.0
10422023-11-14T22:20:15.500Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_84c796c9-7d09-4468-a54a-429863207a8a/root/var/svc/log/oxide-oximeter:default.log (232383 bytes)
10432023-11-14T22:20:15.688Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_84c796c9-7d09-4468-a54a-429863207a8a/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 232383 -> 263152
10442023-11-14T22:20:16.760Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_84c796c9-7d09-4468-a54a-429863207a8a/root/var/svc/log/oxide-oximeter:default.log
10452023-11-14T22:20:16.786Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_e0829317-59c4-4881-a1b7-2adbab351795/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10462023-11-14T22:20:17.884Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_e0829317-59c4-4881-a1b7-2adbab351795/root/var/svc/log/oxide-cockroachdb:default.log
10472023-11-14T22:20:17.909Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_e0829317-59c4-4881-a1b7-2adbab351795/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10482023-11-14T22:20:18.989Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_e0829317-59c4-4881-a1b7-2adbab351795/root/var/svc/log/oxide-cockroachdb:default.log.0
10492023-11-14T22:20:19.015Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7736647c-1c0f-46a9-812d-aa6a13c008ee/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10502023-11-14T22:20:20.086Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7736647c-1c0f-46a9-812d-aa6a13c008ee/root/var/svc/log/oxide-crucible-agent:default.log
10512023-11-14T22:20:20.111Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7736647c-1c0f-46a9-812d-aa6a13c008ee/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10522023-11-14T22:20:21.185Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7736647c-1c0f-46a9-812d-aa6a13c008ee/root/var/svc/log/oxide-crucible-downstairs:default.log
10532023-11-14T22:20:21.209Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_c796928e-ebaa-4f8c-9dd2-1fd074652ff5/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10542023-11-14T22:20:22.281Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_c796928e-ebaa-4f8c-9dd2-1fd074652ff5/root/var/svc/log/oxide-crucible-pantry:default.log
10552023-11-14T22:20:22.321Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_2424179d-ed5b-4163-9bbc-6ed7bbe097d2/root/var/svc/log/oxide-nexus:default.log (4764964 bytes)
10562023-11-14T22:20:23.658Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_2424179d-ed5b-4163-9bbc-6ed7bbe097d2/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4764964 -> 4825578
10572023-11-14T22:20:24.732Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_2424179d-ed5b-4163-9bbc-6ed7bbe097d2/root/var/svc/log/oxide-nexus:default.log
10582023-11-14T22:20:24.778Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_387e4cc7-52fb-42aa-9012-99717813bea6/root/var/svc/log/oxide-clickhouse:default.log (829190 bytes)
10592023-11-14T22:20:25.021Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_387e4cc7-52fb-42aa-9012-99717813bea6/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 829190 -> 979900
10602023-11-14T22:20:26.093Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_387e4cc7-52fb-42aa-9012-99717813bea6/root/var/svc/log/oxide-clickhouse:default.log
10612023-11-14T22:20:26.117Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_b264ac32-6fba-4cd3-afbd-da0df13a9eb2/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10622023-11-14T22:20:27.165Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_b264ac32-6fba-4cd3-afbd-da0df13a9eb2/root/var/svc/log/oxide-cockroachdb:default.log
10632023-11-14T22:20:27.190Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_b264ac32-6fba-4cd3-afbd-da0df13a9eb2/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10642023-11-14T22:20:28.262Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_b264ac32-6fba-4cd3-afbd-da0df13a9eb2/root/var/svc/log/oxide-cockroachdb:default.log.0
10652023-11-14T22:20:28.291Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_0bfb0f0c-d24b-4f86-9d6c-4c931f4fbd17/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10662023-11-14T22:20:29.361Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_0bfb0f0c-d24b-4f86-9d6c-4c931f4fbd17/root/var/svc/log/oxide-crucible-pantry:default.log
10672023-11-14T22:20:29.387Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_80f431f5-5f86-4dbd-81d4-fbc03d6cf8f4/root/var/svc/log/oxide-nexus:default.log (521005 bytes)
10682023-11-14T22:20:29.620Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_80f431f5-5f86-4dbd-81d4-fbc03d6cf8f4/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 521005 -> 582222
10692023-11-14T22:20:30.694Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_80f431f5-5f86-4dbd-81d4-fbc03d6cf8f4/root/var/svc/log/oxide-nexus:default.log
10702023-11-14T22:20:30.719Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_03a615c5-583b-4a12-97d9-58b343047b31/root/var/svc/log/oxide-crucible-agent:default.log (14112 bytes)
10712023-11-14T22:20:31.813Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_03a615c5-583b-4a12-97d9-58b343047b31/root/var/svc/log/oxide-crucible-agent:default.log
10722023-11-14T22:20:31.839Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_03a615c5-583b-4a12-97d9-58b343047b31/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10732023-11-14T22:20:32.907Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_03a615c5-583b-4a12-97d9-58b343047b31/root/var/svc/log/oxide-crucible-downstairs:default.log
10742023-11-14T22:20:32.932Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_03a615c5-583b-4a12-97d9-58b343047b31/root/var/svc/log/oxide-crucible-downstairs:downstairs-f4a558ea-3ff5-498f-85f0-29b436613548.log (5537 bytes)
10752023-11-14T22:20:34.052Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_03a615c5-583b-4a12-97d9-58b343047b31/root/var/svc/log/oxide-crucible-downstairs:downstairs-f4a558ea-3ff5-498f-85f0-29b436613548.log
10762023-11-14T22:20:34.077Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_14314f22-8159-44be-828e-2e2dfe4caa4b/root/var/svc/log/oxide-crucible-agent:default.log (14577 bytes)
10772023-11-14T22:20:35.172Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_14314f22-8159-44be-828e-2e2dfe4caa4b/root/var/svc/log/oxide-crucible-agent:default.log
10782023-11-14T22:20:35.199Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_14314f22-8159-44be-828e-2e2dfe4caa4b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10792023-11-14T22:20:36.267Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_14314f22-8159-44be-828e-2e2dfe4caa4b/root/var/svc/log/oxide-crucible-downstairs:default.log
10802023-11-14T22:20:36.291Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_14314f22-8159-44be-828e-2e2dfe4caa4b/root/var/svc/log/oxide-crucible-downstairs:downstairs-9db4c7ff-b1c7-4910-ac54-87328f6f9802.log (3050 bytes)
10812023-11-14T22:20:37.362Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_14314f22-8159-44be-828e-2e2dfe4caa4b/root/var/svc/log/oxide-crucible-downstairs:downstairs-9db4c7ff-b1c7-4910-ac54-87328f6f9802.log
10822023-11-14T22:20:37.387Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1dc21b94-6dc1-45ff-b37f-45eef4ec3a31/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10832023-11-14T22:20:38.466Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1dc21b94-6dc1-45ff-b37f-45eef4ec3a31/root/var/svc/log/oxide-crucible-agent:default.log
10842023-11-14T22:20:38.492Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1dc21b94-6dc1-45ff-b37f-45eef4ec3a31/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10852023-11-14T22:20:39.568Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1dc21b94-6dc1-45ff-b37f-45eef4ec3a31/root/var/svc/log/oxide-crucible-downstairs:default.log
10862023-11-14T22:20:39.593Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e9903cd0-9cf2-40fc-a7d2-34ffb3ec4ede/root/var/svc/log/oxide-external_dns:default.log (15617 bytes)
10872023-11-14T22:20:39.663Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e9903cd0-9cf2-40fc-a7d2-34ffb3ec4ede/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15617 -> 19394
10882023-11-14T22:20:40.736Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e9903cd0-9cf2-40fc-a7d2-34ffb3ec4ede/root/var/svc/log/oxide-external_dns:default.log
10892023-11-14T22:20:40.761Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_12d4fa79-1d19-4300-b173-75a97e10c949/root/var/svc/log/oxide-internal_dns:default.log (15034 bytes)
10902023-11-14T22:20:40.829Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_12d4fa79-1d19-4300-b173-75a97e10c949/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 15034 -> 18788
10912023-11-14T22:20:41.903Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_12d4fa79-1d19-4300-b173-75a97e10c949/root/var/svc/log/oxide-internal_dns:default.log
10922023-11-14T22:20:41.928Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_12d4fa79-1d19-4300-b173-75a97e10c949/root/var/svc/log/oxide-internal_dns:default.log.0 (98673 bytes)
10932023-11-14T22:20:43.114Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_12d4fa79-1d19-4300-b173-75a97e10c949/root/var/svc/log/oxide-internal_dns:default.log.0
10942023-11-14T22:20:43.139Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_26eecfa2-6e1c-4dbe-afe7-5b9d168269f7/root/var/svc/log/oxide-internal_dns:default.log (15031 bytes)
10952023-11-14T22:20:43.233Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_26eecfa2-6e1c-4dbe-afe7-5b9d168269f7/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 15031 -> 18787
10962023-11-14T22:20:44.313Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_26eecfa2-6e1c-4dbe-afe7-5b9d168269f7/root/var/svc/log/oxide-internal_dns:default.log
10972023-11-14T22:20:44.338Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_26eecfa2-6e1c-4dbe-afe7-5b9d168269f7/root/var/svc/log/oxide-internal_dns:default.log.0 (100277 bytes)
10982023-11-14T22:20:45.525Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_26eecfa2-6e1c-4dbe-afe7-5b9d168269f7/root/var/svc/log/oxide-internal_dns:default.log.0
10992023-11-14T22:20:45.552Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_894c1ff3-1b2e-47a5-8875-c40d112973fb/root/var/svc/log/oxide-external_dns:default.log (15608 bytes)
11002023-11-14T22:20:45.618Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_894c1ff3-1b2e-47a5-8875-c40d112973fb/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15608 -> 19387
11012023-11-14T22:20:46.691Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_894c1ff3-1b2e-47a5-8875-c40d112973fb/root/var/svc/log/oxide-external_dns:default.log
11022023-11-14T22:20:47.127Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8464815a-80a3-449f-b095-cae15a214d2f/root/var/svc/log/oxide-nexus:default.log (525377 bytes)
11032023-11-14T22:20:47.152Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8464815a-80a3-449f-b095-cae15a214d2f/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 525377 -> 587205
11042023-11-14T22:20:48.038Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8464815a-80a3-449f-b095-cae15a214d2f/root/var/svc/log/oxide-nexus:default.log
11052023-11-14T22:20:48.063Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe/root/var/log/chrony/measurements.log (12878 bytes)
11062023-11-14T22:20:48.110Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe/root/var/log/chrony/measurements.log" changed size mid upload: 12878 -> 15070
11072023-11-14T22:20:49.190Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe/root/var/log/chrony/measurements.log
11082023-11-14T22:20:49.215Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe/root/var/log/chrony/statistics.log (7020 bytes)
11092023-11-14T22:20:49.260Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe/root/var/log/chrony/statistics.log" changed size mid upload: 7020 -> 8775
11102023-11-14T22:20:50.332Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe/root/var/log/chrony/statistics.log
11112023-11-14T22:20:50.357Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe/root/var/log/chrony/tracking.log (1855 bytes)
11122023-11-14T22:20:50.381Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe/root/var/log/chrony/tracking.log" changed size mid upload: 1855 -> 1987
11132023-11-14T22:20:51.465Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_aeb7e55c-9743-434c-8c75-532aad5eb3fe/root/var/log/chrony/tracking.log