01HHM2HQBVD261K3RWPNWE6565: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.27",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HHM2J6ZHWYBASH0WCX1XECNH

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-14T12:38:00.131Zjob dependencies complete; ready to run (waiting for 48 m 26 s)
22023-12-14T12:43:37.919Zjob assigned to worker 01HHM5B353GNA85BR3TGWV3B2V (queued for 5 m 37 s)
32023-12-14T12:43:38.143Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-12-14T12:43:38.176Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-12-14T12:43:38.209Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-12-14T12:43:38.243Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-14T12:43:38.276Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-12-14T12:43:38.310ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-14T12:43:38.343Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-14T12:43:38.377ZOxide Helios Version helios-2.0.22333 64-bit
112023-12-14T12:43:38.410ZConfiguring devices.
122023-12-14T12:43:38.443Z##################################################
132023-12-14T12:43:38.477Z#### ############# ##
142023-12-14T12:43:38.514Z### ### ############ ##
152023-12-14T12:43:38.545Z## ### # ## ### ## Oxide Computer Company ##
162023-12-14T12:43:38.578Z## ## ## ### # ### ##
172023-12-14T12:43:38.612Z## # ### #### #### This Station Under ##
182023-12-14T12:43:38.645Z### ### #### # ### Computer Control ##
192023-12-14T12:43:38.679Z#### #### ### ## ##
202023-12-14T12:43:38.712Z##################################################
212023-12-14T12:43:38.745ZHostname: sock
222023-12-14T12:43:38.779ZLoading smf(7) service descriptions:
232023-12-14T12:43:39.774Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-eae955b6030e5419.gz
242023-12-14T12:43:41.017Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-eae955b6030e5419.gz
252023-12-14T12:43:41.017Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-14T12:43:57.825Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-14T12:43:57.825Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-114e4ef0b9f9b2b3.gz
282023-12-14T12:44:15.398Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-114e4ef0b9f9b2b3.gz
292023-12-14T12:44:15.398Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-14T12:44:16.518Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-14T12:44:16.546Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-14T12:44:19.305Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-14T12:44:19.329Zdownloading input: /input/package/work/version.txt
342023-12-14T12:44:19.356Zdownloaded input: /input/package/work/version.txt
352023-12-14T12:44:19.356Zdownloading input: /input/package/work/package.tar.gz
362023-12-14T12:44:33.181Zdownloaded input: /input/package/work/package.tar.gz
372023-12-14T12:44:33.181Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-14T12:44:53.863Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-14T12:44:53.863Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-14T12:45:00.176Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-14T12:45:00.201Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-14T12:45:31.626Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-14T12:45:31.626Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-14T12:46:02.892Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-14T12:46:02.893Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-14T12:46:23.390Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-14T12:46:23.390Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-14T12:46:27.450Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-14T12:46:27.450Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-14T12:46:34.794Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-14T12:46:34.794Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-14T12:46:39.074Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-14T12:46:39.074Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-14T12:46:43.314Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-14T12:46:43.339Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-14T12:46:43.366Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-14T12:46:43.366Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-14T12:46:46.594Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-14T12:46:46.594Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-14T12:46:49.812Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-14T12:46:49.836Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-14T12:47:02.720Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-14T12:47:02.720Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-14T12:47:15.640Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-14T12:47:15.640Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-14T12:47:15.711Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-14T12:47:15.735Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-14T12:47:21.962Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-14T12:47:21.962Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-14T12:47:33.566Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-14T12:47:33.567Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-14T12:48:15.461Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-14T12:48:15.461Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-14T12:48:44.602Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-14T12:48:44.627Zstarting task 0: "setup"
762023-12-14T12:48:44.627Z++ uname -s
772023-12-14T12:48:44.627Z+ kern=SunOS
782023-12-14T12:48:44.627Z+ case "$kern" in
792023-12-14T12:48:44.627Z+ groupadd -g 12345 build
802023-12-14T12:48:44.628Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-14T12:48:46.619Z+ zfs create -o mountpoint=/work rpool/work
822023-12-14T12:48:46.658Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-14T12:48:46.683Z+ home_fs=zfs
842023-12-14T12:48:46.683Z+ [[ zfs == autofs ]]
852023-12-14T12:48:46.683Z+ mkdir -p /home/build
862023-12-14T12:48:46.683Z+ chown build:build /home/build /work
872023-12-14T12:48:47.666Z+ chmod 0700 /home/build /work
882023-12-14T12:48:47.690Zprocess exited: duration 3067 ms, exit code 0
 
892023-12-14T12:48:47.739Zstarting task 1: "authentication"
902023-12-14T12:48:47.785Zprocess exited: duration 43 ms, exit code 0
 
912023-12-14T12:48:47.835Zstarting task 2: "build"
922023-12-14T12:48:47.860Z+ trap _exit_trap EXIT
932023-12-14T12:48:47.860Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-14T12:48:48.040Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-14T12:48:48.040Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-14T12:48:48.040Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-14T12:48:48.040Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-14T12:48:48.040Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-14T12:48:48.065Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-14T12:48:48.065Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-14T12:48:48.065Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-14T12:48:48.065Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-14T12:48:48.065Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-14T12:48:48.065Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-14T12:48:48.065Z+ [[ -d /opt/oxide ]]
1062023-12-14T12:48:48.065Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-14T12:48:48.065Z+ pfexec mkdir /opt/oxide
1082023-12-14T12:48:48.065Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-14T12:48:48.065Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-14T12:48:48.065Z+ cd /opt/oxide-underneath
1112023-12-14T12:48:48.065Z+ pfexec tar ceEp@/f - .
1122023-12-14T12:48:48.065Z+ cd /opt/oxide
1132023-12-14T12:48:48.065Z+ pfexec tar xveEp@/f -
1142023-12-14T12:48:48.169Zx ., 0 bytes, 0 tape blocks
1152023-12-14T12:48:48.169Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-14T12:48:48.169Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-14T12:48:48.169Zx ./opte/bin/opteadm, 52567720 bytes, 102672 tape blocks
1182023-12-14T12:48:48.194Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-14T12:48:48.194Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-14T12:48:48.219Z+ pfexec mkdir /opt/oxide/work
1212023-12-14T12:48:48.219Z+ pfexec chown build:build /opt/oxide/work
1222023-12-14T12:48:48.219Z+ cd /opt/oxide/work
1232023-12-14T12:48:48.219Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-14T12:48:49.751ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-12-14T12:48:49.752Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-12-14T12:48:49.752Zx out/omicron-sled-agent.tar, 111882752 bytes, 218521 tape blocks
1272023-12-14T12:48:49.752Zx out/target/test, 75 bytes, 1 tape blocks
1282023-12-14T12:48:49.752Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1292023-12-14T12:48:49.752Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-12-14T12:48:49.752Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1312023-12-14T12:48:49.752Zx target/release/omicron-package, 42517504 bytes, 83042 tape blocks
1322023-12-14T12:48:49.752Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-12-14T12:48:49.752Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-12-14T12:48:49.752Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-12-14T12:48:49.753Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-12-14T12:48:49.753Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-12-14T12:48:49.753Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-12-14T12:48:49.753Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-12-14T12:48:49.778Z
1402023-12-14T12:48:49.778Zreal 1.542071834
1412023-12-14T12:48:49.778Zuser 1.385218140
1422023-12-14T12:48:49.778Zsys 0.275023927
1432023-12-14T12:48:49.782Ztrap 0.000049572
1442023-12-14T12:48:49.782Ztflt 0.000000000
1452023-12-14T12:48:49.782Zdflt 0.000000000
1462023-12-14T12:48:49.782Zkflt 0.000000000
1472023-12-14T12:48:49.782Zlock 0.000000000
1482023-12-14T12:48:49.782Zslp 1.386339712
1492023-12-14T12:48:49.782Zlat 0.035280913
1502023-12-14T12:48:49.782Zstop 0.000126002
1512023-12-14T12:48:49.782Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-12-14T12:48:50.831Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-12-14T12:48:50.855Z+ mkdir tests
1542023-12-14T12:48:50.855Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-12-14T12:48:50.855Z+ ptime -m gunzip
1562023-12-14T12:48:50.855Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-eae955b6030e5419
1572023-12-14T12:48:50.992Z
1582023-12-14T12:48:51.017Zreal 0.141888683
1592023-12-14T12:48:51.017Zuser 0.128293263
1602023-12-14T12:48:51.017Zsys 0.013370153
1612023-12-14T12:48:51.017Ztrap 0.000011580
1622023-12-14T12:48:51.018Ztflt 0.000000000
1632023-12-14T12:48:51.018Zdflt 0.000000000
1642023-12-14T12:48:51.018Zkflt 0.000000000
1652023-12-14T12:48:51.018Zlock 0.000000000
1662023-12-14T12:48:51.018Zslp 0.000018690
1672023-12-14T12:48:51.018Zlat 0.000015310
1682023-12-14T12:48:51.018Zstop 0.000120312
1692023-12-14T12:48:51.018Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-eae955b6030e5419
1702023-12-14T12:48:51.018Z+ chmod a+x tests/bootstrap-eae955b6030e5419
1712023-12-14T12:48:51.018Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-12-14T12:48:51.018Z+ ptime -m gunzip
1732023-12-14T12:48:51.018Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-12-14T12:48:53.232Z
1752023-12-14T12:48:53.256Zreal 2.227953612
1762023-12-14T12:48:53.256Zuser 2.091418667
1772023-12-14T12:48:53.256Zsys 0.136113376
1782023-12-14T12:48:53.256Ztrap 0.000181173
1792023-12-14T12:48:53.257Ztflt 0.000000000
1802023-12-14T12:48:53.257Zdflt 0.000000000
1812023-12-14T12:48:53.257Zkflt 0.000000000
1822023-12-14T12:48:53.257Zlock 0.000000000
1832023-12-14T12:48:53.257Zslp 0.000021690
1842023-12-14T12:48:53.257Zlat 0.000030490
1852023-12-14T12:48:53.257Zstop 0.000129443
1862023-12-14T12:48:53.257Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-12-14T12:48:53.257Z+ chmod a+x tests/bootstrap
1882023-12-14T12:48:53.257Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-12-14T12:48:53.257Z+ ptime -m gunzip
1902023-12-14T12:48:53.257Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-114e4ef0b9f9b2b3
1912023-12-14T12:48:55.553Z
1922023-12-14T12:48:55.578Zreal 2.309766385
1932023-12-14T12:48:55.578Zuser 2.159627859
1942023-12-14T12:48:55.578Zsys 0.149752547
1952023-12-14T12:48:55.578Ztrap 0.000152563
1962023-12-14T12:48:55.578Ztflt 0.000000000
1972023-12-14T12:48:55.578Zdflt 0.000000000
1982023-12-14T12:48:55.578Zkflt 0.000000000
1992023-12-14T12:48:55.578Zlock 0.000000000
2002023-12-14T12:48:55.578Zslp 0.000020700
2012023-12-14T12:48:55.578Zlat 0.000022740
2022023-12-14T12:48:55.578Zstop 0.000123212
2032023-12-14T12:48:55.578Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-114e4ef0b9f9b2b3
2042023-12-14T12:48:55.578Z+ chmod a+x tests/end_to_end_tests-114e4ef0b9f9b2b3
2052023-12-14T12:48:55.578Z+ bmat address ls
2062023-12-14T12:48:55.579ZNAME CIDR FIRST LAST COUNT
2072023-12-14T12:48:55.579Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-12-14T12:48:55.579Z++ bmat address ls -f extra -Ho first,last,count
2092023-12-14T12:48:55.579Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-12-14T12:48:55.579Z+ EXTRA_IP_START=10.151.2.164
2112023-12-14T12:48:55.579Z+ EXTRA_IP_END=10.151.2.227
2122023-12-14T12:48:55.579Z+ EXTRA_IP_COUNT=64
2132023-12-14T12:48:55.579Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-12-14T12:48:55.579Z+ EXTRA_IP_BASE=10.151.2
2152023-12-14T12:48:55.579Z+ EXTRA_IP_FOCTET=164
2162023-12-14T12:48:55.580Z+ EXTRA_IP_LOCTET=227
2172023-12-14T12:48:55.580Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-12-14T12:48:55.580Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-12-14T12:48:55.580Z+ DNS_IP1=10.151.2.164
2202023-12-14T12:48:55.580Z+ DNS_IP2=10.151.2.165
2212023-12-14T12:48:55.580Z+ UPLINK_IP=10.151.2.174
2222023-12-14T12:48:55.580Z+ IPPOOL_START=10.151.2.175
2232023-12-14T12:48:55.580Z+ IPPOOL_END=10.151.2.227
2242023-12-14T12:48:55.605Z++ ipadm show-addr -po type,addr
2252023-12-14T12:48:55.605Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-12-14T12:48:55.605Z+ GATEWAY_IP=10.151.2.100
2272023-12-14T12:48:55.605Z+ [[ -n 10.151.2.100 ]]
2282023-12-14T12:48:55.605Z+ ping -s 10.151.2.100 56 1
2292023-12-14T12:48:55.605ZPING 10.151.2.100: 56 data bytes
2302023-12-14T12:48:55.605Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.012 ms
2312023-12-14T12:48:55.606Z
2322023-12-14T12:48:55.606Z----10.151.2.100 PING Statistics----
2332023-12-14T12:48:55.606Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-12-14T12:48:55.607Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2352023-12-14T12:48:55.607Z++ arp -an
2362023-12-14T12:48:55.607Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-12-14T12:48:55.607Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-12-14T12:48:55.607Z+ routeadm -e ipv4-forwarding -u
2392023-12-14T12:48:55.717Z+ PXA_START=10.151.2.164
2402023-12-14T12:48:55.717Z+ PXA_END=10.151.2.227
2412023-12-14T12:48:55.717Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-12-14T12:48:55.742Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-12-14T12:48:55.925Z+ ZPOOL_VDEV_DIR=/scratch
2442023-12-14T12:48:55.925Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-12-14T12:48:55.950Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-12-14T12:48:55.950Z++ cd ./tools
2472023-12-14T12:48:55.950Z++ pwd
2482023-12-14T12:48:55.950Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-12-14T12:48:55.950Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-12-14T12:48:55.950Z+ SOFTNPU_MODE=zone
2512023-12-14T12:48:55.950Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-12-14T12:48:55.950Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-12-14T12:48:55.950Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-12-14T12:48:55.950Z++ dladm show-phys -p -o LINK
2552023-12-14T12:48:55.950Z++ head -1
2562023-12-14T12:48:56.008Z+ PHYSICAL_LINK=igb0
2572023-12-14T12:48:56.098Z+ echo 'Using igb0 as physical link'
2582023-12-14T12:48:56.098ZUsing igb0 as physical link
2592023-12-14T12:48:56.098Z+ ensure_run_as_root
2602023-12-14T12:48:56.098Z++ id -u
2612023-12-14T12:48:56.098Z+ [[ 0 -ne 0 ]]
2622023-12-14T12:48:56.098Z+ ensure_zpools
2632023-12-14T12:48:56.098Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-12-14T12:48:56.098Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-12-14T12:48:56.098Z+ readarray -t ZPOOLS
2662023-12-14T12:48:56.098Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-12-14T12:48:56.098Z++ sed 's/[ ",]//g'
2682023-12-14T12:48:56.098Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-12-14T12:48:56.098Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-12-14T12:48:56.098ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-12-14T12:48:56.098Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-12-14T12:48:56.098Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-12-14T12:48:56.098Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-12-14T12:48:56.123Z0+0 records in
2752023-12-14T12:48:56.123Z0+0 records out
2762023-12-14T12:48:56.123Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2772023-12-14T12:48:56.123Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-12-14T12:48:56.123Z+ set +x
2792023-12-14T12:48:56.124ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-12-14T12:48:56.124Z++ zpool list -o name
2812023-12-14T12:48:56.124Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-12-14T12:48:56.124Z+ [[ -z '' ]]
2832023-12-14T12:48:56.124Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-12-14T12:48:56.124Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-12-14T12:48:56.124Z+ set +x
2862023-12-14T12:48:56.124ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-12-14T12:48:56.124Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-12-14T12:48:56.124Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-12-14T12:48:56.124ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-12-14T12:48:56.124Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-12-14T12:48:56.124Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-12-14T12:48:56.124Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-12-14T12:48:56.124Z0+0 records in
2942023-12-14T12:48:56.124Z0+0 records out
2952023-12-14T12:48:56.124Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2962023-12-14T12:48:56.124Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-12-14T12:48:56.126Z+ set +x
2982023-12-14T12:48:56.126ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-12-14T12:48:56.126Z++ zpool list -o name
3002023-12-14T12:48:56.126Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-12-14T12:48:56.126Z+ [[ -z '' ]]
3022023-12-14T12:48:56.126Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-12-14T12:48:56.150Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-12-14T12:48:56.151Z+ set +x
3052023-12-14T12:48:56.151ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-12-14T12:48:56.151Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-12-14T12:48:56.151Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-12-14T12:48:56.151ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3092023-12-14T12:48:56.151Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-12-14T12:48:56.151Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-12-14T12:48:56.151Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-12-14T12:48:56.175Z0+0 records in
3132023-12-14T12:48:56.175Z0+0 records out
3142023-12-14T12:48:56.176Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3152023-12-14T12:48:56.176Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-12-14T12:48:56.176Z+ set +x
3172023-12-14T12:48:56.176ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-12-14T12:48:56.176Z++ zpool list -o name
3192023-12-14T12:48:56.176Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-12-14T12:48:56.176Z+ [[ -z '' ]]
3212023-12-14T12:48:56.176Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-12-14T12:48:56.216Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-12-14T12:48:56.216Z+ set +x
3242023-12-14T12:48:56.216ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-12-14T12:48:56.216Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-12-14T12:48:56.216ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3272023-12-14T12:48:56.216Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3282023-12-14T12:48:56.216Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-12-14T12:48:56.216Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-12-14T12:48:56.216Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-12-14T12:48:56.241Z0+0 records in
3322023-12-14T12:48:56.241Z0+0 records out
3332023-12-14T12:48:56.241Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-12-14T12:48:56.241Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-12-14T12:48:56.241Z+ set +x
3362023-12-14T12:48:56.241ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-12-14T12:48:56.241Z++ zpool list -o name
3382023-12-14T12:48:56.241Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-12-14T12:48:56.241Z+ [[ -z '' ]]
3402023-12-14T12:48:56.241Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-12-14T12:48:56.266Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-12-14T12:48:56.266Z+ set +x
3432023-12-14T12:48:56.266ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-12-14T12:48:56.266Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-12-14T12:48:56.266ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3462023-12-14T12:48:56.267Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3472023-12-14T12:48:56.267Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-12-14T12:48:56.267Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-12-14T12:48:56.267Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-12-14T12:48:56.267Z0+0 records in
3512023-12-14T12:48:56.267Z0+0 records out
3522023-12-14T12:48:56.267Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3532023-12-14T12:48:56.267Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-12-14T12:48:56.267Z+ set +x
3552023-12-14T12:48:56.267ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-12-14T12:48:56.267Z++ zpool list -o name
3572023-12-14T12:48:56.267Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-12-14T12:48:56.267Z+ [[ -z '' ]]
3592023-12-14T12:48:56.267Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-12-14T12:48:56.298Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-12-14T12:48:56.298Z+ set +x
3622023-12-14T12:48:56.323ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-12-14T12:48:56.323Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-12-14T12:48:56.323Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-12-14T12:48:56.323ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-12-14T12:48:56.323Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-12-14T12:48:56.323Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-12-14T12:48:56.323Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-12-14T12:48:56.323Z0+0 records in
3702023-12-14T12:48:56.323Z0+0 records out
3712023-12-14T12:48:56.324Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3722023-12-14T12:48:56.324Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-12-14T12:48:56.324Z+ set +x
3742023-12-14T12:48:56.324ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-12-14T12:48:56.324Z++ zpool list -o name
3762023-12-14T12:48:56.324Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-12-14T12:48:56.324Z+ [[ -z '' ]]
3782023-12-14T12:48:56.324Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-12-14T12:48:56.348Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-12-14T12:48:56.349Z+ set +x
3812023-12-14T12:48:56.349ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-12-14T12:48:56.349Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-12-14T12:48:56.349Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-12-14T12:48:56.349ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-12-14T12:48:56.349Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-12-14T12:48:56.349Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-12-14T12:48:56.349Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-12-14T12:48:56.373Z0+0 records in
3892023-12-14T12:48:56.373Z0+0 records out
3902023-12-14T12:48:56.373Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-12-14T12:48:56.374Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-12-14T12:48:56.374Z+ set +x
3932023-12-14T12:48:56.374ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-12-14T12:48:56.374Z++ zpool list -o name
3952023-12-14T12:48:56.375Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-12-14T12:48:56.375Z+ [[ -z '' ]]
3972023-12-14T12:48:56.375Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-12-14T12:48:56.401Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-12-14T12:48:56.401Z+ set +x
4002023-12-14T12:48:56.425ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-12-14T12:48:56.425Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-12-14T12:48:56.425Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-12-14T12:48:56.425ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-12-14T12:48:56.425Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-12-14T12:48:56.425Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-12-14T12:48:56.426Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-12-14T12:48:56.426Z0+0 records in
4082023-12-14T12:48:56.426Z0+0 records out
4092023-12-14T12:48:56.426Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4102023-12-14T12:48:56.426Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-12-14T12:48:56.426Z+ set +x
4122023-12-14T12:48:56.426ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-12-14T12:48:56.426Z++ zpool list -o name
4142023-12-14T12:48:56.426Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-12-14T12:48:56.426Z+ [[ -z '' ]]
4162023-12-14T12:48:56.426Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-12-14T12:48:56.451Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-12-14T12:48:56.451Z+ set +x
4192023-12-14T12:48:56.451ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-12-14T12:48:56.452Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-12-14T12:48:56.452Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-12-14T12:48:56.452ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-12-14T12:48:56.498Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-12-14T12:48:56.498Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-12-14T12:48:56.498Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-12-14T12:48:56.523Z0+0 records in
4272023-12-14T12:48:56.523Z0+0 records out
4282023-12-14T12:48:56.523Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4292023-12-14T12:48:56.523Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-12-14T12:48:56.523Z+ set +x
4312023-12-14T12:48:56.523ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-12-14T12:48:56.523Z++ zpool list -o name
4332023-12-14T12:48:56.523Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-12-14T12:48:56.523Z+ [[ -z '' ]]
4352023-12-14T12:48:56.523Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-12-14T12:48:56.523Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-12-14T12:48:56.523Z+ set +x
4382023-12-14T12:48:56.523ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-12-14T12:48:56.523Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-12-14T12:48:56.524Z+ readarray -t ZPOOLS
4412023-12-14T12:48:56.524Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-12-14T12:48:56.524Z++ sed 's/[ ",]//g'
4432023-12-14T12:48:56.524Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-12-14T12:48:56.524Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-12-14T12:48:56.524ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-12-14T12:48:56.524Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-12-14T12:48:56.524Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-12-14T12:48:56.524Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-12-14T12:48:56.524Z0+0 records in
4502023-12-14T12:48:56.524Z0+0 records out
4512023-12-14T12:48:56.524Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4522023-12-14T12:48:56.524Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-12-14T12:48:56.524Z+ set +x
4542023-12-14T12:48:56.524ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-12-14T12:48:56.524Z++ zpool list -o name
4562023-12-14T12:48:56.524Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-12-14T12:48:56.549Z+ [[ -z '' ]]
4582023-12-14T12:48:56.549Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-12-14T12:48:56.573Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-12-14T12:48:56.573Z+ set +x
4612023-12-14T12:48:56.573ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-12-14T12:48:56.573Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-12-14T12:48:56.573Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-12-14T12:48:56.573ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-12-14T12:48:56.573Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-12-14T12:48:56.574Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-12-14T12:48:56.574Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-12-14T12:48:56.574Z0+0 records in
4692023-12-14T12:48:56.574Z0+0 records out
4702023-12-14T12:48:56.574Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
4712023-12-14T12:48:56.574Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-12-14T12:48:56.574Z+ set +x
4732023-12-14T12:48:56.574ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-12-14T12:48:56.574Z++ zpool list -o name
4752023-12-14T12:48:56.598Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-12-14T12:48:56.598Z+ [[ -z '' ]]
4772023-12-14T12:48:56.598Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-12-14T12:48:56.623Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-12-14T12:48:56.623Z+ set +x
4802023-12-14T12:48:56.623ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-12-14T12:48:56.623Z+ [[ zone == \z\o\n\e ]]
4822023-12-14T12:48:56.623Z+ ensure_simulated_links igb0
4832023-12-14T12:48:56.623Z+ local PHYSICAL_LINK=igb0
4842023-12-14T12:48:56.623Z+ INDICES=("0" "1")
4852023-12-14T12:48:56.623Z+ for I in "${INDICES[@]}"
4862023-12-14T12:48:56.623Z++ get_simnet_name_if_exists net0
4872023-12-14T12:48:56.623Z++ dladm show-simnet -p -o LINK net0
4882023-12-14T12:48:56.648Z++ echo ''
4892023-12-14T12:48:56.648Z+ [[ -z '' ]]
4902023-12-14T12:48:56.648Z+ dladm create-simnet -t net0
4912023-12-14T12:48:56.648Z+ dladm create-simnet -t sc0_0
4922023-12-14T12:48:56.648Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-12-14T12:48:56.648Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-12-14T12:48:56.648Z+ success 'Simnet net0/sc0_0 exists'
4952023-12-14T12:48:56.648Z+ set +x
4962023-12-14T12:48:56.650ZSimnet net0/sc0_0 exists
4972023-12-14T12:48:56.650Z+ for I in "${INDICES[@]}"
4982023-12-14T12:48:56.650Z++ get_simnet_name_if_exists net1
4992023-12-14T12:48:56.650Z++ dladm show-simnet -p -o LINK net1
5002023-12-14T12:48:56.650Z++ echo ''
5012023-12-14T12:48:56.675Z+ [[ -z '' ]]
5022023-12-14T12:48:56.675Z+ dladm create-simnet -t net1
5032023-12-14T12:48:56.675Z+ dladm create-simnet -t sc1_0
5042023-12-14T12:48:56.675Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-12-14T12:48:56.675Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-12-14T12:48:56.675Z+ success 'Simnet net1/sc1_0 exists'
5072023-12-14T12:48:56.675Z+ set +x
5082023-12-14T12:48:56.675ZSimnet net1/sc1_0 exists
5092023-12-14T12:48:56.675Z++ get_vnic_name_if_exists sc0_1
5102023-12-14T12:48:56.675Z++ dladm show-vnic -p -o LINK sc0_1
5112023-12-14T12:48:56.675Z++ echo ''
5122023-12-14T12:48:56.675Z+ [[ -z '' ]]
5132023-12-14T12:48:56.675Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-12-14T12:48:56.700Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-12-14T12:48:56.700Z+ success 'Vnic sc0_1 exists'
5162023-12-14T12:48:56.700Z+ set +x
5172023-12-14T12:48:56.700ZVnic sc0_1 exists
5182023-12-14T12:48:56.700Z+ warn_if_no_proxy_arp
5192023-12-14T12:48:56.701Z+ [[ -v PXA_START ]]
5202023-12-14T12:48:56.701Z+ [[ -v PXA_END ]]
5212023-12-14T12:48:56.701Z+ ensure_softnpu_zone
5222023-12-14T12:48:56.701Z+ zoneadm list
5232023-12-14T12:48:56.701Z+ grep -q sidecar_softnpu
5242023-12-14T12:48:56.701Z+ [[ -f out/npuzone/npuzone ]]
5252023-12-14T12:48:56.703Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit 45ed98fea5824feb4d42f45bbf218e597dc9fc58 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5262023-12-14T12:48:56.703Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5272023-12-14T12:49:05.569Zdownloading softnpu ...done
5282023-12-14T12:49:05.594Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/libsidecar_lite.so.sha256.txt
5292023-12-14T12:49:11.571Zdownloading asic_program.so ...done
5302023-12-14T12:49:11.596Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/scadm.sha256.txt
5312023-12-14T12:49:22.702Zdownloading scadm ...done
5322023-12-14T12:49:22.727Zcreating port tfportrear0_0 for sc0_0
5332023-12-14T12:49:22.727Zcreating port tfportqsfp0_0 for sc0_1
5342023-12-14T12:49:22.752Zsoftnpu zone setup
5352023-12-14T12:49:22.752Zconfigure zone
5362023-12-14T12:49:22.752Zinstall zone
5372023-12-14T12:49:23.796Zboot zone
5382023-12-14T12:49:26.900Z[sidecar_softnpu] svcs milestone/network
5392023-12-14T12:49:26.927ZSTATE STIME FMRI
5402023-12-14T12:49:26.927Zonline 12:49:25 svc:/milestone/network:default
5412023-12-14T12:49:26.927Z[sidecar_softnpu] /softnpu/npu start
5422023-12-14T12:49:26.952Z
5432023-12-14T12:49:26.977Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-12-14T12:49:26.978Z+ GATEWAY_IP=10.151.2.100
5452023-12-14T12:49:26.978Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-12-14T12:49:26.978ZUsing 10.151.2.100 as gateway ip
5472023-12-14T12:49:26.978Z+ [[ ! -v GATEWAY_MAC ]]
5482023-12-14T12:49:26.978Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-12-14T12:49:26.978Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-12-14T12:49:26.978Z+++ echo 18:c0:4d:0d:a0:2a
5512023-12-14T12:49:26.978Z+ COUNT=1
5522023-12-14T12:49:26.978Z+ [[ 1 -ne 1 ]]
5532023-12-14T12:49:26.978Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-12-14T12:49:26.978ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-12-14T12:49:26.978Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-12-14T12:49:26.978Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-12-14T12:49:27.004Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-12-14T12:49:27.004Z+ [[ -v PXA_START ]]
5592023-12-14T12:49:27.004Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-12-14T12:49:27.004Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-12-14T12:49:27.029Z+ z_scadm dump-state
5622023-12-14T12:49:27.029Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-12-14T12:49:27.054Zlocal v6:
5642023-12-14T12:49:27.054Zlocal v4:
5652023-12-14T12:49:27.054Zrouter v6:
5662023-12-14T12:49:27.054Zrouter v4_idx:
5672023-12-14T12:49:27.054Zrouter v4_routes:
5682023-12-14T12:49:27.054Zresolver v4:
5692023-12-14T12:49:27.054Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-12-14T12:49:27.056Zresolver v6:
5712023-12-14T12:49:27.056Znat v4:
5722023-12-14T12:49:27.056Znat_v6:
5732023-12-14T12:49:27.056Zport_mac:
5742023-12-14T12:49:27.056Zproxy arp:
5752023-12-14T12:49:27.056Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-12-14T12:49:27.056Z+ success 'softnpu zone exists'
5772023-12-14T12:49:27.056Z+ set +x
5782023-12-14T12:49:27.056Zsoftnpu zone exists
5792023-12-14T12:49:27.081Z
5802023-12-14T12:49:27.081Zreal 31.125642279
5812023-12-14T12:49:27.081Zuser 0.722347474
5822023-12-14T12:49:27.081Zsys 1.599771239
5832023-12-14T12:49:27.081Ztrap 0.000094486
5842023-12-14T12:49:27.081Ztflt 0.000194077
5852023-12-14T12:49:27.081Zdflt 0.001354203
5862023-12-14T12:49:27.081Zkflt 0.000000000
5872023-12-14T12:49:27.081Zlock 24:08.344985650
5882023-12-14T12:49:27.081Zslp 1:33.189876631
5892023-12-14T12:49:27.081Zlat 0.057744716
5902023-12-14T12:49:27.081Zstop 0.344854636
5912023-12-14T12:49:27.081Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-12-14T12:49:27.106Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-12-14T12:49:27.107Z+ SILO_NAME=recovery
5942023-12-14T12:49:27.107Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-14T12:49:27.107Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-12-14T12:49:27.107Z+ sed -i~ '
5972023-12-14T12:49:27.107Z /^external_dns_ips/c\
5982023-12-14T12:49:27.107Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-12-14T12:49:27.108Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-12-14T12:49:27.108Z /^first/c\
6012023-12-14T12:49:27.108Zfirst = "10.151.2.164"
6022023-12-14T12:49:27.108Z /^last/c\
6032023-12-14T12:49:27.108Zlast = "10.151.2.173"
6042023-12-14T12:49:27.108Z }
6052023-12-14T12:49:27.108Z /^\[rack_network_config/,/^$/ {
6062023-12-14T12:49:27.108Z /^infra_ip_first/c\
6072023-12-14T12:49:27.108Zinfra_ip_first = "10.151.2.174"
6082023-12-14T12:49:27.108Z /^infra_ip_last/c\
6092023-12-14T12:49:27.108Zinfra_ip_last = "10.151.2.174"
6102023-12-14T12:49:27.108Z }
6112023-12-14T12:49:27.108Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-12-14T12:49:27.108Z /^routes/c\
6132023-12-14T12:49:27.108Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6142023-12-14T12:49:27.108Z /^addresses/c\
6152023-12-14T12:49:27.108Zaddresses = \["10.151.2.174/32"\]
6162023-12-14T12:49:27.108Z }
6172023-12-14T12:49:27.109Z' pkg/config-rss.toml
6182023-12-14T12:49:27.109Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-12-14T12:49:27.109Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-12-14T12:49:27.109Z+++ pkg/config-rss.toml Thu Dec 14 12:49:27 2023
6212023-12-14T12:49:27.109Z@@ -41,7 +41,7 @@
6222023-12-14T12:49:27.109Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-12-14T12:49:27.109Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-12-14T12:49:27.109Z # below.
6252023-12-14T12:49:27.109Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-12-14T12:49:27.109Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6272023-12-14T12:49:27.110Z
6282023-12-14T12:49:27.110Z # Initial TLS certificates for the external API
6292023-12-14T12:49:27.110Z #
6302023-12-14T12:49:27.110Z@@ -77,8 +77,8 @@
6312023-12-14T12:49:27.111Z #
6322023-12-14T12:49:27.111Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-12-14T12:49:27.111Z [[internal_services_ip_pool_ranges]]
6342023-12-14T12:49:27.111Z-first = "192.168.1.20"
6352023-12-14T12:49:27.111Z-last = "192.168.1.29"
6362023-12-14T12:49:27.111Z+first = "10.151.2.164"
6372023-12-14T12:49:27.111Z+last = "10.151.2.173"
6382023-12-14T12:49:27.111Z
6392023-12-14T12:49:27.111Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-12-14T12:49:27.111Z # parameters should be provided to the control plane via wicket, but we need to
6412023-12-14T12:49:27.111Z@@ -92,8 +92,8 @@
6422023-12-14T12:49:27.111Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-12-14T12:49:27.111Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-12-14T12:49:27.111Z # With softnpu, only one address is used.
6452023-12-14T12:49:27.111Z-infra_ip_first = "192.168.1.30"
6462023-12-14T12:49:27.111Z-infra_ip_last = "192.168.1.30"
6472023-12-14T12:49:27.111Z+infra_ip_first = "10.151.2.174"
6482023-12-14T12:49:27.111Z+infra_ip_last = "10.151.2.174"
6492023-12-14T12:49:27.111Z
6502023-12-14T12:49:27.111Z # Configurations for BGP routers to run on the scrimlets.
6512023-12-14T12:49:27.111Z bgp = []
6522023-12-14T12:49:27.111Z@@ -101,9 +101,9 @@
6532023-12-14T12:49:27.111Z # You can configure multiple uplinks by repeating the following stanza
6542023-12-14T12:49:27.111Z [[rack_network_config.ports]]
6552023-12-14T12:49:27.112Z # Routes associated with this port.
6562023-12-14T12:49:27.112Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-12-14T12:49:27.112Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6582023-12-14T12:49:27.112Z # Addresses associated with this port.
6592023-12-14T12:49:27.112Z-addresses = ["192.168.1.30/32"]
6602023-12-14T12:49:27.112Z+addresses = ["10.151.2.174/32"]
6612023-12-14T12:49:27.112Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-12-14T12:49:27.112Z port = "qsfp0"
6632023-12-14T12:49:27.112Z # The speed of this port.
6642023-12-14T12:49:27.112Z+ true
6652023-12-14T12:49:27.112Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-12-14T12:49:27.112Za pkg/config-rss.toml 7K
6672023-12-14T12:49:27.112Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-12-14T12:49:27.137Z+ TLS_NAME=recovery.sys.oxide.test
6692023-12-14T12:49:27.137Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-12-14T12:49:27.384Z....+...+........+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+....+.....+.......+...+..+..........+..+............+..........+.....+.........+.+..................+...+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+.........+.......+......+..+...+.........+.+........+....+............+.........+.....+.......+...+............+........+......+..........+......+........+.+...........+..........+.....+.......+..+................+..+...+....+........+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-12-14T12:49:27.760Z...+.+..+...+.......+...........+.+...+..+.........................+...+........+...+.+...+......+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*............+.....+.+..+...+............+...+...+.+......+.....+.......+...+..+...+.......+..+................+..+..........+..+.+........+..........+.....+.+......+............+........+...+....+..+...+...+....+.....+..........+......+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.................+.................+.+........+...................+...............+............+.....+......+....+...+...+..+.+..............+.+.....+.+...........+...+.+...............+.....+...............+.............+...........+.+........+...+............+.+.....+............+...................+...+...+...............+.........+...+.....+.+..............+.......+........+.+..............+.............+..+.............+........................+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-14T12:49:27.786Z-----
6732023-12-14T12:49:27.786Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-12-14T12:49:27.811Za pkg/initial-tls-cert.pem 2K
6752023-12-14T12:49:27.811Za pkg/initial-tls-key.pem 4K
6762023-12-14T12:49:27.811Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-12-14T12:49:27.811Z+ rmdir pkg
6782023-12-14T12:49:27.811Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-12-14T12:49:27.811Z+ pfexec mkdir -p /usr/oxide
6802023-12-14T12:49:27.811Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6812023-12-14T12:49:28.567Z+ pfexec chmod +x /usr/oxide/oxide
6822023-12-14T12:49:28.592Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6832023-12-14T12:49:46.916Z+ OMICRON_NO_UNINSTALL=1
6842023-12-14T12:49:46.916Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6852023-12-14T12:49:46.941ZDec 14 12:49:47.280 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6862023-12-14T12:49:46.941ZDec 14 12:49:47.282 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6872023-12-14T12:49:46.941ZDec 14 12:49:47.282 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6882023-12-14T12:49:46.941ZDec 14 12:49:47.282 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6892023-12-14T12:49:46.941ZDec 14 12:49:47.282 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6902023-12-14T12:49:46.941ZDec 14 12:49:47.282 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6912023-12-14T12:49:46.941ZDec 14 12:49:47.282 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6922023-12-14T12:49:46.941ZDec 14 12:49:47.283 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6932023-12-14T12:49:46.941ZDec 14 12:49:47.283 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6942023-12-14T12:49:46.941ZDec 14 12:49:47.283 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6952023-12-14T12:49:46.941ZDec 14 12:49:47.283 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6962023-12-14T12:49:46.941ZDec 14 12:49:47.283 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6972023-12-14T12:49:46.941ZDec 14 12:49:47.283 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6982023-12-14T12:49:46.941ZDec 14 12:49:47.283 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6992023-12-14T12:49:46.941ZDec 14 12:49:47.283 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7002023-12-14T12:49:46.941ZDec 14 12:49:47.283 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7012023-12-14T12:49:46.941ZDec 14 12:49:47.283 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
7022023-12-14T12:49:47.152ZDec 14 12:49:47.528 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7032023-12-14T12:49:47.186ZDec 14 12:49:47.563 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7042023-12-14T12:49:47.245ZDec 14 12:49:47.622 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7052023-12-14T12:49:47.373Z
7062023-12-14T12:49:47.398Zreal 0.480121421
7072023-12-14T12:49:47.398Zuser 0.100345836
7082023-12-14T12:49:47.398Zsys 2.024529418
7092023-12-14T12:49:47.398Ztrap 0.000002400
7102023-12-14T12:49:47.399Ztflt 0.000058539
7112023-12-14T12:49:47.399Zdflt 0.001013339
7122023-12-14T12:49:47.400Zkflt 0.000000000
7132023-12-14T12:49:47.400Zlock 43.004606963
7142023-12-14T12:49:47.400Zslp 0.692463227
7152023-12-14T12:49:47.400Zlat 0.003115253
7162023-12-14T12:49:47.400Zstop 0.192239270
7172023-12-14T12:49:47.400Z+ retry=0
7182023-12-14T12:49:47.400Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7192023-12-14T12:49:47.400Z+ [[ 0 -gt 30 ]]
7202023-12-14T12:49:47.400Z+ sleep 1
7212023-12-14T12:49:48.388Z+ retry=1
7222023-12-14T12:49:48.388Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232023-12-14T12:49:48.412Z+ [[ 1 -gt 30 ]]
7242023-12-14T12:49:48.412Z+ sleep 1
7252023-12-14T12:49:49.403Z+ retry=2
7262023-12-14T12:49:49.403Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272023-12-14T12:49:49.428Z+ [[ 2 -gt 30 ]]
7282023-12-14T12:49:49.428Z+ sleep 1
7292023-12-14T12:49:50.417Z+ retry=3
7302023-12-14T12:49:50.417Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312023-12-14T12:49:50.446Z+ [[ 3 -gt 30 ]]
7322023-12-14T12:49:50.446Z+ sleep 1
7332023-12-14T12:49:51.432Z+ retry=4
7342023-12-14T12:49:51.433Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352023-12-14T12:50:16.619Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7362023-12-14T12:50:16.646Zlocal v6:
7372023-12-14T12:50:16.646Zfe80::f89e:b8ff:fe20:ad16
7382023-12-14T12:50:16.646Zfd00:99::1
7392023-12-14T12:50:16.646Zlocal v4:
7402023-12-14T12:50:16.646Z10.151.2.174
7412023-12-14T12:50:16.646Zrouter v6:
7422023-12-14T12:50:16.646Zfd00:1122:3344:1::/64 -> fe80::98ca:82ff:fea4:5ed1 (1)
7432023-12-14T12:50:16.646Zfd00:1122:3344:3::/64 -> fe80::98ca:82ff:fea4:5ed1 (1)
7442023-12-14T12:50:16.646Zfd00:1122:3344:2::/64 -> fe80::98ca:82ff:fea4:5ed1 (1)
7452023-12-14T12:50:16.646Zfd00:1122:3344:101::/64 -> fe80::98ca:82ff:fea4:5ed1 (1)
7462023-12-14T12:50:16.646Zfdb0:18c0:4d0d:a02a::/64 -> fe80::98ca:82ff:fea4:5ed1 (1)
7472023-12-14T12:50:16.646Zrouter v4_idx:
7482023-12-14T12:50:16.646Z0.0.0.0/0 -> 0
7492023-12-14T12:50:16.646Zrouter v4_routes:
7502023-12-14T12:50:16.647Z0 -> 10.151.2.100 (2)
7512023-12-14T12:50:16.647Zresolver v4:
7522023-12-14T12:50:16.647Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7532023-12-14T12:50:16.647Z10.151.2.174 -> 0e:bd:fe:17:06:c2
7542023-12-14T12:50:16.647Zresolver v6:
7552023-12-14T12:50:16.647Zfe80::f89e:b8ff:fe20:ad16 -> fa:9e:b8:20:ad:16
7562023-12-14T12:50:16.647Zfe80::98ca:82ff:fea4:5ed1 -> 9a:ca:82:a4:5e:d1
7572023-12-14T12:50:16.647Znat v4:
7582023-12-14T12:50:16.672Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:af:76
7592023-12-14T12:50:16.672Znat_v6:
7602023-12-14T12:50:16.672Zport_mac:
7612023-12-14T12:50:16.672Z1000: a8:40:25:2d:1c:a2
7622023-12-14T12:50:16.672Z1: a8:40:25:2d:1c:a3
7632023-12-14T12:50:16.672Zproxy arp:
7642023-12-14T12:50:16.672Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7652023-12-14T12:50:16.672Z+ retry=0
7662023-12-14T12:50:16.672Z+++ zoneadm list -n
7672023-12-14T12:50:16.672Z+++ grep oxz_ntp
7682023-12-14T12:50:16.672Z++ pfexec svcs -z oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30 -Hostate oxide/ntp
7692023-12-14T12:50:16.697Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7702023-12-14T12:50:16.697Z++ true
7712023-12-14T12:50:16.697Z+ [[ '' != online ]]
7722023-12-14T12:50:16.697Z+ [[ 0 -gt 60 ]]
7732023-12-14T12:50:16.697Z+ sleep 1
7742023-12-14T12:50:17.680Z+ retry=1
7752023-12-14T12:50:17.705Z+++ zoneadm list -n
7762023-12-14T12:50:17.705Z+++ grep oxz_ntp
7772023-12-14T12:50:17.705Z++ pfexec svcs -z oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30 -Hostate oxide/ntp
7782023-12-14T12:50:17.705Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7792023-12-14T12:50:17.730Z++ true
7802023-12-14T12:50:17.730Z+ [[ '' != online ]]
7812023-12-14T12:50:17.730Z+ [[ 1 -gt 60 ]]
7822023-12-14T12:50:17.730Z+ sleep 1
7832023-12-14T12:50:18.709Z+ retry=2
7842023-12-14T12:50:18.734Z+++ zoneadm list -n
7852023-12-14T12:50:18.734Z+++ grep oxz_ntp
7862023-12-14T12:50:18.734Z++ pfexec svcs -z oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30 -Hostate oxide/ntp
7872023-12-14T12:50:18.759Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7882023-12-14T12:50:18.759Z++ true
7892023-12-14T12:50:18.759Z+ [[ '' != online ]]
7902023-12-14T12:50:18.760Z+ [[ 2 -gt 60 ]]
7912023-12-14T12:50:18.761Z+ sleep 1
7922023-12-14T12:50:19.739Z+ retry=3
7932023-12-14T12:50:19.764Z+++ zoneadm list -n
7942023-12-14T12:50:19.764Z+++ grep oxz_ntp
7952023-12-14T12:50:19.764Z++ pfexec svcs -z oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30 -Hostate oxide/ntp
7962023-12-14T12:50:19.764Z+ [[ disabled != online ]]
7972023-12-14T12:50:19.764Z+ [[ 3 -gt 60 ]]
7982023-12-14T12:50:19.764Z+ sleep 1
7992023-12-14T12:50:20.763Z+ retry=4
8002023-12-14T12:50:20.787Z+++ zoneadm list -n
8012023-12-14T12:50:20.787Z+++ grep oxz_ntp
8022023-12-14T12:50:20.788Z++ pfexec svcs -z oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30 -Hostate oxide/ntp
8032023-12-14T12:50:20.788Z+ [[ disabled != online ]]
8042023-12-14T12:50:20.788Z+ [[ 4 -gt 60 ]]
8052023-12-14T12:50:20.788Z+ sleep 1
8062023-12-14T12:50:21.787Z+ retry=5
8072023-12-14T12:50:21.813Z+++ zoneadm list -n
8082023-12-14T12:50:21.813Z+++ grep oxz_ntp
8092023-12-14T12:50:21.813Z++ pfexec svcs -z oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30 -Hostate oxide/ntp
8102023-12-14T12:50:21.813Z+ [[ disabled != online ]]
8112023-12-14T12:50:21.813Z+ [[ 5 -gt 60 ]]
8122023-12-14T12:50:21.813Z+ sleep 1
8132023-12-14T12:50:22.812Z+ retry=6
8142023-12-14T12:50:22.837Z+++ zoneadm list -n
8152023-12-14T12:50:22.837Z+++ grep oxz_ntp
8162023-12-14T12:50:22.837Z++ pfexec svcs -z oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30 -Hostate oxide/ntp
8172023-12-14T12:50:22.837Z+ [[ disabled != online ]]
8182023-12-14T12:50:22.839Z+ [[ 6 -gt 60 ]]
8192023-12-14T12:50:22.839Z+ sleep 1
8202023-12-14T12:50:23.832Z+ retry=7
8212023-12-14T12:50:23.857Z+++ zoneadm list -n
8222023-12-14T12:50:23.857Z+++ grep oxz_ntp
8232023-12-14T12:50:23.857Z++ pfexec svcs -z oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30 -Hostate oxide/ntp
8242023-12-14T12:50:23.857Z+ [[ disabled != online ]]
8252023-12-14T12:50:23.857Z+ [[ 7 -gt 60 ]]
8262023-12-14T12:50:23.857Z+ sleep 1
8272023-12-14T12:50:24.854Z+ retry=8
8282023-12-14T12:50:24.879Z+++ zoneadm list -n
8292023-12-14T12:50:24.879Z+++ grep oxz_ntp
8302023-12-14T12:50:24.879Z++ pfexec svcs -z oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30 -Hostate oxide/ntp
8312023-12-14T12:50:24.879Z+ [[ disabled != online ]]
8322023-12-14T12:50:24.879Z+ [[ 8 -gt 60 ]]
8332023-12-14T12:50:24.879Z+ sleep 1
8342023-12-14T12:50:25.877Z+ retry=9
8352023-12-14T12:50:25.902Z+++ zoneadm list -n
8362023-12-14T12:50:25.902Z+++ grep oxz_ntp
8372023-12-14T12:50:25.902Z++ pfexec svcs -z oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30 -Hostate oxide/ntp
8382023-12-14T12:50:25.902Z+ [[ disabled != online ]]
8392023-12-14T12:50:25.902Z+ [[ 9 -gt 60 ]]
8402023-12-14T12:50:25.902Z+ sleep 1
 
8412023-12-14T12:50:26.856Zsock console login: Dec 14 12:50:26 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8422023-12-14T12:50:26.903Z+ retry=10
8432023-12-14T12:50:26.927Z+++ zoneadm list -n
8442023-12-14T12:50:26.928Z+++ grep oxz_ntp
8452023-12-14T12:50:26.928Z++ pfexec svcs -z oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30 -Hostate oxide/ntp
8462023-12-14T12:50:26.928Z+ [[ disabled != online ]]
8472023-12-14T12:50:26.928Z+ [[ 10 -gt 60 ]]
8482023-12-14T12:50:26.929Z+ sleep 1
8492023-12-14T12:50:27.923Z+ retry=11
8502023-12-14T12:50:27.948Z+++ zoneadm list -n
8512023-12-14T12:50:27.948Z+++ grep oxz_ntp
8522023-12-14T12:50:27.948Z++ pfexec svcs -z oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30 -Hostate oxide/ntp
8532023-12-14T12:50:27.948Z+ [[ online != online ]]
8542023-12-14T12:50:27.948Z+ echo 'Waited for chrony: 11s'
8552023-12-14T12:50:27.948ZWaited for chrony: 11s
8562023-12-14T12:50:27.948Z+ export RUST_BACKTRACE=1
8572023-12-14T12:50:27.948Z+ RUST_BACKTRACE=1
8582023-12-14T12:50:27.948Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8592023-12-14T12:50:27.948Z++ ./tests/bootstrap
8602023-12-14T12:50:27.972ZUsing external DNS server at 10.151.2.164:53
8612023-12-14T12:50:27.997Z2023-12-14 12:50:28.352578030 UTC: attempting to log into API
8622023-12-14T12:50:43.011Z2023-12-14 12:50:43.140901038 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8632023-12-14T12:50:44.012Z2023-12-14 12:50:44.141787398 UTC: attempting to log into API
8642023-12-14T12:50:59.045Z2023-12-14 12:50:59.174998374 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8652023-12-14T12:51:00.049Z2023-12-14 12:51:00.175660949 UTC: attempting to log into API
8662023-12-14T12:51:15.096Z2023-12-14 12:51:15.226302189 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8672023-12-14T12:51:16.098Z2023-12-14 12:51:16.228328796 UTC: attempting to log into API
8682023-12-14T12:51:31.137Z2023-12-14 12:51:31.264009658 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8692023-12-14T12:51:32.164Z2023-12-14 12:51:32.264831718 UTC: attempting to log into API
8702023-12-14T12:51:47.174Z2023-12-14 12:51:47.280789432 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8712023-12-14T12:51:48.174Z2023-12-14 12:51:48.279066175 UTC: attempting to log into API
 
8722023-12-14T12:51:52.168ZDec 14 12:51:51 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
8732023-12-14T12:51:58.914ZDec 14 12:51:58 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
8742023-12-14T12:52:03.207Z2023-12-14 12:52:03.270520918 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8752023-12-14T12:52:04.208Z2023-12-14 12:52:04.268859062 UTC: attempting to log into API
 
8762023-12-14T12:52:05.957ZDec 14 12:52:05 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
8772023-12-14T12:52:13.532ZDec 14 12:52:12 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
8782023-12-14T12:52:18.810ZDec 14 12:52:18 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
8792023-12-14T12:52:19.241Z2023-12-14 12:52:19.260715903 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8802023-12-14T12:52:20.241Z2023-12-14 12:52:20.259036507 UTC: attempting to log into API
8812023-12-14T12:52:35.273Z2023-12-14 12:52:35.262059041 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8822023-12-14T12:52:36.274Z2023-12-14 12:52:36.263042635 UTC: attempting to log into API
8832023-12-14T12:52:42.069Z2023-12-14 12:52:42.063289675 UTC: login succeeded
8842023-12-14T12:52:42.146Zwaiting for nexus to come up...
8852023-12-14T12:52:42.171Zcreating IP pool... 10.151.2.175 - 10.151.2.227
8862023-12-14T12:52:42.219Zsetting up quotas...
8872023-12-14T12:52:42.270Zensuring datasets are ready...
8882023-12-14T12:52:45.689Zlet's roll.
8892023-12-14T12:52:45.714Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
8902023-12-14T12:52:45.714ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
8912023-12-14T12:52:45.714ZOXIDE_TOKEN=oxide-token-ec35cbb65cfd6cde0cd311aaa74033588ce3d865'
8922023-12-14T12:52:45.714Z++ OXIDE_HOST=https://recovery.sys.oxide.test
8932023-12-14T12:52:45.714Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
8942023-12-14T12:52:45.714Z++ OXIDE_TOKEN=oxide-token-ec35cbb65cfd6cde0cd311aaa74033588ce3d865
8952023-12-14T12:52:45.714Z+ export OXIDE_HOST OXIDE_TOKEN
8962023-12-14T12:52:45.714Z+ retry=0
8972023-12-14T12:52:45.714Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8982023-12-14T12:52:45.786Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
8992023-12-14T12:52:46.343Z{"status":"ok"}success
9002023-12-14T12:52:46.343ZProject {
9012023-12-14T12:52:46.344Z description: "some images",
9022023-12-14T12:52:46.345Z id: 2ef63150-b5d8-4f2e-9360-12ac1dc3a9bf,
9032023-12-14T12:52:46.345Z name: Name(
9042023-12-14T12:52:46.370Z "images",
9052023-12-14T12:52:46.370Z ),
9062023-12-14T12:52:46.370Z time_created: 2023-12-14T12:52:45.860166Z,
9072023-12-14T12:52:46.370Z time_modified: 2023-12-14T12:52:45.860166Z,
9082023-12-14T12:52:46.370Z}
9092023-12-14T12:52:46.370Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
9102023-12-14T12:58:09.273Zdone!
9112023-12-14T12:58:09.376Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
9122023-12-14T12:58:09.476Zsuccess
9132023-12-14T12:58:09.476ZImage {
9142023-12-14T12:58:09.528Z block_size: ByteCount(
9152023-12-14T12:58:09.528Z 512,
9162023-12-14T12:58:09.528Z ),
9172023-12-14T12:58:09.528Z description: "debian 11 original base image",
9182023-12-14T12:58:09.528Z digest: None,
9192023-12-14T12:58:09.528Z id: 8b675305-79fb-4fa9-b10d-90133c01b1df,
9202023-12-14T12:58:09.528Z name: Name(
9212023-12-14T12:58:09.528Z "debian11",
9222023-12-14T12:58:09.528Z ),
9232023-12-14T12:58:09.528Z os: "debian",
9242023-12-14T12:58:09.528Z project_id: None,
9252023-12-14T12:58:09.528Z size: ByteCount(
9262023-12-14T12:58:09.528Z 2147483648,
9272023-12-14T12:58:09.528Z ),
9282023-12-14T12:58:09.528Z time_created: 2023-12-14T12:58:09.209399Z,
9292023-12-14T12:58:09.528Z time_modified: 2023-12-14T12:58:09.460334Z,
9302023-12-14T12:58:09.528Z url: None,
9312023-12-14T12:58:09.529Z version: "11",
9322023-12-14T12:58:09.529Z}
9332023-12-14T12:58:09.529Z+ rm ./tests/bootstrap
9342023-12-14T12:58:09.554Z+ for test_bin in tests/*
9352023-12-14T12:58:09.554Z+ ./tests/bootstrap-eae955b6030e5419
9362023-12-14T12:58:09.554Z
9372023-12-14T12:58:09.555Zrunning 0 tests
9382023-12-14T12:58:09.555Z
9392023-12-14T12:58:09.579Z+ for test_bin in tests/*
9402023-12-14T12:58:09.579Z+ ./tests/end_to_end_tests-114e4ef0b9f9b2b3
9412023-12-14T12:58:09.579Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9422023-12-14T12:58:09.579Z
9432023-12-14T12:58:09.612Z
9442023-12-14T12:58:09.612Zrunning 2 tests
9452023-12-14T12:58:09.836Ztest no_spoof::no_spoof_here ... ok
9462023-12-14T12:58:43.325Ztest instance_launch::instance_launch ... ok
9472023-12-14T12:58:43.326Z
9482023-12-14T12:58:43.326Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 33.71s
9492023-12-14T12:58:43.326Z
9502023-12-14T12:58:43.350Z+ _exit_trap
9512023-12-14T12:58:43.350Z+ local status=0
9522023-12-14T12:58:43.350Z+ [[ 0 -eq 0 ]]
9532023-12-14T12:58:43.350Z+ exit 0
9542023-12-14T12:58:43.350Zprocess exited: duration 595501 ms, exit code 0
 
9552023-12-14T12:58:43.468Zfound 56 output files
9562023-12-14T12:58:43.468Zuploading: /var/svc/log/oxide-sled-agent:default.log (705579 bytes)
9572023-12-14T12:58:45.271Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9582023-12-14T12:58:45.271Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_012e6373-5d13-4772-bd31-14a0202e71a1/root/var/svc/log/oxide-cockroachdb:default.log (5385 bytes)
9592023-12-14T12:58:46.367Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_012e6373-5d13-4772-bd31-14a0202e71a1/root/var/svc/log/oxide-cockroachdb:default.log
9602023-12-14T12:58:46.367Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_28037ff0-f9b8-496e-8178-3313e6b32b6e/root/var/svc/log/oxide-external_dns:default.log (26943 bytes)
9612023-12-14T12:58:47.507Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_28037ff0-f9b8-496e-8178-3313e6b32b6e/root/var/svc/log/oxide-external_dns:default.log
9622023-12-14T12:58:47.507Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_e1332340-23e0-4500-af69-1d831ece5d22/root/var/svc/log/oxide-nexus:default.log (793462 bytes)
9632023-12-14T12:58:49.353Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_e1332340-23e0-4500-af69-1d831ece5d22/root/var/svc/log/oxide-nexus:default.log
9642023-12-14T12:58:49.353Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fcd1e09b-cacc-46ca-af34-7449a971ae2f/root/var/svc/log/oxide-crucible-agent:default.log (24363 bytes)
9652023-12-14T12:58:50.492Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fcd1e09b-cacc-46ca-af34-7449a971ae2f/root/var/svc/log/oxide-crucible-agent:default.log
9662023-12-14T12:58:50.493Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fcd1e09b-cacc-46ca-af34-7449a971ae2f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9672023-12-14T12:58:51.591Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fcd1e09b-cacc-46ca-af34-7449a971ae2f/root/var/svc/log/oxide-crucible-downstairs:default.log
9682023-12-14T12:58:51.591Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fcd1e09b-cacc-46ca-af34-7449a971ae2f/root/var/svc/log/oxide-crucible-downstairs:downstairs-8918216e-b567-49df-b569-b1826c0358ff.log (5535 bytes)
9692023-12-14T12:58:52.687Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fcd1e09b-cacc-46ca-af34-7449a971ae2f/root/var/svc/log/oxide-crucible-downstairs:downstairs-8918216e-b567-49df-b569-b1826c0358ff.log
9702023-12-14T12:58:52.687Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fcd1e09b-cacc-46ca-af34-7449a971ae2f/root/var/svc/log/oxide-crucible-downstairs:downstairs-ff494d16-d1f6-4845-95a3-79c33cb2701c.log (3049 bytes)
9712023-12-14T12:58:53.786Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_fcd1e09b-cacc-46ca-af34-7449a971ae2f/root/var/svc/log/oxide-crucible-downstairs:downstairs-ff494d16-d1f6-4845-95a3-79c33cb2701c.log
9722023-12-14T12:58:53.786Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_c3c11b38-2864-43ca-97b9-02a3fe83f965/root/var/svc/log/oxide-crucible-pantry:default.log (2332 bytes)
9732023-12-14T12:58:54.885Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_c3c11b38-2864-43ca-97b9-02a3fe83f965/root/var/svc/log/oxide-crucible-pantry:default.log
9742023-12-14T12:58:54.885Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_7cdf77db-8c33-4adc-b9b0-629b809e317c/root/var/svc/log/oxide-oximeter:default.log (234052 bytes)
9752023-12-14T12:58:55.069Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_7cdf77db-8c33-4adc-b9b0-629b809e317c/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 234052 -> 238852
9762023-12-14T12:58:56.141Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_7cdf77db-8c33-4adc-b9b0-629b809e317c/root/var/svc/log/oxide-oximeter:default.log
9772023-12-14T12:58:56.142Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_fa345de7-136e-4b8b-88a9-1b76a808db79/root/var/svc/log/oxide-cockroachdb:default.log (5413 bytes)
9782023-12-14T12:58:57.238Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_fa345de7-136e-4b8b-88a9-1b76a808db79/root/var/svc/log/oxide-cockroachdb:default.log
9792023-12-14T12:58:57.238Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_af34766c-c3c1-456b-80b0-575b53687aa6/root/var/svc/log/oxide-clickhouse:default.log (761128 bytes)
9802023-12-14T12:58:57.929Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_af34766c-c3c1-456b-80b0-575b53687aa6/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 761128 -> 775592
9812023-12-14T12:58:59.002Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_af34766c-c3c1-456b-80b0-575b53687aa6/root/var/svc/log/oxide-clickhouse:default.log
9822023-12-14T12:58:59.002Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_75959908-c8f8-4928-b6b8-86c6de2e8b61/root/var/svc/log/oxide-cockroachdb:default.log (5412 bytes)
9832023-12-14T12:59:00.139Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_75959908-c8f8-4928-b6b8-86c6de2e8b61/root/var/svc/log/oxide-cockroachdb:default.log
9842023-12-14T12:59:00.139Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_db12a739-efd3-4194-9fcf-6da16bb3ace2/root/var/svc/log/oxide-crucible-agent:default.log (38512 bytes)
9852023-12-14T12:59:01.278Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_db12a739-efd3-4194-9fcf-6da16bb3ace2/root/var/svc/log/oxide-crucible-agent:default.log
9862023-12-14T12:59:01.279Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_db12a739-efd3-4194-9fcf-6da16bb3ace2/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9872023-12-14T12:59:02.378Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_db12a739-efd3-4194-9fcf-6da16bb3ace2/root/var/svc/log/oxide-crucible-downstairs:default.log
9882023-12-14T12:59:02.378Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_db12a739-efd3-4194-9fcf-6da16bb3ace2/root/var/svc/log/oxide-crucible-downstairs:downstairs-6ea3f129-450c-4dc9-a5b6-520a691c57a6.log (3051 bytes)
9892023-12-14T12:59:03.477Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_db12a739-efd3-4194-9fcf-6da16bb3ace2/root/var/svc/log/oxide-crucible-downstairs:downstairs-6ea3f129-450c-4dc9-a5b6-520a691c57a6.log
9902023-12-14T12:59:03.477Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_db12a739-efd3-4194-9fcf-6da16bb3ace2/root/var/svc/log/oxide-crucible-downstairs:downstairs-7ae6a249-06d6-4021-a2e3-1c7e6cb7e1e4.log (5534 bytes)
9912023-12-14T12:59:04.578Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_db12a739-efd3-4194-9fcf-6da16bb3ace2/root/var/svc/log/oxide-crucible-downstairs:downstairs-7ae6a249-06d6-4021-a2e3-1c7e6cb7e1e4.log
9922023-12-14T12:59:04.578Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_db12a739-efd3-4194-9fcf-6da16bb3ace2/root/var/svc/log/oxide-crucible-downstairs:downstairs-bcb0bcfe-5b8a-4d19-a127-25f0b1546215.log (6433 bytes)
9932023-12-14T12:59:05.677Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_db12a739-efd3-4194-9fcf-6da16bb3ace2/root/var/svc/log/oxide-crucible-downstairs:downstairs-bcb0bcfe-5b8a-4d19-a127-25f0b1546215.log
9942023-12-14T12:59:05.678Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_db12a739-efd3-4194-9fcf-6da16bb3ace2/root/var/svc/log/oxide-crucible-downstairs:snapshot-bcb0bcfe-5b8a-4d19-a127-25f0b1546215-41b140a0-e79e-4028-988c-ae24d62d8118.log (5176 bytes)
9952023-12-14T12:59:06.773Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_db12a739-efd3-4194-9fcf-6da16bb3ace2/root/var/svc/log/oxide-crucible-downstairs:snapshot-bcb0bcfe-5b8a-4d19-a127-25f0b1546215-41b140a0-e79e-4028-988c-ae24d62d8118.log
9962023-12-14T12:59:06.798Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e4f133c0-9cb1-42e3-833a-827732c049fc/root/var/svc/log/oxide-crucible-agent:default.log (23480 bytes)
9972023-12-14T12:59:07.913Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e4f133c0-9cb1-42e3-833a-827732c049fc/root/var/svc/log/oxide-crucible-agent:default.log
9982023-12-14T12:59:07.913Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e4f133c0-9cb1-42e3-833a-827732c049fc/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9992023-12-14T12:59:09.014Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e4f133c0-9cb1-42e3-833a-827732c049fc/root/var/svc/log/oxide-crucible-downstairs:default.log
10002023-12-14T12:59:09.015Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e4f133c0-9cb1-42e3-833a-827732c049fc/root/var/svc/log/oxide-crucible-downstairs:downstairs-8daf16fe-8d8f-4867-ae29-27e146ccba5f.log (2944 bytes)
10012023-12-14T12:59:10.113Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e4f133c0-9cb1-42e3-833a-827732c049fc/root/var/svc/log/oxide-crucible-downstairs:downstairs-8daf16fe-8d8f-4867-ae29-27e146ccba5f.log
10022023-12-14T12:59:10.113Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e4f133c0-9cb1-42e3-833a-827732c049fc/root/var/svc/log/oxide-crucible-downstairs:downstairs-b5a8ae7b-1fcc-4644-8d67-1de084561be2.log (6438 bytes)
10032023-12-14T12:59:11.211Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e4f133c0-9cb1-42e3-833a-827732c049fc/root/var/svc/log/oxide-crucible-downstairs:downstairs-b5a8ae7b-1fcc-4644-8d67-1de084561be2.log
10042023-12-14T12:59:11.212Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e4f133c0-9cb1-42e3-833a-827732c049fc/root/var/svc/log/oxide-crucible-downstairs:snapshot-b5a8ae7b-1fcc-4644-8d67-1de084561be2-41b140a0-e79e-4028-988c-ae24d62d8118.log (5174 bytes)
10052023-12-14T12:59:12.309Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e4f133c0-9cb1-42e3-833a-827732c049fc/root/var/svc/log/oxide-crucible-downstairs:snapshot-b5a8ae7b-1fcc-4644-8d67-1de084561be2-41b140a0-e79e-4028-988c-ae24d62d8118.log
10062023-12-14T12:59:12.309Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_a6263ae8-5d9b-4977-a20b-14b372decf3b/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10072023-12-14T12:59:13.408Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_a6263ae8-5d9b-4977-a20b-14b372decf3b/root/var/svc/log/oxide-crucible-pantry:default.log
10082023-12-14T12:59:13.408Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_350809e7-a0e5-483f-9dfa-b1962f6eccba/root/var/svc/log/oxide-nexus:default.log (35844777 bytes)
10092023-12-14T12:59:25.432Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_350809e7-a0e5-483f-9dfa-b1962f6eccba/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 35844777 -> 35917309
10102023-12-14T12:59:26.504Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_350809e7-a0e5-483f-9dfa-b1962f6eccba/root/var/svc/log/oxide-nexus:default.log
10112023-12-14T12:59:26.504Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6fc75ba6-25d4-41b6-9970-49bca87efc49/root/var/svc/log/oxide-crucible-agent:default.log (17553 bytes)
10122023-12-14T12:59:27.644Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6fc75ba6-25d4-41b6-9970-49bca87efc49/root/var/svc/log/oxide-crucible-agent:default.log
10132023-12-14T12:59:27.644Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6fc75ba6-25d4-41b6-9970-49bca87efc49/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10142023-12-14T12:59:28.742Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6fc75ba6-25d4-41b6-9970-49bca87efc49/root/var/svc/log/oxide-crucible-downstairs:default.log
10152023-12-14T12:59:28.742Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6fc75ba6-25d4-41b6-9970-49bca87efc49/root/var/svc/log/oxide-crucible-downstairs:downstairs-4b99743f-28ea-4b73-8724-3e4fe4905f56.log (6434 bytes)
10162023-12-14T12:59:29.840Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6fc75ba6-25d4-41b6-9970-49bca87efc49/root/var/svc/log/oxide-crucible-downstairs:downstairs-4b99743f-28ea-4b73-8724-3e4fe4905f56.log
10172023-12-14T12:59:29.840Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6fc75ba6-25d4-41b6-9970-49bca87efc49/root/var/svc/log/oxide-crucible-downstairs:snapshot-4b99743f-28ea-4b73-8724-3e4fe4905f56-41b140a0-e79e-4028-988c-ae24d62d8118.log (5177 bytes)
10182023-12-14T12:59:30.936Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6fc75ba6-25d4-41b6-9970-49bca87efc49/root/var/svc/log/oxide-crucible-downstairs:snapshot-4b99743f-28ea-4b73-8724-3e4fe4905f56-41b140a0-e79e-4028-988c-ae24d62d8118.log
10192023-12-14T12:59:30.936Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9a83b406-58fb-4e87-a116-1f2c470b03ce/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10202023-12-14T12:59:32.034Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9a83b406-58fb-4e87-a116-1f2c470b03ce/root/var/svc/log/oxide-crucible-agent:default.log
10212023-12-14T12:59:32.034Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9a83b406-58fb-4e87-a116-1f2c470b03ce/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10222023-12-14T12:59:33.155Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9a83b406-58fb-4e87-a116-1f2c470b03ce/root/var/svc/log/oxide-crucible-downstairs:default.log
10232023-12-14T12:59:33.156Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a0cbafa2-b03b-4975-a26f-43bb52c0796e/root/var/svc/log/oxide-crucible-agent:default.log (31190 bytes)
10242023-12-14T12:59:34.314Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a0cbafa2-b03b-4975-a26f-43bb52c0796e/root/var/svc/log/oxide-crucible-agent:default.log
10252023-12-14T12:59:34.314Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a0cbafa2-b03b-4975-a26f-43bb52c0796e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10262023-12-14T12:59:35.411Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a0cbafa2-b03b-4975-a26f-43bb52c0796e/root/var/svc/log/oxide-crucible-downstairs:default.log
10272023-12-14T12:59:35.411Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a0cbafa2-b03b-4975-a26f-43bb52c0796e/root/var/svc/log/oxide-crucible-downstairs:downstairs-6d80c96d-183c-4714-bb7f-6412f9922c5b.log (5536 bytes)
10282023-12-14T12:59:36.505Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a0cbafa2-b03b-4975-a26f-43bb52c0796e/root/var/svc/log/oxide-crucible-downstairs:downstairs-6d80c96d-183c-4714-bb7f-6412f9922c5b.log
10292023-12-14T12:59:36.505Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a0cbafa2-b03b-4975-a26f-43bb52c0796e/root/var/svc/log/oxide-crucible-downstairs:downstairs-7db95d1c-86db-4898-b740-ec12c343ac09.log (2944 bytes)
10302023-12-14T12:59:37.608Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a0cbafa2-b03b-4975-a26f-43bb52c0796e/root/var/svc/log/oxide-crucible-downstairs:downstairs-7db95d1c-86db-4898-b740-ec12c343ac09.log
10312023-12-14T12:59:37.608Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a0cbafa2-b03b-4975-a26f-43bb52c0796e/root/var/svc/log/oxide-crucible-downstairs:downstairs-c19ffe4d-5803-4417-b52b-24165574917b.log (3051 bytes)
10322023-12-14T12:59:38.703Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a0cbafa2-b03b-4975-a26f-43bb52c0796e/root/var/svc/log/oxide-crucible-downstairs:downstairs-c19ffe4d-5803-4417-b52b-24165574917b.log
10332023-12-14T12:59:38.703Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f1ae9589-293a-4cf1-8126-da6331b2e848/root/var/svc/log/oxide-crucible-agent:default.log (9364 bytes)
10342023-12-14T12:59:39.825Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f1ae9589-293a-4cf1-8126-da6331b2e848/root/var/svc/log/oxide-crucible-agent:default.log
10352023-12-14T12:59:39.825Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f1ae9589-293a-4cf1-8126-da6331b2e848/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10362023-12-14T12:59:40.920Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f1ae9589-293a-4cf1-8126-da6331b2e848/root/var/svc/log/oxide-crucible-downstairs:default.log
10372023-12-14T12:59:40.920Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f1ae9589-293a-4cf1-8126-da6331b2e848/root/var/svc/log/oxide-crucible-downstairs:downstairs-68544e77-958c-43f6-9e2b-65bd1dff4906.log (2945 bytes)
10382023-12-14T12:59:42.019Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f1ae9589-293a-4cf1-8126-da6331b2e848/root/var/svc/log/oxide-crucible-downstairs:downstairs-68544e77-958c-43f6-9e2b-65bd1dff4906.log
10392023-12-14T12:59:42.020Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_92f0b176-ef67-4f20-85d5-fe10a545809f/root/var/svc/log/oxide-external_dns:default.log (26941 bytes)
10402023-12-14T12:59:42.135Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_92f0b176-ef67-4f20-85d5-fe10a545809f/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 26941 -> 30718
10412023-12-14T12:59:43.210Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_92f0b176-ef67-4f20-85d5-fe10a545809f/root/var/svc/log/oxide-external_dns:default.log
10422023-12-14T12:59:43.210Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_e0b667c7-3294-4caa-83d1-4587fea0e314/root/var/svc/log/oxide-internal_dns:default.log (111962 bytes)
10432023-12-14T12:59:43.369Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_e0b667c7-3294-4caa-83d1-4587fea0e314/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 111962 -> 115718
10442023-12-14T12:59:44.440Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_e0b667c7-3294-4caa-83d1-4587fea0e314/root/var/svc/log/oxide-internal_dns:default.log
10452023-12-14T12:59:44.440Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3a3422a9-cef1-4a61-a812-b1da815186b8/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10462023-12-14T12:59:45.572Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3a3422a9-cef1-4a61-a812-b1da815186b8/root/var/svc/log/oxide-crucible-agent:default.log
10472023-12-14T12:59:45.572Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3a3422a9-cef1-4a61-a812-b1da815186b8/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10482023-12-14T12:59:46.672Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3a3422a9-cef1-4a61-a812-b1da815186b8/root/var/svc/log/oxide-crucible-downstairs:default.log
10492023-12-14T12:59:46.672Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_02e67075-447e-4d11-94ae-adecc22a7d6c/root/var/svc/log/oxide-crucible-pantry:default.log (1226908 bytes)
10502023-12-14T12:59:49.261Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_02e67075-447e-4d11-94ae-adecc22a7d6c/root/var/svc/log/oxide-crucible-pantry:default.log
10512023-12-14T12:59:49.285Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_71ba51b3-b363-4d90-b106-e39d21a1717c/root/var/svc/log/oxide-internal_dns:default.log (114811 bytes)
10522023-12-14T12:59:49.420Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_71ba51b3-b363-4d90-b106-e39d21a1717c/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 114811 -> 118563
10532023-12-14T12:59:50.493Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_71ba51b3-b363-4d90-b106-e39d21a1717c/root/var/svc/log/oxide-internal_dns:default.log
10542023-12-14T12:59:50.493Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_67134aae-0878-49d9-8909-8c219f7a48b5/root/var/svc/log/oxide-cockroachdb:default.log (5411 bytes)
10552023-12-14T12:59:51.592Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_67134aae-0878-49d9-8909-8c219f7a48b5/root/var/svc/log/oxide-cockroachdb:default.log
10562023-12-14T12:59:51.592Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6defb047-7c5c-4596-a226-0bf97cb3a6fa/root/var/svc/log/oxide-cockroachdb:default.log (5412 bytes)
10572023-12-14T12:59:52.688Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6defb047-7c5c-4596-a226-0bf97cb3a6fa/root/var/svc/log/oxide-cockroachdb:default.log
10582023-12-14T12:59:52.688Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30/root/var/svc/log/oxide-ntp:default.log (1252 bytes)
10592023-12-14T12:59:53.781Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30/root/var/svc/log/oxide-ntp:default.log
10602023-12-14T12:59:53.817Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_4f316891-3567-4dfc-8498-784cb34a35a3/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10612023-12-14T12:59:54.891Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_4f316891-3567-4dfc-8498-784cb34a35a3/root/var/svc/log/oxide-crucible-agent:default.log
10622023-12-14T12:59:54.891Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_4f316891-3567-4dfc-8498-784cb34a35a3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10632023-12-14T12:59:55.987Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_4f316891-3567-4dfc-8498-784cb34a35a3/root/var/svc/log/oxide-crucible-downstairs:default.log
10642023-12-14T12:59:55.987Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_bffb6824-c26c-497b-8a8b-35381e927553/root/var/svc/log/oxide-internal_dns:default.log (109534 bytes)
10652023-12-14T12:59:56.148Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_bffb6824-c26c-497b-8a8b-35381e927553/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 109534 -> 113290
10662023-12-14T12:59:57.221Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_bffb6824-c26c-497b-8a8b-35381e927553/root/var/svc/log/oxide-internal_dns:default.log
10672023-12-14T12:59:57.221Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_51df8fe6-72d2-4e72-a46d-ba4f88d37d68/root/var/svc/log/oxide-nexus:default.log (904625 bytes)
10682023-12-14T12:59:58.028Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_51df8fe6-72d2-4e72-a46d-ba4f88d37d68/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 904625 -> 982253
10692023-12-14T12:59:59.135Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_51df8fe6-72d2-4e72-a46d-ba4f88d37d68/root/var/svc/log/oxide-nexus:default.log
10702023-12-14T12:59:59.135Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30/root/var/log/chrony/measurements.log (14111 bytes)
10712023-12-14T12:59:59.226Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30/root/var/log/chrony/measurements.log" changed size mid upload: 14111 -> 16166
10722023-12-14T13:00:00.297Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30/root/var/log/chrony/measurements.log
10732023-12-14T13:00:00.298Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30/root/var/log/chrony/statistics.log (9243 bytes)
10742023-12-14T13:00:00.368Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30/root/var/log/chrony/statistics.log" changed size mid upload: 9243 -> 10647
10752023-12-14T13:00:01.439Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30/root/var/log/chrony/statistics.log
10762023-12-14T13:00:01.439Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30/root/var/log/chrony/tracking.log (1981 bytes)
10772023-12-14T13:00:01.490Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30/root/var/log/chrony/tracking.log" changed size mid upload: 1981 -> 2113
10782023-12-14T13:00:02.564Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_286641b0-dd93-4be7-ab5a-d6aef6692e30/root/var/log/chrony/tracking.log