01HGY9MWPK4K7TYSSQGNA9E0K5: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HGY9NH3J8G4QX5RV5YC709TR

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-06T01:36:14.718Zjob dependencies complete; ready to run (waiting for 45 m 49 s)
22023-12-06T01:36:14.746Zjob assigned to worker 01HGYB30RDG7ZPYQAY6AHQBR94 (queued for 0 s)
32023-12-06T01:36:14.915Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-12-06T01:36:14.978Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-12-06T01:36:15.041Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-12-06T01:36:15.098Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-06T01:36:15.153Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-12-06T01:36:15.224ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-06T01:36:15.277Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-06T01:36:15.329ZOxide Helios Version helios-2.0.22318 64-bit
112023-12-06T01:36:15.375ZConfiguring device.
122023-12-06T01:36:15.420Z##################################################
132023-12-06T01:36:15.486Z#### ############# ##
142023-12-06T01:36:15.519Z### ### ############ ##
152023-12-06T01:36:15.564Z## ### # ## ### ## Oxide Computer Company ##
162023-12-06T01:36:15.598Z## ## ## ### # ### ##
172023-12-06T01:36:15.631Z## # ### #### #### This Station Under ##
182023-12-06T01:36:15.664Z### ### #### # ### Computer Control ##
192023-12-06T01:36:15.698Z#### #### ### ## ##
202023-12-06T01:36:15.771Z##################################################
212023-12-06T01:36:15.874ZHostname: buskin
222023-12-06T01:36:15.919ZLoading smf(7) service descriptions:
232023-12-06T01:36:17.308Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-83d41f1b41d1fb89.gz
242023-12-06T01:36:18.521Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-83d41f1b41d1fb89.gz
252023-12-06T01:36:18.521Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-06T01:36:36.973Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-06T01:36:36.974Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8c93950b566420c7.gz
282023-12-06T01:36:55.752Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8c93950b566420c7.gz
292023-12-06T01:36:55.778Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-06T01:36:56.613Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-06T01:36:56.643Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-06T01:36:59.880Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-06T01:36:59.905Zdownloading input: /input/package/work/version.txt
342023-12-06T01:36:59.935Zdownloaded input: /input/package/work/version.txt
352023-12-06T01:36:59.935Zdownloading input: /input/package/work/package.tar.gz
362023-12-06T01:37:14.698Zdownloaded input: /input/package/work/package.tar.gz
372023-12-06T01:37:14.699Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-06T01:37:37.297Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-06T01:37:37.297Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-06T01:37:45.732Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-06T01:37:45.765Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-06T01:38:22.157Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-06T01:38:22.185Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-06T01:38:59.885Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-06T01:38:59.885Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-06T01:39:25.404Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-06T01:39:25.404Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-06T01:39:29.527Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-06T01:39:29.527Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-06T01:39:37.789Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-06T01:39:37.789Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-06T01:39:42.418Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-06T01:39:42.418Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-06T01:39:46.609Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-06T01:39:46.609Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-06T01:39:46.648Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-06T01:39:46.672Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-06T01:39:49.896Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-06T01:39:49.896Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-06T01:39:53.129Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-06T01:39:53.154Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-06T01:40:05.913Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-06T01:40:05.913Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-06T01:40:29.740Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-06T01:40:29.765Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-06T01:40:29.789Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-06T01:40:29.789Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-06T01:40:35.868Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-06T01:40:35.868Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-06T01:40:47.705Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-06T01:40:47.705Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-06T01:41:30.273Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-06T01:41:30.273Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-06T01:41:59.668Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-06T01:41:59.694Zstarting task 0: "setup"
762023-12-06T01:41:59.694Z++ uname -s
772023-12-06T01:41:59.694Z+ kern=SunOS
782023-12-06T01:41:59.694Z+ case "$kern" in
792023-12-06T01:41:59.695Z+ groupadd -g 12345 build
802023-12-06T01:41:59.695Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-06T01:42:01.686Z+ zfs create -o mountpoint=/work rpool/work
822023-12-06T01:42:01.739Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-06T01:42:01.773Z+ home_fs=zfs
842023-12-06T01:42:01.773Z+ [[ zfs == autofs ]]
852023-12-06T01:42:01.773Z+ mkdir -p /home/build
862023-12-06T01:42:01.773Z+ chown build:build /home/build /work
872023-12-06T01:42:03.741Z+ chmod 0700 /home/build /work
882023-12-06T01:42:03.767Zprocess exited: duration 4076 ms, exit code 0
 
892023-12-06T01:42:03.819Zstarting task 1: "authentication"
902023-12-06T01:42:04.272Zprocess exited: duration 453 ms, exit code 0
 
912023-12-06T01:42:04.325Zstarting task 2: "build"
922023-12-06T01:42:04.367Z+ trap _exit_trap EXIT
932023-12-06T01:42:04.367Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-06T01:42:04.531Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-06T01:42:04.531Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-06T01:42:04.531Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-06T01:42:04.531Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-06T01:42:04.532Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-06T01:42:04.532Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-06T01:42:04.532Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-06T01:42:04.532Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-06T01:42:04.532Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-06T01:42:04.532Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-06T01:42:04.562Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-06T01:42:04.562Z+ [[ -d /opt/oxide ]]
1062023-12-06T01:42:04.562Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-06T01:42:04.562Z+ pfexec mkdir /opt/oxide
1082023-12-06T01:42:04.562Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-06T01:42:04.563Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-06T01:42:04.563Z+ cd /opt/oxide-underneath
1112023-12-06T01:42:04.563Z+ pfexec tar ceEp@/f - .
1122023-12-06T01:42:04.563Z+ cd /opt/oxide
1132023-12-06T01:42:04.564Z+ pfexec tar xveEp@/f -
1142023-12-06T01:42:04.648Zx ., 0 bytes, 0 tape blocks
1152023-12-06T01:42:04.648Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-06T01:42:04.648Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-06T01:42:04.648Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-12-06T01:42:04.675Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-06T01:42:04.675Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-06T01:42:04.699Z+ pfexec mkdir /opt/oxide/work
1212023-12-06T01:42:04.700Z+ pfexec chown build:build /opt/oxide/work
1222023-12-06T01:42:04.700Z+ cd /opt/oxide/work
1232023-12-06T01:42:04.700Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-06T01:42:06.222ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-12-06T01:42:06.222Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-12-06T01:42:06.222Zx out/omicron-sled-agent.tar, 111072768 bytes, 216939 tape blocks
1272023-12-06T01:42:06.222Zx out/target/test, 75 bytes, 1 tape blocks
1282023-12-06T01:42:06.222Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-12-06T01:42:06.223Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-12-06T01:42:06.223Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-12-06T01:42:06.223Zx target/release/omicron-package, 42581584 bytes, 83168 tape blocks
1322023-12-06T01:42:06.223Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-12-06T01:42:06.223Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-12-06T01:42:06.223Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-12-06T01:42:06.224Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-12-06T01:42:06.224Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-12-06T01:42:06.224Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-12-06T01:42:06.224Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-12-06T01:42:06.250Z
1402023-12-06T01:42:06.250Zreal 1.533670498
1412023-12-06T01:42:06.250Zuser 1.382448204
1422023-12-06T01:42:06.250Zsys 0.266996478
1432023-12-06T01:42:06.250Ztrap 0.000070170
1442023-12-06T01:42:06.250Ztflt 0.000000000
1452023-12-06T01:42:06.250Zdflt 0.000000000
1462023-12-06T01:42:06.250Zkflt 0.000000000
1472023-12-06T01:42:06.250Zlock 0.000000000
1482023-12-06T01:42:06.250Zslp 1.383331062
1492023-12-06T01:42:06.250Zlat 0.032381903
1502023-12-06T01:42:06.250Zstop 0.000120761
1512023-12-06T01:42:06.250Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-12-06T01:42:07.320Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-12-06T01:42:07.357Z+ mkdir tests
1542023-12-06T01:42:07.358Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-12-06T01:42:07.358Z+ ptime -m gunzip
1562023-12-06T01:42:07.358Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-83d41f1b41d1fb89
1572023-12-06T01:42:07.472Z
1582023-12-06T01:42:07.506Zreal 0.137350886
1592023-12-06T01:42:07.507Zuser 0.126105308
1602023-12-06T01:42:07.507Zsys 0.011016341
1612023-12-06T01:42:07.507Ztrap 0.000014620
1622023-12-06T01:42:07.507Ztflt 0.000000000
1632023-12-06T01:42:07.507Zdflt 0.000000000
1642023-12-06T01:42:07.507Zkflt 0.000000000
1652023-12-06T01:42:07.507Zlock 0.000000000
1662023-12-06T01:42:07.507Zslp 0.000021560
1672023-12-06T01:42:07.507Zlat 0.000019290
1682023-12-06T01:42:07.507Zstop 0.000118372
1692023-12-06T01:42:07.507Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-83d41f1b41d1fb89
1702023-12-06T01:42:07.508Z+ chmod a+x tests/bootstrap-83d41f1b41d1fb89
1712023-12-06T01:42:07.508Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-12-06T01:42:07.508Z+ ptime -m gunzip
1732023-12-06T01:42:07.508Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-12-06T01:42:09.655Z
1752023-12-06T01:42:09.680Zreal 2.174467795
1762023-12-06T01:42:09.681Zuser 2.050525123
1772023-12-06T01:42:09.681Zsys 0.123540944
1782023-12-06T01:42:09.681Ztrap 0.000165873
1792023-12-06T01:42:09.681Ztflt 0.000000000
1802023-12-06T01:42:09.681Zdflt 0.000000000
1812023-12-06T01:42:09.681Zkflt 0.000000000
1822023-12-06T01:42:09.681Zlock 0.000000000
1832023-12-06T01:42:09.681Zslp 0.000020430
1842023-12-06T01:42:09.681Zlat 0.000021800
1852023-12-06T01:42:09.681Zstop 0.000129412
1862023-12-06T01:42:09.681Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-12-06T01:42:09.681Z+ chmod a+x tests/bootstrap
1882023-12-06T01:42:09.681Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-12-06T01:42:09.681Z+ ptime -m gunzip
1902023-12-06T01:42:09.681Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8c93950b566420c7
1912023-12-06T01:42:11.925Z
1922023-12-06T01:42:11.925Zreal 2.250851755
1932023-12-06T01:42:11.962Zuser 2.114289890
1942023-12-06T01:42:11.962Zsys 0.136161456
1952023-12-06T01:42:11.962Ztrap 0.000169714
1962023-12-06T01:42:11.963Ztflt 0.000000000
1972023-12-06T01:42:11.963Zdflt 0.000000000
1982023-12-06T01:42:11.963Zkflt 0.000000000
1992023-12-06T01:42:11.963Zlock 0.000000000
2002023-12-06T01:42:11.963Zslp 0.000019280
2012023-12-06T01:42:11.963Zlat 0.000021380
2022023-12-06T01:42:11.963Zstop 0.000132372
2032023-12-06T01:42:11.964Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8c93950b566420c7
2042023-12-06T01:42:11.964Z+ chmod a+x tests/end_to_end_tests-8c93950b566420c7
2052023-12-06T01:42:11.964Z+ bmat address ls
2062023-12-06T01:42:11.964ZNAME CIDR FIRST LAST COUNT
2072023-12-06T01:42:11.964Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-12-06T01:42:11.964Z++ bmat address ls -f extra -Ho first,last,count
2092023-12-06T01:42:11.964Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-12-06T01:42:11.964Z+ EXTRA_IP_START=10.151.1.164
2112023-12-06T01:42:11.965Z+ EXTRA_IP_END=10.151.1.227
2122023-12-06T01:42:11.965Z+ EXTRA_IP_COUNT=64
2132023-12-06T01:42:11.965Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-12-06T01:42:11.965Z+ EXTRA_IP_BASE=10.151.1
2152023-12-06T01:42:11.965Z+ EXTRA_IP_FOCTET=164
2162023-12-06T01:42:11.965Z+ EXTRA_IP_LOCTET=227
2172023-12-06T01:42:11.966Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-12-06T01:42:11.966Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-12-06T01:42:11.966Z+ DNS_IP1=10.151.1.164
2202023-12-06T01:42:11.966Z+ DNS_IP2=10.151.1.165
2212023-12-06T01:42:11.966Z+ UPLINK_IP=10.151.1.174
2222023-12-06T01:42:11.966Z+ IPPOOL_START=10.151.1.175
2232023-12-06T01:42:11.966Z+ IPPOOL_END=10.151.1.227
2242023-12-06T01:42:11.966Z++ ipadm show-addr -po type,addr
2252023-12-06T01:42:11.967Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-12-06T01:42:11.967Z+ GATEWAY_IP=10.151.1.100
2272023-12-06T01:42:11.967Z+ [[ -n 10.151.1.100 ]]
2282023-12-06T01:42:11.967Z+ ping -s 10.151.1.100 56 1
2292023-12-06T01:42:12.072ZPING 10.151.1.100: 56 data bytes
2302023-12-06T01:42:12.072Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.012 ms
2312023-12-06T01:42:12.072Z
2322023-12-06T01:42:12.073Z----10.151.1.100 PING Statistics----
2332023-12-06T01:42:12.073Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-12-06T01:42:12.073Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2352023-12-06T01:42:12.073Z++ arp -an
2362023-12-06T01:42:12.073Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-12-06T01:42:12.073Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-12-06T01:42:12.073Z+ routeadm -e ipv4-forwarding -u
2392023-12-06T01:42:12.110Z+ PXA_START=10.151.1.164
2402023-12-06T01:42:12.110Z+ PXA_END=10.151.1.227
2412023-12-06T01:42:12.110Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-12-06T01:42:12.110Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-12-06T01:42:12.218Z+ ZPOOL_VDEV_DIR=/scratch
2442023-12-06T01:42:12.218Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-12-06T01:42:12.244Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-12-06T01:42:12.244Z++ cd ./tools
2472023-12-06T01:42:12.244Z++ pwd
2482023-12-06T01:42:12.244Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-12-06T01:42:12.244Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-12-06T01:42:12.245Z+ SOFTNPU_MODE=zone
2512023-12-06T01:42:12.245Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-12-06T01:42:12.245Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-12-06T01:42:12.245Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-12-06T01:42:12.245Z++ dladm show-phys -p -o LINK
2552023-12-06T01:42:12.245Z++ head -1
2562023-12-06T01:42:12.300Z+ PHYSICAL_LINK=igb0
2572023-12-06T01:42:12.300Z+ echo 'Using igb0 as physical link'
2582023-12-06T01:42:12.326ZUsing igb0 as physical link
2592023-12-06T01:42:12.326Z+ ensure_run_as_root
2602023-12-06T01:42:12.326Z++ id -u
2612023-12-06T01:42:12.326Z+ [[ 0 -ne 0 ]]
2622023-12-06T01:42:12.326Z+ ensure_zpools
2632023-12-06T01:42:12.326Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-12-06T01:42:12.326Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-12-06T01:42:12.326Z+ readarray -t ZPOOLS
2662023-12-06T01:42:12.326Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-12-06T01:42:12.326Z++ sed 's/[ ",]//g'
2682023-12-06T01:42:12.326Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-12-06T01:42:12.326Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-12-06T01:42:12.326Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2712023-12-06T01:42:12.326Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2722023-12-06T01:42:12.326ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2732023-12-06T01:42:12.326Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-12-06T01:42:12.354Z0+0 records in
2752023-12-06T01:42:12.354Z0+0 records out
2762023-12-06T01:42:12.354Z0 bytes (0 iB) transferred in 0.000007 secs (0 iB/sec)
2772023-12-06T01:42:12.354Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-12-06T01:42:12.354Z+ set +x
2792023-12-06T01:42:12.354ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-12-06T01:42:12.354Z++ zpool list -o name
2812023-12-06T01:42:12.354Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-12-06T01:42:12.380Z+ [[ -z '' ]]
2832023-12-06T01:42:12.381Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-12-06T01:42:12.412Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-12-06T01:42:12.413Z+ set +x
2862023-12-06T01:42:12.413ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-12-06T01:42:12.413Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-12-06T01:42:12.413Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-12-06T01:42:12.413ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-12-06T01:42:12.413Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-12-06T01:42:12.414Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-12-06T01:42:12.414Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-12-06T01:42:12.414Z0+0 records in
2942023-12-06T01:42:12.414Z0+0 records out
2952023-12-06T01:42:12.414Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-12-06T01:42:12.414Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-12-06T01:42:12.414Z+ set +x
2982023-12-06T01:42:12.414ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-12-06T01:42:12.414Z++ zpool list -o name
3002023-12-06T01:42:12.414Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-12-06T01:42:12.415Z+ [[ -z '' ]]
3022023-12-06T01:42:12.415Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-12-06T01:42:12.443Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-12-06T01:42:12.443Z+ set +x
3052023-12-06T01:42:12.443ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-12-06T01:42:12.443Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-12-06T01:42:12.443ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3082023-12-06T01:42:12.443Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3092023-12-06T01:42:12.443Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-12-06T01:42:12.443Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-12-06T01:42:12.443Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-12-06T01:42:12.443Z0+0 records in
3132023-12-06T01:42:12.443Z0+0 records out
3142023-12-06T01:42:12.443Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3152023-12-06T01:42:12.443Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-12-06T01:42:12.444Z+ set +x
3172023-12-06T01:42:12.444ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-12-06T01:42:12.444Z++ zpool list -o name
3192023-12-06T01:42:12.444Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-12-06T01:42:12.471Z+ [[ -z '' ]]
3212023-12-06T01:42:12.471Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-12-06T01:42:12.512Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-12-06T01:42:12.512Z+ set +x
3242023-12-06T01:42:12.512ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-12-06T01:42:12.512ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3262023-12-06T01:42:12.512Z+ for ZPOOL in "${ZPOOLS[@]}"
3272023-12-06T01:42:12.512Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3282023-12-06T01:42:12.512Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-12-06T01:42:12.512Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-12-06T01:42:12.513Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-12-06T01:42:12.513Z0+0 records in
3322023-12-06T01:42:12.513Z0+0 records out
3332023-12-06T01:42:12.513Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-12-06T01:42:12.513Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-12-06T01:42:12.513Z+ set +x
3362023-12-06T01:42:12.513ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-12-06T01:42:12.514Z++ zpool list -o name
3382023-12-06T01:42:12.514Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-12-06T01:42:12.514Z+ [[ -z '' ]]
3402023-12-06T01:42:12.514Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-12-06T01:42:12.548Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-12-06T01:42:12.548Z+ set +x
3432023-12-06T01:42:12.548ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-12-06T01:42:12.548Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-12-06T01:42:12.548Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-12-06T01:42:12.549ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-12-06T01:42:12.549Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-12-06T01:42:12.549Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-12-06T01:42:12.549Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-12-06T01:42:12.549Z0+0 records in
3512023-12-06T01:42:12.549Z0+0 records out
3522023-12-06T01:42:12.549Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3532023-12-06T01:42:12.549Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-12-06T01:42:12.549Z+ set +x
3552023-12-06T01:42:12.549ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-12-06T01:42:12.549Z++ zpool list -o name
3572023-12-06T01:42:12.549Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-12-06T01:42:12.549Z+ [[ -z '' ]]
3592023-12-06T01:42:12.549Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-12-06T01:42:12.583Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-12-06T01:42:12.583Z+ set +x
3622023-12-06T01:42:12.583ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-12-06T01:42:12.583Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-12-06T01:42:12.584Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-12-06T01:42:12.584ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-12-06T01:42:12.584Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-12-06T01:42:12.584Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-12-06T01:42:12.584Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-12-06T01:42:12.584Z0+0 records in
3702023-12-06T01:42:12.584Z0+0 records out
3712023-12-06T01:42:12.585Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3722023-12-06T01:42:12.585Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-12-06T01:42:12.585Z+ set +x
3742023-12-06T01:42:12.585ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-12-06T01:42:12.585Z++ zpool list -o name
3762023-12-06T01:42:12.585Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-12-06T01:42:12.585Z+ [[ -z '' ]]
3782023-12-06T01:42:12.585Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-12-06T01:42:12.635Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-12-06T01:42:12.636Z+ set +x
3812023-12-06T01:42:12.636ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-12-06T01:42:12.636Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-12-06T01:42:12.636Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-12-06T01:42:12.636ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-12-06T01:42:12.636Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-12-06T01:42:12.636Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-12-06T01:42:12.636Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-12-06T01:42:12.637Z0+0 records in
3892023-12-06T01:42:12.637Z0+0 records out
3902023-12-06T01:42:12.637Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-12-06T01:42:12.637Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-12-06T01:42:12.637Z+ set +x
3932023-12-06T01:42:12.637ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-12-06T01:42:12.637Z++ zpool list -o name
3952023-12-06T01:42:12.637Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-12-06T01:42:12.637Z+ [[ -z '' ]]
3972023-12-06T01:42:12.637Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-12-06T01:42:12.672Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-12-06T01:42:12.672Z+ set +x
4002023-12-06T01:42:12.672ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-12-06T01:42:12.672Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-12-06T01:42:12.673Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-12-06T01:42:12.673ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-12-06T01:42:12.673Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-12-06T01:42:12.673Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-12-06T01:42:12.673Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-12-06T01:42:12.673Z0+0 records in
4082023-12-06T01:42:12.673Z0+0 records out
4092023-12-06T01:42:12.673Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-12-06T01:42:12.673Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-12-06T01:42:12.673Z+ set +x
4122023-12-06T01:42:12.673ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-12-06T01:42:12.673Z++ zpool list -o name
4142023-12-06T01:42:12.673Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-12-06T01:42:12.673Z+ [[ -z '' ]]
4162023-12-06T01:42:12.673Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-12-06T01:42:12.707Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-12-06T01:42:12.707Z+ set +x
4192023-12-06T01:42:12.707ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-12-06T01:42:12.707Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-12-06T01:42:12.707Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-12-06T01:42:12.707ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-12-06T01:42:12.707Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-12-06T01:42:12.708Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-12-06T01:42:12.708Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-12-06T01:42:12.708Z0+0 records in
4272023-12-06T01:42:12.708Z0+0 records out
4282023-12-06T01:42:12.708Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4292023-12-06T01:42:12.708Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-12-06T01:42:12.708Z+ set +x
4312023-12-06T01:42:12.708ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-12-06T01:42:12.708Z++ zpool list -o name
4332023-12-06T01:42:12.708Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-12-06T01:42:12.708Z+ [[ -z '' ]]
4352023-12-06T01:42:12.709Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-12-06T01:42:12.739Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-12-06T01:42:12.739Z+ set +x
4382023-12-06T01:42:12.739ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-12-06T01:42:12.739Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-12-06T01:42:12.739Z+ readarray -t ZPOOLS
4412023-12-06T01:42:12.739Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-12-06T01:42:12.739Z++ sed 's/[ ",]//g'
4432023-12-06T01:42:12.739Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-12-06T01:42:12.740Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-12-06T01:42:12.740ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-12-06T01:42:12.740Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-12-06T01:42:12.740Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-12-06T01:42:12.740Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-12-06T01:42:12.740Z0+0 records in
4502023-12-06T01:42:12.740Z0+0 records out
4512023-12-06T01:42:12.740Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4522023-12-06T01:42:12.740Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-12-06T01:42:12.740Z+ set +x
4542023-12-06T01:42:12.740ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-12-06T01:42:12.772Z++ zpool list -o name
4562023-12-06T01:42:12.772Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-12-06T01:42:12.772Z+ [[ -z '' ]]
4582023-12-06T01:42:12.773Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-12-06T01:42:12.804Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-12-06T01:42:12.804Z+ set +x
4612023-12-06T01:42:12.804ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-12-06T01:42:12.805Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-12-06T01:42:12.805Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-12-06T01:42:12.805ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-12-06T01:42:12.805Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-12-06T01:42:12.805Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-12-06T01:42:12.805Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-12-06T01:42:12.805Z0+0 records in
4692023-12-06T01:42:12.806Z0+0 records out
4702023-12-06T01:42:12.806Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4712023-12-06T01:42:12.806Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-12-06T01:42:12.806Z+ set +x
4732023-12-06T01:42:12.807ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-12-06T01:42:12.807Z++ zpool list -o name
4752023-12-06T01:42:12.807Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-12-06T01:42:12.807Z+ [[ -z '' ]]
4772023-12-06T01:42:12.807Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-12-06T01:42:12.846Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-12-06T01:42:12.846Z+ set +x
4802023-12-06T01:42:12.847ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-12-06T01:42:12.847Z+ [[ zone == \z\o\n\e ]]
4822023-12-06T01:42:12.847Z+ ensure_simulated_links igb0
4832023-12-06T01:42:12.847Z+ local PHYSICAL_LINK=igb0
4842023-12-06T01:42:12.847Z+ INDICES=("0" "1")
4852023-12-06T01:42:12.847Z+ for I in "${INDICES[@]}"
4862023-12-06T01:42:12.847Z++ get_simnet_name_if_exists net0
4872023-12-06T01:42:12.847Z++ dladm show-simnet -p -o LINK net0
4882023-12-06T01:42:12.848Z++ echo ''
4892023-12-06T01:42:12.848Z+ [[ -z '' ]]
4902023-12-06T01:42:12.848Z+ dladm create-simnet -t net0
4912023-12-06T01:42:12.848Z+ dladm create-simnet -t sc0_0
4922023-12-06T01:42:12.848Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-12-06T01:42:12.848Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-12-06T01:42:12.848Z+ success 'Simnet net0/sc0_0 exists'
4952023-12-06T01:42:12.848Z+ set +x
4962023-12-06T01:42:12.848ZSimnet net0/sc0_0 exists
4972023-12-06T01:42:12.848Z+ for I in "${INDICES[@]}"
4982023-12-06T01:42:13.299Z++ get_simnet_name_if_exists net1
4992023-12-06T01:42:13.299Z++ dladm show-simnet -p -o LINK net1
5002023-12-06T01:42:13.299Z++ echo ''
5012023-12-06T01:42:13.300Z+ [[ -z '' ]]
5022023-12-06T01:42:13.300Z+ dladm create-simnet -t net1
5032023-12-06T01:42:13.300Z+ dladm create-simnet -t sc1_0
5042023-12-06T01:42:13.300Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-12-06T01:42:13.301Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-12-06T01:42:13.301Z+ success 'Simnet net1/sc1_0 exists'
5072023-12-06T01:42:13.301Z+ set +x
5082023-12-06T01:42:13.301ZSimnet net1/sc1_0 exists
5092023-12-06T01:42:13.301Z++ get_vnic_name_if_exists sc0_1
5102023-12-06T01:42:13.301Z++ dladm show-vnic -p -o LINK sc0_1
5112023-12-06T01:42:13.301Z++ echo ''
5122023-12-06T01:42:13.302Z+ [[ -z '' ]]
5132023-12-06T01:42:13.302Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-12-06T01:42:13.302Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-12-06T01:42:13.302Z+ success 'Vnic sc0_1 exists'
5162023-12-06T01:42:13.303Z+ set +x
5172023-12-06T01:42:13.303ZVnic sc0_1 exists
5182023-12-06T01:42:13.303Z+ warn_if_no_proxy_arp
5192023-12-06T01:42:13.303Z+ [[ -v PXA_START ]]
5202023-12-06T01:42:13.303Z+ [[ -v PXA_END ]]
5212023-12-06T01:42:13.304Z+ ensure_softnpu_zone
5222023-12-06T01:42:13.304Z+ zoneadm list
5232023-12-06T01:42:13.304Z+ grep -q sidecar_softnpu
5242023-12-06T01:42:13.341Z+ [[ -f out/npuzone/npuzone ]]
5252023-12-06T01:42:13.341Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5262023-12-06T01:42:13.342Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
5272023-12-06T01:42:21.856Zdownloading softnpu ...done
5282023-12-06T01:42:21.884Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5292023-12-06T01:42:28.300Zdownloading asic_program.so ...done
5302023-12-06T01:42:28.324Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-12-06T01:42:35.279Zdownloading scadm ...done
5322023-12-06T01:42:35.308Zcreating port tfportrear0_0 for sc0_0
5332023-12-06T01:42:35.309Zcreating port tfportqsfp0_0 for sc0_1
5342023-12-06T01:42:35.309Zsoftnpu zone setup
5352023-12-06T01:42:35.309Zconfigure zone
5362023-12-06T01:42:35.336Zinstall zone
5372023-12-06T01:42:36.344Zboot zone
5382023-12-06T01:42:39.452Z[sidecar_softnpu] svcs milestone/network
5392023-12-06T01:42:39.480ZSTATE STIME FMRI
5402023-12-06T01:42:39.480Zonline 1:42:37 svc:/milestone/network:default
5412023-12-06T01:42:39.480Z[sidecar_softnpu] /softnpu/npu start
5422023-12-06T01:42:39.506Z
5432023-12-06T01:42:39.531Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-12-06T01:42:39.531Z+ GATEWAY_IP=10.151.1.100
5452023-12-06T01:42:39.531Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-12-06T01:42:39.531ZUsing 10.151.1.100 as gateway ip
5472023-12-06T01:42:39.531Z+ [[ ! -v GATEWAY_MAC ]]
5482023-12-06T01:42:39.531Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-12-06T01:42:39.531Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-12-06T01:42:39.532Z+++ echo 18:c0:4d:0d:9f:b2
5512023-12-06T01:42:39.532Z+ COUNT=1
5522023-12-06T01:42:39.532Z+ [[ 1 -ne 1 ]]
5532023-12-06T01:42:39.532Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-12-06T01:42:39.532ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-12-06T01:42:39.532Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-12-06T01:42:39.532Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-12-06T01:42:39.558Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-12-06T01:42:39.558Z+ [[ -v PXA_START ]]
5592023-12-06T01:42:39.558Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-12-06T01:42:39.558Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-12-06T01:42:39.592Z+ z_scadm dump-state
5622023-12-06T01:42:39.593Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-12-06T01:42:39.635Zlocal v6:
5642023-12-06T01:42:39.636Zlocal v4:
5652023-12-06T01:42:39.636Zrouter v6:
5662023-12-06T01:42:39.636Zrouter v4_idx:
5672023-12-06T01:42:39.636Zrouter v4_routes:
5682023-12-06T01:42:39.636Zresolver v4:
5692023-12-06T01:42:39.636Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-12-06T01:42:39.636Zresolver v6:
5712023-12-06T01:42:39.636Znat v4:
5722023-12-06T01:42:39.637Znat_v6:
5732023-12-06T01:42:39.637Zport_mac:
5742023-12-06T01:42:39.637Zproxy arp:
5752023-12-06T01:42:39.637Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-12-06T01:42:39.637Z+ success 'softnpu zone exists'
5772023-12-06T01:42:39.637Z+ set +x
5782023-12-06T01:42:39.637Zsoftnpu zone exists
5792023-12-06T01:42:39.637Z
5802023-12-06T01:42:39.637Zreal 27.381843183
5812023-12-06T01:42:39.637Zuser 0.692265698
5822023-12-06T01:42:39.637Zsys 1.580825617
5832023-12-06T01:42:39.637Ztrap 0.000082355
5842023-12-06T01:42:39.637Ztflt 0.000147818
5852023-12-06T01:42:39.637Zdflt 0.001119886
5862023-12-06T01:42:39.637Zkflt 0.000000000
5872023-12-06T01:42:39.638Zlock 21:14.196500728
5882023-12-06T01:42:39.638Zslp 1:22.134627249
5892023-12-06T01:42:39.638Zlat 0.074306682
5902023-12-06T01:42:39.638Zstop 0.362646940
5912023-12-06T01:42:39.638Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-12-06T01:42:39.667Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-12-06T01:42:39.667Z+ SILO_NAME=recovery
5942023-12-06T01:42:39.667Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-06T01:42:39.667Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-12-06T01:42:39.667Z+ sed -i~ '
5972023-12-06T01:42:39.667Z /^external_dns_ips/c\
5982023-12-06T01:42:39.667Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-12-06T01:42:39.667Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-12-06T01:42:39.667Z /^first/c\
6012023-12-06T01:42:39.667Zfirst = "10.151.1.164"
6022023-12-06T01:42:39.667Z /^last/c\
6032023-12-06T01:42:39.667Zlast = "10.151.1.173"
6042023-12-06T01:42:39.667Z }
6052023-12-06T01:42:39.667Z /^\[rack_network_config/,/^$/ {
6062023-12-06T01:42:39.667Z /^infra_ip_first/c\
6072023-12-06T01:42:39.667Zinfra_ip_first = "10.151.1.174"
6082023-12-06T01:42:39.667Z /^infra_ip_last/c\
6092023-12-06T01:42:39.667Zinfra_ip_last = "10.151.1.174"
6102023-12-06T01:42:39.667Z }
6112023-12-06T01:42:39.667Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-12-06T01:42:39.667Z /^routes/c\
6132023-12-06T01:42:39.667Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6142023-12-06T01:42:39.668Z /^addresses/c\
6152023-12-06T01:42:39.668Zaddresses = \["10.151.1.174/32"\]
6162023-12-06T01:42:39.668Z }
6172023-12-06T01:42:39.668Z' pkg/config-rss.toml
6182023-12-06T01:42:39.668Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-12-06T01:42:39.668Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-12-06T01:42:39.668Z+++ pkg/config-rss.toml Wed Dec 6 01:42:39 2023
6212023-12-06T01:42:39.668Z@@ -41,7 +41,7 @@
6222023-12-06T01:42:39.668Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-12-06T01:42:39.668Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-12-06T01:42:39.668Z # below.
6252023-12-06T01:42:39.668Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-12-06T01:42:39.668Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6272023-12-06T01:42:39.668Z
6282023-12-06T01:42:39.668Z # Initial TLS certificates for the external API
6292023-12-06T01:42:39.668Z #
6302023-12-06T01:42:39.668Z@@ -77,8 +77,8 @@
6312023-12-06T01:42:39.668Z #
6322023-12-06T01:42:39.668Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-12-06T01:42:39.668Z [[internal_services_ip_pool_ranges]]
6342023-12-06T01:42:39.668Z-first = "192.168.1.20"
6352023-12-06T01:42:39.668Z-last = "192.168.1.29"
6362023-12-06T01:42:39.668Z+first = "10.151.1.164"
6372023-12-06T01:42:39.668Z+last = "10.151.1.173"
6382023-12-06T01:42:39.668Z
6392023-12-06T01:42:39.669Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-12-06T01:42:39.669Z # parameters should be provided to the control plane via wicket, but we need to
6412023-12-06T01:42:39.669Z@@ -92,8 +92,8 @@
6422023-12-06T01:42:39.669Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-12-06T01:42:39.669Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-12-06T01:42:39.669Z # With softnpu, only one address is used.
6452023-12-06T01:42:39.669Z-infra_ip_first = "192.168.1.30"
6462023-12-06T01:42:39.669Z-infra_ip_last = "192.168.1.30"
6472023-12-06T01:42:39.669Z+infra_ip_first = "10.151.1.174"
6482023-12-06T01:42:39.669Z+infra_ip_last = "10.151.1.174"
6492023-12-06T01:42:39.669Z
6502023-12-06T01:42:39.669Z # Configurations for BGP routers to run on the scrimlets.
6512023-12-06T01:42:39.669Z bgp = []
6522023-12-06T01:42:39.669Z@@ -101,9 +101,9 @@
6532023-12-06T01:42:39.669Z # You can configure multiple uplinks by repeating the following stanza
6542023-12-06T01:42:39.669Z [[rack_network_config.ports]]
6552023-12-06T01:42:39.669Z # Routes associated with this port.
6562023-12-06T01:42:39.669Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-12-06T01:42:39.669Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6582023-12-06T01:42:39.669Z # Addresses associated with this port.
6592023-12-06T01:42:39.669Z-addresses = ["192.168.1.30/32"]
6602023-12-06T01:42:39.669Z+addresses = ["10.151.1.174/32"]
6612023-12-06T01:42:39.669Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-12-06T01:42:39.669Z port = "qsfp0"
6632023-12-06T01:42:39.670Z # The speed of this port.
6642023-12-06T01:42:39.670Z+ true
6652023-12-06T01:42:39.670Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-12-06T01:42:39.670Za pkg/config-rss.toml 7K
6672023-12-06T01:42:39.670Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-12-06T01:42:39.670Z+ TLS_NAME=recovery.sys.oxide.test
6692023-12-06T01:42:39.670Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-12-06T01:42:40.283Z.+.....+......+.........+.+...+..+.+.....+.............+..+.......+.....+...+...+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+......+....+..+.........+....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....................+.......+...+...............+.........+...+..+......+.........+............+.+.........+...............+...+........+...+......+...+............+...+.+.........+...+...............+...+............+.......................+..........+..+...+....+...+..+.+...........+...+...+...+.+........+............+.+..............+.....................+....+...........+...+.......+..+.......+..............+....+......+..+.........+.........+..........+......+...............+.....+.+.....+.........+...+.+......+........+.+..............+.............+................................+......+.+...+......+..+.............+.........+..+....+.........+...+.....+....+.........+......+.........+.........+..+...............+...............+.............+.....+.......+...+........+...+....+...+........+.+........+......+..............................+......+.+..................+....................+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-12-06T01:42:42.419Z.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.......+...+..+...+....+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+............+..+................+........................+...........+.......+.....................+......+.....+.........+................+.....+.........+....+...+............+..+...+...............+......+......+.+..+.+...............+.........+.................+......+..........+.........+...+.....+...+...+.......+..+..........+...+..............+.+.......................+....+...........+.......+.....+....+........+.+......+........+.......+...............+.....+...+...+...+.+...+.....+......+.......+.................................+..+...+.+......+...+.....+.......+...+.....+.+......+..+......+............+....+...+...............+.....+............+....+..+............+...+............+......+.........+.......+.....+......+...+.+...........+.......+...+.................+...........................+...+................................................+...+..........+...........+...+...+.......+...+........+.+.....+...+.+..........................+......+....+.....+.+...+............+............+.....+.+..+.......+.................+.+..+......+.+........+...............+.............+...............+..+............+..........+...+...+..+.+.....+......+.+.........+.....+.........+.+..............+....+......+..+.......+...+..+..........+............+...............+.....+.........+.........+.........................+......+...........+....+.........+..+...+................+......+...+.....+.+.....+...............+....+.....+..........+...+...+.....+.....................+.............+...+..+.+.........+.................+....+..+.......+.....+.......+.........+..+...+.........+..........+..+..........+........+...+............+......+.+......+...+.................+.........+.+...............+.................+.............+.....+...+...............+......+....+......+...+..+.+..+......+.......+...+.....+.......+...+..+...............+...+.+....................+...+............+.+.................+.......+...+..+.........+......+......+....+...+........+..........+..+.......+...............+..............+.+...+........+..........+.....+...+......+...+.......+...+.....+.+......+......+............+......+...........+....+.........+..+...+..................+...+..........+..+..................+....+...............+...+..............+..........+...+........+...........................+.+...........+...+...+...+....+...+..+......+.+.....+.+.........+......+..+...+.........+.+...............+.....+...+......+.+...+..............+.....................+......+....+......+...........+...................+...+..+....+............+..+..................+...+.......+............+..................+..+...+.......+...........+......+....+...........+.+.....+.......+.....+.....................+......+...............+......+...+...............+......+.............+..............................+............+...+..+................+..+............+...+......+.+...........+.......+...+..+.......+.....+..........+......+...+.....+....+..+....+...............+.....+....+...+...............+...+..............+.......+.....+......+....+...+........+....+........+.+...........+.+..+......+....+...+..+...+.......+...+...+............+.....+....+...............+...+.....+.+..+.........+.+........+.+......+.....+...+......+...+..........+...+...+..........................+.........+........................+......................+..............+.+..............+................+.....+...+.+......+......+..+...............+......+......+....+....................+.+.......................+...+.......+...........+.+......+.........+.....+.+.................+............+...+............+....+...........+...+.+..+....+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-06T01:42:42.444Z-----
6732023-12-06T01:42:42.444Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-12-06T01:42:42.470Za pkg/initial-tls-cert.pem 2K
6752023-12-06T01:42:42.470Za pkg/initial-tls-key.pem 4K
6762023-12-06T01:42:42.470Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-12-06T01:42:42.470Z+ rmdir pkg
6782023-12-06T01:42:42.470Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-12-06T01:42:42.470Z+ pfexec mkdir -p /usr/oxide
6802023-12-06T01:42:42.470Z+ pfexec rm -f /usr/oxide/tcpproxy
6812023-12-06T01:42:42.471Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6822023-12-06T01:42:42.720Z+ pfexec chmod +x /usr/oxide/tcpproxy
6832023-12-06T01:42:42.746Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6842023-12-06T01:42:42.746Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6852023-12-06T01:42:42.746Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6862023-12-06T01:42:42.842Z+ OMICRON_NO_UNINSTALL=1
6872023-12-06T01:42:42.842Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6882023-12-06T01:42:42.868ZDec 06 01:42:42.849 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6892023-12-06T01:42:42.868ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6902023-12-06T01:42:42.868ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6912023-12-06T01:42:42.868ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6922023-12-06T01:42:42.868ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6932023-12-06T01:42:42.868ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6942023-12-06T01:42:42.868ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6952023-12-06T01:42:42.868ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6962023-12-06T01:42:42.868ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6972023-12-06T01:42:42.868ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6982023-12-06T01:42:42.868ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6992023-12-06T01:42:42.868ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7002023-12-06T01:42:42.868ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7012023-12-06T01:42:42.869ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7022023-12-06T01:42:42.869ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7032023-12-06T01:42:42.869ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7042023-12-06T01:42:42.869ZDec 06 01:42:42.851 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7052023-12-06T01:42:43.093ZDec 06 01:42:43.086 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-12-06T01:42:43.128ZDec 06 01:42:43.121 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-12-06T01:42:43.206ZDec 06 01:42:43.180 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-12-06T01:42:43.316Z
7092023-12-06T01:42:43.342Zreal 0.469747183
7102023-12-06T01:42:43.342Zuser 0.098380871
7112023-12-06T01:42:43.342Zsys 1.885436020
7122023-12-06T01:42:43.342Ztrap 0.000013469
7132023-12-06T01:42:43.342Ztflt 0.000024390
7142023-12-06T01:42:43.342Zdflt 0.000278647
7152023-12-06T01:42:43.342Zkflt 0.000000000
7162023-12-06T01:42:43.342Zlock 42.191630387
7172023-12-06T01:42:43.342Zslp 0.680374173
7182023-12-06T01:42:43.342Zlat 0.003056447
7192023-12-06T01:42:43.342Zstop 0.196288838
7202023-12-06T01:42:43.342Z+ retry=0
7212023-12-06T01:42:43.342Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-12-06T01:42:43.342Z+ [[ 0 -gt 30 ]]
7232023-12-06T01:42:43.342Z+ sleep 1
7242023-12-06T01:42:44.332Z+ retry=1
7252023-12-06T01:42:44.332Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-12-06T01:42:44.357Z+ [[ 1 -gt 30 ]]
7272023-12-06T01:42:44.357Z+ sleep 1
7282023-12-06T01:42:45.345Z+ retry=2
7292023-12-06T01:42:45.345Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-12-06T01:42:45.370Z+ [[ 2 -gt 30 ]]
7312023-12-06T01:42:45.371Z+ sleep 1
7322023-12-06T01:42:46.375Z+ retry=3
7332023-12-06T01:42:46.376Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-12-06T01:42:46.417Z+ [[ 3 -gt 30 ]]
7352023-12-06T01:42:46.417Z+ sleep 1
7362023-12-06T01:42:47.400Z+ retry=4
7372023-12-06T01:42:47.400Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-12-06T01:43:12.546Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7392023-12-06T01:43:12.574Zlocal v6:
7402023-12-06T01:43:12.574Zfd00:99::1
7412023-12-06T01:43:12.574Zfe80::c8f8:6cff:fe4e:82cb
7422023-12-06T01:43:12.575Zlocal v4:
7432023-12-06T01:43:12.575Z10.151.1.174
7442023-12-06T01:43:12.575Zrouter v6:
7452023-12-06T01:43:12.575Zfd00:1122:3344:3::/64 -> fe80::e0e4:f5ff:fe59:673f (1)
7462023-12-06T01:43:12.575Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::e0e4:f5ff:fe59:673f (1)
7472023-12-06T01:43:12.575Zfd00:1122:3344:1::/64 -> fe80::e0e4:f5ff:fe59:673f (1)
7482023-12-06T01:43:12.575Zfd00:1122:3344:101::/64 -> fe80::e0e4:f5ff:fe59:673f (1)
7492023-12-06T01:43:12.575Zfd00:1122:3344:2::/64 -> fe80::e0e4:f5ff:fe59:673f (1)
7502023-12-06T01:43:12.575Zrouter v4_idx:
7512023-12-06T01:43:12.575Z0.0.0.0/0 -> 0
7522023-12-06T01:43:12.575Zrouter v4_routes:
7532023-12-06T01:43:12.600Z0 -> 10.151.1.100 (2)
7542023-12-06T01:43:12.601Zresolver v4:
7552023-12-06T01:43:12.601Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7562023-12-06T01:43:12.601Z10.151.1.174 -> 0e:b8:88:9b:08:55
7572023-12-06T01:43:12.601Zresolver v6:
7582023-12-06T01:43:12.601Zfe80::c8f8:6cff:fe4e:82cb -> ca:f8:6c:4e:82:cb
7592023-12-06T01:43:12.601Zfe80::e0e4:f5ff:fe59:673f -> e2:e4:f5:59:67:3f
7602023-12-06T01:43:12.601Znat v4:
7612023-12-06T01:43:12.601Z10.151.1.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:d5:27
7622023-12-06T01:43:12.601Znat_v6:
7632023-12-06T01:43:12.601Zport_mac:
7642023-12-06T01:43:12.601Z1: a8:40:25:17:57:71
7652023-12-06T01:43:12.601Z1000: a8:40:25:17:57:70
7662023-12-06T01:43:12.601Zproxy arp:
7672023-12-06T01:43:12.601Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7682023-12-06T01:43:12.601Z+ retry=0
7692023-12-06T01:43:12.601Z+++ zoneadm list -n
7702023-12-06T01:43:12.601Z+++ grep oxz_ntp
7712023-12-06T01:43:12.601Z++ pfexec svcs -z oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5 -Hostate oxide/ntp
7722023-12-06T01:43:12.626Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7732023-12-06T01:43:12.626Z++ true
7742023-12-06T01:43:12.627Z+ [[ '' != online ]]
7752023-12-06T01:43:12.627Z+ [[ 0 -gt 60 ]]
7762023-12-06T01:43:12.627Z+ sleep 1
7772023-12-06T01:43:13.611Z+ retry=1
7782023-12-06T01:43:13.636Z+++ zoneadm list -n
7792023-12-06T01:43:13.636Z+++ grep oxz_ntp
7802023-12-06T01:43:13.636Z++ pfexec svcs -z oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5 -Hostate oxide/ntp
7812023-12-06T01:43:13.636Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7822023-12-06T01:43:13.636Z++ true
7832023-12-06T01:43:13.637Z+ [[ '' != online ]]
7842023-12-06T01:43:13.637Z+ [[ 1 -gt 60 ]]
7852023-12-06T01:43:13.637Z+ sleep 1
7862023-12-06T01:43:14.638Z+ retry=2
7872023-12-06T01:43:14.663Z+++ zoneadm list -n
7882023-12-06T01:43:14.663Z+++ grep oxz_ntp
7892023-12-06T01:43:14.663Z++ pfexec svcs -z oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5 -Hostate oxide/ntp
7902023-12-06T01:43:14.663Z+ [[ disabled != online ]]
7912023-12-06T01:43:14.664Z+ [[ 2 -gt 60 ]]
7922023-12-06T01:43:14.664Z+ sleep 1
7932023-12-06T01:43:15.662Z+ retry=3
7942023-12-06T01:43:15.687Z+++ zoneadm list -n
7952023-12-06T01:43:15.687Z+++ grep oxz_ntp
7962023-12-06T01:43:15.687Z++ pfexec svcs -z oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5 -Hostate oxide/ntp
7972023-12-06T01:43:15.687Z+ [[ disabled != online ]]
7982023-12-06T01:43:15.687Z+ [[ 3 -gt 60 ]]
7992023-12-06T01:43:15.687Z+ sleep 1
8002023-12-06T01:43:16.690Z+ retry=4
8012023-12-06T01:43:17.125Z+++ zoneadm list -n
8022023-12-06T01:43:17.125Z+++ grep oxz_ntp
8032023-12-06T01:43:17.125Z++ pfexec svcs -z oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5 -Hostate oxide/ntp
8042023-12-06T01:43:17.125Z+ [[ disabled != online ]]
8052023-12-06T01:43:17.125Z+ [[ 4 -gt 60 ]]
8062023-12-06T01:43:17.125Z+ sleep 1
8072023-12-06T01:43:17.706Z+ retry=5
8082023-12-06T01:43:17.732Z+++ zoneadm list -n
8092023-12-06T01:43:17.732Z+++ grep oxz_ntp
8102023-12-06T01:43:17.732Z++ pfexec svcs -z oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5 -Hostate oxide/ntp
8112023-12-06T01:43:17.732Z+ [[ disabled != online ]]
8122023-12-06T01:43:17.733Z+ [[ 5 -gt 60 ]]
8132023-12-06T01:43:17.733Z+ sleep 1
8142023-12-06T01:43:18.729Z+ retry=6
8152023-12-06T01:43:18.755Z+++ zoneadm list -n
8162023-12-06T01:43:18.755Z+++ grep oxz_ntp
8172023-12-06T01:43:18.755Z++ pfexec svcs -z oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5 -Hostate oxide/ntp
8182023-12-06T01:43:18.755Z+ [[ disabled != online ]]
8192023-12-06T01:43:18.755Z+ [[ 6 -gt 60 ]]
8202023-12-06T01:43:18.755Z+ sleep 1
8212023-12-06T01:43:19.748Z+ retry=7
8222023-12-06T01:43:19.773Z+++ zoneadm list -n
8232023-12-06T01:43:19.774Z+++ grep oxz_ntp
8242023-12-06T01:43:19.774Z++ pfexec svcs -z oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5 -Hostate oxide/ntp
8252023-12-06T01:43:19.774Z+ [[ disabled != online ]]
8262023-12-06T01:43:19.774Z+ [[ 7 -gt 60 ]]
8272023-12-06T01:43:19.774Z+ sleep 1
8282023-12-06T01:43:20.774Z+ retry=8
8292023-12-06T01:43:20.803Z+++ zoneadm list -n
8302023-12-06T01:43:20.803Z+++ grep oxz_ntp
8312023-12-06T01:43:20.803Z++ pfexec svcs -z oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5 -Hostate oxide/ntp
8322023-12-06T01:43:20.803Z+ [[ disabled != online ]]
8332023-12-06T01:43:20.803Z+ [[ 8 -gt 60 ]]
8342023-12-06T01:43:20.803Z+ sleep 1
 
8352023-12-06T01:43:21.622Zbuskin console login: Dec 6 01:43:21 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8362023-12-06T01:43:21.793Z+ retry=9
8372023-12-06T01:43:21.818Z+++ zoneadm list -n
8382023-12-06T01:43:21.818Z+++ grep oxz_ntp
8392023-12-06T01:43:21.818Z++ pfexec svcs -z oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5 -Hostate oxide/ntp
8402023-12-06T01:43:21.818Z+ [[ disabled != online ]]
8412023-12-06T01:43:21.818Z+ [[ 9 -gt 60 ]]
8422023-12-06T01:43:21.818Z+ sleep 1
8432023-12-06T01:43:22.817Z+ retry=10
8442023-12-06T01:43:22.854Z+++ zoneadm list -n
8452023-12-06T01:43:22.854Z+++ grep oxz_ntp
8462023-12-06T01:43:22.855Z++ pfexec svcs -z oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5 -Hostate oxide/ntp
8472023-12-06T01:43:22.855Z+ [[ online != online ]]
8482023-12-06T01:43:22.855Z+ echo 'Waited for chrony: 10s'
8492023-12-06T01:43:22.855ZWaited for chrony: 10s
8502023-12-06T01:43:22.855Z+ export RUST_BACKTRACE=1
8512023-12-06T01:43:22.855Z+ RUST_BACKTRACE=1
8522023-12-06T01:43:22.855Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8532023-12-06T01:43:22.855Z+ ./tests/bootstrap
8542023-12-06T01:43:22.880ZUsing external DNS server at 10.151.1.164:53
8552023-12-06T01:43:22.880Z2023-12-06 01:43:22.858172332 UTC: attempting to log into API
8562023-12-06T01:43:37.902Z2023-12-06 01:43:37.890506328 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8572023-12-06T01:43:38.904Z2023-12-06 01:43:38.891332760 UTC: attempting to log into API
8582023-12-06T01:43:53.936Z2023-12-06 01:43:53.923581186 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8592023-12-06T01:43:54.936Z2023-12-06 01:43:54.924490310 UTC: attempting to log into API
8602023-12-06T01:44:09.969Z2023-12-06 01:44:09.956559672 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8612023-12-06T01:44:10.970Z2023-12-06 01:44:10.957450025 UTC: attempting to log into API
8622023-12-06T01:44:26.006Z2023-12-06 01:44:25.990027708 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8632023-12-06T01:44:27.003Z2023-12-06 01:44:26.990868770 UTC: attempting to log into API
8642023-12-06T01:44:42.048Z2023-12-06 01:44:42.023010048 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8652023-12-06T01:44:43.036Z2023-12-06 01:44:43.024248453 UTC: attempting to log into API
8662023-12-06T01:44:58.075Z2023-12-06 01:44:58.059739618 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8672023-12-06T01:44:59.072Z2023-12-06 01:44:59.060512729 UTC: attempting to log into API
8682023-12-06T01:45:14.120Z2023-12-06 01:45:14.108989094 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8692023-12-06T01:45:15.121Z2023-12-06 01:45:15.110009430 UTC: attempting to log into API
8702023-12-06T01:45:30.159Z2023-12-06 01:45:30.143349557 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8712023-12-06T01:45:31.168Z2023-12-06 01:45:31.144017295 UTC: attempting to log into API
8722023-12-06T01:45:46.213Z2023-12-06 01:45:46.197522327 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
8732023-12-06T01:45:46.983ZDec 6 01:45:46 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
8742023-12-06T01:45:47.210Z2023-12-06 01:45:47.198682796 UTC: attempting to log into API
 
8752023-12-06T01:46:04.576ZDec 6 01:45:52 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
8762023-12-06T01:46:04.912ZDec 6 01:46:00 bukin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
8772023-12-06T01:46:05.208Z2023-12-06 01:46:02.230788702 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8782023-12-06T01:46:05.208Z2023-12-06 01:46:03.231892030 UTC: attempting to log into API
 
8792023-12-06T01:46:07.619ZDec 6 01:46:07 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
8802023-12-06T01:46:14.651ZDec 6 01:46:14 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8812023-12-06T01:46:18.281Z2023-12-06 01:46:18.263908322 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8822023-12-06T01:46:19.280Z2023-12-06 01:46:19.265179633 UTC: attempting to log into API
8832023-12-06T01:46:34.311Z2023-12-06 01:46:34.297584912 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8842023-12-06T01:46:35.311Z2023-12-06 01:46:35.299416395 UTC: attempting to log into API
8852023-12-06T01:46:36.081Z2023-12-06 01:46:36.064737311 UTC: login succeeded
8862023-12-06T01:46:36.122Zwaiting for nexus to come up...
8872023-12-06T01:46:36.195Zcreating IP pool... 10.151.1.175 - 10.151.1.227
8882023-12-06T01:46:36.243Zensuring datasets are ready...
8892023-12-06T01:46:38.539Zlet's roll.
8902023-12-06T01:46:38.565Z+ rm ./tests/bootstrap
8912023-12-06T01:46:38.635Z+ for test_bin in tests/*
8922023-12-06T01:46:38.635Z+ ./tests/bootstrap-83d41f1b41d1fb89
8932023-12-06T01:46:38.663Z
8942023-12-06T01:46:38.663Zrunning 0 tests
8952023-12-06T01:46:38.663Z
8962023-12-06T01:46:38.663Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
8972023-12-06T01:46:38.664Z
8982023-12-06T01:46:38.664Z+ for test_bin in tests/*
8992023-12-06T01:46:38.664Z+ ./tests/end_to_end_tests-8c93950b566420c7
9002023-12-06T01:46:38.664Z
9012023-12-06T01:46:38.664Zrunning 2 tests
9022023-12-06T01:46:38.867Ztest no_spoof::no_spoof_here ... ok
9032023-12-06T01:47:38.655Ztest instance_launch::instance_launch has been running for over 60 seconds
9042023-12-06T01:49:38.678Ztest instance_launch::instance_launch ... ok
9052023-12-06T01:49:38.678Z
9062023-12-06T01:49:38.678Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 178.74s
9072023-12-06T01:49:38.678Z
9082023-12-06T01:49:38.678Z+ _exit_trap
9092023-12-06T01:49:38.678Z+ local status=0
9102023-12-06T01:49:38.679Z+ [[ 0 -eq 0 ]]
9112023-12-06T01:49:38.679Z+ exit 0
9122023-12-06T01:49:38.679Zprocess exited: duration 453077 ms, exit code 0
 
9132023-12-06T01:49:38.795Zfound 56 output files
9142023-12-06T01:49:38.795Zuploading: /var/svc/log/oxide-sled-agent:default.log (677630 bytes)
9152023-12-06T01:49:39.133Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 677630 -> 678030
9162023-12-06T01:49:40.217Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9172023-12-06T01:49:40.217Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7385065-ab98-4fbd-bd59-6646f01e2559/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
9182023-12-06T01:49:41.351Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7385065-ab98-4fbd-bd59-6646f01e2559/root/var/svc/log/oxide-crucible-agent:default.log
9192023-12-06T01:49:41.351Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7385065-ab98-4fbd-bd59-6646f01e2559/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9202023-12-06T01:49:42.486Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7385065-ab98-4fbd-bd59-6646f01e2559/root/var/svc/log/oxide-crucible-downstairs:default.log
9212023-12-06T01:49:42.487Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7385065-ab98-4fbd-bd59-6646f01e2559/root/var/svc/log/oxide-crucible-downstairs:downstairs-55033cef-a64d-4be5-b5f9-ab799ab5a4af.log (5536 bytes)
9222023-12-06T01:49:43.597Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7385065-ab98-4fbd-bd59-6646f01e2559/root/var/svc/log/oxide-crucible-downstairs:downstairs-55033cef-a64d-4be5-b5f9-ab799ab5a4af.log
9232023-12-06T01:49:43.597Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_3e960819-a4b9-40c0-84e7-36c2a42cc1b9/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9242023-12-06T01:49:44.767Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_3e960819-a4b9-40c0-84e7-36c2a42cc1b9/root/var/svc/log/oxide-crucible-pantry:default.log
9252023-12-06T01:49:44.767Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_eee6672e-f908-46d8-ac26-f42e8e8c30b1/root/var/svc/log/oxide-clickhouse:default.log (693856 bytes)
9262023-12-06T01:49:45.129Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_eee6672e-f908-46d8-ac26-f42e8e8c30b1/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 693856 -> 715112
9272023-12-06T01:49:46.225Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_eee6672e-f908-46d8-ac26-f42e8e8c30b1/root/var/svc/log/oxide-clickhouse:default.log
9282023-12-06T01:49:46.225Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_4670e6d0-1b8d-40f9-879c-d59c3cfdd99f/root/var/svc/log/oxide-cockroachdb:default.log (860 bytes)
9292023-12-06T01:49:47.365Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_4670e6d0-1b8d-40f9-879c-d59c3cfdd99f/root/var/svc/log/oxide-cockroachdb:default.log
9302023-12-06T01:49:47.365Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_4670e6d0-1b8d-40f9-879c-d59c3cfdd99f/root/var/svc/log/oxide-cockroachdb:default.log.0 (4508 bytes)
9312023-12-06T01:49:52.533Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_4670e6d0-1b8d-40f9-879c-d59c3cfdd99f/root/var/svc/log/oxide-cockroachdb:default.log.0
9322023-12-06T01:49:52.533Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_bc420a00-86c4-4c4f-b6b9-be2069fb1960/root/var/svc/log/oxide-cockroachdb:default.log (1815 bytes)
9332023-12-06T01:49:53.642Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_bc420a00-86c4-4c4f-b6b9-be2069fb1960/root/var/svc/log/oxide-cockroachdb:default.log
9342023-12-06T01:49:53.643Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_bc420a00-86c4-4c4f-b6b9-be2069fb1960/root/var/svc/log/oxide-cockroachdb:default.log.0 (3554 bytes)
9352023-12-06T01:49:54.748Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_bc420a00-86c4-4c4f-b6b9-be2069fb1960/root/var/svc/log/oxide-cockroachdb:default.log.0
9362023-12-06T01:49:54.748Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_69a29de5-89ac-4785-b2ed-5aeea391c4bc/root/var/svc/log/oxide-internal_dns:default.log (93563 bytes)
9372023-12-06T01:49:56.123Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_69a29de5-89ac-4785-b2ed-5aeea391c4bc/root/var/svc/log/oxide-internal_dns:default.log
9382023-12-06T01:49:56.124Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_69a29de5-89ac-4785-b2ed-5aeea391c4bc/root/var/svc/log/oxide-internal_dns:default.log.0 (17697 bytes)
9392023-12-06T01:49:57.315Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_69a29de5-89ac-4785-b2ed-5aeea391c4bc/root/var/svc/log/oxide-internal_dns:default.log.0
9402023-12-06T01:49:57.316Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_de9d7e4a-779d-460b-a122-0fc475cba6ab/root/var/svc/log/oxide-internal_dns:default.log (93577 bytes)
9412023-12-06T01:49:58.713Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_de9d7e4a-779d-460b-a122-0fc475cba6ab/root/var/svc/log/oxide-internal_dns:default.log
9422023-12-06T01:49:58.713Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_de9d7e4a-779d-460b-a122-0fc475cba6ab/root/var/svc/log/oxide-internal_dns:default.log.0 (19318 bytes)
9432023-12-06T01:49:59.892Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_de9d7e4a-779d-460b-a122-0fc475cba6ab/root/var/svc/log/oxide-internal_dns:default.log.0
9442023-12-06T01:49:59.892Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_9a03324a-da27-4bb5-8a44-117c21079574/root/var/svc/log/oxide-nexus:default.log (4489131 bytes)
9452023-12-06T01:50:07.289Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_9a03324a-da27-4bb5-8a44-117c21079574/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4489131 -> 4494507
9462023-12-06T01:50:08.381Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_9a03324a-da27-4bb5-8a44-117c21079574/root/var/svc/log/oxide-nexus:default.log
9472023-12-06T01:50:08.381Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e477e602-1a29-4fd3-9c5e-17b6df0fc0f0/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9482023-12-06T01:50:09.906Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e477e602-1a29-4fd3-9c5e-17b6df0fc0f0/root/var/svc/log/oxide-crucible-agent:default.log
9492023-12-06T01:50:09.907Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e477e602-1a29-4fd3-9c5e-17b6df0fc0f0/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9502023-12-06T01:50:11.046Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e477e602-1a29-4fd3-9c5e-17b6df0fc0f0/root/var/svc/log/oxide-crucible-downstairs:default.log
9512023-12-06T01:50:11.046Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_589bb911-b0c9-45f2-9736-fa1eb9b3b271/root/var/svc/log/oxide-cockroachdb:default.log (861 bytes)
9522023-12-06T01:50:12.178Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_589bb911-b0c9-45f2-9736-fa1eb9b3b271/root/var/svc/log/oxide-cockroachdb:default.log
9532023-12-06T01:50:12.178Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_589bb911-b0c9-45f2-9736-fa1eb9b3b271/root/var/svc/log/oxide-cockroachdb:default.log.0 (4508 bytes)
9542023-12-06T01:50:13.720Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_589bb911-b0c9-45f2-9736-fa1eb9b3b271/root/var/svc/log/oxide-cockroachdb:default.log.0
9552023-12-06T01:50:13.720Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_c5f84063-a3e1-4120-9162-c569a0254ffe/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
9562023-12-06T01:50:14.432Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_c5f84063-a3e1-4120-9162-c569a0254ffe/root/var/svc/log/oxide-crucible-pantry:default.log
9572023-12-06T01:50:14.433Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_c8a703de-60c8-4e01-8cc5-d48dce4df393/root/var/svc/log/oxide-cockroachdb:default.log (1815 bytes)
9582023-12-06T01:50:15.556Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_c8a703de-60c8-4e01-8cc5-d48dce4df393/root/var/svc/log/oxide-cockroachdb:default.log
9592023-12-06T01:50:15.556Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_c8a703de-60c8-4e01-8cc5-d48dce4df393/root/var/svc/log/oxide-cockroachdb:default.log.0 (3554 bytes)
9602023-12-06T01:50:17.111Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_c8a703de-60c8-4e01-8cc5-d48dce4df393/root/var/svc/log/oxide-cockroachdb:default.log.0
9612023-12-06T01:50:17.111Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_01e8e7f9-6598-4373-93ca-9d120f85354d/root/var/svc/log/oxide-crucible-agent:default.log (13685 bytes)
9622023-12-06T01:50:18.248Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_01e8e7f9-6598-4373-93ca-9d120f85354d/root/var/svc/log/oxide-crucible-agent:default.log
9632023-12-06T01:50:18.249Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_01e8e7f9-6598-4373-93ca-9d120f85354d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9642023-12-06T01:50:20.185Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_01e8e7f9-6598-4373-93ca-9d120f85354d/root/var/svc/log/oxide-crucible-downstairs:default.log
9652023-12-06T01:50:20.185Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_01e8e7f9-6598-4373-93ca-9d120f85354d/root/var/svc/log/oxide-crucible-downstairs:downstairs-8a7ac1b5-853a-4b70-b5d6-2dfd4997a153.log (5532 bytes)
9662023-12-06T01:50:21.372Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_01e8e7f9-6598-4373-93ca-9d120f85354d/root/var/svc/log/oxide-crucible-downstairs:downstairs-8a7ac1b5-853a-4b70-b5d6-2dfd4997a153.log
9672023-12-06T01:50:21.372Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_60f041fc-4b26-4286-866b-6800d9de28b4/root/var/svc/log/oxide-crucible-agent:default.log (13689 bytes)
9682023-12-06T01:50:22.494Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_60f041fc-4b26-4286-866b-6800d9de28b4/root/var/svc/log/oxide-crucible-agent:default.log
9692023-12-06T01:50:22.494Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_60f041fc-4b26-4286-866b-6800d9de28b4/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9702023-12-06T01:50:23.594Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_60f041fc-4b26-4286-866b-6800d9de28b4/root/var/svc/log/oxide-crucible-downstairs:default.log
9712023-12-06T01:50:23.594Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_60f041fc-4b26-4286-866b-6800d9de28b4/root/var/svc/log/oxide-crucible-downstairs:downstairs-0a91d796-8b85-4b78-910b-4515de789ce0.log (3050 bytes)
9722023-12-06T01:50:25.167Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_60f041fc-4b26-4286-866b-6800d9de28b4/root/var/svc/log/oxide-crucible-downstairs:downstairs-0a91d796-8b85-4b78-910b-4515de789ce0.log
9732023-12-06T01:50:25.167Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_2eb084c6-7f30-4827-a4a8-170aa51621c4/root/var/svc/log/oxide-internal_dns:default.log (93574 bytes)
9742023-12-06T01:50:25.192Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_2eb084c6-7f30-4827-a4a8-170aa51621c4/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93574 -> 97332
9752023-12-06T01:50:26.052Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_2eb084c6-7f30-4827-a4a8-170aa51621c4/root/var/svc/log/oxide-internal_dns:default.log
9762023-12-06T01:50:26.052Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_2eb084c6-7f30-4827-a4a8-170aa51621c4/root/var/svc/log/oxide-internal_dns:default.log.0 (21752 bytes)
9772023-12-06T01:50:27.760Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_2eb084c6-7f30-4827-a4a8-170aa51621c4/root/var/svc/log/oxide-internal_dns:default.log.0
9782023-12-06T01:50:27.760Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_8426d077-cb12-44fd-b5cc-ab62195afcdd/root/var/svc/log/oxide-oximeter:default.log (187097 bytes)
9792023-12-06T01:50:28.047Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_8426d077-cb12-44fd-b5cc-ab62195afcdd/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 187097 -> 224832
9802023-12-06T01:50:29.127Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_8426d077-cb12-44fd-b5cc-ab62195afcdd/root/var/svc/log/oxide-oximeter:default.log
9812023-12-06T01:50:29.127Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_3c03854c-9960-4091-a475-6e41f49d34ad/root/var/svc/log/oxide-external_dns:default.log (15613 bytes)
9822023-12-06T01:50:29.219Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_3c03854c-9960-4091-a475-6e41f49d34ad/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15613 -> 19390
9832023-12-06T01:50:30.330Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_3c03854c-9960-4091-a475-6e41f49d34ad/root/var/svc/log/oxide-external_dns:default.log
9842023-12-06T01:50:30.330Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_bd758f05-115b-4b22-a443-e424853ca3ef/root/var/svc/log/oxide-nexus:default.log (873260 bytes)
9852023-12-06T01:50:31.465Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_bd758f05-115b-4b22-a443-e424853ca3ef/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 873260 -> 943375
9862023-12-06T01:50:32.545Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_bd758f05-115b-4b22-a443-e424853ca3ef/root/var/svc/log/oxide-nexus:default.log
9872023-12-06T01:50:32.545Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_176b7585-5339-4913-a528-c6894d494cd8/root/var/svc/log/oxide-cockroachdb:default.log (1789 bytes)
9882023-12-06T01:50:33.660Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_176b7585-5339-4913-a528-c6894d494cd8/root/var/svc/log/oxide-cockroachdb:default.log
9892023-12-06T01:50:33.660Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_176b7585-5339-4913-a528-c6894d494cd8/root/var/svc/log/oxide-cockroachdb:default.log.0 (3554 bytes)
9902023-12-06T01:50:34.762Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_176b7585-5339-4913-a528-c6894d494cd8/root/var/svc/log/oxide-cockroachdb:default.log.0
9912023-12-06T01:50:34.762Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1e7ad78f-589a-4cdb-952a-7ae807752638/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
9922023-12-06T01:50:35.871Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1e7ad78f-589a-4cdb-952a-7ae807752638/root/var/svc/log/oxide-crucible-agent:default.log
9932023-12-06T01:50:35.871Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1e7ad78f-589a-4cdb-952a-7ae807752638/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9942023-12-06T01:50:36.972Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1e7ad78f-589a-4cdb-952a-7ae807752638/root/var/svc/log/oxide-crucible-downstairs:default.log
9952023-12-06T01:50:36.972Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_ad2f57ad-2f8f-4444-b378-389c3d21b852/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9962023-12-06T01:50:38.533Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_ad2f57ad-2f8f-4444-b378-389c3d21b852/root/var/svc/log/oxide-crucible-pantry:default.log
9972023-12-06T01:50:38.533Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_d799d62f-874e-4b0d-a798-1b13418e51f7/root/var/svc/log/oxide-external_dns:default.log (15608 bytes)
9982023-12-06T01:50:39.102Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_d799d62f-874e-4b0d-a798-1b13418e51f7/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15608 -> 19386
9992023-12-06T01:50:40.175Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_d799d62f-874e-4b0d-a798-1b13418e51f7/root/var/svc/log/oxide-external_dns:default.log
10002023-12-06T01:50:40.175Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c331db39-a635-4721-82e1-69e946af9d57/root/var/svc/log/oxide-nexus:default.log (646343 bytes)
10012023-12-06T01:50:41.226Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c331db39-a635-4721-82e1-69e946af9d57/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 646343 -> 715273
10022023-12-06T01:50:42.327Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_c331db39-a635-4721-82e1-69e946af9d57/root/var/svc/log/oxide-nexus:default.log
10032023-12-06T01:50:42.327Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_73589577-b47a-49e7-9278-6b2cb37dbedb/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10042023-12-06T01:50:43.451Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_73589577-b47a-49e7-9278-6b2cb37dbedb/root/var/svc/log/oxide-crucible-agent:default.log
10052023-12-06T01:50:43.451Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_73589577-b47a-49e7-9278-6b2cb37dbedb/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10062023-12-06T01:50:45.438Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_73589577-b47a-49e7-9278-6b2cb37dbedb/root/var/svc/log/oxide-crucible-downstairs:default.log
10072023-12-06T01:50:45.438Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a00b6221-8596-4a6d-ad0f-d55482bdbc7d/root/var/svc/log/oxide-crucible-agent:default.log (24827 bytes)
10082023-12-06T01:50:47.274Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a00b6221-8596-4a6d-ad0f-d55482bdbc7d/root/var/svc/log/oxide-crucible-agent:default.log
10092023-12-06T01:50:47.274Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a00b6221-8596-4a6d-ad0f-d55482bdbc7d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10102023-12-06T01:50:48.309Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a00b6221-8596-4a6d-ad0f-d55482bdbc7d/root/var/svc/log/oxide-crucible-downstairs:default.log
10112023-12-06T01:50:48.309Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a00b6221-8596-4a6d-ad0f-d55482bdbc7d/root/var/svc/log/oxide-crucible-downstairs:downstairs-733e3ac8-b582-4982-be3a-1068ef186d37.log (5529 bytes)
10122023-12-06T01:50:49.902Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a00b6221-8596-4a6d-ad0f-d55482bdbc7d/root/var/svc/log/oxide-crucible-downstairs:downstairs-733e3ac8-b582-4982-be3a-1068ef186d37.log
10132023-12-06T01:50:49.902Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a00b6221-8596-4a6d-ad0f-d55482bdbc7d/root/var/svc/log/oxide-crucible-downstairs:downstairs-c15074e1-a637-47ea-9892-7856be06abab.log (3048 bytes)
10142023-12-06T01:50:51.495Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a00b6221-8596-4a6d-ad0f-d55482bdbc7d/root/var/svc/log/oxide-crucible-downstairs:downstairs-c15074e1-a637-47ea-9892-7856be06abab.log
10152023-12-06T01:50:51.495Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c00a2d53-b181-4f9a-9343-16dcccf8c030/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10162023-12-06T01:50:52.692Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c00a2d53-b181-4f9a-9343-16dcccf8c030/root/var/svc/log/oxide-crucible-agent:default.log
10172023-12-06T01:50:52.692Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c00a2d53-b181-4f9a-9343-16dcccf8c030/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10182023-12-06T01:50:53.769Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c00a2d53-b181-4f9a-9343-16dcccf8c030/root/var/svc/log/oxide-crucible-downstairs:default.log
10192023-12-06T01:50:53.770Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e3bd382d-4e02-4938-8649-468c2966ddfc/root/var/svc/log/oxide-crucible-agent:default.log (13681 bytes)
10202023-12-06T01:50:54.949Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e3bd382d-4e02-4938-8649-468c2966ddfc/root/var/svc/log/oxide-crucible-agent:default.log
10212023-12-06T01:50:54.949Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e3bd382d-4e02-4938-8649-468c2966ddfc/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10222023-12-06T01:50:56.068Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e3bd382d-4e02-4938-8649-468c2966ddfc/root/var/svc/log/oxide-crucible-downstairs:default.log
10232023-12-06T01:50:56.068Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e3bd382d-4e02-4938-8649-468c2966ddfc/root/var/svc/log/oxide-crucible-downstairs:downstairs-8b810b62-1288-4e0b-a806-e9465f28c000.log (3049 bytes)
10242023-12-06T01:50:57.171Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e3bd382d-4e02-4938-8649-468c2966ddfc/root/var/svc/log/oxide-crucible-downstairs:downstairs-8b810b62-1288-4e0b-a806-e9465f28c000.log
10252023-12-06T01:50:57.171Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5/root/var/svc/log/oxide-ntp:default.log (0 bytes)
10262023-12-06T01:50:58.248Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5/root/var/svc/log/oxide-ntp:default.log
10272023-12-06T01:50:58.248Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5/root/var/svc/log/oxide-ntp:default.log.0 (851 bytes)
10282023-12-06T01:50:59.392Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5/root/var/svc/log/oxide-ntp:default.log.0
10292023-12-06T01:50:59.392Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5/root/var/log/chrony/measurements.log (11919 bytes)
10302023-12-06T01:50:59.496Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5/root/var/log/chrony/measurements.log" changed size mid upload: 11919 -> 15755
10312023-12-06T01:51:00.585Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5/root/var/log/chrony/measurements.log
10322023-12-06T01:51:00.585Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5/root/var/log/chrony/statistics.log (7488 bytes)
10332023-12-06T01:51:00.677Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5/root/var/log/chrony/statistics.log" changed size mid upload: 7488 -> 9945
10342023-12-06T01:51:01.751Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5/root/var/log/chrony/statistics.log
10352023-12-06T01:51:01.751Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5/root/var/log/chrony/tracking.log (1587 bytes)
10362023-12-06T01:51:01.800Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5/root/var/log/chrony/tracking.log" changed size mid upload: 1587 -> 1719
10372023-12-06T01:51:02.874Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_dd33efad-b027-488b-9394-8233a991b0c5/root/var/log/chrony/tracking.log