01HHNB2DYFZ3GR0V3X7RRNW2XR: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.27",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HHNB3MY6RJNY5KE9XG3NM2YF

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-15T00:37:58.528Zjob dependencies complete; ready to run (waiting for 59 m 50 s)
22023-12-15T00:58:59.247Zjob assigned to worker 01HHNFFNB05CQ4SWDA1EQ4A677 (queued for 21 m 0 s)
32023-12-15T00:58:59.323Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-12-15T00:58:59.376Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-12-15T00:58:59.412Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-12-15T00:58:59.446Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-15T00:58:59.479Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-12-15T00:58:59.514ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-15T00:58:59.549Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-15T00:58:59.581ZOxide Helios Version helios-2.0.22333 64-bit
112023-12-15T00:58:59.616ZConfiguring devices.
122023-12-15T00:58:59.651Z##################################################
132023-12-15T00:58:59.694Z#### ############# ##
142023-12-15T00:58:59.733Z### ### ############ ##
152023-12-15T00:58:59.776Z## ### # ## ### ## Oxide Computer Company ##
162023-12-15T00:58:59.811Z## ## ## ### # ### ##
172023-12-15T00:58:59.851Z## # ### #### #### This Station Under ##
182023-12-15T00:58:59.891Z### ### #### # ### Computer Control ##
192023-12-15T00:58:59.930Z#### #### ### ## ##
202023-12-15T00:58:59.975Z##################################################
212023-12-15T00:59:00.014ZHostname: sock
222023-12-15T00:59:00.049ZLoading smf(7) service descriptions:
232023-12-15T00:59:01.588Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-eae955b6030e5419.gz
242023-12-15T00:59:05.712Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-eae955b6030e5419.gz
252023-12-15T00:59:06.144Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-15T00:59:30.594Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-15T00:59:30.594Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-114e4ef0b9f9b2b3.gz
282023-12-15T01:00:18.524Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-114e4ef0b9f9b2b3.gz
292023-12-15T01:00:18.576Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-15T01:00:20.085Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-15T01:00:20.121Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-15T01:00:23.089Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-15T01:00:23.115Zdownloading input: /input/package/work/version.txt
342023-12-15T01:00:23.141Zdownloaded input: /input/package/work/version.txt
352023-12-15T01:00:23.141Zdownloading input: /input/package/work/package.tar.gz
362023-12-15T01:00:54.528Zdownloaded input: /input/package/work/package.tar.gz
372023-12-15T01:00:54.528Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-15T01:01:30.304Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-15T01:01:30.304Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-15T01:01:38.500Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-15T01:01:38.548Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-15T01:02:26.775Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-15T01:02:26.775Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-15T01:03:23.247Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-15T01:03:23.249Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-15T01:03:54.270Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-15T01:03:54.271Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-15T01:04:02.026Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-15T01:04:02.056Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-15T01:04:12.633Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-15T01:04:12.633Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-15T01:04:18.564Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-15T01:04:18.564Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-15T01:04:25.968Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-15T01:04:25.968Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-15T01:04:26.039Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-15T01:04:26.065Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-15T01:04:32.313Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-15T01:04:32.412Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-15T01:04:48.117Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-15T01:04:48.148Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-15T01:05:14.297Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-15T01:05:14.298Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-15T01:05:37.770Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-15T01:05:37.770Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-15T01:05:37.860Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-15T01:05:37.860Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-15T01:05:50.058Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-15T01:05:50.059Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-15T01:06:05.689Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-15T01:06:05.690Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-15T01:06:58.521Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-15T01:06:58.522Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-15T01:07:41.957Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-15T01:07:41.985Zstarting task 0: "setup"
762023-12-15T01:07:41.985Z++ uname -s
772023-12-15T01:07:41.985Z+ kern=SunOS
782023-12-15T01:07:41.986Z+ case "$kern" in
792023-12-15T01:07:41.986Z+ groupadd -g 12345 build
802023-12-15T01:07:41.986Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-15T01:07:43.988Z+ zfs create -o mountpoint=/work rpool/work
822023-12-15T01:07:44.459Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-15T01:07:44.459Z+ home_fs=zfs
842023-12-15T01:07:44.462Z+ [[ zfs == autofs ]]
852023-12-15T01:07:44.462Z+ mkdir -p /home/build
862023-12-15T01:07:44.462Z+ chown build:build /home/build /work
872023-12-15T01:07:46.005Z+ chmod 0700 /home/build /work
882023-12-15T01:07:46.038Zprocess exited: duration 4051 ms, exit code 0
 
892023-12-15T01:07:46.097Zstarting task 1: "authentication"
902023-12-15T01:07:46.146Zprocess exited: duration 47 ms, exit code 0
 
912023-12-15T01:07:46.607Zstarting task 2: "build"
922023-12-15T01:07:46.734Z+ trap _exit_trap EXIT
932023-12-15T01:07:46.734Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-15T01:07:46.808Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-15T01:07:46.808Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-15T01:07:46.808Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-15T01:07:46.808Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-15T01:07:46.833Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-15T01:07:46.833Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-15T01:07:46.833Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-15T01:07:46.834Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-15T01:07:46.834Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-15T01:07:46.834Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-15T01:07:46.834Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-15T01:07:46.834Z+ [[ -d /opt/oxide ]]
1062023-12-15T01:07:46.834Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-15T01:07:46.834Z+ pfexec mkdir /opt/oxide
1082023-12-15T01:07:46.834Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-15T01:07:46.834Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-15T01:07:46.834Z+ cd /opt/oxide-underneath
1112023-12-15T01:07:46.834Z+ pfexec tar ceEp@/f - .
1122023-12-15T01:07:46.834Z+ cd /opt/oxide
1132023-12-15T01:07:46.834Z+ pfexec tar xveEp@/f -
1142023-12-15T01:07:46.947Zx ., 0 bytes, 0 tape blocks
1152023-12-15T01:07:46.947Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-15T01:07:46.947Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-15T01:07:46.947Zx ./opte/bin/opteadm, 52567720 bytes, 102672 tape blocks
1182023-12-15T01:07:46.973Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-15T01:07:46.973Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-15T01:07:47.006Z+ pfexec mkdir /opt/oxide/work
1212023-12-15T01:07:47.006Z+ pfexec chown build:build /opt/oxide/work
1222023-12-15T01:07:47.006Z+ cd /opt/oxide/work
1232023-12-15T01:07:47.006Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-15T01:07:48.931ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-12-15T01:07:48.931Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-12-15T01:07:48.931Zx out/omicron-sled-agent.tar, 111881728 bytes, 218519 tape blocks
1272023-12-15T01:07:48.931Zx out/target/test, 75 bytes, 1 tape blocks
1282023-12-15T01:07:48.931Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1292023-12-15T01:07:48.931Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-12-15T01:07:48.932Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1312023-12-15T01:07:48.932Zx target/release/omicron-package, 42517504 bytes, 83042 tape blocks
1322023-12-15T01:07:48.932Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-12-15T01:07:48.932Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-12-15T01:07:48.932Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-12-15T01:07:48.932Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-12-15T01:07:48.932Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-12-15T01:07:48.932Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-12-15T01:07:48.932Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-12-15T01:07:48.963Z
1402023-12-15T01:07:48.964Zreal 1.538648391
1412023-12-15T01:07:48.964Zuser 1.387273172
1422023-12-15T01:07:48.964Zsys 0.261471442
1432023-12-15T01:07:48.964Ztrap 0.000055029
1442023-12-15T01:07:48.964Ztflt 0.000000000
1452023-12-15T01:07:48.964Zdflt 0.000000000
1462023-12-15T01:07:48.964Zkflt 0.000000000
1472023-12-15T01:07:48.964Zlock 0.000000000
1482023-12-15T01:07:48.964Zslp 1.388554366
1492023-12-15T01:07:48.964Zlat 0.037711586
1502023-12-15T01:07:48.964Zstop 0.000147281
1512023-12-15T01:07:48.965Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-12-15T01:07:49.618Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-12-15T01:07:49.651Z+ mkdir tests
1542023-12-15T01:07:49.652Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-12-15T01:07:49.652Z+ ptime -m gunzip
1562023-12-15T01:07:49.652Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-eae955b6030e5419
1572023-12-15T01:07:50.197Z
1582023-12-15T01:07:50.237Zreal 0.140009149
1592023-12-15T01:07:50.237Zuser 0.128473250
1602023-12-15T01:07:50.237Zsys 0.011286364
1612023-12-15T01:07:50.237Ztrap 0.000013139
1622023-12-15T01:07:50.238Ztflt 0.000000000
1632023-12-15T01:07:50.238Zdflt 0.000000000
1642023-12-15T01:07:50.238Zkflt 0.000000000
1652023-12-15T01:07:50.238Zlock 0.000000000
1662023-12-15T01:07:50.238Zslp 0.000024640
1672023-12-15T01:07:50.238Zlat 0.000022360
1682023-12-15T01:07:50.239Zstop 0.000125892
1692023-12-15T01:07:50.239Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-eae955b6030e5419
1702023-12-15T01:07:50.239Z+ chmod a+x tests/bootstrap-eae955b6030e5419
1712023-12-15T01:07:50.239Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-12-15T01:07:50.239Z+ ptime -m gunzip
1732023-12-15T01:07:50.239Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-12-15T01:07:52.017Z
1752023-12-15T01:07:52.049Zreal 2.222477308
1762023-12-15T01:07:52.050Zuser 2.093323796
1772023-12-15T01:07:52.050Zsys 0.128732906
1782023-12-15T01:07:52.050Ztrap 0.000183583
1792023-12-15T01:07:52.050Ztflt 0.000000000
1802023-12-15T01:07:52.050Zdflt 0.000000000
1812023-12-15T01:07:52.051Zkflt 0.000000000
1822023-12-15T01:07:52.051Zlock 0.000000000
1832023-12-15T01:07:52.051Zslp 0.000021530
1842023-12-15T01:07:52.051Zlat 0.000023201
1852023-12-15T01:07:52.051Zstop 0.000130671
1862023-12-15T01:07:52.051Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-12-15T01:07:52.051Z+ chmod a+x tests/bootstrap
1882023-12-15T01:07:52.051Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-12-15T01:07:52.051Z+ ptime -m gunzip
1902023-12-15T01:07:52.051Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-114e4ef0b9f9b2b3
1912023-12-15T01:07:54.369Z
1922023-12-15T01:07:54.393Zreal 2.344895932
1932023-12-15T01:07:54.394Zuser 2.191999147
1942023-12-15T01:07:54.394Zsys 0.152439299
1952023-12-15T01:07:54.395Ztrap 0.000187072
1962023-12-15T01:07:54.395Ztflt 0.000000000
1972023-12-15T01:07:54.395Zdflt 0.000000000
1982023-12-15T01:07:54.395Zkflt 0.000000000
1992023-12-15T01:07:54.395Zlock 0.000000000
2002023-12-15T01:07:54.395Zslp 0.000048730
2012023-12-15T01:07:54.395Zlat 0.000023899
2022023-12-15T01:07:54.395Zstop 0.000126041
2032023-12-15T01:07:54.395Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-114e4ef0b9f9b2b3
2042023-12-15T01:07:54.395Z+ chmod a+x tests/end_to_end_tests-114e4ef0b9f9b2b3
2052023-12-15T01:07:54.395Z+ bmat address ls
2062023-12-15T01:07:54.395ZNAME CIDR FIRST LAST COUNT
2072023-12-15T01:07:54.395Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-12-15T01:07:54.395Z++ bmat address ls -f extra -Ho first,last,count
2092023-12-15T01:07:54.420Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-12-15T01:07:54.420Z+ EXTRA_IP_START=10.151.2.164
2112023-12-15T01:07:54.420Z+ EXTRA_IP_END=10.151.2.227
2122023-12-15T01:07:54.420Z+ EXTRA_IP_COUNT=64
2132023-12-15T01:07:54.420Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-12-15T01:07:54.421Z+ EXTRA_IP_BASE=10.151.2
2152023-12-15T01:07:54.421Z+ EXTRA_IP_FOCTET=164
2162023-12-15T01:07:54.421Z+ EXTRA_IP_LOCTET=227
2172023-12-15T01:07:54.421Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-12-15T01:07:54.421Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-12-15T01:07:54.421Z+ DNS_IP1=10.151.2.164
2202023-12-15T01:07:54.421Z+ DNS_IP2=10.151.2.165
2212023-12-15T01:07:54.421Z+ UPLINK_IP=10.151.2.174
2222023-12-15T01:07:54.421Z+ IPPOOL_START=10.151.2.175
2232023-12-15T01:07:54.422Z+ IPPOOL_END=10.151.2.227
2242023-12-15T01:07:54.422Z++ ipadm show-addr -po type,addr
2252023-12-15T01:07:54.422Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-12-15T01:07:54.422Z+ GATEWAY_IP=10.151.2.100
2272023-12-15T01:07:54.422Z+ [[ -n 10.151.2.100 ]]
2282023-12-15T01:07:54.422Z+ ping -s 10.151.2.100 56 1
2292023-12-15T01:07:54.422ZPING 10.151.2.100: 56 data bytes
2302023-12-15T01:07:54.422Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2312023-12-15T01:07:54.422Z
2322023-12-15T01:07:54.423Z----10.151.2.100 PING Statistics----
2332023-12-15T01:07:54.423Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-12-15T01:07:54.423Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-12-15T01:07:54.423Z++ arp -an
2362023-12-15T01:07:54.423Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-12-15T01:07:54.447Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-12-15T01:07:54.448Z+ routeadm -e ipv4-forwarding -u
2392023-12-15T01:07:54.533Z+ PXA_START=10.151.2.164
2402023-12-15T01:07:54.533Z+ PXA_END=10.151.2.227
2412023-12-15T01:07:54.535Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-12-15T01:07:54.565Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-12-15T01:07:54.670Z+ ZPOOL_VDEV_DIR=/scratch
2442023-12-15T01:07:54.670Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-12-15T01:07:54.707Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-12-15T01:07:54.707Z++ cd ./tools
2472023-12-15T01:07:54.707Z++ pwd
2482023-12-15T01:07:54.707Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-12-15T01:07:54.707Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-12-15T01:07:54.708Z+ SOFTNPU_MODE=zone
2512023-12-15T01:07:54.722Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-12-15T01:07:54.722Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-12-15T01:07:54.723Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-12-15T01:07:54.723Z++ dladm show-phys -p -o LINK
2552023-12-15T01:07:54.723Z++ head -1
2562023-12-15T01:07:54.761Z+ PHYSICAL_LINK=igb0
2572023-12-15T01:07:54.761Z+ echo 'Using igb0 as physical link'
2582023-12-15T01:07:54.761ZUsing igb0 as physical link
2592023-12-15T01:07:54.761Z+ ensure_run_as_root
2602023-12-15T01:07:54.761Z++ id -u
2612023-12-15T01:07:54.761Z+ [[ 0 -ne 0 ]]
2622023-12-15T01:07:54.761Z+ ensure_zpools
2632023-12-15T01:07:54.761Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-12-15T01:07:54.761Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-12-15T01:07:54.761Z+ readarray -t ZPOOLS
2662023-12-15T01:07:54.787Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-12-15T01:07:54.787Z++ sed 's/[ ",]//g'
2682023-12-15T01:07:54.787Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-12-15T01:07:54.787Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-12-15T01:07:54.788ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-12-15T01:07:54.788Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-12-15T01:07:54.788Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-12-15T01:07:54.788Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-12-15T01:07:54.788Z0+0 records in
2752023-12-15T01:07:54.788Z0+0 records out
2762023-12-15T01:07:54.788Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2772023-12-15T01:07:54.788Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-12-15T01:07:54.788Z+ set +x
2792023-12-15T01:07:54.788ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-12-15T01:07:54.814Z++ zpool list -o name
2812023-12-15T01:07:54.814Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-12-15T01:07:54.815Z+ [[ -z '' ]]
2832023-12-15T01:07:54.815Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-12-15T01:07:54.839Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-12-15T01:07:54.839Z+ set +x
2862023-12-15T01:07:54.839ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-12-15T01:07:54.839Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-12-15T01:07:54.839Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-12-15T01:07:54.839ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-12-15T01:07:54.839Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-12-15T01:07:54.840Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-12-15T01:07:54.840Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-12-15T01:07:54.840Z0+0 records in
2942023-12-15T01:07:54.840Z0+0 records out
2952023-12-15T01:07:54.840Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-12-15T01:07:54.840Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-12-15T01:07:54.840Z+ set +x
2982023-12-15T01:07:54.840ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-12-15T01:07:54.864Z++ zpool list -o name
3002023-12-15T01:07:54.864Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-12-15T01:07:54.866Z+ [[ -z '' ]]
3022023-12-15T01:07:54.866Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-12-15T01:07:54.891Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-12-15T01:07:54.892Z+ set +x
3052023-12-15T01:07:54.892ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-12-15T01:07:54.892Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-12-15T01:07:54.892Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-12-15T01:07:54.892ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3092023-12-15T01:07:54.892Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-12-15T01:07:54.892Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-12-15T01:07:54.892Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-12-15T01:07:54.892Z0+0 records in
3132023-12-15T01:07:54.892Z0+0 records out
3142023-12-15T01:07:54.892Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3152023-12-15T01:07:54.892Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-12-15T01:07:54.893Z+ set +x
3172023-12-15T01:07:54.893ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-12-15T01:07:54.893Z++ zpool list -o name
3192023-12-15T01:07:54.893Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-12-15T01:07:54.893Z+ [[ -z '' ]]
3212023-12-15T01:07:54.893Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-12-15T01:07:54.929Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-12-15T01:07:54.929Z+ set +x
3242023-12-15T01:07:54.929ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-12-15T01:07:54.929Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-12-15T01:07:54.930Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-12-15T01:07:54.930ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-12-15T01:07:54.930Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-12-15T01:07:54.930Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-12-15T01:07:54.930Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-12-15T01:07:54.959Z0+0 records in
3322023-12-15T01:07:54.959Z0+0 records out
3332023-12-15T01:07:54.959Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-12-15T01:07:54.959Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-12-15T01:07:54.959Z+ set +x
3362023-12-15T01:07:54.959ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-12-15T01:07:54.959Z++ zpool list -o name
3382023-12-15T01:07:54.960Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-12-15T01:07:54.960Z+ [[ -z '' ]]
3402023-12-15T01:07:54.972Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-12-15T01:07:54.999Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-12-15T01:07:54.999Z+ set +x
3432023-12-15T01:07:54.999ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-12-15T01:07:54.999Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-12-15T01:07:55.000Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-12-15T01:07:55.000ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-12-15T01:07:55.000Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-12-15T01:07:55.000Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-12-15T01:07:55.000Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-12-15T01:07:55.003Z0+0 records in
3512023-12-15T01:07:55.004Z0+0 records out
3522023-12-15T01:07:55.004Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3532023-12-15T01:07:55.004Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-12-15T01:07:55.004Z+ set +x
3552023-12-15T01:07:55.004ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-12-15T01:07:55.004Z++ zpool list -o name
3572023-12-15T01:07:55.004Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-12-15T01:07:55.004Z+ [[ -z '' ]]
3592023-12-15T01:07:55.004Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-12-15T01:07:55.032Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-12-15T01:07:55.033Z+ set +x
3622023-12-15T01:07:55.033ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-12-15T01:07:55.033ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3642023-12-15T01:07:55.033Z+ for ZPOOL in "${ZPOOLS[@]}"
3652023-12-15T01:07:55.033Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3662023-12-15T01:07:55.033Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-12-15T01:07:55.033Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-12-15T01:07:55.033Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-12-15T01:07:55.033Z0+0 records in
3702023-12-15T01:07:55.033Z0+0 records out
3712023-12-15T01:07:55.033Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3722023-12-15T01:07:55.033Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-12-15T01:07:55.033Z+ set +x
3742023-12-15T01:07:55.033ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-12-15T01:07:55.034Z++ zpool list -o name
3762023-12-15T01:07:55.034Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-12-15T01:07:55.034Z+ [[ -z '' ]]
3782023-12-15T01:07:55.034Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-12-15T01:07:55.059Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-12-15T01:07:55.059Z+ set +x
3812023-12-15T01:07:55.059ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-12-15T01:07:55.059Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-12-15T01:07:55.059Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-12-15T01:07:55.059ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-12-15T01:07:55.059Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-12-15T01:07:55.059Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-12-15T01:07:55.059Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-12-15T01:07:55.059Z0+0 records in
3892023-12-15T01:07:55.060Z0+0 records out
3902023-12-15T01:07:55.060Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-12-15T01:07:55.060Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-12-15T01:07:55.060Z+ set +x
3932023-12-15T01:07:55.060ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-12-15T01:07:55.060Z++ zpool list -o name
3952023-12-15T01:07:55.060Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-12-15T01:07:55.060Z+ [[ -z '' ]]
3972023-12-15T01:07:55.060Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-12-15T01:07:55.087Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-12-15T01:07:55.088Z+ set +x
4002023-12-15T01:07:55.115ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-12-15T01:07:55.115Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-12-15T01:07:55.115Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-12-15T01:07:55.116ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-12-15T01:07:55.116Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-12-15T01:07:55.116Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-12-15T01:07:55.116Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-12-15T01:07:55.116Z0+0 records in
4082023-12-15T01:07:55.116Z0+0 records out
4092023-12-15T01:07:55.116Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-12-15T01:07:55.116Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-12-15T01:07:55.116Z+ set +x
4122023-12-15T01:07:55.116ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-12-15T01:07:55.116Z++ zpool list -o name
4142023-12-15T01:07:55.116Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-12-15T01:07:55.116Z+ [[ -z '' ]]
4162023-12-15T01:07:55.116Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-12-15T01:07:55.143Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-12-15T01:07:55.143Z+ set +x
4192023-12-15T01:07:55.143ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-12-15T01:07:55.143Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-12-15T01:07:55.143Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-12-15T01:07:55.143ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-12-15T01:07:55.143Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-12-15T01:07:55.145Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-12-15T01:07:55.145Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-12-15T01:07:55.170Z0+0 records in
4272023-12-15T01:07:55.170Z0+0 records out
4282023-12-15T01:07:55.170Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4292023-12-15T01:07:55.170Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-12-15T01:07:55.170Z+ set +x
4312023-12-15T01:07:55.170ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-12-15T01:07:55.170Z++ zpool list -o name
4332023-12-15T01:07:55.170Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-12-15T01:07:55.170Z+ [[ -z '' ]]
4352023-12-15T01:07:55.171Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-12-15T01:07:55.195Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-12-15T01:07:55.195Z+ set +x
4382023-12-15T01:07:55.195ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-12-15T01:07:55.195Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-12-15T01:07:55.195Z+ readarray -t ZPOOLS
4412023-12-15T01:07:55.195Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-12-15T01:07:55.195Z++ sed 's/[ ",]//g'
4432023-12-15T01:07:55.195Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-12-15T01:07:55.195Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-12-15T01:07:55.195Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4462023-12-15T01:07:55.196ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4472023-12-15T01:07:55.196Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-12-15T01:07:55.196Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-12-15T01:07:55.197Z0+0 records in
4502023-12-15T01:07:55.197Z0+0 records out
4512023-12-15T01:07:55.197Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4522023-12-15T01:07:55.197Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-12-15T01:07:55.197Z+ set +x
4542023-12-15T01:07:55.198ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-12-15T01:07:55.222Z++ zpool list -o name
4562023-12-15T01:07:55.222Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-12-15T01:07:55.222Z+ [[ -z '' ]]
4582023-12-15T01:07:55.222Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-12-15T01:07:55.248Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-12-15T01:07:55.249Z+ set +x
4612023-12-15T01:07:55.249ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-12-15T01:07:55.249Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-12-15T01:07:55.249Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-12-15T01:07:55.249ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-12-15T01:07:55.249Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-12-15T01:07:55.249Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-12-15T01:07:55.249Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-12-15T01:07:55.249Z0+0 records in
4692023-12-15T01:07:55.249Z0+0 records out
4702023-12-15T01:07:55.250Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4712023-12-15T01:07:55.250Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-12-15T01:07:55.250Z+ set +x
4732023-12-15T01:07:55.250ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-12-15T01:07:55.251Z++ zpool list -o name
4752023-12-15T01:07:55.251Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-12-15T01:07:55.276Z+ [[ -z '' ]]
4772023-12-15T01:07:55.276Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-12-15T01:07:55.304Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-12-15T01:07:55.304Z+ set +x
4802023-12-15T01:07:55.304ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-12-15T01:07:55.304Z+ [[ zone == \z\o\n\e ]]
4822023-12-15T01:07:55.304Z+ ensure_simulated_links igb0
4832023-12-15T01:07:55.304Z+ local PHYSICAL_LINK=igb0
4842023-12-15T01:07:55.304Z+ INDICES=("0" "1")
4852023-12-15T01:07:55.304Z+ for I in "${INDICES[@]}"
4862023-12-15T01:07:55.305Z++ get_simnet_name_if_exists net0
4872023-12-15T01:07:55.305Z++ dladm show-simnet -p -o LINK net0
4882023-12-15T01:07:55.305Z++ echo ''
4892023-12-15T01:07:55.305Z+ [[ -z '' ]]
4902023-12-15T01:07:55.305Z+ dladm create-simnet -t net0
4912023-12-15T01:07:55.305Z+ dladm create-simnet -t sc0_0
4922023-12-15T01:07:55.330Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-12-15T01:07:55.330Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-12-15T01:07:55.330Z+ success 'Simnet net0/sc0_0 exists'
4952023-12-15T01:07:55.330Z+ set +x
4962023-12-15T01:07:55.331Z+ for I in "${INDICES[@]}"
4972023-12-15T01:07:55.331ZSimnet net0/sc0_0 exists
4982023-12-15T01:07:55.331Z++ get_simnet_name_if_exists net1
4992023-12-15T01:07:55.331Z++ dladm show-simnet -p -o LINK net1
5002023-12-15T01:07:55.331Z++ echo ''
5012023-12-15T01:07:55.331Z+ [[ -z '' ]]
5022023-12-15T01:07:55.331Z+ dladm create-simnet -t net1
5032023-12-15T01:07:55.331Z+ dladm create-simnet -t sc1_0
5042023-12-15T01:07:55.331Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-12-15T01:07:55.331Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-12-15T01:07:55.356Z+ success 'Simnet net1/sc1_0 exists'
5072023-12-15T01:07:55.356Z+ set +x
5082023-12-15T01:07:55.356ZSimnet net1/sc1_0 exists
5092023-12-15T01:07:55.356Z++ get_vnic_name_if_exists sc0_1
5102023-12-15T01:07:55.356Z++ dladm show-vnic -p -o LINK sc0_1
5112023-12-15T01:07:55.356Z++ echo ''
5122023-12-15T01:07:55.356Z+ [[ -z '' ]]
5132023-12-15T01:07:55.356Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-12-15T01:07:55.356Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-12-15T01:07:55.356Z+ success 'Vnic sc0_1 exists'
5162023-12-15T01:07:55.356Z+ set +x
5172023-12-15T01:07:55.356ZVnic sc0_1 exists
5182023-12-15T01:07:55.356Z+ warn_if_no_proxy_arp
5192023-12-15T01:07:55.356Z+ [[ -v PXA_START ]]
5202023-12-15T01:07:55.356Z+ [[ -v PXA_END ]]
5212023-12-15T01:07:55.356Z+ ensure_softnpu_zone
5222023-12-15T01:07:55.356Z+ zoneadm list
5232023-12-15T01:07:55.357Z+ grep -q sidecar_softnpu
5242023-12-15T01:07:55.357Z+ [[ -f out/npuzone/npuzone ]]
5252023-12-15T01:07:55.357Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit 45ed98fea5824feb4d42f45bbf218e597dc9fc58 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5262023-12-15T01:07:55.381Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5272023-12-15T01:08:11.482Zdownloading softnpu ...done
5282023-12-15T01:08:11.526Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/libsidecar_lite.so.sha256.txt
5292023-12-15T01:08:21.424Zdownloading asic_program.so ...done
5302023-12-15T01:08:21.453Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/scadm.sha256.txt
5312023-12-15T01:08:30.436Zdownloading scadm ...done
5322023-12-15T01:08:30.556Zcreating port tfportrear0_0 for sc0_0
5332023-12-15T01:08:30.556Zcreating port tfportqsfp0_0 for sc0_1
5342023-12-15T01:08:30.577Zsoftnpu zone setup
5352023-12-15T01:08:30.577Zconfigure zone
5362023-12-15T01:08:30.577Zinstall zone
5372023-12-15T01:08:32.030Zboot zone
5382023-12-15T01:08:46.564Z[sidecar_softnpu] svcs milestone/network
5392023-12-15T01:08:46.564ZSTATE STIME FMRI
5402023-12-15T01:08:46.564Zonline 1:08:33 svc:/milestone/network:default
5412023-12-15T01:08:46.564Z[sidecar_softnpu] /softnpu/npu start
5422023-12-15T01:08:46.564Z
5432023-12-15T01:08:46.564Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-12-15T01:08:46.564Z+ GATEWAY_IP=10.151.2.100
5452023-12-15T01:08:46.564Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-12-15T01:08:46.564ZUsing 10.151.2.100 as gateway ip
5472023-12-15T01:08:46.564Z+ [[ ! -v GATEWAY_MAC ]]
5482023-12-15T01:08:46.564Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-12-15T01:08:46.564Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-12-15T01:08:46.564Z+++ echo 18:c0:4d:0d:a0:2a
5512023-12-15T01:08:46.565Z+ COUNT=1
5522023-12-15T01:08:46.565Z+ [[ 1 -ne 1 ]]
5532023-12-15T01:08:46.565Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-12-15T01:08:46.565ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-12-15T01:08:46.565Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-12-15T01:08:46.565Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-12-15T01:08:46.565Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-12-15T01:08:46.565Z+ [[ -v PXA_START ]]
5592023-12-15T01:08:46.565Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-12-15T01:08:46.567Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-12-15T01:08:46.567Z+ z_scadm dump-state
5622023-12-15T01:08:46.567Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-12-15T01:08:46.567Zlocal v6:
5642023-12-15T01:08:46.567Zlocal v4:
5652023-12-15T01:08:46.567Zrouter v6:
5662023-12-15T01:08:46.567Zrouter v4_idx:
5672023-12-15T01:08:46.567Zrouter v4_routes:
5682023-12-15T01:08:46.567Zresolver v4:
5692023-12-15T01:08:46.567Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-12-15T01:08:46.567Zresolver v6:
5712023-12-15T01:08:46.567Znat v4:
5722023-12-15T01:08:46.567Znat_v6:
5732023-12-15T01:08:46.567Zport_mac:
5742023-12-15T01:08:46.567Zproxy arp:
5752023-12-15T01:08:46.567Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-12-15T01:08:46.567Z+ success 'softnpu zone exists'
5772023-12-15T01:08:46.567Z+ set +x
5782023-12-15T01:08:46.567Zsoftnpu zone exists
5792023-12-15T01:08:46.567Z
5802023-12-15T01:08:46.567Zreal 40.130663498
5812023-12-15T01:08:46.567Zuser 0.706781412
5822023-12-15T01:08:46.568Zsys 1.578113436
5832023-12-15T01:08:46.568Ztrap 0.000107983
5842023-12-15T01:08:46.568Ztflt 0.000177960
5852023-12-15T01:08:46.568Zdflt 0.001528000
5862023-12-15T01:08:46.568Zkflt 0.000000000
5872023-12-15T01:08:46.568Zlock 31:26.804760297
5882023-12-15T01:08:46.568Zslp 2:00.321649694
5892023-12-15T01:08:46.568Zlat 0.050699559
5902023-12-15T01:08:46.568Zstop 0.333359171
5912023-12-15T01:08:46.568Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-12-15T01:08:46.568Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-12-15T01:08:46.568Z+ SILO_NAME=recovery
5942023-12-15T01:08:46.568Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-15T01:08:46.568Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-12-15T01:08:46.568Z+ sed -i~ '
5972023-12-15T01:08:46.568Z /^external_dns_ips/c\
5982023-12-15T01:08:46.568Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-12-15T01:08:46.568Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-12-15T01:08:46.568Z /^first/c\
6012023-12-15T01:08:46.568Zfirst = "10.151.2.164"
6022023-12-15T01:08:46.568Z /^last/c\
6032023-12-15T01:08:46.568Zlast = "10.151.2.173"
6042023-12-15T01:08:46.568Z }
6052023-12-15T01:08:46.568Z /^\[rack_network_config/,/^$/ {
6062023-12-15T01:08:46.568Z /^infra_ip_first/c\
6072023-12-15T01:08:46.569Zinfra_ip_first = "10.151.2.174"
6082023-12-15T01:08:46.569Z /^infra_ip_last/c\
6092023-12-15T01:08:46.569Zinfra_ip_last = "10.151.2.174"
6102023-12-15T01:08:46.569Z }
6112023-12-15T01:08:46.569Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-12-15T01:08:46.569Z /^routes/c\
6132023-12-15T01:08:46.569Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6142023-12-15T01:08:46.569Z /^addresses/c\
6152023-12-15T01:08:46.569Zaddresses = \["10.151.2.174/32"\]
6162023-12-15T01:08:46.569Z }
6172023-12-15T01:08:46.569Z' pkg/config-rss.toml
6182023-12-15T01:08:46.571Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-12-15T01:08:46.571Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-12-15T01:08:46.571Z+++ pkg/config-rss.toml Fri Dec 15 01:08:34 2023
6212023-12-15T01:08:46.571Z@@ -41,7 +41,7 @@
6222023-12-15T01:08:46.571Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-12-15T01:08:46.571Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-12-15T01:08:46.571Z # below.
6252023-12-15T01:08:46.571Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-12-15T01:08:46.571Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6272023-12-15T01:08:46.571Z
6282023-12-15T01:08:46.571Z # Initial TLS certificates for the external API
6292023-12-15T01:08:46.571Z #
6302023-12-15T01:08:46.571Z@@ -77,8 +77,8 @@
6312023-12-15T01:08:46.571Z #
6322023-12-15T01:08:46.571Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-12-15T01:08:46.571Z [[internal_services_ip_pool_ranges]]
6342023-12-15T01:08:46.572Z-first = "192.168.1.20"
6352023-12-15T01:08:46.572Z-last = "192.168.1.29"
6362023-12-15T01:08:46.572Z+first = "10.151.2.164"
6372023-12-15T01:08:46.574Z+last = "10.151.2.173"
6382023-12-15T01:08:46.840Z
6392023-12-15T01:08:46.840Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-12-15T01:08:46.841Z # parameters should be provided to the control plane via wicket, but we need to
6412023-12-15T01:08:46.841Z@@ -92,8 +92,8 @@
6422023-12-15T01:08:46.841Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-12-15T01:08:46.841Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-12-15T01:08:46.841Z # With softnpu, only one address is used.
6452023-12-15T01:08:46.841Z-infra_ip_first = "192.168.1.30"
6462023-12-15T01:08:46.841Z-infra_ip_last = "192.168.1.30"
6472023-12-15T01:08:46.842Z+infra_ip_first = "10.151.2.174"
6482023-12-15T01:08:46.842Z+infra_ip_last = "10.151.2.174"
6492023-12-15T01:08:46.842Z
6502023-12-15T01:08:46.842Z # Configurations for BGP routers to run on the scrimlets.
6512023-12-15T01:08:46.842Z bgp = []
6522023-12-15T01:08:46.842Z@@ -101,9 +101,9 @@
6532023-12-15T01:08:46.842Z+ true
6542023-12-15T01:08:46.843Z # You can configure multiple uplinks by repeating the following stanza
6552023-12-15T01:08:46.843Z [[rack_network_config.ports]]
6562023-12-15T01:08:46.843Z # Routes associated with this port.
6572023-12-15T01:08:46.843Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6582023-12-15T01:08:46.843Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6592023-12-15T01:08:46.844Z # Addresses associated with this port.
6602023-12-15T01:08:46.844Z-addresses = ["192.168.1.30/32"]
6612023-12-15T01:08:46.845Z+addresses = ["10.151.2.174/32"]
6622023-12-15T01:08:46.845Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6632023-12-15T01:08:46.845Z port = "qsfp0"
6642023-12-15T01:08:46.846Z # The speed of this port.
6652023-12-15T01:08:46.846Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-12-15T01:08:46.846Za pkg/config-rss.toml 7K
6672023-12-15T01:08:46.848Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-12-15T01:08:46.848Z+ TLS_NAME=recovery.sys.oxide.test
6692023-12-15T01:08:46.848Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-12-15T01:08:46.848Z.+.....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+............+.+.....+.+.........+.....+......+.........+.+...+...........+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+.+.....+.......+..+............+...+.+..............+.+.....+......+...+...+.............+..+.......+........+.............+.................+.............+............+..............+.+........+.+..............+.......+.........+...+..+...+...+...+....+......+..............+.+......+...........+...+.........+......+...............+......+.+..................+..+..........+...+......+..............+.+.....+.+.....+....+........................+.........+...+..+.+.....................+......+..+......+.......+...............+.....+.......+...+........+......+.........+..........+...........+...+...+.......+.....+...+....+.....+...............+...+...+...+.......+...............+...+..+...+......+.+...+......+.........+....................+......+....+.....+.......+.....+...+................+......+...+.....................+........+..........+...............+...........+.........+.+......+...+...............+...+.....+.............+...........+......+...+..........+....................+.........+...+.......+...............+..+.+........+..............................+...................+..+................+..+.+.........+...+..............+....+.................+....+...+..+..........+...+.................+...+.......+.........+.........+..+.+..+......+.+.....................+...+..+.......+......+......+......+........+......................+.....+..................+......+.+.....+...+....+...+..+...............+...+...+.......+...+..+...+......+................+............+..+.........+.+...........+...+......+..................+.......+...+.................+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-12-15T01:08:46.848Z..+.+.....+....+...+...+.....+.+.....+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+........+.......+......+..+.+.....+....+.....+......+.+......+.....+............+...+....+...+...+...+.....+....+.....+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+........+.+......+........+.......+.........+..+...+.........+...+............................+..+............+.+.....+.........+.......+......+.................+......+...................+...+...+.....+......+.+.........+..................+............+...............+......+...+.....+................+......+.....+.........+...+.+..+...............+....+...............+...+..+..........+........+...+...............+...+......+.+.....+...+..........+......+...+..+.........+..........+.....+....+......+............+..+.+.........+........+.......+......+.....+....+..+.+...+...........+......+...+....+......+.....+......+.+....................+.......+..+...+......+......+....+...........+...+...+.............+............+..+.+......+........+......+..........+......+...............+......+...+.................+.......+..+....+...............+....................+.......+...+..............+.......+...+........+............+...+....+...+...........+.........+...+...+.......+...+....................+......+.........+.+......+...+..+............+................+.....+............+.............+...........+...+............+.+.....+.........+.+...............+...+.....................+.........+..+....+...+..+.+...............+...+.....+....+..............................+..+...+..........+..................+..+.+..+....+.....................+.....+............+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-15T01:08:47.015Z-----
6732023-12-15T01:08:47.015Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-12-15T01:08:47.015Za pkg/initial-tls-cert.pem 2K
6752023-12-15T01:08:47.015Za pkg/initial-tls-key.pem 4K
6762023-12-15T01:08:47.015Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-12-15T01:08:47.015Z+ rmdir pkg
6782023-12-15T01:08:47.016Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-12-15T01:08:47.016Z+ pfexec mkdir -p /usr/oxide
6802023-12-15T01:08:47.016Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6812023-12-15T01:08:47.016Z+ pfexec chmod +x /usr/oxide/oxide
6822023-12-15T01:08:47.016Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6832023-12-15T01:08:55.827Z+ OMICRON_NO_UNINSTALL=1
6842023-12-15T01:08:55.827Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6852023-12-15T01:08:55.853ZDec 15 01:08:55.982 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6862023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6872023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6882023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6892023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6902023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6912023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6922023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6932023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6942023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6952023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6962023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6972023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6982023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6992023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7002023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7012023-12-15T01:08:55.853ZDec 15 01:08:55.984 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7022023-12-15T01:08:56.077ZDec 15 01:08:56.217 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7032023-12-15T01:08:56.119ZDec 15 01:08:56.258 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7042023-12-15T01:08:56.180ZDec 15 01:08:56.317 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7052023-12-15T01:08:56.307Z
7062023-12-15T01:08:56.333Zreal 0.474926422
7072023-12-15T01:08:56.333Zuser 0.100685157
7082023-12-15T01:08:56.333Zsys 1.936045276
7092023-12-15T01:08:56.334Ztrap 0.000016160
7102023-12-15T01:08:56.334Ztflt 0.000088147
7112023-12-15T01:08:56.334Zdflt 0.000728166
7122023-12-15T01:08:56.334Zkflt 0.000000000
7132023-12-15T01:08:56.334Zlock 42.547082676
7142023-12-15T01:08:56.334Zslp 0.683804578
7152023-12-15T01:08:56.334Zlat 0.003356164
7162023-12-15T01:08:56.334Zstop 0.221644230
7172023-12-15T01:08:56.334Z+ retry=0
7182023-12-15T01:08:56.334Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7192023-12-15T01:08:56.334Z+ [[ 0 -gt 30 ]]
7202023-12-15T01:08:56.334Z+ sleep 1
7212023-12-15T01:08:57.728Z+ retry=1
7222023-12-15T01:08:57.728Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232023-12-15T01:08:57.753Z+ [[ 1 -gt 30 ]]
7242023-12-15T01:08:57.766Z+ sleep 1
7252023-12-15T01:08:58.343Z+ retry=2
7262023-12-15T01:08:58.343Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272023-12-15T01:08:58.379Z+ [[ 2 -gt 30 ]]
7282023-12-15T01:08:58.380Z+ sleep 1
7292023-12-15T01:08:59.352Z+ retry=3
7302023-12-15T01:08:59.352Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312023-12-15T01:08:59.379Z+ [[ 3 -gt 30 ]]
7322023-12-15T01:08:59.379Z+ sleep 1
7332023-12-15T01:09:00.369Z+ retry=4
7342023-12-15T01:09:00.370Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352023-12-15T01:09:25.546Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7362023-12-15T01:09:25.612Zlocal v6:
7372023-12-15T01:09:25.612Zfe80::605a:15ff:fea0:c066
7382023-12-15T01:09:25.612Zfd00:99::1
7392023-12-15T01:09:25.612Zlocal v4:
7402023-12-15T01:09:25.612Z10.151.2.174
7412023-12-15T01:09:25.612Zrouter v6:
7422023-12-15T01:09:25.612Zfd00:1122:3344:2::/64 -> fe80::f8a3:11ff:fedf:4d32 (1)
7432023-12-15T01:09:25.612Zfd00:1122:3344:3::/64 -> fe80::f8a3:11ff:fedf:4d32 (1)
7442023-12-15T01:09:25.612Zfdb0:18c0:4d0d:a02a::/64 -> fe80::f8a3:11ff:fedf:4d32 (1)
7452023-12-15T01:09:25.612Zfd00:1122:3344:1::/64 -> fe80::f8a3:11ff:fedf:4d32 (1)
7462023-12-15T01:09:25.612Zfd00:1122:3344:101::/64 -> fe80::f8a3:11ff:fedf:4d32 (1)
7472023-12-15T01:09:25.612Zrouter v4_idx:
7482023-12-15T01:09:25.612Z0.0.0.0/0 -> 0
7492023-12-15T01:09:25.612Zrouter v4_routes:
7502023-12-15T01:09:25.612Z0 -> 10.151.2.100 (2)
7512023-12-15T01:09:25.612Zresolver v4:
7522023-12-15T01:09:25.612Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7532023-12-15T01:09:25.612Z10.151.2.174 -> de:3f:7a:13:a0:c7
7542023-12-15T01:09:25.612Zresolver v6:
7552023-12-15T01:09:25.612Zfe80::f8a3:11ff:fedf:4d32 -> fa:a3:11:df:4d:32
7562023-12-15T01:09:25.613Zfe80::605a:15ff:fea0:c066 -> 62:5a:15:a0:c0:66
7572023-12-15T01:09:25.613Znat v4:
7582023-12-15T01:09:25.613Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:da:28
7592023-12-15T01:09:25.613Znat_v6:
7602023-12-15T01:09:25.613Zport_mac:
7612023-12-15T01:09:25.617Z1: a8:40:25:63:e5:53
7622023-12-15T01:09:25.618Z1000: a8:40:25:63:e5:52
7632023-12-15T01:09:25.618Zproxy arp:
7642023-12-15T01:09:25.618Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7652023-12-15T01:09:25.643Z+ retry=0
7662023-12-15T01:09:25.643Z+++ zoneadm list -n
7672023-12-15T01:09:25.671Z+++ grep oxz_ntp
7682023-12-15T01:09:25.671Z++ pfexec svcs -z oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942 -Hostate oxide/ntp
7692023-12-15T01:09:26.181Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7702023-12-15T01:09:26.181Z++ true
7712023-12-15T01:09:26.181Z+ [[ '' != online ]]
7722023-12-15T01:09:26.181Z+ [[ 0 -gt 60 ]]
7732023-12-15T01:09:26.181Z+ sleep 1
7742023-12-15T01:09:27.352Z+ retry=1
7752023-12-15T01:09:27.352Z+++ zoneadm list -n
7762023-12-15T01:09:27.352Z+++ grep oxz_ntp
7772023-12-15T01:09:27.352Z++ pfexec svcs -z oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942 -Hostate oxide/ntp
7782023-12-15T01:09:27.352Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7792023-12-15T01:09:27.353Z++ true
7802023-12-15T01:09:27.353Z+ [[ '' != online ]]
7812023-12-15T01:09:27.353Z+ [[ 1 -gt 60 ]]
7822023-12-15T01:09:27.353Z+ sleep 1
7832023-12-15T01:09:27.738Z+ retry=2
7842023-12-15T01:09:27.764Z+++ zoneadm list -n
7852023-12-15T01:09:27.764Z+++ grep oxz_ntp
7862023-12-15T01:09:27.764Z++ pfexec svcs -z oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942 -Hostate oxide/ntp
7872023-12-15T01:09:27.764Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7882023-12-15T01:09:27.789Z++ true
7892023-12-15T01:09:27.789Z+ [[ '' != online ]]
7902023-12-15T01:09:27.789Z+ [[ 2 -gt 60 ]]
7912023-12-15T01:09:27.789Z+ sleep 1
7922023-12-15T01:09:28.767Z+ retry=3
7932023-12-15T01:09:28.792Z+++ zoneadm list -n
7942023-12-15T01:09:28.792Z+++ grep oxz_ntp
7952023-12-15T01:09:28.792Z++ pfexec svcs -z oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942 -Hostate oxide/ntp
7962023-12-15T01:09:28.816Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7972023-12-15T01:09:28.817Z++ true
7982023-12-15T01:09:28.817Z+ [[ '' != online ]]
7992023-12-15T01:09:28.817Z+ [[ 3 -gt 60 ]]
8002023-12-15T01:09:28.817Z+ sleep 1
8012023-12-15T01:09:30.205Z+ retry=4
8022023-12-15T01:09:30.235Z+++ zoneadm list -n
8032023-12-15T01:09:30.236Z+++ grep oxz_ntp
8042023-12-15T01:09:30.236Z++ pfexec svcs -z oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942 -Hostate oxide/ntp
8052023-12-15T01:09:30.236Z+ [[ disabled != online ]]
8062023-12-15T01:09:30.236Z+ [[ 4 -gt 60 ]]
8072023-12-15T01:09:30.237Z+ sleep 1
8082023-12-15T01:09:30.821Z+ retry=5
8092023-12-15T01:09:30.846Z+++ zoneadm list -n
8102023-12-15T01:09:30.846Z+++ grep oxz_ntp
8112023-12-15T01:09:30.846Z++ pfexec svcs -z oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942 -Hostate oxide/ntp
8122023-12-15T01:09:30.846Z+ [[ disabled != online ]]
8132023-12-15T01:09:30.846Z+ [[ 5 -gt 60 ]]
8142023-12-15T01:09:30.846Z+ sleep 1
8152023-12-15T01:09:31.895Z+ retry=6
8162023-12-15T01:09:31.937Z+++ zoneadm list -n
8172023-12-15T01:09:31.937Z+++ grep oxz_ntp
8182023-12-15T01:09:31.938Z++ pfexec svcs -z oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942 -Hostate oxide/ntp
8192023-12-15T01:09:31.938Z+ [[ disabled != online ]]
8202023-12-15T01:09:31.938Z+ [[ 6 -gt 60 ]]
8212023-12-15T01:09:31.938Z+ sleep 1
8222023-12-15T01:09:32.873Z+ retry=7
8232023-12-15T01:09:32.904Z+++ zoneadm list -n
8242023-12-15T01:09:32.904Z+++ grep oxz_ntp
8252023-12-15T01:09:32.905Z++ pfexec svcs -z oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942 -Hostate oxide/ntp
8262023-12-15T01:09:32.905Z+ [[ disabled != online ]]
8272023-12-15T01:09:32.905Z+ [[ 7 -gt 60 ]]
8282023-12-15T01:09:32.905Z+ sleep 1
8292023-12-15T01:09:33.886Z+ retry=8
8302023-12-15T01:09:33.911Z+++ zoneadm list -n
8312023-12-15T01:09:33.911Z+++ grep oxz_ntp
8322023-12-15T01:09:33.911Z++ pfexec svcs -z oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942 -Hostate oxide/ntp
8332023-12-15T01:09:33.911Z+ [[ disabled != online ]]
8342023-12-15T01:09:33.911Z+ [[ 8 -gt 60 ]]
8352023-12-15T01:09:33.915Z+ sleep 1
8362023-12-15T01:09:34.915Z+ retry=9
8372023-12-15T01:09:34.947Z+++ zoneadm list -n
8382023-12-15T01:09:34.947Z+++ grep oxz_ntp
8392023-12-15T01:09:34.947Z++ pfexec svcs -z oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942 -Hostate oxide/ntp
8402023-12-15T01:09:34.947Z+ [[ disabled != online ]]
8412023-12-15T01:09:34.948Z+ [[ 9 -gt 60 ]]
8422023-12-15T01:09:34.948Z+ sleep 1
8432023-12-15T01:09:35.934Z+ retry=10
8442023-12-15T01:09:36.442Z+++ zoneadm list -n
8452023-12-15T01:09:36.443Z+++ grep oxz_ntp
8462023-12-15T01:09:36.443Z++ pfexec svcs -z oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942 -Hostate oxide/ntp
8472023-12-15T01:09:36.443Z+ [[ disabled != online ]]
8482023-12-15T01:09:36.443Z+ [[ 10 -gt 60 ]]
8492023-12-15T01:09:36.443Z+ sleep 1
 
8502023-12-15T01:09:36.968Zsock console login: Dec 15 01:09:36 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8512023-12-15T01:09:37.418Z+ retry=11
8522023-12-15T01:09:37.418Z+++ zoneadm list -n
8532023-12-15T01:09:37.419Z+++ grep oxz_ntp
8542023-12-15T01:09:37.419Z++ pfexec svcs -z oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942 -Hostate oxide/ntp
8552023-12-15T01:09:37.419Z+ [[ online != online ]]
8562023-12-15T01:09:37.419Z+ echo 'Waited for chrony: 11s'
8572023-12-15T01:09:37.419Z+ export RUST_BACKTRACE=1
8582023-12-15T01:09:37.420ZWaited for chrony: 11s
8592023-12-15T01:09:37.420Z+ RUST_BACKTRACE=1
8602023-12-15T01:09:37.420Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8612023-12-15T01:09:37.420Z++ ./tests/bootstrap
8622023-12-15T01:09:37.420ZUsing external DNS server at 10.151.2.164:53
8632023-12-15T01:09:37.420Z2023-12-15 01:09:37.143420640 UTC: attempting to log into API
8642023-12-15T01:09:52.040Z2023-12-15 01:09:52.304751350 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8652023-12-15T01:09:53.041Z2023-12-15 01:09:53.302941792 UTC: attempting to log into API
8662023-12-15T01:10:08.074Z2023-12-15 01:10:08.294559254 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8672023-12-15T01:10:09.076Z2023-12-15 01:10:09.292497053 UTC: attempting to log into API
8682023-12-15T01:10:24.113Z2023-12-15 01:10:24.284215746 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8692023-12-15T01:10:25.110Z2023-12-15 01:10:25.282233136 UTC: attempting to log into API
8702023-12-15T01:10:40.151Z2023-12-15 01:10:40.275122127 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8712023-12-15T01:10:41.143Z2023-12-15 01:10:41.274349006 UTC: attempting to log into API
8722023-12-15T01:11:13.517Z2023-12-15 01:10:56.266458445 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8732023-12-15T01:11:14.103Z2023-12-15 01:10:57.264512145 UTC: attempting to log into API
8742023-12-15T01:11:14.103Z2023-12-15 01:11:12.256249699 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8752023-12-15T01:11:14.103Z2023-12-15 01:11:13.255275184 UTC: attempting to log into API
8762023-12-15T01:11:28.244Z2023-12-15 01:11:28.248241296 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8772023-12-15T01:11:29.257Z2023-12-15 01:11:29.247136740 UTC: attempting to log into API
8782023-12-15T01:11:48.600Z2023-12-15 01:11:44.266968038 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8792023-12-15T01:11:48.600Z2023-12-15 01:11:45.267695608 UTC: attempting to log into API
8802023-12-15T01:12:00.319Z2023-12-15 01:12:00.301349205 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8812023-12-15T01:12:01.318Z2023-12-15 01:12:01.302230792 UTC: attempting to log into API
8822023-12-15T01:12:27.366Z2023-12-15 01:12:16.335870635 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8832023-12-15T01:12:27.564Z2023-12-15 01:12:17.336684315 UTC: attempting to log into API
8842023-12-15T01:12:32.953Z2023-12-15 01:12:32.369816923 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8852023-12-15T01:12:33.389Z2023-12-15 01:12:33.370931188 UTC: attempting to log into API
8862023-12-15T01:12:48.417Z2023-12-15 01:12:48.407106183 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8872023-12-15T01:12:49.418Z2023-12-15 01:12:49.408608604 UTC: attempting to log into API
8882023-12-15T01:13:09.258Z2023-12-15 01:13:04.457590633 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8892023-12-15T01:13:09.379Z2023-12-15 01:13:05.459608572 UTC: attempting to log into API
 
8902023-12-15T01:13:28.442ZDec 15 01:13:22 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8912023-12-15T01:13:28.639Z2023-12-15 01:13:20.492285605 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8922023-12-15T01:13:28.639Z2023-12-15 01:13:21.493091986 UTC: attempting to log into API
 
8932023-12-15T01:13:30.519ZDec 15 01:13:29 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
8942023-12-15T01:13:36.133ZDec 15 01:13:35 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
8952023-12-15T01:13:36.644Z2023-12-15 01:13:36.525837291 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8962023-12-15T01:13:37.539Z2023-12-15 01:13:37.526774644 UTC: attempting to log into API
 
8972023-12-15T01:13:43.669ZDec 15 01:13:41 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
8982023-12-15T01:13:49.481ZDec 15 01:13:48 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
8992023-12-15T01:13:52.575Z2023-12-15 01:13:52.563394947 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9002023-12-15T01:13:53.577Z2023-12-15 01:13:53.565339656 UTC: attempting to log into API
9012023-12-15T01:14:08.609Z2023-12-15 01:14:08.597729306 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9022023-12-15T01:14:09.615Z2023-12-15 01:14:09.598645459 UTC: attempting to log into API
9032023-12-15T01:14:10.392Z2023-12-15 01:14:10.380937960 UTC: login succeeded
9042023-12-15T01:14:10.425Zwaiting for nexus to come up...
9052023-12-15T01:14:10.493Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9062023-12-15T01:14:10.554Zsetting up quotas...
9072023-12-15T01:14:10.602Zensuring datasets are ready...
9082023-12-15T01:14:13.015Zlet's roll.
9092023-12-15T01:14:13.040Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
9102023-12-15T01:14:13.041ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9112023-12-15T01:14:13.041ZOXIDE_TOKEN=oxide-token-8e9f457e04154d5f3e35d0d0206f40ce782b1afa'
9122023-12-15T01:14:13.041Z++ OXIDE_HOST=https://recovery.sys.oxide.test
9132023-12-15T01:14:13.041Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9142023-12-15T01:14:13.041Z++ OXIDE_TOKEN=oxide-token-8e9f457e04154d5f3e35d0d0206f40ce782b1afa
9152023-12-15T01:14:13.041Z+ export OXIDE_HOST OXIDE_TOKEN
9162023-12-15T01:14:13.041Z+ retry=0
9172023-12-15T01:14:13.041Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9182023-12-15T01:14:13.041Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9192023-12-15T01:14:13.041Z+ [[ 0 -gt 60 ]]
9202023-12-15T01:14:13.041Z+ sleep 1
9212023-12-15T01:14:14.042Z+ retry=1
9222023-12-15T01:14:14.042Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9232023-12-15T01:14:14.072Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9242023-12-15T01:14:14.073Z+ [[ 1 -gt 60 ]]
9252023-12-15T01:14:14.073Z+ sleep 1
9262023-12-15T01:14:15.059Z+ retry=2
9272023-12-15T01:14:15.059Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9282023-12-15T01:14:15.085Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9292023-12-15T01:14:15.085Z+ [[ 2 -gt 60 ]]
9302023-12-15T01:14:15.085Z+ sleep 1
9312023-12-15T01:14:16.079Z+ retry=3
9322023-12-15T01:14:16.079Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9332023-12-15T01:14:16.107Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9342023-12-15T01:14:16.108Z+ [[ 3 -gt 60 ]]
9352023-12-15T01:14:16.108Z+ sleep 1
9362023-12-15T01:14:17.090Z+ retry=4
9372023-12-15T01:14:17.090Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9382023-12-15T01:14:17.115Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9392023-12-15T01:14:17.115Z+ [[ 4 -gt 60 ]]
9402023-12-15T01:14:17.118Z+ sleep 1
9412023-12-15T01:14:18.107Z+ retry=5
9422023-12-15T01:14:18.107Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9432023-12-15T01:14:18.133Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9442023-12-15T01:14:18.133Z+ [[ 5 -gt 60 ]]
9452023-12-15T01:14:18.134Z+ sleep 1
9462023-12-15T01:14:19.122Z+ retry=6
9472023-12-15T01:14:19.122Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9482023-12-15T01:14:19.155Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9492023-12-15T01:14:19.155Z+ [[ 6 -gt 60 ]]
9502023-12-15T01:14:19.155Z+ sleep 1
9512023-12-15T01:14:20.139Z+ retry=7
9522023-12-15T01:14:20.140Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9532023-12-15T01:14:20.170Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9542023-12-15T01:14:20.170Z+ [[ 7 -gt 60 ]]
9552023-12-15T01:14:20.170Z+ sleep 1
9562023-12-15T01:14:21.155Z+ retry=8
9572023-12-15T01:14:21.155Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9582023-12-15T01:14:21.181Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9592023-12-15T01:14:21.181Z+ [[ 8 -gt 60 ]]
9602023-12-15T01:14:21.182Z+ sleep 1
9612023-12-15T01:14:22.171Z+ retry=9
9622023-12-15T01:14:22.171Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9632023-12-15T01:14:22.200Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9642023-12-15T01:14:22.200Z+ [[ 9 -gt 60 ]]
9652023-12-15T01:14:22.200Z+ sleep 1
9662023-12-15T01:14:23.186Z+ retry=10
9672023-12-15T01:14:23.186Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9682023-12-15T01:14:23.212Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9692023-12-15T01:14:23.212Z+ [[ 10 -gt 60 ]]
9702023-12-15T01:14:23.212Z+ sleep 1
9712023-12-15T01:14:24.202Z+ retry=11
9722023-12-15T01:14:24.203Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9732023-12-15T01:14:24.258Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9742023-12-15T01:14:24.258Z+ [[ 11 -gt 60 ]]
9752023-12-15T01:14:24.259Z+ sleep 1
9762023-12-15T01:14:25.295Z+ retry=12
9772023-12-15T01:14:25.295Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9782023-12-15T01:14:25.381Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9792023-12-15T01:14:25.382Z+ [[ 12 -gt 60 ]]
9802023-12-15T01:14:25.382Z+ sleep 1
9812023-12-15T01:14:26.234Z+ retry=13
9822023-12-15T01:14:26.234Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9832023-12-15T01:14:26.267Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9842023-12-15T01:14:26.267Z+ [[ 13 -gt 60 ]]
9852023-12-15T01:14:26.267Z+ sleep 1
9862023-12-15T01:14:35.377Z+ retry=14
9872023-12-15T01:14:35.377Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9882023-12-15T01:14:35.483Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9892023-12-15T01:14:35.483Z+ [[ 14 -gt 60 ]]
9902023-12-15T01:14:35.483Z+ sleep 1
9912023-12-15T01:14:35.483Z+ retry=15
9922023-12-15T01:14:35.483Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9932023-12-15T01:14:35.483Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9942023-12-15T01:14:35.483Z+ [[ 15 -gt 60 ]]
9952023-12-15T01:14:35.483Z+ sleep 1
9962023-12-15T01:14:35.483Z+ retry=16
9972023-12-15T01:14:35.484Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9982023-12-15T01:14:35.484Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9992023-12-15T01:14:35.484Z+ [[ 16 -gt 60 ]]
10002023-12-15T01:14:35.484Z+ sleep 1
10012023-12-15T01:14:35.484Z+ retry=17
10022023-12-15T01:14:35.484Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10032023-12-15T01:14:35.484Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10042023-12-15T01:14:35.484Z+ [[ 17 -gt 60 ]]
10052023-12-15T01:14:35.484Z+ sleep 1
10062023-12-15T01:14:35.484Z+ retry=18
10072023-12-15T01:14:35.484Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10082023-12-15T01:14:35.484Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10092023-12-15T01:14:35.484Z+ [[ 18 -gt 60 ]]
10102023-12-15T01:14:35.485Z+ sleep 1
10112023-12-15T01:14:35.540Z+ retry=19
10122023-12-15T01:14:35.540Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10132023-12-15T01:14:35.540Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10142023-12-15T01:14:35.540Z+ [[ 19 -gt 60 ]]
10152023-12-15T01:14:35.540Z+ sleep 1
10162023-12-15T01:14:35.540Z+ retry=20
10172023-12-15T01:14:35.540Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10182023-12-15T01:14:35.540Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10192023-12-15T01:14:35.540Z+ [[ 20 -gt 60 ]]
10202023-12-15T01:14:35.540Z+ sleep 1
10212023-12-15T01:14:35.540Z+ retry=21
10222023-12-15T01:14:35.540Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10232023-12-15T01:14:35.540Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10242023-12-15T01:14:35.541Z+ [[ 21 -gt 60 ]]
10252023-12-15T01:14:35.553Z+ sleep 1
10262023-12-15T01:14:35.553Z+ retry=22
10272023-12-15T01:14:35.553Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10282023-12-15T01:14:35.553Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10292023-12-15T01:14:35.553Z+ [[ 22 -gt 60 ]]
10302023-12-15T01:14:35.553Z+ sleep 1
10312023-12-15T01:14:36.390Z+ retry=23
10322023-12-15T01:14:36.390Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10332023-12-15T01:14:36.430Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10342023-12-15T01:14:36.431Z+ [[ 23 -gt 60 ]]
10352023-12-15T01:14:36.431Z+ sleep 1
10362023-12-15T01:14:37.421Z+ retry=24
10372023-12-15T01:14:37.421Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10382023-12-15T01:14:37.449Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10392023-12-15T01:14:37.451Z+ [[ 24 -gt 60 ]]
10402023-12-15T01:14:37.452Z+ sleep 1
10412023-12-15T01:14:38.538Z+ retry=25
10422023-12-15T01:14:38.538Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10432023-12-15T01:14:38.566Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10442023-12-15T01:14:38.566Z+ [[ 25 -gt 60 ]]
10452023-12-15T01:14:38.566Z+ sleep 1
10462023-12-15T01:14:39.437Z+ retry=26
10472023-12-15T01:14:39.437Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10482023-12-15T01:14:39.465Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10492023-12-15T01:14:39.465Z+ [[ 26 -gt 60 ]]
10502023-12-15T01:14:39.465Z+ sleep 1
10512023-12-15T01:14:40.454Z+ retry=27
10522023-12-15T01:14:40.454Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10532023-12-15T01:14:40.493Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10542023-12-15T01:14:40.493Z+ [[ 27 -gt 60 ]]
10552023-12-15T01:14:40.533Z+ sleep 1
10562023-12-15T01:14:41.472Z+ retry=28
10572023-12-15T01:14:41.472Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10582023-12-15T01:14:41.509Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10592023-12-15T01:14:41.509Z+ [[ 28 -gt 60 ]]
10602023-12-15T01:14:41.509Z+ sleep 1
10612023-12-15T01:14:42.485Z+ retry=29
10622023-12-15T01:14:42.485Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10632023-12-15T01:14:42.510Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10642023-12-15T01:14:42.511Z+ [[ 29 -gt 60 ]]
10652023-12-15T01:14:42.511Z+ sleep 1
10662023-12-15T01:14:43.500Z+ retry=30
10672023-12-15T01:14:43.500Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10682023-12-15T01:14:43.525Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10692023-12-15T01:14:43.525Z+ [[ 30 -gt 60 ]]
10702023-12-15T01:14:43.525Z+ sleep 1
10712023-12-15T01:14:44.596Z+ retry=31
10722023-12-15T01:14:44.596Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10732023-12-15T01:14:44.688Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10742023-12-15T01:14:44.688Z+ [[ 31 -gt 60 ]]
10752023-12-15T01:14:44.688Z+ sleep 1
10762023-12-15T01:14:45.539Z+ retry=32
10772023-12-15T01:14:45.539Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10782023-12-15T01:14:45.564Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10792023-12-15T01:14:45.564Z+ [[ 32 -gt 60 ]]
10802023-12-15T01:14:45.565Z+ sleep 1
10812023-12-15T01:14:46.547Z+ retry=33
10822023-12-15T01:14:46.547Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10832023-12-15T01:14:46.572Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10842023-12-15T01:14:46.572Z+ [[ 33 -gt 60 ]]
10852023-12-15T01:14:46.572Z+ sleep 1
10862023-12-15T01:14:47.564Z+ retry=34
10872023-12-15T01:14:47.564Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10882023-12-15T01:14:47.589Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10892023-12-15T01:14:47.589Z+ [[ 34 -gt 60 ]]
10902023-12-15T01:14:47.590Z+ sleep 1
10912023-12-15T01:14:48.580Z+ retry=35
10922023-12-15T01:14:48.580Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10932023-12-15T01:14:48.605Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10942023-12-15T01:14:48.605Z+ [[ 35 -gt 60 ]]
10952023-12-15T01:14:48.605Z+ sleep 1
10962023-12-15T01:14:49.595Z+ retry=36
10972023-12-15T01:14:49.595Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10982023-12-15T01:14:49.623Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10992023-12-15T01:14:49.623Z+ [[ 36 -gt 60 ]]
11002023-12-15T01:14:49.624Z+ sleep 1
11012023-12-15T01:14:50.611Z+ retry=37
11022023-12-15T01:14:50.611Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11032023-12-15T01:14:50.693Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
11042023-12-15T01:14:51.316Z{"status":"ok"}success
11052023-12-15T01:14:51.316ZProject {
11062023-12-15T01:14:51.316Z description: "some images",
11072023-12-15T01:14:51.316Z id: c20b6ccf-1e1b-4c55-9253-583760153338,
11082023-12-15T01:14:51.316Z name: Name(
11092023-12-15T01:14:51.316Z "images",
11102023-12-15T01:14:51.321Z ),
11112023-12-15T01:14:51.321Z time_created: 2023-12-15T01:14:50.824400Z,
11122023-12-15T01:14:51.322Z time_modified: 2023-12-15T01:14:50.824400Z,
11132023-12-15T01:14:51.322Z}
11142023-12-15T01:14:51.759Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
11152023-12-15T01:20:24.663Zdone!
11162023-12-15T01:20:24.768Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
11172023-12-15T01:20:24.873Zsuccess
11182023-12-15T01:20:24.873ZImage {
11192023-12-15T01:20:24.874Z block_size: ByteCount(
11202023-12-15T01:20:24.874Z 512,
11212023-12-15T01:20:24.874Z ),
11222023-12-15T01:20:24.874Z description: "debian 11 original base image",
11232023-12-15T01:20:24.874Z digest: None,
11242023-12-15T01:20:24.874Z id: b5455b82-c833-4b3b-8d2b-b2ab5dd47532,
11252023-12-15T01:20:24.924Z name: Name(
11262023-12-15T01:20:24.924Z "debian11",
11272023-12-15T01:20:24.924Z ),
11282023-12-15T01:20:24.924Z os: "debian",
11292023-12-15T01:20:24.924Z project_id: None,
11302023-12-15T01:20:24.924Z size: ByteCount(
11312023-12-15T01:20:24.924Z 2147483648,
11322023-12-15T01:20:24.926Z ),
11332023-12-15T01:20:24.926Z time_created: 2023-12-15T01:20:24.594940Z,
11342023-12-15T01:20:24.926Z time_modified: 2023-12-15T01:20:24.851465Z,
11352023-12-15T01:20:24.926Z url: None,
11362023-12-15T01:20:24.926Z version: "11",
11372023-12-15T01:20:24.926Z}
11382023-12-15T01:20:24.926Z+ rm ./tests/bootstrap
11392023-12-15T01:20:24.952Z+ for test_bin in tests/*
11402023-12-15T01:20:24.952Z+ ./tests/bootstrap-eae955b6030e5419
11412023-12-15T01:20:24.976Z
11422023-12-15T01:20:24.976Zrunning 0 tests
11432023-12-15T01:20:24.976Z
11442023-12-15T01:20:24.977Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
11452023-12-15T01:20:24.977Z
11462023-12-15T01:20:24.977Z+ for test_bin in tests/*
11472023-12-15T01:20:24.977Z+ ./tests/end_to_end_tests-114e4ef0b9f9b2b3
11482023-12-15T01:20:25.002Z
11492023-12-15T01:20:25.002Zrunning 2 tests
11502023-12-15T01:20:25.206Ztest no_spoof::no_spoof_here ... ok
11512023-12-15T01:20:59.109Ztest instance_launch::instance_launch ... ok
11522023-12-15T01:20:59.109Z
11532023-12-15T01:20:59.110Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 34.12s
11542023-12-15T01:20:59.110Z
11552023-12-15T01:20:59.136Z+ _exit_trap
11562023-12-15T01:20:59.136Z+ local status=0
11572023-12-15T01:20:59.136Z+ [[ 0 -eq 0 ]]
11582023-12-15T01:20:59.136Z+ exit 0
11592023-12-15T01:20:59.136Zprocess exited: duration 792508 ms, exit code 0
 
11602023-12-15T01:20:59.249Zfound 99 output files
11612023-12-15T01:20:59.250Zuploading: /var/svc/log/oxide-sled-agent:default.log (767712 bytes)
11622023-12-15T01:21:00.270Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 767712 -> 768110
11632023-12-15T01:21:01.345Zuploaded: /var/svc/log/oxide-sled-agent:default.log
11642023-12-15T01:21:01.369Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c8d17562-2f06-40f3-9824-1f4cb22a7b12/root/var/svc/log/oxide-internal_dns:default.log (22524 bytes)
11652023-12-15T01:21:02.510Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c8d17562-2f06-40f3-9824-1f4cb22a7b12/root/var/svc/log/oxide-internal_dns:default.log
11662023-12-15T01:21:02.510Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c8d17562-2f06-40f3-9824-1f4cb22a7b12/root/var/svc/log/oxide-internal_dns:default.log.0 (108301 bytes)
11672023-12-15T01:21:03.930Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c8d17562-2f06-40f3-9824-1f4cb22a7b12/root/var/svc/log/oxide-internal_dns:default.log.0
11682023-12-15T01:21:03.930Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1756a188-a654-4c2f-b5bb-e2f18ad4474e/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11692023-12-15T01:21:05.004Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1756a188-a654-4c2f-b5bb-e2f18ad4474e/root/var/svc/log/oxide-crucible-agent:default.log
11702023-12-15T01:21:05.004Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1756a188-a654-4c2f-b5bb-e2f18ad4474e/root/var/svc/log/oxide-crucible-agent:default.log.0 (3433 bytes)
11712023-12-15T01:21:06.107Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1756a188-a654-4c2f-b5bb-e2f18ad4474e/root/var/svc/log/oxide-crucible-agent:default.log.0
11722023-12-15T01:21:06.107Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1756a188-a654-4c2f-b5bb-e2f18ad4474e/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11732023-12-15T01:21:08.209Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1756a188-a654-4c2f-b5bb-e2f18ad4474e/root/var/svc/log/oxide-crucible-downstairs:default.log
11742023-12-15T01:21:08.209Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1756a188-a654-4c2f-b5bb-e2f18ad4474e/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11752023-12-15T01:21:08.285Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1756a188-a654-4c2f-b5bb-e2f18ad4474e/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11762023-12-15T01:21:08.310Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17ef7150-35b2-46ab-918a-0b0af0a9b49d/root/var/svc/log/oxide-crucible-agent:default.log (16612 bytes)
11772023-12-15T01:21:09.428Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17ef7150-35b2-46ab-918a-0b0af0a9b49d/root/var/svc/log/oxide-crucible-agent:default.log
11782023-12-15T01:21:09.428Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17ef7150-35b2-46ab-918a-0b0af0a9b49d/root/var/svc/log/oxide-crucible-agent:default.log.0 (3434 bytes)
11792023-12-15T01:21:10.531Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17ef7150-35b2-46ab-918a-0b0af0a9b49d/root/var/svc/log/oxide-crucible-agent:default.log.0
11802023-12-15T01:21:10.531Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17ef7150-35b2-46ab-918a-0b0af0a9b49d/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11812023-12-15T01:21:11.607Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17ef7150-35b2-46ab-918a-0b0af0a9b49d/root/var/svc/log/oxide-crucible-downstairs:default.log
11822023-12-15T01:21:11.610Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17ef7150-35b2-46ab-918a-0b0af0a9b49d/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11832023-12-15T01:21:12.705Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17ef7150-35b2-46ab-918a-0b0af0a9b49d/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11842023-12-15T01:21:12.705Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17ef7150-35b2-46ab-918a-0b0af0a9b49d/root/var/svc/log/oxide-crucible-downstairs:downstairs-11f0680f-0e5a-490a-9cf7-6a0da2b60fd1.log (5532 bytes)
11852023-12-15T01:21:13.825Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17ef7150-35b2-46ab-918a-0b0af0a9b49d/root/var/svc/log/oxide-crucible-downstairs:downstairs-11f0680f-0e5a-490a-9cf7-6a0da2b60fd1.log
11862023-12-15T01:21:13.829Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17ef7150-35b2-46ab-918a-0b0af0a9b49d/root/var/svc/log/oxide-crucible-downstairs:downstairs-796f4442-cd45-4333-b0b3-3865db73e18f.log (2944 bytes)
11872023-12-15T01:21:14.922Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_17ef7150-35b2-46ab-918a-0b0af0a9b49d/root/var/svc/log/oxide-crucible-downstairs:downstairs-796f4442-cd45-4333-b0b3-3865db73e18f.log
11882023-12-15T01:21:14.922Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8b0e0450-abc8-4227-b368-5e4185b52841/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11892023-12-15T01:21:15.994Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8b0e0450-abc8-4227-b368-5e4185b52841/root/var/svc/log/oxide-crucible-agent:default.log
11902023-12-15T01:21:15.994Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8b0e0450-abc8-4227-b368-5e4185b52841/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11912023-12-15T01:21:17.093Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8b0e0450-abc8-4227-b368-5e4185b52841/root/var/svc/log/oxide-crucible-agent:default.log.0
11922023-12-15T01:21:17.093Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8b0e0450-abc8-4227-b368-5e4185b52841/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11932023-12-15T01:21:18.163Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8b0e0450-abc8-4227-b368-5e4185b52841/root/var/svc/log/oxide-crucible-downstairs:default.log
11942023-12-15T01:21:18.189Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8b0e0450-abc8-4227-b368-5e4185b52841/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11952023-12-15T01:21:19.265Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8b0e0450-abc8-4227-b368-5e4185b52841/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11962023-12-15T01:21:19.266Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_cd0082d8-88ea-4123-9209-754395e437e9/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11972023-12-15T01:21:20.337Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_cd0082d8-88ea-4123-9209-754395e437e9/root/var/svc/log/oxide-crucible-pantry:default.log
11982023-12-15T01:21:20.338Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_cd0082d8-88ea-4123-9209-754395e437e9/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2336 bytes)
11992023-12-15T01:21:21.432Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_cd0082d8-88ea-4123-9209-754395e437e9/root/var/svc/log/oxide-crucible-pantry:default.log.0
12002023-12-15T01:21:21.433Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-agent:default.log (7758 bytes)
12012023-12-15T01:21:22.551Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-agent:default.log
12022023-12-15T01:21:22.551Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-agent:default.log.0 (19614 bytes)
12032023-12-15T01:21:23.692Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-agent:default.log.0
12042023-12-15T01:21:23.693Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12052023-12-15T01:21:24.764Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-downstairs:default.log
12062023-12-15T01:21:24.765Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12072023-12-15T01:21:25.859Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12082023-12-15T01:21:25.859Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-downstairs:downstairs-21182df6-e0b8-4b03-bc91-5a9573adff3f.log (0 bytes)
12092023-12-15T01:21:26.931Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-downstairs:downstairs-21182df6-e0b8-4b03-bc91-5a9573adff3f.log
12102023-12-15T01:21:26.931Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-downstairs:downstairs-21182df6-e0b8-4b03-bc91-5a9573adff3f.log.0 (3050 bytes)
12112023-12-15T01:21:28.028Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-downstairs:downstairs-21182df6-e0b8-4b03-bc91-5a9573adff3f.log.0
12122023-12-15T01:21:28.028Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-downstairs:downstairs-33e518c6-a754-452c-8d74-0821ff294795.log (1907 bytes)
12132023-12-15T01:21:29.127Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-downstairs:downstairs-33e518c6-a754-452c-8d74-0821ff294795.log
12142023-12-15T01:21:29.127Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-downstairs:downstairs-33e518c6-a754-452c-8d74-0821ff294795.log.0 (4527 bytes)
12152023-12-15T01:21:30.269Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-downstairs:downstairs-33e518c6-a754-452c-8d74-0821ff294795.log.0
12162023-12-15T01:21:30.269Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-downstairs:snapshot-33e518c6-a754-452c-8d74-0821ff294795-f0243d62-ac0d-4506-b553-a85ecac70171.log (5177 bytes)
12172023-12-15T01:21:31.365Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_04219130-934c-4126-868a-697ce1f64c38/root/var/svc/log/oxide-crucible-downstairs:snapshot-33e518c6-a754-452c-8d74-0821ff294795-f0243d62-ac0d-4506-b553-a85ecac70171.log
12182023-12-15T01:21:31.365Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_cc39462d-3d1b-41ac-8720-1bae501935df/root/var/svc/log/oxide-internal_dns:default.log (22533 bytes)
12192023-12-15T01:21:32.506Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_cc39462d-3d1b-41ac-8720-1bae501935df/root/var/svc/log/oxide-internal_dns:default.log
12202023-12-15T01:21:32.506Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_cc39462d-3d1b-41ac-8720-1bae501935df/root/var/svc/log/oxide-internal_dns:default.log.0 (107157 bytes)
12212023-12-15T01:21:33.715Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_cc39462d-3d1b-41ac-8720-1bae501935df/root/var/svc/log/oxide-internal_dns:default.log.0
12222023-12-15T01:21:33.715Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_00da64ed-baa0-403b-a869-43a61433bbd9/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12232023-12-15T01:21:34.787Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_00da64ed-baa0-403b-a869-43a61433bbd9/root/var/svc/log/oxide-cockroachdb:default.log
12242023-12-15T01:21:34.787Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_00da64ed-baa0-403b-a869-43a61433bbd9/root/var/svc/log/oxide-cockroachdb:default.log.0 (5387 bytes)
12252023-12-15T01:21:35.913Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_00da64ed-baa0-403b-a869-43a61433bbd9/root/var/svc/log/oxide-cockroachdb:default.log.0
12262023-12-15T01:21:35.913Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7fc3d62c-c7ff-41aa-be0f-d678bc85f00a/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12272023-12-15T01:21:37.003Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7fc3d62c-c7ff-41aa-be0f-d678bc85f00a/root/var/svc/log/oxide-cockroachdb:default.log
12282023-12-15T01:21:37.003Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7fc3d62c-c7ff-41aa-be0f-d678bc85f00a/root/var/svc/log/oxide-cockroachdb:default.log.0 (5413 bytes)
12292023-12-15T01:21:38.131Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7fc3d62c-c7ff-41aa-be0f-d678bc85f00a/root/var/svc/log/oxide-cockroachdb:default.log.0
12302023-12-15T01:21:38.131Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_aa7234de-3d19-4031-963b-712389d0323e/root/var/svc/log/oxide-internal_dns:default.log (22530 bytes)
12312023-12-15T01:21:39.270Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_aa7234de-3d19-4031-963b-712389d0323e/root/var/svc/log/oxide-internal_dns:default.log
12322023-12-15T01:21:39.273Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_aa7234de-3d19-4031-963b-712389d0323e/root/var/svc/log/oxide-internal_dns:default.log.0 (111146 bytes)
12332023-12-15T01:21:40.479Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_aa7234de-3d19-4031-963b-712389d0323e/root/var/svc/log/oxide-internal_dns:default.log.0
12342023-12-15T01:21:40.481Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_127efcba-5993-4577-bcb2-f4a98a4520bb/root/var/svc/log/oxide-oximeter:default.log (291980 bytes)
12352023-12-15T01:21:40.710Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_127efcba-5993-4577-bcb2-f4a98a4520bb/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 291980 -> 325010
12362023-12-15T01:21:41.787Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_127efcba-5993-4577-bcb2-f4a98a4520bb/root/var/svc/log/oxide-oximeter:default.log
12372023-12-15T01:21:41.787Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_127efcba-5993-4577-bcb2-f4a98a4520bb/root/var/svc/log/oxide-oximeter:default.log.0 (41228 bytes)
12382023-12-15T01:21:42.957Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_127efcba-5993-4577-bcb2-f4a98a4520bb/root/var/svc/log/oxide-oximeter:default.log.0
12392023-12-15T01:21:42.957Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_700e7b49-f840-40d5-92e8-83ab758ea8a7/root/var/svc/log/oxide-nexus:default.log (558580 bytes)
12402023-12-15T01:21:43.761Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_700e7b49-f840-40d5-92e8-83ab758ea8a7/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 558580 -> 567865
12412023-12-15T01:21:44.849Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_700e7b49-f840-40d5-92e8-83ab758ea8a7/root/var/svc/log/oxide-nexus:default.log
12422023-12-15T01:21:44.849Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_700e7b49-f840-40d5-92e8-83ab758ea8a7/root/var/svc/log/oxide-nexus:default.log.0 (2030697 bytes)
12432023-12-15T01:21:46.676Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_700e7b49-f840-40d5-92e8-83ab758ea8a7/root/var/svc/log/oxide-nexus:default.log.0
12442023-12-15T01:21:46.678Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_7198c8dc-786b-4865-a05e-06e4223227c4/root/var/svc/log/oxide-nexus:default.log (431170 bytes)
12452023-12-15T01:21:46.921Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_7198c8dc-786b-4865-a05e-06e4223227c4/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 431170 -> 439267
12462023-12-15T01:21:48.011Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_7198c8dc-786b-4865-a05e-06e4223227c4/root/var/svc/log/oxide-nexus:default.log
12472023-12-15T01:21:48.012Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_7198c8dc-786b-4865-a05e-06e4223227c4/root/var/svc/log/oxide-nexus:default.log.0 (572882 bytes)
12482023-12-15T01:21:49.778Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_7198c8dc-786b-4865-a05e-06e4223227c4/root/var/svc/log/oxide-nexus:default.log.0
12492023-12-15T01:21:49.779Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_23e76ebf-1083-415c-b1cc-78b70ea0ee28/root/var/svc/log/oxide-clickhouse:default.log (1055849 bytes)
12502023-12-15T01:21:50.100Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_23e76ebf-1083-415c-b1cc-78b70ea0ee28/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 1055849 -> 1199568
12512023-12-15T01:21:51.232Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_23e76ebf-1083-415c-b1cc-78b70ea0ee28/root/var/svc/log/oxide-clickhouse:default.log
12522023-12-15T01:21:51.232Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_23e76ebf-1083-415c-b1cc-78b70ea0ee28/root/var/svc/log/oxide-clickhouse:default.log.0 (190342 bytes)
12532023-12-15T01:21:53.643Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_clickhouse_23e76ebf-1083-415c-b1cc-78b70ea0ee28/root/var/svc/log/oxide-clickhouse:default.log.0
12542023-12-15T01:21:53.643Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_9f6904ab-71f4-4e02-8b40-0deb1fbe77aa/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12552023-12-15T01:21:55.545Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_9f6904ab-71f4-4e02-8b40-0deb1fbe77aa/root/var/svc/log/oxide-cockroachdb:default.log
12562023-12-15T01:21:55.545Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_9f6904ab-71f4-4e02-8b40-0deb1fbe77aa/root/var/svc/log/oxide-cockroachdb:default.log.0 (5413 bytes)
12572023-12-15T01:22:18.303Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_9f6904ab-71f4-4e02-8b40-0deb1fbe77aa/root/var/svc/log/oxide-cockroachdb:default.log.0
12582023-12-15T01:22:18.303Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_3fd5ef26-c842-4d29-8079-afd0b75a8502/root/var/svc/log/oxide-crucible-pantry:default.log (1180220 bytes)
12592023-12-15T01:22:20.750Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_3fd5ef26-c842-4d29-8079-afd0b75a8502/root/var/svc/log/oxide-crucible-pantry:default.log
12602023-12-15T01:22:20.810Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_3fd5ef26-c842-4d29-8079-afd0b75a8502/root/var/svc/log/oxide-crucible-pantry:default.log.0 (47847 bytes)
12612023-12-15T01:22:21.962Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_3fd5ef26-c842-4d29-8079-afd0b75a8502/root/var/svc/log/oxide-crucible-pantry:default.log.0
12622023-12-15T01:22:21.962Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_326e1e2f-55db-4ad4-aa67-d2bb64a872b8/root/var/svc/log/oxide-external_dns:default.log (22660 bytes)
12632023-12-15T01:22:22.087Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_326e1e2f-55db-4ad4-aa67-d2bb64a872b8/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 22660 -> 26437
12642023-12-15T01:22:23.623Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_326e1e2f-55db-4ad4-aa67-d2bb64a872b8/root/var/svc/log/oxide-external_dns:default.log
12652023-12-15T01:22:23.623Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_326e1e2f-55db-4ad4-aa67-d2bb64a872b8/root/var/svc/log/oxide-external_dns:default.log.0 (8059 bytes)
12662023-12-15T01:22:25.996Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_326e1e2f-55db-4ad4-aa67-d2bb64a872b8/root/var/svc/log/oxide-external_dns:default.log.0
12672023-12-15T01:22:26.031Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_b5d8727f-e19d-4672-a2f8-930bec445061/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12682023-12-15T01:22:27.094Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_b5d8727f-e19d-4672-a2f8-930bec445061/root/var/svc/log/oxide-cockroachdb:default.log
12692023-12-15T01:22:27.094Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_b5d8727f-e19d-4672-a2f8-930bec445061/root/var/svc/log/oxide-cockroachdb:default.log.0 (5412 bytes)
12702023-12-15T01:22:28.202Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_b5d8727f-e19d-4672-a2f8-930bec445061/root/var/svc/log/oxide-cockroachdb:default.log.0
12712023-12-15T01:22:28.228Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_df48995f-8d48-4496-b8a4-000ccf1c6fe7/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
12722023-12-15T01:22:29.282Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_df48995f-8d48-4496-b8a4-000ccf1c6fe7/root/var/svc/log/oxide-crucible-pantry:default.log
12732023-12-15T01:22:29.315Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_df48995f-8d48-4496-b8a4-000ccf1c6fe7/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2332 bytes)
12742023-12-15T01:22:30.392Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_df48995f-8d48-4496-b8a4-000ccf1c6fe7/root/var/svc/log/oxide-crucible-pantry:default.log.0
12752023-12-15T01:22:30.392Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2217a76b-b58b-4ada-a4c4-e84eca8d38a4/root/var/svc/log/oxide-crucible-agent:default.log (6360 bytes)
12762023-12-15T01:22:31.505Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2217a76b-b58b-4ada-a4c4-e84eca8d38a4/root/var/svc/log/oxide-crucible-agent:default.log
12772023-12-15T01:22:31.541Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2217a76b-b58b-4ada-a4c4-e84eca8d38a4/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
12782023-12-15T01:22:32.609Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2217a76b-b58b-4ada-a4c4-e84eca8d38a4/root/var/svc/log/oxide-crucible-agent:default.log.0
12792023-12-15T01:22:32.609Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2217a76b-b58b-4ada-a4c4-e84eca8d38a4/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12802023-12-15T01:22:33.681Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2217a76b-b58b-4ada-a4c4-e84eca8d38a4/root/var/svc/log/oxide-crucible-downstairs:default.log
12812023-12-15T01:22:33.681Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2217a76b-b58b-4ada-a4c4-e84eca8d38a4/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12822023-12-15T01:22:35.199Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2217a76b-b58b-4ada-a4c4-e84eca8d38a4/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12832023-12-15T01:22:35.199Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2217a76b-b58b-4ada-a4c4-e84eca8d38a4/root/var/svc/log/oxide-crucible-downstairs:downstairs-5a546296-b061-438b-813b-17940e95615c.log (2944 bytes)
12842023-12-15T01:22:36.328Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2217a76b-b58b-4ada-a4c4-e84eca8d38a4/root/var/svc/log/oxide-crucible-downstairs:downstairs-5a546296-b061-438b-813b-17940e95615c.log
12852023-12-15T01:22:36.357Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c710b263-6e30-4f78-bb21-7a9e382a9c52/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
12862023-12-15T01:22:37.822Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c710b263-6e30-4f78-bb21-7a9e382a9c52/root/var/svc/log/oxide-crucible-agent:default.log
12872023-12-15T01:22:37.822Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c710b263-6e30-4f78-bb21-7a9e382a9c52/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
12882023-12-15T01:22:39.421Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c710b263-6e30-4f78-bb21-7a9e382a9c52/root/var/svc/log/oxide-crucible-agent:default.log.0
12892023-12-15T01:22:39.422Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c710b263-6e30-4f78-bb21-7a9e382a9c52/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12902023-12-15T01:22:40.492Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c710b263-6e30-4f78-bb21-7a9e382a9c52/root/var/svc/log/oxide-crucible-downstairs:default.log
12912023-12-15T01:22:40.492Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c710b263-6e30-4f78-bb21-7a9e382a9c52/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12922023-12-15T01:22:41.606Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c710b263-6e30-4f78-bb21-7a9e382a9c52/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12932023-12-15T01:22:41.607Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942/root/var/svc/log/oxide-ntp:default.log (0 bytes)
12942023-12-15T01:22:43.156Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942/root/var/svc/log/oxide-ntp:default.log
12952023-12-15T01:22:43.156Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942/root/var/svc/log/oxide-ntp:default.log.0 (1250 bytes)
12962023-12-15T01:22:44.274Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942/root/var/svc/log/oxide-ntp:default.log.0
12972023-12-15T01:22:44.274Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c80d7f41-e0c5-46c6-b353-df45d1040963/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12982023-12-15T01:22:45.348Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c80d7f41-e0c5-46c6-b353-df45d1040963/root/var/svc/log/oxide-cockroachdb:default.log
12992023-12-15T01:22:45.348Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c80d7f41-e0c5-46c6-b353-df45d1040963/root/var/svc/log/oxide-cockroachdb:default.log.0 (5413 bytes)
13002023-12-15T01:22:47.292Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c80d7f41-e0c5-46c6-b353-df45d1040963/root/var/svc/log/oxide-cockroachdb:default.log.0
13012023-12-15T01:22:47.292Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-agent:default.log (7759 bytes)
13022023-12-15T01:22:48.892Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-agent:default.log
13032023-12-15T01:22:48.988Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-agent:default.log.0 (19609 bytes)
13042023-12-15T01:22:50.097Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-agent:default.log.0
13052023-12-15T01:22:50.133Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
13062023-12-15T01:22:51.213Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-downstairs:default.log
13072023-12-15T01:22:51.251Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
13082023-12-15T01:22:52.762Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-downstairs:default.log.0
13092023-12-15T01:22:52.762Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-downstairs:downstairs-4e3c5ace-5778-43b5-bbf8-ea3089d319f4.log (1906 bytes)
13102023-12-15T01:22:53.942Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-downstairs:downstairs-4e3c5ace-5778-43b5-bbf8-ea3089d319f4.log
13112023-12-15T01:22:53.971Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-downstairs:downstairs-4e3c5ace-5778-43b5-bbf8-ea3089d319f4.log.0 (4528 bytes)
13122023-12-15T01:22:55.530Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-downstairs:downstairs-4e3c5ace-5778-43b5-bbf8-ea3089d319f4.log.0
13132023-12-15T01:22:55.530Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-downstairs:downstairs-a3dddbe3-deda-4636-8c35-1e6244e965ad.log (0 bytes)
13142023-12-15T01:22:56.641Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-downstairs:downstairs-a3dddbe3-deda-4636-8c35-1e6244e965ad.log
13152023-12-15T01:22:56.642Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-downstairs:downstairs-a3dddbe3-deda-4636-8c35-1e6244e965ad.log.0 (3050 bytes)
13162023-12-15T01:22:57.743Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-downstairs:downstairs-a3dddbe3-deda-4636-8c35-1e6244e965ad.log.0
13172023-12-15T01:22:57.744Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-downstairs:snapshot-4e3c5ace-5778-43b5-bbf8-ea3089d319f4-f0243d62-ac0d-4506-b553-a85ecac70171.log (5175 bytes)
13182023-12-15T01:22:58.844Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_826ad2ba-d609-43a3-9165-9e544e735622/root/var/svc/log/oxide-crucible-downstairs:snapshot-4e3c5ace-5778-43b5-bbf8-ea3089d319f4-f0243d62-ac0d-4506-b553-a85ecac70171.log
13192023-12-15T01:22:58.845Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-agent:default.log (16606 bytes)
13202023-12-15T01:22:59.999Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-agent:default.log
13212023-12-15T01:22:59.999Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-agent:default.log.0 (13685 bytes)
13222023-12-15T01:23:01.126Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-agent:default.log.0
13232023-12-15T01:23:01.126Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
13242023-12-15T01:23:02.203Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-downstairs:default.log
13252023-12-15T01:23:02.203Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
13262023-12-15T01:23:03.857Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-downstairs:default.log.0
13272023-12-15T01:23:03.858Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-downstairs:downstairs-5558c65e-4c52-4aaf-9bd1-4d0d9e4aeed4.log (0 bytes)
13282023-12-15T01:23:04.935Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-downstairs:downstairs-5558c65e-4c52-4aaf-9bd1-4d0d9e4aeed4.log
13292023-12-15T01:23:04.935Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-downstairs:downstairs-5558c65e-4c52-4aaf-9bd1-4d0d9e4aeed4.log.0 (3049 bytes)
13302023-12-15T01:23:06.462Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-downstairs:downstairs-5558c65e-4c52-4aaf-9bd1-4d0d9e4aeed4.log.0
13312023-12-15T01:23:06.462Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-downstairs:downstairs-edaf7a5d-58a2-4ab9-aeda-2dd3904e2338.log (5536 bytes)
13322023-12-15T01:23:07.569Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-downstairs:downstairs-edaf7a5d-58a2-4ab9-aeda-2dd3904e2338.log
13332023-12-15T01:23:07.569Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-downstairs:downstairs-f6bdbea2-4257-4256-aeee-2f1b45609736.log (2944 bytes)
13342023-12-15T01:23:09.556Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_88b23891-1726-4275-973f-1bfd9c7d86da/root/var/svc/log/oxide-crucible-downstairs:downstairs-f6bdbea2-4257-4256-aeee-2f1b45609736.log
13352023-12-15T01:23:09.582Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-agent:default.log (18008 bytes)
13362023-12-15T01:23:10.722Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-agent:default.log
13372023-12-15T01:23:10.723Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-agent:default.log.0 (9793 bytes)
13382023-12-15T01:23:12.335Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-agent:default.log.0
13392023-12-15T01:23:12.335Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
13402023-12-15T01:23:13.416Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-downstairs:default.log
13412023-12-15T01:23:13.444Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
13422023-12-15T01:23:14.521Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-downstairs:default.log.0
13432023-12-15T01:23:14.548Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-81e0b7e6-3b1f-4fb8-a5fe-8da7614ef6e2.log (1903 bytes)
13442023-12-15T01:23:15.628Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-81e0b7e6-3b1f-4fb8-a5fe-8da7614ef6e2.log
13452023-12-15T01:23:15.629Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-81e0b7e6-3b1f-4fb8-a5fe-8da7614ef6e2.log.0 (4531 bytes)
13462023-12-15T01:23:16.753Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-81e0b7e6-3b1f-4fb8-a5fe-8da7614ef6e2.log.0
13472023-12-15T01:23:16.753Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-bf267857-0853-41d7-8516-463f0e083da6.log (5535 bytes)
13482023-12-15T01:23:17.870Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-downstairs:downstairs-bf267857-0853-41d7-8516-463f0e083da6.log
13492023-12-15T01:23:17.870Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-downstairs:snapshot-81e0b7e6-3b1f-4fb8-a5fe-8da7614ef6e2-f0243d62-ac0d-4506-b553-a85ecac70171.log (5176 bytes)
13502023-12-15T01:23:19.079Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_e9d9360e-0097-44f3-aee0-39dd0bc57e9b/root/var/svc/log/oxide-crucible-downstairs:snapshot-81e0b7e6-3b1f-4fb8-a5fe-8da7614ef6e2-f0243d62-ac0d-4506-b553-a85ecac70171.log
13512023-12-15T01:23:19.080Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_5b1f3c9f-bb97-4778-92ff-b6ab4fa6d3f5/root/var/svc/log/oxide-external_dns:default.log (22663 bytes)
13522023-12-15T01:23:19.255Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_5b1f3c9f-bb97-4778-92ff-b6ab4fa6d3f5/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 22663 -> 30218
13532023-12-15T01:23:20.314Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_5b1f3c9f-bb97-4778-92ff-b6ab4fa6d3f5/root/var/svc/log/oxide-external_dns:default.log
13542023-12-15T01:23:20.314Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_5b1f3c9f-bb97-4778-92ff-b6ab4fa6d3f5/root/var/svc/log/oxide-external_dns:default.log.0 (8057 bytes)
13552023-12-15T01:23:21.431Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_5b1f3c9f-bb97-4778-92ff-b6ab4fa6d3f5/root/var/svc/log/oxide-external_dns:default.log.0
13562023-12-15T01:23:21.431Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_72d8f37c-a72e-4800-9c47-39f6b4e0d24b/root/var/svc/log/oxide-nexus:default.log (33005251 bytes)
13572023-12-15T01:23:35.172Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_72d8f37c-a72e-4800-9c47-39f6b4e0d24b/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 33005251 -> 33156662
13582023-12-15T01:23:36.242Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_72d8f37c-a72e-4800-9c47-39f6b4e0d24b/root/var/svc/log/oxide-nexus:default.log
13592023-12-15T01:23:36.242Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_72d8f37c-a72e-4800-9c47-39f6b4e0d24b/root/var/svc/log/oxide-nexus:default.log.0 (1147468 bytes)
13602023-12-15T01:23:38.275Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_72d8f37c-a72e-4800-9c47-39f6b4e0d24b/root/var/svc/log/oxide-nexus:default.log.0
13612023-12-15T01:23:38.275Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942/root/var/log/chrony/measurements.log (22331 bytes)
13622023-12-15T01:23:38.367Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942/root/var/log/chrony/measurements.log" changed size mid upload: 22331 -> 26989
13632023-12-15T01:23:39.438Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942/root/var/log/chrony/measurements.log
13642023-12-15T01:23:39.463Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942/root/var/log/chrony/statistics.log (15327 bytes)
13652023-12-15T01:23:39.530Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942/root/var/log/chrony/statistics.log" changed size mid upload: 15327 -> 19071
13662023-12-15T01:23:40.609Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942/root/var/log/chrony/statistics.log
13672023-12-15T01:23:40.609Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942/root/var/log/chrony/tracking.log (2382 bytes)
13682023-12-15T01:23:40.657Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942/root/var/log/chrony/tracking.log" changed size mid upload: 2382 -> 2646
13692023-12-15T01:23:41.729Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_6e41b21b-bf42-483b-b26f-9cfe96a37942/root/var/log/chrony/tracking.log