01HFFWJ5HNH48TBH16WY52Q5PZ: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HFFWJV54GJY8FNYDDWZJ4ESS

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-18T01:07:10.094Zjob dependencies complete; ready to run (waiting for 50 m 28 s)
22023-11-18T01:13:00.936Zjob assigned to worker 01HFFZF950WQP2XBZJXARJRV8K (queued for 5 m 50 s)
32023-11-18T01:13:01.077Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-11-18T01:13:01.297Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-11-18T01:13:01.472Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-11-18T01:13:01.608Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-18T01:13:01.756Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-11-18T01:13:01.867ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-18T01:13:02.032Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-18T01:13:02.142ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-18T01:13:02.261ZConfiguring devices.
122023-11-18T01:13:02.344Z##################################################
132023-11-18T01:13:02.429Z#### ############# ##
142023-11-18T01:13:02.525Z### ### ############ ##
152023-11-18T01:13:02.609Z## ### # ## ### ## Oxide Computer Company ##
162023-11-18T01:13:02.692Z## ## ## ### # ### ##
172023-11-18T01:13:02.816Z## # ### #### #### This Station Under ##
182023-11-18T01:13:02.891Z### ### #### # ### Computer Control ##
192023-11-18T01:13:02.954Z#### #### ### ## ##
202023-11-18T01:13:03.118Z##################################################
212023-11-18T01:13:03.235ZHostname: buskin
222023-11-18T01:13:03.363ZLoading smf(7) serice descriptions:
232023-11-18T01:13:04.963Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-cd1d99075975e5dc.gz
242023-11-18T01:13:06.673Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-cd1d99075975e5dc.gz
252023-11-18T01:13:06.724Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-18T01:13:30.901Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-18T01:13:30.932Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d1c7f350d876f572.gz
282023-11-18T01:13:54.959Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d1c7f350d876f572.gz
292023-11-18T01:13:54.996Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-18T01:13:56.132Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-18T01:13:56.164Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-18T01:13:59.761Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-18T01:13:59.794Zdownloading input: /input/package/work/version.txt
342023-11-18T01:13:59.827Zdownloaded input: /input/package/work/version.txt
352023-11-18T01:13:59.971Zdownloading input: /input/package/work/package.tar.gz
362023-11-18T01:14:20.966Zdownloaded input: /input/package/work/package.tar.gz
372023-11-18T01:14:21.005Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-18T01:14:55.759Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-18T01:14:55.792Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-18T01:15:06.401Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-18T01:15:06.433Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-18T01:15:59.130Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-18T01:15:59.161Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-18T01:17:00.740Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-18T01:17:00.772Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-18T01:17:37.126Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-18T01:17:37.161Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-18T01:17:44.391Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-18T01:17:44.422Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-18T01:17:57.836Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-18T01:17:57.882Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-18T01:18:05.992Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-18T01:18:06.024Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-18T01:18:13.792Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-18T01:18:13.827Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-18T01:18:13.886Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-18T01:18:13.921Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-18T01:18:18.676Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-18T01:18:18.708Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-18T01:18:24.325Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-18T01:18:24.356Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-18T01:18:53.611Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-18T01:18:53.646Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-18T01:19:12.865Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-18T01:19:12.945Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-18T01:19:13.316Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-18T01:19:13.350Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-18T01:19:23.023Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-18T01:19:23.055Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-18T01:19:41.862Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-18T01:19:41.894Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-18T01:20:45.444Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-18T01:20:45.476Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-18T01:21:33.712Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-18T01:21:33.743Zstarting task 0: "setup"
762023-11-18T01:21:33.784Z++ uname -s
772023-11-18T01:21:33.815Z+ kern=SunOS
782023-11-18T01:21:34.263Z+ case "$kern" in
792023-11-18T01:21:34.305Z+ groupadd -g 12345 build
802023-11-18T01:21:34.342Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-18T01:21:35.793Z+ zfs create -o mountpoint=/work rpool/work
822023-11-18T01:21:35.826Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-18T01:21:35.863Z+ home_fs=zfs
842023-11-18T01:21:35.896Z+ [[ zfs == autofs ]]
852023-11-18T01:21:35.927Z+ mkdir -p /home/build
862023-11-18T01:21:35.958Z+ chown build:build /home/build /work
872023-11-18T01:21:37.841Z+ chmod 0700 /home/build /work
882023-11-18T01:21:38.325Zprocess exited: duration 4063 ms, exit code 0
 
892023-11-18T01:21:38.389Zstarting task 1: "authentication"
902023-11-18T01:21:38.477Zprocess exited: duration 50 ms, exit code 0
 
912023-11-18T01:21:38.539Zstarting task 2: "build"
922023-11-18T01:21:38.577Z+ trap _exit_trap EXIT
932023-11-18T01:21:38.608Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-18T01:21:38.774Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-18T01:21:38.809Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-18T01:21:38.844Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-18T01:21:38.877Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-18T01:21:38.907Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-18T01:21:38.938Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-18T01:21:38.969Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-18T01:21:39.001Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-18T01:21:39.032Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-18T01:21:39.063Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-18T01:21:39.094Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-18T01:21:39.125Z+ [[ -d /opt/oxide ]]
1062023-11-18T01:21:39.156Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-18T01:21:39.188Z+ pfexec mkdir /opt/oxide
1082023-11-18T01:21:39.220Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-18T01:21:39.258Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-18T01:21:39.293Z+ cd /opt/oxide-underneath
1112023-11-18T01:21:39.324Z+ pfexec tar ceEp@/f - .
1122023-11-18T01:21:39.355Z+ cd /opt/oxide
1132023-11-18T01:21:39.385Z+ pfexec tar xveEp@/f -
1142023-11-18T01:21:39.416Zx ., 0 bytes, 0 tape blocks
1152023-11-18T01:21:39.455Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-18T01:21:39.493Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-18T01:21:39.539Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-18T01:21:39.580Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-18T01:21:39.634Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-18T01:21:39.685Z+ pfexec mkdir /opt/oxide/work
1212023-11-18T01:21:39.726Z+ pfexec chown build:build /opt/oxide/work
1222023-11-18T01:21:39.772Z+ cd /opt/oxide/work
1232023-11-18T01:21:39.806Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-18T01:21:40.495ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-18T01:21:40.530Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1262023-11-18T01:21:40.566Zx out/omicron-sled-agent.tar, 111593472 bytes, 217956 tape blocks
1272023-11-18T01:21:40.609Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-18T01:21:40.642Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-11-18T01:21:40.675Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-11-18T01:21:40.711Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-11-18T01:21:40.743Zx target/release/omicron-package, 42419504 bytes, 82851 tape blocks
1322023-11-18T01:21:40.775Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-11-18T01:21:40.806Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-18T01:21:40.837Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-18T01:21:40.869Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-18T01:21:40.900Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-18T01:21:40.931Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-18T01:21:40.962Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-18T01:21:40.993Z
1402023-11-18T01:21:41.024Zreal 1.543981333
1412023-11-18T01:21:41.057Zuser 1.391875620
1422023-11-18T01:21:41.090Zsys 0.270196263
1432023-11-18T01:21:41.126Ztrap 0.000049061
1442023-11-18T01:21:41.157Ztflt 0.000000000
1452023-11-18T01:21:41.191Zdflt 0.000000000
1462023-11-18T01:21:41.222Zkflt 0.000000000
1472023-11-18T01:21:41.253Zlock 0.000000000
1482023-11-18T01:21:41.284Zslp 1.389151344
1492023-11-18T01:21:41.315Zlat 0.034779205
1502023-11-18T01:21:41.346Zstop 0.000116471
1512023-11-18T01:21:41.377Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-18T01:21:41.569Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-18T01:21:42.009Z+ mkdir tests
1542023-11-18T01:21:42.046Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-18T01:21:42.079Z+ ptime -m gunzip
1562023-11-18T01:21:42.111Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-cd1d99075975e5dc
1572023-11-18T01:21:42.149Z
1582023-11-18T01:21:42.183Zreal 0.138490749
1592023-11-18T01:21:42.217Zuser 0.127263655
1602023-11-18T01:21:42.250Zsys 0.010969798
1612023-11-18T01:21:42.281Ztrap 0.000012161
1622023-11-18T01:21:42.312Ztflt 0.000000000
1632023-11-18T01:21:42.343Zdflt 0.000000000
1642023-11-18T01:21:42.373Zkflt 0.000000000
1652023-11-18T01:21:42.404Zlock 0.000000000
1662023-11-18T01:21:42.435Zslp 0.000019950
1672023-11-18T01:21:42.466Zlat 0.000023590
1682023-11-18T01:21:42.497Zstop 0.000126262
1692023-11-18T01:21:42.528Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-cd1d99075975e5dc
1702023-11-18T01:21:42.561Z+ chmod a+x tests/bootstrap-cd1d99075975e5dc
1712023-11-18T01:21:42.593Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-18T01:21:42.624Z+ ptime -m gunzip
1732023-11-18T01:21:42.660Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-18T01:21:43.730Z
1752023-11-18T01:21:43.792Zreal 1.977307116
1762023-11-18T01:21:43.837Zuser 1.860921829
1772023-11-18T01:21:43.868Zsys 0.116014809
1782023-11-18T01:21:43.899Ztrap 0.000146542
1792023-11-18T01:21:43.932Ztflt 0.000000000
1802023-11-18T01:21:43.964Zdflt 0.000000000
1812023-11-18T01:21:43.995Zkflt 0.000000000
1822023-11-18T01:21:44.026Zlock 0.000000000
1832023-11-18T01:21:44.057Zslp 0.000012240
1842023-11-18T01:21:44.088Zlat 0.000024270
1852023-11-18T01:21:44.120Zstop 0.000128093
1862023-11-18T01:21:44.559Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-18T01:21:44.591Z+ chmod a+x tests/bootstrap
1882023-11-18T01:21:44.624Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-18T01:21:44.656Z+ ptime -m gunzip
1902023-11-18T01:21:44.687Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d1c7f350d876f572
1912023-11-18T01:21:45.992Z
1922023-11-18T01:21:46.027Zreal 2.265947251
1932023-11-18T01:21:46.068Zuser 2.123632879
1942023-11-18T01:21:46.103Zsys 0.141910774
1952023-11-18T01:21:46.136Ztrap 0.000167854
1962023-11-18T01:21:46.176Ztflt 0.000000000
1972023-11-18T01:21:46.239Zdflt 0.000000000
1982023-11-18T01:21:46.273Zkflt 0.000000000
1992023-11-18T01:21:46.306Zlock 0.000000000
2002023-11-18T01:21:46.338Zslp 0.000022090
2012023-11-18T01:21:46.380Zlat 0.000024169
2022023-11-18T01:21:46.417Zstop 0.000127212
2032023-11-18T01:21:46.482Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d1c7f350d876f572
2042023-11-18T01:21:46.552Z+ chmod a+x tests/end_to_end_tests-d1c7f350d876f572
2052023-11-18T01:21:46.609Z+ bmat address ls
2062023-11-18T01:21:46.672ZNAME CIDR FIRST LAST COUNT
2072023-11-18T01:21:46.802Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-11-18T01:21:47.063Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-18T01:21:47.288Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-11-18T01:21:47.387Z+ EXTRA_IP_START=10.151.1.164
2112023-11-18T01:21:47.540Z+ EXTRA_IP_END=10.151.1.227
2122023-11-18T01:21:47.584Z+ EXTRA_IP_COUNT=64
2132023-11-18T01:21:47.705Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-18T01:21:48.270Z+ EXTRA_IP_BASE=10.151.1
2152023-11-18T01:21:48.406Z+ EXTRA_IP_FOCTET=164
2162023-11-18T01:21:48.544Z+ EXTRA_IP_LOCTET=227
2172023-11-18T01:21:48.872Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-11-18T01:21:49.965Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-11-18T01:21:51.486Z+ DNS_IP1=10.151.1.164
2202023-11-18T01:21:51.888Z+ DNS_IP2=10.151.1.165
2212023-11-18T01:21:52.090Z+ UPLINK_IP=10.151.1.174
2222023-11-18T01:21:52.250Z+ IPPOOL_START=10.151.1.175
2232023-11-18T01:21:52.462Z+ IPPOOL_END=10.151.1.227
2242023-11-18T01:21:52.888Z++ ipadm show-addr -po type,addr
2252023-11-18T01:21:53.435Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-18T01:21:54.125Z+ GATEWAY_IP=10.151.1.100
2272023-11-18T01:21:54.882Z+ [[ -n 10.151.1.100 ]]
2282023-11-18T01:21:55.756Z+ ping -s 10.151.1.100 56 1
2292023-11-18T01:22:16.970ZPING 10.151.1.100: 56 data bytes
2302023-11-18T01:22:17.092Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.010 ms
2312023-11-18T01:22:17.184Z
2322023-11-18T01:22:17.296Z----10.151.1.100 PING Statistics----
2332023-11-18T01:22:17.348Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-18T01:22:17.407Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2352023-11-18T01:22:17.472Z++ arp -an
2362023-11-18T01:22:17.528Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-11-18T01:22:17.600Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-11-18T01:22:17.659Z+ routeadm -e ipv4-forwarding -u
2392023-11-18T01:22:17.715Z+ PXA_START=10.151.1.164
2402023-11-18T01:22:17.761Z+ PXA_END=10.151.1.227
2412023-11-18T01:22:17.875Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-18T01:22:17.958Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-18T01:22:18.051Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-18T01:22:18.119Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-18T01:22:18.180Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-18T01:22:18.233Z++ cd ./tools
2472023-11-18T01:22:18.305Z++ pwd
2482023-11-18T01:22:18.395Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-18T01:22:18.436Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-18T01:22:18.501Z+ SOFTNPU_MODE=zone
2512023-11-18T01:22:18.533Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-18T01:22:18.566Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-18T01:22:18.614Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-18T01:22:18.659Z++ dladm show-phys -p -o LINK
2552023-11-18T01:22:18.718Z++ head -1
2562023-11-18T01:22:18.755Z+ PHYSICAL_LINK=igb0
2572023-11-18T01:22:18.811Z+ echo 'Using igb0 as physical link'
2582023-11-18T01:22:18.849ZUsing igb0 as physical link
2592023-11-18T01:22:18.891Z+ ensure_run_as_root
2602023-11-18T01:22:18.924Z++ id -u
2612023-11-18T01:22:18.957Z+ [[ 0 -ne 0 ]]
2622023-11-18T01:22:18.992Z+ ensure_zpools
2632023-11-18T01:22:19.041Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-18T01:22:19.083Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-11-18T01:22:19.125Z+ readarray -t ZPOOLS
2662023-11-18T01:22:19.175Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-11-18T01:22:19.232Z++ sed 's/[ ",]//g'
2682023-11-18T01:22:19.292ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2692023-11-18T01:22:19.330Z+ for ZPOOL in "${ZPOOLS[@]}"
2702023-11-18T01:22:19.367ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2712023-11-18T01:22:19.406Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2722023-11-18T01:22:19.451ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2732023-11-18T01:22:19.490Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2742023-11-18T01:22:19.531ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2752023-11-18T01:22:19.564Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2762023-11-18T01:22:19.608ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2772023-11-18T01:22:19.650Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2782023-11-18T01:22:19.686ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2792023-11-18T01:22:19.728Z0+0 records in
2802023-11-18T01:22:19.769ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2812023-11-18T01:22:19.814Z0+0 records out
2822023-11-18T01:22:19.869ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2832023-11-18T01:22:19.910Z0 bytes (0 iB) transferred in 0.000007 secs (0 iB/sec)
2842023-11-18T01:22:19.988ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2852023-11-18T01:22:20.028Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2862023-11-18T01:22:20.472ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2872023-11-18T01:22:20.507Z+ set +x
2882023-11-18T01:22:20.549ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2892023-11-18T01:22:20.601Z++ zpool list -o name
2902023-11-18T01:22:20.650ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2912023-11-18T01:22:20.701Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2922023-11-18T01:22:20.737ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2932023-11-18T01:22:20.778Z+ [[ -z '' ]]
2942023-11-18T01:22:20.816ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2952023-11-18T01:22:20.852Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2962023-11-18T01:22:20.890ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2972023-11-18T01:22:20.928Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2982023-11-18T01:22:20.978ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
2992023-11-18T01:22:21.025Z+ set +x
3002023-11-18T01:22:21.074ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3012023-11-18T01:22:21.135Z+ for ZPOOL in "${ZPOOLS[@]}"
3022023-11-18T01:22:21.193ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3032023-11-18T01:22:21.232Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3042023-11-18T01:22:21.285ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3052023-11-18T01:22:21.319Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3062023-11-18T01:22:21.351ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3072023-11-18T01:22:21.399Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3082023-11-18T01:22:21.433ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3092023-11-18T01:22:21.467Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3102023-11-18T01:22:21.505ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3112023-11-18T01:22:21.542Z0+0 records in
3122023-11-18T01:22:21.579ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3132023-11-18T01:22:21.629Z0+0 records out
3142023-11-18T01:22:21.666ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3152023-11-18T01:22:21.704Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3162023-11-18T01:22:21.737ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3172023-11-18T01:22:21.771Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3182023-11-18T01:22:21.810ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3192023-11-18T01:22:21.853Z+ set +x
3202023-11-18T01:22:21.904ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3212023-11-18T01:22:21.948Z++ zpool list -o name
3222023-11-18T01:22:21.984ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3232023-11-18T01:22:22.018Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3242023-11-18T01:22:22.070ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3252023-11-18T01:22:22.107Z+ [[ -z '' ]]
3262023-11-18T01:22:22.161ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3272023-11-18T01:22:22.193Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3282023-11-18T01:22:22.233ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3292023-11-18T01:22:22.266Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3302023-11-18T01:22:22.308ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3312023-11-18T01:22:22.348Z+ set +x
3322023-11-18T01:22:22.381ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3332023-11-18T01:22:22.821Z+ for ZPOOL in "${ZPOOLS[@]}"
3342023-11-18T01:22:22.872ZSimnet net0/sc0_0 exists
3352023-11-18T01:22:22.924Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3362023-11-18T01:22:22.959ZSimnet net1/sc1_0 exists
3372023-11-18T01:22:23.008Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3382023-11-18T01:22:23.063ZVnic sc0_1 exists
3392023-11-18T01:22:23.110Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3402023-11-18T01:22:23.154Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
3412023-11-18T01:22:23.227Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3422023-11-18T01:22:23.302Zdownloading softnpu ...done
3432023-11-18T01:22:23.337Z0+0 records in
3442023-11-18T01:22:23.381Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
3452023-11-18T01:22:23.426Z0+0 records out
3462023-11-18T01:22:23.887Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3472023-11-18T01:22:23.936Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3482023-11-18T01:22:23.967Z+ set +x
3492023-11-18T01:22:23.999Z++ zpool list -o name
3502023-11-18T01:22:24.045Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3512023-11-18T01:22:24.090Z+ [[ -z '' ]]
3522023-11-18T01:22:24.127Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3532023-11-18T01:22:24.168Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3542023-11-18T01:22:24.200Z+ set +x
3552023-11-18T01:22:24.248Z+ for ZPOOL in "${ZPOOLS[@]}"
3562023-11-18T01:22:24.286Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3572023-11-18T01:22:24.323Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3582023-11-18T01:22:24.356Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3592023-11-18T01:22:24.391Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3602023-11-18T01:22:24.427Z0+0 records in
3612023-11-18T01:22:24.460Z0+0 records out
3622023-11-18T01:22:24.491Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3632023-11-18T01:22:24.523Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3642023-11-18T01:22:24.554Z+ set +x
3652023-11-18T01:22:24.589Z++ zpool list -o name
3662023-11-18T01:22:24.625Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3672023-11-18T01:22:24.656Z+ [[ -z '' ]]
3682023-11-18T01:22:24.687Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3692023-11-18T01:22:24.718Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3702023-11-18T01:22:24.755Z+ set +x
3712023-11-18T01:22:24.786Z+ for ZPOOL in "${ZPOOLS[@]}"
3722023-11-18T01:22:24.817Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3732023-11-18T01:22:24.847Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3742023-11-18T01:22:24.886Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3752023-11-18T01:22:24.920Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3762023-11-18T01:22:24.951Z0+0 records in
3772023-11-18T01:22:25.024Z0+0 records out
3782023-11-18T01:22:25.060Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3792023-11-18T01:22:25.092Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3802023-11-18T01:22:25.124Z+ set +x
3812023-11-18T01:22:25.159Z++ zpool list -o name
3822023-11-18T01:22:25.192Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3832023-11-18T01:22:25.223Z+ [[ -z '' ]]
3842023-11-18T01:22:25.272Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3852023-11-18T01:22:25.307Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3862023-11-18T01:22:25.338Z+ set +x
3872023-11-18T01:22:25.369Z+ for ZPOOL in "${ZPOOLS[@]}"
3882023-11-18T01:22:25.400Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3892023-11-18T01:22:25.431Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3902023-11-18T01:22:25.463Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3912023-11-18T01:22:25.495Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3922023-11-18T01:22:25.533Z0+0 records in
3932023-11-18T01:22:25.565Z0+0 records out
3942023-11-18T01:22:25.607Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3952023-11-18T01:22:25.649Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3962023-11-18T01:22:25.687Z+ set +x
3972023-11-18T01:22:25.738Z++ zpool list -o name
3982023-11-18T01:22:25.777Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3992023-11-18T01:22:25.817Z+ [[ -z '' ]]
4002023-11-18T01:22:25.853Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
4012023-11-18T01:22:25.892Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4022023-11-18T01:22:25.926Z+ set +x
4032023-11-18T01:22:25.965Z+ for ZPOOL in "${ZPOOLS[@]}"
4042023-11-18T01:22:25.999Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4052023-11-18T01:22:26.030Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4062023-11-18T01:22:26.061Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4072023-11-18T01:22:26.092Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4082023-11-18T01:22:26.125Z0+0 records in
4092023-11-18T01:22:26.165Z0+0 records out
4102023-11-18T01:22:26.613Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4112023-11-18T01:22:26.646Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4122023-11-18T01:22:26.676Z+ set +x
4132023-11-18T01:22:26.707Z++ zpool list -o name
4142023-11-18T01:22:26.738Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4152023-11-18T01:22:26.774Z+ [[ -z '' ]]
4162023-11-18T01:22:26.811Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4172023-11-18T01:22:26.844Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4182023-11-18T01:22:26.874Z+ set +x
4192023-11-18T01:22:26.905Z+ for ZPOOL in "${ZPOOLS[@]}"
4202023-11-18T01:22:26.936Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4212023-11-18T01:22:26.967Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4222023-11-18T01:22:26.998Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4232023-11-18T01:22:27.029Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4242023-11-18T01:22:27.060Z0+0 records in
4252023-11-18T01:22:27.094Z0+0 records out
4262023-11-18T01:22:27.126Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4272023-11-18T01:22:27.157Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4282023-11-18T01:22:27.188Z+ set +x
4292023-11-18T01:22:27.226Z++ zpool list -o name
4302023-11-18T01:22:27.265Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4312023-11-18T01:22:27.297Z+ [[ -z '' ]]
4322023-11-18T01:22:27.370Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4332023-11-18T01:22:27.405Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4342023-11-18T01:22:27.437Z+ set +x
4352023-11-18T01:22:27.469Z+ for ZPOOL in "${ZPOOLS[@]}"
4362023-11-18T01:22:27.500Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4372023-11-18T01:22:27.531Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4382023-11-18T01:22:27.565Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4392023-11-18T01:22:27.597Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4402023-11-18T01:22:27.628Z0+0 records in
4412023-11-18T01:22:27.661Z0+0 records out
4422023-11-18T01:22:27.696Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4432023-11-18T01:22:27.729Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4442023-11-18T01:22:27.760Z+ set +x
4452023-11-18T01:22:27.791Z++ zpool list -o name
4462023-11-18T01:22:27.827Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4472023-11-18T01:22:27.858Z+ [[ -z '' ]]
4482023-11-18T01:22:27.889Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4492023-11-18T01:22:27.920Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4502023-11-18T01:22:27.951Z+ set +x
4512023-11-18T01:22:27.982Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4522023-11-18T01:22:28.013Z+ readarray -t ZPOOLS
4532023-11-18T01:22:28.044Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4542023-11-18T01:22:28.075Z++ sed 's/[ ",]//g'
4552023-11-18T01:22:28.110Z+ for ZPOOL in "${ZPOOLS[@]}"
4562023-11-18T01:22:28.151Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4572023-11-18T01:22:28.190Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4582023-11-18T01:22:28.222Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4592023-11-18T01:22:28.258Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4602023-11-18T01:22:28.300Z0+0 records in
4612023-11-18T01:22:28.351Z0+0 records out
4622023-11-18T01:22:28.413Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4632023-11-18T01:22:28.446Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4642023-11-18T01:22:28.481Z+ set +x
4652023-11-18T01:22:28.512Z++ zpool list -o name
4662023-11-18T01:22:28.543Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4672023-11-18T01:22:28.582Z+ [[ -z '' ]]
4682023-11-18T01:22:28.616Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4692023-11-18T01:22:28.648Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4702023-11-18T01:22:28.682Z+ set +x
4712023-11-18T01:22:28.721Z+ for ZPOOL in "${ZPOOLS[@]}"
4722023-11-18T01:22:28.755Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4732023-11-18T01:22:28.789Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4742023-11-18T01:22:28.821Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4752023-11-18T01:22:28.852Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4762023-11-18T01:22:28.884Zdownloading asic_program.so ...done
4772023-11-18T01:22:28.917Z0+0 records in
4782023-11-18T01:22:28.948Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
4792023-11-18T01:22:28.979Z0+0 records out
4802023-11-18T01:22:29.014Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4812023-11-18T01:22:29.051Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4822023-11-18T01:22:29.084Z+ set +x
4832023-11-18T01:22:29.126Z++ zpool list -o name
4842023-11-18T01:22:29.201Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4852023-11-18T01:22:29.240Z+ [[ -z '' ]]
4862023-11-18T01:22:29.281Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4872023-11-18T01:22:29.314Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4882023-11-18T01:22:29.347Z+ set +x
4892023-11-18T01:22:29.379Z+ [[ zone == \z\o\n\e ]]
4902023-11-18T01:22:29.411Z+ ensure_simulated_links igb0
4912023-11-18T01:22:29.474Z+ local PHYSICAL_LINK=igb0
4922023-11-18T01:22:29.519Z+ INDICES=("0" "1")
4932023-11-18T01:22:29.963Z+ for I in "${INDICES[@]}"
4942023-11-18T01:22:30.021Z++ get_simnet_name_if_exists net0
4952023-11-18T01:22:30.090Z++ dladm show-simnet -p -o LINK net0
4962023-11-18T01:22:30.229Z++ echo ''
4972023-11-18T01:22:30.272Z+ [[ -z '' ]]
4982023-11-18T01:22:30.365Z+ dladm create-simnet -t net0
4992023-11-18T01:22:30.467Z+ dladm create-simnet -t sc0_0
5002023-11-18T01:22:30.719Z+ dladm modify-simnet -t -p net0 sc0_0
5012023-11-18T01:22:31.036Z+ dladm set-linkprop -p mtu=9000 sc0_0
5022023-11-18T01:22:31.221Z+ success 'Simnet net0/sc0_0 exists'
5032023-11-18T01:22:31.567Z+ set +x
5042023-11-18T01:22:31.763Z+ for I in "${INDICES[@]}"
5052023-11-18T01:22:32.599Z++ get_simnet_name_if_exists net1
5062023-11-18T01:22:32.910Z++ dladm show-simnet -p -o LINK net1
5072023-11-18T01:22:33.243Z++ echo ''
5082023-11-18T01:22:33.447Z+ [[ -z '' ]]
5092023-11-18T01:22:33.525Z+ dladm create-simnet -t net1
5102023-11-18T01:22:33.582Z+ dladm create-simnet -t sc1_0
5112023-11-18T01:22:33.746Z+ dladm modify-simnet -t -p net1 sc1_0
5122023-11-18T01:22:34.286Z+ dladm set-linkprop -p mtu=9000 sc1_0
5132023-11-18T01:22:34.607Z+ success 'Simnet net1/sc1_0 exists'
5142023-11-18T01:22:34.983Z+ set +x
5152023-11-18T01:22:35.199Z++ get_vnic_name_if_exists sc0_1
5162023-11-18T01:22:35.744Z++ dladm show-vnic -p -o LINK sc0_1
5172023-11-18T01:22:35.856Z++ echo ''
5182023-11-18T01:22:35.982Z+ [[ -z '' ]]
5192023-11-18T01:22:36.265Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5202023-11-18T01:22:36.714Z+ [[ -v PROMISC_FILT_OFF ]]
5212023-11-18T01:22:37.095Z+ success 'Vnic sc0_1 exists'
5222023-11-18T01:22:37.424Z+ set +x
5232023-11-18T01:22:37.715Z+ warn_if_no_proxy_arp
5242023-11-18T01:22:38.346Z+ [[ -v PXA_START ]]
5252023-11-18T01:22:58.438Z+ [[ -v PXA_END ]]
5262023-11-18T01:22:58.709Z+ ensure_softnpu_zone
5272023-11-18T01:22:58.951Z+ zoneadm list
5282023-11-18T01:22:59.988Z+ grep -q sidecar_softnpu
5292023-11-18T01:23:00.329Z+ [[ -f out/npuzone/npuzone ]]
5302023-11-18T01:23:00.393Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5312023-11-18T01:23:00.447Zdownloading scadm ...done
5322023-11-18T01:23:00.526Zcreating port tfportrear0_0 for sc0_0
5332023-11-18T01:23:00.565Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-18T01:23:00.611Zsoftnpu zone setup
5352023-11-18T01:23:00.670Zconfigure zone
5362023-11-18T01:23:00.720Zinstall zone
5372023-11-18T01:23:00.785Zboot zone
5382023-11-18T01:23:00.854Z[sidecar_softnpu] svcs milestone/network
5392023-11-18T01:23:00.916ZSTATE STIME FMRI
5402023-11-18T01:23:00.962Zonline 1:22:37 svc:/milestone/network:default
5412023-11-18T01:23:01.013Z[sidecar_softnpu] /softnpu/npu start
5422023-11-18T01:23:01.110Z
5432023-11-18T01:23:01.181Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-18T01:23:01.248Z+ GATEWAY_IP=10.151.1.100
5452023-11-18T01:23:01.318Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-11-18T01:23:01.372ZUsing 10.151.1.100 as gateway ip
5472023-11-18T01:23:01.417Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-18T01:23:01.469Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-11-18T01:23:01.568Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-18T01:23:01.622Z+++ echo 18:c0:4d:0d:9f:b2
5512023-11-18T01:23:02.112Z+ COUNT=1
5522023-11-18T01:23:02.153Z+ [[ 1 -ne 1 ]]
5532023-11-18T01:23:02.188Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-11-18T01:23:02.238ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-11-18T01:23:02.291Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-11-18T01:23:02.352Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-11-18T01:23:02.397Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-18T01:23:02.445Z+ [[ -v PXA_START ]]
5592023-11-18T01:23:02.496Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-11-18T01:23:02.589Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-11-18T01:23:03.086Z+ z_scadm dump-state
5622023-11-18T01:23:03.138Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-18T01:23:03.212Zlocal v6:
5642023-11-18T01:23:03.261Zlocal v4:
5652023-11-18T01:23:03.748Zrouter v6:
5662023-11-18T01:23:03.864Zrouter v4_idx:
5672023-11-18T01:23:03.911Zrouter v4_routes:
5682023-11-18T01:23:04.018Zresolver v4:
5692023-11-18T01:23:04.074Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-11-18T01:23:04.168Zresolver v6:
5712023-11-18T01:23:04.205Znat v4:
5722023-11-18T01:23:04.254Znat_v6:
5732023-11-18T01:23:04.302Zport_mac:
5742023-11-18T01:23:04.380Zproxy arp:
5752023-11-18T01:23:04.458Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-11-18T01:23:04.514Z+ success 'softnpu zone exists'
5772023-11-18T01:23:04.566Z+ set +x
5782023-11-18T01:23:04.610Zsoftnpu zone exists
5792023-11-18T01:23:04.649Z
5802023-11-18T01:23:04.743Zreal 53.628201930
5812023-11-18T01:23:04.785Zuser 0.721273649
5822023-11-18T01:23:04.828Zsys 1.629647740
5832023-11-18T01:23:04.865Ztrap 0.000097614
5842023-11-18T01:23:04.953Ztflt 0.000094957
5852023-11-18T01:23:05.011Zdflt 0.001212729
5862023-11-18T01:23:05.071Zkflt 0.000000000
5872023-11-18T01:23:05.129Zlock 42:12.240449738
5882023-11-18T01:23:05.160Zslp 2:40.711488417
5892023-11-18T01:23:05.195Zlat 0.083715343
5902023-11-18T01:23:05.227Zstop 0.373703669
5912023-11-18T01:23:05.337Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
5922023-11-18T01:23:05.385Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5932023-11-18T01:23:05.428Z+++ pkg/config-rss.toml Sat Nov 18 01:22:39 2023
5942023-11-18T01:23:05.467Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-18T01:23:05.514Z@@ -41,7 +41,7 @@
5962023-11-18T01:23:05.549Z+ SILO_NAME=recovery
5972023-11-18T01:23:05.586Z # the DNS domain delegated to the rack by the customer. Each of these addresses
5982023-11-18T01:23:05.627Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5992023-11-18T01:23:05.669Z # must be contained in one of the "internal services" IP Pool ranges listed
6002023-11-18T01:23:05.704Z+ EXTERNAL_DNS_DOMAIN=oxide.test
6012023-11-18T01:23:05.788Z # below.
6022023-11-18T01:23:05.833Z+ sed -i~ '
6032023-11-18T01:23:05.880Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6042023-11-18T01:23:05.917Z /^external_dns_ips/c\
6052023-11-18T01:23:05.956Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6062023-11-18T01:23:05.991Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6072023-11-18T01:23:06.023Z
6082023-11-18T01:23:06.054Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6092023-11-18T01:23:06.086Z # Initial TLS certificates for the external API
6102023-11-18T01:23:06.118Z /^first/c\
6112023-11-18T01:23:06.149Z #
6122023-11-18T01:23:06.180Zfirst = "10.151.1.164"
6132023-11-18T01:23:06.211Z@@ -77,8 +77,8 @@
6142023-11-18T01:23:06.242Z /^last/c\
6152023-11-18T01:23:06.275Z #
6162023-11-18T01:23:06.314Zlast = "10.151.1.173"
6172023-11-18T01:23:06.356Z # For more on this and what to put here, see docs/how-to-run.adoc.
6182023-11-18T01:23:06.398Z }
6192023-11-18T01:23:06.437Z [[internal_services_ip_pool_ranges]]
6202023-11-18T01:23:06.475Z /^\[rack_network_config/,/^$/ {
6212023-11-18T01:23:06.509Z-first = "192.168.1.20"
6222023-11-18T01:23:06.541Z /^infra_ip_first/c\
6232023-11-18T01:23:06.575Z-last = "192.168.1.29"
6242023-11-18T01:23:06.611Zinfra_ip_first = "10.151.1.174"
6252023-11-18T01:23:06.645Z+first = "10.151.1.164"
6262023-11-18T01:23:06.676Z /^infra_ip_last/c\
6272023-11-18T01:23:06.708Z+last = "10.151.1.173"
6282023-11-18T01:23:06.739Zinfra_ip_last = "10.151.1.174"
6292023-11-18T01:23:06.775Z
6302023-11-18T01:23:06.848Z }
6312023-11-18T01:23:06.880Z # TODO - this configuration is subject to change going forward. Ultimately these
6322023-11-18T01:23:06.918Z /^\[\[rack_network_config.ports/,/^$/ {
6332023-11-18T01:23:06.949Z # parameters should be provided to the control plane via wicket, but we need to
6342023-11-18T01:23:06.981Z /^routes/c\
6352023-11-18T01:23:07.012Z@@ -92,8 +92,8 @@
6362023-11-18T01:23:07.043Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6372023-11-18T01:23:07.074Z # A range of IP addresses used by Boundary Services on the external network. In
6382023-11-18T01:23:07.107Z /^addresses/c\
6392023-11-18T01:23:07.138Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6402023-11-18T01:23:07.169Zaddresses = \["10.151.1.174/32"\]
6412023-11-18T01:23:07.201Z # With softnpu, only one address is used.
6422023-11-18T01:23:07.237Z }
6432023-11-18T01:23:07.275Z-infra_ip_first = "192.168.1.30"
6442023-11-18T01:23:07.308Z' pkg/config-rss.toml
6452023-11-18T01:23:07.340Z-infra_ip_last = "192.168.1.30"
6462023-11-18T01:23:07.371Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6472023-11-18T01:23:07.402Z+infra_ip_first = "10.151.1.174"
6482023-11-18T01:23:07.438Z+ true
6492023-11-18T01:23:07.471Z+infra_ip_last = "10.151.1.174"
6502023-11-18T01:23:07.502Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6512023-11-18T01:23:07.533Z
6522023-11-18T01:23:07.564Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6532023-11-18T01:23:07.595Z # Configurations for BGP routers to run on the scrimlets.
6542023-11-18T01:23:07.626Z+ TLS_NAME=recovery.sys.oxide.test
6552023-11-18T01:23:07.657Z bgp = []
6562023-11-18T01:23:07.688Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6572023-11-18T01:23:07.720Z@@ -101,9 +101,9 @@
6582023-11-18T01:23:07.754Z.+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*............+.....+....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+...+...........+......+.+.........+..............+....+..+......+...+....+......+......+...+.....+...+.......+...+..+............+..........+.....................+.......................+...................+.....+.......+.........+..+...+......+.........+.......+...+.....+............+.........+.............+......+........................+.....+...............+................................................+..........+...+..+.............+..............+.+.....+......+..................+.+.....+.+.....+................+...+..+...+.........................+.....+.+......+..+......+.+...+............+.........+..+....+......+..............+......+..........+...+.........+..+...+...+...................+...+......+..............+.+..+....+...+...+.....+.......+..+................+.......................+.+..............+....+.................+....+.....+....+...+..+.............+...+.........+..+...+.+...............+.....+...............+.+......+........+...................+..+.........+....+..+.........+....+...+.....+.......+......+............+.........+.....+.+.........+.....+...+.......+.....+......+...+...................+............+........+...............+.............+........+..........+.....+......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6592023-11-18T01:23:08.227Z # You can configure multiple uplinks by repeating the following stanza
6602023-11-18T01:23:08.264Z..+.......+............+.....+.+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+..+.+.....+......+...+...+....+...........+...+....+.................+..........+...........+..........+............+......+.....+.+.....+.........+....+........+....+...........+...+........................+......+.+...+...+............+....................+.+..............+...+....+....................+....+...........+....+............+.....+...+....+...+.........+...........+.......+........+...+.+.....+.+........+..........+.........+.....+.......+...+.....+......+.........+.............+..+............+.............+..+.............+..+.+.........+........+.......+...+........+..................+.+.....+....+...........+...+..........+......+.....+......+.+............+.....+................+............+..+.+.....+.......+.....+...+.......+.........+...+...........+......+.+.........+.....+..................+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6612023-11-18T01:23:08.300Z [[rack_network_config.ports]]
6622023-11-18T01:23:08.334Z-----
6632023-11-18T01:23:08.370Z # Routes associated with this port.
6642023-11-18T01:23:08.405Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6652023-11-18T01:23:08.442Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6662023-11-18T01:23:08.480Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6672023-11-18T01:23:08.516Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6682023-11-18T01:23:08.565Z+ rmdir pkg
6692023-11-18T01:23:08.608Z # Addresses associated with this port.
6702023-11-18T01:23:08.651Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6712023-11-18T01:23:08.708Z-addresses = ["192.168.1.30/32"]
6722023-11-18T01:23:08.754Z+ pfexec mkdir -p /usr/oxide
6732023-11-18T01:23:08.785Z+addresses = ["10.151.1.174/32"]
6742023-11-18T01:23:08.817Z+ pfexec rm -f /usr/oxide/tcpproxy
6752023-11-18T01:23:09.267Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6762023-11-18T01:23:09.298Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6772023-11-18T01:23:09.330Z port = "qsfp0"
6782023-11-18T01:23:09.366Z+ pfexec chmod +x /usr/oxide/tcpproxy
6792023-11-18T01:23:09.402Z # The speed of this port.
6802023-11-18T01:23:09.436Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6812023-11-18T01:23:09.471Za pkg/config-rss.toml 7K
6822023-11-18T01:23:09.504Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6832023-11-18T01:23:09.544Za pkg/initial-tls-cert.pem 2K
6842023-11-18T01:23:09.586Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6852023-11-18T01:23:09.621Za pkg/initial-tls-key.pem 4K
6862023-11-18T01:23:09.652Z+ OMICRON_NO_UNINSTALL=1
6872023-11-18T01:23:09.683Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6882023-11-18T01:23:09.714ZNov 18 01:22:41.308 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6892023-11-18T01:23:09.745ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6902023-11-18T01:23:09.776ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6912023-11-18T01:23:09.808ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6922023-11-18T01:23:09.841ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6932023-11-18T01:23:09.940ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6942023-11-18T01:23:09.971ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6952023-11-18T01:23:10.005ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6962023-11-18T01:23:10.040ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6972023-11-18T01:23:10.079ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6982023-11-18T01:23:10.111ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6992023-11-18T01:23:10.147ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7002023-11-18T01:23:10.179ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7012023-11-18T01:23:10.210ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7022023-11-18T01:23:10.242ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7032023-11-18T01:23:10.275ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7042023-11-18T01:23:10.308ZNov 18 01:22:41.310 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7052023-11-18T01:23:10.340ZNov 18 01:22:41.557 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-11-18T01:23:10.371ZNov 18 01:22:41.590 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-18T01:23:10.403ZNov 18 01:22:41.648 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-18T01:23:10.449Z
7092023-11-18T01:23:10.528Zlocal v6:
7102023-11-18T01:23:10.570Zreal 0.475597846
7112023-11-18T01:23:10.605Zfd00:99::1
7122023-11-18T01:23:10.638Zuser 0.097443229
7132023-11-18T01:23:10.674Zfe80::b811:3fff:fe82:1166
7142023-11-18T01:23:10.717Zsys 2.008380786
7152023-11-18T01:23:10.759Zlocal v4:
7162023-11-18T01:23:10.800Ztrap 0.000007260
7172023-11-18T01:23:10.844Z10.151.1.174
7182023-11-18T01:23:10.888Ztflt 0.000029750
7192023-11-18T01:23:10.926Zrouter v6:
7202023-11-18T01:23:10.972Zdflt 0.001119608
7212023-11-18T01:23:11.008Zrouter v4_idx:
7222023-11-18T01:23:11.461Zkflt 0.000000000
7232023-11-18T01:23:11.504Z0.0.0.0/0 -> 0
7242023-11-18T01:23:11.543Zlock 42.644386246
7252023-11-18T01:23:11.577Zrouter v4_routes:
7262023-11-18T01:23:11.622Zslp 0.680813787
7272023-11-18T01:23:11.657Z0 -> 10.151.1.100 (2)
7282023-11-18T01:23:11.690Zlat 0.003589213
7292023-11-18T01:23:11.735Zresolver v4:
7302023-11-18T01:23:11.773Zstop 0.193248600
7312023-11-18T01:23:11.805Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7322023-11-18T01:23:11.844Z+ retry=0
7332023-11-18T01:23:12.290Zresolver v6:
7342023-11-18T01:23:12.322Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352023-11-18T01:23:12.359Zfe80::1cb3:80ff:fe1b:e9cd -> 1e:b3:80:1b:e9:cd
7362023-11-18T01:23:12.391Z+ [[ 0 -gt 30 ]]
7372023-11-18T01:23:12.422Zfe80::b811:3fff:fe82:1166 -> ba:11:3f:82:11:66
7382023-11-18T01:23:12.454Z+ sleep 1
7392023-11-18T01:23:12.485Znat v4:
7402023-11-18T01:23:12.516Z+ retry=1
7412023-11-18T01:23:12.546Znat_v6:
7422023-11-18T01:23:12.578Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7432023-11-18T01:23:12.608Zport_mac:
7442023-11-18T01:23:12.645Z+ [[ 1 -gt 30 ]]
7452023-11-18T01:23:12.680Z1000: a8:40:25:67:e0:4b
7462023-11-18T01:23:12.715Z+ sleep 1
7472023-11-18T01:23:12.756Z1: a8:40:25:67:e0:4c
7482023-11-18T01:23:12.796Z+ retry=2
7492023-11-18T01:23:12.829Zproxy arp:
7502023-11-18T01:23:12.860Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7512023-11-18T01:23:12.891Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7522023-11-18T01:23:12.922Z+ [[ 2 -gt 30 ]]
7532023-11-18T01:23:12.955Z+ sleep 1
7542023-11-18T01:23:12.986Z+ retry=3
7552023-11-18T01:23:13.025Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7562023-11-18T01:23:13.057Z+ [[ 3 -gt 30 ]]
7572023-11-18T01:23:13.088Z+ sleep 1
7582023-11-18T01:23:13.118Z+ retry=4
7592023-11-18T01:23:13.151Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7602023-11-18T01:23:13.188Z+ [[ 4 -gt 30 ]]
7612023-11-18T01:23:13.219Z+ sleep 1
7622023-11-18T01:23:13.286Z+ retry=5
7632023-11-18T01:23:13.319Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7642023-11-18T01:23:13.350Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7652023-11-18T01:23:13.382Z+ retry=0
7662023-11-18T01:23:13.413Z+++ zoneadm list -n
7672023-11-18T01:23:13.447Z+++ grep oxz_ntp
7682023-11-18T01:23:13.478Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-11-18T01:23:13.509Zsvcs: zone '-Hostate': entity not found
7702023-11-18T01:23:13.540Z++ true
7712023-11-18T01:23:13.571Z+ [[ '' != online ]]
7722023-11-18T01:23:13.605Z+ [[ 0 -gt 60 ]]
7732023-11-18T01:23:14.085Z+ sleep 1
7742023-11-18T01:23:14.116Z+ retry=1
7752023-11-18T01:23:14.147Z+++ zoneadm list -n
7762023-11-18T01:23:14.185Z+++ grep oxz_ntp
7772023-11-18T01:23:14.220Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-11-18T01:23:14.251Zsvcs: zone '-Hostate': entity not found
7792023-11-18T01:23:14.282Z++ true
7802023-11-18T01:23:14.314Z+ [[ '' != online ]]
7812023-11-18T01:23:14.346Z+ [[ 1 -gt 60 ]]
7822023-11-18T01:23:14.377Z+ sleep 1
7832023-11-18T01:23:14.408Z+ retry=2
7842023-11-18T01:23:14.439Z+++ zoneadm list -n
7852023-11-18T01:23:14.470Z+++ grep oxz_ntp
7862023-11-18T01:23:14.501Z++ pfexec svcs -z oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834 -Hostate oxide/ntp
7872023-11-18T01:23:14.533Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7882023-11-18T01:23:14.565Z++ true
7892023-11-18T01:23:14.596Z+ [[ '' != online ]]
7902023-11-18T01:23:14.626Z+ [[ 2 -gt 60 ]]
7912023-11-18T01:23:14.658Z+ sleep 1
7922023-11-18T01:23:14.693Z+ retry=3
7932023-11-18T01:23:14.731Z+++ zoneadm list -n
7942023-11-18T01:23:14.773Z+++ grep oxz_ntp
7952023-11-18T01:23:14.808Z++ pfexec svcs -z oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834 -Hostate oxide/ntp
7962023-11-18T01:23:14.863Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7972023-11-18T01:23:14.910Z++ true
7982023-11-18T01:23:14.952Z+ [[ '' != online ]]
7992023-11-18T01:23:15.086Z+ [[ 3 -gt 60 ]]
8002023-11-18T01:23:15.117Z+ sleep 1
8012023-11-18T01:23:15.148Z+ retry=4
8022023-11-18T01:23:15.179Z+++ zoneadm list -n
8032023-11-18T01:23:15.210Z+++ grep oxz_ntp
8042023-11-18T01:23:15.241Z++ pfexec svcs -z oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834 -Hostate oxide/ntp
8052023-11-18T01:23:15.272Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8062023-11-18T01:23:15.304Z++ true
8072023-11-18T01:23:15.335Z+ [[ '' != online ]]
8082023-11-18T01:23:15.373Z+ [[ 4 -gt 60 ]]
8092023-11-18T01:23:15.406Z+ sleep 1
8102023-11-18T01:23:15.438Z+ retry=5
8112023-11-18T01:23:15.886Z+++ zoneadm list -n
8122023-11-18T01:23:15.921Z+++ grep oxz_ntp
8132023-11-18T01:23:15.959Z++ pfexec svcs -z oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834 -Hostate oxide/ntp
8142023-11-18T01:23:15.995Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8152023-11-18T01:23:16.030Z++ true
8162023-11-18T01:23:16.063Z+ [[ '' != online ]]
8172023-11-18T01:23:16.123Z+ [[ 5 -gt 60 ]]
8182023-11-18T01:23:16.154Z+ sleep 1
8192023-11-18T01:23:16.185Z+ retry=6
8202023-11-18T01:23:16.216Z+++ zoneadm list -n
8212023-11-18T01:23:16.252Z+++ grep oxz_ntp
8222023-11-18T01:23:16.293Z++ pfexec svcs -z oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834 -Hostate oxide/ntp
8232023-11-18T01:23:16.334Z+ [[ disabled != online ]]
8242023-11-18T01:23:16.370Z+ [[ 6 -gt 60 ]]
8252023-11-18T01:23:16.404Z+ sleep 1
8262023-11-18T01:23:16.442Z+ retry=7
8272023-11-18T01:23:16.479Z+++ zoneadm list -n
8282023-11-18T01:23:16.516Z+++ grep oxz_ntp
8292023-11-18T01:23:16.551Z++ pfexec svcs -z oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834 -Hostate oxide/ntp
8302023-11-18T01:23:16.585Z+ [[ disabled != online ]]
8312023-11-18T01:23:16.621Z+ [[ 7 -gt 60 ]]
8322023-11-18T01:23:16.654Z+ sleep 1
8332023-11-18T01:23:16.689Z+ retry=8
8342023-11-18T01:23:16.724Z+++ zoneadm list -n
8352023-11-18T01:23:16.757Z+++ grep oxz_ntp
8362023-11-18T01:23:16.791Z++ pfexec svcs -z oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834 -Hostate oxide/ntp
8372023-11-18T01:23:16.829Z+ [[ disabled != online ]]
8382023-11-18T01:23:16.864Z+ [[ 8 -gt 60 ]]
8392023-11-18T01:23:16.901Z+ sleep 1
8402023-11-18T01:23:16.947Z+ retry=9
8412023-11-18T01:23:16.983Z+++ zoneadm list -n
8422023-11-18T01:23:17.028Z+++ grep oxz_ntp
8432023-11-18T01:23:17.474Z++ pfexec svcs -z oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834 -Hostate oxide/ntp
8442023-11-18T01:23:17.909Z+ [[ disabled != online ]]
8452023-11-18T01:23:17.941Z+ [[ 9 -gt 60 ]]
8462023-11-18T01:23:17.977Z+ sleep 1
8472023-11-18T01:23:18.013Z+ retry=10
8482023-11-18T01:23:18.046Z+++ zoneadm list -n
8492023-11-18T01:23:18.526Z+++ grep oxz_ntp
8502023-11-18T01:23:18.565Z++ pfexec svcs -z oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834 -Hostate oxide/ntp
8512023-11-18T01:23:18.598Z+ [[ disabled != online ]]
8522023-11-18T01:23:18.629Z+ [[ 10 -gt 60 ]]
8532023-11-18T01:23:18.662Z+ sleep 1
8542023-11-18T01:23:18.693Z+ retry=11
8552023-11-18T01:23:18.724Z+++ zoneadm list -n
8562023-11-18T01:23:18.755Z+++ grep oxz_ntp
8572023-11-18T01:23:18.786Z++ pfexec svcs -z oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834 -Hostate oxide/ntp
8582023-11-18T01:23:18.817Z+ [[ disabled != online ]]
8592023-11-18T01:23:18.848Z+ [[ 11 -gt 60 ]]
8602023-11-18T01:23:18.881Z+ sleep 1
8612023-11-18T01:23:19.622Z+ retry=12
8622023-11-18T01:23:19.653Z+++ zoneadm list -n
8632023-11-18T01:23:19.684Z+++ grep oxz_ntp
8642023-11-18T01:23:19.716Z++ pfexec svcs -z oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834 -Hostate oxide/ntp
8652023-11-18T01:23:19.747Z+ [[ disabled != online ]]
8662023-11-18T01:23:19.778Z+ [[ 12 -gt 60 ]]
8672023-11-18T01:23:19.809Z+ sleep 1
8682023-11-18T01:23:20.645Z+ retry=13
8692023-11-18T01:23:20.676Z+++ zoneadm list -n
8702023-11-18T01:23:20.708Z+++ grep oxz_ntp
8712023-11-18T01:23:20.740Z++ pfexec svcs -z oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834 -Hostate oxide/ntp
8722023-11-18T01:23:20.776Z+ [[ disabled != online ]]
8732023-11-18T01:23:20.810Z+ [[ 13 -gt 60 ]]
8742023-11-18T01:23:20.846Z+ sleep 1
 
8752023-11-18T01:23:21.340Zbuskin console login: Nov 18 01:23:19 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8762023-11-18T01:23:21.669Z+ retry=14
8772023-11-18T01:23:21.712Z+++ zoneadm list -n
8782023-11-18T01:23:21.745Z+++ grep oxz_ntp
8792023-11-18T01:23:21.776Z++ pfexec svcs -z oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834 -Hostate oxide/ntp
8802023-11-18T01:23:21.807Z+ [[ online != online ]]
8812023-11-18T01:23:21.837Z+ echo 'Waited for chrony: 14s'
8822023-11-18T01:23:21.869ZWaited for chrony: 14s
8832023-11-18T01:23:21.900Z+ export RUST_BACKTRACE=1
8842023-11-18T01:23:21.930Z+ RUST_BACKTRACE=1
8852023-11-18T01:23:21.961Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8862023-11-18T01:23:21.992Z+ ./tests/bootstrap
8872023-11-18T01:23:22.023ZUsing external DNS server at 10.151.1.164:53
8882023-11-18T01:23:22.055Z2023-11-18 01:23:21.146139451 UTC: attempting to log into API
8892023-11-18T01:23:36.804Z2023-11-18 01:23:37.208450693 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8902023-11-18T01:23:37.759Z2023-11-18 01:23:38.210372301 UTC: attempting to log into API
8912023-11-18T01:23:52.787Z2023-11-18 01:23:53.242895353 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8922023-11-18T01:23:53.788Z2023-11-18 01:23:54.243822532 UTC: attempting to log into API
8932023-11-18T01:24:08.822Z2023-11-18 01:24:09.277602159 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8942023-11-18T01:24:09.824Z2023-11-18 01:24:10.279426776 UTC: attempting to log into API
8952023-11-18T01:24:25.285Z2023-11-18 01:24:25.312257155 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8962023-11-18T01:24:25.858Z2023-11-18 01:24:26.313123912 UTC: attempting to log into API
8972023-11-18T01:24:40.890Z2023-11-18 01:24:41.234260066 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8982023-11-18T01:24:41.901Z2023-11-18 01:24:42.232529545 UTC: attempting to log into API
8992023-11-18T01:24:56.924Z2023-11-18 01:24:57.224456430 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9002023-11-18T01:24:57.935Z2023-11-18 01:24:58.222690489 UTC: attempting to log into API
9012023-11-18T01:25:14.827Z2023-11-18 01:25:13.214673466 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9022023-11-18T01:25:14.894Z2023-11-18 01:25:14.212977906 UTC: attempting to log into API
9032023-11-18T01:25:35.548Z2023-11-18 01:25:29.205988073 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9042023-11-18T01:25:35.757Z2023-11-18 01:25:30.204123829 UTC: attempting to log into API
9052023-11-18T01:25:45.027Z2023-11-18 01:25:45.195967778 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9062023-11-18T01:25:46.027Z2023-11-18 01:25:46.194101193 UTC: attempting to log into API
9072023-11-18T01:26:01.060Z2023-11-18 01:26:01.186892741 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9082023-11-18T01:26:02.061Z2023-11-18 01:26:02.184898234 UTC: attempting to log into API
9092023-11-18T01:26:17.100Z2023-11-18 01:26:17.178344494 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9102023-11-18T01:26:18.097Z2023-11-18 01:26:18.177611473 UTC: attempting to log into API
9112023-11-18T01:26:33.129Z2023-11-18 01:26:33.169745436 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9122023-11-18T01:26:34.130Z2023-11-18 01:26:34.168119271 UTC: attempting to log into API
9132023-11-18T01:26:49.164Z2023-11-18 01:26:49.160981727 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9142023-11-18T01:26:50.165Z2023-11-18 01:26:50.159359119 UTC: attempting to log into API
9152023-11-18T01:27:05.225Z2023-11-18 01:27:05.187282337 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9162023-11-18T01:27:06.199Z2023-11-18 01:27:06.188789676 UTC: attempting to log into API
9172023-11-18T01:27:21.232Z2023-11-18 01:27:21.221909558 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9182023-11-18T01:27:22.233Z2023-11-18 01:27:22.223020949 UTC: attempting to log into API
9192023-11-18T01:27:37.268Z2023-11-18 01:27:37.256679342 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9202023-11-18T01:27:38.267Z2023-11-18 01:27:38.257489348 UTC: attempting to log into API
9212023-11-18T01:27:53.300Z2023-11-18 01:27:53.290758606 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9222023-11-18T01:27:54.302Z2023-11-18 01:27:54.291839047 UTC: attempting to log into API
 
9232023-11-18T01:27:55.229ZNov 18 01:27:54 bukin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9242023-11-18T01:28:01.795ZNov 18 01:28:01 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9252023-11-18T01:28:08.346ZNov 18 01:28:07 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9262023-11-18T01:28:09.335Z2023-11-18 01:28:09.325204375 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9272023-11-18T01:28:10.336Z2023-11-18 01:28:10.326349597 UTC: attempting to log into API
 
9282023-11-18T01:28:14.688ZNov 18 01:28:13 bukin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9292023-11-18T01:28:22.313ZNov 18 01:28:21 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9302023-11-18T01:28:25.373Z2023-11-18 01:28:25.358767696 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9312023-11-18T01:28:26.370Z2023-11-18 01:28:26.359909789 UTC: attempting to log into API
9322023-11-18T01:28:41.406Z2023-11-18 01:28:41.393434830 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9332023-11-18T01:28:42.405Z2023-11-18 01:28:42.395276106 UTC: attempting to log into API
9342023-11-18T01:28:48.219Z2023-11-18 01:28:48.209077727 UTC: login succeeded
9352023-11-18T01:28:48.254Zwaiting for nexus to come up...
9362023-11-18T01:28:48.326Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9372023-11-18T01:28:48.395Zensuring datasets are ready...
9382023-11-18T01:28:51.626Zlet's roll.
9392023-11-18T01:28:51.650Z+ rm ./tests/bootstrap
9402023-11-18T01:28:51.710Z+ for test_bin in tests/*
9412023-11-18T01:28:51.738Z+ ./tests/bootstrap-cd1d99075975e5dc
9422023-11-18T01:28:51.764Z
9432023-11-18T01:28:51.788Zrunning 0 tests
9442023-11-18T01:28:51.816Z
9452023-11-18T01:28:51.840Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9462023-11-18T01:28:51.864Z
9472023-11-18T01:28:51.888Z+ for test_bin in tests/*
9482023-11-18T01:28:51.912Z+ ./tests/end_to_end_tests-d1c7f350d876f572
9492023-11-18T01:28:51.936Z
9502023-11-18T01:28:51.960Zrunning 2 tests
9512023-11-18T01:28:51.988Ztest no_spoof::no_spoof_here ... ok
9522023-11-18T01:29:51.727Ztest instance_launch::instance_launch has been running for over 60 seconds
9532023-11-18T01:31:50.472Ztest instance_launch::instance_launch ... ok
9542023-11-18T01:31:50.500Z
9552023-11-18T01:31:50.529Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 178.75s
9562023-11-18T01:31:50.560Z
9572023-11-18T01:31:50.585Z+ _exit_trap
9582023-11-18T01:31:50.609Z+ local status=0
9592023-11-18T01:31:50.633Z+ [[ 0 -eq 0 ]]
9602023-11-18T01:31:50.657Z+ exit 0
9612023-11-18T01:31:50.681Zprocess exited: duration 611906 ms, exit code 0
 
9622023-11-18T01:31:50.783Zfound 90 output files
9632023-11-18T01:31:50.930Zuploading: /var/svc/log/oxide-sled-agent:default.log (693629 bytes)
9642023-11-18T01:31:53.142Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9652023-11-18T01:31:53.170Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_015e9367-bfb2-4814-9b3b-84bd48998c49/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9662023-11-18T01:31:54.231Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_015e9367-bfb2-4814-9b3b-84bd48998c49/root/var/svc/log/oxide-crucible-agent:default.log
9672023-11-18T01:31:54.341Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_015e9367-bfb2-4814-9b3b-84bd48998c49/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
9682023-11-18T01:31:55.990Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_015e9367-bfb2-4814-9b3b-84bd48998c49/root/var/svc/log/oxide-crucible-agent:default.log.0
9692023-11-18T01:31:56.596Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_015e9367-bfb2-4814-9b3b-84bd48998c49/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9702023-11-18T01:31:57.556Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_015e9367-bfb2-4814-9b3b-84bd48998c49/root/var/svc/log/oxide-crucible-downstairs:default.log
9712023-11-18T01:31:57.860Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_015e9367-bfb2-4814-9b3b-84bd48998c49/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9722023-11-18T01:31:59.219Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_015e9367-bfb2-4814-9b3b-84bd48998c49/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9732023-11-18T01:31:59.336Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac1ba7c-a9d5-48a3-b069-a56bf90670cc/root/var/svc/log/oxide-crucible-agent:default.log (4324 bytes)
9742023-11-18T01:32:00.687Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac1ba7c-a9d5-48a3-b069-a56bf90670cc/root/var/svc/log/oxide-crucible-agent:default.log
9752023-11-18T01:32:00.865Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac1ba7c-a9d5-48a3-b069-a56bf90670cc/root/var/svc/log/oxide-crucible-agent:default.log.0 (9793 bytes)
9762023-11-18T01:32:22.371Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac1ba7c-a9d5-48a3-b069-a56bf90670cc/root/var/svc/log/oxide-crucible-agent:default.log.0
9772023-11-18T01:32:24.610Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac1ba7c-a9d5-48a3-b069-a56bf90670cc/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9782023-11-18T01:32:25.288Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac1ba7c-a9d5-48a3-b069-a56bf90670cc/root/var/svc/log/oxide-crucible-downstairs:default.log
9792023-11-18T01:32:25.752Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac1ba7c-a9d5-48a3-b069-a56bf90670cc/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9802023-11-18T01:32:26.513Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac1ba7c-a9d5-48a3-b069-a56bf90670cc/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9812023-11-18T01:32:26.539Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac1ba7c-a9d5-48a3-b069-a56bf90670cc/root/var/svc/log/oxide-crucible-downstairs:downstairs-1d84b311-66ab-4fbf-8528-09c40ef541dd.log (1006 bytes)
9822023-11-18T01:32:27.641Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac1ba7c-a9d5-48a3-b069-a56bf90670cc/root/var/svc/log/oxide-crucible-downstairs:downstairs-1d84b311-66ab-4fbf-8528-09c40ef541dd.log
9832023-11-18T01:32:27.671Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac1ba7c-a9d5-48a3-b069-a56bf90670cc/root/var/svc/log/oxide-crucible-downstairs:downstairs-1d84b311-66ab-4fbf-8528-09c40ef541dd.log.0 (4528 bytes)
9842023-11-18T01:32:28.748Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac1ba7c-a9d5-48a3-b069-a56bf90670cc/root/var/svc/log/oxide-crucible-downstairs:downstairs-1d84b311-66ab-4fbf-8528-09c40ef541dd.log.0
9852023-11-18T01:32:28.775Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fcc79b86-a2fb-4163-9db5-22968325b018/root/var/svc/log/oxide-crucible-agent:default.log (4789 bytes)
9862023-11-18T01:32:29.865Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fcc79b86-a2fb-4163-9db5-22968325b018/root/var/svc/log/oxide-crucible-agent:default.log
9872023-11-18T01:32:29.892Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fcc79b86-a2fb-4163-9db5-22968325b018/root/var/svc/log/oxide-crucible-agent:default.log.0 (9793 bytes)
9882023-11-18T01:32:30.963Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fcc79b86-a2fb-4163-9db5-22968325b018/root/var/svc/log/oxide-crucible-agent:default.log.0
9892023-11-18T01:32:31.019Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fcc79b86-a2fb-4163-9db5-22968325b018/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9902023-11-18T01:32:32.044Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fcc79b86-a2fb-4163-9db5-22968325b018/root/var/svc/log/oxide-crucible-downstairs:default.log
9912023-11-18T01:32:32.071Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fcc79b86-a2fb-4163-9db5-22968325b018/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9922023-11-18T01:32:33.146Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fcc79b86-a2fb-4163-9db5-22968325b018/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9932023-11-18T01:32:33.171Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fcc79b86-a2fb-4163-9db5-22968325b018/root/var/svc/log/oxide-crucible-downstairs:downstairs-628d39a7-69e3-423d-be9c-66d3af26f47f.log (1006 bytes)
9942023-11-18T01:32:34.243Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fcc79b86-a2fb-4163-9db5-22968325b018/root/var/svc/log/oxide-crucible-downstairs:downstairs-628d39a7-69e3-423d-be9c-66d3af26f47f.log
9952023-11-18T01:32:34.268Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fcc79b86-a2fb-4163-9db5-22968325b018/root/var/svc/log/oxide-crucible-downstairs:downstairs-628d39a7-69e3-423d-be9c-66d3af26f47f.log.0 (4527 bytes)
9962023-11-18T01:32:35.345Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fcc79b86-a2fb-4163-9db5-22968325b018/root/var/svc/log/oxide-crucible-downstairs:downstairs-628d39a7-69e3-423d-be9c-66d3af26f47f.log.0
9972023-11-18T01:32:35.370Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4621a6e4-b443-478f-8e55-738550284b25/root/var/svc/log/oxide-external_dns:default.log (7554 bytes)
9982023-11-18T01:32:35.992Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4621a6e4-b443-478f-8e55-738550284b25/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 7554 -> 11332
9992023-11-18T01:32:37.063Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4621a6e4-b443-478f-8e55-738550284b25/root/var/svc/log/oxide-external_dns:default.log
10002023-11-18T01:32:37.087Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4621a6e4-b443-478f-8e55-738550284b25/root/var/svc/log/oxide-external_dns:default.log.0 (8060 bytes)
10012023-11-18T01:32:38.187Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4621a6e4-b443-478f-8e55-738550284b25/root/var/svc/log/oxide-external_dns:default.log.0
10022023-11-18T01:32:38.219Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bef2b8ae-5644-4674-b728-2a4b9cd96f85/root/var/svc/log/oxide-nexus:default.log (142668 bytes)
10032023-11-18T01:32:39.166Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bef2b8ae-5644-4674-b728-2a4b9cd96f85/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 142668 -> 212779
10042023-11-18T01:32:40.236Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bef2b8ae-5644-4674-b728-2a4b9cd96f85/root/var/svc/log/oxide-nexus:default.log
10052023-11-18T01:32:40.260Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bef2b8ae-5644-4674-b728-2a4b9cd96f85/root/var/svc/log/oxide-nexus:default.log.0 (741888 bytes)
10062023-11-18T01:32:42.256Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bef2b8ae-5644-4674-b728-2a4b9cd96f85/root/var/svc/log/oxide-nexus:default.log.0
10072023-11-18T01:32:42.281Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_e6b79618-15a5-443b-836a-645697a3e8db/root/var/svc/log/oxide-nexus:default.log (133707 bytes)
10082023-11-18T01:32:42.736Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_e6b79618-15a5-443b-836a-645697a3e8db/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 133707 -> 202035
10092023-11-18T01:32:43.806Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_e6b79618-15a5-443b-836a-645697a3e8db/root/var/svc/log/oxide-nexus:default.log
10102023-11-18T01:32:43.831Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_e6b79618-15a5-443b-836a-645697a3e8db/root/var/svc/log/oxide-nexus:default.log.0 (501705 bytes)
10112023-11-18T01:32:46.302Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_e6b79618-15a5-443b-836a-645697a3e8db/root/var/svc/log/oxide-nexus:default.log.0
10122023-11-18T01:32:46.330Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_89b351b3-e9ba-45cd-a0d2-454fcafa702f/root/var/svc/log/oxide-oximeter:default.log (95222 bytes)
10132023-11-18T01:32:46.689Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_89b351b3-e9ba-45cd-a0d2-454fcafa702f/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 95222 -> 141237
10142023-11-18T01:32:47.759Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_89b351b3-e9ba-45cd-a0d2-454fcafa702f/root/var/svc/log/oxide-oximeter:default.log
10152023-11-18T01:32:47.783Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_89b351b3-e9ba-45cd-a0d2-454fcafa702f/root/var/svc/log/oxide-oximeter:default.log.0 (81035 bytes)
10162023-11-18T01:32:49.077Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_89b351b3-e9ba-45cd-a0d2-454fcafa702f/root/var/svc/log/oxide-oximeter:default.log.0
10172023-11-18T01:32:49.116Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-agent:default.log (4787 bytes)
10182023-11-18T01:32:50.190Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-agent:default.log
10192023-11-18T01:32:50.219Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-agent:default.log.0 (20042 bytes)
10202023-11-18T01:32:51.459Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-agent:default.log.0
10212023-11-18T01:32:51.485Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10222023-11-18T01:32:52.531Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-downstairs:default.log
10232023-11-18T01:32:52.555Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10242023-11-18T01:32:53.629Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10252023-11-18T01:32:53.653Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-downstairs:downstairs-29fc1c98-23e9-44c0-a416-56c471a6aea0.log (0 bytes)
10262023-11-18T01:32:54.702Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-downstairs:downstairs-29fc1c98-23e9-44c0-a416-56c471a6aea0.log
10272023-11-18T01:32:54.726Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-downstairs:downstairs-29fc1c98-23e9-44c0-a416-56c471a6aea0.log.0 (3051 bytes)
10282023-11-18T01:32:55.797Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-downstairs:downstairs-29fc1c98-23e9-44c0-a416-56c471a6aea0.log.0
10292023-11-18T01:32:55.821Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-downstairs:downstairs-973a6352-79e3-44bc-a338-5e6d3106f687.log (1006 bytes)
10302023-11-18T01:32:56.890Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-downstairs:downstairs-973a6352-79e3-44bc-a338-5e6d3106f687.log
10312023-11-18T01:32:56.920Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-downstairs:downstairs-973a6352-79e3-44bc-a338-5e6d3106f687.log.0 (4530 bytes)
10322023-11-18T01:32:58.519Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_87689633-f288-4432-b17d-a5fb10126814/root/var/svc/log/oxide-crucible-downstairs:downstairs-973a6352-79e3-44bc-a338-5e6d3106f687.log.0
10332023-11-18T01:32:58.546Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834/root/var/svc/log/oxide-ntp:default.log (0 bytes)
10342023-11-18T01:32:59.590Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834/root/var/svc/log/oxide-ntp:default.log
10352023-11-18T01:32:59.614Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834/root/var/svc/log/oxide-ntp:default.log.0 (1321 bytes)
10362023-11-18T01:33:01.104Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834/root/var/svc/log/oxide-ntp:default.log.0
10372023-11-18T01:33:01.129Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_7de5ddb1-18ff-4baa-848c-599d110dede2/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10382023-11-18T01:33:02.178Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_7de5ddb1-18ff-4baa-848c-599d110dede2/root/var/svc/log/oxide-cockroachdb:default.log
10392023-11-18T01:33:02.203Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_7de5ddb1-18ff-4baa-848c-599d110dede2/root/var/svc/log/oxide-cockroachdb:default.log.0 (5370 bytes)
10402023-11-18T01:33:03.278Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_7de5ddb1-18ff-4baa-848c-599d110dede2/root/var/svc/log/oxide-cockroachdb:default.log.0
10412023-11-18T01:33:03.304Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_892066f8-83b5-4067-9d2e-0a4b1ad50856/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10422023-11-18T01:33:04.350Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_892066f8-83b5-4067-9d2e-0a4b1ad50856/root/var/svc/log/oxide-cockroachdb:default.log
10432023-11-18T01:33:04.377Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_892066f8-83b5-4067-9d2e-0a4b1ad50856/root/var/svc/log/oxide-cockroachdb:default.log.0 (5370 bytes)
10442023-11-18T01:33:05.444Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_892066f8-83b5-4067-9d2e-0a4b1ad50856/root/var/svc/log/oxide-cockroachdb:default.log.0
10452023-11-18T01:33:05.469Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_2b6b213a-4ec2-4110-9caf-98407aaddbbe/root/var/svc/log/oxide-external_dns:default.log (7553 bytes)
10462023-11-18T01:33:05.515Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_2b6b213a-4ec2-4110-9caf-98407aaddbbe/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 7553 -> 11331
10472023-11-18T01:33:06.609Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_2b6b213a-4ec2-4110-9caf-98407aaddbbe/root/var/svc/log/oxide-external_dns:default.log
10482023-11-18T01:33:06.636Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_2b6b213a-4ec2-4110-9caf-98407aaddbbe/root/var/svc/log/oxide-external_dns:default.log.0 (8056 bytes)
10492023-11-18T01:33:07.704Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_2b6b213a-4ec2-4110-9caf-98407aaddbbe/root/var/svc/log/oxide-external_dns:default.log.0
10502023-11-18T01:33:07.729Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_97a36b64-55d1-4a2a-8f81-df6ba5e1b517/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10512023-11-18T01:33:08.777Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_97a36b64-55d1-4a2a-8f81-df6ba5e1b517/root/var/svc/log/oxide-cockroachdb:default.log
10522023-11-18T01:33:08.802Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_97a36b64-55d1-4a2a-8f81-df6ba5e1b517/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10532023-11-18T01:33:09.883Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_97a36b64-55d1-4a2a-8f81-df6ba5e1b517/root/var/svc/log/oxide-cockroachdb:default.log.0
10542023-11-18T01:33:09.909Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7a10051b-0b08-4caa-b169-83e62e4a0080/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10552023-11-18T01:33:10.953Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7a10051b-0b08-4caa-b169-83e62e4a0080/root/var/svc/log/oxide-crucible-agent:default.log
10562023-11-18T01:33:10.980Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7a10051b-0b08-4caa-b169-83e62e4a0080/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10572023-11-18T01:33:12.052Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7a10051b-0b08-4caa-b169-83e62e4a0080/root/var/svc/log/oxide-crucible-agent:default.log.0
10582023-11-18T01:33:12.076Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7a10051b-0b08-4caa-b169-83e62e4a0080/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10592023-11-18T01:33:13.124Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7a10051b-0b08-4caa-b169-83e62e4a0080/root/var/svc/log/oxide-crucible-downstairs:default.log
10602023-11-18T01:33:13.148Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7a10051b-0b08-4caa-b169-83e62e4a0080/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10612023-11-18T01:33:14.227Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_7a10051b-0b08-4caa-b169-83e62e4a0080/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10622023-11-18T01:33:14.260Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_e1119284-61e5-406c-9a30-4fbec515cc1b/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10632023-11-18T01:33:15.781Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_e1119284-61e5-406c-9a30-4fbec515cc1b/root/var/svc/log/oxide-crucible-pantry:default.log
10642023-11-18T01:33:15.805Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_e1119284-61e5-406c-9a30-4fbec515cc1b/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10652023-11-18T01:33:16.873Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_e1119284-61e5-406c-9a30-4fbec515cc1b/root/var/svc/log/oxide-crucible-pantry:default.log.0
10662023-11-18T01:33:16.899Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_e724bd41-b1c6-4349-8f8d-ec789c4037f4/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10672023-11-18T01:33:18.423Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_e724bd41-b1c6-4349-8f8d-ec789c4037f4/root/var/svc/log/oxide-crucible-pantry:default.log
10682023-11-18T01:33:18.447Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_e724bd41-b1c6-4349-8f8d-ec789c4037f4/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10692023-11-18T01:33:19.525Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_e724bd41-b1c6-4349-8f8d-ec789c4037f4/root/var/svc/log/oxide-crucible-pantry:default.log.0
10702023-11-18T01:33:19.549Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_488ac2e5-c209-4809-84b0-db08fd61063c/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10712023-11-18T01:33:20.598Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_488ac2e5-c209-4809-84b0-db08fd61063c/root/var/svc/log/oxide-cockroachdb:default.log
10722023-11-18T01:33:20.626Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_488ac2e5-c209-4809-84b0-db08fd61063c/root/var/svc/log/oxide-cockroachdb:default.log.0 (5343 bytes)
10732023-11-18T01:33:21.698Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_488ac2e5-c209-4809-84b0-db08fd61063c/root/var/svc/log/oxide-cockroachdb:default.log.0
10742023-11-18T01:33:21.728Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_106e3ab4-7cff-4382-9387-591eaed77ee1/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10752023-11-18T01:33:22.774Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_106e3ab4-7cff-4382-9387-591eaed77ee1/root/var/svc/log/oxide-crucible-agent:default.log
10762023-11-18T01:33:22.800Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_106e3ab4-7cff-4382-9387-591eaed77ee1/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
10772023-11-18T01:33:23.872Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_106e3ab4-7cff-4382-9387-591eaed77ee1/root/var/svc/log/oxide-crucible-agent:default.log.0
10782023-11-18T01:33:23.895Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_106e3ab4-7cff-4382-9387-591eaed77ee1/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10792023-11-18T01:33:24.947Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_106e3ab4-7cff-4382-9387-591eaed77ee1/root/var/svc/log/oxide-crucible-downstairs:default.log
10802023-11-18T01:33:24.976Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_106e3ab4-7cff-4382-9387-591eaed77ee1/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10812023-11-18T01:33:26.096Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_106e3ab4-7cff-4382-9387-591eaed77ee1/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10822023-11-18T01:33:26.121Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_58937e4f-8a2b-46e8-9799-9aa7c714f432/root/var/svc/log/oxide-internal_dns:default.log (7508 bytes)
10832023-11-18T01:33:26.177Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_58937e4f-8a2b-46e8-9799-9aa7c714f432/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 7508 -> 15020
10842023-11-18T01:33:27.247Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_58937e4f-8a2b-46e8-9799-9aa7c714f432/root/var/svc/log/oxide-internal_dns:default.log
10852023-11-18T01:33:27.271Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_58937e4f-8a2b-46e8-9799-9aa7c714f432/root/var/svc/log/oxide-internal_dns:default.log.0 (104168 bytes)
10862023-11-18T01:33:28.606Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_58937e4f-8a2b-46e8-9799-9aa7c714f432/root/var/svc/log/oxide-internal_dns:default.log.0
10872023-11-18T01:33:28.631Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_b2d1fed7-2ddf-4e87-909c-32c8db178f7f/root/var/svc/log/oxide-nexus:default.log (1315291 bytes)
10882023-11-18T01:33:38.771Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_b2d1fed7-2ddf-4e87-909c-32c8db178f7f/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1315291 -> 1452558
10892023-11-18T01:33:39.880Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_b2d1fed7-2ddf-4e87-909c-32c8db178f7f/root/var/svc/log/oxide-nexus:default.log
10902023-11-18T01:33:39.906Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_b2d1fed7-2ddf-4e87-909c-32c8db178f7f/root/var/svc/log/oxide-nexus:default.log.0 (3022270 bytes)
10912023-11-18T01:33:43.880Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_b2d1fed7-2ddf-4e87-909c-32c8db178f7f/root/var/svc/log/oxide-nexus:default.log.0
10922023-11-18T01:33:43.923Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_a798af66-c22a-4733-b30a-c4d4603804ca/root/var/svc/log/oxide-clickhouse:default.log (331434 bytes)
10932023-11-18T01:33:45.454Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_a798af66-c22a-4733-b30a-c4d4603804ca/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 331434 -> 658748
10942023-11-18T01:33:46.525Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_a798af66-c22a-4733-b30a-c4d4603804ca/root/var/svc/log/oxide-clickhouse:default.log
10952023-11-18T01:33:46.551Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_a798af66-c22a-4733-b30a-c4d4603804ca/root/var/svc/log/oxide-clickhouse:default.log.0 (341762 bytes)
10962023-11-18T01:33:48.226Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_a798af66-c22a-4733-b30a-c4d4603804ca/root/var/svc/log/oxide-clickhouse:default.log.0
10972023-11-18T01:33:48.250Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_976803c4-6485-460a-99af-cea8584a7953/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10982023-11-18T01:33:49.307Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_976803c4-6485-460a-99af-cea8584a7953/root/var/svc/log/oxide-crucible-agent:default.log
10992023-11-18T01:33:49.331Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_976803c4-6485-460a-99af-cea8584a7953/root/var/svc/log/oxide-crucible-agent:default.log.0 (14116 bytes)
11002023-11-18T01:33:50.445Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_976803c4-6485-460a-99af-cea8584a7953/root/var/svc/log/oxide-crucible-agent:default.log.0
11012023-11-18T01:33:50.472Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_976803c4-6485-460a-99af-cea8584a7953/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11022023-11-18T01:33:51.516Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_976803c4-6485-460a-99af-cea8584a7953/root/var/svc/log/oxide-crucible-downstairs:default.log
11032023-11-18T01:33:51.541Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_976803c4-6485-460a-99af-cea8584a7953/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11042023-11-18T01:33:52.613Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_976803c4-6485-460a-99af-cea8584a7953/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11052023-11-18T01:33:52.637Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_976803c4-6485-460a-99af-cea8584a7953/root/var/svc/log/oxide-crucible-downstairs:downstairs-c5c46bd1-9540-472e-907b-db85fe5b49ae.log (0 bytes)
11062023-11-18T01:33:54.098Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_976803c4-6485-460a-99af-cea8584a7953/root/var/svc/log/oxide-crucible-downstairs:downstairs-c5c46bd1-9540-472e-907b-db85fe5b49ae.log
11072023-11-18T01:33:54.148Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_976803c4-6485-460a-99af-cea8584a7953/root/var/svc/log/oxide-crucible-downstairs:downstairs-c5c46bd1-9540-472e-907b-db85fe5b49ae.log.0 (3051 bytes)
11082023-11-18T01:33:55.226Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_976803c4-6485-460a-99af-cea8584a7953/root/var/svc/log/oxide-crucible-downstairs:downstairs-c5c46bd1-9540-472e-907b-db85fe5b49ae.log.0
11092023-11-18T01:33:55.251Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_12c72f5c-512e-4d48-999d-f560d03ab099/root/var/svc/log/oxide-internal_dns:default.log (7513 bytes)
11102023-11-18T01:33:55.318Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_12c72f5c-512e-4d48-999d-f560d03ab099/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 7513 -> 15021
11112023-11-18T01:33:56.395Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_12c72f5c-512e-4d48-999d-f560d03ab099/root/var/svc/log/oxide-internal_dns:default.log
11122023-11-18T01:33:56.420Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_12c72f5c-512e-4d48-999d-f560d03ab099/root/var/svc/log/oxide-internal_dns:default.log.0 (107409 bytes)
11132023-11-18T01:33:57.830Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_12c72f5c-512e-4d48-999d-f560d03ab099/root/var/svc/log/oxide-internal_dns:default.log.0
11142023-11-18T01:33:57.854Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_be5e85c0-f058-473a-a541-2827fadd32a0/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11152023-11-18T01:33:58.916Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_be5e85c0-f058-473a-a541-2827fadd32a0/root/var/svc/log/oxide-cockroachdb:default.log
11162023-11-18T01:33:58.946Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_be5e85c0-f058-473a-a541-2827fadd32a0/root/var/svc/log/oxide-cockroachdb:default.log.0 (5370 bytes)
11172023-11-18T01:34:00.017Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_be5e85c0-f058-473a-a541-2827fadd32a0/root/var/svc/log/oxide-cockroachdb:default.log.0
11182023-11-18T01:34:00.043Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a95535e2-a23d-4e1c-a3a4-ce6208cda7c5/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11192023-11-18T01:34:01.092Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a95535e2-a23d-4e1c-a3a4-ce6208cda7c5/root/var/svc/log/oxide-crucible-agent:default.log
11202023-11-18T01:34:01.127Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a95535e2-a23d-4e1c-a3a4-ce6208cda7c5/root/var/svc/log/oxide-crucible-agent:default.log.0 (14576 bytes)
11212023-11-18T01:34:02.236Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a95535e2-a23d-4e1c-a3a4-ce6208cda7c5/root/var/svc/log/oxide-crucible-agent:default.log.0
11222023-11-18T01:34:02.262Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a95535e2-a23d-4e1c-a3a4-ce6208cda7c5/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11232023-11-18T01:34:03.308Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a95535e2-a23d-4e1c-a3a4-ce6208cda7c5/root/var/svc/log/oxide-crucible-downstairs:default.log
11242023-11-18T01:34:03.335Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a95535e2-a23d-4e1c-a3a4-ce6208cda7c5/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11252023-11-18T01:34:04.404Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a95535e2-a23d-4e1c-a3a4-ce6208cda7c5/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11262023-11-18T01:34:04.433Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a95535e2-a23d-4e1c-a3a4-ce6208cda7c5/root/var/svc/log/oxide-crucible-downstairs:downstairs-98c996e5-63d7-4534-9d74-81314fdd7701.log (0 bytes)
11272023-11-18T01:34:05.481Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a95535e2-a23d-4e1c-a3a4-ce6208cda7c5/root/var/svc/log/oxide-crucible-downstairs:downstairs-98c996e5-63d7-4534-9d74-81314fdd7701.log
11282023-11-18T01:34:05.509Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a95535e2-a23d-4e1c-a3a4-ce6208cda7c5/root/var/svc/log/oxide-crucible-downstairs:downstairs-98c996e5-63d7-4534-9d74-81314fdd7701.log.0 (3050 bytes)
11292023-11-18T01:34:06.592Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a95535e2-a23d-4e1c-a3a4-ce6208cda7c5/root/var/svc/log/oxide-crucible-downstairs:downstairs-98c996e5-63d7-4534-9d74-81314fdd7701.log.0
11302023-11-18T01:34:06.618Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dc1567fa-cf8f-4045-bdff-8ed2671d7c97/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11312023-11-18T01:34:07.664Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dc1567fa-cf8f-4045-bdff-8ed2671d7c97/root/var/svc/log/oxide-crucible-agent:default.log
11322023-11-18T01:34:07.688Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dc1567fa-cf8f-4045-bdff-8ed2671d7c97/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11332023-11-18T01:34:09.172Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dc1567fa-cf8f-4045-bdff-8ed2671d7c97/root/var/svc/log/oxide-crucible-agent:default.log.0
11342023-11-18T01:34:09.200Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dc1567fa-cf8f-4045-bdff-8ed2671d7c97/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11352023-11-18T01:34:10.672Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dc1567fa-cf8f-4045-bdff-8ed2671d7c97/root/var/svc/log/oxide-crucible-downstairs:default.log
11362023-11-18T01:34:10.698Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dc1567fa-cf8f-4045-bdff-8ed2671d7c97/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11372023-11-18T01:34:11.773Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_dc1567fa-cf8f-4045-bdff-8ed2671d7c97/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11382023-11-18T01:34:11.801Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_9cd147d9-ce2c-4da3-a54e-5a4780e17840/root/var/svc/log/oxide-internal_dns:default.log (7512 bytes)
11392023-11-18T01:34:11.870Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_9cd147d9-ce2c-4da3-a54e-5a4780e17840/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 7512 -> 15023
11402023-11-18T01:34:12.954Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_9cd147d9-ce2c-4da3-a54e-5a4780e17840/root/var/svc/log/oxide-internal_dns:default.log
11412023-11-18T01:34:12.979Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_9cd147d9-ce2c-4da3-a54e-5a4780e17840/root/var/svc/log/oxide-internal_dns:default.log.0 (102933 bytes)
11422023-11-18T01:34:14.338Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_9cd147d9-ce2c-4da3-a54e-5a4780e17840/root/var/svc/log/oxide-internal_dns:default.log.0
11432023-11-18T01:34:14.367Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_fd8d1f08-3061-4686-91d5-7ecdcd0e5752/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11442023-11-18T01:34:15.476Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_fd8d1f08-3061-4686-91d5-7ecdcd0e5752/root/var/svc/log/oxide-crucible-pantry:default.log
11452023-11-18T01:34:15.505Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_fd8d1f08-3061-4686-91d5-7ecdcd0e5752/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
11462023-11-18T01:34:17.006Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_fd8d1f08-3061-4686-91d5-7ecdcd0e5752/root/var/svc/log/oxide-crucible-pantry:default.log.0
11472023-11-18T01:34:17.030Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834/root/var/log/chrony/measurements.log (17673 bytes)
11482023-11-18T01:34:17.121Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834/root/var/log/chrony/measurements.log" changed size mid upload: 17673 -> 23153
11492023-11-18T01:34:18.201Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834/root/var/log/chrony/measurements.log
11502023-11-18T01:34:18.228Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834/root/var/log/chrony/statistics.log (10998 bytes)
11512023-11-18T01:34:18.293Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834/root/var/log/chrony/statistics.log" changed size mid upload: 10998 -> 15678
11522023-11-18T01:34:19.364Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834/root/var/log/chrony/statistics.log
11532023-11-18T01:34:19.389Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834/root/var/log/chrony/tracking.log (2118 bytes)
11542023-11-18T01:34:19.413Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834/root/var/log/chrony/tracking.log" changed size mid upload: 2118 -> 2382
11552023-11-18T01:34:20.483Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1252ff97-3458-4721-9c47-bd819b009834/root/var/log/chrony/tracking.log