01HJWJZ5FY4SDRFECQN5TE5HJP: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.27",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HJWJZQAE845XC7A7ER5FHVF8

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-30T06:12:31.271Zjob dependencies complete; ready to run (waiting for 46 m 20 s)
22023-12-30T06:14:25.391Zjob assigned to worker 01HJWNPPND6R92QCTKXRJJSN0D (queued for 1 m 54 s)
32023-12-30T06:14:25.653Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-12-30T06:14:25.702Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-12-30T06:14:25.740Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-12-30T06:14:25.773Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-30T06:14:25.807Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-12-30T06:14:25.840ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-30T06:14:25.881Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-30T06:14:25.921ZOxide Helios Version helios-2.0.22333 64-bit
112023-12-30T06:14:25.959ZConfiguring devices.
122023-12-30T06:14:25.996Z##################################################
132023-12-30T06:14:26.030Z#### ############# ##
142023-12-30T06:14:26.063Z### ### ############ ##
152023-12-30T06:14:26.097Z## ### # ## ### ## Oxide Computer Company ##
162023-12-30T06:14:26.130Z## ## ## ### # ### ##
172023-12-30T06:14:26.163Z## # ### #### #### This Station Under ##
182023-12-30T06:14:26.197Z### ### #### # ### Computer Control ##
192023-12-30T06:14:26.230Z#### #### ### ## ##
202023-12-30T06:14:26.264Z##################################################
212023-12-30T06:14:26.297ZHostname: sock
222023-12-30T06:14:26.342ZLoading smf(7) service descriptions:
232023-12-30T06:14:27.789Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-73e660426b4fbae9.gz
242023-12-30T06:14:29.011Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-73e660426b4fbae9.gz
252023-12-30T06:14:29.037Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-30T06:14:47.489Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-30T06:14:47.489Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-26f64a1dd7a0c081.gz
282023-12-30T06:15:06.270Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-26f64a1dd7a0c081.gz
292023-12-30T06:15:06.270Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-30T06:15:07.327Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-30T06:15:07.327Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-30T06:15:10.926Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-30T06:15:10.956Zdownloading input: /input/package/work/version.txt
342023-12-30T06:15:11.366Zdownloaded input: /input/package/work/version.txt
352023-12-30T06:15:11.391Zdownloading input: /input/package/work/package.tar.gz
362023-12-30T06:15:28.478Zdownloaded input: /input/package/work/package.tar.gz
372023-12-30T06:15:28.478Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-30T06:15:57.676Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-30T06:15:57.676Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-30T06:16:02.796Zsock console login: Error: No response to keepalive - Terminating session
412023-12-30T06:16:05.396Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
422023-12-30T06:16:05.428Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
432023-12-30T06:16:12.770ZError: No response de-activating SOL payload
442023-12-30T06:16:14.226Z[SOL Session operational. Use ~? for help]
452023-12-30T06:16:41.778Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
462023-12-30T06:16:41.778Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
472023-12-30T06:17:19.329Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
482023-12-30T06:17:19.329Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
492023-12-30T06:17:39.936Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
502023-12-30T06:17:39.936Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
512023-12-30T06:17:43.926Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
522023-12-30T06:17:43.926Zdownloading input: /input/package/work/zones/crucible.tar.gz
532023-12-30T06:17:51.236Zdownloaded input: /input/package/work/zones/crucible.tar.gz
542023-12-30T06:17:51.236Zdownloading input: /input/package/work/zones/external-dns.tar.gz
552023-12-30T06:17:55.945Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
562023-12-30T06:17:55.945Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
572023-12-30T06:18:00.158Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
582023-12-30T06:18:00.158Zdownloading input: /input/package/work/zones/ntp.tar.gz
592023-12-30T06:18:00.196Zdownloaded input: /input/package/work/zones/ntp.tar.gz
602023-12-30T06:18:00.196Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
612023-12-30T06:18:03.431Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
622023-12-30T06:18:03.431Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
632023-12-30T06:18:06.697Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
642023-12-30T06:18:06.698Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
652023-12-30T06:18:19.797Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
662023-12-30T06:18:19.797Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
672023-12-30T06:18:32.998Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
682023-12-30T06:18:32.998Zdownloading input: /input/package/work/zones/overlay.tar.gz
692023-12-30T06:18:33.060Zdownloaded input: /input/package/work/zones/overlay.tar.gz
702023-12-30T06:18:33.061Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
712023-12-30T06:18:39.440Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
722023-12-30T06:18:39.440Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
732023-12-30T06:18:50.944Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
742023-12-30T06:18:50.944Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
752023-12-30T06:19:02.344ZError: No response to keepalive - Terminating session
762023-12-30T06:19:12.559ZError: No response de-activating SOL payload
772023-12-30T06:19:13.480Z[SOL Session operational. Use ~? for help]
782023-12-30T06:19:33.770Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
792023-12-30T06:19:33.770Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
802023-12-30T06:20:03.114Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
812023-12-30T06:20:03.140Zstarting task 0: "setup"
822023-12-30T06:20:03.141Z++ uname -s
832023-12-30T06:20:03.141Z+ kern=SunOS
842023-12-30T06:20:03.141Z+ case "$kern" in
852023-12-30T06:20:03.141Z+ groupadd -g 12345 build
862023-12-30T06:20:03.141Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
872023-12-30T06:20:05.133Z+ zfs create -o mountpoint=/work rpool/work
882023-12-30T06:20:05.169Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
892023-12-30T06:20:05.169Z+ home_fs=zfs
902023-12-30T06:20:05.169Z+ [[ zfs == autofs ]]
912023-12-30T06:20:05.169Z+ mkdir -p /home/build
922023-12-30T06:20:05.195Z+ chown build:build /home/build /work
932023-12-30T06:20:07.164Z+ chmod 0700 /home/build /work
942023-12-30T06:20:07.189Zprocess exited: duration 4052 ms, exit code 0
 
952023-12-30T06:20:07.258Zstarting task 1: "authentication"
962023-12-30T06:20:07.301Zprocess exited: duration 43 ms, exit code 0
 
972023-12-30T06:20:07.351Zstarting task 2: "build"
982023-12-30T06:20:07.381Z+ trap _exit_trap EXIT
992023-12-30T06:20:07.382Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
1002023-12-30T06:20:07.561Zsecpolicy_fs_linkdir: 0x55 = 0x55
1012023-12-30T06:20:07.561Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
1022023-12-30T06:20:07.561Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
1032023-12-30T06:20:07.562Zsecpolicy_fs_linkdir+3: 0 = 0xe5
1042023-12-30T06:20:07.562Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1052023-12-30T06:20:07.562Zsecpolicy_fs_linkdir+5: 0 = 0x1
1062023-12-30T06:20:07.562Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1072023-12-30T06:20:07.562Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1082023-12-30T06:20:07.587Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1092023-12-30T06:20:07.587Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1102023-12-30T06:20:07.587Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1112023-12-30T06:20:07.587Z+ [[ -d /opt/oxide ]]
1122023-12-30T06:20:07.587Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1132023-12-30T06:20:07.587Z+ pfexec mkdir /opt/oxide
1142023-12-30T06:20:07.587Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1152023-12-30T06:20:07.587Z+ [[ -d /opt/oxide-underneath ]]
1162023-12-30T06:20:07.587Z+ cd /opt/oxide-underneath
1172023-12-30T06:20:07.587Z+ pfexec tar ceEp@/f - .
1182023-12-30T06:20:07.587Z+ cd /opt/oxide
1192023-12-30T06:20:07.587Z+ pfexec tar xveEp@/f -
1202023-12-30T06:20:07.683Zx ., 0 bytes, 0 tape blocks
1212023-12-30T06:20:07.683Zx ./opte, 0 bytes, 0 tape blocks
1222023-12-30T06:20:07.683Zx ./opte/bin, 0 bytes, 0 tape blocks
1232023-12-30T06:20:07.683Zx ./opte/bin/opteadm, 52567720 bytes, 102672 tape blocks
1242023-12-30T06:20:07.711Z+ pfexec rm -rf /opt/oxide-underneath
1252023-12-30T06:20:07.711Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1262023-12-30T06:20:07.736Z+ pfexec mkdir /opt/oxide/work
1272023-12-30T06:20:07.761Z+ pfexec chown build:build /opt/oxide/work
1282023-12-30T06:20:07.761Z+ cd /opt/oxide/work
1292023-12-30T06:20:07.761Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1302023-12-30T06:20:09.296ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1312023-12-30T06:20:09.296Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1322023-12-30T06:20:09.296Zx out/omicron-sled-agent.tar, 113966080 bytes, 222590 tape blocks
1332023-12-30T06:20:09.296Zx out/target/test, 75 bytes, 1 tape blocks
1342023-12-30T06:20:09.297Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1352023-12-30T06:20:09.297Zx package-manifest.toml, 22445 bytes, 44 tape blocks
1362023-12-30T06:20:09.297Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1372023-12-30T06:20:09.297Zx target/release/omicron-package, 42803264 bytes, 83601 tape blocks
1382023-12-30T06:20:09.297Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1392023-12-30T06:20:09.297Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1402023-12-30T06:20:09.297Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1412023-12-30T06:20:09.297Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1422023-12-30T06:20:09.297Zx tools/scrimlet/softnpu-init.sh, 2149 bytes, 5 tape blocks
1432023-12-30T06:20:09.297Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1442023-12-30T06:20:09.297Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1452023-12-30T06:20:09.322Z
1462023-12-30T06:20:09.322Zreal 1.549419033
1472023-12-30T06:20:09.322Zuser 1.396000289
1482023-12-30T06:20:09.322Zsys 0.273645487
1492023-12-30T06:20:09.322Ztrap 0.000054030
1502023-12-30T06:20:09.322Ztflt 0.000000000
1512023-12-30T06:20:09.322Zdflt 0.000000000
1522023-12-30T06:20:09.322Zkflt 0.000000000
1532023-12-30T06:20:09.323Zlock 0.000000000
1542023-12-30T06:20:09.323Zslp 1.392063680
1552023-12-30T06:20:09.323Zlat 0.034956968
1562023-12-30T06:20:09.323Zstop 0.000121053
1572023-12-30T06:20:09.323Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1582023-12-30T06:20:10.384Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1592023-12-30T06:20:10.409Z+ mkdir tests
1602023-12-30T06:20:10.409Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1612023-12-30T06:20:10.409Z+ ptime -m gunzip
1622023-12-30T06:20:10.409Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-73e660426b4fbae9
1632023-12-30T06:20:10.541Z
1642023-12-30T06:20:10.566Zreal 0.138338442
1652023-12-30T06:20:10.566Zuser 0.126212665
1662023-12-30T06:20:10.566Zsys 0.011876231
1672023-12-30T06:20:10.566Ztrap 0.000012659
1682023-12-30T06:20:10.566Ztflt 0.000000000
1692023-12-30T06:20:10.566Zdflt 0.000000000
1702023-12-30T06:20:10.566Zkflt 0.000000000
1712023-12-30T06:20:10.566Zlock 0.000000000
1722023-12-30T06:20:10.566Zslp 0.000023830
1732023-12-30T06:20:10.567Zlat 0.000025170
1742023-12-30T06:20:10.575Zstop 0.000126203
1752023-12-30T06:20:10.576Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-73e660426b4fbae9
1762023-12-30T06:20:10.576Z+ chmod a+x tests/bootstrap-73e660426b4fbae9
1772023-12-30T06:20:10.576Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1782023-12-30T06:20:10.576Z+ ptime -m gunzip
1792023-12-30T06:20:10.576Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1802023-12-30T06:20:12.802Z
1812023-12-30T06:20:12.827Zreal 2.249532298
1822023-12-30T06:20:12.827Zuser 2.106409446
1832023-12-30T06:20:12.827Zsys 0.142753004
1842023-12-30T06:20:12.827Ztrap 0.000185974
1852023-12-30T06:20:12.827Ztflt 0.000000000
1862023-12-30T06:20:12.827Zdflt 0.000000000
1872023-12-30T06:20:12.827Zkflt 0.000000000
1882023-12-30T06:20:12.827Zlock 0.000000000
1892023-12-30T06:20:12.827Zslp 0.000021080
1902023-12-30T06:20:12.827Zlat 0.000011870
1912023-12-30T06:20:12.827Zstop 0.000096842
1922023-12-30T06:20:12.828Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1932023-12-30T06:20:12.828Z+ chmod a+x tests/bootstrap
1942023-12-30T06:20:12.828Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1952023-12-30T06:20:12.828Z+ ptime -m gunzip
1962023-12-30T06:20:12.828Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-26f64a1dd7a0c081
1972023-12-30T06:20:15.134Z
1982023-12-30T06:20:15.159Zreal 2.320470855
1992023-12-30T06:20:15.159Zuser 2.174531171
2002023-12-30T06:20:15.159Zsys 0.145504303
2012023-12-30T06:20:15.159Ztrap 0.000175244
2022023-12-30T06:20:15.159Ztflt 0.000000000
2032023-12-30T06:20:15.159Zdflt 0.000000000
2042023-12-30T06:20:15.159Zkflt 0.000000000
2052023-12-30T06:20:15.159Zlock 0.000000000
2062023-12-30T06:20:15.159Zslp 0.000019620
2072023-12-30T06:20:15.159Zlat 0.000024590
2082023-12-30T06:20:15.159Zstop 0.000158433
2092023-12-30T06:20:15.159Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-26f64a1dd7a0c081
2102023-12-30T06:20:15.159Z+ chmod a+x tests/end_to_end_tests-26f64a1dd7a0c081
2112023-12-30T06:20:15.160Z+ bmat address ls
2122023-12-30T06:20:15.160ZNAME CIDR FIRST LAST COUNT
2132023-12-30T06:20:15.160Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2142023-12-30T06:20:15.160Z++ bmat address ls -f extra -Ho first,last,count
2152023-12-30T06:20:15.184Z+ set -- 10.151.2.164 10.151.2.227 64
2162023-12-30T06:20:15.184Z+ EXTRA_IP_START=10.151.2.164
2172023-12-30T06:20:15.184Z+ EXTRA_IP_END=10.151.2.227
2182023-12-30T06:20:15.184Z+ EXTRA_IP_COUNT=64
2192023-12-30T06:20:15.185Z+ (( EXTRA_IP_COUNT >= 32 ))
2202023-12-30T06:20:15.185Z+ EXTRA_IP_BASE=10.151.2
2212023-12-30T06:20:15.185Z+ EXTRA_IP_FOCTET=164
2222023-12-30T06:20:15.185Z+ EXTRA_IP_LOCTET=227
2232023-12-30T06:20:15.185Z+ SERVICE_IP_POOL_START=10.151.2.164
2242023-12-30T06:20:15.185Z+ SERVICE_IP_POOL_END=10.151.2.173
2252023-12-30T06:20:15.185Z+ DNS_IP1=10.151.2.164
2262023-12-30T06:20:15.185Z+ DNS_IP2=10.151.2.165
2272023-12-30T06:20:15.186Z+ UPLINK_IP=10.151.2.174
2282023-12-30T06:20:15.186Z+ IPPOOL_START=10.151.2.175
2292023-12-30T06:20:15.186Z+ IPPOOL_END=10.151.2.227
2302023-12-30T06:20:15.186Z++ ipadm show-addr -po type,addr
2312023-12-30T06:20:15.186Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2322023-12-30T06:20:15.186Z+ GATEWAY_IP=10.151.2.100
2332023-12-30T06:20:15.186Z+ [[ -n 10.151.2.100 ]]
2342023-12-30T06:20:15.187Z+ ping -s 10.151.2.100 56 1
2352023-12-30T06:20:15.187ZPING 10.151.2.100: 56 data bytes
2362023-12-30T06:20:15.187Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2372023-12-30T06:20:15.187Z
2382023-12-30T06:20:15.187Z----10.151.2.100 PING Statistics----
2392023-12-30T06:20:15.187Z1 packets transmitted, 1 packets received, 0% packet loss
2402023-12-30T06:20:15.187Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2412023-12-30T06:20:15.187Z++ arp -an
2422023-12-30T06:20:15.187Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2432023-12-30T06:20:15.211Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2442023-12-30T06:20:15.211Z+ routeadm -e ipv4-forwarding -u
2452023-12-30T06:20:15.298Z+ PXA_START=10.151.2.164
2462023-12-30T06:20:15.299Z+ PXA_END=10.151.2.227
2472023-12-30T06:20:15.299Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2482023-12-30T06:20:15.323Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2492023-12-30T06:20:15.431Z+ ZPOOL_VDEV_DIR=/scratch
2502023-12-30T06:20:15.431Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2512023-12-30T06:20:15.487Z+++ dirname ./tools/create_virtual_hardware.sh
2522023-12-30T06:20:15.487Z++ cd ./tools
2532023-12-30T06:20:15.487Z++ pwd
2542023-12-30T06:20:15.487Z+ SOURCE_DIR=/opt/oxide/work/tools
2552023-12-30T06:20:15.487Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2562023-12-30T06:20:15.487Z+ SOFTNPU_MODE=zone
2572023-12-30T06:20:15.487Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2582023-12-30T06:20:15.487Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2592023-12-30T06:20:15.487Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2602023-12-30T06:20:15.487Z++ dladm show-phys -p -o LINK
2612023-12-30T06:20:15.487Z++ head -1
2622023-12-30T06:20:15.516Z+ PHYSICAL_LINK=igb0
2632023-12-30T06:20:15.541Z+ echo 'Using igb0 as physical link'
2642023-12-30T06:20:15.541ZUsing igb0 as physical link
2652023-12-30T06:20:15.541Z+ ensure_run_as_root
2662023-12-30T06:20:15.541Z++ id -u
2672023-12-30T06:20:15.541Z+ [[ 0 -ne 0 ]]
2682023-12-30T06:20:15.541Z+ ensure_zpools
2692023-12-30T06:20:15.542Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2702023-12-30T06:20:15.542Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2712023-12-30T06:20:15.542Z+ readarray -t ZPOOLS
2722023-12-30T06:20:15.542Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2732023-12-30T06:20:15.542Z++ sed 's/[ ",]//g'
2742023-12-30T06:20:15.542Z+ for ZPOOL in "${ZPOOLS[@]}"
2752023-12-30T06:20:15.542Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2762023-12-30T06:20:15.542ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2772023-12-30T06:20:15.542Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2782023-12-30T06:20:15.542Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2792023-12-30T06:20:15.542Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2802023-12-30T06:20:15.567Z0+0 records in
2812023-12-30T06:20:15.567Z0+0 records out
2822023-12-30T06:20:15.569Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2832023-12-30T06:20:15.569Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2842023-12-30T06:20:15.569Z+ set +x
2852023-12-30T06:20:15.569ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2862023-12-30T06:20:15.569Z++ zpool list -o name
2872023-12-30T06:20:15.569Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2882023-12-30T06:20:15.569Z+ [[ -z '' ]]
2892023-12-30T06:20:15.569Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2902023-12-30T06:20:15.596Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2912023-12-30T06:20:15.621Z+ set +x
2922023-12-30T06:20:15.621ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2932023-12-30T06:20:15.621Z+ for ZPOOL in "${ZPOOLS[@]}"
2942023-12-30T06:20:15.621Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2952023-12-30T06:20:15.621ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2962023-12-30T06:20:15.621Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2972023-12-30T06:20:15.621Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2982023-12-30T06:20:15.621Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2992023-12-30T06:20:15.621Z0+0 records in
3002023-12-30T06:20:15.621Z0+0 records out
3012023-12-30T06:20:15.621Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3022023-12-30T06:20:15.621Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3032023-12-30T06:20:15.621Z+ set +x
3042023-12-30T06:20:15.621ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3052023-12-30T06:20:15.621Z++ zpool list -o name
3062023-12-30T06:20:15.621Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3072023-12-30T06:20:15.621Z+ [[ -z '' ]]
3082023-12-30T06:20:15.621Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3092023-12-30T06:20:15.646Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3102023-12-30T06:20:15.646Z+ set +x
3112023-12-30T06:20:15.646ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3122023-12-30T06:20:15.646Z+ for ZPOOL in "${ZPOOLS[@]}"
3132023-12-30T06:20:15.647Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3142023-12-30T06:20:15.647ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3152023-12-30T06:20:15.647Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3162023-12-30T06:20:15.647Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3172023-12-30T06:20:15.647Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3182023-12-30T06:20:15.647Z0+0 records in
3192023-12-30T06:20:15.647Z0+0 records out
3202023-12-30T06:20:15.647Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3212023-12-30T06:20:15.647Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3222023-12-30T06:20:15.647Z+ set +x
3232023-12-30T06:20:15.647ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3242023-12-30T06:20:15.647Z++ zpool list -o name
3252023-12-30T06:20:15.647Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3262023-12-30T06:20:15.672Z+ [[ -z '' ]]
3272023-12-30T06:20:15.672Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3282023-12-30T06:20:15.697Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3292023-12-30T06:20:15.697Z+ set +x
3302023-12-30T06:20:15.697ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3312023-12-30T06:20:15.697Z+ for ZPOOL in "${ZPOOLS[@]}"
3322023-12-30T06:20:15.697Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3332023-12-30T06:20:15.697ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3342023-12-30T06:20:15.697Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3352023-12-30T06:20:15.697Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3362023-12-30T06:20:15.698Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3372023-12-30T06:20:15.698Z0+0 records in
3382023-12-30T06:20:15.698Z0+0 records out
3392023-12-30T06:20:15.698Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3402023-12-30T06:20:15.698Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3412023-12-30T06:20:15.698Z+ set +x
3422023-12-30T06:20:15.698ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3432023-12-30T06:20:15.698Z++ zpool list -o name
3442023-12-30T06:20:15.698Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3452023-12-30T06:20:15.698Z+ [[ -z '' ]]
3462023-12-30T06:20:15.698Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3472023-12-30T06:20:15.723Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3482023-12-30T06:20:15.723Z+ set +x
3492023-12-30T06:20:15.723ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3502023-12-30T06:20:15.723Z+ for ZPOOL in "${ZPOOLS[@]}"
3512023-12-30T06:20:15.723Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3522023-12-30T06:20:15.723ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3532023-12-30T06:20:15.723Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3542023-12-30T06:20:15.723Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3552023-12-30T06:20:15.723Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3562023-12-30T06:20:15.723Z0+0 records in
3572023-12-30T06:20:15.723Z0+0 records out
3582023-12-30T06:20:15.723Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3592023-12-30T06:20:15.723Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3602023-12-30T06:20:15.723Z+ set +x
3612023-12-30T06:20:15.723ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3622023-12-30T06:20:15.748Z++ zpool list -o name
3632023-12-30T06:20:15.748Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-12-30T06:20:15.748Z+ [[ -z '' ]]
3652023-12-30T06:20:15.748Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-12-30T06:20:15.773Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-12-30T06:20:15.773Z+ set +x
3682023-12-30T06:20:15.773ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3692023-12-30T06:20:15.774Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-12-30T06:20:15.774Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3712023-12-30T06:20:15.774ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3722023-12-30T06:20:15.774Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3732023-12-30T06:20:15.774Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3742023-12-30T06:20:15.774Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3752023-12-30T06:20:15.774Z0+0 records in
3762023-12-30T06:20:15.774Z0+0 records out
3772023-12-30T06:20:15.774Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3782023-12-30T06:20:15.774Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3792023-12-30T06:20:15.774Z+ set +x
3802023-12-30T06:20:15.774ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3812023-12-30T06:20:15.774Z++ zpool list -o name
3822023-12-30T06:20:15.774Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3832023-12-30T06:20:15.800Z+ [[ -z '' ]]
3842023-12-30T06:20:15.800Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3852023-12-30T06:20:15.825Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3862023-12-30T06:20:15.825Z+ set +x
3872023-12-30T06:20:15.825ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3882023-12-30T06:20:15.825Z+ for ZPOOL in "${ZPOOLS[@]}"
3892023-12-30T06:20:15.825Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3902023-12-30T06:20:15.825ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3912023-12-30T06:20:15.825Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3922023-12-30T06:20:15.825Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3932023-12-30T06:20:15.825Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3942023-12-30T06:20:15.825Z0+0 records in
3952023-12-30T06:20:15.825Z0+0 records out
3962023-12-30T06:20:15.825Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
3972023-12-30T06:20:15.825Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3982023-12-30T06:20:15.825Z+ set +x
3992023-12-30T06:20:15.825ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
4002023-12-30T06:20:15.825Z++ zpool list -o name
4012023-12-30T06:20:15.826Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4022023-12-30T06:20:15.826Z+ [[ -z '' ]]
4032023-12-30T06:20:15.826Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-12-30T06:20:15.853Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4052023-12-30T06:20:15.879Z+ set +x
4062023-12-30T06:20:15.879ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4072023-12-30T06:20:15.879Z+ for ZPOOL in "${ZPOOLS[@]}"
4082023-12-30T06:20:15.879Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4092023-12-30T06:20:15.879ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4102023-12-30T06:20:15.879Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4112023-12-30T06:20:15.879Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4122023-12-30T06:20:15.879Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4132023-12-30T06:20:15.879Z0+0 records in
4142023-12-30T06:20:15.879Z0+0 records out
4152023-12-30T06:20:15.879Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4162023-12-30T06:20:15.879Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4172023-12-30T06:20:15.879Z+ set +x
4182023-12-30T06:20:15.879ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4192023-12-30T06:20:15.879Z++ zpool list -o name
4202023-12-30T06:20:15.879Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4212023-12-30T06:20:15.879Z+ [[ -z '' ]]
4222023-12-30T06:20:15.879Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4232023-12-30T06:20:15.910Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4242023-12-30T06:20:15.910Z+ set +x
4252023-12-30T06:20:15.910ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4262023-12-30T06:20:15.910Z+ for ZPOOL in "${ZPOOLS[@]}"
4272023-12-30T06:20:15.910Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4282023-12-30T06:20:15.910ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4292023-12-30T06:20:15.910Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4302023-12-30T06:20:15.910Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4312023-12-30T06:20:15.910Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4322023-12-30T06:20:15.910Z0+0 records in
4332023-12-30T06:20:15.910Z0+0 records out
4342023-12-30T06:20:15.910Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4352023-12-30T06:20:15.910Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4362023-12-30T06:20:15.910Z+ set +x
4372023-12-30T06:20:15.910ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4382023-12-30T06:20:15.910Z++ zpool list -o name
4392023-12-30T06:20:15.910Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4402023-12-30T06:20:15.935Z+ [[ -z '' ]]
4412023-12-30T06:20:15.935Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4422023-12-30T06:20:15.959Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4432023-12-30T06:20:15.959Z+ set +x
4442023-12-30T06:20:15.959ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4452023-12-30T06:20:15.959Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4462023-12-30T06:20:15.959Z+ readarray -t ZPOOLS
4472023-12-30T06:20:15.959Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4482023-12-30T06:20:15.960Z++ sed 's/[ ",]//g'
4492023-12-30T06:20:15.960Z+ for ZPOOL in "${ZPOOLS[@]}"
4502023-12-30T06:20:15.960Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4512023-12-30T06:20:15.960ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4522023-12-30T06:20:15.960Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4532023-12-30T06:20:15.960Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4542023-12-30T06:20:15.960Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4552023-12-30T06:20:15.960Z0+0 records in
4562023-12-30T06:20:15.960Z0+0 records out
4572023-12-30T06:20:15.960Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4582023-12-30T06:20:15.960Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4592023-12-30T06:20:15.960Z+ set +x
4602023-12-30T06:20:15.960ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4612023-12-30T06:20:15.960Z++ zpool list -o name
4622023-12-30T06:20:15.960Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4632023-12-30T06:20:15.985Z+ [[ -z '' ]]
4642023-12-30T06:20:15.985Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4652023-12-30T06:20:16.009Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4662023-12-30T06:20:16.010Z+ set +x
4672023-12-30T06:20:16.010ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4682023-12-30T06:20:16.010Z+ for ZPOOL in "${ZPOOLS[@]}"
4692023-12-30T06:20:16.010Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4702023-12-30T06:20:16.010ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4712023-12-30T06:20:16.010Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4722023-12-30T06:20:16.010Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4732023-12-30T06:20:16.010Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4742023-12-30T06:20:16.010Z0+0 records in
4752023-12-30T06:20:16.010Z0+0 records out
4762023-12-30T06:20:16.010Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4772023-12-30T06:20:16.010Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4782023-12-30T06:20:16.010Z+ set +x
4792023-12-30T06:20:16.010ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4802023-12-30T06:20:16.010Z++ zpool list -o name
4812023-12-30T06:20:16.010Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4822023-12-30T06:20:16.010Z+ [[ -z '' ]]
4832023-12-30T06:20:16.010Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4842023-12-30T06:20:16.038Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4852023-12-30T06:20:16.038Z+ set +x
4862023-12-30T06:20:16.038ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4872023-12-30T06:20:16.070Z+ [[ zone == \z\o\n\e ]]
4882023-12-30T06:20:16.070Z+ ensure_simulated_links igb0
4892023-12-30T06:20:16.070Z+ local PHYSICAL_LINK=igb0
4902023-12-30T06:20:16.070Z+ INDICES=("0" "1")
4912023-12-30T06:20:16.070Z+ for I in "${INDICES[@]}"
4922023-12-30T06:20:16.070Z++ get_simnet_name_if_exists net0
4932023-12-30T06:20:16.070Z++ dladm show-simnet -p -o LINK net0
4942023-12-30T06:20:16.070Z++ echo ''
4952023-12-30T06:20:16.070Z+ [[ -z '' ]]
4962023-12-30T06:20:16.070Z+ dladm create-simnet -t net0
4972023-12-30T06:20:16.070Z+ dladm create-simnet -t sc0_0
4982023-12-30T06:20:16.070Z+ dladm modify-simnet -t -p net0 sc0_0
4992023-12-30T06:20:16.070Z+ dladm set-linkprop -p mtu=9000 sc0_0
5002023-12-30T06:20:16.070Z+ success 'Simnet net0/sc0_0 exists'
5012023-12-30T06:20:16.070Z+ set +x
5022023-12-30T06:20:16.070ZSimnet net0/sc0_0 exists
5032023-12-30T06:20:16.070Z+ for I in "${INDICES[@]}"
5042023-12-30T06:20:16.095Z++ get_simnet_name_if_exists net1
5052023-12-30T06:20:16.096Z++ dladm show-simnet -p -o LINK net1
5062023-12-30T06:20:16.096Z++ echo ''
5072023-12-30T06:20:16.097Z+ [[ -z '' ]]
5082023-12-30T06:20:16.098Z+ dladm create-simnet -t net1
5092023-12-30T06:20:16.098Z+ dladm create-simnet -t sc1_0
5102023-12-30T06:20:16.098Z+ dladm modify-simnet -t -p net1 sc1_0
5112023-12-30T06:20:16.098Z+ dladm set-linkprop -p mtu=9000 sc1_0
5122023-12-30T06:20:16.098Z+ success 'Simnet net1/sc1_0 exists'
5132023-12-30T06:20:16.098Z+ set +x
5142023-12-30T06:20:16.098ZSimnet net1/sc1_0 exists
5152023-12-30T06:20:16.098Z++ get_vnic_name_if_exists sc0_1
5162023-12-30T06:20:16.098Z++ dladm show-vnic -p -o LINK sc0_1
5172023-12-30T06:20:16.098Z++ echo ''
5182023-12-30T06:20:16.098Z+ [[ -z '' ]]
5192023-12-30T06:20:16.098Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5202023-12-30T06:20:16.098Z+ [[ -v PROMISC_FILT_OFF ]]
5212023-12-30T06:20:16.098Z+ success 'Vnic sc0_1 exists'
5222023-12-30T06:20:16.098Z+ set +x
5232023-12-30T06:20:16.098ZVnic sc0_1 exists
5242023-12-30T06:20:16.098Z+ warn_if_no_proxy_arp
5252023-12-30T06:20:16.098Z+ [[ -v PXA_START ]]
5262023-12-30T06:20:16.098Z+ [[ -v PXA_END ]]
5272023-12-30T06:20:16.099Z+ ensure_softnpu_zone
5282023-12-30T06:20:16.123Z+ zoneadm list
5292023-12-30T06:20:16.124Z+ grep -q sidecar_softnpu
5302023-12-30T06:20:16.124Z+ [[ -f out/npuzone/npuzone ]]
5312023-12-30T06:20:16.124Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit e3ea4b495ba0a71801ded0776ae4bbd31df57e26 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5322023-12-30T06:20:16.124Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5332023-12-30T06:20:22.249Zdownloading softnpu ...done
5342023-12-30T06:20:22.273Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/libsidecar_lite.so.sha256.txt
5352023-12-30T06:20:27.101Zdownloading asic_program.so ...done
5362023-12-30T06:20:27.125Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/scadm.sha256.txt
5372023-12-30T06:20:33.577Zdownloading scadm ...done
5382023-12-30T06:20:33.602Zcreating port tfportrear0_0 for sc0_0
5392023-12-30T06:20:33.602Zcreating port tfportqsfp0_0 for sc0_1
5402023-12-30T06:20:33.602Zsoftnpu zone setup
5412023-12-30T06:20:33.602Zconfigure zone
5422023-12-30T06:20:33.626Zinstall zone
5432023-12-30T06:20:34.675Zboot zone
5442023-12-30T06:20:37.779Z[sidecar_softnpu] svcs milestone/network
5452023-12-30T06:20:37.806ZSTATE STIME FMRI
5462023-12-30T06:20:37.806Zonline 6:20:36 svc:/milestone/network:default
5472023-12-30T06:20:37.806Z[sidecar_softnpu] /softnpu/npu start
5482023-12-30T06:20:37.832Z
5492023-12-30T06:20:37.856Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5502023-12-30T06:20:37.856Z+ GATEWAY_IP=10.151.2.100
5512023-12-30T06:20:37.857Z+ echo 'Using 10.151.2.100 as gateway ip'
5522023-12-30T06:20:37.857ZUsing 10.151.2.100 as gateway ip
5532023-12-30T06:20:37.857Z+ [[ ! -v GATEWAY_MAC ]]
5542023-12-30T06:20:37.857Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5552023-12-30T06:20:37.857Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5562023-12-30T06:20:37.857Z+++ echo 18:c0:4d:0d:a0:2a
5572023-12-30T06:20:37.857Z+ COUNT=1
5582023-12-30T06:20:37.857Z+ [[ 1 -ne 1 ]]
5592023-12-30T06:20:37.857Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5602023-12-30T06:20:37.857ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5612023-12-30T06:20:37.857Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5622023-12-30T06:20:37.857Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5632023-12-30T06:20:37.882Z+ PXA_MAC=a8:e1:de:01:70:1d
5642023-12-30T06:20:37.882Z+ [[ -v PXA_START ]]
5652023-12-30T06:20:37.882Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5662023-12-30T06:20:37.882Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5672023-12-30T06:20:37.909Z+ z_scadm dump-state
5682023-12-30T06:20:37.910Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5692023-12-30T06:20:37.935Zlocal v6:
5702023-12-30T06:20:37.935Zlocal v4:
5712023-12-30T06:20:37.935Zrouter v6:
5722023-12-30T06:20:37.935Zrouter v4_idx:
5732023-12-30T06:20:37.935Zrouter v4_routes:
5742023-12-30T06:20:37.935Zresolver v4:
5752023-12-30T06:20:37.935Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5762023-12-30T06:20:37.935Zresolver v6:
5772023-12-30T06:20:37.935Znat v4:
5782023-12-30T06:20:37.935Znat_v6:
5792023-12-30T06:20:37.936Zport_mac:
5802023-12-30T06:20:37.936Zproxy arp:
5812023-12-30T06:20:37.936Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5822023-12-30T06:20:37.936Z+ success 'softnpu zone exists'
5832023-12-30T06:20:37.936Z+ set +x
5842023-12-30T06:20:37.936Zsoftnpu zone exists
5852023-12-30T06:20:37.936Z
5862023-12-30T06:20:37.936Zreal 22.497697034
5872023-12-30T06:20:37.936Zuser 0.706109173
5882023-12-30T06:20:37.936Zsys 1.599096560
5892023-12-30T06:20:37.936Ztrap 0.000077265
5902023-12-30T06:20:37.936Ztflt 0.000152894
5912023-12-30T06:20:37.936Zdflt 0.000854676
5922023-12-30T06:20:37.936Zkflt 0.000000000
5932023-12-30T06:20:37.937Zlock 17:20.775327036
5942023-12-30T06:20:37.937Zslp 1:07.452159192
5952023-12-30T06:20:37.938Zlat 0.081226387
5962023-12-30T06:20:37.938Zstop 0.340675113
5972023-12-30T06:20:37.938Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5982023-12-30T06:20:37.967Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5992023-12-30T06:20:37.993Z+ SILO_NAME=recovery
6002023-12-30T06:20:37.993Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
6012023-12-30T06:20:37.993Z+ EXTERNAL_DNS_DOMAIN=oxide.test
6022023-12-30T06:20:37.993Z+ sed -i~ '
6032023-12-30T06:20:37.993Z /^external_dns_ips/c\
6042023-12-30T06:20:37.993Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6052023-12-30T06:20:37.993Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6062023-12-30T06:20:37.993Z /^first/c\
6072023-12-30T06:20:37.993Zfirst = "10.151.2.164"
6082023-12-30T06:20:37.993Z /^last/c\
6092023-12-30T06:20:37.993Zlast = "10.151.2.173"
6102023-12-30T06:20:37.993Z }
6112023-12-30T06:20:37.993Z /^\[rack_network_config/,/^$/ {
6122023-12-30T06:20:37.993Z /^infra_ip_first/c\
6132023-12-30T06:20:37.993Zinfra_ip_first = "10.151.2.174"
6142023-12-30T06:20:37.993Z /^infra_ip_last/c\
6152023-12-30T06:20:37.993Zinfra_ip_last = "10.151.2.174"
6162023-12-30T06:20:37.993Z }
6172023-12-30T06:20:37.993Z /^\[\[rack_network_config.ports/,/^$/ {
6182023-12-30T06:20:37.993Z /^routes/c\
6192023-12-30T06:20:37.993Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6202023-12-30T06:20:37.994Z /^addresses/c\
6212023-12-30T06:20:37.994Zaddresses = \["10.151.2.174/32"\]
6222023-12-30T06:20:37.994Z }
6232023-12-30T06:20:37.994Z' pkg/config-rss.toml
6242023-12-30T06:20:37.994Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6252023-12-30T06:20:37.994Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6262023-12-30T06:20:37.994Z+++ pkg/config-rss.toml Sat Dec 30 06:20:38 2023
6272023-12-30T06:20:37.994Z@@ -41,7 +41,7 @@
6282023-12-30T06:20:37.994Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6292023-12-30T06:20:37.994Z # must be contained in one of the "internal services" IP Pool ranges listed
6302023-12-30T06:20:37.994Z # below.
6312023-12-30T06:20:37.994Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6322023-12-30T06:20:37.994Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6332023-12-30T06:20:37.994Z
6342023-12-30T06:20:37.994Z # Initial TLS certificates for the external API
6352023-12-30T06:20:37.994Z #
6362023-12-30T06:20:37.994Z@@ -77,8 +77,8 @@
6372023-12-30T06:20:37.994Z #
6382023-12-30T06:20:37.994Z # For more on this and what to put here, see docs/how-to-run.adoc.
6392023-12-30T06:20:37.994Z [[internal_services_ip_pool_ranges]]
6402023-12-30T06:20:37.994Z-first = "192.168.1.20"
6412023-12-30T06:20:37.994Z-last = "192.168.1.29"
6422023-12-30T06:20:37.994Z+first = "10.151.2.164"
6432023-12-30T06:20:37.995Z+last = "10.151.2.173"
6442023-12-30T06:20:37.995Z
6452023-12-30T06:20:37.995Z # TODO - this configuration is subject to change going forward. Ultimately these
6462023-12-30T06:20:37.995Z # parameters should be provided to the control plane via wicket, but we need to
6472023-12-30T06:20:37.995Z@@ -92,8 +92,8 @@
6482023-12-30T06:20:37.995Z # A range of IP addresses used by Boundary Services on the external network. In
6492023-12-30T06:20:37.995Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6502023-12-30T06:20:37.995Z # With softnpu, only one address is used.
6512023-12-30T06:20:37.995Z-infra_ip_first = "192.168.1.30"
6522023-12-30T06:20:37.995Z-infra_ip_last = "192.168.1.30"
6532023-12-30T06:20:37.995Z+infra_ip_first = "10.151.2.174"
6542023-12-30T06:20:37.996Z+infra_ip_last = "10.151.2.174"
6552023-12-30T06:20:37.996Z
6562023-12-30T06:20:37.996Z # Configurations for BGP routers to run on the scrimlets.
6572023-12-30T06:20:37.996Z bgp = []
6582023-12-30T06:20:37.996Z@@ -101,9 +101,9 @@
6592023-12-30T06:20:37.996Z # You can configure multiple uplinks by repeating the following stanza
6602023-12-30T06:20:37.996Z [[rack_network_config.ports]]
6612023-12-30T06:20:37.996Z # Routes associated with this port.
6622023-12-30T06:20:37.996Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6632023-12-30T06:20:37.996Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6642023-12-30T06:20:37.996Z # Addresses associated with this port.
6652023-12-30T06:20:37.996Z-addresses = ["192.168.1.30/32"]
6662023-12-30T06:20:37.996Z+addresses = ["10.151.2.174/32"]
6672023-12-30T06:20:37.996Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6682023-12-30T06:20:37.996Z port = "qsfp0"
6692023-12-30T06:20:37.996Z # The speed of this port.
6702023-12-30T06:20:37.997Z+ true
6712023-12-30T06:20:37.997Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6722023-12-30T06:20:37.997Za pkg/config-rss.toml 7K
6732023-12-30T06:20:37.997Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6742023-12-30T06:20:37.997Z+ TLS_NAME=recovery.sys.oxide.test
6752023-12-30T06:20:37.997Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6762023-12-30T06:20:38.368Z...+.............+..+....+........+......+....+..+.........+......+.+.....+....+...+..+.............+......+............+..+.+..+.......+..+..........+......+............+...+..+......+.+.....+.+..+...+...+.........+.......+..................+..+...+.+......+.....+....+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+.........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...+............+....................+..................+.+.....+....+...+..+...+...+......+.+............+...............+........+....+.........+.....+......+...............+.+..............+.+..............+....+..+..........+.....+.........+......+....+.........+...........+...+.............+...+..+...+.+...........+....+...+...........+.......+..+..........+...........+..........+.....+......+...+.+.....................+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6772023-12-30T06:20:39.617Z....+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.+...+.................+.........+......................+........+.......+...+............+..+...+...................+...+...........+.+.....+.+.................+.+..+...+......+......+....+.........+.........+...+.....+.........+...+...........................+..........+........+.+......+...+.....+...+...+....+..+......+...+....+...+..+..........+...........+..........+.........+......+...........+....+..+............+..........+........+...+.....................+.........+.+.........+.....+......+...+.+......+........+..........+.....+.......+......+...+.....+.......+.....+....+........+......+....+......+.....+......+.........+......+...+..........+.....+......+.+....................+.+...+......+...+........+...+.+......+..+.+.........+.....+..................+..........+..+............+.+........+....+...+.........+........+.............+..+.+..+....+.........+......+......+...+...........+......+.......+...............+.....+..........+..+....+..+...+....+...+........+......+.+...+......+..............+.+......+.....+.......+..+....+.....+....+......+..............+.............+.........+......+...+......+.....+...+.+...........+.........+..........+.........+.....+.+..............+...............+............+....+...........+.........+.+...+..........................+.......+..+.+................................+...+......+.+..................+.....+............+....+..................+...+..+...+.......+..........................+.+...+............+..............+....+...+.....+....+......+.............................+....+....................+................+........+......+......+......+...+.+...........+...+......+.+.........+........+......+.+...........+.......+......+............+..+.+..+...+.........................+..+.........+...+.+......+...+...........+................+...+...........+......+...+...+..........+...............+.....+...+.......+.........+...+..+................+............+...+...+......+...............+...+...+.....+.......+........+...+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6782023-12-30T06:20:39.642Z-----
6792023-12-30T06:20:39.642Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6802023-12-30T06:20:39.667Za pkg/initial-tls-cert.pem 2K
6812023-12-30T06:20:39.667Za pkg/initial-tls-key.pem 4K
6822023-12-30T06:20:39.667Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6832023-12-30T06:20:39.667Z+ rmdir pkg
6842023-12-30T06:20:39.667Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6852023-12-30T06:20:39.667Z+ pfexec mkdir -p /usr/oxide
6862023-12-30T06:20:39.667Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6872023-12-30T06:20:40.461Z+ pfexec chmod +x /usr/oxide/oxide
6882023-12-30T06:20:40.487Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6892023-12-30T06:20:58.748Z+ OMICRON_NO_UNINSTALL=1
6902023-12-30T06:20:58.748Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6912023-12-30T06:20:58.773ZDec 30 06:20:59.423 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6922023-12-30T06:20:58.773ZDec 30 06:20:59.425 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6932023-12-30T06:20:58.773ZDec 30 06:20:59.425 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6942023-12-30T06:20:58.774ZDec 30 06:20:59.425 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6952023-12-30T06:20:58.774ZDec 30 06:20:59.425 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6962023-12-30T06:20:58.774ZDec 30 06:20:59.425 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6972023-12-30T06:20:58.774ZDec 30 06:20:59.426 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6982023-12-30T06:20:58.774ZDec 30 06:20:59.426 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6992023-12-30T06:20:58.774ZDec 30 06:20:59.426 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7002023-12-30T06:20:58.775ZDec 30 06:20:59.426 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7012023-12-30T06:20:58.775ZDec 30 06:20:59.426 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7022023-12-30T06:20:58.775ZDec 30 06:20:59.426 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
7032023-12-30T06:20:58.775ZDec 30 06:20:59.426 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7042023-12-30T06:20:58.775ZDec 30 06:20:59.426 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7052023-12-30T06:20:58.775ZDec 30 06:20:59.426 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7062023-12-30T06:20:58.775ZDec 30 06:20:59.426 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7072023-12-30T06:20:58.775ZDec 30 06:20:59.426 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7082023-12-30T06:20:59.013ZDec 30 06:20:59.674 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7092023-12-30T06:20:59.050ZDec 30 06:20:59.711 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7102023-12-30T06:20:59.113ZDec 30 06:20:59.774 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7112023-12-30T06:20:59.242Z
7122023-12-30T06:20:59.267Zreal 0.489160748
7132023-12-30T06:20:59.267Zuser 0.099926687
7142023-12-30T06:20:59.267Zsys 1.994036054
7152023-12-30T06:20:59.267Ztrap 0.000001009
7162023-12-30T06:20:59.267Ztflt 0.000036378
7172023-12-30T06:20:59.267Zdflt 0.000933753
7182023-12-30T06:20:59.267Zkflt 0.000005060
7192023-12-30T06:20:59.267Zlock 43.931581525
7202023-12-30T06:20:59.267Zslp 0.702891812
7212023-12-30T06:20:59.267Zlat 0.003206009
7222023-12-30T06:20:59.267Zstop 0.203600561
7232023-12-30T06:20:59.267Z+ retry=0
7242023-12-30T06:20:59.267Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7252023-12-30T06:20:59.267Z+ [[ 0 -gt 30 ]]
7262023-12-30T06:20:59.267Z+ sleep 1
7272023-12-30T06:21:00.257Z+ retry=1
7282023-12-30T06:21:00.257Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7292023-12-30T06:21:00.282Z+ [[ 1 -gt 30 ]]
7302023-12-30T06:21:00.282Z+ sleep 1
7312023-12-30T06:21:01.273Z+ retry=2
7322023-12-30T06:21:01.273Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7332023-12-30T06:21:01.298Z+ [[ 2 -gt 30 ]]
7342023-12-30T06:21:01.298Z+ sleep 1
7352023-12-30T06:21:02.287Z+ retry=3
7362023-12-30T06:21:02.287Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7372023-12-30T06:21:02.313Z+ [[ 3 -gt 30 ]]
7382023-12-30T06:21:02.313Z+ sleep 1
7392023-12-30T06:21:03.301Z+ retry=4
7402023-12-30T06:21:03.301Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7412023-12-30T06:21:28.471Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7422023-12-30T06:21:28.498Zlocal v6:
7432023-12-30T06:21:28.498Zfe80::7c20:e4ff:fe79:731a
7442023-12-30T06:21:28.499Zfd00:99::1
7452023-12-30T06:21:28.499Zlocal v4:
7462023-12-30T06:21:28.499Z10.151.2.174
7472023-12-30T06:21:28.499Zrouter v6:
7482023-12-30T06:21:28.499Zfd00:1122:3344:101::/64 -> fe80::102b:14ff:fe4e:22b7 (1)
7492023-12-30T06:21:28.499Zfd00:1122:3344:3::/64 -> fe80::102b:14ff:fe4e:22b7 (1)
7502023-12-30T06:21:28.499Zfdb0:18c0:4d0d:a02a::/64 -> fe80::102b:14ff:fe4e:22b7 (1)
7512023-12-30T06:21:28.499Zfd00:1122:3344:1::/64 -> fe80::102b:14ff:fe4e:22b7 (1)
7522023-12-30T06:21:28.499Zfd00:1122:3344:2::/64 -> fe80::102b:14ff:fe4e:22b7 (1)
7532023-12-30T06:21:28.499Zrouter v4_idx:
7542023-12-30T06:21:28.499Z0.0.0.0/0 -> 0
7552023-12-30T06:21:28.499Zrouter v4_routes:
7562023-12-30T06:21:28.499Z0 -> 10.151.2.100 (2)
7572023-12-30T06:21:28.499Zresolver v4:
7582023-12-30T06:21:28.499Z10.151.2.174 -> 6e:eb:e6:9b:c5:97
7592023-12-30T06:21:28.499Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7602023-12-30T06:21:28.499Zresolver v6:
7612023-12-30T06:21:28.499Zfe80::102b:14ff:fe4e:22b7 -> 12:2b:14:4e:22:b7
7622023-12-30T06:21:28.499Zfe80::7c20:e4ff:fe79:731a -> 7e:20:e4:79:73:1a
7632023-12-30T06:21:28.499Znat v4:
7642023-12-30T06:21:28.524Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:ae:14
7652023-12-30T06:21:28.524Znat_v6:
7662023-12-30T06:21:28.524Zport_mac:
7672023-12-30T06:21:28.524Z1: a8:40:25:42:38:43
7682023-12-30T06:21:28.524Z1000: a8:40:25:42:38:42
7692023-12-30T06:21:28.524Zproxy arp:
7702023-12-30T06:21:28.524Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7712023-12-30T06:21:28.524Z+ retry=0
7722023-12-30T06:21:28.524Z+++ zoneadm list -n
7732023-12-30T06:21:28.524Z+++ grep oxz_ntp
7742023-12-30T06:21:28.524Z++ pfexec svcs -z oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb -Hostate oxide/ntp
7752023-12-30T06:21:28.549Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7762023-12-30T06:21:28.549Z++ true
7772023-12-30T06:21:28.549Z+ [[ '' != online ]]
7782023-12-30T06:21:28.549Z+ [[ 0 -gt 60 ]]
7792023-12-30T06:21:28.549Z+ sleep 1
7802023-12-30T06:21:29.532Z+ retry=1
7812023-12-30T06:21:29.557Z+++ zoneadm list -n
7822023-12-30T06:21:29.557Z+++ grep oxz_ntp
7832023-12-30T06:21:29.557Z++ pfexec svcs -z oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb -Hostate oxide/ntp
7842023-12-30T06:21:29.581Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7852023-12-30T06:21:29.581Z++ true
7862023-12-30T06:21:29.581Z+ [[ '' != online ]]
7872023-12-30T06:21:29.581Z+ [[ 1 -gt 60 ]]
7882023-12-30T06:21:29.581Z+ sleep 1
7892023-12-30T06:21:30.562Z+ retry=2
7902023-12-30T06:21:30.587Z+++ zoneadm list -n
7912023-12-30T06:21:30.587Z+++ grep oxz_ntp
7922023-12-30T06:21:30.587Z++ pfexec svcs -z oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb -Hostate oxide/ntp
7932023-12-30T06:21:30.587Z+ [[ disabled != online ]]
7942023-12-30T06:21:30.587Z+ [[ 2 -gt 60 ]]
7952023-12-30T06:21:30.587Z+ sleep 1
7962023-12-30T06:21:31.586Z+ retry=3
7972023-12-30T06:21:31.611Z+++ zoneadm list -n
7982023-12-30T06:21:31.611Z+++ grep oxz_ntp
7992023-12-30T06:21:31.611Z++ pfexec svcs -z oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb -Hostate oxide/ntp
8002023-12-30T06:21:31.611Z+ [[ disabled != online ]]
8012023-12-30T06:21:31.611Z+ [[ 3 -gt 60 ]]
8022023-12-30T06:21:31.611Z+ sleep 1
8032023-12-30T06:21:32.609Z+ retry=4
8042023-12-30T06:21:32.633Z+++ zoneadm list -n
8052023-12-30T06:21:32.633Z+++ grep oxz_ntp
8062023-12-30T06:21:32.633Z++ pfexec svcs -z oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb -Hostate oxide/ntp
8072023-12-30T06:21:32.633Z+ [[ disabled != online ]]
8082023-12-30T06:21:32.633Z+ [[ 4 -gt 60 ]]
8092023-12-30T06:21:32.633Z+ sleep 1
8102023-12-30T06:21:33.630Z+ retry=5
8112023-12-30T06:21:33.655Z+++ zoneadm list -n
8122023-12-30T06:21:33.655Z+++ grep oxz_ntp
8132023-12-30T06:21:33.655Z++ pfexec svcs -z oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb -Hostate oxide/ntp
8142023-12-30T06:21:33.655Z+ [[ disabled != online ]]
8152023-12-30T06:21:33.655Z+ [[ 5 -gt 60 ]]
8162023-12-30T06:21:33.655Z+ sleep 1
8172023-12-30T06:21:34.651Z+ retry=6
8182023-12-30T06:21:34.675Z+++ zoneadm list -n
8192023-12-30T06:21:34.675Z+++ grep oxz_ntp
8202023-12-30T06:21:34.676Z++ pfexec svcs -z oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb -Hostate oxide/ntp
8212023-12-30T06:21:34.676Z+ [[ disabled != online ]]
8222023-12-30T06:21:34.676Z+ [[ 6 -gt 60 ]]
8232023-12-30T06:21:34.676Z+ sleep 1
8242023-12-30T06:21:35.673Z+ retry=7
8252023-12-30T06:21:35.698Z+++ zoneadm list -n
8262023-12-30T06:21:35.698Z+++ grep oxz_ntp
8272023-12-30T06:21:35.698Z++ pfexec svcs -z oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb -Hostate oxide/ntp
8282023-12-30T06:21:35.698Z+ [[ disabled != online ]]
8292023-12-30T06:21:35.698Z+ [[ 7 -gt 60 ]]
8302023-12-30T06:21:35.698Z+ sleep 1
8312023-12-30T06:21:36.695Z+ retry=8
8322023-12-30T06:21:36.720Z+++ zoneadm list -n
8332023-12-30T06:21:36.720Z+++ grep oxz_ntp
8342023-12-30T06:21:36.720Z++ pfexec svcs -z oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb -Hostate oxide/ntp
8352023-12-30T06:21:36.720Z+ [[ disabled != online ]]
8362023-12-30T06:21:36.720Z+ [[ 8 -gt 60 ]]
8372023-12-30T06:21:36.720Z+ sleep 1
8382023-12-30T06:21:37.742Z+ retry=9
8392023-12-30T06:21:37.742Z+++ zoneadm list -n
8402023-12-30T06:21:37.742Z+++ grep oxz_ntp
8412023-12-30T06:21:37.742Z++ pfexec svcs -z oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb -Hostate oxide/ntp
8422023-12-30T06:21:37.742Z+ [[ disabled != online ]]
8432023-12-30T06:21:37.742Z+ [[ 9 -gt 60 ]]
8442023-12-30T06:21:37.742Z+ sleep 1
8452023-12-30T06:21:38.739Z+ retry=10
8462023-12-30T06:21:38.764Z+++ zoneadm list -n
8472023-12-30T06:21:38.764Z+++ grep oxz_ntp
8482023-12-30T06:21:38.764Z++ pfexec svcs -z oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb -Hostate oxide/ntp
8492023-12-30T06:21:38.764Z+ [[ online != online ]]
8502023-12-30T06:21:38.764Z+ echo 'Waited for chrony: 10s'
8512023-12-30T06:21:38.764Z+ export RUST_BACKTRACE=1
8522023-12-30T06:21:38.764ZWaited for chrony: 10s
8532023-12-30T06:21:38.764Z+ RUST_BACKTRACE=1
8542023-12-30T06:21:38.764Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8552023-12-30T06:21:38.764Z++ ./tests/bootstrap
8562023-12-30T06:21:38.789ZUsing external DNS server at 10.151.2.164:53
8572023-12-30T06:21:38.813Z2023-12-30 06:21:39.450944425 UTC: attempting to log into API
8582023-12-30T06:21:53.828Z2023-12-30 06:21:53.345401125 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8592023-12-30T06:21:54.828Z2023-12-30 06:21:54.346312151 UTC: attempting to log into API
 
8602023-12-30T06:22:02.315ZError: No response to keepalive - Terminating session
 
8612023-12-30T06:22:09.862Z2023-12-30 06:22:09.379628125 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8622023-12-30T06:22:10.862Z2023-12-30 06:22:10.380441609 UTC: attempting to log into API
 
8632023-12-30T06:22:12.444ZError: No response de-activating SOL payload
8642023-12-30T06:22:13.640Z[SOL Session operational. Use ~? for help]
 
8652023-12-30T06:22:25.895Z2023-12-30 06:22:25.413543507 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8662023-12-30T06:22:26.896Z2023-12-30 06:22:26.414595996 UTC: attempting to log into API
8672023-12-30T06:22:41.959Z2023-12-30 06:22:41.477714927 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8682023-12-30T06:22:42.961Z2023-12-30 06:22:42.479029283 UTC: attempting to log into API
 
8692023-12-30T06:22:53.279ZDec 30 06:22:52 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8702023-12-30T06:22:58.002Z2023-12-30 06:22:57.540875041 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8712023-12-30T06:22:59.006Z2023-12-30 06:22:58.544222381 UTC: attempting to log into API
 
8722023-12-30T06:22:59.934ZDec 30 06:22:59 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
8732023-12-30T06:23:06.579ZDec 30 06:23:05 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
8742023-12-30T06:23:13.548ZDec 30 06:23:12 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
8752023-12-30T06:23:14.036Z2023-12-30 06:23:13.618089046 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8762023-12-30T06:23:15.036Z2023-12-30 06:23:14.621505443 UTC: attempting to log into API
 
8772023-12-30T06:23:20.196ZDec 30 06:23:19 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
8782023-12-30T06:23:30.070Z2023-12-30 06:23:29.695398126 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8792023-12-30T06:23:31.071Z2023-12-30 06:23:30.699423899 UTC: attempting to log into API
8802023-12-30T06:23:46.106Z2023-12-30 06:23:45.775765296 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8812023-12-30T06:23:47.107Z2023-12-30 06:23:46.779229236 UTC: attempting to log into API
8822023-12-30T06:23:52.879Z2023-12-30 06:23:52.566233340 UTC: login succeeded
8832023-12-30T06:23:52.911Zwaiting for nexus to come up...
8842023-12-30T06:23:52.978Zcreating IP pool... 10.151.2.175 - 10.151.2.227
8852023-12-30T06:23:53.029Zsetting up quotas...
8862023-12-30T06:23:53.081Zensuring datasets are ready...
8872023-12-30T06:23:55.578Zlet's roll.
8882023-12-30T06:23:55.603Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
8892023-12-30T06:23:55.603ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
8902023-12-30T06:23:55.603ZOXIDE_TOKEN=oxide-token-bcc506cae464ce2bbb9bb83333750e6f3e909920'
8912023-12-30T06:23:55.603Z++ OXIDE_HOST=https://recovery.sys.oxide.test
8922023-12-30T06:23:55.603Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
8932023-12-30T06:23:55.603Z++ OXIDE_TOKEN=oxide-token-bcc506cae464ce2bbb9bb83333750e6f3e909920
8942023-12-30T06:23:55.603Z+ export OXIDE_HOST OXIDE_TOKEN
8952023-12-30T06:23:55.603Z+ retry=0
8962023-12-30T06:23:55.603Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8972023-12-30T06:23:55.627Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
8982023-12-30T06:23:55.628Z+ [[ 0 -gt 60 ]]
8992023-12-30T06:23:55.628Z+ sleep 1
9002023-12-30T06:23:56.608Z+ retry=1
9012023-12-30T06:23:56.608Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9022023-12-30T06:23:56.633Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9032023-12-30T06:23:56.633Z+ [[ 1 -gt 60 ]]
9042023-12-30T06:23:56.633Z+ sleep 1
9052023-12-30T06:23:57.624Z+ retry=2
9062023-12-30T06:23:57.624Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9072023-12-30T06:23:57.648Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9082023-12-30T06:23:57.648Z+ [[ 2 -gt 60 ]]
9092023-12-30T06:23:57.648Z+ sleep 1
9102023-12-30T06:23:58.640Z+ retry=3
9112023-12-30T06:23:58.640Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9122023-12-30T06:23:58.664Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9132023-12-30T06:23:58.664Z+ [[ 3 -gt 60 ]]
9142023-12-30T06:23:58.664Z+ sleep 1
9152023-12-30T06:23:59.655Z+ retry=4
9162023-12-30T06:23:59.655Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9172023-12-30T06:23:59.679Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9182023-12-30T06:23:59.679Z+ [[ 4 -gt 60 ]]
9192023-12-30T06:23:59.679Z+ sleep 1
9202023-12-30T06:24:00.670Z+ retry=5
9212023-12-30T06:24:00.670Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9222023-12-30T06:24:00.700Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9232023-12-30T06:24:00.700Z+ [[ 5 -gt 60 ]]
9242023-12-30T06:24:00.700Z+ sleep 1
9252023-12-30T06:24:01.686Z+ retry=6
9262023-12-30T06:24:01.686Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9272023-12-30T06:24:01.712Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9282023-12-30T06:24:01.712Z+ [[ 6 -gt 60 ]]
9292023-12-30T06:24:01.712Z+ sleep 1
9302023-12-30T06:24:02.707Z+ retry=7
9312023-12-30T06:24:02.707Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9322023-12-30T06:24:02.759Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9332023-12-30T06:24:02.760Z+ [[ 7 -gt 60 ]]
9342023-12-30T06:24:02.760Z+ sleep 1
9352023-12-30T06:24:03.719Z+ retry=8
9362023-12-30T06:24:03.719Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9372023-12-30T06:24:03.745Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9382023-12-30T06:24:03.745Z+ [[ 8 -gt 60 ]]
9392023-12-30T06:24:03.745Z+ sleep 1
9402023-12-30T06:24:04.734Z+ retry=9
9412023-12-30T06:24:04.734Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9422023-12-30T06:24:04.759Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9432023-12-30T06:24:04.759Z+ [[ 9 -gt 60 ]]
9442023-12-30T06:24:04.759Z+ sleep 1
9452023-12-30T06:24:05.748Z+ retry=10
9462023-12-30T06:24:05.748Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9472023-12-30T06:24:05.772Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9482023-12-30T06:24:05.773Z+ [[ 10 -gt 60 ]]
9492023-12-30T06:24:05.773Z+ sleep 1
9502023-12-30T06:24:06.778Z+ retry=11
9512023-12-30T06:24:06.778Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9522023-12-30T06:24:06.803Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9532023-12-30T06:24:06.803Z+ [[ 11 -gt 60 ]]
9542023-12-30T06:24:06.803Z+ sleep 1
9552023-12-30T06:24:07.810Z+ retry=12
9562023-12-30T06:24:07.834Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9572023-12-30T06:24:07.834Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9582023-12-30T06:24:07.834Z+ [[ 12 -gt 60 ]]
9592023-12-30T06:24:07.834Z+ sleep 1
9602023-12-30T06:24:08.795Z+ retry=13
9612023-12-30T06:24:08.795Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9622023-12-30T06:24:08.842Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9632023-12-30T06:24:08.842Z+ [[ 13 -gt 60 ]]
9642023-12-30T06:24:08.842Z+ sleep 1
9652023-12-30T06:24:09.811Z+ retry=14
9662023-12-30T06:24:09.811Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9672023-12-30T06:24:09.835Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9682023-12-30T06:24:09.835Z+ [[ 14 -gt 60 ]]
9692023-12-30T06:24:09.836Z+ sleep 1
9702023-12-30T06:24:10.826Z+ retry=15
9712023-12-30T06:24:10.826Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9722023-12-30T06:24:10.851Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9732023-12-30T06:24:10.851Z+ [[ 15 -gt 60 ]]
9742023-12-30T06:24:10.851Z+ sleep 1
9752023-12-30T06:24:11.841Z+ retry=16
9762023-12-30T06:24:11.841Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9772023-12-30T06:24:11.866Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9782023-12-30T06:24:11.866Z+ [[ 16 -gt 60 ]]
9792023-12-30T06:24:11.866Z+ sleep 1
9802023-12-30T06:24:12.856Z+ retry=17
9812023-12-30T06:24:12.856Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9822023-12-30T06:24:12.881Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9832023-12-30T06:24:12.881Z+ [[ 17 -gt 60 ]]
9842023-12-30T06:24:12.881Z+ sleep 1
9852023-12-30T06:24:13.871Z+ retry=18
9862023-12-30T06:24:13.871Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9872023-12-30T06:24:13.896Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9882023-12-30T06:24:13.896Z+ [[ 18 -gt 60 ]]
9892023-12-30T06:24:13.896Z+ sleep 1
9902023-12-30T06:24:14.887Z+ retry=19
9912023-12-30T06:24:14.887Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9922023-12-30T06:24:14.912Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9932023-12-30T06:24:14.912Z+ [[ 19 -gt 60 ]]
9942023-12-30T06:24:14.912Z+ sleep 1
9952023-12-30T06:24:15.902Z+ retry=20
9962023-12-30T06:24:15.902Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9972023-12-30T06:24:15.927Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9982023-12-30T06:24:15.927Z+ [[ 20 -gt 60 ]]
9992023-12-30T06:24:15.927Z+ sleep 1
10002023-12-30T06:24:16.918Z+ retry=21
10012023-12-30T06:24:16.918Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10022023-12-30T06:24:16.942Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10032023-12-30T06:24:16.942Z+ [[ 21 -gt 60 ]]
10042023-12-30T06:24:16.942Z+ sleep 1
10052023-12-30T06:24:17.933Z+ retry=22
10062023-12-30T06:24:17.933Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10072023-12-30T06:24:17.958Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10082023-12-30T06:24:17.958Z+ [[ 22 -gt 60 ]]
10092023-12-30T06:24:17.958Z+ sleep 1
10102023-12-30T06:24:18.947Z+ retry=23
10112023-12-30T06:24:18.948Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10122023-12-30T06:24:18.972Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10132023-12-30T06:24:18.972Z+ [[ 23 -gt 60 ]]
10142023-12-30T06:24:18.972Z+ sleep 1
10152023-12-30T06:24:19.963Z+ retry=24
10162023-12-30T06:24:19.963Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10172023-12-30T06:24:19.987Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10182023-12-30T06:24:19.987Z+ [[ 24 -gt 60 ]]
10192023-12-30T06:24:19.987Z+ sleep 1
10202023-12-30T06:24:20.978Z+ retry=25
10212023-12-30T06:24:20.979Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10222023-12-30T06:24:21.004Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10232023-12-30T06:24:21.004Z+ [[ 25 -gt 60 ]]
10242023-12-30T06:24:21.004Z+ sleep 1
10252023-12-30T06:24:21.995Z+ retry=26
10262023-12-30T06:24:21.995Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10272023-12-30T06:24:22.080Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
10282023-12-30T06:24:22.799Z{"status":"ok"}success
10292023-12-30T06:24:22.799ZProject {
10302023-12-30T06:24:22.827Z description: "some images",
10312023-12-30T06:24:22.827Z id: 276d2b75-424f-4da1-8257-045d177d637c,
10322023-12-30T06:24:22.827Z name: Name(
10332023-12-30T06:24:22.827Z "images",
10342023-12-30T06:24:22.827Z ),
10352023-12-30T06:24:22.827Z time_created: 2023-12-30T06:24:21.983441Z,
10362023-12-30T06:24:22.827Z time_modified: 2023-12-30T06:24:21.983441Z,
10372023-12-30T06:24:22.827Z}
10382023-12-30T06:24:22.827Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
 
10392023-12-30T06:25:18.553ZError: No response to keepalive - Terminating session
10402023-12-30T06:25:28.690ZError: No response de-activating SOL payload
10412023-12-30T06:25:29.872Z[SOL Session operational. Use ~? for help]
10422023-12-30T06:28:18.490ZError: No response to keepalive - Terminating session
10432023-12-30T06:28:28.591ZError: No response de-activating SOL payload
10442023-12-30T06:28:29.793Z[SOL Session operational. Use ~? for help]
 
10452023-12-30T06:29:56.973Zdone!
10462023-12-30T06:29:57.073Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
10472023-12-30T06:29:57.179Zsuccess
10482023-12-30T06:29:57.179ZImage {
10492023-12-30T06:29:57.203Z block_size: ByteCount(
10502023-12-30T06:29:57.204Z 512,
10512023-12-30T06:29:57.204Z ),
10522023-12-30T06:29:57.204Z description: "debian 11 original base image",
10532023-12-30T06:29:57.204Z digest: None,
10542023-12-30T06:29:57.204Z id: 2ef40287-dc5e-4da5-919f-3ddb4401756c,
10552023-12-30T06:29:57.204Z name: Name(
10562023-12-30T06:29:57.204Z "debian11",
10572023-12-30T06:29:57.204Z ),
10582023-12-30T06:29:57.204Z os: "debian",
10592023-12-30T06:29:57.204Z project_id: None,
10602023-12-30T06:29:57.204Z size: ByteCount(
10612023-12-30T06:29:57.204Z 2147483648,
10622023-12-30T06:29:57.204Z ),
10632023-12-30T06:29:57.204Z time_created: 2023-12-30T06:29:56.907858Z,
10642023-12-30T06:29:57.204Z time_modified: 2023-12-30T06:29:57.162045Z,
10652023-12-30T06:29:57.204Z url: None,
10662023-12-30T06:29:57.204Z version: "11",
10672023-12-30T06:29:57.204Z}
10682023-12-30T06:29:57.204Z+ rm ./tests/bootstrap
10692023-12-30T06:29:57.259Z+ for test_bin in tests/*
10702023-12-30T06:29:57.259Z+ ./tests/bootstrap-73e660426b4fbae9
10712023-12-30T06:29:57.283Z
10722023-12-30T06:29:57.283Zrunning 0 tests
10732023-12-30T06:29:57.283Z
10742023-12-30T06:29:57.283Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10752023-12-30T06:29:57.283Z
10762023-12-30T06:29:57.283Z+ for test_bin in tests/*
10772023-12-30T06:29:57.283Z+ ./tests/end_to_end_tests-26f64a1dd7a0c081
10782023-12-30T06:29:57.318Z
10792023-12-30T06:29:57.318Zrunning 2 tests
10802023-12-30T06:29:57.539Ztest no_spoof::no_spoof_here ... ok
10812023-12-30T06:30:32.048Ztest instance_launch::instance_launch ... ok
10822023-12-30T06:30:32.048Z
10832023-12-30T06:30:32.048Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 34.73s
10842023-12-30T06:30:32.048Z
10852023-12-30T06:30:32.073Z+ _exit_trap
10862023-12-30T06:30:32.074Z+ local status=0
10872023-12-30T06:30:32.074Z+ [[ 0 -eq 0 ]]
10882023-12-30T06:30:32.074Z+ exit 0
10892023-12-30T06:30:32.074Zprocess exited: duration 624708 ms, exit code 0
 
10902023-12-30T06:30:32.176Zfound 108 output files
10912023-12-30T06:30:32.177Zuploading: /var/svc/log/oxide-sled-agent:default.log (666503 bytes)
10922023-12-30T06:30:33.467Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10932023-12-30T06:30:33.467Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_8022db93-0429-4451-aabb-553cad3ab24f/root/var/svc/log/oxide-clickhouse:default.log (60633 bytes)
10942023-12-30T06:30:34.654Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_8022db93-0429-4451-aabb-553cad3ab24f/root/var/svc/log/oxide-clickhouse:default.log
10952023-12-30T06:30:34.654Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_8022db93-0429-4451-aabb-553cad3ab24f/root/var/svc/log/oxide-clickhouse:default.log.0 (824445 bytes)
10962023-12-30T06:30:36.595Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_8022db93-0429-4451-aabb-553cad3ab24f/root/var/svc/log/oxide-clickhouse:default.log.0
10972023-12-30T06:30:36.595Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_41198fce-880f-4967-b520-bb2dd2496cf0/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10982023-12-30T06:30:37.671Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_41198fce-880f-4967-b520-bb2dd2496cf0/root/var/svc/log/oxide-cockroachdb:default.log
10992023-12-30T06:30:37.671Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_41198fce-880f-4967-b520-bb2dd2496cf0/root/var/svc/log/oxide-cockroachdb:default.log.0 (5411 bytes)
11002023-12-30T06:30:38.770Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_41198fce-880f-4967-b520-bb2dd2496cf0/root/var/svc/log/oxide-cockroachdb:default.log.0
11012023-12-30T06:30:38.770Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2ddf6fbf-a718-4c8c-ba8e-f59525692cf8/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11022023-12-30T06:30:39.840Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2ddf6fbf-a718-4c8c-ba8e-f59525692cf8/root/var/svc/log/oxide-crucible-agent:default.log
11032023-12-30T06:30:39.840Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2ddf6fbf-a718-4c8c-ba8e-f59525692cf8/root/var/svc/log/oxide-crucible-agent:default.log.0 (9793 bytes)
11042023-12-30T06:30:40.956Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2ddf6fbf-a718-4c8c-ba8e-f59525692cf8/root/var/svc/log/oxide-crucible-agent:default.log.0
11052023-12-30T06:30:40.982Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2ddf6fbf-a718-4c8c-ba8e-f59525692cf8/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11062023-12-30T06:30:42.028Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2ddf6fbf-a718-4c8c-ba8e-f59525692cf8/root/var/svc/log/oxide-crucible-downstairs:default.log
11072023-12-30T06:30:42.028Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2ddf6fbf-a718-4c8c-ba8e-f59525692cf8/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11082023-12-30T06:30:43.124Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2ddf6fbf-a718-4c8c-ba8e-f59525692cf8/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11092023-12-30T06:30:43.124Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2ddf6fbf-a718-4c8c-ba8e-f59525692cf8/root/var/svc/log/oxide-crucible-downstairs:downstairs-b916c0f9-9c0e-4d48-852d-07542b09ea89.log (0 bytes)
11102023-12-30T06:30:44.221Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2ddf6fbf-a718-4c8c-ba8e-f59525692cf8/root/var/svc/log/oxide-crucible-downstairs:downstairs-b916c0f9-9c0e-4d48-852d-07542b09ea89.log
11112023-12-30T06:30:44.221Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2ddf6fbf-a718-4c8c-ba8e-f59525692cf8/root/var/svc/log/oxide-crucible-downstairs:downstairs-b916c0f9-9c0e-4d48-852d-07542b09ea89.log.0 (2945 bytes)
11122023-12-30T06:30:45.322Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2ddf6fbf-a718-4c8c-ba8e-f59525692cf8/root/var/svc/log/oxide-crucible-downstairs:downstairs-b916c0f9-9c0e-4d48-852d-07542b09ea89.log.0
11132023-12-30T06:30:45.322Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11142023-12-30T06:30:46.395Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-agent:default.log
11152023-12-30T06:30:46.395Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-agent:default.log.0 (17550 bytes)
11162023-12-30T06:30:47.536Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-agent:default.log.0
11172023-12-30T06:30:47.537Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11182023-12-30T06:30:48.608Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-downstairs:default.log
11192023-12-30T06:30:48.608Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11202023-12-30T06:30:49.716Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11212023-12-30T06:30:49.740Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-downstairs:downstairs-7f6e5b02-e5d9-44ed-a178-4baee8ca43ec.log (0 bytes)
11222023-12-30T06:30:50.788Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-downstairs:downstairs-7f6e5b02-e5d9-44ed-a178-4baee8ca43ec.log
11232023-12-30T06:30:50.823Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-downstairs:downstairs-7f6e5b02-e5d9-44ed-a178-4baee8ca43ec.log.0 (6438 bytes)
11242023-12-30T06:30:51.882Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-downstairs:downstairs-7f6e5b02-e5d9-44ed-a178-4baee8ca43ec.log.0
11252023-12-30T06:30:51.883Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-downstairs:snapshot-7f6e5b02-e5d9-44ed-a178-4baee8ca43ec-4bd36969-7ce7-4479-bc7f-aebaf430c842.log (2182 bytes)
11262023-12-30T06:30:52.977Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-downstairs:snapshot-7f6e5b02-e5d9-44ed-a178-4baee8ca43ec-4bd36969-7ce7-4479-bc7f-aebaf430c842.log
11272023-12-30T06:30:52.977Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-downstairs:snapshot-7f6e5b02-e5d9-44ed-a178-4baee8ca43ec-4bd36969-7ce7-4479-bc7f-aebaf430c842.log.0 (2995 bytes)
11282023-12-30T06:30:54.072Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_ef06a8a0-5440-4b00-85eb-a158a0bb5dfa/root/var/svc/log/oxide-crucible-downstairs:snapshot-7f6e5b02-e5d9-44ed-a178-4baee8ca43ec-4bd36969-7ce7-4479-bc7f-aebaf430c842.log.0
11292023-12-30T06:30:54.072Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_2cc67b7e-dbe8-4f42-a254-89b75f6e24de/root/var/svc/log/oxide-internal_dns:default.log (3755 bytes)
11302023-12-30T06:30:55.166Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_2cc67b7e-dbe8-4f42-a254-89b75f6e24de/root/var/svc/log/oxide-internal_dns:default.log
11312023-12-30T06:30:55.166Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_2cc67b7e-dbe8-4f42-a254-89b75f6e24de/root/var/svc/log/oxide-internal_dns:default.log.0 (129920 bytes)
11322023-12-30T06:30:56.392Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_2cc67b7e-dbe8-4f42-a254-89b75f6e24de/root/var/svc/log/oxide-internal_dns:default.log.0
11332023-12-30T06:30:56.392Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_ff370632-6232-4a1a-b0ff-e5ee061503a1/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11342023-12-30T06:30:57.465Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_ff370632-6232-4a1a-b0ff-e5ee061503a1/root/var/svc/log/oxide-cockroachdb:default.log
11352023-12-30T06:30:57.465Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_ff370632-6232-4a1a-b0ff-e5ee061503a1/root/var/svc/log/oxide-cockroachdb:default.log.0 (5413 bytes)
11362023-12-30T06:30:58.559Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_ff370632-6232-4a1a-b0ff-e5ee061503a1/root/var/svc/log/oxide-cockroachdb:default.log.0
11372023-12-30T06:30:58.559Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_16116a53-42de-4f42-a8a0-e8f288c5f6f4/root/var/svc/log/oxide-crucible-agent:default.log (4323 bytes)
11382023-12-30T06:30:59.684Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_16116a53-42de-4f42-a8a0-e8f288c5f6f4/root/var/svc/log/oxide-crucible-agent:default.log
11392023-12-30T06:30:59.708Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_16116a53-42de-4f42-a8a0-e8f288c5f6f4/root/var/svc/log/oxide-crucible-agent:default.log.0 (9795 bytes)
11402023-12-30T06:31:00.800Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_16116a53-42de-4f42-a8a0-e8f288c5f6f4/root/var/svc/log/oxide-crucible-agent:default.log.0
11412023-12-30T06:31:00.801Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_16116a53-42de-4f42-a8a0-e8f288c5f6f4/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11422023-12-30T06:31:01.877Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_16116a53-42de-4f42-a8a0-e8f288c5f6f4/root/var/svc/log/oxide-crucible-downstairs:default.log
11432023-12-30T06:31:01.877Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_16116a53-42de-4f42-a8a0-e8f288c5f6f4/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11442023-12-30T06:31:02.973Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_16116a53-42de-4f42-a8a0-e8f288c5f6f4/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11452023-12-30T06:31:02.973Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_16116a53-42de-4f42-a8a0-e8f288c5f6f4/root/var/svc/log/oxide-crucible-downstairs:downstairs-4d857476-f820-47dd-ad79-ff25c1e8ce86.log (2592 bytes)
11462023-12-30T06:31:04.069Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_16116a53-42de-4f42-a8a0-e8f288c5f6f4/root/var/svc/log/oxide-crucible-downstairs:downstairs-4d857476-f820-47dd-ad79-ff25c1e8ce86.log
11472023-12-30T06:31:04.069Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_16116a53-42de-4f42-a8a0-e8f288c5f6f4/root/var/svc/log/oxide-crucible-downstairs:downstairs-4d857476-f820-47dd-ad79-ff25c1e8ce86.log.0 (2945 bytes)
11482023-12-30T06:31:05.164Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_16116a53-42de-4f42-a8a0-e8f288c5f6f4/root/var/svc/log/oxide-crucible-downstairs:downstairs-4d857476-f820-47dd-ad79-ff25c1e8ce86.log.0
11492023-12-30T06:31:05.164Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_deb875a4-116b-4593-947f-889f1ba1be8b/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11502023-12-30T06:31:06.236Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_deb875a4-116b-4593-947f-889f1ba1be8b/root/var/svc/log/oxide-crucible-agent:default.log
11512023-12-30T06:31:06.236Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_deb875a4-116b-4593-947f-889f1ba1be8b/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11522023-12-30T06:31:07.331Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_deb875a4-116b-4593-947f-889f1ba1be8b/root/var/svc/log/oxide-crucible-agent:default.log.0
11532023-12-30T06:31:07.331Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_deb875a4-116b-4593-947f-889f1ba1be8b/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11542023-12-30T06:31:08.405Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_deb875a4-116b-4593-947f-889f1ba1be8b/root/var/svc/log/oxide-crucible-downstairs:default.log
11552023-12-30T06:31:08.405Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_deb875a4-116b-4593-947f-889f1ba1be8b/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11562023-12-30T06:31:09.499Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_deb875a4-116b-4593-947f-889f1ba1be8b/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11572023-12-30T06:31:09.499Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_3ee05eb4-b633-42e6-b43b-bf1291d4cfb0/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11582023-12-30T06:31:10.574Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_3ee05eb4-b633-42e6-b43b-bf1291d4cfb0/root/var/svc/log/oxide-crucible-pantry:default.log
11592023-12-30T06:31:10.599Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_3ee05eb4-b633-42e6-b43b-bf1291d4cfb0/root/var/svc/log/oxide-crucible-pantry:default.log.0 (1228957 bytes)
11602023-12-30T06:31:12.367Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_3ee05eb4-b633-42e6-b43b-bf1291d4cfb0/root/var/svc/log/oxide-crucible-pantry:default.log.0
11612023-12-30T06:31:12.394Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_d31aa598-00cb-45f4-af68-31845f024fb3/root/var/svc/log/oxide-internal_dns:default.log (3756 bytes)
11622023-12-30T06:31:13.464Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_d31aa598-00cb-45f4-af68-31845f024fb3/root/var/svc/log/oxide-internal_dns:default.log
11632023-12-30T06:31:13.464Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_d31aa598-00cb-45f4-af68-31845f024fb3/root/var/svc/log/oxide-internal_dns:default.log.0 (125936 bytes)
11642023-12-30T06:31:14.669Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_d31aa598-00cb-45f4-af68-31845f024fb3/root/var/svc/log/oxide-internal_dns:default.log.0
11652023-12-30T06:31:14.669Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_d4e260cc-74b5-4a2d-8fb5-e37943d8e002/root/var/svc/log/oxide-nexus:default.log (914201 bytes)
11662023-12-30T06:31:14.945Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_d4e260cc-74b5-4a2d-8fb5-e37943d8e002/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 914201 -> 921691
11672023-12-30T06:31:16.019Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_d4e260cc-74b5-4a2d-8fb5-e37943d8e002/root/var/svc/log/oxide-nexus:default.log
11682023-12-30T06:31:16.019Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_d4e260cc-74b5-4a2d-8fb5-e37943d8e002/root/var/svc/log/oxide-nexus:default.log.0 (33151896 bytes)
11692023-12-30T06:31:18.774ZError: No response to keepalive - Terminating session
11702023-12-30T06:31:26.579Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_d4e260cc-74b5-4a2d-8fb5-e37943d8e002/root/var/svc/log/oxide-nexus:default.log.0
11712023-12-30T06:31:26.580Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb/root/var/svc/log/oxide-ntp:default.log (0 bytes)
11722023-12-30T06:31:27.650Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb/root/var/svc/log/oxide-ntp:default.log
11732023-12-30T06:31:27.650Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb/root/var/svc/log/oxide-ntp:default.log.0 (1322 bytes)
11742023-12-30T06:31:28.646ZError: No response de-activating SOL payload
11752023-12-30T06:31:28.745Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb/root/var/svc/log/oxide-ntp:default.log.0
11762023-12-30T06:31:28.745Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_dcd2dca2-af93-482c-84fe-2805eceaacdc/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11772023-12-30T06:31:29.816Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_dcd2dca2-af93-482c-84fe-2805eceaacdc/root/var/svc/log/oxide-cockroachdb:default.log
11782023-12-30T06:31:29.817Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_dcd2dca2-af93-482c-84fe-2805eceaacdc/root/var/svc/log/oxide-cockroachdb:default.log.0 (5412 bytes)
11792023-12-30T06:31:29.849Z[SOL Session operational. Use ~? for help]
11802023-12-30T06:31:30.913Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_dcd2dca2-af93-482c-84fe-2805eceaacdc/root/var/svc/log/oxide-cockroachdb:default.log.0
11812023-12-30T06:31:30.913Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_ef608542-3261-49b3-8a1b-358adf45b96a/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11822023-12-30T06:31:31.986Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_ef608542-3261-49b3-8a1b-358adf45b96a/root/var/svc/log/oxide-crucible-pantry:default.log
11832023-12-30T06:31:31.986Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_ef608542-3261-49b3-8a1b-358adf45b96a/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
11842023-12-30T06:31:33.082Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_ef608542-3261-49b3-8a1b-358adf45b96a/root/var/svc/log/oxide-crucible-pantry:default.log.0
11852023-12-30T06:31:33.083Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_0b8bfc22-c753-4a17-a70f-ba26af17a184/root/var/svc/log/oxide-internal_dns:default.log (18866 bytes)
11862023-12-30T06:31:33.175Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_0b8bfc22-c753-4a17-a70f-ba26af17a184/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 18866 -> 22621
11872023-12-30T06:31:34.247Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_0b8bfc22-c753-4a17-a70f-ba26af17a184/root/var/svc/log/oxide-internal_dns:default.log
11882023-12-30T06:31:34.271Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_0b8bfc22-c753-4a17-a70f-ba26af17a184/root/var/svc/log/oxide-internal_dns:default.log.0 (112849 bytes)
11892023-12-30T06:31:35.459Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_0b8bfc22-c753-4a17-a70f-ba26af17a184/root/var/svc/log/oxide-internal_dns:default.log.0
11902023-12-30T06:31:35.460Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_3de07407-b466-457d-ab72-2488d391a245/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11912023-12-30T06:31:36.532Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_3de07407-b466-457d-ab72-2488d391a245/root/var/svc/log/oxide-cockroachdb:default.log
11922023-12-30T06:31:36.532Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_3de07407-b466-457d-ab72-2488d391a245/root/var/svc/log/oxide-cockroachdb:default.log.0 (5386 bytes)
11932023-12-30T06:31:37.631Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_3de07407-b466-457d-ab72-2488d391a245/root/var/svc/log/oxide-cockroachdb:default.log.0
11942023-12-30T06:31:37.631Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_c98f6bec-d7a0-4a8e-8cb2-229c8445b635/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11952023-12-30T06:31:38.703Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_c98f6bec-d7a0-4a8e-8cb2-229c8445b635/root/var/svc/log/oxide-cockroachdb:default.log
11962023-12-30T06:31:38.703Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_c98f6bec-d7a0-4a8e-8cb2-229c8445b635/root/var/svc/log/oxide-cockroachdb:default.log.0 (5410 bytes)
11972023-12-30T06:31:39.799Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_c98f6bec-d7a0-4a8e-8cb2-229c8445b635/root/var/svc/log/oxide-cockroachdb:default.log.0
11982023-12-30T06:31:39.799Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_103fe16d-906c-4d70-9eed-5cd0ce3cc079/root/var/svc/log/oxide-oximeter:default.log (22127 bytes)
11992023-12-30T06:31:39.941Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_103fe16d-906c-4d70-9eed-5cd0ce3cc079/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 22127 -> 60953
12002023-12-30T06:31:41.036Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_103fe16d-906c-4d70-9eed-5cd0ce3cc079/root/var/svc/log/oxide-oximeter:default.log
12012023-12-30T06:31:41.036Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_103fe16d-906c-4d70-9eed-5cd0ce3cc079/root/var/svc/log/oxide-oximeter:default.log.0 (252006 bytes)
12022023-12-30T06:31:42.294Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_103fe16d-906c-4d70-9eed-5cd0ce3cc079/root/var/svc/log/oxide-oximeter:default.log.0
12032023-12-30T06:31:42.319Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6b5905e6-d01e-4ee0-a010-aa942ef479ed/root/var/svc/log/oxide-crucible-agent:default.log (4324 bytes)
12042023-12-30T06:31:43.392Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6b5905e6-d01e-4ee0-a010-aa942ef479ed/root/var/svc/log/oxide-crucible-agent:default.log
12052023-12-30T06:31:43.392Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6b5905e6-d01e-4ee0-a010-aa942ef479ed/root/var/svc/log/oxide-crucible-agent:default.log.0 (9362 bytes)
12062023-12-30T06:31:44.511Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6b5905e6-d01e-4ee0-a010-aa942ef479ed/root/var/svc/log/oxide-crucible-agent:default.log.0
12072023-12-30T06:31:44.511Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6b5905e6-d01e-4ee0-a010-aa942ef479ed/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12082023-12-30T06:31:45.584Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6b5905e6-d01e-4ee0-a010-aa942ef479ed/root/var/svc/log/oxide-crucible-downstairs:default.log
12092023-12-30T06:31:45.584Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6b5905e6-d01e-4ee0-a010-aa942ef479ed/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12102023-12-30T06:31:46.683Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6b5905e6-d01e-4ee0-a010-aa942ef479ed/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12112023-12-30T06:31:46.683Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6b5905e6-d01e-4ee0-a010-aa942ef479ed/root/var/svc/log/oxide-crucible-downstairs:downstairs-3de963cc-2db5-4e5b-9029-35a97f731cb0.log (2592 bytes)
12122023-12-30T06:31:47.780Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6b5905e6-d01e-4ee0-a010-aa942ef479ed/root/var/svc/log/oxide-crucible-downstairs:downstairs-3de963cc-2db5-4e5b-9029-35a97f731cb0.log
12132023-12-30T06:31:47.780Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6b5905e6-d01e-4ee0-a010-aa942ef479ed/root/var/svc/log/oxide-crucible-downstairs:downstairs-3de963cc-2db5-4e5b-9029-35a97f731cb0.log.0 (2945 bytes)
12142023-12-30T06:31:48.874Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6b5905e6-d01e-4ee0-a010-aa942ef479ed/root/var/svc/log/oxide-crucible-downstairs:downstairs-3de963cc-2db5-4e5b-9029-35a97f731cb0.log.0
12152023-12-30T06:31:48.875Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_7c8432c2-1cfe-47f4-9a67-d3c2709cbceb/root/var/svc/log/oxide-nexus:default.log (170433 bytes)
12162023-12-30T06:31:49.082Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_7c8432c2-1cfe-47f4-9a67-d3c2709cbceb/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 170433 -> 246580
12172023-12-30T06:31:50.153Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_7c8432c2-1cfe-47f4-9a67-d3c2709cbceb/root/var/svc/log/oxide-nexus:default.log
12182023-12-30T06:31:50.153Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_7c8432c2-1cfe-47f4-9a67-d3c2709cbceb/root/var/svc/log/oxide-nexus:default.log.0 (2630254 bytes)
12192023-12-30T06:31:52.797Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_7c8432c2-1cfe-47f4-9a67-d3c2709cbceb/root/var/svc/log/oxide-nexus:default.log.0
12202023-12-30T06:31:52.797Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_e4aa2c60-952e-4093-8472-26be63544fe6/root/var/svc/log/oxide-nexus:default.log (68035 bytes)
12212023-12-30T06:31:52.978Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_e4aa2c60-952e-4093-8472-26be63544fe6/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 68035 -> 146259
12222023-12-30T06:31:54.046Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_e4aa2c60-952e-4093-8472-26be63544fe6/root/var/svc/log/oxide-nexus:default.log
12232023-12-30T06:31:54.046Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_e4aa2c60-952e-4093-8472-26be63544fe6/root/var/svc/log/oxide-nexus:default.log.0 (774247 bytes)
12242023-12-30T06:31:55.372Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_e4aa2c60-952e-4093-8472-26be63544fe6/root/var/svc/log/oxide-nexus:default.log.0
12252023-12-30T06:31:55.372Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
12262023-12-30T06:31:56.446Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-agent:default.log
12272023-12-30T06:31:56.446Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-agent:default.log.0 (27798 bytes)
12282023-12-30T06:31:57.580Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-agent:default.log.0
12292023-12-30T06:31:57.580Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12302023-12-30T06:31:58.650Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:default.log
12312023-12-30T06:31:58.650Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12322023-12-30T06:31:59.743Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12332023-12-30T06:31:59.744Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:downstairs-69c10c18-b360-467f-ad00-564c984bf576.log (0 bytes)
12342023-12-30T06:32:00.814Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:downstairs-69c10c18-b360-467f-ad00-564c984bf576.log
12352023-12-30T06:32:00.814Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:downstairs-69c10c18-b360-467f-ad00-564c984bf576.log.0 (6435 bytes)
12362023-12-30T06:32:01.909Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:downstairs-69c10c18-b360-467f-ad00-564c984bf576.log.0
12372023-12-30T06:32:01.909Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:downstairs-7b95296d-4d6b-4d14-83f9-428dbd56abfd.log (0 bytes)
12382023-12-30T06:32:02.983Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:downstairs-7b95296d-4d6b-4d14-83f9-428dbd56abfd.log
12392023-12-30T06:32:02.983Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:downstairs-7b95296d-4d6b-4d14-83f9-428dbd56abfd.log.0 (3048 bytes)
12402023-12-30T06:32:04.080Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:downstairs-7b95296d-4d6b-4d14-83f9-428dbd56abfd.log.0
12412023-12-30T06:32:04.080Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:snapshot-69c10c18-b360-467f-ad00-564c984bf576-4bd36969-7ce7-4479-bc7f-aebaf430c842.log (2182 bytes)
12422023-12-30T06:32:05.177Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:snapshot-69c10c18-b360-467f-ad00-564c984bf576-4bd36969-7ce7-4479-bc7f-aebaf430c842.log
12432023-12-30T06:32:05.177Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:snapshot-69c10c18-b360-467f-ad00-564c984bf576-4bd36969-7ce7-4479-bc7f-aebaf430c842.log.0 (2996 bytes)
12442023-12-30T06:32:06.272Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_84bd1ff1-d683-4f80-b048-b9283bd8891f/root/var/svc/log/oxide-crucible-downstairs:snapshot-69c10c18-b360-467f-ad00-564c984bf576-4bd36969-7ce7-4479-bc7f-aebaf430c842.log.0
12452023-12-30T06:32:06.272Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-agent:default.log (4784 bytes)
12462023-12-30T06:32:07.401Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-agent:default.log
12472023-12-30T06:32:07.401Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-agent:default.log.0 (40545 bytes)
12482023-12-30T06:32:08.601Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-agent:default.log.0
12492023-12-30T06:32:08.601Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12502023-12-30T06:32:09.671Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:default.log
12512023-12-30T06:32:09.672Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12522023-12-30T06:32:10.765Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12532023-12-30T06:32:10.766Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-32a11cc6-8c7b-4189-82f5-48e133a29b3e.log (0 bytes)
12542023-12-30T06:32:11.836Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-32a11cc6-8c7b-4189-82f5-48e133a29b3e.log
12552023-12-30T06:32:11.836Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-32a11cc6-8c7b-4189-82f5-48e133a29b3e.log.0 (6437 bytes)
12562023-12-30T06:32:12.931Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-32a11cc6-8c7b-4189-82f5-48e133a29b3e.log.0
12572023-12-30T06:32:12.931Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-898ba538-1006-4c88-af3e-63e6eb7b2c28.log (0 bytes)
12582023-12-30T06:32:14.038Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-898ba538-1006-4c88-af3e-63e6eb7b2c28.log
12592023-12-30T06:32:14.064Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-898ba538-1006-4c88-af3e-63e6eb7b2c28.log.0 (2945 bytes)
12602023-12-30T06:32:15.143Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-898ba538-1006-4c88-af3e-63e6eb7b2c28.log.0
12612023-12-30T06:32:15.143Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-a1c87332-7363-47d3-aba7-d96831992027.log (2592 bytes)
12622023-12-30T06:32:16.238Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-a1c87332-7363-47d3-aba7-d96831992027.log
12632023-12-30T06:32:16.238Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-a1c87332-7363-47d3-aba7-d96831992027.log.0 (2944 bytes)
12642023-12-30T06:32:17.334Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-a1c87332-7363-47d3-aba7-d96831992027.log.0
12652023-12-30T06:32:17.335Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-ec099177-3e7c-4422-9406-cb5dd6df8f26.log (0 bytes)
12662023-12-30T06:32:18.404Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-ec099177-3e7c-4422-9406-cb5dd6df8f26.log
12672023-12-30T06:32:18.405Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-ec099177-3e7c-4422-9406-cb5dd6df8f26.log.0 (3050 bytes)
12682023-12-30T06:32:19.498Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:downstairs-ec099177-3e7c-4422-9406-cb5dd6df8f26.log.0
12692023-12-30T06:32:19.522Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:snapshot-32a11cc6-8c7b-4189-82f5-48e133a29b3e-4bd36969-7ce7-4479-bc7f-aebaf430c842.log (2182 bytes)
12702023-12-30T06:32:20.592Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:snapshot-32a11cc6-8c7b-4189-82f5-48e133a29b3e-4bd36969-7ce7-4479-bc7f-aebaf430c842.log
12712023-12-30T06:32:20.592Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:snapshot-32a11cc6-8c7b-4189-82f5-48e133a29b3e-4bd36969-7ce7-4479-bc7f-aebaf430c842.log.0 (2994 bytes)
12722023-12-30T06:32:21.686Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d102ca9b-f329-4934-aebc-2f9b906f83a7/root/var/svc/log/oxide-crucible-downstairs:snapshot-32a11cc6-8c7b-4189-82f5-48e133a29b3e-4bd36969-7ce7-4479-bc7f-aebaf430c842.log.0
12732023-12-30T06:32:21.686Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_90367355-5819-460f-a629-f5c03d4d4681/root/var/svc/log/oxide-external_dns:default.log (3779 bytes)
12742023-12-30T06:32:21.757Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_90367355-5819-460f-a629-f5c03d4d4681/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 3779 -> 11333
12752023-12-30T06:32:22.828Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_90367355-5819-460f-a629-f5c03d4d4681/root/var/svc/log/oxide-external_dns:default.log
12762023-12-30T06:32:22.828Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_90367355-5819-460f-a629-f5c03d4d4681/root/var/svc/log/oxide-external_dns:default.log.0 (26943 bytes)
12772023-12-30T06:32:23.973Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_90367355-5819-460f-a629-f5c03d4d4681/root/var/svc/log/oxide-external_dns:default.log.0
12782023-12-30T06:32:23.973Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_aaedf38a-af39-488c-b0c7-08f5645ac4dd/root/var/svc/log/oxide-external_dns:default.log (3779 bytes)
12792023-12-30T06:32:24.043Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_aaedf38a-af39-488c-b0c7-08f5645ac4dd/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 3779 -> 11333
12802023-12-30T06:32:25.113Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_aaedf38a-af39-488c-b0c7-08f5645ac4dd/root/var/svc/log/oxide-external_dns:default.log
12812023-12-30T06:32:25.113Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_aaedf38a-af39-488c-b0c7-08f5645ac4dd/root/var/svc/log/oxide-external_dns:default.log.0 (26943 bytes)
12822023-12-30T06:32:26.251Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_aaedf38a-af39-488c-b0c7-08f5645ac4dd/root/var/svc/log/oxide-external_dns:default.log.0
12832023-12-30T06:32:26.277Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_80ab5f5b-d305-4db2-9a09-ba351c0a3e90/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
12842023-12-30T06:32:27.323Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_80ab5f5b-d305-4db2-9a09-ba351c0a3e90/root/var/svc/log/oxide-crucible-agent:default.log
12852023-12-30T06:32:27.323Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_80ab5f5b-d305-4db2-9a09-ba351c0a3e90/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
12862023-12-30T06:32:28.420Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_80ab5f5b-d305-4db2-9a09-ba351c0a3e90/root/var/svc/log/oxide-crucible-agent:default.log.0
12872023-12-30T06:32:28.420Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_80ab5f5b-d305-4db2-9a09-ba351c0a3e90/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12882023-12-30T06:32:29.517Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_80ab5f5b-d305-4db2-9a09-ba351c0a3e90/root/var/svc/log/oxide-crucible-downstairs:default.log
12892023-12-30T06:32:29.517Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_80ab5f5b-d305-4db2-9a09-ba351c0a3e90/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12902023-12-30T06:32:30.612Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_80ab5f5b-d305-4db2-9a09-ba351c0a3e90/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12912023-12-30T06:32:30.612Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
12922023-12-30T06:32:31.684Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-agent:default.log
12932023-12-30T06:32:31.684Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-agent:default.log.0 (20048 bytes)
12942023-12-30T06:32:32.837Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-agent:default.log.0
12952023-12-30T06:32:32.837Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12962023-12-30T06:32:33.909Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-downstairs:default.log
12972023-12-30T06:32:33.909Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12982023-12-30T06:32:35.004Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12992023-12-30T06:32:35.029Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-downstairs:downstairs-03bf3a15-5107-4c4a-9c47-f5d9cc4e9754.log (0 bytes)
13002023-12-30T06:32:36.076Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-downstairs:downstairs-03bf3a15-5107-4c4a-9c47-f5d9cc4e9754.log
13012023-12-30T06:32:36.104Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-downstairs:downstairs-03bf3a15-5107-4c4a-9c47-f5d9cc4e9754.log.0 (3050 bytes)
13022023-12-30T06:32:37.170Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-downstairs:downstairs-03bf3a15-5107-4c4a-9c47-f5d9cc4e9754.log.0
13032023-12-30T06:32:37.170Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-downstairs:downstairs-208261b8-0f10-41c4-a4e6-152e5c0af30c.log (0 bytes)
13042023-12-30T06:32:38.245Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-downstairs:downstairs-208261b8-0f10-41c4-a4e6-152e5c0af30c.log
13052023-12-30T06:32:38.246Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-downstairs:downstairs-208261b8-0f10-41c4-a4e6-152e5c0af30c.log.0 (2945 bytes)
13062023-12-30T06:32:39.344Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f7fec8b1-2b0b-402e-acd4-bc8c6b9864e1/root/var/svc/log/oxide-crucible-downstairs:downstairs-208261b8-0f10-41c4-a4e6-152e5c0af30c.log.0
13072023-12-30T06:32:39.344Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b4028ad3-f0f2-4915-b9b9-23e3f71d0915/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
13082023-12-30T06:32:40.415Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b4028ad3-f0f2-4915-b9b9-23e3f71d0915/root/var/svc/log/oxide-crucible-pantry:default.log
13092023-12-30T06:32:40.415Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b4028ad3-f0f2-4915-b9b9-23e3f71d0915/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
13102023-12-30T06:32:41.509Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b4028ad3-f0f2-4915-b9b9-23e3f71d0915/root/var/svc/log/oxide-crucible-pantry:default.log.0
13112023-12-30T06:32:41.510Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb/root/var/log/chrony/measurements.log (19865 bytes)
13122023-12-30T06:32:41.601Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb/root/var/log/chrony/measurements.log" changed size mid upload: 19865 -> 23975
13132023-12-30T06:32:42.672Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb/root/var/log/chrony/measurements.log
13142023-12-30T06:32:42.672Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb/root/var/log/chrony/statistics.log (11232 bytes)
13152023-12-30T06:32:42.741Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb/root/var/log/chrony/statistics.log" changed size mid upload: 11232 -> 14976
13162023-12-30T06:32:43.811Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb/root/var/log/chrony/statistics.log
13172023-12-30T06:32:43.811Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb/root/var/log/chrony/tracking.log (2383 bytes)
13182023-12-30T06:32:43.858Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb/root/var/log/chrony/tracking.log" changed size mid upload: 2383 -> 2647
13192023-12-30T06:32:44.932Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_5181c903-1d19-427f-996f-f2ebb150e3bb/root/var/log/chrony/tracking.log