01HNGEHGHQTRPMYDZK34FFKP6N: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.28",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HNGEKK2T3QCT7A9GZP3HD5W7

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12024-01-31T20:07:21.783Zjob dependencies complete; ready to run (waiting for 1 h 1 m 30 s)
22024-01-31T20:57:22.722Zjob assigned to worker 01HNGMXJ4MZBD582R5DDGAS2B1 (queued for 50 m 0 s)
32024-01-31T20:57:22.899Zhttp://172.20.0.5:9969/ipxe/sock... ok
42024-01-31T20:57:23.359Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52024-01-31T20:57:23.402Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62024-01-31T20:57:23.443Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72024-01-31T20:57:23.532Zhttp://172.20.0.5:9969/postboot/sock... ok
82024-01-31T20:57:23.574ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92024-01-31T20:57:23.612Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102024-01-31T20:57:23.657ZOxide Helios Version helios-2.0.22418 64-bit
112024-01-31T20:57:23.700ZConfiguring devices.
122024-01-31T20:57:23.775Z##################################################
132024-01-31T20:57:23.845Z#### ############# ##
142024-01-31T20:57:23.893Z### ### ############ ##
152024-01-31T20:57:23.939Z## ### # ## ### ## Oxide Computer Company ##
162024-01-31T20:57:23.987Z## ## ## ### # ### ##
172024-01-31T20:57:24.026Z## # ### #### #### This Station Under ##
182024-01-31T20:57:24.084Z### ### #### # ### Computer Control ##
192024-01-31T20:57:24.127Z#### #### ### ## ##
202024-01-31T20:57:24.168Z##################################################
212024-01-31T20:57:24.221ZHostname: sock
222024-01-31T20:57:24.267ZLoading smf(7) service descriptions:
232024-01-31T20:57:25.224Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-485102d0e6124cf0.gz
242024-01-31T20:57:30.420Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-485102d0e6124cf0.gz
252024-01-31T20:57:30.447Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262024-01-31T20:58:19.271Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272024-01-31T20:58:19.271Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8616f2942d631494.gz
282024-01-31T20:58:54.808Zsock console login: Error: No response to keepalive - Terminating session
292024-01-31T20:59:04.858ZError: No response de-activating SOL payload
302024-01-31T20:59:05.993Z[SOL Session operational. Use ~? for help]
312024-01-31T20:59:08.055Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8616f2942d631494.gz
322024-01-31T20:59:08.055Zdownloading input: /input/ci-tools/work/caboose-util.gz
332024-01-31T20:59:09.521Zdownloaded input: /input/ci-tools/work/caboose-util.gz
342024-01-31T20:59:09.521Zdownloading input: /input/ci-tools/work/tufaceous.gz
352024-01-31T20:59:37.460Zdownloaded input: /input/ci-tools/work/tufaceous.gz
362024-01-31T20:59:37.550Zdownloading input: /input/package/work/version.txt
372024-01-31T20:59:37.584Zdownloaded input: /input/package/work/version.txt
382024-01-31T20:59:37.584Zdownloading input: /input/package/work/package.tar.gz
392024-01-31T21:00:17.758Zdownloaded input: /input/package/work/package.tar.gz
402024-01-31T21:00:17.758Zdownloading input: /input/package/work/global-zone-packages.tar.gz
412024-01-31T21:01:29.912Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
422024-01-31T21:01:29.913Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
432024-01-31T21:01:51.691Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
442024-01-31T21:01:51.909Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
452024-01-31T21:02:10.769ZError: No response to keepalive - Terminating session
462024-01-31T21:02:20.431ZError: No response de-activating SOL payload
472024-01-31T21:02:21.548Z[SOL Session operational. Use ~? for help]
482024-01-31T21:05:10.721ZError: No response to keepalive - Terminating session
492024-01-31T21:05:14.542Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
502024-01-31T21:05:14.576Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
512024-01-31T21:05:20.695ZError: No response de-activating SOL payload
522024-01-31T21:05:28.369Z[SOL Session operational. Use ~? for help]
532024-01-31T21:06:38.611Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
542024-01-31T21:06:38.611Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
552024-01-31T21:07:42.687Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
562024-01-31T21:07:42.687Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
572024-01-31T21:07:52.816Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
582024-01-31T21:07:52.816Zdownloading input: /input/package/work/zones/crucible.tar.gz
592024-01-31T21:08:09.469Zdownloaded input: /input/package/work/zones/crucible.tar.gz
602024-01-31T21:08:09.469Zdownloading input: /input/package/work/zones/external-dns.tar.gz
612024-01-31T21:08:10.859ZError: No response to keepalive - Terminating session
622024-01-31T21:08:17.546Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
632024-01-31T21:08:17.546Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
642024-01-31T21:08:20.778ZError: No response de-activating SOL payload
652024-01-31T21:08:22.286Z[SOL Session operational. Use ~? for help]
662024-01-31T21:08:25.379Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
672024-01-31T21:08:25.404Zdownloading input: /input/package/work/zones/ntp.tar.gz
682024-01-31T21:08:25.442Zdownloaded input: /input/package/work/zones/ntp.tar.gz
692024-01-31T21:08:25.442Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
702024-01-31T21:08:30.468Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
712024-01-31T21:08:30.468Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
722024-01-31T21:08:54.792Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
732024-01-31T21:08:54.792Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
742024-01-31T21:09:31.233Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
752024-01-31T21:09:31.233Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
762024-01-31T21:10:11.776Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
772024-01-31T21:10:11.777Zdownloading input: /input/package/work/zones/overlay.tar.gz
782024-01-31T21:10:11.846Zdownloaded input: /input/package/work/zones/overlay.tar.gz
792024-01-31T21:10:11.870Zdownloading input: /input/package/work/zones/oximeter.tar.gz
802024-01-31T21:10:43.132Zdownloaded input: /input/package/work/zones/oximeter.tar.gz
812024-01-31T21:10:43.133Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
822024-01-31T21:11:12.912Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
832024-01-31T21:11:13.319Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
842024-01-31T21:11:28.543ZError: No response to keepalive - Terminating session
852024-01-31T21:11:37.012ZError: No response de-activating SOL payload
862024-01-31T21:11:38.578Z[SOL Session operational. Use ~? for help]
872024-01-31T21:13:27.948Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
882024-01-31T21:13:27.949Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
892024-01-31T21:14:34.711ZError: No response to keepalive - Terminating session
902024-01-31T21:14:36.379ZError: No response de-activating SOL payload
912024-01-31T21:14:37.909Z[SOL Session operational. Use ~? for help]
922024-01-31T21:14:39.530Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
932024-01-31T21:14:39.531Zstarting task 0: "setup"
942024-01-31T21:14:39.598Z++ uname -s
952024-01-31T21:14:39.598Z+ kern=SunOS
962024-01-31T21:14:39.599Z+ case "$kern" in
972024-01-31T21:14:39.599Z+ groupadd -g 12345 build
982024-01-31T21:14:39.599Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
992024-01-31T21:14:41.668Z+ zfs create -o mountpoint=/work rpool/work
1002024-01-31T21:14:41.716Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
1012024-01-31T21:14:41.716Z+ home_fs=zfs
1022024-01-31T21:14:41.716Z+ [[ zfs == autofs ]]
1032024-01-31T21:14:41.716Z+ mkdir -p /home/build
1042024-01-31T21:14:41.716Z+ chown build:build /home/build /work
1052024-01-31T21:14:43.450Z+ chmod 0700 /home/build /work
1062024-01-31T21:14:43.716Zprocess exited: duration 4069 ms, exit code 0
 
1072024-01-31T21:14:44.196Zstarting task 1: "authentication"
1082024-01-31T21:14:44.566Zprocess exited: duration 53 ms, exit code 0
 
1092024-01-31T21:14:44.796Zstarting task 2: "build"
1102024-01-31T21:14:45.025Z+ trap _exit_trap EXIT
1112024-01-31T21:14:45.025Z+ set +x
1122024-01-31T21:15:56.851Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
1132024-01-31T21:15:57.102Zsecpolicy_fs_linkdir: 0x55 = 0x55
1142024-01-31T21:15:57.102Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
1152024-01-31T21:15:57.102Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
1162024-01-31T21:15:57.102Zsecpolicy_fs_linkdir+3: 0 = 0xe5
1172024-01-31T21:15:57.103Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1182024-01-31T21:15:57.103Zsecpolicy_fs_linkdir+5: 0 = 0x1
1192024-01-31T21:15:57.103Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1202024-01-31T21:15:57.145Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1212024-01-31T21:15:57.146Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1222024-01-31T21:15:57.146Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1232024-01-31T21:15:57.146Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1242024-01-31T21:15:57.146Z+ [[ -d /opt/oxide ]]
1252024-01-31T21:15:57.147Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1262024-01-31T21:15:57.147Z+ pfexec mkdir /opt/oxide
1272024-01-31T21:15:57.147Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1282024-01-31T21:15:57.147Z+ [[ -d /opt/oxide-underneath ]]
1292024-01-31T21:15:57.147Z+ cd /opt/oxide-underneath
1302024-01-31T21:15:57.152Z+ pfexec tar ceEp@/f - .
1312024-01-31T21:15:57.152Z+ cd /opt/oxide
1322024-01-31T21:15:57.152Z+ pfexec tar xveEp@/f -
1332024-01-31T21:15:57.189Zx ., 0 bytes, 0 tape blocks
1342024-01-31T21:15:57.189Zx ./opte, 0 bytes, 0 tape blocks
1352024-01-31T21:15:57.189Zx ./opte/bin, 0 bytes, 0 tape blocks
1362024-01-31T21:15:57.189Zx ./opte/bin/opteadm, 51799224 bytes, 101171 tape blocks
1372024-01-31T21:15:57.189Z+ pfexec rm -rf /opt/oxide-underneath
1382024-01-31T21:15:57.189Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1392024-01-31T21:15:57.218Z+ pfexec mkdir /opt/oxide/work
1402024-01-31T21:15:57.218Z+ pfexec chown build:build /opt/oxide/work
1412024-01-31T21:15:57.218Z+ cd /opt/oxide/work
1422024-01-31T21:15:57.218Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1432024-01-31T21:15:58.882ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1442024-01-31T21:15:58.882Zx out/mg-ddm-gz.tar, 63063552 bytes, 123171 tape blocks
1452024-01-31T21:15:58.882Zx out/omicron-sled-agent.tar, 122210816 bytes, 238693 tape blocks
1462024-01-31T21:15:58.882Zx out/oxlog.tar, 8382976 bytes, 16373 tape blocks
1472024-01-31T21:15:58.883Zx out/target/test, 75 bytes, 1 tape blocks
1482024-01-31T21:15:58.883Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1492024-01-31T21:15:58.884Zx package-manifest.toml, 23732 bytes, 47 tape blocks
1502024-01-31T21:15:58.884Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1512024-01-31T21:15:58.884Zx target/release/omicron-package, 44586080 bytes, 87083 tape blocks
1522024-01-31T21:15:58.884Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1532024-01-31T21:15:58.884Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1542024-01-31T21:15:58.884Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1552024-01-31T21:15:58.884Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1562024-01-31T21:15:58.884Zx tools/scrimlet/softnpu-init.sh, 2149 bytes, 5 tape blocks
1572024-01-31T21:15:58.884Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1582024-01-31T21:15:58.884Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1592024-01-31T21:15:59.348Z
1602024-01-31T21:15:59.348Zreal 1.673423070
1612024-01-31T21:15:59.349Zuser 1.504859533
1622024-01-31T21:15:59.349Zsys 0.298213669
1632024-01-31T21:15:59.349Ztrap 0.000079020
1642024-01-31T21:15:59.349Ztflt 0.000000000
1652024-01-31T21:15:59.349Zdflt 0.000000000
1662024-01-31T21:15:59.350Zkflt 0.000000000
1672024-01-31T21:15:59.350Zlock 0.000000000
1682024-01-31T21:15:59.350Zslp 1.504805209
1692024-01-31T21:15:59.350Zlat 0.036707205
1702024-01-31T21:15:59.351Zstop 0.000129830
1712024-01-31T21:15:59.351Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1722024-01-31T21:16:00.068Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1732024-01-31T21:16:00.094Z+ mkdir tests
1742024-01-31T21:16:00.094Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1752024-01-31T21:16:00.094Z+ ptime -m gunzip
1762024-01-31T21:16:00.094Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-485102d0e6124cf0
1772024-01-31T21:16:00.330Z
1782024-01-31T21:16:00.365Zreal 0.238765323
1792024-01-31T21:16:00.365Zuser 0.221190343
1802024-01-31T21:16:00.366Zsys 0.017310977
1812024-01-31T21:16:00.366Ztrap 0.000021740
1822024-01-31T21:16:00.366Ztflt 0.000000000
1832024-01-31T21:16:00.366Zdflt 0.000000000
1842024-01-31T21:16:00.366Zkflt 0.000000000
1852024-01-31T21:16:00.366Zlock 0.000000000
1862024-01-31T21:16:00.366Zslp 0.000021410
1872024-01-31T21:16:00.366Zlat 0.000023030
1882024-01-31T21:16:00.366Zstop 0.000131361
1892024-01-31T21:16:00.366Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-485102d0e6124cf0
1902024-01-31T21:16:00.366Z+ chmod a+x tests/bootstrap-485102d0e6124cf0
1912024-01-31T21:16:00.366Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1922024-01-31T21:16:00.366Z+ ptime -m gunzip
1932024-01-31T21:16:00.366Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1942024-01-31T21:16:02.879Z
1952024-01-31T21:16:02.926Zreal 2.512212985
1962024-01-31T21:16:02.926Zuser 2.347966836
1972024-01-31T21:16:02.926Zsys 0.163810255
1982024-01-31T21:16:02.926Ztrap 0.000193592
1992024-01-31T21:16:02.926Ztflt 0.000000000
2002024-01-31T21:16:02.926Zdflt 0.000000000
2012024-01-31T21:16:02.926Zkflt 0.000000000
2022024-01-31T21:16:02.926Zlock 0.000000000
2032024-01-31T21:16:02.926Zslp 0.000021509
2042024-01-31T21:16:02.927Zlat 0.000023610
2052024-01-31T21:16:02.927Zstop 0.000128600
2062024-01-31T21:16:02.927Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
2072024-01-31T21:16:02.927Z+ chmod a+x tests/bootstrap
2082024-01-31T21:16:02.927Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
2092024-01-31T21:16:02.927Z+ ptime -m gunzip
2102024-01-31T21:16:02.927Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8616f2942d631494
2112024-01-31T21:16:05.479Z
2122024-01-31T21:16:05.513Zreal 2.610838900
2132024-01-31T21:16:05.513Zuser 2.463917275
2142024-01-31T21:16:05.513Zsys 0.146488503
2152024-01-31T21:16:05.513Ztrap 0.000199622
2162024-01-31T21:16:05.513Ztflt 0.000000000
2172024-01-31T21:16:05.514Zdflt 0.000000000
2182024-01-31T21:16:05.514Zkflt 0.000000000
2192024-01-31T21:16:05.514Zlock 0.000000000
2202024-01-31T21:16:05.514Zslp 0.000021660
2212024-01-31T21:16:05.514Zlat 0.000023140
2222024-01-31T21:16:05.514Zstop 0.000129301
2232024-01-31T21:16:05.515Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8616f2942d631494
2242024-01-31T21:16:05.515Z+ chmod a+x tests/end_to_end_tests-8616f2942d631494
2252024-01-31T21:16:05.515Z+ bmat address ls
2262024-01-31T21:16:05.515ZNAME CIDR FIRST LAST COUNT
2272024-01-31T21:16:05.515Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2282024-01-31T21:16:05.515Z++ bmat address ls -f extra -Ho first,last,count
2292024-01-31T21:16:05.516Z+ set -- 10.151.2.164 10.151.2.227 64
2302024-01-31T21:16:05.516Z+ EXTRA_IP_START=10.151.2.164
2312024-01-31T21:16:05.516Z+ EXTRA_IP_END=10.151.2.227
2322024-01-31T21:16:05.517Z+ EXTRA_IP_COUNT=64
2332024-01-31T21:16:05.517Z+ (( EXTRA_IP_COUNT >= 32 ))
2342024-01-31T21:16:05.517Z+ EXTRA_IP_BASE=10.151.2
2352024-01-31T21:16:05.517Z+ EXTRA_IP_FOCTET=164
2362024-01-31T21:16:05.517Z+ EXTRA_IP_LOCTET=227
2372024-01-31T21:16:05.517Z+ SERVICE_IP_POOL_START=10.151.2.164
2382024-01-31T21:16:05.517Z+ SERVICE_IP_POOL_END=10.151.2.173
2392024-01-31T21:16:05.518Z+ DNS_IP1=10.151.2.164
2402024-01-31T21:16:05.518Z+ DNS_IP2=10.151.2.165
2412024-01-31T21:16:05.518Z+ UPLINK_IP=10.151.2.174
2422024-01-31T21:16:05.518Z+ IPPOOL_START=10.151.2.175
2432024-01-31T21:16:05.518Z+ IPPOOL_END=10.151.2.227
2442024-01-31T21:16:05.519Z++ ipadm show-addr -po type,addr
2452024-01-31T21:16:05.519Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2462024-01-31T21:16:05.519Z+ GATEWAY_IP=10.151.2.100
2472024-01-31T21:16:05.519Z+ [[ -n 10.151.2.100 ]]
2482024-01-31T21:16:05.519Z+ ping -s 10.151.2.100 56 1
2492024-01-31T21:16:05.564ZPING 10.151.2.100: 56 data bytes
2502024-01-31T21:16:05.564Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.010 ms
2512024-01-31T21:16:05.566Z
2522024-01-31T21:16:05.566Z----10.151.2.100 PING Statistics----
2532024-01-31T21:16:05.566Z1 packets transmitted, 1 packets received, 0% packet loss
2542024-01-31T21:16:05.566Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2552024-01-31T21:16:05.566Z++ arp -an
2562024-01-31T21:16:05.566Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2572024-01-31T21:16:05.567Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2582024-01-31T21:16:05.567Z+ routeadm -e ipv4-forwarding -u
2592024-01-31T21:16:05.641Z+ PXA_START=10.151.2.164
2602024-01-31T21:16:05.641Z+ PXA_END=10.151.2.227
2612024-01-31T21:16:05.670Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2622024-01-31T21:16:05.670Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2632024-01-31T21:16:05.840Z+ ZPOOL_VDEV_DIR=/scratch
2642024-01-31T21:16:05.841Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2652024-01-31T21:16:05.877Z+++ dirname ./tools/create_virtual_hardware.sh
2662024-01-31T21:16:05.877Z++ cd ./tools
2672024-01-31T21:16:05.878Z++ pwd
2682024-01-31T21:16:05.878Z+ SOURCE_DIR=/opt/oxide/work/tools
2692024-01-31T21:16:05.878Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2702024-01-31T21:16:05.878Z+ SOFTNPU_MODE=zone
2712024-01-31T21:16:05.878Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2722024-01-31T21:16:05.878Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2732024-01-31T21:16:05.878Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2742024-01-31T21:16:05.879Z++ dladm show-phys -p -o LINK
2752024-01-31T21:16:05.880Z++ head -1
2762024-01-31T21:16:05.880Z+ PHYSICAL_LINK=igb0
2772024-01-31T21:16:05.880Z+ echo 'Using igb0 as physical link'
2782024-01-31T21:16:05.880ZUsing igb0 as physical link
2792024-01-31T21:16:05.880Z+ ensure_run_as_root
2802024-01-31T21:16:05.880Z++ id -u
2812024-01-31T21:16:05.881Z+ [[ 0 -ne 0 ]]
2822024-01-31T21:16:05.881Z+ ensure_zpools
2832024-01-31T21:16:05.881Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2842024-01-31T21:16:05.881Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2852024-01-31T21:16:05.881Z+ readarray -t ZPOOLS
2862024-01-31T21:16:05.881Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2872024-01-31T21:16:05.881Z++ sed 's/[ ",]//g'
2882024-01-31T21:16:05.912Z+ for ZPOOL in "${ZPOOLS[@]}"
2892024-01-31T21:16:05.915Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2902024-01-31T21:16:05.915Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2912024-01-31T21:16:05.915ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2922024-01-31T21:16:05.915Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2932024-01-31T21:16:05.916Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2942024-01-31T21:16:05.916Z0+0 records in
2952024-01-31T21:16:05.916Z0+0 records out
2962024-01-31T21:16:05.916Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2972024-01-31T21:16:05.916Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2982024-01-31T21:16:05.916Z+ set +x
2992024-01-31T21:16:05.916ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3002024-01-31T21:16:05.916Z++ zpool list -o name
3012024-01-31T21:16:05.916Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
3022024-01-31T21:16:05.916Z+ [[ -z '' ]]
3032024-01-31T21:16:05.916Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
3042024-01-31T21:16:05.951Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
3052024-01-31T21:16:05.951Z+ set +x
3062024-01-31T21:16:05.952ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3072024-01-31T21:16:05.952Z+ for ZPOOL in "${ZPOOLS[@]}"
3082024-01-31T21:16:05.952Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3092024-01-31T21:16:05.952ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3102024-01-31T21:16:05.952Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3112024-01-31T21:16:05.952Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3122024-01-31T21:16:05.952Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3132024-01-31T21:16:05.952Z0+0 records in
3142024-01-31T21:16:05.952Z0+0 records out
3152024-01-31T21:16:05.952Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3162024-01-31T21:16:05.952Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3172024-01-31T21:16:05.952Z+ set +x
3182024-01-31T21:16:05.953ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3192024-01-31T21:16:05.953Z++ zpool list -o name
3202024-01-31T21:16:05.953Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3212024-01-31T21:16:05.953Z+ [[ -z '' ]]
3222024-01-31T21:16:05.953Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3232024-01-31T21:16:05.984Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3242024-01-31T21:16:05.984Z+ set +x
3252024-01-31T21:16:05.984ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3262024-01-31T21:16:05.984Z+ for ZPOOL in "${ZPOOLS[@]}"
3272024-01-31T21:16:05.984Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3282024-01-31T21:16:05.984ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3292024-01-31T21:16:05.984Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3302024-01-31T21:16:05.984Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3312024-01-31T21:16:05.984Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3322024-01-31T21:16:05.985Z0+0 records in
3332024-01-31T21:16:05.985Z0+0 records out
3342024-01-31T21:16:05.985Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3352024-01-31T21:16:05.985Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3362024-01-31T21:16:05.985Z+ set +x
3372024-01-31T21:16:05.985ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3382024-01-31T21:16:05.985Z++ zpool list -o name
3392024-01-31T21:16:05.985Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3402024-01-31T21:16:05.985Z+ [[ -z '' ]]
3412024-01-31T21:16:05.985Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3422024-01-31T21:16:06.026Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3432024-01-31T21:16:06.026Z+ set +x
3442024-01-31T21:16:06.027ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3452024-01-31T21:16:06.027Z+ for ZPOOL in "${ZPOOLS[@]}"
3462024-01-31T21:16:06.027Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3472024-01-31T21:16:06.027ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3482024-01-31T21:16:06.027Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3492024-01-31T21:16:06.027Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3502024-01-31T21:16:06.027Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3512024-01-31T21:16:06.033Z0+0 records in
3522024-01-31T21:16:06.033Z0+0 records out
3532024-01-31T21:16:06.033Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3542024-01-31T21:16:06.033Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3552024-01-31T21:16:06.033Z+ set +x
3562024-01-31T21:16:06.034ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3572024-01-31T21:16:06.034Z++ zpool list -o name
3582024-01-31T21:16:06.034Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3592024-01-31T21:16:06.034Z+ [[ -z '' ]]
3602024-01-31T21:16:06.034Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3612024-01-31T21:16:06.074Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3622024-01-31T21:16:06.074Z+ set +x
3632024-01-31T21:16:06.074ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3642024-01-31T21:16:06.074Z+ for ZPOOL in "${ZPOOLS[@]}"
3652024-01-31T21:16:06.074Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3662024-01-31T21:16:06.075ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3672024-01-31T21:16:06.075Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3682024-01-31T21:16:06.075Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3692024-01-31T21:16:06.075Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3702024-01-31T21:16:06.075Z0+0 records in
3712024-01-31T21:16:06.075Z0+0 records out
3722024-01-31T21:16:06.075Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3732024-01-31T21:16:06.075Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3742024-01-31T21:16:06.075Z+ set +x
3752024-01-31T21:16:06.075ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3762024-01-31T21:16:06.076Z++ zpool list -o name
3772024-01-31T21:16:06.076Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3782024-01-31T21:16:06.076Z+ [[ -z '' ]]
3792024-01-31T21:16:06.076Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3802024-01-31T21:16:06.128Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3812024-01-31T21:16:06.128Z+ set +x
3822024-01-31T21:16:06.128ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3832024-01-31T21:16:06.129Z+ for ZPOOL in "${ZPOOLS[@]}"
3842024-01-31T21:16:06.129Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3852024-01-31T21:16:06.129ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3862024-01-31T21:16:06.129Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872024-01-31T21:16:06.129Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882024-01-31T21:16:06.130Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892024-01-31T21:16:06.130Z0+0 records in
3902024-01-31T21:16:06.130Z0+0 records out
3912024-01-31T21:16:06.130Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922024-01-31T21:16:06.130Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932024-01-31T21:16:06.130Z+ set +x
3942024-01-31T21:16:06.131ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3952024-01-31T21:16:06.131Z++ zpool list -o name
3962024-01-31T21:16:06.131Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972024-01-31T21:16:06.131Z+ [[ -z '' ]]
3982024-01-31T21:16:06.131Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992024-01-31T21:16:06.166Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002024-01-31T21:16:06.167Z+ set +x
4012024-01-31T21:16:06.167ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
4022024-01-31T21:16:06.167Z+ for ZPOOL in "${ZPOOLS[@]}"
4032024-01-31T21:16:06.167Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4042024-01-31T21:16:06.167ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
4052024-01-31T21:16:06.167Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4062024-01-31T21:16:06.168Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4072024-01-31T21:16:06.168Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4082024-01-31T21:16:06.168Z0+0 records in
4092024-01-31T21:16:06.168Z0+0 records out
4102024-01-31T21:16:06.168Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4112024-01-31T21:16:06.168Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4122024-01-31T21:16:06.168Z+ set +x
4132024-01-31T21:16:06.169ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
4142024-01-31T21:16:06.169Z++ zpool list -o name
4152024-01-31T21:16:06.169Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4162024-01-31T21:16:06.169Z+ [[ -z '' ]]
4172024-01-31T21:16:06.169Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4182024-01-31T21:16:06.201Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4192024-01-31T21:16:06.201Z+ set +x
4202024-01-31T21:16:06.201ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4212024-01-31T21:16:06.201Z+ for ZPOOL in "${ZPOOLS[@]}"
4222024-01-31T21:16:06.201Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4232024-01-31T21:16:06.202ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4242024-01-31T21:16:06.202Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4252024-01-31T21:16:06.202Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4262024-01-31T21:16:06.202Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4272024-01-31T21:16:06.202Z0+0 records in
4282024-01-31T21:16:06.203Z0+0 records out
4292024-01-31T21:16:06.203Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4302024-01-31T21:16:06.203Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4312024-01-31T21:16:06.203Z+ set +x
4322024-01-31T21:16:06.203ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4332024-01-31T21:16:06.203Z++ zpool list -o name
4342024-01-31T21:16:06.204Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4352024-01-31T21:16:06.204Z+ [[ -z '' ]]
4362024-01-31T21:16:06.204Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4372024-01-31T21:16:06.239Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4382024-01-31T21:16:06.239Z+ set +x
4392024-01-31T21:16:06.240ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4402024-01-31T21:16:06.240Z+ for ZPOOL in "${ZPOOLS[@]}"
4412024-01-31T21:16:06.240Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4422024-01-31T21:16:06.240ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4432024-01-31T21:16:06.240Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4442024-01-31T21:16:06.240Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4452024-01-31T21:16:06.240Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4462024-01-31T21:16:06.240Z0+0 records in
4472024-01-31T21:16:06.240Z0+0 records out
4482024-01-31T21:16:06.241Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4492024-01-31T21:16:06.241Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4502024-01-31T21:16:06.241Z+ set +x
4512024-01-31T21:16:06.241ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4522024-01-31T21:16:06.241Z++ zpool list -o name
4532024-01-31T21:16:06.241Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4542024-01-31T21:16:06.241Z+ [[ -z '' ]]
4552024-01-31T21:16:06.241Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4562024-01-31T21:16:06.274Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4572024-01-31T21:16:06.274Z+ set +x
4582024-01-31T21:16:06.274ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4592024-01-31T21:16:06.275Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4602024-01-31T21:16:06.275Z+ readarray -t ZPOOLS
4612024-01-31T21:16:06.275Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4622024-01-31T21:16:06.275Z++ sed 's/[ ",]//g'
4632024-01-31T21:16:06.275Z+ for ZPOOL in "${ZPOOLS[@]}"
4642024-01-31T21:16:06.275Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4652024-01-31T21:16:06.275ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4662024-01-31T21:16:06.275Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4672024-01-31T21:16:06.276Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4682024-01-31T21:16:06.276Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4692024-01-31T21:16:06.276Z0+0 records in
4702024-01-31T21:16:06.276Z0+0 records out
4712024-01-31T21:16:06.276Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4722024-01-31T21:16:06.276Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4732024-01-31T21:16:06.276Z+ set +x
4742024-01-31T21:16:06.277ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4752024-01-31T21:16:06.277Z++ zpool list -o name
4762024-01-31T21:16:06.277Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4772024-01-31T21:16:06.277Z+ [[ -z '' ]]
4782024-01-31T21:16:06.278Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4792024-01-31T21:16:06.310Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4802024-01-31T21:16:06.310Z+ set +x
4812024-01-31T21:16:06.310ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4822024-01-31T21:16:06.311Z+ for ZPOOL in "${ZPOOLS[@]}"
4832024-01-31T21:16:06.311Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4842024-01-31T21:16:06.311ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4852024-01-31T21:16:06.311Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4862024-01-31T21:16:06.311Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4872024-01-31T21:16:06.311Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4882024-01-31T21:16:06.312Z0+0 records in
4892024-01-31T21:16:06.312Z0+0 records out
4902024-01-31T21:16:06.312Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4912024-01-31T21:16:06.312Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4922024-01-31T21:16:06.312Z+ set +x
4932024-01-31T21:16:06.313ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4942024-01-31T21:16:06.313Z++ zpool list -o name
4952024-01-31T21:16:06.313Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4962024-01-31T21:16:06.758Z+ [[ -z '' ]]
4972024-01-31T21:16:06.758Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4982024-01-31T21:16:06.758Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4992024-01-31T21:16:06.759Z+ set +x
5002024-01-31T21:16:06.759ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
5012024-01-31T21:16:06.759Z+ [[ zone == \z\o\n\e ]]
5022024-01-31T21:16:06.759Z+ ensure_simulated_links igb0
5032024-01-31T21:16:06.759Z+ local PHYSICAL_LINK=igb0
5042024-01-31T21:16:06.759Z+ INDICES=("0" "1")
5052024-01-31T21:16:06.759Z+ for I in "${INDICES[@]}"
5062024-01-31T21:16:06.759Z++ get_simnet_name_if_exists net0
5072024-01-31T21:16:06.759Z++ dladm show-simnet -p -o LINK net0
5082024-01-31T21:16:06.793Z++ echo ''
5092024-01-31T21:16:06.793Z+ [[ -z '' ]]
5102024-01-31T21:16:06.793Z+ dladm create-simnet -t net0
5112024-01-31T21:16:06.793Z+ dladm create-simnet -t sc0_0
5122024-01-31T21:16:06.794Z+ dladm modify-simnet -t -p net0 sc0_0
5132024-01-31T21:16:06.794Z+ dladm set-linkprop -p mtu=9000 sc0_0
5142024-01-31T21:16:06.794Z+ success 'Simnet net0/sc0_0 exists'
5152024-01-31T21:16:06.794Z+ set +x
5162024-01-31T21:16:06.795ZSimnet net0/sc0_0 exists
5172024-01-31T21:16:06.795Z+ for I in "${INDICES[@]}"
5182024-01-31T21:16:06.795Z++ get_simnet_name_if_exists net1
5192024-01-31T21:16:06.795Z++ dladm show-simnet -p -o LINK net1
5202024-01-31T21:16:06.795Z++ echo ''
5212024-01-31T21:16:06.796Z+ [[ -z '' ]]
5222024-01-31T21:16:06.796Z+ dladm create-simnet -t net1
5232024-01-31T21:16:06.796Z+ dladm create-simnet -t sc1_0
5242024-01-31T21:16:06.796Z+ dladm modify-simnet -t -p net1 sc1_0
5252024-01-31T21:16:06.796Z+ dladm set-linkprop -p mtu=9000 sc1_0
5262024-01-31T21:16:06.796Z+ success 'Simnet net1/sc1_0 exists'
5272024-01-31T21:16:06.796Z+ set +x
5282024-01-31T21:16:06.796ZSimnet net1/sc1_0 exists
5292024-01-31T21:16:06.796Z++ get_vnic_name_if_exists sc0_1
5302024-01-31T21:16:06.796Z++ dladm show-vnic -p -o LINK sc0_1
5312024-01-31T21:16:06.796Z++ echo ''
5322024-01-31T21:16:06.796Z+ [[ -z '' ]]
5332024-01-31T21:16:06.797Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5342024-01-31T21:16:06.797Z+ [[ -v PROMISC_FILT_OFF ]]
5352024-01-31T21:16:06.797Z+ success 'Vnic sc0_1 exists'
5362024-01-31T21:16:06.797Z+ set +x
5372024-01-31T21:16:06.797ZVnic sc0_1 exists
5382024-01-31T21:16:06.797Z+ warn_if_no_proxy_arp
5392024-01-31T21:16:06.797Z+ [[ -v PXA_START ]]
5402024-01-31T21:16:06.797Z+ [[ -v PXA_END ]]
5412024-01-31T21:16:06.797Z+ ensure_softnpu_zone
5422024-01-31T21:16:06.797Z+ zoneadm list
5432024-01-31T21:16:06.798Z+ grep -q sidecar_softnpu
5442024-01-31T21:16:06.798Z+ [[ -f out/npuzone/npuzone ]]
5452024-01-31T21:16:06.798Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit e3ea4b495ba0a71801ded0776ae4bbd31df57e26 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5462024-01-31T21:16:06.798Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5472024-01-31T21:16:18.479Zdownloading softnpu ...done
5482024-01-31T21:16:18.505Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/libsidecar_lite.so.sha256.txt
5492024-01-31T21:16:31.631Zdownloading asic_program.so ...done
5502024-01-31T21:16:31.666Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/scadm.sha256.txt
5512024-01-31T21:16:46.560Zdownloading scadm ...done
5522024-01-31T21:16:46.594Zcreating port tfportrear0_0 for sc0_0
5532024-01-31T21:16:46.594Zcreating port tfportqsfp0_0 for sc0_1
5542024-01-31T21:16:46.594Zsoftnpu zone setup
5552024-01-31T21:16:46.594Zconfigure zone
5562024-01-31T21:16:46.628Zinstall zone
5572024-01-31T21:16:47.653Zboot zone
5582024-01-31T21:16:50.769Z[sidecar_softnpu] svcs milestone/network
5592024-01-31T21:16:50.817ZSTATE STIME FMRI
5602024-01-31T21:16:50.817Zonline 21:16:48 svc:/milestone/network:default
5612024-01-31T21:16:50.818Z[sidecar_softnpu] /softnpu/npu start
5622024-01-31T21:16:50.864Z
5632024-01-31T21:16:50.864Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5642024-01-31T21:16:50.865Z+ GATEWAY_IP=10.151.2.100
5652024-01-31T21:16:50.865Z+ echo 'Using 10.151.2.100 as gateway ip'
5662024-01-31T21:16:50.866ZUsing 10.151.2.100 as gateway ip
5672024-01-31T21:16:50.866Z+ [[ ! -v GATEWAY_MAC ]]
5682024-01-31T21:16:50.868Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5692024-01-31T21:16:50.869Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5702024-01-31T21:16:50.869Z+++ echo 18:c0:4d:0d:a0:2a
5712024-01-31T21:16:50.870Z+ COUNT=1
5722024-01-31T21:16:50.870Z+ [[ 1 -ne 1 ]]
5732024-01-31T21:16:50.870Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5742024-01-31T21:16:50.870ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5752024-01-31T21:16:50.870Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5762024-01-31T21:16:50.871Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5772024-01-31T21:16:50.941Z+ PXA_MAC=a8:e1:de:01:70:1d
5782024-01-31T21:16:50.941Z+ [[ -v PXA_START ]]
5792024-01-31T21:16:50.941Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5802024-01-31T21:16:50.942Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5812024-01-31T21:16:50.942Z+ z_scadm dump-state
5822024-01-31T21:16:50.942Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5832024-01-31T21:16:50.942Zlocal v6:
5842024-01-31T21:16:50.942Zlocal v4:
5852024-01-31T21:16:50.942Zrouter v6:
5862024-01-31T21:16:50.942Zrouter v4_idx:
5872024-01-31T21:16:50.943Zrouter v4_routes:
5882024-01-31T21:16:50.943Zresolver v4:
5892024-01-31T21:16:50.943Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5902024-01-31T21:16:50.943Zresolver v6:
5912024-01-31T21:16:50.943Znat v4:
5922024-01-31T21:16:50.943Znat_v6:
5932024-01-31T21:16:50.943Zport_mac:
5942024-01-31T21:16:50.943Zproxy arp:
5952024-01-31T21:16:50.943Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5962024-01-31T21:16:50.943Z+ success 'softnpu zone exists'
5972024-01-31T21:16:50.943Z+ set +x
5982024-01-31T21:16:50.943Zsoftnpu zone exists
5992024-01-31T21:16:50.944Z
6002024-01-31T21:16:50.944Zreal 45.069040075
6012024-01-31T21:16:50.944Zuser 0.690702000
6022024-01-31T21:16:50.944Zsys 1.540106604
6032024-01-31T21:16:50.944Ztrap 0.000088475
6042024-01-31T21:16:50.944Ztflt 0.000156684
6052024-01-31T21:16:50.944Zdflt 0.001239880
6062024-01-31T21:16:50.944Zkflt 0.000000000
6072024-01-31T21:16:50.945Zlock 35:28.417889459
6082024-01-31T21:16:50.945Zslp 2:14.962544752
6092024-01-31T21:16:50.945Zlat 0.069851137
6102024-01-31T21:16:50.945Zstop 0.367426308
6112024-01-31T21:16:50.945Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
6122024-01-31T21:16:50.980Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
6132024-01-31T21:16:50.980Z+ SILO_NAME=recovery
6142024-01-31T21:16:50.981Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
6152024-01-31T21:16:50.981Z+ EXTERNAL_DNS_DOMAIN=oxide.test
6162024-01-31T21:16:50.981Z+ sed -i~ '
6172024-01-31T21:16:50.981Z /^external_dns_ips/c\
6182024-01-31T21:16:50.982Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6192024-01-31T21:16:50.982Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6202024-01-31T21:16:50.982Z /^first/c\
6212024-01-31T21:16:50.982Zfirst = "10.151.2.164"
6222024-01-31T21:16:50.983Z /^last/c\
6232024-01-31T21:16:50.983Zlast = "10.151.2.173"
6242024-01-31T21:16:50.983Z }
6252024-01-31T21:16:50.983Z /^\[rack_network_config/,/^$/ {
6262024-01-31T21:16:50.984Z /^infra_ip_first/c\
6272024-01-31T21:16:50.984Zinfra_ip_first = "10.151.2.174"
6282024-01-31T21:16:50.984Z /^infra_ip_last/c\
6292024-01-31T21:16:50.984Zinfra_ip_last = "10.151.2.174"
6302024-01-31T21:16:50.984Z }
6312024-01-31T21:16:50.985Z /^\[\[rack_network_config.ports/,/^$/ {
6322024-01-31T21:16:50.985Z /^routes/c\
6332024-01-31T21:16:50.985Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6342024-01-31T21:16:50.985Z /^addresses/c\
6352024-01-31T21:16:50.985Zaddresses = \["10.151.2.174/24"\]
6362024-01-31T21:16:50.990Z }
6372024-01-31T21:16:50.990Z' pkg/config-rss.toml
6382024-01-31T21:16:50.990Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6392024-01-31T21:16:50.990Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6402024-01-31T21:16:50.990Z+++ pkg/config-rss.toml Wed Jan 31 21:16:50 2024
6412024-01-31T21:16:50.991Z@@ -41,7 +41,7 @@
6422024-01-31T21:16:50.991Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6432024-01-31T21:16:50.991Z # must be contained in one of the "internal services" IP Pool ranges listed
6442024-01-31T21:16:50.992Z # below.
6452024-01-31T21:16:50.992Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6462024-01-31T21:16:50.992Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6472024-01-31T21:16:50.992Z
6482024-01-31T21:16:50.992Z # Initial TLS certificates for the external API
6492024-01-31T21:16:50.992Z #
6502024-01-31T21:16:50.993Z@@ -77,8 +77,8 @@
6512024-01-31T21:16:50.993Z #
6522024-01-31T21:16:50.993Z # For more on this and what to put here, see docs/how-to-run.adoc.
6532024-01-31T21:16:50.993Z [[internal_services_ip_pool_ranges]]
6542024-01-31T21:16:50.993Z-first = "192.168.1.20"
6552024-01-31T21:16:50.993Z-last = "192.168.1.29"
6562024-01-31T21:16:50.993Z+first = "10.151.2.164"
6572024-01-31T21:16:50.993Z+last = "10.151.2.173"
6582024-01-31T21:16:50.993Z
6592024-01-31T21:16:50.994Z # TODO - this configuration is subject to change going forward. Ultimately these
6602024-01-31T21:16:50.994Z # parameters should be provided to the control plane via wicket, but we need to
6612024-01-31T21:16:50.996Z@@ -92,8 +92,8 @@
6622024-01-31T21:16:50.996Z # A range of IP addresses used by Boundary Services on the external network. In
6632024-01-31T21:16:50.998Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6642024-01-31T21:16:50.998Z # With softnpu, only one address is used.
6652024-01-31T21:16:50.998Z-infra_ip_first = "192.168.1.30"
6662024-01-31T21:16:50.998Z-infra_ip_last = "192.168.1.30"
6672024-01-31T21:16:50.999Z+infra_ip_first = "10.151.2.174"
6682024-01-31T21:16:50.999Z+infra_ip_last = "10.151.2.174"
6692024-01-31T21:16:50.999Z
6702024-01-31T21:16:50.999Z # Configurations for BGP routers to run on the scrimlets.
6712024-01-31T21:16:50.999Z bgp = []
6722024-01-31T21:16:51.000Z@@ -101,9 +101,9 @@
6732024-01-31T21:16:51.000Z # You can configure multiple uplinks by repeating the following stanza
6742024-01-31T21:16:51.000Z [[rack_network_config.ports]]
6752024-01-31T21:16:51.000Z # Routes associated with this port.
6762024-01-31T21:16:51.000Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6772024-01-31T21:16:51.000Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6782024-01-31T21:16:51.004Z # Addresses associated with this port.
6792024-01-31T21:16:51.004Z-addresses = ["192.168.1.30/24"]
6802024-01-31T21:16:51.004Z+addresses = ["10.151.2.174/24"]
6812024-01-31T21:16:51.005Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6822024-01-31T21:16:51.005Z port = "qsfp0"
6832024-01-31T21:16:51.005Z # The speed of this port.
6842024-01-31T21:16:51.006Z+ true
6852024-01-31T21:16:51.006Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6862024-01-31T21:16:51.006Za pkg/config-rss.toml 7K
6872024-01-31T21:16:51.006Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6882024-01-31T21:16:51.006Z+ TLS_NAME=recovery.sys.oxide.test
6892024-01-31T21:16:51.007Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6902024-01-31T21:16:51.463Z.+.....+.........+.............+...+.....+.+........+...+....+...+...+.....+...+.+...+..+....+.....+....+...+...+......+...+..+...+...............+.+.....+....+..+.+..............+.+..+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+.+.....+.......+..+.+...............+.....+...+.......+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+.+...........+..........+......+..............+.+.........+.....+......+...............+...+..........+..+...+....+.....+...+....+...+.....+.+.........+...+........................+.........+............+..+.......+...+..+.+...............+.....+....+........+.+..............+......+....+.....+.+.....+.+...........+...+.......+............+......+.................+.........+...+...+.......+....................+.......+...........+....+.....+.........+..........+.....+.......+..+.+......+...........+....+...+.....+.......+......+..+.+......+...........+..........+..+...+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6912024-01-31T21:16:51.665Z.........+.........+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..........+...+......+.........+..+...+..........+...........+....+........+...+......+.+...+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+...........................+.....+....+......+...........+...+.+...+..+...+......+............+...+....+...............+........+...+.........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6922024-01-31T21:16:51.698Z-----
6932024-01-31T21:16:51.698Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6942024-01-31T21:16:51.699Za pkg/initial-tls-cert.pem 2K
6952024-01-31T21:16:51.699Za pkg/initial-tls-key.pem 4K
6962024-01-31T21:16:51.699Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6972024-01-31T21:16:51.699Z+ rmdir pkg
6982024-01-31T21:16:51.699Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6992024-01-31T21:16:51.700Z+ pfexec mkdir -p /usr/oxide
7002024-01-31T21:16:51.700Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
7012024-01-31T21:16:52.395Z+ pfexec chmod +x /usr/oxide/oxide
7022024-01-31T21:16:52.426Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
7032024-01-31T21:17:10.899Z+ OMICRON_NO_UNINSTALL=1
7042024-01-31T21:17:10.900Z+ ptime -m pfexec ./target/release/omicron-package -t test install
7052024-01-31T21:17:10.900ZJan 31 21:17:10.498 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
7062024-01-31T21:17:10.900ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
7072024-01-31T21:17:10.901ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7082024-01-31T21:17:10.901ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7092024-01-31T21:17:10.901ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7102024-01-31T21:17:10.904ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7112024-01-31T21:17:10.904ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7122024-01-31T21:17:10.905ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7132024-01-31T21:17:10.905ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/oxlog.tar, src: out/oxlog.tar
7142024-01-31T21:17:10.905ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7152024-01-31T21:17:10.906ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7162024-01-31T21:17:10.906ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7172024-01-31T21:17:10.906ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7182024-01-31T21:17:10.906ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter.tar.gz
7192024-01-31T21:17:10.906ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7202024-01-31T21:17:10.928ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
7212024-01-31T21:17:10.929ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7222024-01-31T21:17:10.929ZJan 31 21:17:10.500 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7232024-01-31T21:17:10.993ZJan 31 21:17:10.769 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7242024-01-31T21:17:11.043ZJan 31 21:17:10.808 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7252024-01-31T21:17:11.099ZJan 31 21:17:10.874 INFO Unpacking service tarball, service_path: /opt/oxide/oxlog, tar_path: /opt/oxide/oxlog.tar
7262024-01-31T21:17:11.193ZJan 31 21:17:10.879 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7272024-01-31T21:17:11.367Z
7282024-01-31T21:17:11.367Zreal 0.522638036
7292024-01-31T21:17:11.367Zuser 0.109052012
7302024-01-31T21:17:11.368Zsys 2.143251105
7312024-01-31T21:17:11.368Ztrap 0.000006929
7322024-01-31T21:17:11.368Ztflt 0.000013210
7332024-01-31T21:17:11.368Zdflt 0.001010101
7342024-01-31T21:17:11.368Zkflt 0.000000000
7352024-01-31T21:17:11.368Zlock 46.971595418
7362024-01-31T21:17:11.380Zslp 0.739416660
7372024-01-31T21:17:11.380Zlat 0.003118134
7382024-01-31T21:17:11.380Zstop 0.206401449
7392024-01-31T21:17:11.381Z+ retry=0
7402024-01-31T21:17:11.381Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7412024-01-31T21:17:11.444Z+ [[ 0 -gt 30 ]]
7422024-01-31T21:17:11.444Z+ sleep 1
7432024-01-31T21:17:12.242Z+ retry=1
7442024-01-31T21:17:12.242Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7452024-01-31T21:17:12.331Z+ [[ 1 -gt 30 ]]
7462024-01-31T21:17:12.331Z+ sleep 1
7472024-01-31T21:17:13.275Z+ retry=2
7482024-01-31T21:17:13.276Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7492024-01-31T21:17:13.317Z+ [[ 2 -gt 30 ]]
7502024-01-31T21:17:13.318Z+ sleep 1
7512024-01-31T21:17:14.280Z+ retry=3
7522024-01-31T21:17:14.280Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7532024-01-31T21:17:14.323Z+ [[ 3 -gt 30 ]]
7542024-01-31T21:17:14.346Z+ sleep 1
7552024-01-31T21:17:15.294Z+ retry=4
7562024-01-31T21:17:15.294Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7572024-01-31T21:17:40.495Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7582024-01-31T21:17:40.562Zlocal v6:
7592024-01-31T21:17:40.562Zfd39:9611:fcc3:5163::1
7602024-01-31T21:17:40.562Zfe80::6e:44ff:feb9:e764
7612024-01-31T21:17:40.562Zlocal v4:
7622024-01-31T21:17:40.562Z10.151.2.174
7632024-01-31T21:17:40.562Zrouter v6:
7642024-01-31T21:17:40.562Zfd00:1122:3344:1::/64 -> fe80::78a9:83ff:fe7d:15ad (1)
7652024-01-31T21:17:40.563Zfd00:1122:3344:2::/64 -> fe80::78a9:83ff:fe7d:15ad (1)
7662024-01-31T21:17:40.563Zfdb0:18c0:4d0d:a02a::/64 -> fe80::78a9:83ff:fe7d:15ad (1)
7672024-01-31T21:17:40.563Zfd00:1122:3344:3::/64 -> fe80::78a9:83ff:fe7d:15ad (1)
7682024-01-31T21:17:40.563Zfd00:1122:3344:101::/64 -> fe80::78a9:83ff:fe7d:15ad (1)
7692024-01-31T21:17:40.564Zrouter v4_idx:
7702024-01-31T21:17:40.564Z0.0.0.0/0 -> 0
7712024-01-31T21:17:40.564Zrouter v4_routes:
7722024-01-31T21:17:40.565Z0 -> 10.151.2.100 (2)
7732024-01-31T21:17:40.565Zresolver v4:
7742024-01-31T21:17:40.565Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7752024-01-31T21:17:40.565Z10.151.2.174 -> 9a:6c:bc:58:22:fb
7762024-01-31T21:17:40.565Zresolver v6:
7772024-01-31T21:17:40.565Zfe80::6e:44ff:feb9:e764 -> 02:6e:44:b9:e7:64
7782024-01-31T21:17:40.565Zfe80::78a9:83ff:fe7d:15ad -> 7a:a9:83:7d:15:ad
7792024-01-31T21:17:40.565Znat v4:
7802024-01-31T21:17:40.565Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:fc:61
7812024-01-31T21:17:40.565Znat_v6:
7822024-01-31T21:17:40.565Zport_mac:
7832024-01-31T21:17:40.565Z1: a8:40:25:0e:15:e3
7842024-01-31T21:17:40.565Z1000: a8:40:25:0e:15:e2
7852024-01-31T21:17:40.565Zproxy arp:
7862024-01-31T21:17:40.565Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7872024-01-31T21:17:40.565Z+ retry=0
7882024-01-31T21:17:40.566Z+++ zoneadm list -n
7892024-01-31T21:17:40.566Z+++ grep oxz_ntp
7902024-01-31T21:17:40.566Z++ pfexec svcs -z oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b -Hostate oxide/ntp
7912024-01-31T21:17:40.566Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7922024-01-31T21:17:40.566Z++ true
7932024-01-31T21:17:40.566Z+ [[ '' != online ]]
7942024-01-31T21:17:40.566Z+ [[ 0 -gt 60 ]]
7952024-01-31T21:17:40.566Z+ sleep 1
7962024-01-31T21:17:41.543Z+ retry=1
7972024-01-31T21:17:41.593Z+++ zoneadm list -n
7982024-01-31T21:17:41.593Z+++ grep oxz_ntp
7992024-01-31T21:17:41.593Z++ pfexec svcs -z oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b -Hostate oxide/ntp
8002024-01-31T21:17:41.593Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8012024-01-31T21:17:41.594Z++ true
8022024-01-31T21:17:41.594Z+ [[ '' != online ]]
8032024-01-31T21:17:41.594Z+ [[ 1 -gt 60 ]]
8042024-01-31T21:17:41.594Z+ sleep 1
8052024-01-31T21:17:42.579Z+ retry=2
8062024-01-31T21:17:42.632Z+++ zoneadm list -n
8072024-01-31T21:17:42.633Z+++ grep oxz_ntp
8082024-01-31T21:17:42.633Z++ pfexec svcs -z oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b -Hostate oxide/ntp
8092024-01-31T21:17:42.633Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8102024-01-31T21:17:42.633Z++ true
8112024-01-31T21:17:42.633Z+ [[ '' != online ]]
8122024-01-31T21:17:42.633Z+ [[ 2 -gt 60 ]]
8132024-01-31T21:17:42.633Z+ sleep 1
 
8142024-01-31T21:17:42.906ZError: No response to keepalive - Terminating session
 
8152024-01-31T21:17:43.601Z+ retry=3
8162024-01-31T21:17:44.318Z+++ zoneadm list -n
8172024-01-31T21:17:44.319Z+++ grep oxz_ntp
8182024-01-31T21:17:44.319Z++ pfexec svcs -z oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b -Hostate oxide/ntp
8192024-01-31T21:17:44.319Z+ [[ disabled != online ]]
8202024-01-31T21:17:44.319Z+ [[ 3 -gt 60 ]]
8212024-01-31T21:17:44.320Z+ sleep 1
8222024-01-31T21:17:44.621Z+ retry=4
8232024-01-31T21:17:44.659Z+++ zoneadm list -n
8242024-01-31T21:17:44.659Z+++ grep oxz_ntp
8252024-01-31T21:17:44.659Z++ pfexec svcs -z oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b -Hostate oxide/ntp
8262024-01-31T21:17:44.659Z+ [[ disabled != online ]]
8272024-01-31T21:17:44.659Z+ [[ 4 -gt 60 ]]
8282024-01-31T21:17:44.659Z+ sleep 1
8292024-01-31T21:17:45.638Z+ retry=5
8302024-01-31T21:17:45.684Z+++ zoneadm list -n
8312024-01-31T21:17:45.684Z+++ grep oxz_ntp
8322024-01-31T21:17:45.684Z++ pfexec svcs -z oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b -Hostate oxide/ntp
8332024-01-31T21:17:45.685Z+ [[ disabled != online ]]
8342024-01-31T21:17:45.685Z+ [[ 5 -gt 60 ]]
8352024-01-31T21:17:45.685Z+ sleep 1
8362024-01-31T21:17:46.670Z+ retry=6
8372024-01-31T21:17:46.703Z+++ zoneadm list -n
8382024-01-31T21:17:46.703Z+++ grep oxz_ntp
8392024-01-31T21:17:46.703Z++ pfexec svcs -z oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b -Hostate oxide/ntp
8402024-01-31T21:17:46.703Z+ [[ disabled != online ]]
8412024-01-31T21:17:46.703Z+ [[ 6 -gt 60 ]]
8422024-01-31T21:17:46.706Z+ sleep 1
8432024-01-31T21:17:47.688Z+ retry=7
8442024-01-31T21:17:47.723Z+++ zoneadm list -n
8452024-01-31T21:17:47.724Z+++ grep oxz_ntp
8462024-01-31T21:17:47.724Z++ pfexec svcs -z oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b -Hostate oxide/ntp
8472024-01-31T21:17:47.726Z+ [[ disabled != online ]]
8482024-01-31T21:17:47.726Z+ [[ 7 -gt 60 ]]
8492024-01-31T21:17:47.726Z+ sleep 1
8502024-01-31T21:17:48.712Z+ retry=8
8512024-01-31T21:17:48.746Z+++ zoneadm list -n
8522024-01-31T21:17:48.746Z+++ grep oxz_ntp
8532024-01-31T21:17:48.747Z++ pfexec svcs -z oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b -Hostate oxide/ntp
8542024-01-31T21:17:48.748Z+ [[ disabled != online ]]
8552024-01-31T21:17:48.748Z+ [[ 8 -gt 60 ]]
8562024-01-31T21:17:48.748Z+ sleep 1
8572024-01-31T21:17:49.725Z+ retry=9
8582024-01-31T21:17:49.758Z+++ zoneadm list -n
8592024-01-31T21:17:49.759Z+++ grep oxz_ntp
8602024-01-31T21:17:49.759Z++ pfexec svcs -z oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b -Hostate oxide/ntp
8612024-01-31T21:17:49.759Z+ [[ disabled != online ]]
8622024-01-31T21:17:49.759Z+ [[ 9 -gt 60 ]]
8632024-01-31T21:17:49.759Z+ sleep 1
8642024-01-31T21:17:50.744Z+ retry=10
8652024-01-31T21:17:50.774Z+++ zoneadm list -n
8662024-01-31T21:17:50.774Z+++ grep oxz_ntp
8672024-01-31T21:17:50.774Z++ pfexec svcs -z oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b -Hostate oxide/ntp
8682024-01-31T21:17:50.774Z+ [[ disabled != online ]]
8692024-01-31T21:17:50.774Z+ [[ 10 -gt 60 ]]
8702024-01-31T21:17:50.774Z+ sleep 1
8712024-01-31T21:17:51.775Z+ retry=11
8722024-01-31T21:17:51.813Z+++ zoneadm list -n
8732024-01-31T21:17:51.813Z+++ grep oxz_ntp
8742024-01-31T21:17:51.814Z++ pfexec svcs -z oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b -Hostate oxide/ntp
8752024-01-31T21:17:51.814Z+ [[ online != online ]]
8762024-01-31T21:17:51.814Z+ echo 'Waited for chrony: 11s'
8772024-01-31T21:17:51.814ZWaited for chrony: 11s
8782024-01-31T21:17:51.814Z+ export RUST_BACKTRACE=1
8792024-01-31T21:17:51.814Z+ RUST_BACKTRACE=1
8802024-01-31T21:17:51.814Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8812024-01-31T21:17:51.814Z++ ./tests/bootstrap
8822024-01-31T21:17:51.846ZUsing external DNS server at 10.151.2.164:53
8832024-01-31T21:17:51.846Z2024-01-31 21:17:51.603023461 UTC: attempting to log into API
 
8842024-01-31T21:17:52.867ZError: No response de-activating SOL payload
8852024-01-31T21:17:54.273Z[SOL Session operational. Use ~? for help]
 
8862024-01-31T21:18:06.859Z2024-01-31 21:18:06.870686885 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8872024-01-31T21:18:07.856Z2024-01-31 21:18:07.869078144 UTC: attempting to log into API
8882024-01-31T21:18:23.757Z2024-01-31 21:18:22.881738417 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8892024-01-31T21:18:23.905Z2024-01-31 21:18:23.883583733 UTC: attempting to log into API
8902024-01-31T21:18:38.954Z2024-01-31 21:18:38.932091220 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8912024-01-31T21:18:40.074Z2024-01-31 21:18:39.933183150 UTC: attempting to log into API
8922024-01-31T21:18:54.987Z2024-01-31 21:18:54.966552677 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8932024-01-31T21:18:55.984Z2024-01-31 21:18:55.967339543 UTC: attempting to log into API
8942024-01-31T21:19:11.030Z2024-01-31 21:19:11.003615203 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8952024-01-31T21:19:12.089Z2024-01-31 21:19:12.035876341 UTC: attempting to log into API
8962024-01-31T21:19:27.149Z2024-01-31 21:19:27.128303819 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8972024-01-31T21:19:28.149Z2024-01-31 21:19:28.129910912 UTC: attempting to log into API
8982024-01-31T21:19:43.194Z2024-01-31 21:19:43.169594859 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8992024-01-31T21:19:44.202Z2024-01-31 21:19:44.170775699 UTC: attempting to log into API
9002024-01-31T21:19:59.224Z2024-01-31 21:19:59.206706926 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9012024-01-31T21:20:00.237Z2024-01-31 21:20:00.207816805 UTC: attempting to log into API
9022024-01-31T21:20:15.587Z2024-01-31 21:20:15.243338043 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9032024-01-31T21:20:33.310Z2024-01-31 21:20:16.244430592 UTC: attempting to log into API
9042024-01-31T21:20:35.505Z2024-01-31 21:20:31.278444144 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9052024-01-31T21:20:35.505Z2024-01-31 21:20:32.280333530 UTC: attempting to log into API
 
9062024-01-31T21:20:42.531ZError: No response to keepalive - Terminating session
 
9072024-01-31T21:20:47.326Z2024-01-31 21:20:47.314206251 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9082024-01-31T21:20:48.999Z2024-01-31 21:20:48.315346410 UTC: attempting to log into API
 
9092024-01-31T21:20:52.525ZError: No response de-activating SOL payload
9102024-01-31T21:20:53.783Z[SOL Session operational. Use ~? for help]
 
9112024-01-31T21:21:06.777Z2024-01-31 21:21:03.349552564 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9122024-01-31T21:21:07.027Z2024-01-31 21:21:04.350699004 UTC: attempting to log into API
9132024-01-31T21:21:10.147Z2024-01-31 21:21:10.126020968 UTC: login succeeded
9142024-01-31T21:21:10.191Zwaiting for nexus to come up...
9152024-01-31T21:21:10.237Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9162024-01-31T21:21:10.400Zsetting up quotas...
9172024-01-31T21:21:10.470Zensuring datasets are ready...
9182024-01-31T21:21:12.547Zlet's roll.
9192024-01-31T21:21:12.575Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
9202024-01-31T21:21:12.575ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9212024-01-31T21:21:12.575ZOXIDE_TOKEN=oxide-token-21d6b18b3e42d68e84b179e2142d1ec457296b2f'
9222024-01-31T21:21:12.576Z++ OXIDE_HOST=https://recovery.sys.oxide.test
9232024-01-31T21:21:12.576Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9242024-01-31T21:21:12.586Z++ OXIDE_TOKEN=oxide-token-21d6b18b3e42d68e84b179e2142d1ec457296b2f
9252024-01-31T21:21:12.586Z+ export OXIDE_HOST OXIDE_TOKEN
9262024-01-31T21:21:12.586Z+ retry=0
9272024-01-31T21:21:12.586Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9282024-01-31T21:21:12.621Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9292024-01-31T21:21:12.621Z+ [[ 0 -gt 60 ]]
9302024-01-31T21:21:12.621Z+ sleep 1
9312024-01-31T21:21:13.591Z+ retry=1
9322024-01-31T21:21:13.592Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9332024-01-31T21:21:13.676Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
9342024-01-31T21:21:14.007Z{"status":"ok"}success
9352024-01-31T21:21:14.008ZProject {
9362024-01-31T21:21:14.037Z description: "some images",
9372024-01-31T21:21:14.037Z id: f26afe37-6912-48e0-8048-106f20741c78,
9382024-01-31T21:21:14.038Z name: Name(
9392024-01-31T21:21:14.040Z "images",
9402024-01-31T21:21:14.041Z ),
9412024-01-31T21:21:14.041Z time_created: 2024-01-31T21:21:13.774900Z,
9422024-01-31T21:21:14.041Z time_modified: 2024-01-31T21:21:13.774900Z,
9432024-01-31T21:21:14.042Z}
9442024-01-31T21:21:14.042Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
 
9452024-01-31T21:23:42.280ZError: No response to keepalive - Terminating session
9462024-01-31T21:23:52.727ZError: No response de-activating SOL payload
9472024-01-31T21:23:53.986Z[SOL Session operational. Use ~? for help]
9482024-01-31T21:26:42.560ZError: No response to keepalive - Terminating session
9492024-01-31T21:26:52.517ZError: No response de-activating SOL payload
9502024-01-31T21:26:53.685Z[SOL Session operational. Use ~? for help]
 
9512024-01-31T21:27:01.243Zdone!
9522024-01-31T21:27:01.315Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
9532024-01-31T21:27:01.453Zsuccess
9542024-01-31T21:27:01.453ZImage {
9552024-01-31T21:27:01.453Z block_size: ByteCount(
9562024-01-31T21:27:01.477Z 512,
9572024-01-31T21:27:01.477Z ),
9582024-01-31T21:27:01.478Z description: "debian 11 original base image",
9592024-01-31T21:27:01.478Z digest: None,
9602024-01-31T21:27:01.479Z id: 27ea45cf-242e-4656-87c5-713842e40d6b,
9612024-01-31T21:27:01.479Z name: Name(
9622024-01-31T21:27:01.479Z "debian11",
9632024-01-31T21:27:01.479Z ),
9642024-01-31T21:27:01.479Z os: "debian",
9652024-01-31T21:27:01.479Z project_id: None,
9662024-01-31T21:27:01.479Z size: ByteCount(
9672024-01-31T21:27:01.479Z 2147483648,
9682024-01-31T21:27:01.479Z ),
9692024-01-31T21:27:01.479Z time_created: 2024-01-31T21:27:01.159450Z,
9702024-01-31T21:27:01.479Z time_modified: 2024-01-31T21:27:01.415959Z,
9712024-01-31T21:27:01.479Z url: None,
9722024-01-31T21:27:01.479Z version: "11",
9732024-01-31T21:27:01.479Z}
9742024-01-31T21:27:01.479Z+ rm ./tests/bootstrap
9752024-01-31T21:27:01.537Z+ for test_bin in tests/*
9762024-01-31T21:27:01.537Z+ ./tests/bootstrap-485102d0e6124cf0
9772024-01-31T21:27:01.663Z
9782024-01-31T21:27:01.664Zrunning 0 tests
9792024-01-31T21:27:01.692Z
9802024-01-31T21:27:01.693Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9812024-01-31T21:27:01.693Z
9822024-01-31T21:27:01.693Z+ for test_bin in tests/*
9832024-01-31T21:27:01.693Z+ ./tests/end_to_end_tests-8616f2942d631494
9842024-01-31T21:27:01.952Z
9852024-01-31T21:27:01.953Zrunning 2 tests
9862024-01-31T21:27:02.228Ztest no_spoof::no_spoof_here ... ok
9872024-01-31T21:27:42.166Ztest instance_launch::instance_launch ... ok
9882024-01-31T21:27:42.251Z
9892024-01-31T21:27:42.252Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 37.51s
9902024-01-31T21:27:42.252Z
9912024-01-31T21:27:42.252Z+ _exit_trap
9922024-01-31T21:27:42.253Z+ local status=0
9932024-01-31T21:27:42.253Z+ [[ 0 -eq 0 ]]
9942024-01-31T21:27:42.253Z+ exit 0
9952024-01-31T21:27:42.253Zprocess exited: duration 774659 ms, exit code 0
 
9962024-01-31T21:27:42.400Zfound 72 output files
9972024-01-31T21:27:42.436Zuploading: /var/svc/log/oxide-sled-agent:default.log (1562594 bytes)
9982024-01-31T21:27:44.264Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9992024-01-31T21:27:44.264Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log (2590321 bytes)
10002024-01-31T21:27:46.343Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log" changed size mid upload: 2590321 -> 2606467
10012024-01-31T21:27:47.436Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log
10022024-01-31T21:27:47.436Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log (5622 bytes)
10032024-01-31T21:27:48.982Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log
10042024-01-31T21:27:48.982Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log (6791 bytes)
10052024-01-31T21:27:50.103Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log
10062024-01-31T21:27:50.130Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log (1349056 bytes)
10072024-01-31T21:27:51.317Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log" changed size mid upload: 1349056 -> 1361042
10082024-01-31T21:27:52.398Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log
10092024-01-31T21:27:52.399Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log (821291 bytes)
10102024-01-31T21:27:53.601Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log" changed size mid upload: 821291 -> 833757
10112024-01-31T21:27:54.673Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log
10122024-01-31T21:27:54.674Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log (1388 bytes)
10132024-01-31T21:27:55.776Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log
10142024-01-31T21:27:55.777Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log (288436 bytes)
10152024-01-31T21:27:56.089Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log" changed size mid upload: 288436 -> 294972
10162024-01-31T21:27:57.174Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log
10172024-01-31T21:27:57.174Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log (617 bytes)
10182024-01-31T21:27:58.306Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log
10192024-01-31T21:27:58.306Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log (21913 bytes)
10202024-01-31T21:27:59.449Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log
10212024-01-31T21:27:59.478Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_dded1cbe-0834-4039-a61f-1d0b58b5c960/root/var/svc/log/oxide-clickhouse:default.log (1301120 bytes)
10222024-01-31T21:28:00.402Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_dded1cbe-0834-4039-a61f-1d0b58b5c960/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 1301120 -> 1364580
10232024-01-31T21:28:01.499Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_dded1cbe-0834-4039-a61f-1d0b58b5c960/root/var/svc/log/oxide-clickhouse:default.log
10242024-01-31T21:28:01.499Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_dded1cbe-0834-4039-a61f-1d0b58b5c960/root/var/svc/log/oxide-zone-network-setup:default.log (1709 bytes)
10252024-01-31T21:28:02.626Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_dded1cbe-0834-4039-a61f-1d0b58b5c960/root/var/svc/log/oxide-zone-network-setup:default.log
10262024-01-31T21:28:02.626Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1a774a11-68fd-4d7a-a7bc-4b561c3d90c5/root/var/svc/log/oxide-cockroachdb:default.log (4380 bytes)
10272024-01-31T21:28:03.751Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1a774a11-68fd-4d7a-a7bc-4b561c3d90c5/root/var/svc/log/oxide-cockroachdb:default.log
10282024-01-31T21:28:03.787Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1a774a11-68fd-4d7a-a7bc-4b561c3d90c5/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
10292024-01-31T21:28:04.885Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_1a774a11-68fd-4d7a-a7bc-4b561c3d90c5/root/var/svc/log/oxide-zone-network-setup:default.log
10302024-01-31T21:28:04.886Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63107cc5-31ce-4044-a49a-c8ddaf7ef5d5/root/var/svc/log/oxide-crucible-agent:default.log (23482 bytes)
10312024-01-31T21:28:06.922Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63107cc5-31ce-4044-a49a-c8ddaf7ef5d5/root/var/svc/log/oxide-crucible-agent:default.log
10322024-01-31T21:28:06.949Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63107cc5-31ce-4044-a49a-c8ddaf7ef5d5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10332024-01-31T21:28:08.445Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63107cc5-31ce-4044-a49a-c8ddaf7ef5d5/root/var/svc/log/oxide-crucible-downstairs:default.log
10342024-01-31T21:28:08.445Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63107cc5-31ce-4044-a49a-c8ddaf7ef5d5/root/var/svc/log/oxide-crucible-downstairs:downstairs-6dd33390-695e-4ad8-a8c0-cfc75ed59c84.log (5617 bytes)
10352024-01-31T21:28:09.618Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63107cc5-31ce-4044-a49a-c8ddaf7ef5d5/root/var/svc/log/oxide-crucible-downstairs:downstairs-6dd33390-695e-4ad8-a8c0-cfc75ed59c84.log
10362024-01-31T21:28:09.619Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63107cc5-31ce-4044-a49a-c8ddaf7ef5d5/root/var/svc/log/oxide-crucible-downstairs:downstairs-f2b9a1bc-5fab-48d5-a67d-02707c7ddf7a.log (2944 bytes)
10372024-01-31T21:28:10.745Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63107cc5-31ce-4044-a49a-c8ddaf7ef5d5/root/var/svc/log/oxide-crucible-downstairs:downstairs-f2b9a1bc-5fab-48d5-a67d-02707c7ddf7a.log
10382024-01-31T21:28:10.745Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63107cc5-31ce-4044-a49a-c8ddaf7ef5d5/root/var/svc/log/oxide-crucible-downstairs:snapshot-6dd33390-695e-4ad8-a8c0-cfc75ed59c84-3059f5d0-7848-47d3-98f8-2e259ba9f020.log (5177 bytes)
10392024-01-31T21:28:12.345Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63107cc5-31ce-4044-a49a-c8ddaf7ef5d5/root/var/svc/log/oxide-crucible-downstairs:snapshot-6dd33390-695e-4ad8-a8c0-cfc75ed59c84-3059f5d0-7848-47d3-98f8-2e259ba9f020.log
10402024-01-31T21:28:12.345Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_186430bb-79f5-40fd-9a99-6190a91ebdbd/root/var/svc/log/oxide-external_dns:default.log (30724 bytes)
10412024-01-31T21:28:13.870Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_186430bb-79f5-40fd-9a99-6190a91ebdbd/root/var/svc/log/oxide-external_dns:default.log
10422024-01-31T21:28:13.870Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f473ba50-ce02-4cc5-8c17-fab2874f1adf/root/var/svc/log/oxide-internal_dns:default.log (112010 bytes)
10432024-01-31T21:28:14.211Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f473ba50-ce02-4cc5-8c17-fab2874f1adf/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 112010 -> 113262
10442024-01-31T21:28:15.731Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f473ba50-ce02-4cc5-8c17-fab2874f1adf/root/var/svc/log/oxide-internal_dns:default.log
10452024-01-31T21:28:15.731Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b/root/var/svc/log/oxide-ntp:default.log (1119 bytes)
10462024-01-31T21:28:16.838Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b/root/var/svc/log/oxide-ntp:default.log
10472024-01-31T21:28:16.838Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_5c4cb3cb-4e80-4d50-babe-f5c43bcc5330/root/var/svc/log/oxide-cockroachdb:default.log (4405 bytes)
10482024-01-31T21:28:18.455Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_5c4cb3cb-4e80-4d50-babe-f5c43bcc5330/root/var/svc/log/oxide-cockroachdb:default.log
10492024-01-31T21:28:18.455Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_5c4cb3cb-4e80-4d50-babe-f5c43bcc5330/root/var/svc/log/oxide-zone-network-setup:default.log (1704 bytes)
10502024-01-31T21:28:19.562Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_5c4cb3cb-4e80-4d50-babe-f5c43bcc5330/root/var/svc/log/oxide-zone-network-setup:default.log
10512024-01-31T21:28:19.563Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_47d371a4-0cda-4a3f-8bd7-b72d3c15b32d/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10522024-01-31T21:28:22.005Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_47d371a4-0cda-4a3f-8bd7-b72d3c15b32d/root/var/svc/log/oxide-crucible-agent:default.log
10532024-01-31T21:28:22.005Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_47d371a4-0cda-4a3f-8bd7-b72d3c15b32d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10542024-01-31T21:28:23.132Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_47d371a4-0cda-4a3f-8bd7-b72d3c15b32d/root/var/svc/log/oxide-crucible-downstairs:default.log
10552024-01-31T21:28:23.132Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_2aad55a2-124d-4a13-a8e1-89ca75fc7444/root/var/svc/log/oxide-external_dns:default.log (30720 bytes)
10562024-01-31T21:28:23.294Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_2aad55a2-124d-4a13-a8e1-89ca75fc7444/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 30720 -> 34496
10572024-01-31T21:28:24.916Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_2aad55a2-124d-4a13-a8e1-89ca75fc7444/root/var/svc/log/oxide-external_dns:default.log
10582024-01-31T21:28:24.916Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c53eb488-afb5-45e0-aae4-351c59ce1812/root/var/svc/log/oxide-internal_dns:default.log (113647 bytes)
10592024-01-31T21:28:25.333Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c53eb488-afb5-45e0-aae4-351c59ce1812/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 113647 -> 117403
10602024-01-31T21:28:26.447Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c53eb488-afb5-45e0-aae4-351c59ce1812/root/var/svc/log/oxide-internal_dns:default.log
10612024-01-31T21:28:26.447Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_d0289539-921c-4bf2-920a-04bbdc4c125f/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
10622024-01-31T21:28:28.053Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_d0289539-921c-4bf2-920a-04bbdc4c125f/root/var/svc/log/oxide-cockroachdb:default.log
10632024-01-31T21:28:28.053Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_d0289539-921c-4bf2-920a-04bbdc4c125f/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
10642024-01-31T21:28:29.155Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_d0289539-921c-4bf2-920a-04bbdc4c125f/root/var/svc/log/oxide-zone-network-setup:default.log
10652024-01-31T21:28:29.155Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_0da92414-7fee-4c7c-98b4-5596856b4bd9/root/var/svc/log/oxide-crucible-agent:default.log (15008 bytes)
10662024-01-31T21:28:30.789Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_0da92414-7fee-4c7c-98b4-5596856b4bd9/root/var/svc/log/oxide-crucible-agent:default.log
10672024-01-31T21:28:30.790Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_0da92414-7fee-4c7c-98b4-5596856b4bd9/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10682024-01-31T21:28:32.770Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_0da92414-7fee-4c7c-98b4-5596856b4bd9/root/var/svc/log/oxide-crucible-downstairs:default.log
10692024-01-31T21:28:32.770Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_0da92414-7fee-4c7c-98b4-5596856b4bd9/root/var/svc/log/oxide-crucible-downstairs:downstairs-c70727f7-40f3-427a-a4d6-317cc70cd4a4.log (5536 bytes)
10702024-01-31T21:28:33.897Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_0da92414-7fee-4c7c-98b4-5596856b4bd9/root/var/svc/log/oxide-crucible-downstairs:downstairs-c70727f7-40f3-427a-a4d6-317cc70cd4a4.log
10712024-01-31T21:28:33.898Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_0b757ae8-d715-46c9-990f-40e8f57b714b/root/var/svc/log/oxide-crucible-pantry:default.log (1225975 bytes)
10722024-01-31T21:28:36.402Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_0b757ae8-d715-46c9-990f-40e8f57b714b/root/var/svc/log/oxide-crucible-pantry:default.log
10732024-01-31T21:28:36.402Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_25cd3ebd-efff-40ff-a395-e1b4462d1c70/root/var/svc/log/oxide-internal_dns:default.log (115266 bytes)
10742024-01-31T21:28:36.576Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_25cd3ebd-efff-40ff-a395-e1b4462d1c70/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 115266 -> 119022
10752024-01-31T21:28:37.651Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_25cd3ebd-efff-40ff-a395-e1b4462d1c70/root/var/svc/log/oxide-internal_dns:default.log
10762024-01-31T21:28:37.652Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_e1a1727b-d1f2-4594-8693-39abbc53897f/root/var/svc/log/oxide-nexus:default.log (1161055 bytes)
10772024-01-31T21:28:38.006Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_e1a1727b-d1f2-4594-8693-39abbc53897f/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1161055 -> 1248023
10782024-01-31T21:28:39.092Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_e1a1727b-d1f2-4594-8693-39abbc53897f/root/var/svc/log/oxide-nexus:default.log
10792024-01-31T21:28:39.092Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_3e5ac74f-f729-4c04-8429-ce45ea26ba68/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
10802024-01-31T21:28:40.216Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_3e5ac74f-f729-4c04-8429-ce45ea26ba68/root/var/svc/log/oxide-cockroachdb:default.log
10812024-01-31T21:28:40.216Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_3e5ac74f-f729-4c04-8429-ce45ea26ba68/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
10822024-01-31T21:28:41.358Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_3e5ac74f-f729-4c04-8429-ce45ea26ba68/root/var/svc/log/oxide-zone-network-setup:default.log
10832024-01-31T21:28:41.358Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_677592f4-75b6-4a2e-a87e-694572ba9755/root/var/svc/log/oxide-crucible-agent:default.log (13687 bytes)
10842024-01-31T21:28:42.942Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_677592f4-75b6-4a2e-a87e-694572ba9755/root/var/svc/log/oxide-crucible-agent:default.log
10852024-01-31T21:28:42.942Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_677592f4-75b6-4a2e-a87e-694572ba9755/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10862024-01-31T21:28:44.061Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_677592f4-75b6-4a2e-a87e-694572ba9755/root/var/svc/log/oxide-crucible-downstairs:default.log
10872024-01-31T21:28:44.061Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_677592f4-75b6-4a2e-a87e-694572ba9755/root/var/svc/log/oxide-crucible-downstairs:downstairs-8c68b3d5-728e-4fbd-9772-0e07cf2a848d.log (3049 bytes)
10882024-01-31T21:28:45.156Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_677592f4-75b6-4a2e-a87e-694572ba9755/root/var/svc/log/oxide-crucible-downstairs:downstairs-8c68b3d5-728e-4fbd-9772-0e07cf2a848d.log
10892024-01-31T21:28:45.156Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_293e4627-e0e7-4b31-853e-740b7878beb5/root/var/svc/log/oxide-cockroachdb:default.log (4406 bytes)
10902024-01-31T21:28:46.257Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_293e4627-e0e7-4b31-853e-740b7878beb5/root/var/svc/log/oxide-cockroachdb:default.log
10912024-01-31T21:28:46.258Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_293e4627-e0e7-4b31-853e-740b7878beb5/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
10922024-01-31T21:28:47.403Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_293e4627-e0e7-4b31-853e-740b7878beb5/root/var/svc/log/oxide-zone-network-setup:default.log
10932024-01-31T21:28:47.404Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_bfd201eb-7daf-46b1-97cb-928d7695eeed/root/var/svc/log/oxide-crucible-agent:default.log (17553 bytes)
10942024-01-31T21:28:48.539Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_bfd201eb-7daf-46b1-97cb-928d7695eeed/root/var/svc/log/oxide-crucible-agent:default.log
10952024-01-31T21:28:48.539Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_bfd201eb-7daf-46b1-97cb-928d7695eeed/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10962024-01-31T21:28:49.645Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_bfd201eb-7daf-46b1-97cb-928d7695eeed/root/var/svc/log/oxide-crucible-downstairs:default.log
10972024-01-31T21:28:49.645Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_bfd201eb-7daf-46b1-97cb-928d7695eeed/root/var/svc/log/oxide-crucible-downstairs:downstairs-7b4cb353-1515-4f75-a041-6665ef6e820f.log (5620 bytes)
10982024-01-31T21:28:50.780Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_bfd201eb-7daf-46b1-97cb-928d7695eeed/root/var/svc/log/oxide-crucible-downstairs:downstairs-7b4cb353-1515-4f75-a041-6665ef6e820f.log
10992024-01-31T21:28:50.780Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_bfd201eb-7daf-46b1-97cb-928d7695eeed/root/var/svc/log/oxide-crucible-downstairs:snapshot-7b4cb353-1515-4f75-a041-6665ef6e820f-3059f5d0-7848-47d3-98f8-2e259ba9f020.log (5176 bytes)
11002024-01-31T21:28:51.878Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_bfd201eb-7daf-46b1-97cb-928d7695eeed/root/var/svc/log/oxide-crucible-downstairs:snapshot-7b4cb353-1515-4f75-a041-6665ef6e820f-3059f5d0-7848-47d3-98f8-2e259ba9f020.log
11012024-01-31T21:28:51.904Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_b49728b9-39be-4b43-985f-3cfc6bf15c54/root/var/svc/log/oxide-crucible-agent:default.log (10226 bytes)
11022024-01-31T21:28:52.996Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_b49728b9-39be-4b43-985f-3cfc6bf15c54/root/var/svc/log/oxide-crucible-agent:default.log
11032024-01-31T21:28:52.996Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_b49728b9-39be-4b43-985f-3cfc6bf15c54/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11042024-01-31T21:28:54.105Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_b49728b9-39be-4b43-985f-3cfc6bf15c54/root/var/svc/log/oxide-crucible-downstairs:default.log
11052024-01-31T21:28:54.105Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_b49728b9-39be-4b43-985f-3cfc6bf15c54/root/var/svc/log/oxide-crucible-downstairs:downstairs-50b1d2d5-2d0d-44f0-a5ce-bedb7ef24abb.log (2943 bytes)
11062024-01-31T21:28:55.207Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_b49728b9-39be-4b43-985f-3cfc6bf15c54/root/var/svc/log/oxide-crucible-downstairs:downstairs-50b1d2d5-2d0d-44f0-a5ce-bedb7ef24abb.log
11072024-01-31T21:28:55.207Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_6baf4aa0-14d0-4012-950b-d0f19be44a86/root/var/svc/log/oxide-nexus:default.log (34569433 bytes)
11082024-01-31T21:29:17.584Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_6baf4aa0-14d0-4012-950b-d0f19be44a86/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 34569433 -> 34743401
11092024-01-31T21:29:18.660Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_6baf4aa0-14d0-4012-950b-d0f19be44a86/root/var/svc/log/oxide-nexus:default.log
11102024-01-31T21:29:18.660Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_608f1e18-3502-4610-af6e-13e94039eb49/root/var/svc/log/oxide-crucible-agent:default.log (30722 bytes)
11112024-01-31T21:29:19.830Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_608f1e18-3502-4610-af6e-13e94039eb49/root/var/svc/log/oxide-crucible-agent:default.log
11122024-01-31T21:29:19.830Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_608f1e18-3502-4610-af6e-13e94039eb49/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11132024-01-31T21:29:20.941Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_608f1e18-3502-4610-af6e-13e94039eb49/root/var/svc/log/oxide-crucible-downstairs:default.log
11142024-01-31T21:29:20.942Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_608f1e18-3502-4610-af6e-13e94039eb49/root/var/svc/log/oxide-crucible-downstairs:downstairs-20952443-a437-4525-ac69-361ffe9946b8.log (3050 bytes)
11152024-01-31T21:29:22.044Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_608f1e18-3502-4610-af6e-13e94039eb49/root/var/svc/log/oxide-crucible-downstairs:downstairs-20952443-a437-4525-ac69-361ffe9946b8.log
11162024-01-31T21:29:22.044Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_608f1e18-3502-4610-af6e-13e94039eb49/root/var/svc/log/oxide-crucible-downstairs:downstairs-32584863-4e48-4a15-ab20-9c922cbe24b4.log (2945 bytes)
11172024-01-31T21:29:23.568Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_608f1e18-3502-4610-af6e-13e94039eb49/root/var/svc/log/oxide-crucible-downstairs:downstairs-32584863-4e48-4a15-ab20-9c922cbe24b4.log
11182024-01-31T21:29:23.568Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_608f1e18-3502-4610-af6e-13e94039eb49/root/var/svc/log/oxide-crucible-downstairs:downstairs-f6af4d9f-b4e8-463a-8610-46ddfd398696.log (5537 bytes)
11192024-01-31T21:29:24.263Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_608f1e18-3502-4610-af6e-13e94039eb49/root/var/svc/log/oxide-crucible-downstairs:downstairs-f6af4d9f-b4e8-463a-8610-46ddfd398696.log
11202024-01-31T21:29:24.263Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_bc2d4b57-b658-4cfd-8b99-243e666e8c3f/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
11212024-01-31T21:29:25.372Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_bc2d4b57-b658-4cfd-8b99-243e666e8c3f/root/var/svc/log/oxide-crucible-pantry:default.log
11222024-01-31T21:29:25.372Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_c3d41cf7-d750-4355-91d2-66a34e791e1e/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
11232024-01-31T21:29:26.482Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_c3d41cf7-d750-4355-91d2-66a34e791e1e/root/var/svc/log/oxide-crucible-pantry:default.log
11242024-01-31T21:29:26.512Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_1dc163a1-5af7-48a8-9cd2-e21722e1f8af/root/var/svc/log/oxide-nexus:default.log (2523373 bytes)
11252024-01-31T21:29:28.718Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_1dc163a1-5af7-48a8-9cd2-e21722e1f8af/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 2523373 -> 2703323
11262024-01-31T21:29:29.803Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_1dc163a1-5af7-48a8-9cd2-e21722e1f8af/root/var/svc/log/oxide-nexus:default.log
11272024-01-31T21:29:29.803Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1d1f1cb7-5d14-4b53-b47a-fcb65d8a1f9f/root/var/svc/log/oxide-crucible-agent:default.log (27797 bytes)
11282024-01-31T21:29:30.948Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1d1f1cb7-5d14-4b53-b47a-fcb65d8a1f9f/root/var/svc/log/oxide-crucible-agent:default.log
11292024-01-31T21:29:30.950Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1d1f1cb7-5d14-4b53-b47a-fcb65d8a1f9f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11302024-01-31T21:29:32.058Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1d1f1cb7-5d14-4b53-b47a-fcb65d8a1f9f/root/var/svc/log/oxide-crucible-downstairs:default.log
11312024-01-31T21:29:32.058Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1d1f1cb7-5d14-4b53-b47a-fcb65d8a1f9f/root/var/svc/log/oxide-crucible-downstairs:downstairs-298c457e-3a97-43ad-b67e-7619aa70229b.log (5614 bytes)
11322024-01-31T21:29:33.185Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1d1f1cb7-5d14-4b53-b47a-fcb65d8a1f9f/root/var/svc/log/oxide-crucible-downstairs:downstairs-298c457e-3a97-43ad-b67e-7619aa70229b.log
11332024-01-31T21:29:33.213Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1d1f1cb7-5d14-4b53-b47a-fcb65d8a1f9f/root/var/svc/log/oxide-crucible-downstairs:downstairs-ab1558d0-21a8-4ff1-8217-4e6bbb4ba38a.log (3048 bytes)
11342024-01-31T21:29:34.285Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1d1f1cb7-5d14-4b53-b47a-fcb65d8a1f9f/root/var/svc/log/oxide-crucible-downstairs:downstairs-ab1558d0-21a8-4ff1-8217-4e6bbb4ba38a.log
11352024-01-31T21:29:34.310Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1d1f1cb7-5d14-4b53-b47a-fcb65d8a1f9f/root/var/svc/log/oxide-crucible-downstairs:snapshot-298c457e-3a97-43ad-b67e-7619aa70229b-3059f5d0-7848-47d3-98f8-2e259ba9f020.log (5174 bytes)
11362024-01-31T21:29:35.388Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1d1f1cb7-5d14-4b53-b47a-fcb65d8a1f9f/root/var/svc/log/oxide-crucible-downstairs:snapshot-298c457e-3a97-43ad-b67e-7619aa70229b-3059f5d0-7848-47d3-98f8-2e259ba9f020.log
11372024-01-31T21:29:35.388Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_d4b9aca8-6d8e-4b9e-aadb-3d13456cad48/root/var/svc/log/oxide-oximeter:default.log (345517 bytes)
11382024-01-31T21:29:35.838Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_d4b9aca8-6d8e-4b9e-aadb-3d13456cad48/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 345517 -> 438455
11392024-01-31T21:29:36.925Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_d4b9aca8-6d8e-4b9e-aadb-3d13456cad48/root/var/svc/log/oxide-oximeter:default.log
11402024-01-31T21:29:36.925Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_d4b9aca8-6d8e-4b9e-aadb-3d13456cad48/root/var/svc/log/oxide-zone-network-setup:default.log (1697 bytes)
11412024-01-31T21:29:38.073Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_d4b9aca8-6d8e-4b9e-aadb-3d13456cad48/root/var/svc/log/oxide-zone-network-setup:default.log
11422024-01-31T21:29:38.073Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7aede8e5-c7f6-40c3-9245-60ecd570f175/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
11432024-01-31T21:29:39.226Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7aede8e5-c7f6-40c3-9245-60ecd570f175/root/var/svc/log/oxide-crucible-agent:default.log
11442024-01-31T21:29:39.226Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7aede8e5-c7f6-40c3-9245-60ecd570f175/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11452024-01-31T21:29:40.346Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7aede8e5-c7f6-40c3-9245-60ecd570f175/root/var/svc/log/oxide-crucible-downstairs:default.log
11462024-01-31T21:29:40.346Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7aede8e5-c7f6-40c3-9245-60ecd570f175/root/var/svc/log/oxide-crucible-downstairs:downstairs-9e374b92-20ce-4223-8660-664b59ea103a.log (5538 bytes)
11472024-01-31T21:29:41.872Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7aede8e5-c7f6-40c3-9245-60ecd570f175/root/var/svc/log/oxide-crucible-downstairs:downstairs-9e374b92-20ce-4223-8660-664b59ea103a.log
11482024-01-31T21:29:41.872Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b/root/var/log/chrony/measurements.log (24797 bytes)
11492024-01-31T21:29:41.988Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b/root/var/log/chrony/measurements.log" changed size mid upload: 24797 -> 27948
11502024-01-31T21:29:43.077Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b/root/var/log/chrony/measurements.log
11512024-01-31T21:29:43.077Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b/root/var/log/chrony/statistics.log (15093 bytes)
11522024-01-31T21:29:43.619Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b/root/var/log/chrony/statistics.log" changed size mid upload: 15093 -> 17901
11532024-01-31T21:29:44.696Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b/root/var/log/chrony/statistics.log
11542024-01-31T21:29:44.697Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b/root/var/log/chrony/tracking.log (2113 bytes)
11552024-01-31T21:29:44.744Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b/root/var/log/chrony/tracking.log" changed size mid upload: 2113 -> 2377
11562024-01-31T21:29:45.821Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1ee5ad27-d0bc-4132-b9fc-e3f09de9de3b/root/var/log/chrony/tracking.log