01HMVNGG26GWKRVP1AKDXHDQD2: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.27",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HMVNHDZ12VXMEA90RVN4079K

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12024-01-23T18:31:47.463Zjob dependencies complete; ready to run (waiting for 1 h 8 m 49 s)
22024-01-23T18:31:47.477Zjob assigned to worker 01HMVRCPMM90RBH3YVY4MM5B57 (queued for 0 s)
32024-01-23T18:31:47.530Zhttp://172.20.0.5:9969/ipxe/sock... ok
42024-01-23T18:31:47.593Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52024-01-23T18:31:47.638Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... 28%Error: No response to keepalive - Terminating session
62024-01-23T18:31:47.693ZError: No response de-activating SOL payload
72024-01-23T18:31:47.739Z[SOL Session operational. Use ~? for help]
82024-01-23T18:31:47.808Z 89%Error: No response to keepalive - Terminating session
92024-01-23T18:31:47.855ZError: No response de-activating SOL payload
102024-01-23T18:31:47.900Z[SOL Session operational. Use ~? for help]
112024-01-23T18:31:47.956ZError: No response to keepalive - Terminating session
122024-01-23T18:31:48.063ZError: No response de-activating SOL payload
132024-01-23T18:31:48.114Z[SOL Session operational. Use ~? for help]
142024-01-23T18:31:48.172ZError: No response to keepalive - Terminating session
152024-01-23T18:31:48.225ZError: No response de-activating SOL payload
162024-01-23T18:31:48.275Z[SOL Session operational. Use ~? for help]
172024-01-23T18:31:48.327ZError: No response to keepalive - Terminating session
182024-01-23T18:31:48.383ZError: No response de-activating SOL payload
192024-01-23T18:31:48.430Z[SOL Session operational. Use ~? for help]
202024-01-23T18:31:48.480ZError: No response to keepalive - Terminating session
212024-01-23T18:31:48.525ZError: No response de-activating SOL payload
222024-01-23T18:31:48.568Z[SOL Session operational. Use ~? for help]
232024-01-23T18:31:49.356Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-ae34befe45ae3da4.gz
242024-01-23T18:31:51.096Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-ae34befe45ae3da4.gz
252024-01-23T18:31:51.136Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262024-01-23T18:32:23.707Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272024-01-23T18:32:23.770Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-fe5f3f977d30e70c.gz
282024-01-23T18:32:55.722Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-fe5f3f977d30e70c.gz
292024-01-23T18:32:55.722Zdownloading input: /input/ci-tools/work/caboose-util.gz
302024-01-23T18:32:57.102Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312024-01-23T18:32:57.102Zdownloading input: /input/ci-tools/work/tufaceous.gz
322024-01-23T18:33:02.362Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332024-01-23T18:33:02.408Zdownloading input: /input/package/work/version.txt
342024-01-23T18:33:02.408Zdownloaded input: /input/package/work/version.txt
352024-01-23T18:33:02.408Zdownloading input: /input/package/work/package.tar.gz
362024-01-23T18:33:35.822Zdownloaded input: /input/package/work/package.tar.gz
372024-01-23T18:33:35.822Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382024-01-23T18:34:05.696ZError: No response to keepalive - Terminating session
392024-01-23T18:34:15.889ZError: No response de-activating SOL payload
402024-01-23T18:34:17.020Z[SOL Session operational. Use ~? for help]
412024-01-23T18:34:24.898Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
422024-01-23T18:34:24.898Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
432024-01-23T18:34:38.128Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
442024-01-23T18:34:38.170Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
452024-01-23T18:35:47.308Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
462024-01-23T18:35:47.309Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
472024-01-23T18:36:58.723Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
482024-01-23T18:36:58.817Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
492024-01-23T18:37:05.699ZError: No response to keepalive - Terminating session
502024-01-23T18:37:15.895ZError: No response de-activating SOL payload
512024-01-23T18:37:17.296Z[SOL Session operational. Use ~? for help]
522024-01-23T18:37:42.288Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
532024-01-23T18:37:42.288Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
542024-01-23T18:37:47.240Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
552024-01-23T18:37:47.240Zdownloading input: /input/package/work/zones/crucible.tar.gz
562024-01-23T18:37:56.380Zdownloaded input: /input/package/work/zones/crucible.tar.gz
572024-01-23T18:37:56.420Zdownloading input: /input/package/work/zones/external-dns.tar.gz
582024-01-23T18:38:04.027Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
592024-01-23T18:38:04.105Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
602024-01-23T18:38:12.688Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
612024-01-23T18:38:12.688Zdownloading input: /input/package/work/zones/ntp.tar.gz
622024-01-23T18:38:12.859Zdownloaded input: /input/package/work/zones/ntp.tar.gz
632024-01-23T18:38:13.358Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
642024-01-23T18:38:18.879Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
652024-01-23T18:38:18.879Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
662024-01-23T18:38:28.024Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
672024-01-23T18:38:28.024Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
682024-01-23T18:38:52.640Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
692024-01-23T18:38:52.640Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
702024-01-23T18:39:20.735Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
712024-01-23T18:39:20.777Zdownloading input: /input/package/work/zones/overlay.tar.gz
722024-01-23T18:39:20.851Zdownloaded input: /input/package/work/zones/overlay.tar.gz
732024-01-23T18:39:20.892Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
742024-01-23T18:39:33.181Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
752024-01-23T18:39:33.181Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
762024-01-23T18:39:53.273Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
772024-01-23T18:39:53.273Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
782024-01-23T18:40:22.548ZError: No response to keepalive - Terminating session
792024-01-23T18:40:32.550ZError: No response de-activating SOL payload
802024-01-23T18:40:33.824Z[SOL Session operational. Use ~? for help]
812024-01-23T18:41:32.183Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
822024-01-23T18:41:32.665Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
832024-01-23T18:42:52.647Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
842024-01-23T18:42:52.647Zstarting task 0: "setup"
852024-01-23T18:42:52.690Z++ uname -s
862024-01-23T18:42:52.690Z+ kern=SunOS
872024-01-23T18:42:52.690Z+ case "$kern" in
882024-01-23T18:42:52.690Z+ groupadd -g 12345 build
892024-01-23T18:42:52.690Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
902024-01-23T18:42:54.666Z+ zfs create -o mountpoint=/work rpool/work
912024-01-23T18:42:54.708Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
922024-01-23T18:42:54.708Z+ home_fs=zfs
932024-01-23T18:42:54.708Z+ [[ zfs == autofs ]]
942024-01-23T18:42:54.709Z+ mkdir -p /home/build
952024-01-23T18:42:54.709Z+ chown build:build /home/build /work
962024-01-23T18:42:56.693Z+ chmod 0700 /home/build /work
972024-01-23T18:42:56.738Zprocess exited: duration 4045 ms, exit code 0
 
982024-01-23T18:42:56.833Zstarting task 1: "authentication"
992024-01-23T18:42:56.895Zprocess exited: duration 65 ms, exit code 0
 
1002024-01-23T18:42:56.983Zstarting task 2: "build"
1012024-01-23T18:42:57.027Z+ trap _exit_trap EXIT
1022024-01-23T18:42:57.027Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
1032024-01-23T18:42:57.210Zsecpolicy_fs_linkdir: 0x55 = 0x55
1042024-01-23T18:42:57.210Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
1052024-01-23T18:42:57.211Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
1062024-01-23T18:42:57.211Zsecpolicy_fs_linkdir+3: 0 = 0xe5
1072024-01-23T18:42:57.211Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1082024-01-23T18:42:57.211Zsecpolicy_fs_linkdir+5: 0 = 0x1
1092024-01-23T18:42:57.211Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1102024-01-23T18:42:57.211Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1112024-01-23T18:42:57.296Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1122024-01-23T18:42:57.296Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1132024-01-23T18:42:57.296Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1142024-01-23T18:42:57.296Z+ [[ -d /opt/oxide ]]
1152024-01-23T18:42:57.296Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1162024-01-23T18:42:57.296Z+ pfexec mkdir /opt/oxide
1172024-01-23T18:42:57.296Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1182024-01-23T18:42:57.296Z+ [[ -d /opt/oxide-underneath ]]
1192024-01-23T18:42:57.297Z+ cd /opt/oxide-underneath
1202024-01-23T18:42:57.297Z+ pfexec tar ceEp@/f - .
1212024-01-23T18:42:57.297Z+ cd /opt/oxide
1222024-01-23T18:42:57.297Z+ pfexec tar xveEp@/f -
1232024-01-23T18:42:57.338Zx ., 0 bytes, 0 tape blocks
1242024-01-23T18:42:57.338Zx ./opte, 0 bytes, 0 tape blocks
1252024-01-23T18:42:57.338Zx ./opte/bin, 0 bytes, 0 tape blocks
1262024-01-23T18:42:57.338Zx ./opte/bin/opteadm, 52567720 bytes, 102672 tape blocks
1272024-01-23T18:42:57.338Z+ pfexec rm -rf /opt/oxide-underneath
1282024-01-23T18:42:57.339Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1292024-01-23T18:42:57.379Z+ pfexec mkdir /opt/oxide/work
1302024-01-23T18:42:57.379Z+ pfexec chown build:build /opt/oxide/work
1312024-01-23T18:42:57.379Z+ cd /opt/oxide/work
1322024-01-23T18:42:57.379Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1332024-01-23T18:42:58.971ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1342024-01-23T18:42:58.971Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1352024-01-23T18:42:58.971Zx out/omicron-sled-agent.tar, 120804864 bytes, 235947 tape blocks
1362024-01-23T18:42:58.971Zx out/target/test, 75 bytes, 1 tape blocks
1372024-01-23T18:42:58.971Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1382024-01-23T18:42:58.971Zx package-manifest.toml, 23296 bytes, 46 tape blocks
1392024-01-23T18:42:58.971Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1402024-01-23T18:42:58.971Zx target/release/omicron-package, 44507824 bytes, 86930 tape blocks
1412024-01-23T18:42:58.971Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1422024-01-23T18:42:58.971Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1432024-01-23T18:42:58.971Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1442024-01-23T18:42:58.971Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1452024-01-23T18:42:58.971Zx tools/scrimlet/softnpu-init.sh, 2149 bytes, 5 tape blocks
1462024-01-23T18:42:58.971Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1472024-01-23T18:42:58.971Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1482024-01-23T18:42:58.971Z
1492024-01-23T18:42:58.971Zreal 1.596918816
1502024-01-23T18:42:58.972Zuser 1.440461124
1512024-01-23T18:42:58.972Zsys 0.280018405
1522024-01-23T18:42:58.972Ztrap 0.000065812
1532024-01-23T18:42:58.972Ztflt 0.000000000
1542024-01-23T18:42:58.972Zdflt 0.000000000
1552024-01-23T18:42:58.972Zkflt 0.000000000
1562024-01-23T18:42:58.972Zlock 0.000000000
1572024-01-23T18:42:58.972Zslp 1.437627192
1582024-01-23T18:42:58.972Zlat 0.033620927
1592024-01-23T18:42:58.972Zstop 0.000119943
1602024-01-23T18:42:58.972Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1612024-01-23T18:43:00.119Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1622024-01-23T18:43:00.158Z+ mkdir tests
1632024-01-23T18:43:00.158Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1642024-01-23T18:43:00.159Z+ ptime -m gunzip
1652024-01-23T18:43:00.159Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-ae34befe45ae3da4
1662024-01-23T18:43:00.275Z
1672024-01-23T18:43:00.314Zreal 0.138737766
1682024-01-23T18:43:00.314Zuser 0.126434873
1692024-01-23T18:43:00.314Zsys 0.012068977
1702024-01-23T18:43:00.314Ztrap 0.000008850
1712024-01-23T18:43:00.314Ztflt 0.000000000
1722024-01-23T18:43:00.314Zdflt 0.000000000
1732024-01-23T18:43:00.314Zkflt 0.000000000
1742024-01-23T18:43:00.314Zlock 0.000000000
1752024-01-23T18:43:00.314Zslp 0.000022070
1762024-01-23T18:43:00.314Zlat 0.000015880
1772024-01-23T18:43:00.314Zstop 0.000120273
1782024-01-23T18:43:00.314Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-ae34befe45ae3da4
1792024-01-23T18:43:00.315Z+ chmod a+x tests/bootstrap-ae34befe45ae3da4
1802024-01-23T18:43:00.315Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1812024-01-23T18:43:00.315Z+ ptime -m gunzip
1822024-01-23T18:43:00.315Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1832024-01-23T18:43:02.688Z
1842024-01-23T18:43:02.732Zreal 2.394595161
1852024-01-23T18:43:02.732Zuser 2.243832955
1862024-01-23T18:43:02.732Zsys 0.150348636
1872024-01-23T18:43:02.732Ztrap 0.000173114
1882024-01-23T18:43:02.732Ztflt 0.000000000
1892024-01-23T18:43:02.733Zdflt 0.000000000
1902024-01-23T18:43:02.733Zkflt 0.000000000
1912024-01-23T18:43:02.733Zlock 0.000000000
1922024-01-23T18:43:02.733Zslp 0.000018860
1932024-01-23T18:43:02.733Zlat 0.000023760
1942024-01-23T18:43:02.733Zstop 0.000128762
1952024-01-23T18:43:02.733Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1962024-01-23T18:43:02.733Z+ chmod a+x tests/bootstrap
1972024-01-23T18:43:02.734Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1982024-01-23T18:43:02.734Z+ ptime -m gunzip
1992024-01-23T18:43:02.734Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-fe5f3f977d30e70c
2002024-01-23T18:43:05.160Z
2012024-01-23T18:43:05.216Zreal 2.457598633
2022024-01-23T18:43:05.216Zuser 2.305687438
2032024-01-23T18:43:05.216Zsys 0.151440894
2042024-01-23T18:43:05.216Ztrap 0.000184544
2052024-01-23T18:43:05.216Ztflt 0.000000000
2062024-01-23T18:43:05.216Zdflt 0.000000000
2072024-01-23T18:43:05.217Zkflt 0.000000000
2082024-01-23T18:43:05.217Zlock 0.000000000
2092024-01-23T18:43:05.217Zslp 0.000034851
2102024-01-23T18:43:05.217Zlat 0.000073062
2112024-01-23T18:43:05.217Zstop 0.000120912
2122024-01-23T18:43:05.217Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-fe5f3f977d30e70c
2132024-01-23T18:43:05.217Z+ chmod a+x tests/end_to_end_tests-fe5f3f977d30e70c
2142024-01-23T18:43:05.218Z+ bmat address ls
2152024-01-23T18:43:05.218ZNAME CIDR FIRST LAST COUNT
2162024-01-23T18:43:05.218Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2172024-01-23T18:43:05.218Z++ bmat address ls -f extra -Ho first,last,count
2182024-01-23T18:43:05.218Z+ set -- 10.151.2.164 10.151.2.227 64
2192024-01-23T18:43:05.218Z+ EXTRA_IP_START=10.151.2.164
2202024-01-23T18:43:05.218Z+ EXTRA_IP_END=10.151.2.227
2212024-01-23T18:43:05.219Z+ EXTRA_IP_COUNT=64
2222024-01-23T18:43:05.219Z+ (( EXTRA_IP_COUNT >= 32 ))
2232024-01-23T18:43:05.219Z+ EXTRA_IP_BASE=10.151.2
2242024-01-23T18:43:05.219Z+ EXTRA_IP_FOCTET=164
2252024-01-23T18:43:05.219Z+ EXTRA_IP_LOCTET=227
2262024-01-23T18:43:05.219Z+ SERVICE_IP_POOL_START=10.151.2.164
2272024-01-23T18:43:05.219Z+ SERVICE_IP_POOL_END=10.151.2.173
2282024-01-23T18:43:05.219Z+ DNS_IP1=10.151.2.164
2292024-01-23T18:43:05.219Z+ DNS_IP2=10.151.2.165
2302024-01-23T18:43:05.220Z+ UPLINK_IP=10.151.2.174
2312024-01-23T18:43:05.220Z+ IPPOOL_START=10.151.2.175
2322024-01-23T18:43:05.220Z+ IPPOOL_END=10.151.2.227
2332024-01-23T18:43:05.220Z++ ipadm show-addr -po type,addr
2342024-01-23T18:43:05.220Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2352024-01-23T18:43:05.220Z+ GATEWAY_IP=10.151.2.100
2362024-01-23T18:43:05.220Z+ [[ -n 10.151.2.100 ]]
2372024-01-23T18:43:05.220Z+ ping -s 10.151.2.100 56 1
2382024-01-23T18:43:05.220ZPING 10.151.2.100: 56 data bytes
2392024-01-23T18:43:05.273Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.010 ms
2402024-01-23T18:43:05.273Z
2412024-01-23T18:43:05.273Z----10.151.2.100 PING Statistics----
2422024-01-23T18:43:05.274Z1 packets transmitted, 1 packets received, 0% packet loss
2432024-01-23T18:43:05.274Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2442024-01-23T18:43:05.274Z++ arp -an
2452024-01-23T18:43:05.274Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2462024-01-23T18:43:05.274Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2472024-01-23T18:43:05.275Z+ routeadm -e ipv4-forwarding -u
2482024-01-23T18:43:05.334Z+ PXA_START=10.151.2.164
2492024-01-23T18:43:05.334Z+ PXA_END=10.151.2.227
2502024-01-23T18:43:05.386Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2512024-01-23T18:43:05.386Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2522024-01-23T18:43:05.475Z+ ZPOOL_VDEV_DIR=/scratch
2532024-01-23T18:43:05.475Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2542024-01-23T18:43:05.527Z+++ dirname ./tools/create_virtual_hardware.sh
2552024-01-23T18:43:05.527Z++ cd ./tools
2562024-01-23T18:43:05.527Z++ pwd
2572024-01-23T18:43:05.527Z+ SOURCE_DIR=/opt/oxide/work/tools
2582024-01-23T18:43:05.528Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2592024-01-23T18:43:05.528Z+ SOFTNPU_MODE=zone
2602024-01-23T18:43:05.528Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2612024-01-23T18:43:05.528Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2622024-01-23T18:43:05.528Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2632024-01-23T18:43:05.529Z++ dladm show-phys -p -o LINK
2642024-01-23T18:43:05.529Z++ head -1
2652024-01-23T18:43:05.579Z+ PHYSICAL_LINK=igb0
2662024-01-23T18:43:05.579Z+ echo 'Using igb0 as physical link'
2672024-01-23T18:43:05.579ZUsing igb0 as physical link
2682024-01-23T18:43:05.580Z+ ensure_run_as_root
2692024-01-23T18:43:05.580Z++ id -u
2702024-01-23T18:43:05.580Z+ [[ 0 -ne 0 ]]
2712024-01-23T18:43:05.580Z+ ensure_zpools
2722024-01-23T18:43:05.580Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2732024-01-23T18:43:05.580Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2742024-01-23T18:43:05.580Z+ readarray -t ZPOOLS
2752024-01-23T18:43:05.580Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2762024-01-23T18:43:05.580Z++ sed 's/[ ",]//g'
2772024-01-23T18:43:05.581Z+ for ZPOOL in "${ZPOOLS[@]}"
2782024-01-23T18:43:05.581Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2792024-01-23T18:43:05.581ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2802024-01-23T18:43:05.581Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2812024-01-23T18:43:05.581Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2822024-01-23T18:43:05.581Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2832024-01-23T18:43:05.635Z0+0 records in
2842024-01-23T18:43:05.635Z0+0 records out
2852024-01-23T18:43:05.635Z0 bytes (0 iB) transferred in 0.000007 secs (0 iB/sec)
2862024-01-23T18:43:05.635Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2872024-01-23T18:43:05.636Z+ set +x
2882024-01-23T18:43:05.636ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2892024-01-23T18:43:05.636Z++ zpool list -o name
2902024-01-23T18:43:05.636Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2912024-01-23T18:43:05.636Z+ [[ -z '' ]]
2922024-01-23T18:43:05.637Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2932024-01-23T18:43:05.689Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2942024-01-23T18:43:05.689Z+ set +x
2952024-01-23T18:43:05.689ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2962024-01-23T18:43:05.689Z+ for ZPOOL in "${ZPOOLS[@]}"
2972024-01-23T18:43:05.689Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2982024-01-23T18:43:05.689ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2992024-01-23T18:43:05.689Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3002024-01-23T18:43:05.690Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3012024-01-23T18:43:05.690Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3022024-01-23T18:43:05.690Z0+0 records in
3032024-01-23T18:43:05.690Z0+0 records out
3042024-01-23T18:43:05.690Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3052024-01-23T18:43:05.690Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3062024-01-23T18:43:05.690Z+ set +x
3072024-01-23T18:43:05.690ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3082024-01-23T18:43:05.690Z++ zpool list -o name
3092024-01-23T18:43:05.690Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3102024-01-23T18:43:05.691Z+ [[ -z '' ]]
3112024-01-23T18:43:05.691Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3122024-01-23T18:43:05.790Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3132024-01-23T18:43:05.790Z+ set +x
3142024-01-23T18:43:05.790ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3152024-01-23T18:43:05.790Z+ for ZPOOL in "${ZPOOLS[@]}"
3162024-01-23T18:43:05.791Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3172024-01-23T18:43:05.791ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3182024-01-23T18:43:05.791Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3192024-01-23T18:43:05.791Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3202024-01-23T18:43:05.791Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3212024-01-23T18:43:05.791Z0+0 records in
3222024-01-23T18:43:05.792Z0+0 records out
3232024-01-23T18:43:05.792Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3242024-01-23T18:43:05.792Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3252024-01-23T18:43:05.792Z+ set +x
3262024-01-23T18:43:05.792ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3272024-01-23T18:43:05.792Z++ zpool list -o name
3282024-01-23T18:43:05.792Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3292024-01-23T18:43:05.792Z+ [[ -z '' ]]
3302024-01-23T18:43:05.792Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3312024-01-23T18:43:05.842Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3322024-01-23T18:43:05.842Z+ set +x
3332024-01-23T18:43:05.843ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3342024-01-23T18:43:05.843Z+ for ZPOOL in "${ZPOOLS[@]}"
3352024-01-23T18:43:05.843Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3362024-01-23T18:43:05.843ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3372024-01-23T18:43:05.843Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3382024-01-23T18:43:05.843Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3392024-01-23T18:43:05.843Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3402024-01-23T18:43:05.843Z0+0 records in
3412024-01-23T18:43:05.843Z0+0 records out
3422024-01-23T18:43:05.844Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3432024-01-23T18:43:05.844Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3442024-01-23T18:43:05.844Z+ set +x
3452024-01-23T18:43:05.844ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3462024-01-23T18:43:05.844Z++ zpool list -o name
3472024-01-23T18:43:05.844Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3482024-01-23T18:43:05.844Z+ [[ -z '' ]]
3492024-01-23T18:43:05.844Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502024-01-23T18:43:05.845Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512024-01-23T18:43:05.845Z+ set +x
3522024-01-23T18:43:05.845ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3532024-01-23T18:43:05.845Z+ for ZPOOL in "${ZPOOLS[@]}"
3542024-01-23T18:43:05.845Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552024-01-23T18:43:05.845ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3562024-01-23T18:43:05.845Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3572024-01-23T18:43:05.845Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3582024-01-23T18:43:05.845Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3592024-01-23T18:43:05.846Z0+0 records in
3602024-01-23T18:43:05.846Z0+0 records out
3612024-01-23T18:43:05.846Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3622024-01-23T18:43:05.846Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3632024-01-23T18:43:05.846Z+ set +x
3642024-01-23T18:43:05.846ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3652024-01-23T18:43:05.846Z++ zpool list -o name
3662024-01-23T18:43:05.846Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3672024-01-23T18:43:05.846Z+ [[ -z '' ]]
3682024-01-23T18:43:05.846Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3692024-01-23T18:43:05.887Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3702024-01-23T18:43:05.888Z+ set +x
3712024-01-23T18:43:05.888ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3722024-01-23T18:43:05.888Z+ for ZPOOL in "${ZPOOLS[@]}"
3732024-01-23T18:43:05.888Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3742024-01-23T18:43:05.888ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3752024-01-23T18:43:05.889Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3762024-01-23T18:43:05.889Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3772024-01-23T18:43:05.889Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3782024-01-23T18:43:05.889Z0+0 records in
3792024-01-23T18:43:05.889Z0+0 records out
3802024-01-23T18:43:05.889Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3812024-01-23T18:43:05.889Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3822024-01-23T18:43:05.889Z+ set +x
3832024-01-23T18:43:05.890ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3842024-01-23T18:43:05.890Z++ zpool list -o name
3852024-01-23T18:43:05.890Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3862024-01-23T18:43:05.890Z+ [[ -z '' ]]
3872024-01-23T18:43:05.890Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882024-01-23T18:43:05.931Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3892024-01-23T18:43:05.931Z+ set +x
3902024-01-23T18:43:05.931ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3912024-01-23T18:43:05.931Z+ for ZPOOL in "${ZPOOLS[@]}"
3922024-01-23T18:43:05.931Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3932024-01-23T18:43:05.931ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3942024-01-23T18:43:05.932Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3952024-01-23T18:43:05.932Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3962024-01-23T18:43:05.932Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3972024-01-23T18:43:05.932Z0+0 records in
3982024-01-23T18:43:05.932Z0+0 records out
3992024-01-23T18:43:05.932Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4002024-01-23T18:43:05.932Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4012024-01-23T18:43:05.932Z+ set +x
4022024-01-23T18:43:05.932ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
4032024-01-23T18:43:05.933Z++ zpool list -o name
4042024-01-23T18:43:05.933Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4052024-01-23T18:43:05.933Z+ [[ -z '' ]]
4062024-01-23T18:43:05.933Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4072024-01-23T18:43:05.974Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4082024-01-23T18:43:05.974Z+ set +x
4092024-01-23T18:43:05.974ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4102024-01-23T18:43:05.975Z+ for ZPOOL in "${ZPOOLS[@]}"
4112024-01-23T18:43:05.975Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4122024-01-23T18:43:05.975ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4132024-01-23T18:43:05.975Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4142024-01-23T18:43:05.975Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4152024-01-23T18:43:05.975Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4162024-01-23T18:43:05.975Z0+0 records in
4172024-01-23T18:43:05.976Z0+0 records out
4182024-01-23T18:43:05.976Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4192024-01-23T18:43:05.976Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4202024-01-23T18:43:05.976Z+ set +x
4212024-01-23T18:43:05.976ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4222024-01-23T18:43:05.976Z++ zpool list -o name
4232024-01-23T18:43:05.976Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4242024-01-23T18:43:05.976Z+ [[ -z '' ]]
4252024-01-23T18:43:05.976Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4262024-01-23T18:43:06.019Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4272024-01-23T18:43:06.019Z+ set +x
4282024-01-23T18:43:06.019ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4292024-01-23T18:43:06.019Z+ for ZPOOL in "${ZPOOLS[@]}"
4302024-01-23T18:43:06.019Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4312024-01-23T18:43:06.020ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4322024-01-23T18:43:06.020Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4332024-01-23T18:43:06.020Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4342024-01-23T18:43:06.020Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4352024-01-23T18:43:06.020Z0+0 records in
4362024-01-23T18:43:06.020Z0+0 records out
4372024-01-23T18:43:06.020Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4382024-01-23T18:43:06.020Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4392024-01-23T18:43:06.020Z+ set +x
4402024-01-23T18:43:06.020ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4412024-01-23T18:43:06.020Z++ zpool list -o name
4422024-01-23T18:43:06.021Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4432024-01-23T18:43:06.021Z+ [[ -z '' ]]
4442024-01-23T18:43:06.021Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4452024-01-23T18:43:06.063Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4462024-01-23T18:43:06.063Z+ set +x
4472024-01-23T18:43:06.064ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4482024-01-23T18:43:06.064Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4492024-01-23T18:43:06.064Z+ readarray -t ZPOOLS
4502024-01-23T18:43:06.064Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4512024-01-23T18:43:06.065Z++ sed 's/[ ",]//g'
4522024-01-23T18:43:06.065Z+ for ZPOOL in "${ZPOOLS[@]}"
4532024-01-23T18:43:06.065Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4542024-01-23T18:43:06.065ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4552024-01-23T18:43:06.066Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4562024-01-23T18:43:06.066Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4572024-01-23T18:43:06.066Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4582024-01-23T18:43:06.066Z0+0 records in
4592024-01-23T18:43:06.066Z0+0 records out
4602024-01-23T18:43:06.066Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4612024-01-23T18:43:06.067Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4622024-01-23T18:43:06.067Z+ set +x
4632024-01-23T18:43:06.067ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4642024-01-23T18:43:06.067Z++ zpool list -o name
4652024-01-23T18:43:06.067Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4662024-01-23T18:43:06.116Z+ [[ -z '' ]]
4672024-01-23T18:43:06.116Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4682024-01-23T18:43:06.116Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4692024-01-23T18:43:06.116Z+ set +x
4702024-01-23T18:43:06.116ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4712024-01-23T18:43:06.116Z+ for ZPOOL in "${ZPOOLS[@]}"
4722024-01-23T18:43:06.117Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4732024-01-23T18:43:06.117ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4742024-01-23T18:43:06.117Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4752024-01-23T18:43:06.117Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4762024-01-23T18:43:06.117Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4772024-01-23T18:43:06.117Z0+0 records in
4782024-01-23T18:43:06.117Z0+0 records out
4792024-01-23T18:43:06.117Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4802024-01-23T18:43:06.117Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4812024-01-23T18:43:06.117Z+ set +x
4822024-01-23T18:43:06.118ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4832024-01-23T18:43:06.118Z++ zpool list -o name
4842024-01-23T18:43:06.118Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4852024-01-23T18:43:06.166Z+ [[ -z '' ]]
4862024-01-23T18:43:06.166Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4872024-01-23T18:43:06.166Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4882024-01-23T18:43:06.167Z+ set +x
4892024-01-23T18:43:06.167ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4902024-01-23T18:43:06.167Z+ [[ zone == \z\o\n\e ]]
4912024-01-23T18:43:06.167Z+ ensure_simulated_links igb0
4922024-01-23T18:43:06.167Z+ local PHYSICAL_LINK=igb0
4932024-01-23T18:43:06.167Z+ INDICES=("0" "1")
4942024-01-23T18:43:06.168Z+ for I in "${INDICES[@]}"
4952024-01-23T18:43:06.168Z++ get_simnet_name_if_exists net0
4962024-01-23T18:43:06.168Z++ dladm show-simnet -p -o LINK net0
4972024-01-23T18:43:06.168Z++ echo ''
4982024-01-23T18:43:06.168Z+ [[ -z '' ]]
4992024-01-23T18:43:06.168Z+ dladm create-simnet -t net0
5002024-01-23T18:43:06.218Z+ dladm create-simnet -t sc0_0
5012024-01-23T18:43:06.218Z+ dladm modify-simnet -t -p net0 sc0_0
5022024-01-23T18:43:06.218Z+ dladm set-linkprop -p mtu=9000 sc0_0
5032024-01-23T18:43:06.219Z+ success 'Simnet net0/sc0_0 exists'
5042024-01-23T18:43:06.219Z+ set +x
5052024-01-23T18:43:06.219ZSimnet net0/sc0_0 exists
5062024-01-23T18:43:06.220Z+ for I in "${INDICES[@]}"
5072024-01-23T18:43:06.220Z++ get_simnet_name_if_exists net1
5082024-01-23T18:43:06.220Z++ dladm show-simnet -p -o LINK net1
5092024-01-23T18:43:06.220Z++ echo ''
5102024-01-23T18:43:06.220Z+ [[ -z '' ]]
5112024-01-23T18:43:06.220Z+ dladm create-simnet -t net1
5122024-01-23T18:43:06.220Z+ dladm create-simnet -t sc1_0
5132024-01-23T18:43:06.221Z+ dladm modify-simnet -t -p net1 sc1_0
5142024-01-23T18:43:06.221Z+ dladm set-linkprop -p mtu=9000 sc1_0
5152024-01-23T18:43:06.221Z+ success 'Simnet net1/sc1_0 exists'
5162024-01-23T18:43:06.222Z+ set +x
5172024-01-23T18:43:06.222ZSimnet net1/sc1_0 exists
5182024-01-23T18:43:06.223Z++ get_vnic_name_if_exists sc0_1
5192024-01-23T18:43:06.223Z++ dladm show-vnic -p -o LINK sc0_1
5202024-01-23T18:43:06.223Z++ echo ''
5212024-01-23T18:43:06.223Z+ [[ -z '' ]]
5222024-01-23T18:43:06.223Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5232024-01-23T18:43:06.283Z+ [[ -v PROMISC_FILT_OFF ]]
5242024-01-23T18:43:06.283Z+ success 'Vnic sc0_1 exists'
5252024-01-23T18:43:06.283Z+ set +x
5262024-01-23T18:43:06.283ZVnic sc0_1 exists
5272024-01-23T18:43:06.284Z+ warn_if_no_proxy_arp
5282024-01-23T18:43:06.284Z+ [[ -v PXA_START ]]
5292024-01-23T18:43:06.284Z+ [[ -v PXA_END ]]
5302024-01-23T18:43:06.284Z+ ensure_softnpu_zone
5312024-01-23T18:43:06.285Z+ zoneadm list
5322024-01-23T18:43:06.285Z+ grep -q sidecar_softnpu
5332024-01-23T18:43:06.285Z+ [[ -f out/npuzone/npuzone ]]
5342024-01-23T18:43:06.285Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit e3ea4b495ba0a71801ded0776ae4bbd31df57e26 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5352024-01-23T18:43:06.285Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
 
5362024-01-23T18:43:23.136ZError: No response to keepalive - Terminating session
 
5372024-01-23T18:43:23.331Zdownloading softnpu ...done
5382024-01-23T18:43:23.372Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/libsidecar_lite.so.sha256.txt
5392024-01-23T18:43:30.588Zdownloading asic_program.so ...done
5402024-01-23T18:43:30.629Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/scadm.sha256.txt
 
5412024-01-23T18:43:32.512ZError: No response de-activating SOL payload
5422024-01-23T18:43:33.818Z[SOL Session operational. Use ~? for help]
 
5432024-01-23T18:43:38.707Zdownloading scadm ...done
5442024-01-23T18:43:38.748Zcreating port tfportrear0_0 for sc0_0
5452024-01-23T18:43:38.749Zcreating port tfportqsfp0_0 for sc0_1
5462024-01-23T18:43:38.749Zsoftnpu zone setup
5472024-01-23T18:43:38.749Zconfigure zone
5482024-01-23T18:43:38.749Zinstall zone
5492024-01-23T18:43:39.796Zboot zone
5502024-01-23T18:43:42.904Z[sidecar_softnpu] svcs milestone/network
5512024-01-23T18:43:42.945ZSTATE STIME FMRI
5522024-01-23T18:43:42.945Zonline 18:43:41 svc:/milestone/network:default
5532024-01-23T18:43:42.946Z[sidecar_softnpu] /softnpu/npu start
5542024-01-23T18:43:42.986Z
5552024-01-23T18:43:42.986Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5562024-01-23T18:43:42.986Z+ GATEWAY_IP=10.151.2.100
5572024-01-23T18:43:42.986Z+ echo 'Using 10.151.2.100 as gateway ip'
5582024-01-23T18:43:42.986ZUsing 10.151.2.100 as gateway ip
5592024-01-23T18:43:42.986Z+ [[ ! -v GATEWAY_MAC ]]
5602024-01-23T18:43:42.986Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5612024-01-23T18:43:42.986Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5622024-01-23T18:43:42.987Z+++ echo 18:c0:4d:0d:a0:2a
5632024-01-23T18:43:42.987Z+ COUNT=1
5642024-01-23T18:43:42.987Z+ [[ 1 -ne 1 ]]
5652024-01-23T18:43:42.987Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5662024-01-23T18:43:42.987ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5672024-01-23T18:43:42.987Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5682024-01-23T18:43:42.987Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5692024-01-23T18:43:43.026Z+ PXA_MAC=a8:e1:de:01:70:1d
5702024-01-23T18:43:43.026Z+ [[ -v PXA_START ]]
5712024-01-23T18:43:43.027Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5722024-01-23T18:43:43.027Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5732024-01-23T18:43:43.027Z+ z_scadm dump-state
5742024-01-23T18:43:43.027Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5752024-01-23T18:43:43.187Zlocal v6:
5762024-01-23T18:43:43.187Zlocal v4:
5772024-01-23T18:43:43.188Zrouter v6:
5782024-01-23T18:43:43.188Zrouter v4_idx:
5792024-01-23T18:43:43.188Zrouter v4_routes:
5802024-01-23T18:43:43.188Zresolver v4:
5812024-01-23T18:43:43.188Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5822024-01-23T18:43:43.189Zresolver v6:
5832024-01-23T18:43:43.189Znat v4:
5842024-01-23T18:43:43.189Znat_v6:
5852024-01-23T18:43:43.189Zport_mac:
5862024-01-23T18:43:43.189Zproxy arp:
5872024-01-23T18:43:43.189Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5882024-01-23T18:43:43.190Z+ success 'softnpu zone exists'
5892024-01-23T18:43:43.190Z+ set +x
5902024-01-23T18:43:43.190Zsoftnpu zone exists
5912024-01-23T18:43:43.190Z
5922024-01-23T18:43:43.190Zreal 37.582691667
5932024-01-23T18:43:43.190Zuser 0.721017018
5942024-01-23T18:43:43.190Zsys 1.585287625
5952024-01-23T18:43:43.190Ztrap 0.000105017
5962024-01-23T18:43:43.190Ztflt 0.000097916
5972024-01-23T18:43:43.191Zdflt 0.001527965
5982024-01-23T18:43:43.191Zkflt 0.000000000
5992024-01-23T18:43:43.191Zlock 29:19.448363483
6002024-01-23T18:43:43.191Zslp 1:52.382969658
6012024-01-23T18:43:43.191Zlat 0.058002244
6022024-01-23T18:43:43.192Zstop 0.348767092
6032024-01-23T18:43:43.192Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
6042024-01-23T18:43:43.233Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
6052024-01-23T18:43:43.233Z+ SILO_NAME=recovery
6062024-01-23T18:43:43.233Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
6072024-01-23T18:43:43.233Z+ EXTERNAL_DNS_DOMAIN=oxide.test
6082024-01-23T18:43:43.233Z+ sed -i~ '
6092024-01-23T18:43:43.233Z /^external_dns_ips/c\
6102024-01-23T18:43:43.233Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6112024-01-23T18:43:43.234Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6122024-01-23T18:43:43.234Z /^first/c\
6132024-01-23T18:43:43.234Zfirst = "10.151.2.164"
6142024-01-23T18:43:43.234Z /^last/c\
6152024-01-23T18:43:43.234Zlast = "10.151.2.173"
6162024-01-23T18:43:43.234Z }
6172024-01-23T18:43:43.234Z /^\[rack_network_config/,/^$/ {
6182024-01-23T18:43:43.234Z /^infra_ip_first/c\
6192024-01-23T18:43:43.234Zinfra_ip_first = "10.151.2.174"
6202024-01-23T18:43:43.234Z /^infra_ip_last/c\
6212024-01-23T18:43:43.235Zinfra_ip_last = "10.151.2.174"
6222024-01-23T18:43:43.235Z }
6232024-01-23T18:43:43.235Z /^\[\[rack_network_config.ports/,/^$/ {
6242024-01-23T18:43:43.235Z /^routes/c\
6252024-01-23T18:43:43.235Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6262024-01-23T18:43:43.235Z /^addresses/c\
6272024-01-23T18:43:43.235Zaddresses = \["10.151.2.174/32"\]
6282024-01-23T18:43:43.235Z }
6292024-01-23T18:43:43.235Z' pkg/config-rss.toml
6302024-01-23T18:43:43.235Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6312024-01-23T18:43:43.235Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6322024-01-23T18:43:43.236Z+++ pkg/config-rss.toml Tue Jan 23 18:43:43 2024
6332024-01-23T18:43:43.236Z@@ -41,7 +41,7 @@
6342024-01-23T18:43:43.236Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6352024-01-23T18:43:43.236Z # must be contained in one of the "internal services" IP Pool ranges listed
6362024-01-23T18:43:43.236Z # below.
6372024-01-23T18:43:43.236Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6382024-01-23T18:43:43.236Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6392024-01-23T18:43:43.236Z
6402024-01-23T18:43:43.236Z # Initial TLS certificates for the external API
6412024-01-23T18:43:43.236Z #
6422024-01-23T18:43:43.237Z@@ -77,8 +77,8 @@
6432024-01-23T18:43:43.237Z #
6442024-01-23T18:43:43.237Z # For more on this and what to put here, see docs/how-to-run.adoc.
6452024-01-23T18:43:43.237Z [[internal_services_ip_pool_ranges]]
6462024-01-23T18:43:43.237Z-first = "192.168.1.20"
6472024-01-23T18:43:43.237Z-last = "192.168.1.29"
6482024-01-23T18:43:43.237Z+first = "10.151.2.164"
6492024-01-23T18:43:43.237Z+last = "10.151.2.173"
6502024-01-23T18:43:43.237Z
6512024-01-23T18:43:43.237Z # TODO - this configuration is subject to change going forward. Ultimately these
6522024-01-23T18:43:43.237Z # parameters should be provided to the control plane via wicket, but we need to
6532024-01-23T18:43:43.238Z@@ -92,8 +92,8 @@
6542024-01-23T18:43:43.238Z # A range of IP addresses used by Boundary Services on the external network. In
6552024-01-23T18:43:43.238Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6562024-01-23T18:43:43.238Z # With softnpu, only one address is used.
6572024-01-23T18:43:43.238Z-infra_ip_first = "192.168.1.30"
6582024-01-23T18:43:43.238Z-infra_ip_last = "192.168.1.30"
6592024-01-23T18:43:43.238Z+infra_ip_first = "10.151.2.174"
6602024-01-23T18:43:43.238Z+infra_ip_last = "10.151.2.174"
6612024-01-23T18:43:43.238Z
6622024-01-23T18:43:43.238Z # Configurations for BGP routers to run on the scrimlets.
6632024-01-23T18:43:43.239Z bgp = []
6642024-01-23T18:43:43.239Z@@ -101,9 +101,9 @@
6652024-01-23T18:43:43.239Z # You can configure multiple uplinks by repeating the following stanza
6662024-01-23T18:43:43.239Z [[rack_network_config.ports]]
6672024-01-23T18:43:43.239Z # Routes associated with this port.
6682024-01-23T18:43:43.239Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6692024-01-23T18:43:43.239Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6702024-01-23T18:43:43.239Z # Addresses associated with this port.
6712024-01-23T18:43:43.239Z-addresses = ["192.168.1.30/32"]
6722024-01-23T18:43:43.239Z+addresses = ["10.151.2.174/32"]
6732024-01-23T18:43:43.240Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6742024-01-23T18:43:43.240Z port = "qsfp0"
6752024-01-23T18:43:43.240Z # The speed of this port.
6762024-01-23T18:43:43.240Z+ true
6772024-01-23T18:43:43.240Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6782024-01-23T18:43:43.240Za pkg/config-rss.toml 7K
6792024-01-23T18:43:43.241Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6802024-01-23T18:43:43.241Z+ TLS_NAME=recovery.sys.oxide.test
6812024-01-23T18:43:43.241Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6822024-01-23T18:43:43.531Z..+.+......+............+...+..+.......+......+..+...+...+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.+.....+...+......+......+...+......+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+......+............+..+.+.....+......+..........+......+..+...+.......+........+...+.+.....+......+...+....+...+..............+..........+...........+..........+..................+..+...+...................+...........+....+..+...+...+....+...............+...+.....+...............+.+.....+.+.........+..+.........+...+.............+..............+.+...+..+.......+...............+........+.......+............+........+.+..............+..........+...............+.....+......................+...........+............+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6832024-01-23T18:43:44.338Z...............+.....+.......+.....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+............+..+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+.+..............+.+....................+.+......+..+...+....+.....+.........+....+.....+.....................+............+.......+...+...+.....+.......+..+...+....+.....+......+...+.+...+...........+....+............+..............+.........+..........+.....+.+..............+............+.+..+..............................................+..+.............+..+...+....+...+..+.........................+............+.....................+........................+........+......+...............+.......+...+......+...+........+...+..........+..+...+......+.+.....+.......+............+...+......+............+....................+...+....+..+.........+.+.....+.+........................+...+............+...+...+.....+...+....+..+....+......+..............+.......+........+..........+............+.....+.......+.....+......+.+...............+........+.+......+.....+...+...+............+............+......+.......+...+.........+........+......+....+...........+.........+......+.........+.+.........+.....+..........+......+..+.......+..+......+....+..............+.+..................+......+............+.....+.......+..+.+..+.............+.................+.......+.....+...+.+.................+..................+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6842024-01-23T18:43:44.380Z-----
6852024-01-23T18:43:44.380Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6862024-01-23T18:43:44.380Za pkg/initial-tls-cert.pem 2K
6872024-01-23T18:43:44.380Za pkg/initial-tls-key.pem 4K
6882024-01-23T18:43:44.380Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6892024-01-23T18:43:44.380Z+ rmdir pkg
6902024-01-23T18:43:44.380Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6912024-01-23T18:43:44.380Z+ pfexec mkdir -p /usr/oxide
6922024-01-23T18:43:44.380Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6932024-01-23T18:43:45.599Z+ pfexec chmod +x /usr/oxide/oxide
6942024-01-23T18:43:45.640Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6952024-01-23T18:44:03.456Z+ OMICRON_NO_UNINSTALL=1
6962024-01-23T18:44:03.456Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6972024-01-23T18:44:03.590ZJan 23 18:44:03.577 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6982024-01-23T18:44:03.590ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6992024-01-23T18:44:03.590ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7002024-01-23T18:44:03.590ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7012024-01-23T18:44:03.590ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
7022024-01-23T18:44:03.590ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7032024-01-23T18:44:03.590ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7042024-01-23T18:44:03.590ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7052024-01-23T18:44:03.590ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7062024-01-23T18:44:03.590ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7072024-01-23T18:44:03.590ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7082024-01-23T18:44:03.591ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7092024-01-23T18:44:03.591ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7102024-01-23T18:44:03.591ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7112024-01-23T18:44:03.591ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
7122024-01-23T18:44:03.591ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7132024-01-23T18:44:03.591ZJan 23 18:44:03.579 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7142024-01-23T18:44:03.728ZJan 23 18:44:03.833 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7152024-01-23T18:44:03.770ZJan 23 18:44:03.875 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7162024-01-23T18:44:03.836ZJan 23 18:44:03.941 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7172024-01-23T18:44:03.974Z
7182024-01-23T18:44:03.974Zreal 0.504573081
7192024-01-23T18:44:03.974Zuser 0.102619923
7202024-01-23T18:44:03.975Zsys 1.974459101
7212024-01-23T18:44:03.975Ztrap 0.000001409
7222024-01-23T18:44:03.975Ztflt 0.000124740
7232024-01-23T18:44:03.975Zdflt 0.000895668
7242024-01-23T18:44:03.975Zkflt 0.000007649
7252024-01-23T18:44:03.975Zlock 45.401434155
7262024-01-23T18:44:03.975Zslp 0.718526390
7272024-01-23T18:44:03.975Zlat 0.002905471
7282024-01-23T18:44:03.976Zstop 0.190193932
7292024-01-23T18:44:03.976Z+ retry=0
7302024-01-23T18:44:03.976Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312024-01-23T18:44:04.017Z+ [[ 0 -gt 30 ]]
7322024-01-23T18:44:04.017Z+ sleep 1
7332024-01-23T18:44:04.981Z+ retry=1
7342024-01-23T18:44:04.981Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352024-01-23T18:44:05.024Z+ [[ 1 -gt 30 ]]
7362024-01-23T18:44:05.024Z+ sleep 1
7372024-01-23T18:44:05.994Z+ retry=2
7382024-01-23T18:44:05.994Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7392024-01-23T18:44:06.036Z+ [[ 2 -gt 30 ]]
7402024-01-23T18:44:06.036Z+ sleep 1
7412024-01-23T18:44:07.008Z+ retry=3
7422024-01-23T18:44:07.009Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7432024-01-23T18:44:07.048Z+ [[ 3 -gt 30 ]]
7442024-01-23T18:44:07.048Z+ sleep 1
7452024-01-23T18:44:08.022Z+ retry=4
7462024-01-23T18:44:08.022Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7472024-01-23T18:44:33.204Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7482024-01-23T18:44:33.247Zlocal v6:
7492024-01-23T18:44:33.247Zfd00:99::1
7502024-01-23T18:44:33.248Zfe80::7c9c:1fff:fee0:bd59
7512024-01-23T18:44:33.248Zlocal v4:
7522024-01-23T18:44:33.248Z10.151.2.174
7532024-01-23T18:44:33.248Zrouter v6:
7542024-01-23T18:44:33.248Zfd00:1122:3344:2::/64 -> fe80::b0a8:8aff:fe77:9120 (1)
7552024-01-23T18:44:33.249Zfd00:1122:3344:1::/64 -> fe80::b0a8:8aff:fe77:9120 (1)
7562024-01-23T18:44:33.249Zfdb0:18c0:4d0d:a02a::/64 -> fe80::b0a8:8aff:fe77:9120 (1)
7572024-01-23T18:44:33.249Zfd00:1122:3344:3::/64 -> fe80::b0a8:8aff:fe77:9120 (1)
7582024-01-23T18:44:33.249Zfd00:1122:3344:101::/64 -> fe80::b0a8:8aff:fe77:9120 (1)
7592024-01-23T18:44:33.249Zrouter v4_idx:
7602024-01-23T18:44:33.249Z0.0.0.0/0 -> 0
7612024-01-23T18:44:33.250Zrouter v4_routes:
7622024-01-23T18:44:33.250Z0 -> 10.151.2.100 (2)
7632024-01-23T18:44:33.250Zresolver v4:
7642024-01-23T18:44:33.250Z10.151.2.174 -> f2:a4:88:c3:52:e6
7652024-01-23T18:44:33.250Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7662024-01-23T18:44:33.250Zresolver v6:
7672024-01-23T18:44:33.251Zfe80::b0a8:8aff:fe77:9120 -> b2:a8:8a:77:91:20
7682024-01-23T18:44:33.251Zfe80::7c9c:1fff:fee0:bd59 -> 7e:9c:1f:e0:bd:59
7692024-01-23T18:44:33.251Znat v4:
7702024-01-23T18:44:33.251Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:e7:89
7712024-01-23T18:44:33.251Znat_v6:
7722024-01-23T18:44:33.252Zport_mac:
7732024-01-23T18:44:33.252Z1: a8:40:25:63:eb:49
7742024-01-23T18:44:33.252Z1000: a8:40:25:63:eb:48
7752024-01-23T18:44:33.252Zproxy arp:
7762024-01-23T18:44:33.252Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7772024-01-23T18:44:33.252Z+ retry=0
7782024-01-23T18:44:33.253Z+++ zoneadm list -n
7792024-01-23T18:44:33.253Z+++ grep oxz_ntp
7802024-01-23T18:44:33.253Z++ pfexec svcs -z oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c -Hostate oxide/ntp
7812024-01-23T18:44:33.302Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7822024-01-23T18:44:33.302Z++ true
7832024-01-23T18:44:33.302Z+ [[ '' != online ]]
7842024-01-23T18:44:33.302Z+ [[ 0 -gt 60 ]]
7852024-01-23T18:44:33.303Z+ sleep 1
7862024-01-23T18:44:34.263Z+ retry=1
7872024-01-23T18:44:34.303Z+++ zoneadm list -n
7882024-01-23T18:44:34.303Z+++ grep oxz_ntp
7892024-01-23T18:44:34.303Z++ pfexec svcs -z oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c -Hostate oxide/ntp
7902024-01-23T18:44:34.303Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7912024-01-23T18:44:34.303Z++ true
7922024-01-23T18:44:34.303Z+ [[ '' != online ]]
7932024-01-23T18:44:34.303Z+ [[ 1 -gt 60 ]]
7942024-01-23T18:44:34.303Z+ sleep 1
7952024-01-23T18:44:35.298Z+ retry=2
7962024-01-23T18:44:35.337Z+++ zoneadm list -n
7972024-01-23T18:44:35.338Z+++ grep oxz_ntp
7982024-01-23T18:44:35.338Z++ pfexec svcs -z oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c -Hostate oxide/ntp
7992024-01-23T18:44:35.338Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8002024-01-23T18:44:35.338Z++ true
8012024-01-23T18:44:35.338Z+ [[ '' != online ]]
8022024-01-23T18:44:35.338Z+ [[ 2 -gt 60 ]]
8032024-01-23T18:44:35.338Z+ sleep 1
8042024-01-23T18:44:36.314Z+ retry=3
8052024-01-23T18:44:36.353Z+++ zoneadm list -n
8062024-01-23T18:44:36.354Z+++ grep oxz_ntp
8072024-01-23T18:44:36.354Z++ pfexec svcs -z oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c -Hostate oxide/ntp
8082024-01-23T18:44:36.354Z+ [[ disabled != online ]]
8092024-01-23T18:44:36.354Z+ [[ 3 -gt 60 ]]
8102024-01-23T18:44:36.354Z+ sleep 1
8112024-01-23T18:44:37.335Z+ retry=4
8122024-01-23T18:44:37.377Z+++ zoneadm list -n
8132024-01-23T18:44:37.377Z+++ grep oxz_ntp
8142024-01-23T18:44:37.377Z++ pfexec svcs -z oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c -Hostate oxide/ntp
8152024-01-23T18:44:37.377Z+ [[ disabled != online ]]
8162024-01-23T18:44:37.377Z+ [[ 4 -gt 60 ]]
8172024-01-23T18:44:37.377Z+ sleep 1
8182024-01-23T18:44:38.359Z+ retry=5
8192024-01-23T18:44:38.400Z+++ zoneadm list -n
8202024-01-23T18:44:38.401Z+++ grep oxz_ntp
8212024-01-23T18:44:38.401Z++ pfexec svcs -z oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c -Hostate oxide/ntp
8222024-01-23T18:44:38.401Z+ [[ disabled != online ]]
8232024-01-23T18:44:38.401Z+ [[ 5 -gt 60 ]]
8242024-01-23T18:44:38.401Z+ sleep 1
8252024-01-23T18:44:39.379Z+ retry=6
8262024-01-23T18:44:39.421Z+++ zoneadm list -n
8272024-01-23T18:44:39.421Z+++ grep oxz_ntp
8282024-01-23T18:44:39.422Z++ pfexec svcs -z oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c -Hostate oxide/ntp
8292024-01-23T18:44:39.422Z+ [[ disabled != online ]]
8302024-01-23T18:44:39.422Z+ [[ 6 -gt 60 ]]
8312024-01-23T18:44:39.422Z+ sleep 1
8322024-01-23T18:44:40.407Z+ retry=7
8332024-01-23T18:44:40.447Z+++ zoneadm list -n
8342024-01-23T18:44:40.447Z+++ grep oxz_ntp
8352024-01-23T18:44:40.447Z++ pfexec svcs -z oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c -Hostate oxide/ntp
8362024-01-23T18:44:40.447Z+ [[ disabled != online ]]
8372024-01-23T18:44:40.447Z+ [[ 7 -gt 60 ]]
8382024-01-23T18:44:40.447Z+ sleep 1
8392024-01-23T18:44:41.420Z+ retry=8
8402024-01-23T18:44:41.459Z+++ zoneadm list -n
8412024-01-23T18:44:41.459Z+++ grep oxz_ntp
8422024-01-23T18:44:41.459Z++ pfexec svcs -z oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c -Hostate oxide/ntp
8432024-01-23T18:44:41.459Z+ [[ disabled != online ]]
8442024-01-23T18:44:41.459Z+ [[ 8 -gt 60 ]]
8452024-01-23T18:44:41.459Z+ sleep 1
8462024-01-23T18:44:42.443Z+ retry=9
8472024-01-23T18:44:42.488Z+++ zoneadm list -n
8482024-01-23T18:44:42.488Z+++ grep oxz_ntp
8492024-01-23T18:44:42.488Z++ pfexec svcs -z oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c -Hostate oxide/ntp
8502024-01-23T18:44:42.489Z+ [[ disabled != online ]]
8512024-01-23T18:44:42.489Z+ [[ 9 -gt 60 ]]
8522024-01-23T18:44:42.489Z+ sleep 1
 
8532024-01-23T18:44:43.045ZJan 23 18:44:42 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8542024-01-23T18:44:43.465Z+ retry=10
8552024-01-23T18:44:43.523Z+++ zoneadm list -n
8562024-01-23T18:44:43.523Z+++ grep oxz_ntp
8572024-01-23T18:44:43.523Z++ pfexec svcs -z oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c -Hostate oxide/ntp
8582024-01-23T18:44:43.523Z+ [[ online != online ]]
8592024-01-23T18:44:43.523Z+ echo 'Waited for chrony: 10s'
8602024-01-23T18:44:43.523Z+ export RUST_BACKTRACE=1
8612024-01-23T18:44:43.523Z+ RUST_BACKTRACE=1
8622024-01-23T18:44:43.523ZWaited for chrony: 10s
8632024-01-23T18:44:43.524Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8642024-01-23T18:44:43.524Z++ ./tests/bootstrap
8652024-01-23T18:44:43.524ZUsing external DNS server at 10.151.2.164:53
8662024-01-23T18:44:43.575Z2024-01-23 18:44:43.621534723 UTC: attempting to log into API
8672024-01-23T18:44:58.551Z2024-01-23 18:44:58.164539402 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8682024-01-23T18:44:59.553Z2024-01-23 18:44:59.165609206 UTC: attempting to log into API
8692024-01-23T18:45:14.593Z2024-01-23 18:45:14.199180618 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8702024-01-23T18:45:15.588Z2024-01-23 18:45:15.200483808 UTC: attempting to log into API
8712024-01-23T18:45:31.290Z2024-01-23 18:45:30.903725509 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8722024-01-23T18:45:32.295Z2024-01-23 18:45:31.904967858 UTC: attempting to log into API
8732024-01-23T18:45:47.325Z2024-01-23 18:45:46.938441157 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8742024-01-23T18:45:48.327Z2024-01-23 18:45:47.939525262 UTC: attempting to log into API
8752024-01-23T18:46:03.363Z2024-01-23 18:46:03.586369563 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8762024-01-23T18:46:04.365Z2024-01-23 18:46:04.584466521 UTC: attempting to log into API
8772024-01-23T18:46:19.396Z2024-01-23 18:46:19.577565229 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8782024-01-23T18:46:20.393Z2024-01-23 18:46:20.576019855 UTC: attempting to log into API
 
8792024-01-23T18:46:22.583ZError: No response to keepalive - Terminating session
8802024-01-23T18:46:32.499ZError: No response de-activating SOL payload
8812024-01-23T18:46:33.891Z[SOL Session operational. Use ~? for help]
 
8822024-01-23T18:46:35.427Z2024-01-23 18:46:35.569306556 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8832024-01-23T18:46:36.430Z2024-01-23 18:46:36.567574398 UTC: attempting to log into API
8842024-01-23T18:46:51.462Z2024-01-23 18:46:51.560452537 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8852024-01-23T18:46:52.463Z2024-01-23 18:46:52.558651626 UTC: attempting to log into API
8862024-01-23T18:47:07.513Z2024-01-23 18:47:07.550819380 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8872024-01-23T18:47:08.496Z2024-01-23 18:47:08.549189025 UTC: attempting to log into API
8882024-01-23T18:47:23.530Z2024-01-23 18:47:23.540271673 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8892024-01-23T18:47:24.532Z2024-01-23 18:47:24.538576127 UTC: attempting to log into API
8902024-01-23T18:47:39.563Z2024-01-23 18:47:39.541037309 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8912024-01-23T18:47:40.564Z2024-01-23 18:47:40.542347309 UTC: attempting to log into API
8922024-01-23T18:47:55.608Z2024-01-23 18:47:55.582767138 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8932024-01-23T18:47:56.608Z2024-01-23 18:47:56.583847193 UTC: attempting to log into API
8942024-01-23T18:48:11.645Z2024-01-23 18:48:11.622539159 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8952024-01-23T18:48:12.646Z2024-01-23 18:48:12.623249155 UTC: attempting to log into API
8962024-01-23T18:48:27.697Z2024-01-23 18:48:27.673739883 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8972024-01-23T18:48:28.697Z2024-01-23 18:48:28.674890860 UTC: attempting to log into API
8982024-01-23T18:48:43.735Z2024-01-23 18:48:43.706678642 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8992024-01-23T18:48:44.732Z2024-01-23 18:48:44.707785689 UTC: attempting to log into API
9002024-01-23T18:48:49.772Z2024-01-23 18:48:49.748443321 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: tcp connect error: Connection refused (os error 146)
9012024-01-23T18:48:50.778Z2024-01-23 18:48:50.749947997 UTC: attempting to log into API
9022024-01-23T18:48:50.833Z2024-01-23 18:48:50.783406061 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: tcp connect error: Connection refused (os error 146)
9032024-01-23T18:48:51.816Z2024-01-23 18:48:51.784577669 UTC: attempting to log into API
9042024-01-23T18:48:52.578Z2024-01-23 18:48:52.553627396 UTC: login succeeded
9052024-01-23T18:48:52.627Zwaiting for nexus to come up...
9062024-01-23T18:48:52.681Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9072024-01-23T18:48:52.857Zsetting up quotas...
9082024-01-23T18:48:52.903Zensuring datasets are ready...
9092024-01-23T18:48:55.670Zlet's roll.
9102024-01-23T18:48:55.708Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
9112024-01-23T18:48:55.709ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9122024-01-23T18:48:55.709ZOXIDE_TOKEN=oxide-token-702272d749eb2f27472d110d42b62f0dadac997b'
9132024-01-23T18:48:55.709Z++ OXIDE_HOST=https://recovery.sys.oxide.test
9142024-01-23T18:48:55.709Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9152024-01-23T18:48:55.709Z++ OXIDE_TOKEN=oxide-token-702272d749eb2f27472d110d42b62f0dadac997b
9162024-01-23T18:48:55.709Z+ export OXIDE_HOST OXIDE_TOKEN
9172024-01-23T18:48:55.709Z+ retry=0
9182024-01-23T18:48:55.709Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9192024-01-23T18:48:55.780Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
9202024-01-23T18:48:56.071Z{"status":"ok"}success
9212024-01-23T18:48:56.071ZProject {
9222024-01-23T18:48:56.071Z description: "some images",
9232024-01-23T18:48:56.071Z id: 9ceb1303-834b-4733-a8f9-69d49610464c,
9242024-01-23T18:48:56.071Z name: Name(
9252024-01-23T18:48:56.071Z "images",
9262024-01-23T18:48:56.071Z ),
9272024-01-23T18:48:56.072Z time_created: 2024-01-23T18:48:55.829021Z,
9282024-01-23T18:48:56.072Z time_modified: 2024-01-23T18:48:55.829021Z,
9292024-01-23T18:48:56.072Z}
9302024-01-23T18:48:56.111Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
 
9312024-01-23T18:49:38.619ZError: No response to keepalive - Terminating session
9322024-01-23T18:49:48.536ZError: No response de-activating SOL payload
9332024-01-23T18:49:49.925Z[SOL Session operational. Use ~? for help]
9342024-01-23T18:52:38.674ZError: No response to keepalive - Terminating session
9352024-01-23T18:52:48.622ZError: No response de-activating SOL payload
9362024-01-23T18:52:50.035Z[SOL Session operational. Use ~? for help]
 
9372024-01-23T18:54:38.769Zdone!
9382024-01-23T18:54:38.959Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
9392024-01-23T18:54:39.121Zsuccess
9402024-01-23T18:54:39.121ZImage {
9412024-01-23T18:54:39.121Z block_size: ByteCount(
9422024-01-23T18:54:39.121Z 512,
9432024-01-23T18:54:39.121Z ),
9442024-01-23T18:54:39.121Z description: "debian 11 original base image",
9452024-01-23T18:54:39.121Z digest: None,
9462024-01-23T18:54:39.121Z id: 7624e98b-8a75-42dd-8576-5b72888de664,
9472024-01-23T18:54:39.121Z name: Name(
9482024-01-23T18:54:39.121Z "debian11",
9492024-01-23T18:54:39.122Z ),
9502024-01-23T18:54:39.122Z os: "debian",
9512024-01-23T18:54:39.122Z project_id: None,
9522024-01-23T18:54:39.122Z size: ByteCount(
9532024-01-23T18:54:39.122Z 2147483648,
9542024-01-23T18:54:39.122Z ),
9552024-01-23T18:54:39.122Z time_created: 2024-01-23T18:54:38.715934Z,
9562024-01-23T18:54:39.122Z time_modified: 2024-01-23T18:54:38.937843Z,
9572024-01-23T18:54:39.122Z url: None,
9582024-01-23T18:54:39.122Z version: "11",
9592024-01-23T18:54:39.122Z}
9602024-01-23T18:54:39.122Z+ rm ./tests/bootstrap
9612024-01-23T18:54:39.122Z+ for test_bin in tests/*
9622024-01-23T18:54:39.122Z+ ./tests/bootstrap-ae34befe45ae3da4
9632024-01-23T18:54:39.122Z
9642024-01-23T18:54:39.122Zrunning 0 tests
9652024-01-23T18:54:39.122Z
9662024-01-23T18:54:39.122Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9672024-01-23T18:54:39.122Z
9682024-01-23T18:54:39.122Z+ for test_bin in tests/*
9692024-01-23T18:54:39.122Z+ ./tests/end_to_end_tests-fe5f3f977d30e70c
9702024-01-23T18:54:39.122Z
9712024-01-23T18:54:39.122Zrunning 2 tests
9722024-01-23T18:54:39.318Ztest no_spoof::no_spoof_here ... ok
9732024-01-23T18:55:11.996Ztest instance_launch::instance_launch ... ok
9742024-01-23T18:55:11.997Z
9752024-01-23T18:55:11.997Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 32.91s
9762024-01-23T18:55:11.997Z
9772024-01-23T18:55:12.041Z+ _exit_trap
9782024-01-23T18:55:12.042Z+ local status=0
9792024-01-23T18:55:12.042Z+ [[ 0 -eq 0 ]]
9802024-01-23T18:55:12.042Z+ exit 0
9812024-01-23T18:55:12.042Zprocess exited: duration 735029 ms, exit code 0
 
9822024-01-23T18:55:12.170Zfound 66 output files
9832024-01-23T18:55:12.171Zuploading: /var/svc/log/oxide-sled-agent:default.log (1873036 bytes)
9842024-01-23T18:55:14.161Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9852024-01-23T18:55:14.161Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_a1ef6a3e-ac80-4791-a742-d8a3fe216fce/root/var/svc/log/oxide-clickhouse:default.log (891555 bytes)
9862024-01-23T18:55:16.879Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_a1ef6a3e-ac80-4791-a742-d8a3fe216fce/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 891555 -> 908984
9872024-01-23T18:55:18.007Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_a1ef6a3e-ac80-4791-a742-d8a3fe216fce/root/var/svc/log/oxide-clickhouse:default.log
9882024-01-23T18:55:18.007Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_a1ef6a3e-ac80-4791-a742-d8a3fe216fce/root/var/svc/log/oxide-zone-network-setup:default.log (1707 bytes)
9892024-01-23T18:55:19.176Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_a1ef6a3e-ac80-4791-a742-d8a3fe216fce/root/var/svc/log/oxide-zone-network-setup:default.log
9902024-01-23T18:55:19.177Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a4bf3c41-0811-4e14-ac6a-bb33f7cb5608/root/var/svc/log/oxide-cockroachdb:default.log (4482 bytes)
9912024-01-23T18:55:20.343Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a4bf3c41-0811-4e14-ac6a-bb33f7cb5608/root/var/svc/log/oxide-cockroachdb:default.log
9922024-01-23T18:55:20.344Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a4bf3c41-0811-4e14-ac6a-bb33f7cb5608/root/var/svc/log/oxide-zone-network-setup:default.log (1711 bytes)
9932024-01-23T18:55:21.510Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a4bf3c41-0811-4e14-ac6a-bb33f7cb5608/root/var/svc/log/oxide-zone-network-setup:default.log
9942024-01-23T18:55:21.511Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3dfe586e-e625-4813-89e4-1708a187a944/root/var/svc/log/oxide-crucible-agent:default.log (24362 bytes)
9952024-01-23T18:55:22.855Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3dfe586e-e625-4813-89e4-1708a187a944/root/var/svc/log/oxide-crucible-agent:default.log
9962024-01-23T18:55:22.894Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3dfe586e-e625-4813-89e4-1708a187a944/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9972024-01-23T18:55:24.387Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3dfe586e-e625-4813-89e4-1708a187a944/root/var/svc/log/oxide-crucible-downstairs:default.log
9982024-01-23T18:55:24.388Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3dfe586e-e625-4813-89e4-1708a187a944/root/var/svc/log/oxide-crucible-downstairs:downstairs-3e53a12f-85a7-4b8a-823f-57c346260c78.log (3051 bytes)
9992024-01-23T18:55:25.197Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3dfe586e-e625-4813-89e4-1708a187a944/root/var/svc/log/oxide-crucible-downstairs:downstairs-3e53a12f-85a7-4b8a-823f-57c346260c78.log
10002024-01-23T18:55:25.197Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3dfe586e-e625-4813-89e4-1708a187a944/root/var/svc/log/oxide-crucible-downstairs:downstairs-845e3fdf-bf84-437f-bff9-0d035e9cd321.log (5497 bytes)
10012024-01-23T18:55:26.397Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3dfe586e-e625-4813-89e4-1708a187a944/root/var/svc/log/oxide-crucible-downstairs:downstairs-845e3fdf-bf84-437f-bff9-0d035e9cd321.log
10022024-01-23T18:55:26.397Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_03b23ff8-daf2-4d73-9672-28011985d5c7/root/var/svc/log/oxide-external_dns:default.log (26946 bytes)
10032024-01-23T18:55:27.718Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_03b23ff8-daf2-4d73-9672-28011985d5c7/root/var/svc/log/oxide-external_dns:default.log
10042024-01-23T18:55:27.765Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b1bf6595-177b-4c78-bd39-164d60f7d632/root/var/svc/log/oxide-internal_dns:default.log (103159 bytes)
10052024-01-23T18:55:31.124Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b1bf6595-177b-4c78-bd39-164d60f7d632/root/var/svc/log/oxide-internal_dns:default.log
10062024-01-23T18:55:31.124Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b1bf6595-177b-4c78-bd39-164d60f7d632/root/var/svc/log/oxide-internal_dns:default.log.0 (4288 bytes)
10072024-01-23T18:55:32.351Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b1bf6595-177b-4c78-bd39-164d60f7d632/root/var/svc/log/oxide-internal_dns:default.log.0
10082024-01-23T18:55:32.351Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9182b4bd-b43b-4a28-8280-be6d9a163ad5/root/var/svc/log/oxide-cockroachdb:default.log (4482 bytes)
10092024-01-23T18:55:33.533Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9182b4bd-b43b-4a28-8280-be6d9a163ad5/root/var/svc/log/oxide-cockroachdb:default.log
10102024-01-23T18:55:33.534Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9182b4bd-b43b-4a28-8280-be6d9a163ad5/root/var/svc/log/oxide-zone-network-setup:default.log (1709 bytes)
10112024-01-23T18:55:35.116Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9182b4bd-b43b-4a28-8280-be6d9a163ad5/root/var/svc/log/oxide-zone-network-setup:default.log
10122024-01-23T18:55:35.116Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_de3885ea-3b65-4286-947d-03a26fe370ec/root/var/svc/log/oxide-crucible-agent:default.log (9791 bytes)
10132024-01-23T18:55:36.316Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_de3885ea-3b65-4286-947d-03a26fe370ec/root/var/svc/log/oxide-crucible-agent:default.log
10142024-01-23T18:55:36.317Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_de3885ea-3b65-4286-947d-03a26fe370ec/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10152024-01-23T18:55:37.490Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_de3885ea-3b65-4286-947d-03a26fe370ec/root/var/svc/log/oxide-crucible-downstairs:default.log
10162024-01-23T18:55:37.490Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_de3885ea-3b65-4286-947d-03a26fe370ec/root/var/svc/log/oxide-crucible-downstairs:downstairs-44a71be9-1b1a-4c09-b969-2f19b3238374.log (2921 bytes)
10172024-01-23T18:55:38.586ZError: No response to keepalive - Terminating session
10182024-01-23T18:55:39.395Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_de3885ea-3b65-4286-947d-03a26fe370ec/root/var/svc/log/oxide-crucible-downstairs:downstairs-44a71be9-1b1a-4c09-b969-2f19b3238374.log
10192024-01-23T18:55:39.395Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_73e2aff0-6c6b-4d3d-b111-47dc723580f4/root/var/svc/log/oxide-external_dns:default.log (26942 bytes)
10202024-01-23T18:55:39.396Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_73e2aff0-6c6b-4d3d-b111-47dc723580f4/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 26942 -> 28200
10212024-01-23T18:55:40.418Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_73e2aff0-6c6b-4d3d-b111-47dc723580f4/root/var/svc/log/oxide-external_dns:default.log
10222024-01-23T18:55:40.462Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4128e2f8-965f-4ce0-a75a-d5e114753460/root/var/svc/log/oxide-internal_dns:default.log (103158 bytes)
10232024-01-23T18:55:40.692Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4128e2f8-965f-4ce0-a75a-d5e114753460/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 103158 -> 104410
10242024-01-23T18:55:41.815Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4128e2f8-965f-4ce0-a75a-d5e114753460/root/var/svc/log/oxide-internal_dns:default.log
10252024-01-23T18:55:41.815Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4128e2f8-965f-4ce0-a75a-d5e114753460/root/var/svc/log/oxide-internal_dns:default.log.0 (9973 bytes)
10262024-01-23T18:55:43.040Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4128e2f8-965f-4ce0-a75a-d5e114753460/root/var/svc/log/oxide-internal_dns:default.log.0
10272024-01-23T18:55:43.041Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_13b5c6fe-b598-4362-abbf-2797af265d97/root/var/svc/log/oxide-nexus:default.log (35511781 bytes)
10282024-01-23T18:55:48.654ZError: No response de-activating SOL payload
10292024-01-23T18:55:50.036Z[SOL Session operational. Use ~? for help]
10302024-01-23T18:55:59.673Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_13b5c6fe-b598-4362-abbf-2797af265d97/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 35511781 -> 35582539
10312024-01-23T18:56:00.818Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_13b5c6fe-b598-4362-abbf-2797af265d97/root/var/svc/log/oxide-nexus:default.log
10322024-01-23T18:56:00.818Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_4e066324-a926-429a-9fcd-2ffa2736fc63/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
10332024-01-23T18:56:02.004Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_4e066324-a926-429a-9fcd-2ffa2736fc63/root/var/svc/log/oxide-cockroachdb:default.log
10342024-01-23T18:56:02.004Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_4e066324-a926-429a-9fcd-2ffa2736fc63/root/var/svc/log/oxide-zone-network-setup:default.log (1711 bytes)
10352024-01-23T18:56:03.190Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_4e066324-a926-429a-9fcd-2ffa2736fc63/root/var/svc/log/oxide-zone-network-setup:default.log
10362024-01-23T18:56:03.190Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_99e97100-14a0-47dd-915f-0b788895fea8/root/var/svc/log/oxide-crucible-agent:default.log (38041 bytes)
10372024-01-23T18:56:04.476Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_99e97100-14a0-47dd-915f-0b788895fea8/root/var/svc/log/oxide-crucible-agent:default.log
10382024-01-23T18:56:04.477Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_99e97100-14a0-47dd-915f-0b788895fea8/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10392024-01-23T18:56:06.080Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_99e97100-14a0-47dd-915f-0b788895fea8/root/var/svc/log/oxide-crucible-downstairs:default.log
10402024-01-23T18:56:06.080Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_99e97100-14a0-47dd-915f-0b788895fea8/root/var/svc/log/oxide-crucible-downstairs:downstairs-07b2ffa8-330e-465d-b942-7fe866f95e75.log (5495 bytes)
10412024-01-23T18:56:07.255Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_99e97100-14a0-47dd-915f-0b788895fea8/root/var/svc/log/oxide-crucible-downstairs:downstairs-07b2ffa8-330e-465d-b942-7fe866f95e75.log
10422024-01-23T18:56:07.255Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_99e97100-14a0-47dd-915f-0b788895fea8/root/var/svc/log/oxide-crucible-downstairs:downstairs-203071d8-3cec-4f4e-a915-1d04ca48d5ae.log (3051 bytes)
10432024-01-23T18:56:08.428Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_99e97100-14a0-47dd-915f-0b788895fea8/root/var/svc/log/oxide-crucible-downstairs:downstairs-203071d8-3cec-4f4e-a915-1d04ca48d5ae.log
10442024-01-23T18:56:08.428Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_99e97100-14a0-47dd-915f-0b788895fea8/root/var/svc/log/oxide-crucible-downstairs:downstairs-3fa84898-ce1e-4102-856a-f037d15d0c30.log (5620 bytes)
10452024-01-23T18:56:09.598Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_99e97100-14a0-47dd-915f-0b788895fea8/root/var/svc/log/oxide-crucible-downstairs:downstairs-3fa84898-ce1e-4102-856a-f037d15d0c30.log
10462024-01-23T18:56:09.598Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_99e97100-14a0-47dd-915f-0b788895fea8/root/var/svc/log/oxide-crucible-downstairs:snapshot-3fa84898-ce1e-4102-856a-f037d15d0c30-e4f2903b-6fa7-434c-aa8b-fd694ad2f3fe.log (5139 bytes)
10472024-01-23T18:56:10.788Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_99e97100-14a0-47dd-915f-0b788895fea8/root/var/svc/log/oxide-crucible-downstairs:snapshot-3fa84898-ce1e-4102-856a-f037d15d0c30-e4f2903b-6fa7-434c-aa8b-fd694ad2f3fe.log
10482024-01-23T18:56:10.788Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_7c9a2d16-9fac-4548-a228-c433e5573a79/root/var/svc/log/oxide-internal_dns:default.log (103172 bytes)
10492024-01-23T18:56:11.086Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_7c9a2d16-9fac-4548-a228-c433e5573a79/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 103172 -> 106928
10502024-01-23T18:56:12.302Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_7c9a2d16-9fac-4548-a228-c433e5573a79/root/var/svc/log/oxide-internal_dns:default.log
10512024-01-23T18:56:12.302Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_7c9a2d16-9fac-4548-a228-c433e5573a79/root/var/svc/log/oxide-internal_dns:default.log.0 (6314 bytes)
10522024-01-23T18:56:13.893Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_7c9a2d16-9fac-4548-a228-c433e5573a79/root/var/svc/log/oxide-internal_dns:default.log.0
10532024-01-23T18:56:13.893Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_1628de23-16dd-472e-aeda-1d1e07fdf98d/root/var/svc/log/oxide-oximeter:default.log (256589 bytes)
10542024-01-23T18:56:14.764Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_1628de23-16dd-472e-aeda-1d1e07fdf98d/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 256589 -> 294764
10552024-01-23T18:56:15.904Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_1628de23-16dd-472e-aeda-1d1e07fdf98d/root/var/svc/log/oxide-oximeter:default.log
10562024-01-23T18:56:15.904Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_d274347f-9762-4775-9e9c-790294b65946/root/var/svc/log/oxide-cockroachdb:default.log (4406 bytes)
10572024-01-23T18:56:17.097Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_d274347f-9762-4775-9e9c-790294b65946/root/var/svc/log/oxide-cockroachdb:default.log
10582024-01-23T18:56:17.097Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_d274347f-9762-4775-9e9c-790294b65946/root/var/svc/log/oxide-zone-network-setup:default.log (1709 bytes)
10592024-01-23T18:56:18.310Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_d274347f-9762-4775-9e9c-790294b65946/root/var/svc/log/oxide-zone-network-setup:default.log
10602024-01-23T18:56:18.310Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_a6637c4e-e614-4e72-97c4-892fda21bf2d/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10612024-01-23T18:56:19.505Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_a6637c4e-e614-4e72-97c4-892fda21bf2d/root/var/svc/log/oxide-crucible-agent:default.log
10622024-01-23T18:56:19.505Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_a6637c4e-e614-4e72-97c4-892fda21bf2d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10632024-01-23T18:56:20.661Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_a6637c4e-e614-4e72-97c4-892fda21bf2d/root/var/svc/log/oxide-crucible-downstairs:default.log
10642024-01-23T18:56:20.661Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_350adeb5-d7cc-4499-ae9e-37e20c416744/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10652024-01-23T18:56:22.044Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_350adeb5-d7cc-4499-ae9e-37e20c416744/root/var/svc/log/oxide-crucible-pantry:default.log
10662024-01-23T18:56:22.044Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_03aed299-9852-4660-9da9-34a21aec3294/root/var/svc/log/oxide-cockroachdb:default.log (4382 bytes)
10672024-01-23T18:56:23.206Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_03aed299-9852-4660-9da9-34a21aec3294/root/var/svc/log/oxide-cockroachdb:default.log
10682024-01-23T18:56:23.206Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_03aed299-9852-4660-9da9-34a21aec3294/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
10692024-01-23T18:56:24.360Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_03aed299-9852-4660-9da9-34a21aec3294/root/var/svc/log/oxide-zone-network-setup:default.log
10702024-01-23T18:56:24.360Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d25f6f12-cd7b-47ee-a556-ce796fb0e398/root/var/svc/log/oxide-crucible-agent:default.log (9795 bytes)
10712024-01-23T18:56:25.554Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d25f6f12-cd7b-47ee-a556-ce796fb0e398/root/var/svc/log/oxide-crucible-agent:default.log
10722024-01-23T18:56:25.554Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d25f6f12-cd7b-47ee-a556-ce796fb0e398/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10732024-01-23T18:56:26.715Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d25f6f12-cd7b-47ee-a556-ce796fb0e398/root/var/svc/log/oxide-crucible-downstairs:default.log
10742024-01-23T18:56:26.715Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d25f6f12-cd7b-47ee-a556-ce796fb0e398/root/var/svc/log/oxide-crucible-downstairs:downstairs-b8fc4e12-a0a6-44ae-846a-a47dd76cc552.log (2919 bytes)
10752024-01-23T18:56:27.901Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_d25f6f12-cd7b-47ee-a556-ce796fb0e398/root/var/svc/log/oxide-crucible-downstairs:downstairs-b8fc4e12-a0a6-44ae-846a-a47dd76cc552.log
10762024-01-23T18:56:27.902Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b23d433-a9ad-4d04-8459-cde29b53680a/root/var/svc/log/oxide-crucible-agent:default.log (23908 bytes)
10772024-01-23T18:56:29.145Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b23d433-a9ad-4d04-8459-cde29b53680a/root/var/svc/log/oxide-crucible-agent:default.log
10782024-01-23T18:56:29.145Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b23d433-a9ad-4d04-8459-cde29b53680a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10792024-01-23T18:56:30.312Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b23d433-a9ad-4d04-8459-cde29b53680a/root/var/svc/log/oxide-crucible-downstairs:default.log
10802024-01-23T18:56:30.312Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b23d433-a9ad-4d04-8459-cde29b53680a/root/var/svc/log/oxide-crucible-downstairs:downstairs-840fd73f-8c36-48f7-8875-26a6fdaa2123.log (5619 bytes)
10812024-01-23T18:56:31.495Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b23d433-a9ad-4d04-8459-cde29b53680a/root/var/svc/log/oxide-crucible-downstairs:downstairs-840fd73f-8c36-48f7-8875-26a6fdaa2123.log
10822024-01-23T18:56:31.495Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b23d433-a9ad-4d04-8459-cde29b53680a/root/var/svc/log/oxide-crucible-downstairs:downstairs-8e941b44-e152-4704-8049-084b5e53d642.log (2921 bytes)
10832024-01-23T18:56:34.077Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b23d433-a9ad-4d04-8459-cde29b53680a/root/var/svc/log/oxide-crucible-downstairs:downstairs-8e941b44-e152-4704-8049-084b5e53d642.log
10842024-01-23T18:56:34.078Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b23d433-a9ad-4d04-8459-cde29b53680a/root/var/svc/log/oxide-crucible-downstairs:snapshot-840fd73f-8c36-48f7-8875-26a6fdaa2123-e4f2903b-6fa7-434c-aa8b-fd694ad2f3fe.log (5136 bytes)
10852024-01-23T18:56:35.271Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_0b23d433-a9ad-4d04-8459-cde29b53680a/root/var/svc/log/oxide-crucible-downstairs:snapshot-840fd73f-8c36-48f7-8875-26a6fdaa2123-e4f2903b-6fa7-434c-aa8b-fd694ad2f3fe.log
10862024-01-23T18:56:35.271Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_40336d05-1a90-470f-b21c-ac35f3bd8a38/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10872024-01-23T18:56:36.467Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_40336d05-1a90-470f-b21c-ac35f3bd8a38/root/var/svc/log/oxide-crucible-pantry:default.log
10882024-01-23T18:56:36.467Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c/root/var/svc/log/oxide-ntp:default.log (342 bytes)
10892024-01-23T18:56:37.633Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c/root/var/svc/log/oxide-ntp:default.log
10902024-01-23T18:56:37.633Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c/root/var/svc/log/oxide-ntp:default.log.0 (914 bytes)
10912024-01-23T18:56:38.796Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c/root/var/svc/log/oxide-ntp:default.log.0
10922024-01-23T18:56:38.797Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3dfef7f9-5740-4aed-9fdb-dcf7e7cc60c5/root/var/svc/log/oxide-crucible-agent:default.log (27798 bytes)
10932024-01-23T18:56:40.061Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3dfef7f9-5740-4aed-9fdb-dcf7e7cc60c5/root/var/svc/log/oxide-crucible-agent:default.log
10942024-01-23T18:56:40.062Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3dfef7f9-5740-4aed-9fdb-dcf7e7cc60c5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10952024-01-23T18:56:41.232Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3dfef7f9-5740-4aed-9fdb-dcf7e7cc60c5/root/var/svc/log/oxide-crucible-downstairs:default.log
10962024-01-23T18:56:41.233Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3dfef7f9-5740-4aed-9fdb-dcf7e7cc60c5/root/var/svc/log/oxide-crucible-downstairs:downstairs-10a561e2-8496-4148-95d5-c8eaa6143416.log (3050 bytes)
10972024-01-23T18:56:42.496Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3dfef7f9-5740-4aed-9fdb-dcf7e7cc60c5/root/var/svc/log/oxide-crucible-downstairs:downstairs-10a561e2-8496-4148-95d5-c8eaa6143416.log
10982024-01-23T18:56:42.496Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3dfef7f9-5740-4aed-9fdb-dcf7e7cc60c5/root/var/svc/log/oxide-crucible-downstairs:downstairs-af8e6c1a-ed98-4c15-8eb2-3e2bebec0f5a.log (5619 bytes)
10992024-01-23T18:56:43.703Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3dfef7f9-5740-4aed-9fdb-dcf7e7cc60c5/root/var/svc/log/oxide-crucible-downstairs:downstairs-af8e6c1a-ed98-4c15-8eb2-3e2bebec0f5a.log
11002024-01-23T18:56:43.704Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3dfef7f9-5740-4aed-9fdb-dcf7e7cc60c5/root/var/svc/log/oxide-crucible-downstairs:snapshot-af8e6c1a-ed98-4c15-8eb2-3e2bebec0f5a-e4f2903b-6fa7-434c-aa8b-fd694ad2f3fe.log (5138 bytes)
11012024-01-23T18:56:44.939Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_3dfef7f9-5740-4aed-9fdb-dcf7e7cc60c5/root/var/svc/log/oxide-crucible-downstairs:snapshot-af8e6c1a-ed98-4c15-8eb2-3e2bebec0f5a-e4f2903b-6fa7-434c-aa8b-fd694ad2f3fe.log
11022024-01-23T18:56:44.940Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_e2c9d400-d96e-44da-934d-5a77388a5e38/root/var/svc/log/oxide-nexus:default.log (1263997 bytes)
11032024-01-23T18:56:47.148Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_e2c9d400-d96e-44da-934d-5a77388a5e38/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1263997 -> 1406464
11042024-01-23T18:56:48.309Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_e2c9d400-d96e-44da-934d-5a77388a5e38/root/var/svc/log/oxide-nexus:default.log
11052024-01-23T18:56:48.310Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6a5927f5-2d68-4d95-916b-fd351c9ecee5/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
11062024-01-23T18:56:49.552Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6a5927f5-2d68-4d95-916b-fd351c9ecee5/root/var/svc/log/oxide-crucible-agent:default.log
11072024-01-23T18:56:49.552Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6a5927f5-2d68-4d95-916b-fd351c9ecee5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11082024-01-23T18:56:51.141Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6a5927f5-2d68-4d95-916b-fd351c9ecee5/root/var/svc/log/oxide-crucible-downstairs:default.log
11092024-01-23T18:56:51.141Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_18dd51ec-08bf-41a1-a470-d97821f88210/root/var/svc/log/oxide-crucible-pantry:default.log (1218737 bytes)
11102024-01-23T18:56:52.842Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_18dd51ec-08bf-41a1-a470-d97821f88210/root/var/svc/log/oxide-crucible-pantry:default.log" changed size mid upload: 1218737 -> 1250431
11112024-01-23T18:56:53.973Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_18dd51ec-08bf-41a1-a470-d97821f88210/root/var/svc/log/oxide-crucible-pantry:default.log
11122024-01-23T18:56:53.973Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_85b9e42d-c193-4b12-8ba4-644c4a0cb970/root/var/svc/log/oxide-crucible-agent:default.log (14122 bytes)
11132024-01-23T18:56:55.168Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_85b9e42d-c193-4b12-8ba4-644c4a0cb970/root/var/svc/log/oxide-crucible-agent:default.log
11142024-01-23T18:56:55.168Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_85b9e42d-c193-4b12-8ba4-644c4a0cb970/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11152024-01-23T18:56:56.331Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_85b9e42d-c193-4b12-8ba4-644c4a0cb970/root/var/svc/log/oxide-crucible-downstairs:default.log
11162024-01-23T18:56:56.331Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_85b9e42d-c193-4b12-8ba4-644c4a0cb970/root/var/svc/log/oxide-crucible-downstairs:downstairs-f11a03f1-241b-4ac8-8e61-15fb02cd9cbd.log (5494 bytes)
11172024-01-23T18:56:57.486Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_85b9e42d-c193-4b12-8ba4-644c4a0cb970/root/var/svc/log/oxide-crucible-downstairs:downstairs-f11a03f1-241b-4ac8-8e61-15fb02cd9cbd.log
11182024-01-23T18:56:57.486Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_36ae8cbb-41a4-40db-90bd-03dd97a63b4b/root/var/svc/log/oxide-nexus:default.log (926668 bytes)
11192024-01-23T18:56:59.335Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_36ae8cbb-41a4-40db-90bd-03dd97a63b4b/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 926668 -> 1069978
11202024-01-23T18:57:00.555Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_36ae8cbb-41a4-40db-90bd-03dd97a63b4b/root/var/svc/log/oxide-nexus:default.log
11212024-01-23T18:57:00.555Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c/root/var/log/chrony/measurements.log (20824 bytes)
11222024-01-23T18:57:00.719Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c/root/var/log/chrony/measurements.log" changed size mid upload: 20824 -> 25208
11232024-01-23T18:57:01.844Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c/root/var/log/chrony/measurements.log
11242024-01-23T18:57:01.844Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c/root/var/log/chrony/statistics.log (12051 bytes)
11252024-01-23T18:57:01.963Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c/root/var/log/chrony/statistics.log" changed size mid upload: 12051 -> 15444
11262024-01-23T18:57:03.078Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c/root/var/log/chrony/statistics.log
11272024-01-23T18:57:03.078Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c/root/var/log/chrony/tracking.log (1984 bytes)
11282024-01-23T18:57:03.158Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c/root/var/log/chrony/tracking.log" changed size mid upload: 1984 -> 2248
11292024-01-23T18:57:04.280Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_ntp_4fad6e5b-668e-4eae-b808-20ad50eff99c/root/var/log/chrony/tracking.log