01HNKJEM7AQ95B29KEC9PAC4WB: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.28",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HNKJFRSWMK8F9V2W3X5D7AJD

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12024-02-02T01:25:15.599Zjob dependencies complete; ready to run (waiting for 1 h 13 m 51 s)
22024-02-02T02:17:04.351Zjob assigned to worker 01HNKSAEDAX9YJ1DJRR39GY1ZH (queued for 51 m 48 s)
32024-02-02T02:17:04.507Zhttp://172.20.0.5:9969/ipxe/sock... ok
42024-02-02T02:17:04.556Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52024-02-02T02:17:04.596Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... 4%Error: No response to keepalive - Terminating session
62024-02-02T02:17:04.634ZError: No response de-activating SOL payload
72024-02-02T02:17:04.667Z[SOL Session operational. Use ~? for help]
82024-02-02T02:17:04.706Z 64%Error: No response to keepalive - Terminating session
92024-02-02T02:17:04.742ZError: No response de-activating SOL payload
102024-02-02T02:17:04.779Z[SOL Session operational. Use ~? for help]
112024-02-02T02:17:04.813Z ok
122024-02-02T02:17:04.847Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
132024-02-02T02:17:04.880Zhttp://172.20.0.5:9969/postboot/sock... ok
142024-02-02T02:17:04.914ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
152024-02-02T02:17:04.947Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
162024-02-02T02:17:04.981ZOxide Helios Version helios-2.0.22418 64-bit
172024-02-02T02:17:05.014ZConfiguring devices.
182024-02-02T02:17:05.047Z##################################################
192024-02-02T02:17:05.082Z#### ############# ##
202024-02-02T02:17:05.114Z### ### ############ ##
212024-02-02T02:17:05.156Z## ### # ## ### ## Oxide Computer Company ##
222024-02-02T02:17:05.192Z## ## ## ### # ### ##
232024-02-02T02:17:05.635Z## # ### #### #### This Station Under ##
242024-02-02T02:17:05.670Z### ### #### # ### Computer Control ##
252024-02-02T02:17:05.703Z#### #### ### ## ##
262024-02-02T02:17:05.738Z##################################################
272024-02-02T02:17:05.866ZHostname: sock
282024-02-02T02:17:05.904ZLoading smf(7) service descriptions:
292024-02-02T02:17:06.588Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-0096297e5e63ab8e.gz
302024-02-02T02:17:09.007Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-0096297e5e63ab8e.gz
312024-02-02T02:17:09.007Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
322024-02-02T02:17:33.563Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
332024-02-02T02:17:33.588Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-6d4886eeea56eeb3.gz
342024-02-02T02:18:00.703Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-6d4886eeea56eeb3.gz
352024-02-02T02:18:00.704Zdownloading input: /input/ci-tools/work/caboose-util.gz
362024-02-02T02:18:02.283Zdownloaded input: /input/ci-tools/work/caboose-util.gz
372024-02-02T02:18:02.283Zdownloading input: /input/ci-tools/work/tufaceous.gz
382024-02-02T02:18:06.257Zdownloaded input: /input/ci-tools/work/tufaceous.gz
392024-02-02T02:18:06.283Zdownloading input: /input/package/work/version.txt
402024-02-02T02:18:06.308Zdownloaded input: /input/package/work/version.txt
412024-02-02T02:18:06.309Zdownloading input: /input/package/work/package.tar.gz
422024-02-02T02:18:28.223Zdownloaded input: /input/package/work/package.tar.gz
432024-02-02T02:18:28.223Zdownloading input: /input/package/work/global-zone-packages.tar.gz
442024-02-02T02:18:53.823Zsock console login: Error: No response to keepalive - Terminating session
452024-02-02T02:19:03.093Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
462024-02-02T02:19:03.122Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
472024-02-02T02:19:03.764ZError: No response de-activating SOL payload
482024-02-02T02:19:05.003Z[SOL Session operational. Use ~? for help]
492024-02-02T02:19:17.459Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
502024-02-02T02:19:17.542Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
512024-02-02T02:20:05.332Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
522024-02-02T02:20:05.332Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
532024-02-02T02:21:14.828Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
542024-02-02T02:21:14.829Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
552024-02-02T02:21:54.063ZError: No response to keepalive - Terminating session
562024-02-02T02:22:01.683Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
572024-02-02T02:22:01.710Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
582024-02-02T02:22:03.841ZError: No response de-activating SOL payload
592024-02-02T02:22:04.948Z[SOL Session operational. Use ~? for help]
602024-02-02T02:22:10.624Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
612024-02-02T02:22:10.625Zdownloading input: /input/package/work/zones/crucible.tar.gz
622024-02-02T02:22:24.660Zdownloaded input: /input/package/work/zones/crucible.tar.gz
632024-02-02T02:22:24.660Zdownloading input: /input/package/work/zones/external-dns.tar.gz
642024-02-02T02:22:36.426Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
652024-02-02T02:22:36.450Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
662024-02-02T02:22:48.761Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
672024-02-02T02:22:48.761Zdownloading input: /input/package/work/zones/ntp.tar.gz
682024-02-02T02:22:48.795Zdownloaded input: /input/package/work/zones/ntp.tar.gz
692024-02-02T02:22:48.795Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
702024-02-02T02:22:54.842Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
712024-02-02T02:22:54.842Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
722024-02-02T02:23:07.518Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
732024-02-02T02:23:07.518Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
742024-02-02T02:23:36.618Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
752024-02-02T02:23:36.644Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
762024-02-02T02:23:59.725Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
772024-02-02T02:23:59.725Zdownloading input: /input/package/work/zones/overlay.tar.gz
782024-02-02T02:23:59.804Zdownloaded input: /input/package/work/zones/overlay.tar.gz
792024-02-02T02:23:59.830Zdownloading input: /input/package/work/zones/oximeter.tar.gz
802024-02-02T02:24:17.039Zdownloaded input: /input/package/work/zones/oximeter.tar.gz
812024-02-02T02:24:17.039Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
822024-02-02T02:24:46.084Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
832024-02-02T02:24:46.122Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
842024-02-02T02:25:09.841ZError: No response to keepalive - Terminating session
852024-02-02T02:25:19.918ZError: No response de-activating SOL payload
862024-02-02T02:25:21.027Z[SOL Session operational. Use ~? for help]
872024-02-02T02:26:20.006Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
882024-02-02T02:26:20.007Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
892024-02-02T02:27:35.363Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
902024-02-02T02:27:35.363Zstarting task 0: "setup"
912024-02-02T02:27:35.388Z++ uname -s
922024-02-02T02:27:35.388Z+ kern=SunOS
932024-02-02T02:27:35.388Z+ case "$kern" in
942024-02-02T02:27:35.388Z+ groupadd -g 12345 build
952024-02-02T02:27:35.388Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
962024-02-02T02:27:37.387Z+ zfs create -o mountpoint=/work rpool/work
972024-02-02T02:27:37.418Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
982024-02-02T02:27:37.418Z+ home_fs=zfs
992024-02-02T02:27:37.419Z+ [[ zfs == autofs ]]
1002024-02-02T02:27:37.419Z+ mkdir -p /home/build
1012024-02-02T02:27:37.419Z+ chown build:build /home/build /work
1022024-02-02T02:27:39.422Z+ chmod 0700 /home/build /work
1032024-02-02T02:27:39.448Zprocess exited: duration 4054 ms, exit code 0
 
1042024-02-02T02:27:39.912Zstarting task 1: "authentication"
1052024-02-02T02:27:39.972Zprocess exited: duration 58 ms, exit code 0
 
1062024-02-02T02:27:40.052Zstarting task 2: "build"
1072024-02-02T02:27:40.109Z+ trap _exit_trap EXIT
1082024-02-02T02:27:40.110Z+ set +x
 
1092024-02-02T02:28:09.706ZError: No response to keepalive - Terminating session
1102024-02-02T02:28:19.858ZError: No response de-activating SOL payload
1112024-02-02T02:28:21.367Z[SOL Session operational. Use ~? for help]
 
1122024-02-02T02:28:22.457Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
1132024-02-02T02:28:22.947Zsecpolicy_fs_linkdir: 0x55 = 0x55
1142024-02-02T02:28:22.947Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
1152024-02-02T02:28:22.948Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
1162024-02-02T02:28:22.948Zsecpolicy_fs_linkdir+3: 0 = 0xe5
1172024-02-02T02:28:22.948Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1182024-02-02T02:28:22.948Zsecpolicy_fs_linkdir+5: 0 = 0x1
1192024-02-02T02:28:22.948Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1202024-02-02T02:28:22.948Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1212024-02-02T02:28:22.999Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1222024-02-02T02:28:22.999Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1232024-02-02T02:28:22.999Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1242024-02-02T02:28:22.999Z+ [[ -d /opt/oxide ]]
1252024-02-02T02:28:22.999Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1262024-02-02T02:28:23.000Z+ pfexec mkdir /opt/oxide
1272024-02-02T02:28:23.000Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1282024-02-02T02:28:23.000Z+ [[ -d /opt/oxide-underneath ]]
1292024-02-02T02:28:23.000Z+ cd /opt/oxide-underneath
1302024-02-02T02:28:23.000Z+ pfexec tar ceEp@/f - .
1312024-02-02T02:28:23.000Z+ cd /opt/oxide
1322024-02-02T02:28:23.000Z+ pfexec tar xveEp@/f -
1332024-02-02T02:28:23.000Zx ., 0 bytes, 0 tape blocks
1342024-02-02T02:28:23.000Zx ./opte, 0 bytes, 0 tape blocks
1352024-02-02T02:28:23.000Zx ./opte/bin, 0 bytes, 0 tape blocks
1362024-02-02T02:28:23.000Zx ./opte/bin/opteadm, 51799224 bytes, 101171 tape blocks
1372024-02-02T02:28:23.000Z+ pfexec rm -rf /opt/oxide-underneath
1382024-02-02T02:28:23.000Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1392024-02-02T02:28:23.000Z+ pfexec mkdir /opt/oxide/work
1402024-02-02T02:28:23.000Z+ pfexec chown build:build /opt/oxide/work
1412024-02-02T02:28:23.000Z+ cd /opt/oxide/work
1422024-02-02T02:28:23.003Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1432024-02-02T02:28:23.004Ztar: blocksize = 10
1442024-02-02T02:28:24.476ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1452024-02-02T02:28:24.476Zx out/mg-ddm-gz.tar, 63076864 bytes, 123197 tape blocks
1462024-02-02T02:28:24.476Zx out/omicron-sled-agent.tar, 122173952 bytes, 238621 tape blocks
1472024-02-02T02:28:24.476Zx out/oxlog.tar, 8382464 bytes, 16372 tape blocks
1482024-02-02T02:28:24.476Zx out/target/test, 75 bytes, 1 tape blocks
1492024-02-02T02:28:24.476Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1502024-02-02T02:28:24.476Zx package-manifest.toml, 23732 bytes, 47 tape blocks
1512024-02-02T02:28:24.476Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1522024-02-02T02:28:24.476Zx target/release/omicron-package, 44581856 bytes, 87074 tape blocks
1532024-02-02T02:28:24.476Zx tools/create_virtual_hardware.sh, 3184 bytes, 7 tape blocks
1542024-02-02T02:28:24.477Zx tools/virtual_hardware.sh, 3416 bytes, 7 tape blocks
1552024-02-02T02:28:24.477Zx tools/scrimlet/create-softnpu-zone.sh, 627 bytes, 2 tape blocks
1562024-02-02T02:28:24.477Zx tools/scrimlet/destroy-softnpu-zone.sh, 203 bytes, 1 tape blocks
1572024-02-02T02:28:24.477Zx tools/scrimlet/softnpu-init.sh, 2157 bytes, 5 tape blocks
1582024-02-02T02:28:24.477Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1592024-02-02T02:28:24.477Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1602024-02-02T02:28:24.504Z
1612024-02-02T02:28:24.504Zreal 1.669172383
1622024-02-02T02:28:24.505Zuser 1.504537036
1632024-02-02T02:28:24.505Zsys 0.322017130
1642024-02-02T02:28:24.505Ztrap 0.000057819
1652024-02-02T02:28:24.505Ztflt 0.000000000
1662024-02-02T02:28:24.505Zdflt 0.000000000
1672024-02-02T02:28:24.505Zkflt 0.000000000
1682024-02-02T02:28:24.505Zlock 0.000000000
1692024-02-02T02:28:24.505Zslp 1.469221040
1702024-02-02T02:28:24.505Zlat 0.040332685
1712024-02-02T02:28:24.505Zstop 0.000129372
1722024-02-02T02:28:24.505Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1732024-02-02T02:28:25.651Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1742024-02-02T02:28:25.680Z+ mkdir tests
1752024-02-02T02:28:25.680Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1762024-02-02T02:28:25.680Z+ ptime -m gunzip
1772024-02-02T02:28:25.681Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-0096297e5e63ab8e
1782024-02-02T02:28:25.908Z
1792024-02-02T02:28:25.937Zreal 0.233793490
1802024-02-02T02:28:25.938Zuser 0.213984015
1812024-02-02T02:28:25.938Zsys 0.019547827
1822024-02-02T02:28:25.938Ztrap 0.000023230
1832024-02-02T02:28:25.938Ztflt 0.000000000
1842024-02-02T02:28:25.938Zdflt 0.000000000
1852024-02-02T02:28:25.938Zkflt 0.000000000
1862024-02-02T02:28:25.938Zlock 0.000000000
1872024-02-02T02:28:25.938Zslp 0.000022920
1882024-02-02T02:28:25.938Zlat 0.000024420
1892024-02-02T02:28:25.938Zstop 0.000127703
1902024-02-02T02:28:25.938Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-0096297e5e63ab8e
1912024-02-02T02:28:25.938Z+ chmod a+x tests/bootstrap-0096297e5e63ab8e
1922024-02-02T02:28:25.938Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1932024-02-02T02:28:25.938Z+ ptime -m gunzip
1942024-02-02T02:28:25.939Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1952024-02-02T02:28:28.426Z
1962024-02-02T02:28:28.473Zreal 2.499883394
1972024-02-02T02:28:28.474Zuser 2.332648203
1982024-02-02T02:28:28.474Zsys 0.166837823
1992024-02-02T02:28:28.474Ztrap 0.000175903
2002024-02-02T02:28:28.474Ztflt 0.000000000
2012024-02-02T02:28:28.475Zdflt 0.000000000
2022024-02-02T02:28:28.475Zkflt 0.000000000
2032024-02-02T02:28:28.475Zlock 0.000000000
2042024-02-02T02:28:28.475Zslp 0.000020710
2052024-02-02T02:28:28.475Zlat 0.000017690
2062024-02-02T02:28:28.476Zstop 0.000122033
2072024-02-02T02:28:28.476Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
2082024-02-02T02:28:28.476Z+ chmod a+x tests/bootstrap
2092024-02-02T02:28:28.476Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
2102024-02-02T02:28:28.476Z+ ptime -m gunzip
2112024-02-02T02:28:28.477Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-6d4886eeea56eeb3
2122024-02-02T02:28:30.998Z
2132024-02-02T02:28:31.048Zreal 2.551433944
2142024-02-02T02:28:31.049Zuser 2.404617724
2152024-02-02T02:28:31.049Zsys 0.146405388
2162024-02-02T02:28:31.049Ztrap 0.000180744
2172024-02-02T02:28:31.049Ztflt 0.000000000
2182024-02-02T02:28:31.049Zdflt 0.000000000
2192024-02-02T02:28:31.049Zkflt 0.000000000
2202024-02-02T02:28:31.049Zlock 0.000000000
2212024-02-02T02:28:31.049Zslp 0.000023250
2222024-02-02T02:28:31.050Zlat 0.000020181
2232024-02-02T02:28:31.050Zstop 0.000117922
2242024-02-02T02:28:31.050Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-6d4886eeea56eeb3
2252024-02-02T02:28:31.050Z+ chmod a+x tests/end_to_end_tests-6d4886eeea56eeb3
2262024-02-02T02:28:31.050Z+ bmat address ls
2272024-02-02T02:28:31.050ZNAME CIDR FIRST LAST COUNT
2282024-02-02T02:28:31.050Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2292024-02-02T02:28:31.050Z++ bmat address ls -f extra -Ho first,last,count
2302024-02-02T02:28:31.051Z+ set -- 10.151.2.164 10.151.2.227 64
2312024-02-02T02:28:31.051Z+ EXTRA_IP_START=10.151.2.164
2322024-02-02T02:28:31.051Z+ EXTRA_IP_END=10.151.2.227
2332024-02-02T02:28:31.051Z+ EXTRA_IP_COUNT=64
2342024-02-02T02:28:31.051Z+ (( EXTRA_IP_COUNT >= 32 ))
2352024-02-02T02:28:31.051Z+ EXTRA_IP_BASE=10.151.2
2362024-02-02T02:28:31.052Z+ EXTRA_IP_FOCTET=164
2372024-02-02T02:28:31.052Z+ EXTRA_IP_LOCTET=227
2382024-02-02T02:28:31.052Z+ SERVICE_IP_POOL_START=10.151.2.164
2392024-02-02T02:28:31.052Z+ SERVICE_IP_POOL_END=10.151.2.173
2402024-02-02T02:28:31.052Z+ DNS_IP1=10.151.2.164
2412024-02-02T02:28:31.052Z+ DNS_IP2=10.151.2.165
2422024-02-02T02:28:31.052Z+ UPLINK_IP=10.151.2.174
2432024-02-02T02:28:31.052Z+ IPPOOL_START=10.151.2.175
2442024-02-02T02:28:31.052Z+ IPPOOL_END=10.151.2.227
2452024-02-02T02:28:31.052Z++ ipadm show-addr -po type,addr
2462024-02-02T02:28:31.053Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2472024-02-02T02:28:31.053Z+ GATEWAY_IP=10.151.2.100
2482024-02-02T02:28:31.053Z+ [[ -n 10.151.2.100 ]]
2492024-02-02T02:28:31.053Z+ ping -s 10.151.2.100 56 1
2502024-02-02T02:28:31.053ZPING 10.151.2.100: 56 data bytes
2512024-02-02T02:28:31.053Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2522024-02-02T02:28:31.053Z
2532024-02-02T02:28:31.053Z----10.151.2.100 PING Statistics----
2542024-02-02T02:28:31.053Z1 packets transmitted, 1 packets received, 0% packet loss
2552024-02-02T02:28:31.053Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2562024-02-02T02:28:31.054Z++ arp -an
2572024-02-02T02:28:31.054Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2582024-02-02T02:28:31.054Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2592024-02-02T02:28:31.054Z+ routeadm -e ipv4-forwarding -u
2602024-02-02T02:28:31.150Z+ PXA_START=10.151.2.164
2612024-02-02T02:28:31.150Z+ PXA_END=10.151.2.227
2622024-02-02T02:28:31.150Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2632024-02-02T02:28:31.179Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2642024-02-02T02:28:31.355Z+ ZPOOL_VDEV_DIR=/scratch
2652024-02-02T02:28:31.355Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2662024-02-02T02:28:31.394Z+++ dirname ./tools/create_virtual_hardware.sh
2672024-02-02T02:28:31.394Z++ cd ./tools
2682024-02-02T02:28:31.394Z++ pwd
2692024-02-02T02:28:31.394Z+ SOURCE_DIR=/opt/oxide/work/tools
2702024-02-02T02:28:31.394Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2712024-02-02T02:28:31.394Z+ SOFTNPU_MODE=zone
2722024-02-02T02:28:31.394Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2732024-02-02T02:28:31.394Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2742024-02-02T02:28:31.395Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2752024-02-02T02:28:31.395Z++ dladm show-phys -p -o LINK
2762024-02-02T02:28:31.395Z++ head -1
2772024-02-02T02:28:31.395Z+ PHYSICAL_LINK=igb0
2782024-02-02T02:28:31.395Z+ echo 'Using igb0 as physical link'
2792024-02-02T02:28:31.395ZUsing igb0 as physical link
2802024-02-02T02:28:31.395Z+ ensure_run_as_root
2812024-02-02T02:28:31.395Z++ id -u
2822024-02-02T02:28:31.395Z+ [[ 0 -ne 0 ]]
2832024-02-02T02:28:31.395Z+ ensure_zpools
2842024-02-02T02:28:31.395Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2852024-02-02T02:28:31.395Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2862024-02-02T02:28:31.395Z+ readarray -t ZPOOLS
2872024-02-02T02:28:31.395Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2882024-02-02T02:28:31.395Z++ sed 's/[ ",]//g'
2892024-02-02T02:28:31.395Z+ for ZPOOL in "${ZPOOLS[@]}"
2902024-02-02T02:28:31.395Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2912024-02-02T02:28:31.396ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2922024-02-02T02:28:31.396Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2932024-02-02T02:28:31.396Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2942024-02-02T02:28:31.396Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2952024-02-02T02:28:31.396Z0+0 records in
2962024-02-02T02:28:31.396Z0+0 records out
2972024-02-02T02:28:31.396Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2982024-02-02T02:28:31.396Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2992024-02-02T02:28:31.396Z+ set +x
3002024-02-02T02:28:31.396ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3012024-02-02T02:28:31.432Z++ zpool list -o name
3022024-02-02T02:28:31.432Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
3032024-02-02T02:28:31.432Z+ [[ -z '' ]]
3042024-02-02T02:28:31.432Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
3052024-02-02T02:28:31.508Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
3062024-02-02T02:28:31.508Z+ set +x
3072024-02-02T02:28:31.508ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3082024-02-02T02:28:31.508Z+ for ZPOOL in "${ZPOOLS[@]}"
3092024-02-02T02:28:31.508ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3102024-02-02T02:28:31.508Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3112024-02-02T02:28:31.508Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3122024-02-02T02:28:31.508Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3132024-02-02T02:28:31.509Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3142024-02-02T02:28:31.509Z0+0 records in
3152024-02-02T02:28:31.509Z0+0 records out
3162024-02-02T02:28:31.509Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3172024-02-02T02:28:31.509Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3182024-02-02T02:28:31.509Z+ set +x
3192024-02-02T02:28:31.509ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3202024-02-02T02:28:31.509Z++ zpool list -o name
3212024-02-02T02:28:31.509Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3222024-02-02T02:28:31.509Z+ [[ -z '' ]]
3232024-02-02T02:28:31.510Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3242024-02-02T02:28:31.510Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3252024-02-02T02:28:31.510Z+ set +x
3262024-02-02T02:28:31.510ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3272024-02-02T02:28:31.510Z+ for ZPOOL in "${ZPOOLS[@]}"
3282024-02-02T02:28:31.510ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3292024-02-02T02:28:31.510Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3302024-02-02T02:28:31.510Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3312024-02-02T02:28:31.510Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3322024-02-02T02:28:31.510Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3332024-02-02T02:28:31.510Z0+0 records in
3342024-02-02T02:28:31.510Z0+0 records out
3352024-02-02T02:28:31.510Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3362024-02-02T02:28:31.510Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3372024-02-02T02:28:31.511Z+ set +x
3382024-02-02T02:28:31.511ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3392024-02-02T02:28:31.511Z++ zpool list -o name
3402024-02-02T02:28:31.511Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3412024-02-02T02:28:31.511Z+ [[ -z '' ]]
3422024-02-02T02:28:31.511Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3432024-02-02T02:28:31.562Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3442024-02-02T02:28:31.562Z+ set +x
3452024-02-02T02:28:31.562ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3462024-02-02T02:28:31.562Z+ for ZPOOL in "${ZPOOLS[@]}"
3472024-02-02T02:28:31.562Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3482024-02-02T02:28:31.562ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3492024-02-02T02:28:31.562Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502024-02-02T02:28:31.562Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3512024-02-02T02:28:31.562Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3522024-02-02T02:28:31.562Z0+0 records in
3532024-02-02T02:28:31.562Z0+0 records out
3542024-02-02T02:28:31.562Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3552024-02-02T02:28:31.562Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3562024-02-02T02:28:31.562Z+ set +x
3572024-02-02T02:28:31.563ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3582024-02-02T02:28:31.563Z++ zpool list -o name
3592024-02-02T02:28:31.563Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3602024-02-02T02:28:31.563Z+ [[ -z '' ]]
3612024-02-02T02:28:31.563Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3622024-02-02T02:28:31.592Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3632024-02-02T02:28:31.592Z+ set +x
3642024-02-02T02:28:31.592ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3652024-02-02T02:28:31.592Z+ for ZPOOL in "${ZPOOLS[@]}"
3662024-02-02T02:28:31.592Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3672024-02-02T02:28:31.592ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3682024-02-02T02:28:31.592Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3692024-02-02T02:28:31.592Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3702024-02-02T02:28:31.592Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3712024-02-02T02:28:31.592Z0+0 records in
3722024-02-02T02:28:31.592Z0+0 records out
3732024-02-02T02:28:31.592Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3742024-02-02T02:28:31.592Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3752024-02-02T02:28:31.592Z+ set +x
3762024-02-02T02:28:31.592ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3772024-02-02T02:28:31.592Z++ zpool list -o name
3782024-02-02T02:28:31.592Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3792024-02-02T02:28:31.592Z+ [[ -z '' ]]
3802024-02-02T02:28:31.593Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3812024-02-02T02:28:31.636Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3822024-02-02T02:28:31.636Z+ set +x
3832024-02-02T02:28:31.636ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3842024-02-02T02:28:31.636Z+ for ZPOOL in "${ZPOOLS[@]}"
3852024-02-02T02:28:31.636Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862024-02-02T02:28:31.636ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3872024-02-02T02:28:31.636Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882024-02-02T02:28:31.636Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892024-02-02T02:28:31.636Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902024-02-02T02:28:31.636Z0+0 records in
3912024-02-02T02:28:31.636Z0+0 records out
3922024-02-02T02:28:31.637Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3932024-02-02T02:28:31.637Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942024-02-02T02:28:31.637Z+ set +x
3952024-02-02T02:28:31.637ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3962024-02-02T02:28:31.637Z++ zpool list -o name
3972024-02-02T02:28:31.637Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3982024-02-02T02:28:31.637Z+ [[ -z '' ]]
3992024-02-02T02:28:31.637Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
4002024-02-02T02:28:31.684Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4012024-02-02T02:28:31.684Z+ set +x
4022024-02-02T02:28:31.685ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
4032024-02-02T02:28:31.685Z+ for ZPOOL in "${ZPOOLS[@]}"
4042024-02-02T02:28:31.685Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4052024-02-02T02:28:31.685ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
4062024-02-02T02:28:31.685Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4072024-02-02T02:28:31.686Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4082024-02-02T02:28:31.688Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4092024-02-02T02:28:31.688Z0+0 records in
4102024-02-02T02:28:31.688Z0+0 records out
4112024-02-02T02:28:31.689Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4122024-02-02T02:28:31.689Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4132024-02-02T02:28:31.689Z+ set +x
4142024-02-02T02:28:31.689ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
4152024-02-02T02:28:31.689Z++ zpool list -o name
4162024-02-02T02:28:31.690Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4172024-02-02T02:28:31.690Z+ [[ -z '' ]]
4182024-02-02T02:28:31.690Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4192024-02-02T02:28:31.729Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4202024-02-02T02:28:31.729Z+ set +x
4212024-02-02T02:28:31.729ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4222024-02-02T02:28:31.731Z+ for ZPOOL in "${ZPOOLS[@]}"
4232024-02-02T02:28:31.731Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4242024-02-02T02:28:31.731ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4252024-02-02T02:28:31.731Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4262024-02-02T02:28:31.732Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4272024-02-02T02:28:31.732Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4282024-02-02T02:28:31.734Z0+0 records in
4292024-02-02T02:28:31.734Z0+0 records out
4302024-02-02T02:28:31.734Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4312024-02-02T02:28:31.734Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4322024-02-02T02:28:31.734Z+ set +x
4332024-02-02T02:28:31.734ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4342024-02-02T02:28:31.734Z++ zpool list -o name
4352024-02-02T02:28:31.735Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4362024-02-02T02:28:31.735Z+ [[ -z '' ]]
4372024-02-02T02:28:31.735Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4382024-02-02T02:28:31.763Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4392024-02-02T02:28:31.764Z+ set +x
4402024-02-02T02:28:31.764ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4412024-02-02T02:28:31.764Z+ for ZPOOL in "${ZPOOLS[@]}"
4422024-02-02T02:28:31.764Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4432024-02-02T02:28:31.764ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4442024-02-02T02:28:31.765Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4452024-02-02T02:28:31.765Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4462024-02-02T02:28:31.765Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4472024-02-02T02:28:31.765Z0+0 records in
4482024-02-02T02:28:31.765Z0+0 records out
4492024-02-02T02:28:31.765Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4502024-02-02T02:28:31.765Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4512024-02-02T02:28:31.766Z+ set +x
4522024-02-02T02:28:31.766ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4532024-02-02T02:28:31.766Z++ zpool list -o name
4542024-02-02T02:28:31.766Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4552024-02-02T02:28:31.766Z+ [[ -z '' ]]
4562024-02-02T02:28:31.766Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4572024-02-02T02:28:41.278Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4582024-02-02T02:28:41.278Z+ set +x
4592024-02-02T02:28:41.278ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4602024-02-02T02:28:41.278Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4612024-02-02T02:28:41.278Z+ readarray -t ZPOOLS
4622024-02-02T02:28:41.278Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4632024-02-02T02:28:41.278Z++ sed 's/[ ",]//g'
4642024-02-02T02:28:41.278Z+ for ZPOOL in "${ZPOOLS[@]}"
4652024-02-02T02:28:41.278Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4662024-02-02T02:28:41.278ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4672024-02-02T02:28:41.278Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4682024-02-02T02:28:41.279Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4692024-02-02T02:28:41.279Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4702024-02-02T02:28:41.279Z0+0 records in
4712024-02-02T02:28:41.279Z0+0 records out
4722024-02-02T02:28:41.279Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4732024-02-02T02:28:41.279Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4742024-02-02T02:28:41.279Z+ set +x
4752024-02-02T02:28:41.279ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4762024-02-02T02:28:41.279Z++ zpool list -o name
4772024-02-02T02:28:41.279Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4782024-02-02T02:28:41.279Z+ [[ -z '' ]]
4792024-02-02T02:28:41.279Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4802024-02-02T02:28:41.279Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4812024-02-02T02:28:41.280Z+ set +x
4822024-02-02T02:28:41.280ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4832024-02-02T02:28:41.280Z+ for ZPOOL in "${ZPOOLS[@]}"
4842024-02-02T02:28:41.280ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4852024-02-02T02:28:41.280Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4862024-02-02T02:28:41.280Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4872024-02-02T02:28:41.280Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4882024-02-02T02:28:41.280Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4892024-02-02T02:28:41.280Z0+0 records in
4902024-02-02T02:28:41.280Z0+0 records out
4912024-02-02T02:28:41.280Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4922024-02-02T02:28:41.280Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4932024-02-02T02:28:41.280Z+ set +x
4942024-02-02T02:28:41.281ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4952024-02-02T02:28:41.281Z++ zpool list -o name
4962024-02-02T02:28:41.281Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4972024-02-02T02:28:41.281Z+ [[ -z '' ]]
4982024-02-02T02:28:41.281Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4992024-02-02T02:28:41.281Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
5002024-02-02T02:28:41.281Z+ set +x
5012024-02-02T02:28:41.281ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
5022024-02-02T02:28:41.281Z+ [[ zone == \z\o\n\e ]]
5032024-02-02T02:28:41.281Z+ ensure_simulated_links igb0
5042024-02-02T02:28:41.281Z+ local PHYSICAL_LINK=igb0
5052024-02-02T02:28:41.281Z+ INDICES=("0" "1")
5062024-02-02T02:28:41.281Z+ for I in "${INDICES[@]}"
5072024-02-02T02:28:41.281Z++ get_simnet_name_if_exists net0
5082024-02-02T02:28:41.282Z++ dladm show-simnet -p -o LINK net0
5092024-02-02T02:28:41.282Z++ echo ''
5102024-02-02T02:28:41.282Z+ [[ -z '' ]]
5112024-02-02T02:28:41.282Z+ dladm create-simnet -t net0
5122024-02-02T02:28:41.282Z+ dladm create-simnet -t sc0_0
5132024-02-02T02:28:41.282Z+ dladm modify-simnet -t -p net0 sc0_0
5142024-02-02T02:28:41.282Z+ dladm set-linkprop -p mtu=9000 sc0_0
5152024-02-02T02:28:41.282Z+ success 'Simnet net0/sc0_0 exists'
5162024-02-02T02:28:41.282Z+ set +x
5172024-02-02T02:28:41.282ZSimnet net0/sc0_0 exists
5182024-02-02T02:28:41.282Z+ for I in "${INDICES[@]}"
5192024-02-02T02:28:41.282Z++ get_simnet_name_if_exists net1
5202024-02-02T02:28:41.282Z++ dladm show-simnet -p -o LINK net1
5212024-02-02T02:28:41.282Z++ echo ''
5222024-02-02T02:28:41.282Z+ [[ -z '' ]]
5232024-02-02T02:28:41.283Z+ dladm create-simnet -t net1
5242024-02-02T02:28:41.283Z+ dladm create-simnet -t sc1_0
5252024-02-02T02:28:41.283Z+ dladm modify-simnet -t -p net1 sc1_0
5262024-02-02T02:28:41.283Z+ dladm set-linkprop -p mtu=9000 sc1_0
5272024-02-02T02:28:41.283Z+ success 'Simnet net1/sc1_0 exists'
5282024-02-02T02:28:41.283Z+ set +x
5292024-02-02T02:28:41.283ZSimnet net1/sc1_0 exists
5302024-02-02T02:28:41.283Z++ get_vnic_name_if_exists sc0_1
5312024-02-02T02:28:41.283Z++ dladm show-vnic -p -o LINK sc0_1
5322024-02-02T02:28:41.283Z++ echo ''
5332024-02-02T02:28:41.283Z+ [[ -z '' ]]
5342024-02-02T02:28:41.283Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5352024-02-02T02:28:41.283Z+ [[ -v PROMISC_FILT_OFF ]]
5362024-02-02T02:28:41.283Z+ success 'Vnic sc0_1 exists'
5372024-02-02T02:28:41.284Z+ set +x
5382024-02-02T02:28:41.284ZVnic sc0_1 exists
5392024-02-02T02:28:41.284Z+ warn_if_no_proxy_arp
5402024-02-02T02:28:41.284Z+ [[ -v PXA_START ]]
5412024-02-02T02:28:41.284Z+ [[ -v PXA_END ]]
5422024-02-02T02:28:41.284Z+ ensure_softnpu_zone
5432024-02-02T02:28:41.284Z+ zoneadm list
5442024-02-02T02:28:41.284Z+ grep -q sidecar_softnpu
5452024-02-02T02:28:41.284Z+ [[ -f out/npuzone/npuzone ]]
5462024-02-02T02:28:41.284Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit e3ea4b495ba0a71801ded0776ae4bbd31df57e26 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5472024-02-02T02:28:41.284Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5482024-02-02T02:28:58.502Zdownloading softnpu ...done
5492024-02-02T02:28:58.527Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/libsidecar_lite.so.sha256.txt
5502024-02-02T02:29:10.822Zdownloading asic_program.so ...done
5512024-02-02T02:29:10.854Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/scadm.sha256.txt
5522024-02-02T02:29:23.601Zdownloading scadm ...done
5532024-02-02T02:29:23.635Zcreating port tfportrear0_0 for sc0_0
5542024-02-02T02:29:23.635Zcreating port tfportqsfp0_0 for sc0_1
5552024-02-02T02:29:23.636Zsoftnpu zone setup
5562024-02-02T02:29:23.636Zconfigure zone
5572024-02-02T02:29:23.663Zinstall zone
5582024-02-02T02:29:24.735Zboot zone
5592024-02-02T02:29:27.851Z[sidecar_softnpu] svcs milestone/network
5602024-02-02T02:29:27.878ZSTATE STIME FMRI
5612024-02-02T02:29:27.878Zonline 2:29:26 svc:/milestone/network:default
5622024-02-02T02:29:27.878Z[sidecar_softnpu] /softnpu/npu start
5632024-02-02T02:29:27.903Z
5642024-02-02T02:29:27.904Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5652024-02-02T02:29:27.929Z+ GATEWAY_IP=10.151.2.100
5662024-02-02T02:29:27.929Z+ echo 'Using 10.151.2.100 as gateway ip'
5672024-02-02T02:29:27.929ZUsing 10.151.2.100 as gateway ip
5682024-02-02T02:29:27.929Z+ [[ ! -v GATEWAY_MAC ]]
5692024-02-02T02:29:27.929Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5702024-02-02T02:29:27.929Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5712024-02-02T02:29:27.929Z+++ echo 18:c0:4d:0d:a0:2a
5722024-02-02T02:29:27.929Z+ COUNT=1
5732024-02-02T02:29:27.929Z+ [[ 1 -ne 1 ]]
5742024-02-02T02:29:27.929Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5752024-02-02T02:29:27.929ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5762024-02-02T02:29:27.929Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5772024-02-02T02:29:27.929Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5782024-02-02T02:29:27.954Z+ PXA_MAC=a8:e1:de:01:70:1d
5792024-02-02T02:29:27.955Z+ [[ -v PXA_START ]]
5802024-02-02T02:29:27.955Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5812024-02-02T02:29:27.955Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5822024-02-02T02:29:27.988Z+ z_scadm dump-state
5832024-02-02T02:29:27.988Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5842024-02-02T02:29:28.017Zlocal v6:
5852024-02-02T02:29:28.017Zlocal v4:
5862024-02-02T02:29:28.017Zrouter v6:
5872024-02-02T02:29:28.017Zrouter v4_idx:
5882024-02-02T02:29:28.017Zrouter v4_routes:
5892024-02-02T02:29:28.017Zresolver v4:
5902024-02-02T02:29:28.018Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5912024-02-02T02:29:28.018Zresolver v6:
5922024-02-02T02:29:28.018Znat v4:
5932024-02-02T02:29:28.018Znat_v6:
5942024-02-02T02:29:28.018Zport_mac:
5952024-02-02T02:29:28.018Zproxy arp:
5962024-02-02T02:29:28.018Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5972024-02-02T02:29:28.018Z+ success 'softnpu zone exists'
5982024-02-02T02:29:28.018Z+ set +x
5992024-02-02T02:29:28.018Zsoftnpu zone exists
6002024-02-02T02:29:28.018Z
6012024-02-02T02:29:28.018Zreal 56.637738955
6022024-02-02T02:29:28.018Zuser 0.696209519
6032024-02-02T02:29:28.018Zsys 1.577192829
6042024-02-02T02:29:28.018Ztrap 0.000068003
6052024-02-02T02:29:28.019Ztflt 0.000082207
6062024-02-02T02:29:28.019Zdflt 0.001142132
6072024-02-02T02:29:28.019Zkflt 0.000000000
6082024-02-02T02:29:28.019Zlock 44:33.873820798
6092024-02-02T02:29:28.019Zslp 2:49.691841802
6102024-02-02T02:29:28.019Zlat 0.067971948
6112024-02-02T02:29:28.019Zstop 0.334486223
6122024-02-02T02:29:28.019Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
6132024-02-02T02:29:28.049Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
6142024-02-02T02:29:28.049Z+ SILO_NAME=recovery
6152024-02-02T02:29:28.049Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
6162024-02-02T02:29:28.049Z+ EXTERNAL_DNS_DOMAIN=oxide.test
6172024-02-02T02:29:28.049Z+ sed -i~ '
6182024-02-02T02:29:28.049Z /^external_dns_ips/c\
6192024-02-02T02:29:28.049Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6202024-02-02T02:29:28.050Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6212024-02-02T02:29:28.050Z /^first/c\
6222024-02-02T02:29:28.050Zfirst = "10.151.2.164"
6232024-02-02T02:29:28.050Z /^last/c\
6242024-02-02T02:29:28.050Zlast = "10.151.2.173"
6252024-02-02T02:29:28.050Z }
6262024-02-02T02:29:28.050Z /^\[rack_network_config/,/^$/ {
6272024-02-02T02:29:28.050Z /^infra_ip_first/c\
6282024-02-02T02:29:28.050Zinfra_ip_first = "10.151.2.174"
6292024-02-02T02:29:28.050Z /^infra_ip_last/c\
6302024-02-02T02:29:28.050Zinfra_ip_last = "10.151.2.174"
6312024-02-02T02:29:28.050Z }
6322024-02-02T02:29:28.050Z /^\[\[rack_network_config.ports/,/^$/ {
6332024-02-02T02:29:28.050Z /^routes/c\
6342024-02-02T02:29:28.050Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6352024-02-02T02:29:28.050Z /^addresses/c\
6362024-02-02T02:29:28.050Zaddresses = \["10.151.2.174/24"\]
6372024-02-02T02:29:28.050Z }
6382024-02-02T02:29:28.050Z' pkg/config-rss.toml
6392024-02-02T02:29:28.050Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6402024-02-02T02:29:28.085Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6412024-02-02T02:29:28.085Z+++ pkg/config-rss.toml Fri Feb 2 02:29:28 2024
6422024-02-02T02:29:28.085Z@@ -41,7 +41,7 @@
6432024-02-02T02:29:28.085Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6442024-02-02T02:29:28.085Z # must be contained in one of the "internal services" IP Pool ranges listed
6452024-02-02T02:29:28.085Z # below.
6462024-02-02T02:29:28.085Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6472024-02-02T02:29:28.085Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6482024-02-02T02:29:28.085Z
6492024-02-02T02:29:28.085Z # Initial TLS certificates for the external API
6502024-02-02T02:29:28.085Z #
6512024-02-02T02:29:28.085Z@@ -77,8 +77,8 @@
6522024-02-02T02:29:28.085Z #
6532024-02-02T02:29:28.086Z # For more on this and what to put here, see docs/how-to-run.adoc.
6542024-02-02T02:29:28.086Z [[internal_services_ip_pool_ranges]]
6552024-02-02T02:29:28.086Z-first = "192.168.1.20"
6562024-02-02T02:29:28.086Z-last = "192.168.1.29"
6572024-02-02T02:29:28.086Z+first = "10.151.2.164"
6582024-02-02T02:29:28.086Z+last = "10.151.2.173"
6592024-02-02T02:29:28.086Z
6602024-02-02T02:29:28.086Z # TODO - this configuration is subject to change going forward. Ultimately these
6612024-02-02T02:29:28.086Z # parameters should be provided to the control plane via wicket, but we need to
6622024-02-02T02:29:28.086Z@@ -92,8 +92,8 @@
6632024-02-02T02:29:28.086Z # A range of IP addresses used by Boundary Services on the external network. In
6642024-02-02T02:29:28.086Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6652024-02-02T02:29:28.086Z # With softnpu, only one address is used.
6662024-02-02T02:29:28.086Z-infra_ip_first = "192.168.1.30"
6672024-02-02T02:29:28.086Z-infra_ip_last = "192.168.1.30"
6682024-02-02T02:29:28.086Z+infra_ip_first = "10.151.2.174"
6692024-02-02T02:29:28.086Z+infra_ip_last = "10.151.2.174"
6702024-02-02T02:29:28.086Z
6712024-02-02T02:29:28.086Z # Configurations for BGP routers to run on the scrimlets.
6722024-02-02T02:29:28.086Z bgp = []
6732024-02-02T02:29:28.086Z@@ -101,9 +101,9 @@
6742024-02-02T02:29:28.086Z # You can configure multiple uplinks by repeating the following stanza
6752024-02-02T02:29:28.087Z [[rack_network_config.ports]]
6762024-02-02T02:29:28.087Z # Routes associated with this port.
6772024-02-02T02:29:28.087Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6782024-02-02T02:29:28.087Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6792024-02-02T02:29:28.087Z # Addresses associated with this port.
6802024-02-02T02:29:28.087Z-addresses = ["192.168.1.30/24"]
6812024-02-02T02:29:28.087Z+addresses = ["10.151.2.174/24"]
6822024-02-02T02:29:28.087Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6832024-02-02T02:29:28.087Z port = "qsfp0"
6842024-02-02T02:29:28.087Z # The speed of this port.
6852024-02-02T02:29:28.087Z+ true
6862024-02-02T02:29:28.087Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6872024-02-02T02:29:28.087Za pkg/config-rss.toml 7K
6882024-02-02T02:29:28.087Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6892024-02-02T02:29:28.087Z+ TLS_NAME=recovery.sys.oxide.test
6902024-02-02T02:29:28.087Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6912024-02-02T02:29:28.795Z...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.......+........................+.....+................+...+..+......+...+....+......+.....+.......+........+.......+...+.........+..+.............+...............+..+.+..+..........+........+...+.+......+........+..........+.....+......+.+.....+.........+.............+...............+..+......+....+.........+............+..+......+.+...+..+...................+..+....+........+......+.+...+...........+......+...+................+..+..........+...+...........................+.....+....+........+...+..........+...+..+.........+....+.........+........+..........+..+...............+.......+......+..+.............+........+.+...+......+..+...+.......+..+.+......+.....+......+...............+....+......+...+.....+................+...+....................................+..+..................+.......+........+.......+..+.+..............+..........+..+...+......+............+..........+..+..........+..+...+.+......+........+.......+.....+...+................+..+.........+..........+...+..+.+.....+.......+...+..+............+....+....................+................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6922024-02-02T02:29:29.008Z........+....+...+.................+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+.+.....+..........+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+....+............+...+........+............+............+...+......+................+.....+...+.+.....+.......+........+................+...+......+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6932024-02-02T02:29:29.033Z-----
6942024-02-02T02:29:29.033Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6952024-02-02T02:29:29.033Za pkg/initial-tls-cert.pem 2K
6962024-02-02T02:29:29.033Za pkg/initial-tls-key.pem 4K
6972024-02-02T02:29:29.033Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6982024-02-02T02:29:29.059Z+ rmdir pkg
6992024-02-02T02:29:29.059Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
7002024-02-02T02:29:29.059Z+ pfexec mkdir -p /usr/oxide
7012024-02-02T02:29:29.059Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
7022024-02-02T02:29:29.774Z+ pfexec chmod +x /usr/oxide/oxide
7032024-02-02T02:29:29.811Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
7042024-02-02T02:29:48.071Z+ OMICRON_NO_UNINSTALL=1
7052024-02-02T02:29:48.071Z+ ptime -m pfexec ./target/release/omicron-package -t test install
7062024-02-02T02:29:48.098ZFeb 02 02:29:48.623 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
7072024-02-02T02:29:48.098ZFeb 02 02:29:48.625 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
7082024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7092024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
7102024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7112024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7122024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7132024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7142024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7152024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7162024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7172024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7182024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7192024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter.tar.gz
7202024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7212024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7222024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/oxlog.tar, src: out/oxlog.tar
7232024-02-02T02:29:48.098ZFeb 02 02:29:48.626 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7242024-02-02T02:29:48.378ZFeb 02 02:29:48.912 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7252024-02-02T02:29:48.424ZFeb 02 02:29:48.955 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7262024-02-02T02:29:48.484ZFeb 02 02:29:49.023 INFO Unpacking service tarball, service_path: /opt/oxide/oxlog, tar_path: /opt/oxide/oxlog.tar
7272024-02-02T02:29:48.550ZFeb 02 02:29:49.028 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7282024-02-02T02:29:48.616Z
7292024-02-02T02:29:48.641Zreal 0.540671927
7302024-02-02T02:29:48.641Zuser 0.108170957
7312024-02-02T02:29:48.641Zsys 2.294979478
7322024-02-02T02:29:48.641Ztrap 0.000007738
7332024-02-02T02:29:48.641Ztflt 0.000138052
7342024-02-02T02:29:48.641Zdflt 0.001141149
7352024-02-02T02:29:48.641Zkflt 0.000008349
7362024-02-02T02:29:48.641Zlock 48.606681489
7372024-02-02T02:29:48.641Zslp 0.749337228
7382024-02-02T02:29:48.641Zlat 0.002945555
7392024-02-02T02:29:48.641Zstop 0.194426376
7402024-02-02T02:29:48.641Z+ retry=0
7412024-02-02T02:29:48.641Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7422024-02-02T02:29:48.641Z+ [[ 0 -gt 30 ]]
7432024-02-02T02:29:48.641Z+ sleep 1
7442024-02-02T02:29:49.637Z+ retry=1
7452024-02-02T02:29:49.638Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7462024-02-02T02:29:49.685Z+ [[ 1 -gt 30 ]]
7472024-02-02T02:29:49.685Z+ sleep 1
7482024-02-02T02:29:50.647Z+ retry=2
7492024-02-02T02:29:50.647Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7502024-02-02T02:29:50.701Z+ [[ 2 -gt 30 ]]
7512024-02-02T02:29:50.701Z+ sleep 1
7522024-02-02T02:29:51.663Z+ retry=3
7532024-02-02T02:29:51.664Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7542024-02-02T02:29:51.725Z+ [[ 3 -gt 30 ]]
7552024-02-02T02:29:51.726Z+ sleep 1
7562024-02-02T02:29:52.686Z+ retry=4
7572024-02-02T02:29:52.686Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7582024-02-02T02:30:17.845Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7592024-02-02T02:30:17.905Zlocal v6:
7602024-02-02T02:30:17.905Zfdaf:fdfc:9529:b0cc::1
7612024-02-02T02:30:17.905Zfe80::a820:7ff:fe7f:e102
7622024-02-02T02:30:17.905Zlocal v4:
7632024-02-02T02:30:17.905Z10.151.2.174
7642024-02-02T02:30:17.905Zrouter v6:
7652024-02-02T02:30:17.905Zfd00:1122:3344:1::/64 -> fe80::54c4:aeff:fef3:2f76 (1)
7662024-02-02T02:30:17.905Zfd00:1122:3344:101::/64 -> fe80::54c4:aeff:fef3:2f76 (1)
7672024-02-02T02:30:17.905Zfd00:1122:3344:3::/64 -> fe80::54c4:aeff:fef3:2f76 (1)
7682024-02-02T02:30:17.905Zfd00:1122:3344:2::/64 -> fe80::54c4:aeff:fef3:2f76 (1)
7692024-02-02T02:30:17.905Zfdb0:18c0:4d0d:a02a::/64 -> fe80::54c4:aeff:fef3:2f76 (1)
7702024-02-02T02:30:17.905Zrouter v4_idx:
7712024-02-02T02:30:17.905Z0.0.0.0/0 -> 0
7722024-02-02T02:30:17.905Zrouter v4_routes:
7732024-02-02T02:30:17.905Z0 -> 10.151.2.100 (2)
7742024-02-02T02:30:17.906Zresolver v4:
7752024-02-02T02:30:17.906Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7762024-02-02T02:30:17.906Z10.151.2.174 -> 62:e8:51:de:00:48
7772024-02-02T02:30:17.906Zresolver v6:
7782024-02-02T02:30:17.906Zfe80::54c4:aeff:fef3:2f76 -> 56:c4:ae:f3:2f:76
7792024-02-02T02:30:17.906Zfe80::a820:7ff:fe7f:e102 -> aa:20:07:7f:e1:02
7802024-02-02T02:30:17.906Znat v4:
7812024-02-02T02:30:17.906Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:f4:f6
7822024-02-02T02:30:17.906Znat_v6:
7832024-02-02T02:30:17.907Zport_mac:
7842024-02-02T02:30:17.907Z1: a8:40:25:5f:f9:d7
7852024-02-02T02:30:17.907Z1000: a8:40:25:5f:f9:d6
7862024-02-02T02:30:17.907Zproxy arp:
7872024-02-02T02:30:17.907Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7882024-02-02T02:30:17.907Z+ retry=0
7892024-02-02T02:30:17.907Z+++ zoneadm list -n
7902024-02-02T02:30:17.907Z+++ grep oxz_ntp
7912024-02-02T02:30:17.907Z++ pfexec svcs -z oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a -Hostate oxide/ntp
7922024-02-02T02:30:17.907Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7932024-02-02T02:30:17.907Z++ true
7942024-02-02T02:30:17.907Z+ [[ '' != online ]]
7952024-02-02T02:30:17.907Z+ [[ 0 -gt 60 ]]
7962024-02-02T02:30:17.907Z+ sleep 1
7972024-02-02T02:30:18.914Z+ retry=1
7982024-02-02T02:30:18.957Z+++ zoneadm list -n
7992024-02-02T02:30:18.958Z+++ grep oxz_ntp
8002024-02-02T02:30:18.958Z++ pfexec svcs -z oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a -Hostate oxide/ntp
8012024-02-02T02:30:18.959Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8022024-02-02T02:30:18.959Z++ true
8032024-02-02T02:30:18.959Z+ [[ '' != online ]]
8042024-02-02T02:30:18.960Z+ [[ 1 -gt 60 ]]
8052024-02-02T02:30:18.960Z+ sleep 1
8062024-02-02T02:30:19.936Z+ retry=2
8072024-02-02T02:30:20.384Z+++ zoneadm list -n
8082024-02-02T02:30:20.384Z+++ grep oxz_ntp
8092024-02-02T02:30:20.384Z++ pfexec svcs -z oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a -Hostate oxide/ntp
8102024-02-02T02:30:20.384Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8112024-02-02T02:30:20.385Z++ true
8122024-02-02T02:30:20.385Z+ [[ '' != online ]]
8132024-02-02T02:30:20.385Z+ [[ 2 -gt 60 ]]
8142024-02-02T02:30:20.385Z+ sleep 1
8152024-02-02T02:30:20.973Z+ retry=3
8162024-02-02T02:30:21.032Z+++ zoneadm list -n
8172024-02-02T02:30:21.032Z+++ grep oxz_ntp
8182024-02-02T02:30:21.032Z++ pfexec svcs -z oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a -Hostate oxide/ntp
8192024-02-02T02:30:21.032Z+ [[ disabled != online ]]
8202024-02-02T02:30:21.032Z+ [[ 3 -gt 60 ]]
8212024-02-02T02:30:21.033Z+ sleep 1
8222024-02-02T02:30:21.992Z+ retry=4
8232024-02-02T02:30:22.039Z+++ zoneadm list -n
8242024-02-02T02:30:22.039Z+++ grep oxz_ntp
8252024-02-02T02:30:22.039Z++ pfexec svcs -z oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a -Hostate oxide/ntp
8262024-02-02T02:30:22.039Z+ [[ disabled != online ]]
8272024-02-02T02:30:22.039Z+ [[ 4 -gt 60 ]]
8282024-02-02T02:30:22.039Z+ sleep 1
8292024-02-02T02:30:32.967Z+ retry=5
8302024-02-02T02:30:33.218Z+++ zoneadm list -n
8312024-02-02T02:30:33.218Z+++ grep oxz_ntp
8322024-02-02T02:30:33.218Z++ pfexec svcs -z oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a -Hostate oxide/ntp
8332024-02-02T02:30:33.218Z+ [[ disabled != online ]]
8342024-02-02T02:30:33.219Z+ [[ 5 -gt 60 ]]
8352024-02-02T02:30:33.219Z+ sleep 1
8362024-02-02T02:30:33.219Z+ retry=6
8372024-02-02T02:30:33.219Z+++ zoneadm list -n
8382024-02-02T02:30:33.219Z+++ grep oxz_ntp
8392024-02-02T02:30:33.219Z++ pfexec svcs -z oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a -Hostate oxide/ntp
8402024-02-02T02:30:33.220Z+ [[ disabled != online ]]
8412024-02-02T02:30:33.220Z+ [[ 6 -gt 60 ]]
8422024-02-02T02:30:33.220Z+ sleep 1
8432024-02-02T02:30:33.220Z+ retry=7
8442024-02-02T02:30:33.220Z+++ zoneadm list -n
8452024-02-02T02:30:33.220Z+++ grep oxz_ntp
8462024-02-02T02:30:33.220Z++ pfexec svcs -z oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a -Hostate oxide/ntp
8472024-02-02T02:30:33.220Z+ [[ disabled != online ]]
8482024-02-02T02:30:33.221Z+ [[ 7 -gt 60 ]]
8492024-02-02T02:30:33.221Z+ sleep 1
8502024-02-02T02:30:33.221Z+ retry=8
8512024-02-02T02:30:33.221Z+++ zoneadm list -n
8522024-02-02T02:30:33.221Z+++ grep oxz_ntp
8532024-02-02T02:30:33.221Z++ pfexec svcs -z oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a -Hostate oxide/ntp
8542024-02-02T02:30:33.221Z+ [[ disabled != online ]]
8552024-02-02T02:30:33.221Z+ [[ 8 -gt 60 ]]
8562024-02-02T02:30:33.222Z+ sleep 1
8572024-02-02T02:30:33.418Z+ retry=9
8582024-02-02T02:30:33.565Z+++ zoneadm list -n
8592024-02-02T02:30:33.565Z+++ grep oxz_ntp
8602024-02-02T02:30:33.565Z++ pfexec svcs -z oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a -Hostate oxide/ntp
8612024-02-02T02:30:33.565Z+ [[ disabled != online ]]
8622024-02-02T02:30:33.565Z+ [[ 9 -gt 60 ]]
8632024-02-02T02:30:33.565Z+ sleep 1
8642024-02-02T02:30:33.565Z+ retry=10
8652024-02-02T02:30:33.565Z+++ zoneadm list -n
8662024-02-02T02:30:33.565Z+++ grep oxz_ntp
8672024-02-02T02:30:33.565Z++ pfexec svcs -z oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a -Hostate oxide/ntp
8682024-02-02T02:30:33.565Z+ [[ online != online ]]
8692024-02-02T02:30:33.565Z+ echo 'Waited for chrony: 10s'
8702024-02-02T02:30:33.565Z+ export RUST_BACKTRACE=1
8712024-02-02T02:30:33.565ZWaited for chrony: 10s
8722024-02-02T02:30:33.565Z+ RUST_BACKTRACE=1
8732024-02-02T02:30:33.565Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8742024-02-02T02:30:33.565Z++ ./tests/bootstrap
8752024-02-02T02:30:33.566ZUsing external DNS server at 10.151.2.164:53
8762024-02-02T02:30:33.566Z2024-02-02 02:30:28.711826414 UTC: attempting to log into API
8772024-02-02T02:30:43.221Z2024-02-02 02:30:43.183964369 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8782024-02-02T02:30:44.227Z2024-02-02 02:30:44.187580120 UTC: attempting to log into API
8792024-02-02T02:30:59.245Z2024-02-02 02:30:59.228359005 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8802024-02-02T02:31:00.246Z2024-02-02 02:31:00.229384425 UTC: attempting to log into API
8812024-02-02T02:31:15.280Z2024-02-02 02:31:15.263090799 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8822024-02-02T02:31:16.281Z2024-02-02 02:31:16.265053073 UTC: attempting to log into API
 
8832024-02-02T02:31:25.921ZError: No response to keepalive - Terminating session
 
8842024-02-02T02:31:31.314Z2024-02-02 02:31:31.297644349 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8852024-02-02T02:31:32.315Z2024-02-02 02:31:32.298565997 UTC: attempting to log into API
 
8862024-02-02T02:31:35.838ZError: No response de-activating SOL payload
8872024-02-02T02:31:37.230Z[SOL Session operational. Use ~? for help]
 
8882024-02-02T02:31:47.348Z2024-02-02 02:31:47.337505886 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8892024-02-02T02:31:48.349Z2024-02-02 02:31:48.339217424 UTC: attempting to log into API
8902024-02-02T02:32:03.394Z2024-02-02 02:32:03.382632707 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8912024-02-02T02:32:04.385Z2024-02-02 02:32:04.384068582 UTC: attempting to log into API
8922024-02-02T02:32:42.226Z2024-02-02 02:32:19.439672799 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8932024-02-02T02:32:42.571Z2024-02-02 02:32:20.441186118 UTC: attempting to log into API
8942024-02-02T02:32:43.321Z2024-02-02 02:32:35.482249477 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8952024-02-02T02:32:43.321Z2024-02-02 02:32:36.483875707 UTC: attempting to log into API
8962024-02-02T02:32:52.980Z2024-02-02 02:32:52.481348037 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8972024-02-02T02:32:53.493Z2024-02-02 02:32:53.480601910 UTC: attempting to log into API
8982024-02-02T02:33:08.679Z2024-02-02 02:33:08.655018577 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8992024-02-02T02:33:09.672Z2024-02-02 02:33:09.656066598 UTC: attempting to log into API
9002024-02-02T02:33:24.710Z2024-02-02 02:33:24.694159793 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9012024-02-02T02:33:25.711Z2024-02-02 02:33:25.695186893 UTC: attempting to log into API
9022024-02-02T02:33:40.748Z2024-02-02 02:33:40.728708825 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9032024-02-02T02:33:41.747Z2024-02-02 02:33:41.730744121 UTC: attempting to log into API
9042024-02-02T02:33:56.788Z2024-02-02 02:33:56.764970904 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9052024-02-02T02:33:57.783Z2024-02-02 02:33:57.766892536 UTC: attempting to log into API
9062024-02-02T02:34:12.816Z2024-02-02 02:34:12.801055715 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9072024-02-02T02:34:13.817Z2024-02-02 02:34:13.802204498 UTC: attempting to log into API
9082024-02-02T02:34:25.838Z2024-02-02 02:34:24.582733070 UTC: login succeeded
9092024-02-02T02:34:25.838Zwaiting for nexus to come up...
9102024-02-02T02:34:25.838Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9112024-02-02T02:34:25.838Zsetting up quotas...
9122024-02-02T02:34:25.839Zensuring datasets are ready...
 
9132024-02-02T02:34:25.948ZError: No response to keepalive - Terminating session
 
9142024-02-02T02:34:27.139Zlet's roll.
9152024-02-02T02:34:27.164Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
9162024-02-02T02:34:27.164ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9172024-02-02T02:34:27.164ZOXIDE_TOKEN=oxide-token-ca802f56556c39caafe89eb2a368ed17a39de536'
9182024-02-02T02:34:27.164Z++ OXIDE_HOST=https://recovery.sys.oxide.test
9192024-02-02T02:34:27.164Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9202024-02-02T02:34:27.164Z++ OXIDE_TOKEN=oxide-token-ca802f56556c39caafe89eb2a368ed17a39de536
9212024-02-02T02:34:27.164Z+ export OXIDE_HOST OXIDE_TOKEN
9222024-02-02T02:34:27.164Z+ retry=0
9232024-02-02T02:34:27.164Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9242024-02-02T02:34:27.236Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
9252024-02-02T02:34:27.471Z{"status":"ok"}success
9262024-02-02T02:34:27.471ZProject {
9272024-02-02T02:34:27.471Z description: "some images",
9282024-02-02T02:34:27.471Z id: 80403229-2318-49cb-aa5c-3ba10b72a2bd,
9292024-02-02T02:34:27.471Z name: Name(
9302024-02-02T02:34:27.471Z "images",
9312024-02-02T02:34:27.497Z ),
9322024-02-02T02:34:27.497Z time_created: 2024-02-02T02:34:27.286943Z,
9332024-02-02T02:34:27.497Z time_modified: 2024-02-02T02:34:27.286943Z,
9342024-02-02T02:34:27.497Z}
9352024-02-02T02:34:27.497Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
 
9362024-02-02T02:34:35.749ZError: No response de-activating SOL payload
9372024-02-02T02:34:37.111Z[SOL Session operational. Use ~? for help]
9382024-02-02T02:37:25.488ZError: No response to keepalive - Terminating session
9392024-02-02T02:37:35.563ZError: No response de-activating SOL payload
9402024-02-02T02:37:36.670Z[SOL Session operational. Use ~? for help]
 
9412024-02-02T02:40:11.928Zdone!
9422024-02-02T02:40:11.928Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
9432024-02-02T02:40:11.929Zsuccess
9442024-02-02T02:40:11.929ZImage {
9452024-02-02T02:40:11.929Z block_size: ByteCount(
9462024-02-02T02:40:11.929Z 512,
9472024-02-02T02:40:11.929Z ),
9482024-02-02T02:40:11.929Z description: "debian 11 original base image",
9492024-02-02T02:40:11.929Z digest: None,
9502024-02-02T02:40:11.929Z id: 6abf2eaf-4f6f-4993-91d8-f38323603a51,
9512024-02-02T02:40:11.930Z name: Name(
9522024-02-02T02:40:11.930Z "debian11",
9532024-02-02T02:40:11.930Z ),
9542024-02-02T02:40:11.930Z os: "debian",
9552024-02-02T02:40:11.930Z project_id: None,
9562024-02-02T02:40:11.930Z size: ByteCount(
9572024-02-02T02:40:11.931Z 2147483648,
9582024-02-02T02:40:11.931Z ),
9592024-02-02T02:40:11.931Z time_created: 2024-02-02T02:40:06.812820Z,
9602024-02-02T02:40:11.931Z time_modified: 2024-02-02T02:40:07.042648Z,
9612024-02-02T02:40:11.931Z url: None,
9622024-02-02T02:40:11.931Z version: "11",
9632024-02-02T02:40:11.931Z}
9642024-02-02T02:40:11.931Z+ rm ./tests/bootstrap
9652024-02-02T02:40:11.931Z+ for test_bin in tests/*
9662024-02-02T02:40:11.931Z+ ./tests/bootstrap-0096297e5e63ab8e
9672024-02-02T02:40:11.932Z
9682024-02-02T02:40:11.932Zrunning 0 tests
9692024-02-02T02:40:11.932Z
9702024-02-02T02:40:11.932Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9712024-02-02T02:40:11.932Z
9722024-02-02T02:40:11.932Z+ for test_bin in tests/*
9732024-02-02T02:40:11.932Z+ ./tests/end_to_end_tests-6d4886eeea56eeb3
9742024-02-02T02:40:11.932Z
9752024-02-02T02:40:11.933Zrunning 2 tests
9762024-02-02T02:40:11.933Ztest no_spoof::no_spoof_here ... ok
 
9772024-02-02T02:40:25.405ZError: No response to keepalive - Terminating session
9782024-02-02T02:40:35.607ZError: No response de-activating SOL payload
9792024-02-02T02:40:37.410Z[SOL Session operational. Use ~? for help]
 
9802024-02-02T02:40:40.939Ztest instance_launch::instance_launch ... ok
9812024-02-02T02:40:40.939Z
9822024-02-02T02:40:40.939Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 33.67s
9832024-02-02T02:40:40.939Z
9842024-02-02T02:40:40.968Z+ _exit_trap
9852024-02-02T02:40:40.968Z+ local status=0
9862024-02-02T02:40:40.968Z+ [[ 0 -eq 0 ]]
9872024-02-02T02:40:40.968Z+ exit 0
9882024-02-02T02:40:40.969Zprocess exited: duration 780915 ms, exit code 0
 
9892024-02-02T02:40:41.100Zfound 72 output files
9902024-02-02T02:40:41.100Zuploading: /var/svc/log/oxide-sled-agent:default.log (842432 bytes)
9912024-02-02T02:40:42.893Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 842432 -> 842958
9922024-02-02T02:40:43.969Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9932024-02-02T02:40:43.969Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log (2690540 bytes)
9942024-02-02T02:40:46.183Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log" changed size mid upload: 2690540 -> 2710241
9952024-02-02T02:40:47.267Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log
9962024-02-02T02:40:47.267Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log (92544 bytes)
9972024-02-02T02:40:47.527Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log" changed size mid upload: 92544 -> 93517
9982024-02-02T02:40:48.599Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log
9992024-02-02T02:40:48.599Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log (117219 bytes)
10002024-02-02T02:40:48.880Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log" changed size mid upload: 117219 -> 118410
10012024-02-02T02:40:49.953Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log
10022024-02-02T02:40:49.953Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log (1357158 bytes)
10032024-02-02T02:40:51.226Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log" changed size mid upload: 1357158 -> 1378724
10042024-02-02T02:40:52.304Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log
10052024-02-02T02:40:52.304Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log (832343 bytes)
10062024-02-02T02:40:53.021Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log" changed size mid upload: 832343 -> 845515
10072024-02-02T02:40:54.103Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log
10082024-02-02T02:40:54.103Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log (1388 bytes)
10092024-02-02T02:40:55.206Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log
10102024-02-02T02:40:55.206Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log (304501 bytes)
10112024-02-02T02:40:55.577Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log" changed size mid upload: 304501 -> 312563
10122024-02-02T02:40:56.668Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log
10132024-02-02T02:40:56.668Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log (617 bytes)
10142024-02-02T02:40:58.171Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log
10152024-02-02T02:40:58.171Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log (22283 bytes)
10162024-02-02T02:40:59.314Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log
10172024-02-02T02:40:59.314Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_24dd9778-e2bb-40f6-80bd-6f4f19562b54/root/var/svc/log/oxide-clickhouse:default.log (775284 bytes)
10182024-02-02T02:41:00.297Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_24dd9778-e2bb-40f6-80bd-6f4f19562b54/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 775284 -> 802009
10192024-02-02T02:41:01.375Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_24dd9778-e2bb-40f6-80bd-6f4f19562b54/root/var/svc/log/oxide-clickhouse:default.log
10202024-02-02T02:41:01.400Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_24dd9778-e2bb-40f6-80bd-6f4f19562b54/root/var/svc/log/oxide-zone-network-setup:default.log (1708 bytes)
10212024-02-02T02:41:02.474Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_24dd9778-e2bb-40f6-80bd-6f4f19562b54/root/var/svc/log/oxide-zone-network-setup:default.log
10222024-02-02T02:41:02.474Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_4e25e20f-6e69-473b-96ba-512d4e6da97e/root/var/svc/log/oxide-cockroachdb:default.log (4406 bytes)
10232024-02-02T02:41:03.591Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_4e25e20f-6e69-473b-96ba-512d4e6da97e/root/var/svc/log/oxide-cockroachdb:default.log
10242024-02-02T02:41:03.616Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_4e25e20f-6e69-473b-96ba-512d4e6da97e/root/var/svc/log/oxide-zone-network-setup:default.log (1705 bytes)
10252024-02-02T02:41:04.691Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_4e25e20f-6e69-473b-96ba-512d4e6da97e/root/var/svc/log/oxide-zone-network-setup:default.log
10262024-02-02T02:41:04.691Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7ad3498-0054-48c8-ab6b-7c2e110a8cfe/root/var/svc/log/oxide-crucible-agent:default.log (38504 bytes)
10272024-02-02T02:41:05.867Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7ad3498-0054-48c8-ab6b-7c2e110a8cfe/root/var/svc/log/oxide-crucible-agent:default.log
10282024-02-02T02:41:05.891Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7ad3498-0054-48c8-ab6b-7c2e110a8cfe/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10292024-02-02T02:41:06.962Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7ad3498-0054-48c8-ab6b-7c2e110a8cfe/root/var/svc/log/oxide-crucible-downstairs:default.log
10302024-02-02T02:41:06.962Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7ad3498-0054-48c8-ab6b-7c2e110a8cfe/root/var/svc/log/oxide-crucible-downstairs:downstairs-1f771e14-a636-4132-aa03-ab19ee2067ee.log (3049 bytes)
10312024-02-02T02:41:08.095Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7ad3498-0054-48c8-ab6b-7c2e110a8cfe/root/var/svc/log/oxide-crucible-downstairs:downstairs-1f771e14-a636-4132-aa03-ab19ee2067ee.log
10322024-02-02T02:41:08.096Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7ad3498-0054-48c8-ab6b-7c2e110a8cfe/root/var/svc/log/oxide-crucible-downstairs:downstairs-23b569b8-2028-49a3-8a78-281eb5c72e3a.log (5620 bytes)
10332024-02-02T02:41:09.205Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7ad3498-0054-48c8-ab6b-7c2e110a8cfe/root/var/svc/log/oxide-crucible-downstairs:downstairs-23b569b8-2028-49a3-8a78-281eb5c72e3a.log
10342024-02-02T02:41:09.205Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7ad3498-0054-48c8-ab6b-7c2e110a8cfe/root/var/svc/log/oxide-crucible-downstairs:downstairs-c6d1f399-715d-4127-b750-6b5afc719b0e.log (5533 bytes)
10352024-02-02T02:41:10.340Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7ad3498-0054-48c8-ab6b-7c2e110a8cfe/root/var/svc/log/oxide-crucible-downstairs:downstairs-c6d1f399-715d-4127-b750-6b5afc719b0e.log
10362024-02-02T02:41:10.340Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7ad3498-0054-48c8-ab6b-7c2e110a8cfe/root/var/svc/log/oxide-crucible-downstairs:snapshot-23b569b8-2028-49a3-8a78-281eb5c72e3a-043d67e6-7244-453f-b2d4-b5c90f72c8d3.log (5178 bytes)
10372024-02-02T02:41:11.445Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c7ad3498-0054-48c8-ab6b-7c2e110a8cfe/root/var/svc/log/oxide-crucible-downstairs:snapshot-23b569b8-2028-49a3-8a78-281eb5c72e3a-043d67e6-7244-453f-b2d4-b5c90f72c8d3.log
10382024-02-02T02:41:11.445Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_90c5a2b5-0e85-47a8-9b62-f32aa19a66f1/root/var/svc/log/oxide-external_dns:default.log (26944 bytes)
10392024-02-02T02:41:11.613Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_90c5a2b5-0e85-47a8-9b62-f32aa19a66f1/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 26944 -> 30721
10402024-02-02T02:41:12.687Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_90c5a2b5-0e85-47a8-9b62-f32aa19a66f1/root/var/svc/log/oxide-external_dns:default.log
10412024-02-02T02:41:12.687Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_1d065971-2686-487a-8f5e-511347efa8cd/root/var/svc/log/oxide-internal_dns:default.log (111914 bytes)
10422024-02-02T02:41:13.037Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_1d065971-2686-487a-8f5e-511347efa8cd/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 111914 -> 115669
10432024-02-02T02:41:14.132Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_1d065971-2686-487a-8f5e-511347efa8cd/root/var/svc/log/oxide-internal_dns:default.log
10442024-02-02T02:41:14.132Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f3a20ef2-a5b9-41fb-83e2-b4ad96e61f0f/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
10452024-02-02T02:41:15.651Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f3a20ef2-a5b9-41fb-83e2-b4ad96e61f0f/root/var/svc/log/oxide-cockroachdb:default.log
10462024-02-02T02:41:15.679Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f3a20ef2-a5b9-41fb-83e2-b4ad96e61f0f/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
10472024-02-02T02:41:16.356Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f3a20ef2-a5b9-41fb-83e2-b4ad96e61f0f/root/var/svc/log/oxide-zone-network-setup:default.log
10482024-02-02T02:41:16.356Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_081b7722-c721-4816-a7b0-0154d7710ffc/root/var/svc/log/oxide-crucible-agent:default.log (20034 bytes)
10492024-02-02T02:41:17.508Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_081b7722-c721-4816-a7b0-0154d7710ffc/root/var/svc/log/oxide-crucible-agent:default.log
10502024-02-02T02:41:17.536Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_081b7722-c721-4816-a7b0-0154d7710ffc/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10512024-02-02T02:41:18.629Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_081b7722-c721-4816-a7b0-0154d7710ffc/root/var/svc/log/oxide-crucible-downstairs:default.log
10522024-02-02T02:41:18.629Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_081b7722-c721-4816-a7b0-0154d7710ffc/root/var/svc/log/oxide-crucible-downstairs:downstairs-4c33b0ae-d187-4a92-b70d-bb1b49e3f6c4.log (2945 bytes)
10532024-02-02T02:41:20.155Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_081b7722-c721-4816-a7b0-0154d7710ffc/root/var/svc/log/oxide-crucible-downstairs:downstairs-4c33b0ae-d187-4a92-b70d-bb1b49e3f6c4.log
10542024-02-02T02:41:20.155Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_081b7722-c721-4816-a7b0-0154d7710ffc/root/var/svc/log/oxide-crucible-downstairs:downstairs-e605d087-604e-4454-b81d-e5a17195cfd0.log (3049 bytes)
10552024-02-02T02:41:21.960Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_081b7722-c721-4816-a7b0-0154d7710ffc/root/var/svc/log/oxide-crucible-downstairs:downstairs-e605d087-604e-4454-b81d-e5a17195cfd0.log
10562024-02-02T02:41:21.960Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_628862e6-710a-4328-9fcc-f513ee96ea86/root/var/svc/log/oxide-crucible-pantry:default.log (1226258 bytes)
10572024-02-02T02:41:23.842Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_628862e6-710a-4328-9fcc-f513ee96ea86/root/var/svc/log/oxide-crucible-pantry:default.log
10582024-02-02T02:41:23.872Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_b73919b3-a3ba-40de-85b0-3e773d575848/root/var/svc/log/oxide-external_dns:default.log (26932 bytes)
10592024-02-02T02:41:23.955Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_b73919b3-a3ba-40de-85b0-3e773d575848/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 26932 -> 30709
10602024-02-02T02:41:25.046Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_b73919b3-a3ba-40de-85b0-3e773d575848/root/var/svc/log/oxide-external_dns:default.log
10612024-02-02T02:41:25.046Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_31ab0d1c-b92d-47bd-b5b1-007e18282b6a/root/var/svc/log/oxide-internal_dns:default.log (111204 bytes)
10622024-02-02T02:41:25.299Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_31ab0d1c-b92d-47bd-b5b1-007e18282b6a/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 111204 -> 114960
10632024-02-02T02:41:26.393Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_31ab0d1c-b92d-47bd-b5b1-007e18282b6a/root/var/svc/log/oxide-internal_dns:default.log
10642024-02-02T02:41:26.393Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f4b1b827-f83d-47c1-b3b9-b6d642591523/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
10652024-02-02T02:41:27.549Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f4b1b827-f83d-47c1-b3b9-b6d642591523/root/var/svc/log/oxide-cockroachdb:default.log
10662024-02-02T02:41:27.549Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f4b1b827-f83d-47c1-b3b9-b6d642591523/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
10672024-02-02T02:41:28.717Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f4b1b827-f83d-47c1-b3b9-b6d642591523/root/var/svc/log/oxide-zone-network-setup:default.log
10682024-02-02T02:41:28.717Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_d7f54cfd-6f98-4918-ac58-a0b1ac19ada0/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10692024-02-02T02:41:29.839Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_d7f54cfd-6f98-4918-ac58-a0b1ac19ada0/root/var/svc/log/oxide-crucible-agent:default.log
10702024-02-02T02:41:29.840Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_d7f54cfd-6f98-4918-ac58-a0b1ac19ada0/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10712024-02-02T02:41:32.586Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_d7f54cfd-6f98-4918-ac58-a0b1ac19ada0/root/var/svc/log/oxide-crucible-downstairs:default.log
10722024-02-02T02:41:32.586Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_6c42df6c-4bd7-4f93-bb73-e41cf005a093/root/var/svc/log/oxide-internal_dns:default.log (111313 bytes)
10732024-02-02T02:41:33.014Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_6c42df6c-4bd7-4f93-bb73-e41cf005a093/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 111313 -> 115067
10742024-02-02T02:41:34.097Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_6c42df6c-4bd7-4f93-bb73-e41cf005a093/root/var/svc/log/oxide-internal_dns:default.log
10752024-02-02T02:41:34.131Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_35d16852-24e6-4638-afff-e4014ff0066e/root/var/svc/log/oxide-oximeter:default.log (237820 bytes)
10762024-02-02T02:41:34.711Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_35d16852-24e6-4638-afff-e4014ff0066e/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 237820 -> 271541
10772024-02-02T02:41:35.877Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_35d16852-24e6-4638-afff-e4014ff0066e/root/var/svc/log/oxide-oximeter:default.log
10782024-02-02T02:41:35.877Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_35d16852-24e6-4638-afff-e4014ff0066e/root/var/svc/log/oxide-zone-network-setup:default.log (1697 bytes)
10792024-02-02T02:41:37.010Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_35d16852-24e6-4638-afff-e4014ff0066e/root/var/svc/log/oxide-zone-network-setup:default.log
10802024-02-02T02:41:37.010Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_6f281b3e-c68a-480e-998b-77e2438eb4a2/root/var/svc/log/oxide-cockroachdb:default.log (4407 bytes)
10812024-02-02T02:41:38.109Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_6f281b3e-c68a-480e-998b-77e2438eb4a2/root/var/svc/log/oxide-cockroachdb:default.log
10822024-02-02T02:41:38.109Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_6f281b3e-c68a-480e-998b-77e2438eb4a2/root/var/svc/log/oxide-zone-network-setup:default.log (1705 bytes)
10832024-02-02T02:41:39.214Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_6f281b3e-c68a-480e-998b-77e2438eb4a2/root/var/svc/log/oxide-zone-network-setup:default.log
10842024-02-02T02:41:39.214Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_93302d5a-1953-467e-8410-c5af44cdcf40/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10852024-02-02T02:41:40.349Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_93302d5a-1953-467e-8410-c5af44cdcf40/root/var/svc/log/oxide-crucible-agent:default.log
10862024-02-02T02:41:40.403Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_93302d5a-1953-467e-8410-c5af44cdcf40/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10872024-02-02T02:41:41.504Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_93302d5a-1953-467e-8410-c5af44cdcf40/root/var/svc/log/oxide-crucible-downstairs:default.log
10882024-02-02T02:41:41.504Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_33ffd49e-b20b-448c-b783-fa94dc371807/root/var/svc/log/oxide-cockroachdb:default.log (4381 bytes)
10892024-02-02T02:41:42.788Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_33ffd49e-b20b-448c-b783-fa94dc371807/root/var/svc/log/oxide-cockroachdb:default.log
10902024-02-02T02:41:42.788Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_33ffd49e-b20b-448c-b783-fa94dc371807/root/var/svc/log/oxide-zone-network-setup:default.log (1706 bytes)
10912024-02-02T02:41:43.904Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_33ffd49e-b20b-448c-b783-fa94dc371807/root/var/svc/log/oxide-zone-network-setup:default.log
10922024-02-02T02:41:43.904Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_670beb6f-4fc9-4fed-b03f-652a8d62fc10/root/var/svc/log/oxide-crucible-agent:default.log (27795 bytes)
10932024-02-02T02:41:45.152Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_670beb6f-4fc9-4fed-b03f-652a8d62fc10/root/var/svc/log/oxide-crucible-agent:default.log
10942024-02-02T02:41:45.152Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_670beb6f-4fc9-4fed-b03f-652a8d62fc10/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10952024-02-02T02:41:46.370Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_670beb6f-4fc9-4fed-b03f-652a8d62fc10/root/var/svc/log/oxide-crucible-downstairs:default.log
10962024-02-02T02:41:46.371Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_670beb6f-4fc9-4fed-b03f-652a8d62fc10/root/var/svc/log/oxide-crucible-downstairs:downstairs-cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea.log (5539 bytes)
10972024-02-02T02:41:47.516Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_670beb6f-4fc9-4fed-b03f-652a8d62fc10/root/var/svc/log/oxide-crucible-downstairs:downstairs-cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea.log
10982024-02-02T02:41:47.516Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_670beb6f-4fc9-4fed-b03f-652a8d62fc10/root/var/svc/log/oxide-crucible-downstairs:downstairs-e4cfe51f-c926-4d1a-a47a-20a931173f1a.log (5622 bytes)
10992024-02-02T02:42:03.988Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_670beb6f-4fc9-4fed-b03f-652a8d62fc10/root/var/svc/log/oxide-crucible-downstairs:downstairs-e4cfe51f-c926-4d1a-a47a-20a931173f1a.log
11002024-02-02T02:42:03.988Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_670beb6f-4fc9-4fed-b03f-652a8d62fc10/root/var/svc/log/oxide-crucible-downstairs:snapshot-e4cfe51f-c926-4d1a-a47a-20a931173f1a-043d67e6-7244-453f-b2d4-b5c90f72c8d3.log (5178 bytes)
11012024-02-02T02:42:05.078Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_670beb6f-4fc9-4fed-b03f-652a8d62fc10/root/var/svc/log/oxide-crucible-downstairs:snapshot-e4cfe51f-c926-4d1a-a47a-20a931173f1a-043d67e6-7244-453f-b2d4-b5c90f72c8d3.log
11022024-02-02T02:42:05.078Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a/root/var/svc/log/oxide-ntp:default.log (1051 bytes)
11032024-02-02T02:42:06.177Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a/root/var/svc/log/oxide-ntp:default.log
11042024-02-02T02:42:06.177Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6b974e8c-42be-44c5-901f-a458a8dc7988/root/var/svc/log/oxide-crucible-agent:default.log (3433 bytes)
11052024-02-02T02:42:07.602Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6b974e8c-42be-44c5-901f-a458a8dc7988/root/var/svc/log/oxide-crucible-agent:default.log
11062024-02-02T02:42:07.602Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6b974e8c-42be-44c5-901f-a458a8dc7988/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11072024-02-02T02:42:08.404Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6b974e8c-42be-44c5-901f-a458a8dc7988/root/var/svc/log/oxide-crucible-downstairs:default.log
11082024-02-02T02:42:08.404Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_fbed8bbd-e751-4e14-a41f-f2c95e5df4b7/root/var/svc/log/oxide-crucible-pantry:default.log (2332 bytes)
11092024-02-02T02:42:09.519Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_fbed8bbd-e751-4e14-a41f-f2c95e5df4b7/root/var/svc/log/oxide-crucible-pantry:default.log
11102024-02-02T02:42:09.555Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_55bf155d-6674-42cb-b501-3d2634fe68fc/root/var/svc/log/oxide-nexus:default.log (1190858 bytes)
11112024-02-02T02:42:10.263Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_55bf155d-6674-42cb-b501-3d2634fe68fc/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1190858 -> 1391535
11122024-02-02T02:42:11.342Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_55bf155d-6674-42cb-b501-3d2634fe68fc/root/var/svc/log/oxide-nexus:default.log
11132024-02-02T02:42:11.342Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_220f853f-cd29-408e-bbaa-98a0bc967e5c/root/var/svc/log/oxide-crucible-agent:default.log (23475 bytes)
11142024-02-02T02:42:12.512Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_220f853f-cd29-408e-bbaa-98a0bc967e5c/root/var/svc/log/oxide-crucible-agent:default.log
11152024-02-02T02:42:12.512Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_220f853f-cd29-408e-bbaa-98a0bc967e5c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11162024-02-02T02:42:13.615Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_220f853f-cd29-408e-bbaa-98a0bc967e5c/root/var/svc/log/oxide-crucible-downstairs:default.log
11172024-02-02T02:42:13.615Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_220f853f-cd29-408e-bbaa-98a0bc967e5c/root/var/svc/log/oxide-crucible-downstairs:downstairs-373c4fb4-2eda-4fdd-a321-af2a9205b904.log (5619 bytes)
11182024-02-02T02:42:14.738Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_220f853f-cd29-408e-bbaa-98a0bc967e5c/root/var/svc/log/oxide-crucible-downstairs:downstairs-373c4fb4-2eda-4fdd-a321-af2a9205b904.log
11192024-02-02T02:42:14.738Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_220f853f-cd29-408e-bbaa-98a0bc967e5c/root/var/svc/log/oxide-crucible-downstairs:downstairs-50d30a45-abea-4203-852f-cd0fad5c43fd.log (2945 bytes)
11202024-02-02T02:42:16.258Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_220f853f-cd29-408e-bbaa-98a0bc967e5c/root/var/svc/log/oxide-crucible-downstairs:downstairs-50d30a45-abea-4203-852f-cd0fad5c43fd.log
11212024-02-02T02:42:16.258Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_220f853f-cd29-408e-bbaa-98a0bc967e5c/root/var/svc/log/oxide-crucible-downstairs:snapshot-373c4fb4-2eda-4fdd-a321-af2a9205b904-043d67e6-7244-453f-b2d4-b5c90f72c8d3.log (5177 bytes)
11222024-02-02T02:42:17.363Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_220f853f-cd29-408e-bbaa-98a0bc967e5c/root/var/svc/log/oxide-crucible-downstairs:snapshot-373c4fb4-2eda-4fdd-a321-af2a9205b904-043d67e6-7244-453f-b2d4-b5c90f72c8d3.log
11232024-02-02T02:42:17.363Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_ee3aefdb-02f1-4fd0-a1df-ca09e33df7cf/root/var/svc/log/oxide-nexus:default.log (1202024 bytes)
11242024-02-02T02:42:18.108Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_ee3aefdb-02f1-4fd0-a1df-ca09e33df7cf/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1202024 -> 1406604
11252024-02-02T02:42:19.182Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_ee3aefdb-02f1-4fd0-a1df-ca09e33df7cf/root/var/svc/log/oxide-nexus:default.log
11262024-02-02T02:42:19.208Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d12933ea-f85c-4c37-af14-1cbabd25e66d/root/var/svc/log/oxide-crucible-agent:default.log (19610 bytes)
11272024-02-02T02:42:20.338Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d12933ea-f85c-4c37-af14-1cbabd25e66d/root/var/svc/log/oxide-crucible-agent:default.log
11282024-02-02T02:42:20.338Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d12933ea-f85c-4c37-af14-1cbabd25e66d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11292024-02-02T02:42:21.439Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d12933ea-f85c-4c37-af14-1cbabd25e66d/root/var/svc/log/oxide-crucible-downstairs:default.log
11302024-02-02T02:42:21.439Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d12933ea-f85c-4c37-af14-1cbabd25e66d/root/var/svc/log/oxide-crucible-downstairs:downstairs-628fd1ac-454b-4411-a471-9ffc067a31b1.log (2945 bytes)
11312024-02-02T02:42:22.563Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d12933ea-f85c-4c37-af14-1cbabd25e66d/root/var/svc/log/oxide-crucible-downstairs:downstairs-628fd1ac-454b-4411-a471-9ffc067a31b1.log
11322024-02-02T02:42:22.563Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d12933ea-f85c-4c37-af14-1cbabd25e66d/root/var/svc/log/oxide-crucible-downstairs:downstairs-ce3f3a05-4c4f-4760-a8be-e486600e925a.log (3050 bytes)
11332024-02-02T02:42:23.677Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d12933ea-f85c-4c37-af14-1cbabd25e66d/root/var/svc/log/oxide-crucible-downstairs:downstairs-ce3f3a05-4c4f-4760-a8be-e486600e925a.log
11342024-02-02T02:42:23.677Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_7cfc4997-b9b5-4dec-9cb4-d4900dea0fc9/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
11352024-02-02T02:42:25.218Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_7cfc4997-b9b5-4dec-9cb4-d4900dea0fc9/root/var/svc/log/oxide-crucible-pantry:default.log
11362024-02-02T02:42:25.218Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d45660a7-4dd2-4024-90f6-52d9a78db041/root/var/svc/log/oxide-crucible-agent:default.log (14580 bytes)
11372024-02-02T02:42:26.380Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d45660a7-4dd2-4024-90f6-52d9a78db041/root/var/svc/log/oxide-crucible-agent:default.log
11382024-02-02T02:42:26.380Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d45660a7-4dd2-4024-90f6-52d9a78db041/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11392024-02-02T02:42:27.488Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d45660a7-4dd2-4024-90f6-52d9a78db041/root/var/svc/log/oxide-crucible-downstairs:default.log
11402024-02-02T02:42:27.488Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d45660a7-4dd2-4024-90f6-52d9a78db041/root/var/svc/log/oxide-crucible-downstairs:downstairs-f2edd028-334a-4d75-85cc-afbde81225b7.log (5537 bytes)
11412024-02-02T02:42:28.582Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d45660a7-4dd2-4024-90f6-52d9a78db041/root/var/svc/log/oxide-crucible-downstairs:downstairs-f2edd028-334a-4d75-85cc-afbde81225b7.log
11422024-02-02T02:42:28.582Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5389af38-0610-47be-932d-c3afed3ab71c/root/var/svc/log/oxide-nexus:default.log (36098756 bytes)
11432024-02-02T02:42:51.556Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5389af38-0610-47be-932d-c3afed3ab71c/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 36098756 -> 36337855
11442024-02-02T02:42:52.634Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5389af38-0610-47be-932d-c3afed3ab71c/root/var/svc/log/oxide-nexus:default.log
11452024-02-02T02:42:52.634Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a/root/var/log/chrony/measurements.log (20002 bytes)
11462024-02-02T02:42:52.767Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a/root/var/log/chrony/measurements.log" changed size mid upload: 20002 -> 23838
11472024-02-02T02:42:53.841Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a/root/var/log/chrony/measurements.log
11482024-02-02T02:42:53.841Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a/root/var/log/chrony/statistics.log (13221 bytes)
11492024-02-02T02:42:53.932Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a/root/var/log/chrony/statistics.log" changed size mid upload: 13221 -> 16848
11502024-02-02T02:42:55.023Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a/root/var/log/chrony/statistics.log
11512024-02-02T02:42:55.054Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a/root/var/log/chrony/tracking.log (1983 bytes)
11522024-02-02T02:42:55.079Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a/root/var/log/chrony/tracking.log" changed size mid upload: 1983 -> 2247
11532024-02-02T02:42:56.147Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_2aeb7627-e9ea-44c0-93b4-69a902bdde6a/root/var/log/chrony/tracking.log