01HQJS9DGDTP3C32MHRPMNTRHS: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.28",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HQJS9XPNQHDHED3RQ8H7D0ZB

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12024-02-26T14:01:36.608Zjob dependencies complete; ready to run (waiting for 38 m 55 s)
22024-02-26T14:03:57.481Zjob assigned to worker 01HQJVK5VTC5YR9TJA8F66KMA2 (queued for 2 m 20 s)
32024-02-26T14:03:57.604Zhttp://172.20.0.5:9969/ipxe/sock... ok
42024-02-26T14:03:57.652Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52024-02-26T14:03:58.753Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-22e5a97eec9bb5c6.gz
62024-02-26T14:04:00.922Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-22e5a97eec9bb5c6.gz
72024-02-26T14:04:00.922Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
82024-02-26T14:04:20.723Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... 94%Error: No response to keepalive - Terminating session
92024-02-26T14:04:27.195Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
102024-02-26T14:04:27.229Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-25e7d53a8e782c84.gz
112024-02-26T14:04:30.680ZError: No response de-activating SOL payload
122024-02-26T14:04:32.004Z[SOL Session operational. Use ~? for help]
132024-02-26T14:04:33.361Z ok
142024-02-26T14:04:33.392Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
152024-02-26T14:04:33.437Zhttp://172.20.0.5:9969/postboot/sock... ok
162024-02-26T14:04:33.493ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
172024-02-26T14:04:33.538Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
182024-02-26T14:04:33.577ZOxide Helios Version helios-2.0.22451 64-bit
192024-02-26T14:04:33.616ZConfiguring devices.
202024-02-26T14:04:33.649Z##################################################
212024-02-26T14:04:33.682Z#### ############# ##
222024-02-26T14:04:33.716Z### ### ############ ##
232024-02-26T14:04:33.749Z## ### # ## ### ## Oxide Computer Company ##
242024-02-26T14:04:33.785Z## ## ## ### # ### ##
252024-02-26T14:04:33.832Z## # ### #### #### This Station Under ##
262024-02-26T14:04:33.874Z### ### #### # ### Computer Control ##
272024-02-26T14:04:33.909Z#### #### ### ## ##
282024-02-26T14:04:33.940Z##################################################
292024-02-26T14:04:33.972ZHostname: sock
302024-02-26T14:04:34.011ZLoading smf(7) service descriptions:
312024-02-26T14:04:54.393Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-25e7d53a8e782c84.gz
322024-02-26T14:04:54.418Zdownloading input: /input/ci-tools/work/caboose-util.gz
332024-02-26T14:04:55.365Zdownloaded input: /input/ci-tools/work/caboose-util.gz
342024-02-26T14:04:55.365Zdownloading input: /input/ci-tools/work/tufaceous.gz
352024-02-26T14:04:59.264Zdownloaded input: /input/ci-tools/work/tufaceous.gz
362024-02-26T14:04:59.293Zdownloading input: /input/package/work/version.txt
372024-02-26T14:04:59.321Zdownloaded input: /input/package/work/version.txt
382024-02-26T14:04:59.321Zdownloading input: /input/package/work/package.tar.gz
392024-02-26T14:05:18.300Zdownloaded input: /input/package/work/package.tar.gz
402024-02-26T14:05:18.301Zdownloading input: /input/package/work/global-zone-packages.tar.gz
412024-02-26T14:05:52.136Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
422024-02-26T14:05:52.136Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
432024-02-26T14:06:02.937Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
442024-02-26T14:06:02.965Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
452024-02-26T14:06:38.329Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
462024-02-26T14:06:38.329Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
472024-02-26T14:07:22.234Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
482024-02-26T14:07:22.234Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
492024-02-26T14:07:36.767Zsock console login: Error: No response to keepalive - Terminating session
502024-02-26T14:07:46.794ZError: No response de-activating SOL payload
512024-02-26T14:07:46.894Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
522024-02-26T14:07:46.894Zdownloading input: /input/package/work/zones/crucible-pantry-zone.tar.gz
532024-02-26T14:07:48.034Z[SOL Session operational. Use ~? for help]
542024-02-26T14:07:52.702Zdownloaded input: /input/package/work/zones/crucible-pantry-zone.tar.gz
552024-02-26T14:07:52.702Zdownloading input: /input/package/work/zones/crucible-zone.tar.gz
562024-02-26T14:08:01.001Zdownloaded input: /input/package/work/zones/crucible-zone.tar.gz
572024-02-26T14:08:01.001Zdownloading input: /input/package/work/zones/external-dns.tar.gz
582024-02-26T14:08:07.627Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
592024-02-26T14:08:07.627Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
602024-02-26T14:08:12.304Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
612024-02-26T14:08:12.304Zdownloading input: /input/package/work/zones/ntp.tar.gz
622024-02-26T14:08:12.359Zdownloaded input: /input/package/work/zones/ntp.tar.gz
632024-02-26T14:08:12.384Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
642024-02-26T14:08:16.012Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
652024-02-26T14:08:16.012Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
662024-02-26T14:08:19.620Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
672024-02-26T14:08:19.647Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
682024-02-26T14:08:36.836Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
692024-02-26T14:08:36.836Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
702024-02-26T14:08:53.043Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
712024-02-26T14:08:53.044Zdownloading input: /input/package/work/zones/overlay.tar.gz
722024-02-26T14:08:53.115Zdownloaded input: /input/package/work/zones/overlay.tar.gz
732024-02-26T14:08:53.143Zdownloading input: /input/package/work/zones/oximeter.tar.gz
742024-02-26T14:09:05.341Zdownloaded input: /input/package/work/zones/oximeter.tar.gz
752024-02-26T14:09:05.366Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
762024-02-26T14:09:17.835Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
772024-02-26T14:09:17.835Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
782024-02-26T14:10:10.046Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
792024-02-26T14:10:10.046Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
802024-02-26T14:10:36.516ZError: No response to keepalive - Terminating session
812024-02-26T14:10:46.563ZError: No response de-activating SOL payload
822024-02-26T14:10:47.775Z[SOL Session operational. Use ~? for help]
832024-02-26T14:10:49.156Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
842024-02-26T14:10:49.157Zstarting task 0: "setup"
852024-02-26T14:10:49.185Z++ uname -s
862024-02-26T14:10:49.185Z+ kern=SunOS
872024-02-26T14:10:49.185Z+ case "$kern" in
882024-02-26T14:10:49.185Z+ groupadd -g 12345 build
892024-02-26T14:10:49.185Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
902024-02-26T14:10:51.171Z+ zfs create -o mountpoint=/work rpool/work
912024-02-26T14:10:51.217Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
922024-02-26T14:10:51.243Z+ home_fs=zfs
932024-02-26T14:10:51.243Z+ [[ zfs == autofs ]]
942024-02-26T14:10:51.243Z+ mkdir -p /home/build
952024-02-26T14:10:51.243Z+ chown build:build /home/build /work
962024-02-26T14:10:53.225Z+ chmod 0700 /home/build /work
972024-02-26T14:10:53.282Zprocess exited: duration 4076 ms, exit code 0
 
982024-02-26T14:10:53.340Zstarting task 1: "authentication"
992024-02-26T14:10:53.384Zprocess exited: duration 44 ms, exit code 0
 
1002024-02-26T14:10:53.449Zstarting task 2: "build"
1012024-02-26T14:10:53.485Z+ trap _exit_trap EXIT
1022024-02-26T14:10:53.485Z+ set +x
1032024-02-26T14:11:27.714Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
1042024-02-26T14:11:27.959Zsecpolicy_fs_linkdir: 0x55 = 0x55
1052024-02-26T14:11:27.959Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
1062024-02-26T14:11:27.959Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
1072024-02-26T14:11:27.959Zsecpolicy_fs_linkdir+3: 0 = 0xe5
1082024-02-26T14:11:27.984Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1092024-02-26T14:11:27.984Zsecpolicy_fs_linkdir+5: 0 = 0x1
1102024-02-26T14:11:27.984Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1112024-02-26T14:11:27.984Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1122024-02-26T14:11:27.984Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1132024-02-26T14:11:27.984Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1142024-02-26T14:11:27.984Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1152024-02-26T14:11:27.984Z+ [[ -d /opt/oxide ]]
1162024-02-26T14:11:27.984Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1172024-02-26T14:11:27.984Z+ pfexec mkdir /opt/oxide
1182024-02-26T14:11:27.984Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1192024-02-26T14:11:27.985Z+ [[ -d /opt/oxide-underneath ]]
1202024-02-26T14:11:27.985Z+ cd /opt/oxide-underneath
1212024-02-26T14:11:27.985Z+ pfexec tar ceEp@/f - .
1222024-02-26T14:11:27.985Z+ cd /opt/oxide
1232024-02-26T14:11:27.985Z+ pfexec tar xveEp@/f -
1242024-02-26T14:11:28.021Zx ., 0 bytes, 0 tape blocks
1252024-02-26T14:11:28.021Zx ./opte, 0 bytes, 0 tape blocks
1262024-02-26T14:11:28.021Zx ./opte/bin, 0 bytes, 0 tape blocks
1272024-02-26T14:11:28.021Zx ./opte/bin/opteadm, 51799224 bytes, 101171 tape blocks
1282024-02-26T14:11:28.046Z+ pfexec rm -rf /opt/oxide-underneath
1292024-02-26T14:11:28.046Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1302024-02-26T14:11:28.071Z+ pfexec mkdir /opt/oxide/work
1312024-02-26T14:11:28.071Z+ pfexec chown build:build /opt/oxide/work
1322024-02-26T14:11:28.071Z+ cd /opt/oxide/work
1332024-02-26T14:11:28.071Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1342024-02-26T14:11:29.780ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1352024-02-26T14:11:29.780Zx out/mg-ddm-gz.tar, 67661824 bytes, 132152 tape blocks
1362024-02-26T14:11:29.780Zx out/omicron-sled-agent.tar, 123044352 bytes, 240321 tape blocks
1372024-02-26T14:11:29.780Zx out/oxlog.tar, 8557568 bytes, 16714 tape blocks
1382024-02-26T14:11:29.780Zx out/target/test, 75 bytes, 1 tape blocks
1392024-02-26T14:11:29.780Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1402024-02-26T14:11:29.780Zx package-manifest.toml, 25053 bytes, 49 tape blocks
1412024-02-26T14:11:29.780Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1422024-02-26T14:11:29.780Zx target/release/omicron-package, 45894056 bytes, 89637 tape blocks
1432024-02-26T14:11:29.780Zx tools/create_virtual_hardware.sh, 3184 bytes, 7 tape blocks
1442024-02-26T14:11:29.780Zx tools/virtual_hardware.sh, 3416 bytes, 7 tape blocks
1452024-02-26T14:11:29.780Zx tools/scrimlet/create-softnpu-zone.sh, 627 bytes, 2 tape blocks
1462024-02-26T14:11:29.780Zx tools/scrimlet/destroy-softnpu-zone.sh, 203 bytes, 1 tape blocks
1472024-02-26T14:11:29.780Zx tools/scrimlet/softnpu-init.sh, 2157 bytes, 5 tape blocks
1482024-02-26T14:11:29.780Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1492024-02-26T14:11:29.780Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1502024-02-26T14:11:29.806Z
1512024-02-26T14:11:29.806Zreal 1.715796718
1522024-02-26T14:11:29.806Zuser 1.539074841
1532024-02-26T14:11:29.806Zsys 0.311674308
1542024-02-26T14:11:29.806Ztrap 0.000072970
1552024-02-26T14:11:29.806Ztflt 0.000000000
1562024-02-26T14:11:29.806Zdflt 0.000000000
1572024-02-26T14:11:29.806Zkflt 0.000000000
1582024-02-26T14:11:29.806Zlock 0.000000000
1592024-02-26T14:11:29.806Zslp 1.540519477
1602024-02-26T14:11:29.806Zlat 0.038357532
1612024-02-26T14:11:29.806Zstop 0.000121382
1622024-02-26T14:11:29.806Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry-zone.tar.gz /input/package/work/zones/crucible-zone.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1632024-02-26T14:11:30.993Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1642024-02-26T14:11:31.018Z+ mkdir tests
1652024-02-26T14:11:31.018Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1662024-02-26T14:11:31.019Z+ ptime -m gunzip
1672024-02-26T14:11:31.019Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-22e5a97eec9bb5c6
1682024-02-26T14:11:31.250Z
1692024-02-26T14:11:31.275Zreal 0.235828585
1702024-02-26T14:11:31.275Zuser 0.218492982
1712024-02-26T14:11:31.276Zsys 0.017076757
1722024-02-26T14:11:31.276Ztrap 0.000022260
1732024-02-26T14:11:31.276Ztflt 0.000000000
1742024-02-26T14:11:31.276Zdflt 0.000000000
1752024-02-26T14:11:31.276Zkflt 0.000000000
1762024-02-26T14:11:31.276Zlock 0.000000000
1772024-02-26T14:11:31.276Zslp 0.000020400
1782024-02-26T14:11:31.276Zlat 0.000021801
1792024-02-26T14:11:31.276Zstop 0.000126021
1802024-02-26T14:11:31.276Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-22e5a97eec9bb5c6
1812024-02-26T14:11:31.276Z+ chmod a+x tests/bootstrap-22e5a97eec9bb5c6
1822024-02-26T14:11:31.276Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1832024-02-26T14:11:31.276Z+ ptime -m gunzip
1842024-02-26T14:11:31.276Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1852024-02-26T14:11:33.778Z
1862024-02-26T14:11:33.804Zreal 2.517487855
1872024-02-26T14:11:33.804Zuser 2.356878451
1882024-02-26T14:11:33.804Zsys 0.160193417
1892024-02-26T14:11:33.804Ztrap 0.000177483
1902024-02-26T14:11:33.804Ztflt 0.000000000
1912024-02-26T14:11:33.804Zdflt 0.000000000
1922024-02-26T14:11:33.804Zkflt 0.000000000
1932024-02-26T14:11:33.804Zlock 0.000000000
1942024-02-26T14:11:33.804Zslp 0.000022650
1952024-02-26T14:11:33.804Zlat 0.000032290
1962024-02-26T14:11:33.804Zstop 0.000120422
1972024-02-26T14:11:33.804Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1982024-02-26T14:11:33.804Z+ chmod a+x tests/bootstrap
1992024-02-26T14:11:33.804Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
2002024-02-26T14:11:33.804Z+ ptime -m gunzip
2012024-02-26T14:11:33.804Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-25e7d53a8e782c84
2022024-02-26T14:11:36.374Z
2032024-02-26T14:11:36.399Zreal 2.583134031
2042024-02-26T14:11:36.399Zuser 2.418530913
2052024-02-26T14:11:36.399Zsys 0.164151881
2062024-02-26T14:11:36.399Ztrap 0.000196691
2072024-02-26T14:11:36.399Ztflt 0.000000000
2082024-02-26T14:11:36.399Zdflt 0.000000000
2092024-02-26T14:11:36.399Zkflt 0.000000000
2102024-02-26T14:11:36.399Zlock 0.000000000
2112024-02-26T14:11:36.400Zslp 0.000024120
2122024-02-26T14:11:36.400Zlat 0.000043130
2132024-02-26T14:11:36.400Zstop 0.000124032
2142024-02-26T14:11:36.400Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-25e7d53a8e782c84
2152024-02-26T14:11:36.400Z+ chmod a+x tests/end_to_end_tests-25e7d53a8e782c84
2162024-02-26T14:11:36.400Z+ bmat address ls
2172024-02-26T14:11:36.400ZNAME CIDR FIRST LAST COUNT
2182024-02-26T14:11:36.400Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2192024-02-26T14:11:36.400Z++ bmat address ls -f extra -Ho first,last,count
2202024-02-26T14:11:36.400Z+ set -- 10.151.2.164 10.151.2.227 64
2212024-02-26T14:11:36.400Z+ EXTRA_IP_START=10.151.2.164
2222024-02-26T14:11:36.400Z+ EXTRA_IP_END=10.151.2.227
2232024-02-26T14:11:36.400Z+ EXTRA_IP_COUNT=64
2242024-02-26T14:11:36.400Z+ (( EXTRA_IP_COUNT >= 32 ))
2252024-02-26T14:11:36.400Z+ EXTRA_IP_BASE=10.151.2
2262024-02-26T14:11:36.400Z+ EXTRA_IP_FOCTET=164
2272024-02-26T14:11:36.400Z+ EXTRA_IP_LOCTET=227
2282024-02-26T14:11:36.400Z+ SERVICE_IP_POOL_START=10.151.2.164
2292024-02-26T14:11:36.400Z+ SERVICE_IP_POOL_END=10.151.2.173
2302024-02-26T14:11:36.400Z+ DNS_IP1=10.151.2.164
2312024-02-26T14:11:36.400Z+ DNS_IP2=10.151.2.165
2322024-02-26T14:11:36.400Z+ UPLINK_IP=10.151.2.174
2332024-02-26T14:11:36.400Z+ IPPOOL_START=10.151.2.175
2342024-02-26T14:11:36.400Z+ IPPOOL_END=10.151.2.227
2352024-02-26T14:11:36.426Z++ ipadm show-addr -po type,addr
2362024-02-26T14:11:36.426Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2372024-02-26T14:11:36.426Z+ GATEWAY_IP=10.151.2.100
2382024-02-26T14:11:36.426Z+ [[ -n 10.151.2.100 ]]
2392024-02-26T14:11:36.426Z+ ping -s 10.151.2.100 56 1
2402024-02-26T14:11:36.426ZPING 10.151.2.100: 56 data bytes
2412024-02-26T14:11:36.426Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.010 ms
2422024-02-26T14:11:36.426Z
2432024-02-26T14:11:36.426Z----10.151.2.100 PING Statistics----
2442024-02-26T14:11:36.426Z1 packets transmitted, 1 packets received, 0% packet loss
2452024-02-26T14:11:36.426Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2462024-02-26T14:11:36.426Z++ arp -an
2472024-02-26T14:11:36.426Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2482024-02-26T14:11:36.426Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2492024-02-26T14:11:36.426Z+ routeadm -e ipv4-forwarding -u
2502024-02-26T14:11:36.533Z+ PXA_START=10.151.2.164
2512024-02-26T14:11:36.533Z+ PXA_END=10.151.2.227
2522024-02-26T14:11:36.533Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2532024-02-26T14:11:36.558Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2542024-02-26T14:11:36.732Z+ ZPOOL_VDEV_DIR=/scratch
2552024-02-26T14:11:36.732Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2562024-02-26T14:11:36.757Z+++ dirname ./tools/create_virtual_hardware.sh
2572024-02-26T14:11:36.757Z++ cd ./tools
2582024-02-26T14:11:36.757Z++ pwd
2592024-02-26T14:11:36.757Z+ SOURCE_DIR=/opt/oxide/work/tools
2602024-02-26T14:11:36.757Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2612024-02-26T14:11:36.757Z+ SOFTNPU_MODE=zone
2622024-02-26T14:11:36.757Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2632024-02-26T14:11:36.757Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2642024-02-26T14:11:36.757Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2652024-02-26T14:11:36.757Z++ dladm show-phys -p -o LINK
2662024-02-26T14:11:36.757Z++ head -1
2672024-02-26T14:11:36.757Z+ PHYSICAL_LINK=igb0
2682024-02-26T14:11:36.757Z+ echo 'Using igb0 as physical link'
2692024-02-26T14:11:36.757ZUsing igb0 as physical link
2702024-02-26T14:11:36.757Z+ ensure_run_as_root
2712024-02-26T14:11:36.783Z++ id -u
2722024-02-26T14:11:36.783Z+ [[ 0 -ne 0 ]]
2732024-02-26T14:11:36.783Z+ ensure_zpools
2742024-02-26T14:11:36.783Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2752024-02-26T14:11:36.783Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2762024-02-26T14:11:36.784Z+ readarray -t ZPOOLS
2772024-02-26T14:11:36.784Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2782024-02-26T14:11:36.784Z++ sed 's/[ ",]//g'
2792024-02-26T14:11:36.784Z+ for ZPOOL in "${ZPOOLS[@]}"
2802024-02-26T14:11:36.784Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2812024-02-26T14:11:36.784ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2822024-02-26T14:11:36.784Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2832024-02-26T14:11:36.784Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2842024-02-26T14:11:36.784Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2852024-02-26T14:11:36.784Z0+0 records in
2862024-02-26T14:11:36.784Z0+0 records out
2872024-02-26T14:11:36.784Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2882024-02-26T14:11:36.784Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2892024-02-26T14:11:36.784Z+ set +x
2902024-02-26T14:11:36.784ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2912024-02-26T14:11:36.784Z++ zpool list -o name
2922024-02-26T14:11:36.784Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2932024-02-26T14:11:36.784Z+ [[ -z '' ]]
2942024-02-26T14:11:36.784Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2952024-02-26T14:11:36.810Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2962024-02-26T14:11:36.835Z+ set +x
2972024-02-26T14:11:36.835ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2982024-02-26T14:11:36.835Z+ for ZPOOL in "${ZPOOLS[@]}"
2992024-02-26T14:11:36.835Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3002024-02-26T14:11:36.835ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3012024-02-26T14:11:36.835Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3022024-02-26T14:11:36.835Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3032024-02-26T14:11:36.835Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3042024-02-26T14:11:36.835Z0+0 records in
3052024-02-26T14:11:36.835Z0+0 records out
3062024-02-26T14:11:36.835Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3072024-02-26T14:11:36.835Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3082024-02-26T14:11:36.835Z+ set +x
3092024-02-26T14:11:36.835ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3102024-02-26T14:11:36.835Z++ zpool list -o name
3112024-02-26T14:11:36.835Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3122024-02-26T14:11:36.836Z+ [[ -z '' ]]
3132024-02-26T14:11:36.836Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3142024-02-26T14:11:36.872Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3152024-02-26T14:11:36.872Z+ set +x
3162024-02-26T14:11:36.872ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3172024-02-26T14:11:36.873Z+ for ZPOOL in "${ZPOOLS[@]}"
3182024-02-26T14:11:36.873Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3192024-02-26T14:11:36.873Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3202024-02-26T14:11:36.873ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3212024-02-26T14:11:36.873Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3222024-02-26T14:11:36.873Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3232024-02-26T14:11:36.873Z0+0 records in
3242024-02-26T14:11:36.873Z0+0 records out
3252024-02-26T14:11:36.873Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3262024-02-26T14:11:36.873Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3272024-02-26T14:11:36.873Z+ set +x
3282024-02-26T14:11:36.873ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3292024-02-26T14:11:36.873Z++ zpool list -o name
3302024-02-26T14:11:36.874Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3312024-02-26T14:11:36.900Z+ [[ -z '' ]]
3322024-02-26T14:11:36.900Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3332024-02-26T14:11:36.900Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3342024-02-26T14:11:36.900Z+ set +x
3352024-02-26T14:11:36.901ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3362024-02-26T14:11:36.901Z+ for ZPOOL in "${ZPOOLS[@]}"
3372024-02-26T14:11:36.901Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3382024-02-26T14:11:36.901ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3392024-02-26T14:11:36.901Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3402024-02-26T14:11:36.901Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3412024-02-26T14:11:36.901Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3422024-02-26T14:11:36.901Z0+0 records in
3432024-02-26T14:11:36.901Z0+0 records out
3442024-02-26T14:11:36.901Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3452024-02-26T14:11:36.901Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3462024-02-26T14:11:36.901Z+ set +x
3472024-02-26T14:11:36.901ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3482024-02-26T14:11:36.901Z++ zpool list -o name
3492024-02-26T14:11:36.926Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3502024-02-26T14:11:36.926Z+ [[ -z '' ]]
3512024-02-26T14:11:36.926Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3522024-02-26T14:11:36.952Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3532024-02-26T14:11:36.952Z+ set +x
3542024-02-26T14:11:36.952ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3552024-02-26T14:11:36.952ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3562024-02-26T14:11:36.952Z+ for ZPOOL in "${ZPOOLS[@]}"
3572024-02-26T14:11:36.952Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3582024-02-26T14:11:36.952Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3592024-02-26T14:11:36.952Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3602024-02-26T14:11:36.952Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3612024-02-26T14:11:36.952Z0+0 records in
3622024-02-26T14:11:36.952Z0+0 records out
3632024-02-26T14:11:36.952Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3642024-02-26T14:11:36.952Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3652024-02-26T14:11:36.952Z+ set +x
3662024-02-26T14:11:36.952ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3672024-02-26T14:11:36.952Z++ zpool list -o name
3682024-02-26T14:11:36.952Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3692024-02-26T14:11:36.952Z+ [[ -z '' ]]
3702024-02-26T14:11:36.952Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712024-02-26T14:11:36.980Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3722024-02-26T14:11:36.980Z+ set +x
3732024-02-26T14:11:37.005ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3742024-02-26T14:11:37.005Z+ for ZPOOL in "${ZPOOLS[@]}"
3752024-02-26T14:11:37.005Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3762024-02-26T14:11:37.005ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3772024-02-26T14:11:37.006Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3782024-02-26T14:11:37.006Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3792024-02-26T14:11:37.006Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3802024-02-26T14:11:37.006Z0+0 records in
3812024-02-26T14:11:37.006Z0+0 records out
3822024-02-26T14:11:37.006Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3832024-02-26T14:11:37.006Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3842024-02-26T14:11:37.006Z+ set +x
3852024-02-26T14:11:37.006ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3862024-02-26T14:11:37.006Z++ zpool list -o name
3872024-02-26T14:11:37.006Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3882024-02-26T14:11:37.006Z+ [[ -z '' ]]
3892024-02-26T14:11:37.006Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3902024-02-26T14:11:37.031Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3912024-02-26T14:11:37.031Z+ set +x
3922024-02-26T14:11:37.031ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3932024-02-26T14:11:37.031Z+ for ZPOOL in "${ZPOOLS[@]}"
3942024-02-26T14:11:37.031Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3952024-02-26T14:11:37.031ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3962024-02-26T14:11:37.031Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3972024-02-26T14:11:37.031Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3982024-02-26T14:11:37.031Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3992024-02-26T14:11:37.031Z0+0 records in
4002024-02-26T14:11:37.031Z0+0 records out
4012024-02-26T14:11:37.032Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4022024-02-26T14:11:37.032Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4032024-02-26T14:11:37.032Z+ set +x
4042024-02-26T14:11:37.032ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
4052024-02-26T14:11:37.057Z++ zpool list -o name
4062024-02-26T14:11:37.057Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4072024-02-26T14:11:37.057Z+ [[ -z '' ]]
4082024-02-26T14:11:37.057Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4092024-02-26T14:11:37.084Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4102024-02-26T14:11:37.084Z+ set +x
4112024-02-26T14:11:37.084ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4122024-02-26T14:11:37.084Z+ for ZPOOL in "${ZPOOLS[@]}"
4132024-02-26T14:11:37.084Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4142024-02-26T14:11:37.084ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4152024-02-26T14:11:37.084Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4162024-02-26T14:11:37.084Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4172024-02-26T14:11:37.084Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4182024-02-26T14:11:37.084Z0+0 records in
4192024-02-26T14:11:37.084Z0+0 records out
4202024-02-26T14:11:37.084Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4212024-02-26T14:11:37.084Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4222024-02-26T14:11:37.084Z+ set +x
4232024-02-26T14:11:37.084ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4242024-02-26T14:11:37.084Z++ zpool list -o name
4252024-02-26T14:11:37.085Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4262024-02-26T14:11:37.110Z+ [[ -z '' ]]
4272024-02-26T14:11:37.110Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4282024-02-26T14:11:37.135Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4292024-02-26T14:11:37.135Z+ set +x
4302024-02-26T14:11:37.135ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4312024-02-26T14:11:37.135Z+ for ZPOOL in "${ZPOOLS[@]}"
4322024-02-26T14:11:37.135Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4332024-02-26T14:11:37.135ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4342024-02-26T14:11:37.135Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4352024-02-26T14:11:37.135Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4362024-02-26T14:11:37.135Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4372024-02-26T14:11:37.135Z0+0 records in
4382024-02-26T14:11:37.135Z0+0 records out
4392024-02-26T14:11:37.135Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4402024-02-26T14:11:37.136Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4412024-02-26T14:11:37.136Z+ set +x
4422024-02-26T14:11:37.136ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4432024-02-26T14:11:37.136Z++ zpool list -o name
4442024-02-26T14:11:37.136Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4452024-02-26T14:11:37.136Z+ [[ -z '' ]]
4462024-02-26T14:11:37.136Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4472024-02-26T14:11:37.161Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4482024-02-26T14:11:37.161Z+ set +x
4492024-02-26T14:11:37.161ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4502024-02-26T14:11:37.161Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4512024-02-26T14:11:37.161Z+ readarray -t ZPOOLS
4522024-02-26T14:11:37.161Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4532024-02-26T14:11:37.161Z++ sed 's/[ ",]//g'
4542024-02-26T14:11:37.190Z+ for ZPOOL in "${ZPOOLS[@]}"
4552024-02-26T14:11:37.190Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4562024-02-26T14:11:37.190ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4572024-02-26T14:11:37.190Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4582024-02-26T14:11:37.190Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4592024-02-26T14:11:37.190Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4602024-02-26T14:11:37.190Z0+0 records in
4612024-02-26T14:11:37.190Z0+0 records out
4622024-02-26T14:11:37.190Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4632024-02-26T14:11:37.190Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4642024-02-26T14:11:37.190Z+ set +x
4652024-02-26T14:11:37.190ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4662024-02-26T14:11:37.190Z++ zpool list -o name
4672024-02-26T14:11:37.190Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4682024-02-26T14:11:37.191Z+ [[ -z '' ]]
4692024-02-26T14:11:37.191Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4702024-02-26T14:11:37.218Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4712024-02-26T14:11:37.218Z+ set +x
4722024-02-26T14:11:37.218ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4732024-02-26T14:11:37.218Z+ for ZPOOL in "${ZPOOLS[@]}"
4742024-02-26T14:11:37.218Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4752024-02-26T14:11:37.218ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4762024-02-26T14:11:37.219Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4772024-02-26T14:11:37.219Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4782024-02-26T14:11:37.219Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4792024-02-26T14:11:37.219Z0+0 records in
4802024-02-26T14:11:37.219Z0+0 records out
4812024-02-26T14:11:37.219Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4822024-02-26T14:11:37.219Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4832024-02-26T14:11:37.219Z+ set +x
4842024-02-26T14:11:37.219ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4852024-02-26T14:11:37.219Z++ zpool list -o name
4862024-02-26T14:11:37.219Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4872024-02-26T14:11:37.244Z+ [[ -z '' ]]
4882024-02-26T14:11:37.244Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4892024-02-26T14:11:37.270Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4902024-02-26T14:11:37.270Z+ set +x
4912024-02-26T14:11:37.270ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4922024-02-26T14:11:37.270Z+ [[ zone == \z\o\n\e ]]
4932024-02-26T14:11:37.270Z+ ensure_simulated_links igb0
4942024-02-26T14:11:37.270Z+ local PHYSICAL_LINK=igb0
4952024-02-26T14:11:37.270Z+ INDICES=("0" "1")
4962024-02-26T14:11:37.270Z+ for I in "${INDICES[@]}"
4972024-02-26T14:11:37.270Z++ get_simnet_name_if_exists net0
4982024-02-26T14:11:37.271Z++ dladm show-simnet -p -o LINK net0
4992024-02-26T14:11:37.271Z++ echo ''
5002024-02-26T14:11:37.271Z+ [[ -z '' ]]
5012024-02-26T14:11:37.271Z+ dladm create-simnet -t net0
5022024-02-26T14:11:37.271Z+ dladm create-simnet -t sc0_0
5032024-02-26T14:11:37.271Z+ dladm modify-simnet -t -p net0 sc0_0
5042024-02-26T14:11:37.271Z+ dladm set-linkprop -p mtu=9000 sc0_0
5052024-02-26T14:11:37.298Z+ success 'Simnet net0/sc0_0 exists'
5062024-02-26T14:11:37.298Z+ set +x
5072024-02-26T14:11:37.298ZSimnet net0/sc0_0 exists
5082024-02-26T14:11:37.298Z+ for I in "${INDICES[@]}"
5092024-02-26T14:11:37.298Z++ get_simnet_name_if_exists net1
5102024-02-26T14:11:37.298Z++ dladm show-simnet -p -o LINK net1
5112024-02-26T14:11:37.298Z++ echo ''
5122024-02-26T14:11:37.298Z+ [[ -z '' ]]
5132024-02-26T14:11:37.298Z+ dladm create-simnet -t net1
5142024-02-26T14:11:37.298Z+ dladm create-simnet -t sc1_0
5152024-02-26T14:11:37.298Z+ dladm modify-simnet -t -p net1 sc1_0
5162024-02-26T14:11:37.298Z+ dladm set-linkprop -p mtu=9000 sc1_0
5172024-02-26T14:11:37.298Z+ success 'Simnet net1/sc1_0 exists'
5182024-02-26T14:11:37.298Z+ set +x
5192024-02-26T14:11:37.298ZSimnet net1/sc1_0 exists
5202024-02-26T14:11:37.298Z++ get_vnic_name_if_exists sc0_1
5212024-02-26T14:11:37.298Z++ dladm show-vnic -p -o LINK sc0_1
5222024-02-26T14:11:37.298Z++ echo ''
5232024-02-26T14:11:37.298Z+ [[ -z '' ]]
5242024-02-26T14:11:37.298Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5252024-02-26T14:11:37.324Z+ [[ -v PROMISC_FILT_OFF ]]
5262024-02-26T14:11:37.324Z+ success 'Vnic sc0_1 exists'
5272024-02-26T14:11:37.324Z+ set +x
5282024-02-26T14:11:37.324ZVnic sc0_1 exists
5292024-02-26T14:11:37.324Z+ warn_if_no_proxy_arp
5302024-02-26T14:11:37.324Z+ [[ -v PXA_START ]]
5312024-02-26T14:11:37.324Z+ [[ -v PXA_END ]]
5322024-02-26T14:11:37.324Z+ ensure_softnpu_zone
5332024-02-26T14:11:37.324Z+ zoneadm list
5342024-02-26T14:11:37.324Z+ grep -q sidecar_softnpu
5352024-02-26T14:11:37.324Z+ [[ -f out/npuzone/npuzone ]]
5362024-02-26T14:11:37.324Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit e3ea4b495ba0a71801ded0776ae4bbd31df57e26 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5372024-02-26T14:11:37.351Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5382024-02-26T14:11:41.976Zdownloading softnpu ...done
5392024-02-26T14:11:42.004Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/libsidecar_lite.so.sha256.txt
5402024-02-26T14:11:46.735Zdownloading asic_program.so ...done
5412024-02-26T14:11:46.762Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/scadm.sha256.txt
5422024-02-26T14:11:53.143Zdownloading scadm ...done
5432024-02-26T14:11:53.168Zcreating port tfportrear0_0 for sc0_0
5442024-02-26T14:11:53.168Zcreating port tfportqsfp0_0 for sc0_1
5452024-02-26T14:11:53.193Zsoftnpu zone setup
5462024-02-26T14:11:53.193Zconfigure zone
5472024-02-26T14:11:53.193Zinstall zone
5482024-02-26T14:11:54.269Zboot zone
5492024-02-26T14:11:57.368Z[sidecar_softnpu] svcs milestone/network
5502024-02-26T14:11:57.394ZSTATE STIME FMRI
5512024-02-26T14:11:57.394Zonline 14:11:55 svc:/milestone/network:default
5522024-02-26T14:11:57.394Z[sidecar_softnpu] /softnpu/npu start
5532024-02-26T14:11:57.419Z
5542024-02-26T14:11:57.445Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5552024-02-26T14:11:57.445Z+ GATEWAY_IP=10.151.2.100
5562024-02-26T14:11:57.445Z+ echo 'Using 10.151.2.100 as gateway ip'
5572024-02-26T14:11:57.445ZUsing 10.151.2.100 as gateway ip
5582024-02-26T14:11:57.445Z+ [[ ! -v GATEWAY_MAC ]]
5592024-02-26T14:11:57.445Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5602024-02-26T14:11:57.445Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5612024-02-26T14:11:57.445Z+++ echo 18:c0:4d:0d:a0:2a
5622024-02-26T14:11:57.445Z+ COUNT=1
5632024-02-26T14:11:57.445Z+ [[ 1 -ne 1 ]]
5642024-02-26T14:11:57.446Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5652024-02-26T14:11:57.446ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5662024-02-26T14:11:57.446Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5672024-02-26T14:11:57.446Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5682024-02-26T14:11:57.475Z+ PXA_MAC=a8:e1:de:01:70:1d
5692024-02-26T14:11:57.475Z+ [[ -v PXA_START ]]
5702024-02-26T14:11:57.475Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5712024-02-26T14:11:57.475Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5722024-02-26T14:11:57.501Z+ z_scadm dump-state
5732024-02-26T14:11:57.501Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5742024-02-26T14:11:57.527Zlocal v6:
5752024-02-26T14:11:57.527Zlocal v4:
5762024-02-26T14:11:57.528Zrouter v6:
5772024-02-26T14:11:57.528Zrouter v4_idx:
5782024-02-26T14:11:57.528Zrouter v4_routes:
5792024-02-26T14:11:57.528Zresolver v4:
5802024-02-26T14:11:57.528Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5812024-02-26T14:11:57.528Zresolver v6:
5822024-02-26T14:11:57.528Znat v4:
5832024-02-26T14:11:57.528Znat_v6:
5842024-02-26T14:11:57.528Zport_mac:
5852024-02-26T14:11:57.528Zproxy arp:
5862024-02-26T14:11:57.528Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5872024-02-26T14:11:57.528Z+ success 'softnpu zone exists'
5882024-02-26T14:11:57.528Z+ set +x
5892024-02-26T14:11:57.529Zsoftnpu zone exists
5902024-02-26T14:11:57.529Z
5912024-02-26T14:11:57.529Zreal 20.785549786
5922024-02-26T14:11:57.529Zuser 0.708612987
5932024-02-26T14:11:57.529Zsys 1.582080770
5942024-02-26T14:11:57.529Ztrap 0.000090398
5952024-02-26T14:11:57.529Ztflt 0.000125444
5962024-02-26T14:11:57.529Zdflt 0.001330298
5972024-02-26T14:11:57.529Zkflt 0.000000000
5982024-02-26T14:11:57.529Zlock 16:03.624819545
5992024-02-26T14:11:57.529Zslp 1:02.082365494
6002024-02-26T14:11:57.529Zlat 0.080741231
6012024-02-26T14:11:57.529Zstop 0.361481106
6022024-02-26T14:11:57.530Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
6032024-02-26T14:11:57.562Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
6042024-02-26T14:11:57.587Z+ SILO_NAME=recovery
6052024-02-26T14:11:57.587Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
6062024-02-26T14:11:57.587Z+ EXTERNAL_DNS_DOMAIN=oxide.test
6072024-02-26T14:11:57.587Z+ sed -i~ '
6082024-02-26T14:11:57.587Z /^external_dns_ips/c\
6092024-02-26T14:11:57.587Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6102024-02-26T14:11:57.588Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6112024-02-26T14:11:57.588Z /^first/c\
6122024-02-26T14:11:57.588Zfirst = "10.151.2.164"
6132024-02-26T14:11:57.588Z /^last/c\
6142024-02-26T14:11:57.588Zlast = "10.151.2.173"
6152024-02-26T14:11:57.588Z }
6162024-02-26T14:11:57.588Z /^\[rack_network_config/,/^$/ {
6172024-02-26T14:11:57.588Z /^infra_ip_first/c\
6182024-02-26T14:11:57.588Zinfra_ip_first = "10.151.2.174"
6192024-02-26T14:11:57.588Z /^infra_ip_last/c\
6202024-02-26T14:11:57.588Zinfra_ip_last = "10.151.2.174"
6212024-02-26T14:11:57.588Z }
6222024-02-26T14:11:57.588Z /^\[\[rack_network_config.ports/,/^$/ {
6232024-02-26T14:11:57.588Z /^routes/c\
6242024-02-26T14:11:57.588Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6252024-02-26T14:11:57.588Z /^addresses/c\
6262024-02-26T14:11:57.588Zaddresses = \["10.151.2.174/24"\]
6272024-02-26T14:11:57.588Z }
6282024-02-26T14:11:57.588Z' pkg/config-rss.toml
6292024-02-26T14:11:57.588Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6302024-02-26T14:11:57.588Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6312024-02-26T14:11:57.588Z+++ pkg/config-rss.toml Mon Feb 26 14:11:57 2024
6322024-02-26T14:11:57.588Z@@ -33,7 +33,7 @@
6332024-02-26T14:11:57.588Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6342024-02-26T14:11:57.589Z # must be contained in one of the "internal services" IP Pool ranges listed
6352024-02-26T14:11:57.589Z # below.
6362024-02-26T14:11:57.589Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6372024-02-26T14:11:57.589Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6382024-02-26T14:11:57.589Z
6392024-02-26T14:11:57.589Z # Initial TLS certificates for the external API
6402024-02-26T14:11:57.589Z #
6412024-02-26T14:11:57.589Z@@ -69,8 +69,8 @@
6422024-02-26T14:11:57.589Z #
6432024-02-26T14:11:57.589Z # For more on this and what to put here, see docs/how-to-run.adoc.
6442024-02-26T14:11:57.589Z [[internal_services_ip_pool_ranges]]
6452024-02-26T14:11:57.589Z-first = "192.168.1.20"
6462024-02-26T14:11:57.589Z-last = "192.168.1.29"
6472024-02-26T14:11:57.589Z+first = "10.151.2.164"
6482024-02-26T14:11:57.589Z+last = "10.151.2.173"
6492024-02-26T14:11:57.589Z
6502024-02-26T14:11:57.589Z # TODO - this configuration is subject to change going forward. Ultimately these
6512024-02-26T14:11:57.589Z # parameters should be provided to the control plane via wicket, but we need to
6522024-02-26T14:11:57.589Z@@ -100,9 +100,9 @@
6532024-02-26T14:11:57.589Z # You can configure multiple uplinks by repeating the following stanza
6542024-02-26T14:11:57.589Z [[rack_network_config.ports]]
6552024-02-26T14:11:57.589Z # Routes associated with this port.
6562024-02-26T14:11:57.589Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572024-02-26T14:11:57.589Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6582024-02-26T14:11:57.589Z # Addresses associated with this port.
6592024-02-26T14:11:57.590Z-addresses = ["192.168.1.30/24"]
6602024-02-26T14:11:57.590Z+addresses = ["10.151.2.174/24"]
6612024-02-26T14:11:57.590Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622024-02-26T14:11:57.590Z port = "qsfp0"
6632024-02-26T14:11:57.590Z # The speed of this port.
6642024-02-26T14:11:57.590Z+ true
6652024-02-26T14:11:57.590Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662024-02-26T14:11:57.590Za pkg/config-rss.toml 7K
6672024-02-26T14:11:57.590Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682024-02-26T14:11:57.590Z+ TLS_NAME=recovery.sys.oxide.test
6692024-02-26T14:11:57.590Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702024-02-26T14:11:58.023Z..........+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.........+......+...+..........+..+.+.....+..........+............+......+..+............+.+..+....+........+.......+...+..+.........+.........+.+..+...+....+..+......+.......+..+....+.........+..+.+..............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+..+..........+....................+....+...........+....+.....+...+.......+.........+.........+........+......................+............+...+...............+...........+....+...........+.......+..+.+.....+.+......+..+...+........................+.+...+..............................+.....+.......+........+.+......+.......................+......+....+...........+.+..+...+.........+...+......+.........+...+...+.......+..+.+......+......+...+...........+......+...+....+..+....+.................+............+......+....+.....+...+....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712024-02-26T14:11:58.553Z..+......+...+......+..+....+...+........+...+......+...+.+.....+......+......+...+....+...+............+........+.......+...............+.....+...+..........+..+.......+...+......+...........+.+..+.......+.....+.+..+...+.............+..+...+..........+..+.......+..+...+..........+...+............+...+........+.......+...+.....+.+...+...+..................+.....+.......+..+...+...................+........+.+..+....+.........+......+.....+.........+.+...+..+..........+...+...............+...+...+...........+....+.................+...+....+...+..+.+......+.....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.....+...+..........+...+..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+...+.+...+........+.......+........+............+......+....+.....................+......+........+...+..........+.........+......+........+.......+..+.+......+........+.+..+.............+.....+.......+........+.......+........+....+.....+................+..............................+.........+...+..+.............+..............+.+..+.+..............+........................................+.....+...+....+........+...+...+..........+.....+......+......+.......+..+...+............+.+........+.......+...+..+....+.....+....+......+...+..........................+.+........+...+...+.......+............+.................+.+.........+...+..+.........+.+..................+......+......+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722024-02-26T14:11:58.579Z-----
6732024-02-26T14:11:58.579Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742024-02-26T14:11:58.579Za pkg/initial-tls-cert.pem 2K
6752024-02-26T14:11:58.579Za pkg/initial-tls-key.pem 4K
6762024-02-26T14:11:58.579Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772024-02-26T14:11:58.579Z+ rmdir pkg
6782024-02-26T14:11:58.579Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792024-02-26T14:11:58.579Z+ pfexec mkdir -p /usr/oxide
6802024-02-26T14:11:58.604Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6812024-02-26T14:11:59.291Z+ pfexec chmod +x /usr/oxide/oxide
6822024-02-26T14:11:59.317Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6832024-02-26T14:12:17.587Z+ OMICRON_NO_UNINSTALL=1
6842024-02-26T14:12:17.587Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6852024-02-26T14:12:17.613ZLogging to: /opt/oxide/work/out/LOG
6862024-02-26T14:12:18.135Z
6872024-02-26T14:12:18.160Zreal 0.542108089
6882024-02-26T14:12:18.160Zuser 0.110766942
6892024-02-26T14:12:18.160Zsys 2.450167863
6902024-02-26T14:12:18.160Ztrap 0.000006650
6912024-02-26T14:12:18.160Ztflt 0.000156382
6922024-02-26T14:12:18.160Zdflt 0.000315580
6932024-02-26T14:12:18.160Zkflt 0.000008070
6942024-02-26T14:12:18.160Zlock 49.013348090
6952024-02-26T14:12:18.160Zslp 0.745773659
6962024-02-26T14:12:18.160Zlat 0.003549866
6972024-02-26T14:12:18.160Zstop 0.206514266
6982024-02-26T14:12:18.161Z+ retry=0
6992024-02-26T14:12:18.161Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7002024-02-26T14:12:18.161Z+ [[ 0 -gt 30 ]]
7012024-02-26T14:12:18.161Z+ sleep 1
7022024-02-26T14:12:19.149Z+ retry=1
7032024-02-26T14:12:19.149Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7042024-02-26T14:12:19.176Z+ [[ 1 -gt 30 ]]
7052024-02-26T14:12:19.176Z+ sleep 1
7062024-02-26T14:12:20.164Z+ retry=2
7072024-02-26T14:12:20.164Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7082024-02-26T14:12:20.189Z+ [[ 2 -gt 30 ]]
7092024-02-26T14:12:20.189Z+ sleep 1
7102024-02-26T14:12:21.179Z+ retry=3
7112024-02-26T14:12:21.179Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7122024-02-26T14:12:21.204Z+ [[ 3 -gt 30 ]]
7132024-02-26T14:12:21.204Z+ sleep 1
7142024-02-26T14:12:22.192Z+ retry=4
7152024-02-26T14:12:22.192Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7162024-02-26T14:12:47.364Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7172024-02-26T14:12:47.391Zlocal v6:
7182024-02-26T14:12:47.391Zfdb0:8dce:60a:dc2d::1
7192024-02-26T14:12:47.391Zfe80::886f:ebff:fe81:660
7202024-02-26T14:12:47.391Zlocal v4:
7212024-02-26T14:12:47.391Z10.151.2.174
7222024-02-26T14:12:47.391Zrouter v6:
7232024-02-26T14:12:47.391Zfdb0:18c0:4d0d:a02a::/64 -> fe80::101c:fff:fe3e:20ba (1)
7242024-02-26T14:12:47.391Zfd00:1122:3344:3::/64 -> fe80::101c:fff:fe3e:20ba (1)
7252024-02-26T14:12:47.391Zfd00:1122:3344:101::/64 -> fe80::101c:fff:fe3e:20ba (1)
7262024-02-26T14:12:47.391Zfd00:1122:3344:1::/64 -> fe80::101c:fff:fe3e:20ba (1)
7272024-02-26T14:12:47.391Zfd00:1122:3344:2::/64 -> fe80::101c:fff:fe3e:20ba (1)
7282024-02-26T14:12:47.391Zrouter v4_idx:
7292024-02-26T14:12:47.391Z0.0.0.0/0 -> 0
7302024-02-26T14:12:47.391Zrouter v4_routes:
7312024-02-26T14:12:47.391Z0 -> 10.151.2.100 (2)
7322024-02-26T14:12:47.417Zresolver v4:
7332024-02-26T14:12:47.417Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7342024-02-26T14:12:47.417Z10.151.2.174 -> 4e:bc:64:a3:5c:85
7352024-02-26T14:12:47.417Zresolver v6:
7362024-02-26T14:12:47.417Zfe80::886f:ebff:fe81:660 -> 8a:6f:eb:81:06:60
7372024-02-26T14:12:47.417Zfe80::101c:fff:fe3e:20ba -> 12:1c:0f:3e:20:ba
7382024-02-26T14:12:47.417Znat v4:
7392024-02-26T14:12:47.417Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:fa:ea
7402024-02-26T14:12:47.417Znat_v6:
7412024-02-26T14:12:47.417Zport_mac:
7422024-02-26T14:12:47.417Z2: a8:40:25:38:07:b4
7432024-02-26T14:12:47.417Z1000: a8:40:25:38:07:93
7442024-02-26T14:12:47.417Z1: a8:40:25:38:07:94
7452024-02-26T14:12:47.417Zproxy arp:
7462024-02-26T14:12:47.417Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7472024-02-26T14:12:47.417Z+ retry=0
7482024-02-26T14:12:47.417Z+++ zoneadm list -n
7492024-02-26T14:12:47.417Z+++ grep oxz_ntp
7502024-02-26T14:12:47.417Z++ pfexec svcs -z oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7 -Hostate oxide/ntp
7512024-02-26T14:12:47.445Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7522024-02-26T14:12:47.445Z++ true
7532024-02-26T14:12:47.446Z+ [[ '' != online ]]
7542024-02-26T14:12:47.446Z+ [[ 0 -gt 60 ]]
7552024-02-26T14:12:47.446Z+ sleep 1
7562024-02-26T14:12:48.424Z+ retry=1
7572024-02-26T14:12:48.449Z+++ zoneadm list -n
7582024-02-26T14:12:48.449Z+++ grep oxz_ntp
7592024-02-26T14:12:48.449Z++ pfexec svcs -z oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7 -Hostate oxide/ntp
7602024-02-26T14:12:48.449Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7612024-02-26T14:12:48.474Z++ true
7622024-02-26T14:12:48.474Z+ [[ '' != online ]]
7632024-02-26T14:12:48.474Z+ [[ 1 -gt 60 ]]
7642024-02-26T14:12:48.474Z+ sleep 1
7652024-02-26T14:12:49.455Z+ retry=2
7662024-02-26T14:12:49.484Z+++ zoneadm list -n
7672024-02-26T14:12:49.484Z+++ grep oxz_ntp
7682024-02-26T14:12:49.484Z++ pfexec svcs -z oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7 -Hostate oxide/ntp
7692024-02-26T14:12:49.484Z+ [[ disabled != online ]]
7702024-02-26T14:12:49.484Z+ [[ 2 -gt 60 ]]
7712024-02-26T14:12:49.484Z+ sleep 1
7722024-02-26T14:12:50.475Z+ retry=3
7732024-02-26T14:12:50.501Z+++ zoneadm list -n
7742024-02-26T14:12:50.501Z+++ grep oxz_ntp
7752024-02-26T14:12:50.501Z++ pfexec svcs -z oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7 -Hostate oxide/ntp
7762024-02-26T14:12:50.501Z+ [[ disabled != online ]]
7772024-02-26T14:12:50.501Z+ [[ 3 -gt 60 ]]
7782024-02-26T14:12:50.501Z+ sleep 1
7792024-02-26T14:12:51.497Z+ retry=4
7802024-02-26T14:12:51.523Z+++ zoneadm list -n
7812024-02-26T14:12:51.523Z+++ grep oxz_ntp
7822024-02-26T14:12:51.523Z++ pfexec svcs -z oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7 -Hostate oxide/ntp
7832024-02-26T14:12:51.523Z+ [[ disabled != online ]]
7842024-02-26T14:12:51.523Z+ [[ 4 -gt 60 ]]
7852024-02-26T14:12:51.523Z+ sleep 1
7862024-02-26T14:12:52.519Z+ retry=5
7872024-02-26T14:12:52.545Z+++ zoneadm list -n
7882024-02-26T14:12:52.545Z+++ grep oxz_ntp
7892024-02-26T14:12:52.545Z++ pfexec svcs -z oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7 -Hostate oxide/ntp
7902024-02-26T14:12:52.545Z+ [[ disabled != online ]]
7912024-02-26T14:12:52.545Z+ [[ 5 -gt 60 ]]
7922024-02-26T14:12:52.545Z+ sleep 1
7932024-02-26T14:12:53.541Z+ retry=6
7942024-02-26T14:12:53.566Z+++ zoneadm list -n
7952024-02-26T14:12:53.567Z+++ grep oxz_ntp
7962024-02-26T14:12:53.567Z++ pfexec svcs -z oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7 -Hostate oxide/ntp
7972024-02-26T14:12:53.567Z+ [[ disabled != online ]]
7982024-02-26T14:12:53.567Z+ [[ 6 -gt 60 ]]
7992024-02-26T14:12:53.567Z+ sleep 1
8002024-02-26T14:12:54.563Z+ retry=7
8012024-02-26T14:12:54.589Z+++ zoneadm list -n
8022024-02-26T14:12:54.589Z+++ grep oxz_ntp
8032024-02-26T14:12:54.589Z++ pfexec svcs -z oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7 -Hostate oxide/ntp
8042024-02-26T14:12:54.589Z+ [[ disabled != online ]]
8052024-02-26T14:12:54.589Z+ [[ 7 -gt 60 ]]
8062024-02-26T14:12:54.589Z+ sleep 1
8072024-02-26T14:12:55.586Z+ retry=8
8082024-02-26T14:12:55.611Z+++ zoneadm list -n
8092024-02-26T14:12:55.611Z+++ grep oxz_ntp
8102024-02-26T14:12:55.611Z++ pfexec svcs -z oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7 -Hostate oxide/ntp
8112024-02-26T14:12:55.611Z+ [[ disabled != online ]]
8122024-02-26T14:12:55.611Z+ [[ 8 -gt 60 ]]
8132024-02-26T14:12:55.611Z+ sleep 1
8142024-02-26T14:12:56.608Z+ retry=9
8152024-02-26T14:12:56.634Z+++ zoneadm list -n
8162024-02-26T14:12:56.634Z+++ grep oxz_ntp
8172024-02-26T14:12:56.634Z++ pfexec svcs -z oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7 -Hostate oxide/ntp
8182024-02-26T14:12:56.634Z+ [[ online != online ]]
8192024-02-26T14:12:56.634Z+ echo 'Waited for chrony: 9s'
8202024-02-26T14:12:56.634Z+ export RUST_BACKTRACE=1
8212024-02-26T14:12:56.634ZWaited for chrony: 9s
8222024-02-26T14:12:56.634Z+ RUST_BACKTRACE=1
8232024-02-26T14:12:56.634Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8242024-02-26T14:12:56.634Z++ ./tests/bootstrap
8252024-02-26T14:12:56.659ZUsing external DNS server at 10.151.2.164:53
8262024-02-26T14:12:56.687Z2024-02-26 14:12:56.672935322 UTC: attempting to log into API
8272024-02-26T14:13:11.699Z2024-02-26 14:13:11.682906784 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8282024-02-26T14:13:12.697Z2024-02-26 14:13:12.683016249 UTC: attempting to log into API
8292024-02-26T14:13:27.730Z2024-02-26 14:13:27.715699940 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8302024-02-26T14:13:28.731Z2024-02-26 14:13:28.716372709 UTC: attempting to log into API
 
8312024-02-26T14:13:36.710ZError: No response to keepalive - Terminating session
 
8322024-02-26T14:13:43.763Z2024-02-26 14:13:43.752911728 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8332024-02-26T14:13:44.765Z2024-02-26 14:13:44.753980944 UTC: attempting to log into API
 
8342024-02-26T14:13:46.516ZError: No response de-activating SOL payload
8352024-02-26T14:13:48.026Z[SOL Session operational. Use ~? for help]
 
8362024-02-26T14:13:59.800Z2024-02-26 14:13:59.785260740 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8372024-02-26T14:14:00.797Z2024-02-26 14:14:00.786277716 UTC: attempting to log into API
8382024-02-26T14:14:15.833Z2024-02-26 14:14:15.817834927 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8392024-02-26T14:14:16.831Z2024-02-26 14:14:16.818840122 UTC: attempting to log into API
8402024-02-26T14:14:34.096Z2024-02-26 14:14:31.852195154 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8412024-02-26T14:14:34.096Z2024-02-26 14:14:32.853642918 UTC: attempting to log into API
8422024-02-26T14:14:47.947Z2024-02-26 14:14:47.908979652 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8432024-02-26T14:14:48.934Z2024-02-26 14:14:48.917361866 UTC: attempting to log into API
8442024-02-26T14:15:06.443Z2024-02-26 14:15:04.611717720 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8452024-02-26T14:15:06.512Z2024-02-26 14:15:06.191262840 UTC: attempting to log into API
8462024-02-26T14:15:21.596Z2024-02-26 14:15:21.584538511 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8472024-02-26T14:15:22.597Z2024-02-26 14:15:22.585290392 UTC: attempting to log into API
8482024-02-26T14:15:37.629Z2024-02-26 14:15:37.617703096 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8492024-02-26T14:15:38.629Z2024-02-26 14:15:38.618534459 UTC: attempting to log into API
8502024-02-26T14:15:53.661Z2024-02-26 14:15:53.649390058 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8512024-02-26T14:15:54.661Z2024-02-26 14:15:54.650425684 UTC: attempting to log into API
8522024-02-26T14:16:09.695Z2024-02-26 14:16:09.682377513 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8532024-02-26T14:16:10.695Z2024-02-26 14:16:10.683360549 UTC: attempting to log into API
8542024-02-26T14:16:11.510Z2024-02-26 14:16:11.498925959 UTC: login succeeded
8552024-02-26T14:16:11.542Zwaiting for nexus to come up...
8562024-02-26T14:16:11.616Zcreating IP pool... 10.151.2.175 - 10.151.2.227
8572024-02-26T14:16:11.811Zsetting up quotas...
8582024-02-26T14:16:11.869Zensuring datasets are ready...
8592024-02-26T14:16:13.968Zlet's roll.
8602024-02-26T14:16:13.997Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
8612024-02-26T14:16:13.997ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
8622024-02-26T14:16:13.997ZOXIDE_TOKEN=oxide-token-c840537cfb80aca98a17b783b9ef9a4ccbdfa729'
8632024-02-26T14:16:13.997Z++ OXIDE_HOST=https://recovery.sys.oxide.test
8642024-02-26T14:16:13.997Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
8652024-02-26T14:16:13.997Z++ OXIDE_TOKEN=oxide-token-c840537cfb80aca98a17b783b9ef9a4ccbdfa729
8662024-02-26T14:16:13.998Z+ export OXIDE_HOST OXIDE_TOKEN
8672024-02-26T14:16:13.998Z+ retry=0
8682024-02-26T14:16:13.998Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8692024-02-26T14:16:14.024Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
8702024-02-26T14:16:14.024Z+ [[ 0 -gt 60 ]]
8712024-02-26T14:16:14.024Z+ sleep 1
8722024-02-26T14:16:15.017Z+ retry=1
8732024-02-26T14:16:15.017Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8742024-02-26T14:16:15.044Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
8752024-02-26T14:16:15.044Z+ [[ 1 -gt 60 ]]
8762024-02-26T14:16:15.044Z+ sleep 1
8772024-02-26T14:16:16.031Z+ retry=2
8782024-02-26T14:16:16.031Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8792024-02-26T14:16:16.057Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
8802024-02-26T14:16:16.057Z+ [[ 2 -gt 60 ]]
8812024-02-26T14:16:16.057Z+ sleep 1
8822024-02-26T14:16:17.047Z+ retry=3
8832024-02-26T14:16:17.048Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8842024-02-26T14:16:17.073Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
8852024-02-26T14:16:17.074Z+ [[ 3 -gt 60 ]]
8862024-02-26T14:16:17.074Z+ sleep 1
8872024-02-26T14:16:18.063Z+ retry=4
8882024-02-26T14:16:18.064Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8892024-02-26T14:16:18.089Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
8902024-02-26T14:16:18.089Z+ [[ 4 -gt 60 ]]
8912024-02-26T14:16:18.090Z+ sleep 1
8922024-02-26T14:16:19.080Z+ retry=5
8932024-02-26T14:16:19.081Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8942024-02-26T14:16:19.114Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
8952024-02-26T14:16:19.114Z+ [[ 5 -gt 60 ]]
8962024-02-26T14:16:19.114Z+ sleep 1
8972024-02-26T14:16:20.095Z+ retry=6
8982024-02-26T14:16:20.095Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8992024-02-26T14:16:20.121Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9002024-02-26T14:16:20.121Z+ [[ 6 -gt 60 ]]
9012024-02-26T14:16:20.121Z+ sleep 1
9022024-02-26T14:16:21.111Z+ retry=7
9032024-02-26T14:16:21.111Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9042024-02-26T14:16:21.136Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9052024-02-26T14:16:21.137Z+ [[ 7 -gt 60 ]]
9062024-02-26T14:16:21.137Z+ sleep 1
9072024-02-26T14:16:22.127Z+ retry=8
9082024-02-26T14:16:22.127Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9092024-02-26T14:16:22.152Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9102024-02-26T14:16:22.152Z+ [[ 8 -gt 60 ]]
9112024-02-26T14:16:22.152Z+ sleep 1
9122024-02-26T14:16:23.142Z+ retry=9
9132024-02-26T14:16:23.142Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9142024-02-26T14:16:23.168Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9152024-02-26T14:16:23.168Z+ [[ 9 -gt 60 ]]
9162024-02-26T14:16:23.168Z+ sleep 1
9172024-02-26T14:16:24.158Z+ retry=10
9182024-02-26T14:16:24.158Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9192024-02-26T14:16:24.184Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9202024-02-26T14:16:24.184Z+ [[ 10 -gt 60 ]]
9212024-02-26T14:16:24.185Z+ sleep 1
9222024-02-26T14:16:25.175Z+ retry=11
9232024-02-26T14:16:25.175Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9242024-02-26T14:16:25.212Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9252024-02-26T14:16:25.212Z+ [[ 11 -gt 60 ]]
9262024-02-26T14:16:25.212Z+ sleep 1
9272024-02-26T14:16:26.189Z+ retry=12
9282024-02-26T14:16:26.189Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9292024-02-26T14:16:26.215Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9302024-02-26T14:16:26.215Z+ [[ 12 -gt 60 ]]
9312024-02-26T14:16:26.216Z+ sleep 1
9322024-02-26T14:16:27.204Z+ retry=13
9332024-02-26T14:16:27.204Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9342024-02-26T14:16:27.231Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9352024-02-26T14:16:27.231Z+ [[ 13 -gt 60 ]]
9362024-02-26T14:16:27.231Z+ sleep 1
9372024-02-26T14:16:28.219Z+ retry=14
9382024-02-26T14:16:28.219Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9392024-02-26T14:16:28.259Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9402024-02-26T14:16:28.260Z+ [[ 14 -gt 60 ]]
9412024-02-26T14:16:28.260Z+ sleep 1
9422024-02-26T14:16:29.235Z+ retry=15
9432024-02-26T14:16:29.235Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9442024-02-26T14:16:29.262Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9452024-02-26T14:16:29.262Z+ [[ 15 -gt 60 ]]
9462024-02-26T14:16:29.262Z+ sleep 1
9472024-02-26T14:16:30.250Z+ retry=16
9482024-02-26T14:16:30.250Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9492024-02-26T14:16:30.275Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9502024-02-26T14:16:30.275Z+ [[ 16 -gt 60 ]]
9512024-02-26T14:16:30.275Z+ sleep 1
9522024-02-26T14:16:31.267Z+ retry=17
9532024-02-26T14:16:31.267Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9542024-02-26T14:16:31.296Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9552024-02-26T14:16:31.296Z+ [[ 17 -gt 60 ]]
9562024-02-26T14:16:31.296Z+ sleep 1
9572024-02-26T14:16:32.284Z+ retry=18
9582024-02-26T14:16:32.284Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9592024-02-26T14:16:32.311Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9602024-02-26T14:16:32.311Z+ [[ 18 -gt 60 ]]
9612024-02-26T14:16:32.311Z+ sleep 1
9622024-02-26T14:16:33.298Z+ retry=19
9632024-02-26T14:16:33.299Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9642024-02-26T14:16:33.325Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9652024-02-26T14:16:33.325Z+ [[ 19 -gt 60 ]]
9662024-02-26T14:16:33.325Z+ sleep 1
9672024-02-26T14:16:34.315Z+ retry=20
9682024-02-26T14:16:34.315Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9692024-02-26T14:16:34.342Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9702024-02-26T14:16:34.343Z+ [[ 20 -gt 60 ]]
9712024-02-26T14:16:34.343Z+ sleep 1
9722024-02-26T14:16:35.331Z+ retry=21
9732024-02-26T14:16:35.331Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9742024-02-26T14:16:35.357Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9752024-02-26T14:16:35.357Z+ [[ 21 -gt 60 ]]
9762024-02-26T14:16:35.357Z+ sleep 1
9772024-02-26T14:16:36.347Z+ retry=22
9782024-02-26T14:16:36.347Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9792024-02-26T14:16:36.374Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9802024-02-26T14:16:36.374Z+ [[ 22 -gt 60 ]]
9812024-02-26T14:16:36.374Z+ sleep 1
 
9822024-02-26T14:16:36.688ZError: No response to keepalive - Terminating session
 
9832024-02-26T14:16:37.364Z+ retry=23
9842024-02-26T14:16:37.364Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9852024-02-26T14:16:37.390Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9862024-02-26T14:16:37.390Z+ [[ 23 -gt 60 ]]
9872024-02-26T14:16:37.390Z+ sleep 1
9882024-02-26T14:16:38.378Z+ retry=24
9892024-02-26T14:16:38.378Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9902024-02-26T14:16:38.403Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9912024-02-26T14:16:38.403Z+ [[ 24 -gt 60 ]]
9922024-02-26T14:16:38.403Z+ sleep 1
9932024-02-26T14:16:39.394Z+ retry=25
9942024-02-26T14:16:39.394Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9952024-02-26T14:16:39.431Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9962024-02-26T14:16:39.431Z+ [[ 25 -gt 60 ]]
9972024-02-26T14:16:39.431Z+ sleep 1
9982024-02-26T14:16:40.409Z+ retry=26
9992024-02-26T14:16:40.409Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10002024-02-26T14:16:40.435Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10012024-02-26T14:16:40.435Z+ [[ 26 -gt 60 ]]
10022024-02-26T14:16:40.435Z+ sleep 1
10032024-02-26T14:16:41.424Z+ retry=27
10042024-02-26T14:16:41.424Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10052024-02-26T14:16:41.449Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10062024-02-26T14:16:41.449Z+ [[ 27 -gt 60 ]]
10072024-02-26T14:16:41.449Z+ sleep 1
10082024-02-26T14:16:42.439Z+ retry=28
10092024-02-26T14:16:42.440Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10102024-02-26T14:16:42.465Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10112024-02-26T14:16:42.466Z+ [[ 28 -gt 60 ]]
10122024-02-26T14:16:42.466Z+ sleep 1
10132024-02-26T14:16:43.454Z+ retry=29
10142024-02-26T14:16:43.454Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10152024-02-26T14:16:43.481Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10162024-02-26T14:16:43.481Z+ [[ 29 -gt 60 ]]
10172024-02-26T14:16:43.482Z+ sleep 1
10182024-02-26T14:16:44.470Z+ retry=30
10192024-02-26T14:16:44.471Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10202024-02-26T14:16:44.496Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10212024-02-26T14:16:44.497Z+ [[ 30 -gt 60 ]]
10222024-02-26T14:16:44.497Z+ sleep 1
10232024-02-26T14:16:45.486Z+ retry=31
10242024-02-26T14:16:45.486Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10252024-02-26T14:16:45.512Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10262024-02-26T14:16:45.512Z+ [[ 31 -gt 60 ]]
10272024-02-26T14:16:45.512Z+ sleep 1
10282024-02-26T14:16:46.501Z+ retry=32
10292024-02-26T14:16:46.501Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10302024-02-26T14:16:46.587Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
 
10312024-02-26T14:16:46.716ZError: No response de-activating SOL payload
 
10322024-02-26T14:16:46.967Z{"status":"ok"}success
10332024-02-26T14:16:46.968ZProject {
10342024-02-26T14:16:46.968Z description: "some images",
10352024-02-26T14:16:46.968Z id: fcab6d4a-87bd-4e73-a21e-8ddeb539fc0b,
10362024-02-26T14:16:46.968Z name: Name(
10372024-02-26T14:16:46.968Z "images",
10382024-02-26T14:16:46.969Z ),
10392024-02-26T14:16:46.969Z time_created: 2024-02-26T14:16:46.673517Z,
10402024-02-26T14:16:46.969Z time_modified: 2024-02-26T14:16:46.673517Z,
10412024-02-26T14:16:46.969Z}
10422024-02-26T14:16:46.994Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
 
10432024-02-26T14:16:47.955Z[SOL Session operational. Use ~? for help]
10442024-02-26T14:19:52.678ZError: No response to keepalive - Terminating session
10452024-02-26T14:20:02.724ZError: No response de-activating SOL payload
10462024-02-26T14:20:03.944Z[SOL Session operational. Use ~? for help]
 
10472024-02-26T14:22:11.663Zdone!
10482024-02-26T14:22:11.771Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
10492024-02-26T14:22:11.877Zsuccess
10502024-02-26T14:22:11.877ZImage {
10512024-02-26T14:22:11.877Z block_size: ByteCount(
10522024-02-26T14:22:11.904Z 512,
10532024-02-26T14:22:11.904Z ),
10542024-02-26T14:22:11.904Z description: "debian 11 original base image",
10552024-02-26T14:22:11.904Z digest: None,
10562024-02-26T14:22:11.904Z id: 28f79948-59ee-4916-ba4e-6f9c6046f0f0,
10572024-02-26T14:22:11.904Z name: Name(
10582024-02-26T14:22:11.904Z "debian11",
10592024-02-26T14:22:11.904Z ),
10602024-02-26T14:22:11.904Z os: "debian",
10612024-02-26T14:22:11.904Z project_id: None,
10622024-02-26T14:22:11.904Z size: ByteCount(
10632024-02-26T14:22:11.904Z 2147483648,
10642024-02-26T14:22:11.904Z ),
10652024-02-26T14:22:11.904Z time_created: 2024-02-26T14:22:11.623562Z,
10662024-02-26T14:22:11.904Z time_modified: 2024-02-26T14:22:11.859754Z,
10672024-02-26T14:22:11.904Z url: None,
10682024-02-26T14:22:11.904Z version: "11",
10692024-02-26T14:22:11.904Z}
10702024-02-26T14:22:11.904Z+ rm ./tests/bootstrap
10712024-02-26T14:22:11.955Z+ for test_bin in tests/*
10722024-02-26T14:22:11.955Z+ ./tests/bootstrap-22e5a97eec9bb5c6
10732024-02-26T14:22:12.018Z
10742024-02-26T14:22:12.018Zrunning 0 tests
10752024-02-26T14:22:12.042Z
10762024-02-26T14:22:12.042Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10772024-02-26T14:22:12.042Z
10782024-02-26T14:22:12.042Z+ for test_bin in tests/*
10792024-02-26T14:22:12.042Z+ ./tests/end_to_end_tests-25e7d53a8e782c84
10802024-02-26T14:22:12.366Z
10812024-02-26T14:22:12.366Zrunning 2 tests
10822024-02-26T14:22:12.671Ztest no_spoof::no_spoof_here ... ok
 
10832024-02-26T14:22:52.832ZError: No response to keepalive - Terminating session
10842024-02-26T14:23:02.728ZError: No response de-activating SOL payload
10852024-02-26T14:23:03.929Z[SOL Session operational. Use ~? for help]
 
10862024-02-26T14:23:05.622Ztest instance_launch::instance_launch ... ok
10872024-02-26T14:23:05.622Z
10882024-02-26T14:23:05.622Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 53.26s
10892024-02-26T14:23:05.622Z
10902024-02-26T14:23:05.647Z+ _exit_trap
10912024-02-26T14:23:05.648Z+ local status=0
10922024-02-26T14:23:05.648Z+ [[ 0 -eq 0 ]]
10932024-02-26T14:23:05.648Z+ exit 0
10942024-02-26T14:23:05.648Zprocess exited: duration 732184 ms, exit code 0
 
10952024-02-26T14:23:05.743Zfound 111 output files
10962024-02-26T14:23:05.801Zuploading: /var/svc/log/oxide-sled-agent:default.log (1865700 bytes)
10972024-02-26T14:23:06.518Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 1865700 -> 1866554
10982024-02-26T14:23:07.593Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10992024-02-26T14:23:07.593Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log (2010092 bytes)
11002024-02-26T14:23:08.382Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log" changed size mid upload: 2010092 -> 2020084
11012024-02-26T14:23:09.460Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log
11022024-02-26T14:23:09.460Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log.0 (627827 bytes)
11032024-02-26T14:23:10.770Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log.0
11042024-02-26T14:23:10.771Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log (0 bytes)
11052024-02-26T14:23:11.842Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log
11062024-02-26T14:23:11.842Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log.0 (5858 bytes)
11072024-02-26T14:23:12.951Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log.0
11082024-02-26T14:23:12.951Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log (17426 bytes)
11092024-02-26T14:23:14.125Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log
11102024-02-26T14:23:14.125Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log.0 (8075 bytes)
11112024-02-26T14:23:15.221Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log.0
11122024-02-26T14:23:15.221Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log (1281363 bytes)
11132024-02-26T14:23:16.801Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log" changed size mid upload: 1281363 -> 1302927
11142024-02-26T14:23:17.875Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log
11152024-02-26T14:23:17.875Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log.0 (313634 bytes)
11162024-02-26T14:23:19.596Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log.0
11172024-02-26T14:23:19.596Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log (754724 bytes)
11182024-02-26T14:23:20.583Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log" changed size mid upload: 754724 -> 773406
11192024-02-26T14:23:21.654Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log
11202024-02-26T14:23:21.654Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log.0 (182285 bytes)
11212024-02-26T14:23:22.951Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log.0
11222024-02-26T14:23:22.951Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log (0 bytes)
11232024-02-26T14:23:24.021Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log
11242024-02-26T14:23:24.021Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log.0 (1388 bytes)
11252024-02-26T14:23:25.125Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log.0
11262024-02-26T14:23:25.125Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log (236192 bytes)
11272024-02-26T14:23:25.423Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log" changed size mid upload: 236192 -> 246261
11282024-02-26T14:23:26.498Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log
11292024-02-26T14:23:26.498Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log.0 (67868 bytes)
11302024-02-26T14:23:27.685Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log.0
11312024-02-26T14:23:27.685Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log (0 bytes)
11322024-02-26T14:23:28.756Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log
11332024-02-26T14:23:28.756Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log.0 (617 bytes)
11342024-02-26T14:23:29.852Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log.0
11352024-02-26T14:23:29.852Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log (0 bytes)
11362024-02-26T14:23:30.925Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log
11372024-02-26T14:23:30.953Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log.0 (5759 bytes)
11382024-02-26T14:23:32.028Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log.0
11392024-02-26T14:23:32.055Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_b702bd51-be07-4d6e-bc95-e5b363d696d4/root/var/svc/log/oxide-clickhouse:default.log (978739 bytes)
11402024-02-26T14:23:33.252Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_b702bd51-be07-4d6e-bc95-e5b363d696d4/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 978739 -> 1027444
11412024-02-26T14:23:34.333Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_b702bd51-be07-4d6e-bc95-e5b363d696d4/root/var/svc/log/oxide-clickhouse:default.log
11422024-02-26T14:23:34.333Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_b702bd51-be07-4d6e-bc95-e5b363d696d4/root/var/svc/log/oxide-zone-network-setup:default.log (1713 bytes)
11432024-02-26T14:23:35.428Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_b702bd51-be07-4d6e-bc95-e5b363d696d4/root/var/svc/log/oxide-zone-network-setup:default.log
11442024-02-26T14:23:35.428Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a3119140-f60e-4ac0-a7f7-1921a58da3ed/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11452024-02-26T14:23:36.499Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a3119140-f60e-4ac0-a7f7-1921a58da3ed/root/var/svc/log/oxide-cockroachdb:default.log
11462024-02-26T14:23:36.523Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a3119140-f60e-4ac0-a7f7-1921a58da3ed/root/var/svc/log/oxide-cockroachdb:default.log.0 (4406 bytes)
11472024-02-26T14:23:37.605Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a3119140-f60e-4ac0-a7f7-1921a58da3ed/root/var/svc/log/oxide-cockroachdb:default.log.0
11482024-02-26T14:23:37.605Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a3119140-f60e-4ac0-a7f7-1921a58da3ed/root/var/svc/log/oxide-zone-network-setup:default.log (0 bytes)
11492024-02-26T14:23:38.727Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a3119140-f60e-4ac0-a7f7-1921a58da3ed/root/var/svc/log/oxide-zone-network-setup:default.log
11502024-02-26T14:23:38.727Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a3119140-f60e-4ac0-a7f7-1921a58da3ed/root/var/svc/log/oxide-zone-network-setup:default.log.0 (1711 bytes)
11512024-02-26T14:23:39.778Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a3119140-f60e-4ac0-a7f7-1921a58da3ed/root/var/svc/log/oxide-zone-network-setup:default.log.0
11522024-02-26T14:23:39.778Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_878c6e2d-47ce-4550-8d53-0b55eb9c0bb3/root/var/svc/log/oxide-crucible-agent:default.log (18046 bytes)
11532024-02-26T14:23:40.918Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_878c6e2d-47ce-4550-8d53-0b55eb9c0bb3/root/var/svc/log/oxide-crucible-agent:default.log
11542024-02-26T14:23:40.942Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_878c6e2d-47ce-4550-8d53-0b55eb9c0bb3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11552024-02-26T14:23:42.014Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_878c6e2d-47ce-4550-8d53-0b55eb9c0bb3/root/var/svc/log/oxide-crucible-downstairs:default.log
11562024-02-26T14:23:42.014Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_878c6e2d-47ce-4550-8d53-0b55eb9c0bb3/root/var/svc/log/oxide-crucible-downstairs:downstairs-eb2b3110-c94b-4a80-ae16-1fd1ac99a007.log (5694 bytes)
11572024-02-26T14:23:43.109Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_878c6e2d-47ce-4550-8d53-0b55eb9c0bb3/root/var/svc/log/oxide-crucible-downstairs:downstairs-eb2b3110-c94b-4a80-ae16-1fd1ac99a007.log
11582024-02-26T14:23:43.109Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_878c6e2d-47ce-4550-8d53-0b55eb9c0bb3/root/var/svc/log/oxide-crucible-downstairs:snapshot-eb2b3110-c94b-4a80-ae16-1fd1ac99a007-70bd21ab-457d-4e7d-be36-876d510cb8cf.log (5286 bytes)
11592024-02-26T14:23:44.206Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_878c6e2d-47ce-4550-8d53-0b55eb9c0bb3/root/var/svc/log/oxide-crucible-downstairs:snapshot-eb2b3110-c94b-4a80-ae16-1fd1ac99a007-70bd21ab-457d-4e7d-be36-876d510cb8cf.log
11602024-02-26T14:23:44.230Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_878c6e2d-47ce-4550-8d53-0b55eb9c0bb3/root/var/svc/log/oxide-zone-network-setup:default.log (1703 bytes)
11612024-02-26T14:23:45.307Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_878c6e2d-47ce-4550-8d53-0b55eb9c0bb3/root/var/svc/log/oxide-zone-network-setup:default.log
11622024-02-26T14:23:45.331Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_48ad4bca-df24-4d26-b150-a76ad7ccb695/root/var/svc/log/oxide-external_dns:default.log (30638 bytes)
11632024-02-26T14:23:46.477Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_48ad4bca-df24-4d26-b150-a76ad7ccb695/root/var/svc/log/oxide-external_dns:default.log
11642024-02-26T14:23:46.477Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_48ad4bca-df24-4d26-b150-a76ad7ccb695/root/var/svc/log/oxide-opte-interface-setup:default.log (1158 bytes)
11652024-02-26T14:23:47.575Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_48ad4bca-df24-4d26-b150-a76ad7ccb695/root/var/svc/log/oxide-opte-interface-setup:default.log
11662024-02-26T14:23:47.575Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_48ad4bca-df24-4d26-b150-a76ad7ccb695/root/var/svc/log/oxide-zone-network-setup:default.log (1725 bytes)
11672024-02-26T14:23:48.670Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_48ad4bca-df24-4d26-b150-a76ad7ccb695/root/var/svc/log/oxide-zone-network-setup:default.log
11682024-02-26T14:23:48.670Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_3e85505b-ff86-4823-ad10-5f9c28b3c026/root/var/svc/log/oxide-internal_dns:default.log (32581 bytes)
11692024-02-26T14:23:48.788Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_3e85505b-ff86-4823-ad10-5f9c28b3c026/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 32581 -> 36334
11702024-02-26T14:23:49.862Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_3e85505b-ff86-4823-ad10-5f9c28b3c026/root/var/svc/log/oxide-internal_dns:default.log
11712024-02-26T14:23:49.862Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_3e85505b-ff86-4823-ad10-5f9c28b3c026/root/var/svc/log/oxide-internal_dns:default.log.0 (84861 bytes)
11722024-02-26T14:23:51.100Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_3e85505b-ff86-4823-ad10-5f9c28b3c026/root/var/svc/log/oxide-internal_dns:default.log.0
11732024-02-26T14:23:51.100Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7/root/var/svc/log/oxide-ntp:default.log (67 bytes)
11742024-02-26T14:23:52.196Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7/root/var/svc/log/oxide-ntp:default.log
11752024-02-26T14:23:52.196Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7/root/var/svc/log/oxide-ntp:default.log.0 (916 bytes)
11762024-02-26T14:23:53.302Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7/root/var/svc/log/oxide-ntp:default.log.0
11772024-02-26T14:23:53.303Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b7bdbc49-7e2d-4c9a-ae30-5d6da79e518d/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11782024-02-26T14:23:54.404Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b7bdbc49-7e2d-4c9a-ae30-5d6da79e518d/root/var/svc/log/oxide-cockroachdb:default.log
11792024-02-26T14:23:54.404Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b7bdbc49-7e2d-4c9a-ae30-5d6da79e518d/root/var/svc/log/oxide-cockroachdb:default.log.0 (4405 bytes)
11802024-02-26T14:23:55.499Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b7bdbc49-7e2d-4c9a-ae30-5d6da79e518d/root/var/svc/log/oxide-cockroachdb:default.log.0
11812024-02-26T14:23:55.499Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b7bdbc49-7e2d-4c9a-ae30-5d6da79e518d/root/var/svc/log/oxide-zone-network-setup:default.log (0 bytes)
11822024-02-26T14:23:56.572Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b7bdbc49-7e2d-4c9a-ae30-5d6da79e518d/root/var/svc/log/oxide-zone-network-setup:default.log
11832024-02-26T14:23:56.572Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b7bdbc49-7e2d-4c9a-ae30-5d6da79e518d/root/var/svc/log/oxide-zone-network-setup:default.log.0 (1711 bytes)
11842024-02-26T14:23:57.666Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b7bdbc49-7e2d-4c9a-ae30-5d6da79e518d/root/var/svc/log/oxide-zone-network-setup:default.log.0
11852024-02-26T14:23:57.666Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c65cd3-a9ff-4018-b75b-75d614672435/root/var/svc/log/oxide-crucible-agent:default.log (33961 bytes)
11862024-02-26T14:23:58.833Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c65cd3-a9ff-4018-b75b-75d614672435/root/var/svc/log/oxide-crucible-agent:default.log
11872024-02-26T14:23:58.833Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c65cd3-a9ff-4018-b75b-75d614672435/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11882024-02-26T14:23:59.938Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c65cd3-a9ff-4018-b75b-75d614672435/root/var/svc/log/oxide-crucible-downstairs:default.log
11892024-02-26T14:23:59.938Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c65cd3-a9ff-4018-b75b-75d614672435/root/var/svc/log/oxide-crucible-downstairs:downstairs-6d939e3d-e0d9-4a08-930b-433ec2fc2c41.log (5592 bytes)
11902024-02-26T14:24:01.066Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c65cd3-a9ff-4018-b75b-75d614672435/root/var/svc/log/oxide-crucible-downstairs:downstairs-6d939e3d-e0d9-4a08-930b-433ec2fc2c41.log
11912024-02-26T14:24:01.067Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c65cd3-a9ff-4018-b75b-75d614672435/root/var/svc/log/oxide-crucible-downstairs:downstairs-bb2a1408-0adc-42e9-8cca-7d538307dd05.log (5695 bytes)
11922024-02-26T14:24:02.168Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c65cd3-a9ff-4018-b75b-75d614672435/root/var/svc/log/oxide-crucible-downstairs:downstairs-bb2a1408-0adc-42e9-8cca-7d538307dd05.log
11932024-02-26T14:24:02.168Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c65cd3-a9ff-4018-b75b-75d614672435/root/var/svc/log/oxide-crucible-downstairs:snapshot-bb2a1408-0adc-42e9-8cca-7d538307dd05-70bd21ab-457d-4e7d-be36-876d510cb8cf.log (5284 bytes)
11942024-02-26T14:24:03.266Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c65cd3-a9ff-4018-b75b-75d614672435/root/var/svc/log/oxide-crucible-downstairs:snapshot-bb2a1408-0adc-42e9-8cca-7d538307dd05-70bd21ab-457d-4e7d-be36-876d510cb8cf.log
11952024-02-26T14:24:03.266Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c65cd3-a9ff-4018-b75b-75d614672435/root/var/svc/log/oxide-zone-network-setup:default.log (1704 bytes)
11962024-02-26T14:24:04.382Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41c65cd3-a9ff-4018-b75b-75d614672435/root/var/svc/log/oxide-zone-network-setup:default.log
11972024-02-26T14:24:04.382Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_0b7196d9-6e1c-464d-8b59-51b50861c571/root/var/svc/log/oxide-external_dns:default.log (30637 bytes)
11982024-02-26T14:24:04.612Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_0b7196d9-6e1c-464d-8b59-51b50861c571/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 30637 -> 34415
11992024-02-26T14:24:05.682Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_0b7196d9-6e1c-464d-8b59-51b50861c571/root/var/svc/log/oxide-external_dns:default.log
12002024-02-26T14:24:05.683Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_0b7196d9-6e1c-464d-8b59-51b50861c571/root/var/svc/log/oxide-opte-interface-setup:default.log (1158 bytes)
12012024-02-26T14:24:06.782Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_0b7196d9-6e1c-464d-8b59-51b50861c571/root/var/svc/log/oxide-opte-interface-setup:default.log
12022024-02-26T14:24:06.783Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_0b7196d9-6e1c-464d-8b59-51b50861c571/root/var/svc/log/oxide-zone-network-setup:default.log (1726 bytes)
12032024-02-26T14:24:07.877Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_0b7196d9-6e1c-464d-8b59-51b50861c571/root/var/svc/log/oxide-zone-network-setup:default.log
12042024-02-26T14:24:07.877Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4004b87f-b74c-4032-9e94-ba287665c3c9/root/var/svc/log/oxide-internal_dns:default.log (32567 bytes)
12052024-02-26T14:24:07.993Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4004b87f-b74c-4032-9e94-ba287665c3c9/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 32567 -> 36323
12062024-02-26T14:24:09.074Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4004b87f-b74c-4032-9e94-ba287665c3c9/root/var/svc/log/oxide-internal_dns:default.log
12072024-02-26T14:24:09.075Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4004b87f-b74c-4032-9e94-ba287665c3c9/root/var/svc/log/oxide-internal_dns:default.log.0 (88108 bytes)
12082024-02-26T14:24:10.279Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4004b87f-b74c-4032-9e94-ba287665c3c9/root/var/svc/log/oxide-internal_dns:default.log.0
12092024-02-26T14:24:10.280Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_8c5e1792-bf9d-47e0-9cb7-8c6f1653e827/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12102024-02-26T14:24:11.350Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_8c5e1792-bf9d-47e0-9cb7-8c6f1653e827/root/var/svc/log/oxide-cockroachdb:default.log
12112024-02-26T14:24:11.350Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_8c5e1792-bf9d-47e0-9cb7-8c6f1653e827/root/var/svc/log/oxide-cockroachdb:default.log.0 (4407 bytes)
12122024-02-26T14:24:12.448Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_8c5e1792-bf9d-47e0-9cb7-8c6f1653e827/root/var/svc/log/oxide-cockroachdb:default.log.0
12132024-02-26T14:24:12.448Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_8c5e1792-bf9d-47e0-9cb7-8c6f1653e827/root/var/svc/log/oxide-zone-network-setup:default.log (0 bytes)
12142024-02-26T14:24:13.527Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_8c5e1792-bf9d-47e0-9cb7-8c6f1653e827/root/var/svc/log/oxide-zone-network-setup:default.log
12152024-02-26T14:24:13.527Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_8c5e1792-bf9d-47e0-9cb7-8c6f1653e827/root/var/svc/log/oxide-zone-network-setup:default.log.0 (1711 bytes)
12162024-02-26T14:24:14.632Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_8c5e1792-bf9d-47e0-9cb7-8c6f1653e827/root/var/svc/log/oxide-zone-network-setup:default.log.0
12172024-02-26T14:24:14.632Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7279c449-e435-466d-8ef2-781c1c3c566b/root/var/svc/log/oxide-crucible-agent:default.log (17506 bytes)
12182024-02-26T14:24:15.773Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7279c449-e435-466d-8ef2-781c1c3c566b/root/var/svc/log/oxide-crucible-agent:default.log
12192024-02-26T14:24:15.773Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7279c449-e435-466d-8ef2-781c1c3c566b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12202024-02-26T14:24:16.870Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7279c449-e435-466d-8ef2-781c1c3c566b/root/var/svc/log/oxide-crucible-downstairs:default.log
12212024-02-26T14:24:16.870Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7279c449-e435-466d-8ef2-781c1c3c566b/root/var/svc/log/oxide-crucible-downstairs:downstairs-7d4cb344-e9eb-4ea8-92a5-7565e1b704de.log (3006 bytes)
12222024-02-26T14:24:17.966Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7279c449-e435-466d-8ef2-781c1c3c566b/root/var/svc/log/oxide-crucible-downstairs:downstairs-7d4cb344-e9eb-4ea8-92a5-7565e1b704de.log
12232024-02-26T14:24:17.966Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7279c449-e435-466d-8ef2-781c1c3c566b/root/var/svc/log/oxide-crucible-downstairs:downstairs-df08ee3f-6e1e-4e64-80c3-6c0777acc366.log (3125 bytes)
12242024-02-26T14:24:19.063Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7279c449-e435-466d-8ef2-781c1c3c566b/root/var/svc/log/oxide-crucible-downstairs:downstairs-df08ee3f-6e1e-4e64-80c3-6c0777acc366.log
12252024-02-26T14:24:19.063Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7279c449-e435-466d-8ef2-781c1c3c566b/root/var/svc/log/oxide-zone-network-setup:default.log (1703 bytes)
12262024-02-26T14:24:20.611Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7279c449-e435-466d-8ef2-781c1c3c566b/root/var/svc/log/oxide-zone-network-setup:default.log
12272024-02-26T14:24:20.611Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_eec58ed4-9191-44d9-adcb-46c67e4fc77d/root/var/svc/log/oxide-internal_dns:default.log (32573 bytes)
12282024-02-26T14:24:20.726Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_eec58ed4-9191-44d9-adcb-46c67e4fc77d/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 32573 -> 36326
12292024-02-26T14:24:21.799Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_eec58ed4-9191-44d9-adcb-46c67e4fc77d/root/var/svc/log/oxide-internal_dns:default.log
12302024-02-26T14:24:21.799Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_eec58ed4-9191-44d9-adcb-46c67e4fc77d/root/var/svc/log/oxide-internal_dns:default.log.0 (84532 bytes)
12312024-02-26T14:24:23.010Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_eec58ed4-9191-44d9-adcb-46c67e4fc77d/root/var/svc/log/oxide-internal_dns:default.log.0
12322024-02-26T14:24:23.010Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_51a6bf59-61ff-491b-af37-2dfd05ea6440/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12332024-02-26T14:24:24.089Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_51a6bf59-61ff-491b-af37-2dfd05ea6440/root/var/svc/log/oxide-cockroachdb:default.log
12342024-02-26T14:24:24.089Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_51a6bf59-61ff-491b-af37-2dfd05ea6440/root/var/svc/log/oxide-cockroachdb:default.log.0 (4381 bytes)
12352024-02-26T14:24:25.193Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_51a6bf59-61ff-491b-af37-2dfd05ea6440/root/var/svc/log/oxide-cockroachdb:default.log.0
12362024-02-26T14:24:25.193Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_51a6bf59-61ff-491b-af37-2dfd05ea6440/root/var/svc/log/oxide-zone-network-setup:default.log (0 bytes)
12372024-02-26T14:24:26.276Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_51a6bf59-61ff-491b-af37-2dfd05ea6440/root/var/svc/log/oxide-zone-network-setup:default.log
12382024-02-26T14:24:26.276Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_51a6bf59-61ff-491b-af37-2dfd05ea6440/root/var/svc/log/oxide-zone-network-setup:default.log.0 (1711 bytes)
12392024-02-26T14:24:27.374Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_51a6bf59-61ff-491b-af37-2dfd05ea6440/root/var/svc/log/oxide-zone-network-setup:default.log.0
12402024-02-26T14:24:27.374Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b6c90105-fbc3-42d0-960a-63e7c9025138/root/var/svc/log/oxide-crucible-agent:default.log (12008 bytes)
12412024-02-26T14:24:28.514Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b6c90105-fbc3-42d0-960a-63e7c9025138/root/var/svc/log/oxide-crucible-agent:default.log
12422024-02-26T14:24:28.541Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b6c90105-fbc3-42d0-960a-63e7c9025138/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12432024-02-26T14:24:29.618Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b6c90105-fbc3-42d0-960a-63e7c9025138/root/var/svc/log/oxide-crucible-downstairs:default.log
12442024-02-26T14:24:29.643Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b6c90105-fbc3-42d0-960a-63e7c9025138/root/var/svc/log/oxide-crucible-downstairs:downstairs-0ab7663a-eb89-4da2-8ef4-270bc3503aa4.log (5588 bytes)
12452024-02-26T14:24:30.715Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b6c90105-fbc3-42d0-960a-63e7c9025138/root/var/svc/log/oxide-crucible-downstairs:downstairs-0ab7663a-eb89-4da2-8ef4-270bc3503aa4.log
12462024-02-26T14:24:30.715Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b6c90105-fbc3-42d0-960a-63e7c9025138/root/var/svc/log/oxide-zone-network-setup:default.log (1704 bytes)
12472024-02-26T14:24:31.813Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b6c90105-fbc3-42d0-960a-63e7c9025138/root/var/svc/log/oxide-zone-network-setup:default.log
12482024-02-26T14:24:31.813Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_cdbcf2db-f02f-49d1-b6be-9a8125034dbc/root/var/svc/log/oxide-crucible-pantry:default.log (783 bytes)
12492024-02-26T14:24:32.912Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_cdbcf2db-f02f-49d1-b6be-9a8125034dbc/root/var/svc/log/oxide-crucible-pantry:default.log
12502024-02-26T14:24:32.912Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_cdbcf2db-f02f-49d1-b6be-9a8125034dbc/root/var/svc/log/oxide-zone-network-setup:default.log (1743 bytes)
12512024-02-26T14:24:34.042Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_cdbcf2db-f02f-49d1-b6be-9a8125034dbc/root/var/svc/log/oxide-zone-network-setup:default.log
12522024-02-26T14:24:34.043Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_37e97d92-dac4-45ef-994e-e60f121531a7/root/var/svc/log/oxide-nexus:default.log (34757711 bytes)
12532024-02-26T14:24:48.851Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_37e97d92-dac4-45ef-994e-e60f121531a7/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 34757711 -> 34991687
12542024-02-26T14:24:49.921Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_37e97d92-dac4-45ef-994e-e60f121531a7/root/var/svc/log/oxide-nexus:default.log
12552024-02-26T14:24:49.921Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_d0813305-f45c-4d8d-9b73-bd94436fc777/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12562024-02-26T14:24:50.997Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_d0813305-f45c-4d8d-9b73-bd94436fc777/root/var/svc/log/oxide-cockroachdb:default.log
12572024-02-26T14:24:50.997Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_d0813305-f45c-4d8d-9b73-bd94436fc777/root/var/svc/log/oxide-cockroachdb:default.log.0 (4407 bytes)
12582024-02-26T14:24:52.094Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_d0813305-f45c-4d8d-9b73-bd94436fc777/root/var/svc/log/oxide-cockroachdb:default.log.0
12592024-02-26T14:24:52.124Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_d0813305-f45c-4d8d-9b73-bd94436fc777/root/var/svc/log/oxide-zone-network-setup:default.log (0 bytes)
12602024-02-26T14:24:53.172Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_d0813305-f45c-4d8d-9b73-bd94436fc777/root/var/svc/log/oxide-zone-network-setup:default.log
12612024-02-26T14:24:53.172Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_d0813305-f45c-4d8d-9b73-bd94436fc777/root/var/svc/log/oxide-zone-network-setup:default.log.0 (1711 bytes)
12622024-02-26T14:24:54.266Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_d0813305-f45c-4d8d-9b73-bd94436fc777/root/var/svc/log/oxide-zone-network-setup:default.log.0
12632024-02-26T14:24:54.290Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa3da73-3b30-4c75-8f0f-7f5b5b5ad47e/root/var/svc/log/oxide-crucible-agent:default.log (22723 bytes)
12642024-02-26T14:24:55.407Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa3da73-3b30-4c75-8f0f-7f5b5b5ad47e/root/var/svc/log/oxide-crucible-agent:default.log
12652024-02-26T14:24:55.407Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa3da73-3b30-4c75-8f0f-7f5b5b5ad47e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12662024-02-26T14:24:56.512Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa3da73-3b30-4c75-8f0f-7f5b5b5ad47e/root/var/svc/log/oxide-crucible-downstairs:default.log
12672024-02-26T14:24:56.512Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa3da73-3b30-4c75-8f0f-7f5b5b5ad47e/root/var/svc/log/oxide-crucible-downstairs:downstairs-1e7cf599-8c63-49e5-8873-f73652fce458.log (3126 bytes)
12682024-02-26T14:24:57.607Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa3da73-3b30-4c75-8f0f-7f5b5b5ad47e/root/var/svc/log/oxide-crucible-downstairs:downstairs-1e7cf599-8c63-49e5-8873-f73652fce458.log
12692024-02-26T14:24:57.607Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa3da73-3b30-4c75-8f0f-7f5b5b5ad47e/root/var/svc/log/oxide-crucible-downstairs:downstairs-39e657c0-64dc-4f5d-a8ef-58dd3ca95008.log (5593 bytes)
12702024-02-26T14:24:58.727Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa3da73-3b30-4c75-8f0f-7f5b5b5ad47e/root/var/svc/log/oxide-crucible-downstairs:downstairs-39e657c0-64dc-4f5d-a8ef-58dd3ca95008.log
12712024-02-26T14:24:58.727Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa3da73-3b30-4c75-8f0f-7f5b5b5ad47e/root/var/svc/log/oxide-zone-network-setup:default.log (1704 bytes)
12722024-02-26T14:24:59.795Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0aa3da73-3b30-4c75-8f0f-7f5b5b5ad47e/root/var/svc/log/oxide-zone-network-setup:default.log
12732024-02-26T14:24:59.795Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_37d28ef3-41e1-458c-a3d8-51477fad056e/root/var/svc/log/oxide-nexus:default.log (5273036 bytes)
12742024-02-26T14:25:01.253Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_37d28ef3-41e1-458c-a3d8-51477fad056e/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 5273036 -> 5676797
12752024-02-26T14:25:02.324Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_37d28ef3-41e1-458c-a3d8-51477fad056e/root/var/svc/log/oxide-nexus:default.log
12762024-02-26T14:25:02.324Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cdabffe4-769c-4b02-b55e-25255583376f/root/var/svc/log/oxide-crucible-agent:default.log (1327 bytes)
12772024-02-26T14:25:03.422Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cdabffe4-769c-4b02-b55e-25255583376f/root/var/svc/log/oxide-crucible-agent:default.log
12782024-02-26T14:25:03.422Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cdabffe4-769c-4b02-b55e-25255583376f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12792024-02-26T14:25:04.527Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cdabffe4-769c-4b02-b55e-25255583376f/root/var/svc/log/oxide-crucible-downstairs:default.log
12802024-02-26T14:25:04.527Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cdabffe4-769c-4b02-b55e-25255583376f/root/var/svc/log/oxide-zone-network-setup:default.log (1704 bytes)
12812024-02-26T14:25:05.633Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cdabffe4-769c-4b02-b55e-25255583376f/root/var/svc/log/oxide-zone-network-setup:default.log
12822024-02-26T14:25:05.633Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_faa45480-e10d-48eb-be2b-17da0373f2b7/root/var/svc/log/oxide-crucible-agent:default.log (7686 bytes)
12832024-02-26T14:25:06.751Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_faa45480-e10d-48eb-be2b-17da0373f2b7/root/var/svc/log/oxide-crucible-agent:default.log
12842024-02-26T14:25:06.751Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_faa45480-e10d-48eb-be2b-17da0373f2b7/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
12852024-02-26T14:25:07.849Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_faa45480-e10d-48eb-be2b-17da0373f2b7/root/var/svc/log/oxide-crucible-downstairs:default.log
12862024-02-26T14:25:07.849Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_faa45480-e10d-48eb-be2b-17da0373f2b7/root/var/svc/log/oxide-crucible-downstairs:downstairs-d05f2282-f24f-4c32-a4f1-614ad6c91905.log (3006 bytes)
12872024-02-26T14:25:08.944Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_faa45480-e10d-48eb-be2b-17da0373f2b7/root/var/svc/log/oxide-crucible-downstairs:downstairs-d05f2282-f24f-4c32-a4f1-614ad6c91905.log
12882024-02-26T14:25:08.944Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_faa45480-e10d-48eb-be2b-17da0373f2b7/root/var/svc/log/oxide-zone-network-setup:default.log (1704 bytes)
12892024-02-26T14:25:10.039Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_faa45480-e10d-48eb-be2b-17da0373f2b7/root/var/svc/log/oxide-zone-network-setup:default.log
12902024-02-26T14:25:10.039Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_41355962-ac7d-4e04-870d-8289638ef330/root/var/svc/log/oxide-crucible-pantry:default.log (1214057 bytes)
12912024-02-26T14:25:11.393Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_41355962-ac7d-4e04-870d-8289638ef330/root/var/svc/log/oxide-crucible-pantry:default.log
12922024-02-26T14:25:11.420Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_41355962-ac7d-4e04-870d-8289638ef330/root/var/svc/log/oxide-zone-network-setup:default.log (1743 bytes)
12932024-02-26T14:25:12.487Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_41355962-ac7d-4e04-870d-8289638ef330/root/var/svc/log/oxide-zone-network-setup:default.log
12942024-02-26T14:25:12.487Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_c4e156ab-c1ae-4342-b581-84d6d44e005a/root/var/svc/log/oxide-oximeter:default.log (295625 bytes)
12952024-02-26T14:25:12.714Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_c4e156ab-c1ae-4342-b581-84d6d44e005a/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 295625 -> 374380
12962024-02-26T14:25:13.783Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_c4e156ab-c1ae-4342-b581-84d6d44e005a/root/var/svc/log/oxide-oximeter:default.log
12972024-02-26T14:25:13.784Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_c4e156ab-c1ae-4342-b581-84d6d44e005a/root/var/svc/log/oxide-zone-network-setup:default.log (1702 bytes)
12982024-02-26T14:25:14.880Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_c4e156ab-c1ae-4342-b581-84d6d44e005a/root/var/svc/log/oxide-zone-network-setup:default.log
12992024-02-26T14:25:14.880Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_940597d1-79b0-4ba7-bd28-8513ddedab6b/root/var/svc/log/oxide-crucible-agent:default.log (26569 bytes)
13002024-02-26T14:25:16.021Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_940597d1-79b0-4ba7-bd28-8513ddedab6b/root/var/svc/log/oxide-crucible-agent:default.log
13012024-02-26T14:25:16.021Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_940597d1-79b0-4ba7-bd28-8513ddedab6b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
13022024-02-26T14:25:17.128Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_940597d1-79b0-4ba7-bd28-8513ddedab6b/root/var/svc/log/oxide-crucible-downstairs:default.log
13032024-02-26T14:25:17.128Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_940597d1-79b0-4ba7-bd28-8513ddedab6b/root/var/svc/log/oxide-crucible-downstairs:downstairs-5323b831-15d0-4f17-93cf-71a23feb45b0.log (3003 bytes)
13042024-02-26T14:25:18.225Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_940597d1-79b0-4ba7-bd28-8513ddedab6b/root/var/svc/log/oxide-crucible-downstairs:downstairs-5323b831-15d0-4f17-93cf-71a23feb45b0.log
13052024-02-26T14:25:18.226Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_940597d1-79b0-4ba7-bd28-8513ddedab6b/root/var/svc/log/oxide-crucible-downstairs:downstairs-536ae11c-af48-442d-b5bb-7f6cd3f97604.log (5695 bytes)
13062024-02-26T14:25:19.315Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_940597d1-79b0-4ba7-bd28-8513ddedab6b/root/var/svc/log/oxide-crucible-downstairs:downstairs-536ae11c-af48-442d-b5bb-7f6cd3f97604.log
13072024-02-26T14:25:19.315Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_940597d1-79b0-4ba7-bd28-8513ddedab6b/root/var/svc/log/oxide-crucible-downstairs:snapshot-536ae11c-af48-442d-b5bb-7f6cd3f97604-70bd21ab-457d-4e7d-be36-876d510cb8cf.log (5283 bytes)
13082024-02-26T14:25:20.409Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_940597d1-79b0-4ba7-bd28-8513ddedab6b/root/var/svc/log/oxide-crucible-downstairs:snapshot-536ae11c-af48-442d-b5bb-7f6cd3f97604-70bd21ab-457d-4e7d-be36-876d510cb8cf.log
13092024-02-26T14:25:20.409Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_940597d1-79b0-4ba7-bd28-8513ddedab6b/root/var/svc/log/oxide-zone-network-setup:default.log (1704 bytes)
13102024-02-26T14:25:21.505Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_940597d1-79b0-4ba7-bd28-8513ddedab6b/root/var/svc/log/oxide-zone-network-setup:default.log
13112024-02-26T14:25:21.505Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_f0970254-d6d4-4596-b14c-20641e09df17/root/var/svc/log/oxide-crucible-pantry:default.log (783 bytes)
13122024-02-26T14:25:22.619Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_f0970254-d6d4-4596-b14c-20641e09df17/root/var/svc/log/oxide-crucible-pantry:default.log
13132024-02-26T14:25:22.619Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_f0970254-d6d4-4596-b14c-20641e09df17/root/var/svc/log/oxide-zone-network-setup:default.log (1746 bytes)
13142024-02-26T14:25:23.713Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_f0970254-d6d4-4596-b14c-20641e09df17/root/var/svc/log/oxide-zone-network-setup:default.log
13152024-02-26T14:25:23.713Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5375fdbc-7e5d-4015-92c5-659c92d1c82c/root/var/svc/log/oxide-nexus:default.log (1296171 bytes)
13162024-02-26T14:25:24.563Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5375fdbc-7e5d-4015-92c5-659c92d1c82c/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1296171 -> 1550046
13172024-02-26T14:25:25.641Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5375fdbc-7e5d-4015-92c5-659c92d1c82c/root/var/svc/log/oxide-nexus:default.log
13182024-02-26T14:25:25.641Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c6acb1b0-18b8-4efb-b7ea-921bc52077a5/root/var/svc/log/oxide-crucible-agent:default.log (12037 bytes)
13192024-02-26T14:25:26.761Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c6acb1b0-18b8-4efb-b7ea-921bc52077a5/root/var/svc/log/oxide-crucible-agent:default.log
13202024-02-26T14:25:26.761Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c6acb1b0-18b8-4efb-b7ea-921bc52077a5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
13212024-02-26T14:25:27.859Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c6acb1b0-18b8-4efb-b7ea-921bc52077a5/root/var/svc/log/oxide-crucible-downstairs:default.log
13222024-02-26T14:25:27.883Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c6acb1b0-18b8-4efb-b7ea-921bc52077a5/root/var/svc/log/oxide-crucible-downstairs:downstairs-f5c7a7c8-79ee-47bb-9d9f-36f8cb0c42ef.log (3126 bytes)
13232024-02-26T14:25:28.957Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c6acb1b0-18b8-4efb-b7ea-921bc52077a5/root/var/svc/log/oxide-crucible-downstairs:downstairs-f5c7a7c8-79ee-47bb-9d9f-36f8cb0c42ef.log
13242024-02-26T14:25:28.958Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c6acb1b0-18b8-4efb-b7ea-921bc52077a5/root/var/svc/log/oxide-zone-network-setup:default.log (1703 bytes)
13252024-02-26T14:25:30.057Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c6acb1b0-18b8-4efb-b7ea-921bc52077a5/root/var/svc/log/oxide-zone-network-setup:default.log
13262024-02-26T14:25:30.057Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7/root/var/log/chrony/measurements.log (22331 bytes)
13272024-02-26T14:25:30.149Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7/root/var/log/chrony/measurements.log" changed size mid upload: 22331 -> 27400
13282024-02-26T14:25:31.232Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7/root/var/log/chrony/measurements.log
13292024-02-26T14:25:31.232Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7/root/var/log/chrony/statistics.log (15093 bytes)
13302024-02-26T14:25:31.330Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7/root/var/log/chrony/statistics.log" changed size mid upload: 15093 -> 18954
13312024-02-26T14:25:32.401Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7/root/var/log/chrony/statistics.log
13322024-02-26T14:25:32.401Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7/root/var/log/chrony/tracking.log (1852 bytes)
13332024-02-26T14:25:32.462Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7/root/var/log/chrony/tracking.log" changed size mid upload: 1852 -> 2116
13342024-02-26T14:25:33.540Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38c83a19-4122-4643-a6c1-4b27a8cf5fc7/root/var/log/chrony/tracking.log